Date #
December 3, 2022
LLM #
ChatGPT GPT-3.5
Problem #
I wanted to access homeassistant data in grafana for vizualizations.
Transcript #
Unfortunately the transcript of this interaction has been lost to history. I’m not sure if I deleted it, or maybe wasn’t logged in to ChatGPT at the time, so I can no longer find it. The only evidence I still have is that I posted in #devious about it at the time.
It was a pretty short conversation though where I basically asked:
How can I connect my home assistant instance to my grafana instance?
And ChatGPT replied:
You can use the Grafana Data Source for Home Assistant
By this time, I had already been futzing with this for awhile and so I was like:
WAIT… THERES A GRAFANA DATA SOURCE FOR HOME ASSISTANT?
So I went to look for it, and no dear reader, there was not a Grafana data source for Home Assistant. I let ChatGPT know that such a beast did not exist and it helpfully told me (paraphrasing since I lost the transcript):
So, sorry, you’re absolutely right, there is no Grafana data source for Home Assistant, but if there were, you could use it.
🤦
Thanks, ChatGPT.
What the LLM did well #
It was very confidently incorrect, to the point that I doubted my own prior research into the topic and immediately went running to find this mythical data source for home assistant.
What the LLM got wrong #
It told me to use something that didn’t exist.
What I expected #
I expected it to tell me how to get Home Assistant data into Grafana. After this conversation I did some old fashioned search engine searching and found that Home Assistant supports exporting via prometheus out of the box. I configured that and was able to scape metrics into grafana like that.
Update (2025-08) #
I’ve tried this query a few times now on more modern LLMs and the results are a bit more promising, but deepseek still starts out telling me to install something that doesn’t exist:
deepseek v3 wanted me to install things that didn’t exist “Home Assistant Grafana plugin”, and “Home Assistant data source in Grafana”. It also told me the wrong place to create a long lived access token for Home Assistant. The second section (Using a Time-Seriew Database) looks correct, but I didn’t validate it, as I ended up using prometheus which Home Assistant supports out of the box.
ChatGPT - o4 got it right on the first try.
Reply by Email