Using ChatGPT for Data Analytics: What could go wrong?

Today I was using a custom GPT provided by Open AI.

The Data Analyst by ChatGPT.

My goal: Analyse the content published on a LinkedIn page for a full year.

TL;DR: The results generated by ChatGPT have been wrong. I ended up analyzing the data on my own in Google Sheets but with the help of ChatGPT.

The data is from a client project, so I had to pixelate some of the content in the images.

Data preparation

I exported all LinkedIn post data for the last 365 days (The maximum date range LinkedIn provides data for).

I got an XLS file that holds both the metric data (impressions, engagement, and so on) and the actual post content on a second tab. Here is a screenshot of the Excel file.

I saved the post data as a CSV and opened ChatGPT

Start of analysis in ChatGPT

In ChatGPT, I browsed to the custom GPT directory, selected Data Analyst, and started the conversion with the upload of the CSV and a first prompt.

This CSV contains the posts to our LinkedIn page over the last 12 months. Please analyse the tone and voice of the posts.

“Error analyzing”

First learning: ChatGPT often replies with “Error analyzing”. It restarts the process, and sometimes, on the second or third try, it is able to generate results.

This error was shown multiple times during the conversation. Note that I did not re-upload the CSV file or change it in any way. Simply by retrying, ChatGPT was able to work with the file provided.

The results

I aimed to analyze the voice and tone of the published posts on that LinkedIn page.

The first feedback I got is this:

So far so basic. Nothing that blows me away but a starting point for the analysation.

Then I wanted to dig deeper with more follow up questions. Which posts are more formal? List outliers? And more questions.

Digging deeper

Now, after having some basic understanding about the tone and voice used on that LinkedIn page, I wanted to know who are the most mentioned partners on their page.

LinkedIn is not showing the @ symbol for mentions, so I provided a list of partners that I know this client works with across all projects.

My goal was to get a list of how often each partner is mentioned to give the client an understanding of how much support each partner receives from them.

The results of this text comparison are quite helpful. I provided the names of all the partners, and ChatGPT looked in the posting texts to see how often each partner was mentioned.

Wrong results

Before working with these result I did a manual check. With the search feature in Google sheets I searched for the partner names. And to my surprise, I discovered that not a single partner mention count done by ChatGPT was right.

I first thought this issue could be related to how the counting was done (case sensitive, blanks in partner names, etc.).

But as this screenshot showed, ChatGPT basically gave up and asked me to do the analysis on my own.

Taking the detour

Realizing I couldn’t use ChatGPT to complete the analysis, I changed my goal from using ChatGPT for the analysis to using It to guide me to do the analysis by myself in Google Sheets.

Getting the correct numbers on how often each partner is mentioned is a worthwhile knowledge nugget.

Here is how I proceeded in ChatGPT:

I ended up using ChatGPT as a Google Sheets tutor. How is that different from using Google to find the formulas I need to do it?

ChatGPT knows the underlying data file, and the generated output already matches the exact structure of my Google sheet.

My Learnings by doing this exercise with ChatGPT

  • Custom GPTs often throw errors (this is not only with the Data Analysis GPT, I observed this across different GPTs)
  • Text pattern comparisons are not working with the Data Analysis GPT.
  • During each ChatGPT session, I get at least one interesting nugget of knowledge that I can use.
  • Confidently wrong: The results generated by Data Analytics GPT have been entirely wrong for some questions (text comparison and counting), but it does not remined you to double-check. That is up to you.
  • I got my result, but by taking a detour using a formula in Google Sheets directly. I was still quicker doing it that way compared to starting the analysis without the help of AI.

All these learnings will be applied to future data analysis tasks I run in ChatGPT.

The best content, tips, and tools for curious creators. Twice a month.