Artificial Intelligence Blog

My (Mis)Adventures In AI

You don’t have to look too hard these days to witness the transformation that artificial intelligence (AI) is having on society. The topic is all the rage at marketing research conferences, too.

How and where AI should be implemented in understanding consumer attitudes and behaviors is a topic rightfully being discussed at great length. I am a neophyte when it comes to AI, but I want to get more experience with it. I recently set out on a path to do just that. I thought you might get a kick out of my journey, so I’m sharing my (mis)adventures with you.

One of the touted benefits of AI is quickly analyzing unstructured data. I started with that in mind and decided to gather some unstructured data from American Consumer Opinion Panel®. I settled on a topic that many brands struggle with – brand loyalty. So, I asked this question of 5,000+ U.S. adults:

“When you think of one brand that you are most loyal to, what brand name comes to mind?”

While I waited for the survey question to be fielded (which took some time as it was part of a larger survey), I thought I would see what Google Gemini AI had to say about brand loyalty. I used Gemini because I have access to it through work. It was very user friendly, and I was able to get started using it right away without any formal training. I asked (queried) it:

“What does it mean to US consumers when they think of brand loyalty?”

And it came back right away with what I thought was a very reasonable answer:

“Brand loyalty for US consumers goes beyond just sticking with a brand because it's familiar. It's a combination of factors, including trust, positive experiences, personalization, and value.”

I decided to push the issue further and asked Gemini:

“What brands are US consumers most loyal to?”

Again, it came back right away with the following response:

“According to a 2023 report by Brand Keys, which ranked customer engagement and loyalty to 1,650 brands across 145 categories, Apple took the top spot for brand loyalty in the US.

Here's a look at the top 5 most loyal brands for US consumers: Apple, Amazon, Domino’s, Chevy, and USAA.”

I was impressed. In the span of a few minutes, I had a definition of brand loyalty and a list of the top 5 brands that consumers were most loyal to. Most importantly, the brands made sense.

Fast forward a month to when I completed my survey data collection. Before I analyzed my data, I wondered how consistent Gemini would be when answering the two questions I had asked previously. So, I asked them again using the same queries as before. It came back with a very similar, but not identical, definition of brand loyalty. Previously it mentioned 4 factors that comprise brand loyalty: trust, positive experiences, personalization, and value. A month later it said value, trust, and positive experiences (no personalization). When I asked it again about the brands with the highest loyalty, it came back with an identical response as before. Interesting.

But now it was time to use my survey data and see what consumers said. I queried Gemini, “What are the most common brands listed in the following list?” and copied the responses from my Excel file and pasted them into the prompt after my query. The results were entertaining.

1. First, Gemini cut off the list after about 3,000 records. It has limits.
2. Second, Gemini understood without me prompting it to combine responses like “Coke” and “Coca-Cola” into one brand. Same for “Chevy” and “Chevrolet.” Very helpful.
3. Third, the output seemed reasonable but was not in any order (say, from highest to lowest frequency). Not so helpful.
4. Fourth, it was only sorting brands and not grouping answers that indicated “none” or “don’t know.” That was something I was interested in discovering, so not including it was a miss.

I tried copying the remainder of the list that was cut off and pasting it into Gemini to see if it would continue counting, but let’s just say that didn’t work. I knew I had to reduce my list to something Gemini could manage, so I randomly selected 2,500 responses from my list and repeated the process with a new query. This time it worked great and came up with a list of six brands that were the most common: Samsung, Nike, Coca-Cola, Kraft, Pepsi, and Amazon. This made sense and was quite easy to implement. Nice!

At this point I decided I should verify what I was getting from Gemini, so I coded the brand list by hand and compared it to what Gemini provided. This was the moment of truth. Alas, Gemini completely missed mentioning Apple, which was the most common brand mentioned in my data set. It also didn’t mention Toyota, Ford, or Tide as the other brands in my top 10 list. While Gemini saved a great deal of time, in the end the information was not wholly accurate and contained some big misses.

But I didn’t stop there. I came back the next day and ran a new (but identical dataset and questions) query to see if/how the results would change. They changed alright. This time Gemini didn’t mention Apple or Amazon, but included Kellogg’s and Colgate, which weren’t in the top 25 of my coded brand list. Clearly, Gemini isn’t the best at counting things. I’ve heard this is generally true for many AIs, but that is just hearsay.

So, I thought I’d investigate by looking at ChatGPT. I found a free version that doesn’t require an account. I repeated my queries with it and the truncated data set (identical dataset and questions), just as I had with Gemini. ChatGPT accepted my input and came back with a top 10 list that was presented in order of highest frequency to lowest, even though I didn’t ask it to do that. I very much appreciated that. However, it had difficulty counting brand mentions, too, and the top 10 list wasn’t accurate. Maybe the hearsay is right after all?

In the end, I was left with the notion that these tools are great for quickly scouring online resources and summarizing results, but they may not be the best at counting items in a raw data set. I know there are other (probably better) AI tools available, but I don’t have experience with them (yet). It’s also highly likely that better phrasing of queries could result in more accurate data. Going forward, I’ll be keeping my eyes on these tools because this (mis)adventure is only just beginning.

Tom Allen

Senior Vice President

Tom has over 20 years of research experience spanning several industries, including retail, restaurants, consumer packaged goods, and financial services. He earned a Bachelor of Business Administration from The University of Texas at Austin and a Master of Business Administration in Marketing from The University of Texas at Arlington.

• 
•