AI field report: ChatGPT for competitor analysis
Burrito boxing, how ChatGPT works, and more
Howdy fellow adventurers.
In the recent past, leveraging artificial intelligence at work required deep technical knowledge of machine learning algorithms and hands-on programming expertise. The great news is that the latest batch of AI tools (e.g. ChatGPT, Claude, and other Large Language Models aka LLMs) are so user-friendly, that anyone who is familiar with using a chatbot can get started. Even you and me!
In this issue:
Case study: Use ChatGPT to identify a company’s key competitors
Spotlight: How ChatGPT Works
What we’re reading this week
Read to the bottom to learn why AI Agents are the New Excel. Let’s get to it.
Case study: How to identify a company’s key competitors
Product managers, market researchers, and investors, among others, often want to build a list of a company’s key competitors as the starting point for comparative analysis. If you don’t have access to proprietary market data, then this task usually involves online keyword searches, assembling a list of names, and then populating a table with data and info you track down. Can ChatGPT do it any better or quicker?
Adventure Club note: As a general rule, there are 3 key stages to how we interact with AI tools:
Generation stage: i.e. the initial prompt. Prompt engineering is the process of designing and optimizing your prompts, in order to return better AI output.
Refinement stage: any follow-up requests (prompts) to get the AI tool to focus in on the right output.
Evaluation stage: check that all output passes our common sense and accuracy tests before we use it. Stress-testing results can help us build better prompts.
Caution: We cannot blindly trust the AI output from any given tool, so we must remain skeptical and use critical thinking to validate all output.
Back to the example. Let’s say we’re interested in researching the restaurant chain Chipotle. For this example, let’s simply interact with ChatGPT in a very casual manner, asking questions as they come to mind.
Note: In later weeks, we’ll investigate the discipline of ‘Prompt Engineering’, which involves some additional upfront work to write more comprehensive prompts that should yield stronger, more accurate responses compared to just asking simple questions.
Generation stage: I ask ChatGPT to pull a competitor list:
Please pull a list of competitors for Chipotle
Let’s evaluate the output critically. Of course, there’s both an art and a science to selecting competitors. Competitors can be selected based on their overlap with the target company along several dimensions:
Restaurant format (e.g. fast casual or quick service)
Cuisine style (e.g. Mexican)
Geography (e.g. mostly in the United States)
Price point (lower price, under $15 for an entree)
Other key features (such as better for you, etc.).
Note that competitors do not need to match the target company on every dimension, although the more overlap there is, the more direct the level of competition.
The restaurants on ChatGPT’s list largely overlap with Chipotle across each of the key strategic dimensions above. Missing from this list are restaurants that match Chipotle on only one or two dimensions…for example, high end U.S. Mexican restaurants. Other prominent lower price restaurant chains that fall outside of fast-casual or Mexican-themed food are not listed but could still be considered as indirect competitors.
Based on the response above, my sense is that ChatGPT did a pretty solid job of identifying competitors that closely match Chipotle’s key strategic dimensions.
Refinement stage: How do we feel about these results? How successful was our prompt? I had 3 initial thoughts on how to refine the output:
The rank order of the restaurants is unclear: It is not ranked by store count, which would be useful.
The listed competitors are national chains only: Smaller regional competitors are missing.
Only restaurant chains in the Mexican-style cuisine sector are included: But according to Chipotle’s 2023 annual report, “Our competition includes a variety of restaurants in each of these segments [fast-casual, quick-service, and casual dining segments of the restaurant industry], including locally owned restaurants, as well as national and regional chains.” Based on this, the initial competitor list focusing on national Mexican-style chains seems too narrow.
So, I next ran a few more prompts to broaden and clean up the analysis:
Are there any smaller chains? What about restaurants that don't serve Mexican food? My sense is this prompt encouraged ChatGPT to “think” more broadly - as the longer list below reflects - bringing in smaller Mexican-style chains as well as non-Mexican restaurants that, like Chipotle, feature healthy, fresh, and sustainably sourced ingredients.
Please summarize both lists of competitors into a table. The first column should be the restaurant name. The second column should be a short description. The third column should be the number of stores. Include Chipotle as the top row.
Add a column for the regions where a restaurant operates.
A few seconds later, I had a pretty good table ready to export.
Evaluation stage: Now let’s validate the output!
Common sense test: Overall, this looks like a comprehensive list. Big chains like McDonald’s didn’t make the list, but I feel ok with this given Chipotle’s strategic focus on Mexican and freshly prepared, customizable foods. I recognize most of the chains on this list and quickly Googled the ones I didn’t…to make sure they exist and that ChatGPT wasn’t hallucinating.
Stress test: I next ‘stress tested’ the initial prompt (“Please pull a list of competitors for Chipotle”) by asking ChatGPT a similar but different prompt: asking it to imagine it was a regular Chipotle customer and asking it to compile a list of alternatives. That produced output similar to the table above but also included the ‘Del Taco’ chain. After checking out the company online, I would add it to the table above before finalizing the analysis.
Adventure Club note: There’s a lot of focus in the GenAI space surrounding prompt engineering - aka writing optimal prompts to yield better AI responses.
This is a simple example of the impact of prompt engineering: Simply by tweaking the language or asking the AI bot to act as a different party, you can oftentimes benefit from more diverse responses. At this stage, rather than only focusing on writing the perfect one-time prompt, think through ways to come at your questions from different angles to get the best answers.
Data accuracy tests:
The store counts listed aren’t specific numbers, just estimates or comparisons, e.g. “more than” 700. I will need to pull the actual numbers as I would not present “over 700” to an executive (it would call into question the accuracy of my research).
Nearly every store count ends in a ‘0’ which is statistically unlikely. This suggests that the data shared is being rounded or approximated.
Row-by-row checks: Given the above concerns, I plan to validate each row. Let’s audit Chipotle’s row here as an example.
Description (reflects 20% of product categories): according to Chipotle’s 2023 annual report, it features a “menu of burritos, burrito bowls (a burrito without the tortilla), quesadillas, tacos, and salads.” ChatGPT’s description (“bowls”) misses the mark. What about the burritos?! Miss.
Number of stores (7% error): turning again to the annual report, an authoritative source, Chipotle notes a meaningfully higher number (3,437) vs. ChatGPT (3,200+).
Regions (0% error): the store locator on Chipotle’s website shows the same regions.
Overall: ChatGPT performed quite well at generating a list of competitors quickly, but I wouldn’t feel comfortable using the data before validating it manually.
The list of companies is reasonable and a good entry point for further strategy or financial analysis. The info and data cited, however, was not authoritative and showed errors in several places. Given the current model performance, I personally would still need to spend ~5 minutes validating each row to feel comfortable.
Time to complete task analysis:
AI-enabled: 5 mins on prompts + 60 mins validation (5 mins per row x 12 rows) = 65 mins.
Human-only: ~15 mins to create competitor list via Google searches + 84 mins data population (7 mins to find and enter data per row x 12 rows) = 99 mins. Some of the extra time here is just for typing all the info into the table.
Advantage: ChatGPT, which saves us ~35 mins per analysis, a ~33% time savings even after validating the data.
Key takeaways:
LLM chatbots like ChatGPT are not perfect. In this example, the tool did not pull completely accurate, recent, authoritative data for all rows. This suggests that there is still a need for companies who provide proprietary, authoritative market intelligence products.
A human analyst is also still needed to validate all information returned by these chatbots and to make revisions as needed.
Use follow-up prompt modifications and stress tests to ensure you have a broad enough list of smaller or indirect competitors. Ask ChatGPT follow-up questions to get better results.
Leverage ChatGPT’s ability to make, sort, and filter data tables to save time and effort.
Prompt engineering can have meaningful impact on the responses we receive.
Know someone who wants to save 30 mins per analysis using AI methods like the one explored above? Click to share!
Spotlight
What we’re reading this week
AI Slop Is Flooding Medium [Wired]
Seven principles for effectively partnering with Generative AI for teaching and learning [SSRN]
Using ChatGPT-4o with Excel [Journal of Accountancy]
More than a quarter of new code at Google is generated by AI [The Verge]
Microsoft CEO: AI Agents are the New Excel, not ChatGPT [ERP Today]
Adventure on.
This is fantastic. I’ll apply what I learned about prompt engineering and creating and editing data tables