I’m Not Afraid of ChatGPT
January 2023
Joe Madden, Mobile Experts
Several journalists have written articles recently about ChatGPT replacing them. It’s a reasonable fear—this tool is pretty good at taking publicly available information and synthesizing a coherent article. Some journalists are likely to lose their jobs, especially in high-volume data-based news areas such as financial news or sports news.
In the market research game, we’re likely to see heavy use of tools like ChatGPT as well. Some research can be completely replaced by ChatGPT and will become essentially free. Other research will continue to have value. Here’s my breakdown as a 15-year veteran:
1. GIGO Research (garbage in, garbage out): There’s a class of “reports” that can be purchased, mostly from companies based in India, which have essentially zero insight into the market. They’re simply a compilation of the publicly available data on the key competitors in a market segment. This category is likely to be completely replaced by ChatGPT.
2. Hype Research: You’ve probably seen these reports: The charts always go up on the right-hand side. Maybe there’s helium in their Excel model, because the market always shows a strong growth rate, no matter how ridiculous the product is in the real world. This kind of research is generally produced by affable people in business suits without hands-on technical and product experience. My dad would say that they couldn’t find their butt with both hands. I’m a little less anatomical and more cynical: I think that some people knowingly buy market research with optimistic numbers so that they can justify the project to investors or their CFO.
3. Market Surveys: It’s very common to see charts pointing out that “80% of survey respondents will be using XX technology next year.” The problem with this approach is that gathering data from hundreds of companies can result in a machine-gun approach: create an online survey and accept any input from a company, whether it’s from a C-suite executive or a truck driver. The truck driver has more time on his hands than the CTO, so the uninformed responses swamp out the really insightful responses. This approach is useful in finding consumer preferences, but not for complex technical or business-model questions.
4. Pay to play: Want to be listed as a market leader? Pay for a subscription. This disgusting practice is far too common, and the problems are obvious.
5. Hard-nosed analysis: In other words, market analysis that makes a serious attempt to provide true analysis and most-likely predictions of future market dynamics. This is where I live. Sometimes we produce reports (like our 2012 femtocell report, or our 2013 Cloud RAN report) that show the market going nowhere. Other times, we produce analysis (like our 2022 Private 5G research) showing a strong growth cycle. The market prediction depends on technical factors and on the expected ROI for all parties in the ecosystem.
ChatGPT can replace Category 1 and even Category 2 analysts, because they essentially follow a formula: Collect information that is offered up by the vendors, add up the estimates, and extrapolate into the future. This can be reduced to an algorithm.
On the other hand, good market research follows a very different method. My company collects component shipment data to track radio deployment. This is not the type of information that can be found on the Web…it’s only available through personal relationships with chip suppliers. The suppliers only provide this information to us because they trust us to protect their confidential data.
Similarly, a good method involves data collection from a second, independent source. We collect data from chip vendors and compare it with inputs from mobile operators and OEMs. If the numbers don’t match, we know that some information is missing or wrong. We’ve used this approach to create a RAN revenue and market share tracking service that is more accurate than any other scoreboard…you can read about our methodology here.
Beyond the methodology, good research involves a great deal of judgment. A good analyst has 20 years of experience in the battle to introduce a technology product, and his/her battle scars remind him/her that subjective factors in the business world can outweigh the cost analysis or the ROI calculation. I don’t think that this kind of judgment can be reduced to an algorithm. Heck, if I simply added up the market share estimates given to me by RAN vendors, I would be tracking a total of 150%.
Who is defining a slightly different market segment to make themselves look better? Who is being conservative to avoid setting sales targets too high for themselves? Are the numbers from China trustworthy? These are very human judgment calls, and that’s why people will continue to pay for high-quality insight.
Today, almost 200 companies use Mobile Experts research in planning their approach to the mobile market. Almost every RF component vendor, small cell vendor, and network equipment vendor uses our research because of its accuracy.
Bring on ChatGPT: it doesn’t scare me.
Joe Madden
Chief Analyst
Mobile Experts