Carolyn Geason-Beissel/MIT SMR | Getty Images
To understand their customers and markets, a growing number of customer-oriented companies are using generative AI tools, alongside the language and reasoning capabilities of popular large language models (LLMs), to access and analyze their own internal content. These hybrid knowledge approaches, which typically employ a technique called retrieval-augmented generation (RAG), allow the integration of a company’s own customer insights with the general knowledge base on which an LLM was trained. Companies taking this approach to what was previously called “knowledge management” reap several benefits, such as enabling employees to access and summarize content in natural language. That capability is particularly important in large organizations — where employees searching for insights often have no idea where those insights originated or how they might be found.
Organizations gather and attend to insights about what their customers want, how they want to be sold to, and what products and services they have interest in. Customer insights typically originate from market research departments or external market research agencies, but they can also be found in sales interactions, customer letters and emails, website behaviors, social media interactions, customer service tickets, purchase patterns, focus groups, and other channels. It can add up to a lot of structured and unstructured information, so tools to help summarize, categorize, store, and access it certainly make sense.
However, companies that focus exclusively on storing and accessing knowledge are making the same error that many organizations made in the earlier generation of knowledge management: That focus is too narrow. Instead, companies should also address knowledge flows — how customer and market insights are created, analyzed, stored, and accessed. New technologies — using generative AI to at least some degree — can assist with all of those steps.
While earlier tools for knowledge management (such as Lotus Notes and Microsoft SharePoint) allowed broad access to customer and market insight content, many organizations did not experience a revolution in insights access and usage. The technology provided more access, but cultural challenges — difficulty in organizing and retrieving the knowledge, indifference to the content, organizational silos and overlaps, lack of collaboration with external agencies — often prevailed. Today, organizations still face those same cultural obstacles when tackling knowledge management.
To learn how generative AI can and can’t help leaders beat knowledge management obstacles, we spoke with customer and market insight specialists or leaders at eight different consumer-oriented companies. Some of these leaders had responsibility for creating and overseeing the generative AI system; others had broader responsibility for market research or insights-oriented technology. We also spoke with several vendors of GenAI-based technology for customer and market insights, and one market research agency that makes extensive use of GenAI-based qualitative research technology.
How Generative AI Tools Help
Companies can combine GenAI tools with customer and market insight content on their own (though it may be difficult), use external software vendors to simplify the work, or use a combination of both approaches. Procter & Gamble, for example, uses vendor-supplied software for knowledge storage and access but has created its own system for GenAI-based analysis and categorization of the content. As Kirti Singh, P&G’s chief analytics, insights and media officer, noted, this way they get “sharp, pointed answers from GenAI” rather than just links to documents.
Among the tool vendors, focus differs. Some primarily focus on the storage of and access to insights, but the tools’ value goes beyond providing simple automated virtual filing cabinets. Their functions include automated curation of documents, integration of diverse content types, on-demand analysis done in response to queries, and synthesized answers to prompts. This type of tool, however, assumes that data analysis has already been done and that insights are waiting to be found. Analysis of quantitative data is done by analytical software, and the capacity to analyze qualitative data is limited.
Other vendors concentrate on the analysis of qualitative customer data and documents for customer insights. Still other vendors emphasize rapid testing of consumer responses to advertising. Today, there is overlap among these categories, and most vendors are attempting to address the broad process of identifying or creating customer insights, curating and categorizing them, and making them available for later access. We expect that at some point, broad customer insights platforms will emerge, employing generative AI and other capabilities to address the entire process.
If the vendor’s primary function is insight storage and access, the approach typically involves adding the customer’s custom and proprietary content on top of large language models. The content stored can include structured data (quantitative market research results, spreadsheets, or customer satisfaction ratings, for example) and unstructured data (such as transcripts of interviews, social media comments, or focus group results) from both internal and external sources.
In most cases, organizations pursuing this route centralize as much customer and market content as possible in one system. Curation is required to reduce content overlaps, eliminate obsolete/irrelevant documents, and generally maintain quality. As one customer insights leader told us, “AI is only as useful as the data it learns from.” To make sense of insights, the GenAI tools typically must tackle categorization, summarization, and content tagging. Tagging the content makes it more likely to be retrieved later. Some companies use manual tagging, while some systems employ GenAI-based tagging using a predefined taxonomy.
Novartis offers an example of a company successfully revamping insight storage and access using GenAI tools. Working with an external vendor, the company developed a customer and market insights system, called Sherlock, for its consumer business. After users pose questions, the system gives answers by pointing to a specific line of text or a time stamp in a video. Sherlock also incorporates expert-curated microsites, known as Knowledge Zones, on particular topics, such as packaging. Users who add content to the system must adhere to strict governance guidelines about document formats and quality. Novartis’s research vendors can upload project deliverables directly into Sherlock.
The system helps Novartis avoid spending on redundant insights services across its business and helps employees find relevant insights quickly, without overgeneralization. (For example, it could flag results that were based on patient data from Europe only, using a feature called WatchOut.) The results have added up: Novartis saved more than $29 million in primary market research costs in just one year. Such use of GenAI to enhance insight storage and access can facilitate the democratization of information by helping employees find both knowledge and knowledgeable people.
