Independent study finds AI usage patterns differ sharply from company reports

A new research project called the AI Observatory aggregated and analyzed 24,521 real AI conversations from seven existing datasets collected between 2023 and 2025. These conversations involved 5,000 users interacting with 52 different models, including ChatGPT, Gemini, Claude, and Grok.

The study found that AI usage patterns differ significantly from what major AI companies report. When researchers applied Anthropic’s filtering methods to their dataset, they found that 48% of conversations would have been excluded because they were not work-related. Among those excluded conversations, 44.2% involved health and relationships, 7.9% involved adult or illicit topics, 27.5% involved harassment and hate, and 16.7% involved sexual content.

The AI Observatory’s analysis also showed that people used different models for distinct purposes: Grok and Gemini were used more for information retrieval, Anthropic for coding, Gemini for social and roleplay, and ChatGPT for homework assistance. Conversation length and structure varied by model and version, with ChatGPT conversations becoming longer and more iterative when powered by GPT-4o compared to GPT-3.5.

Researchers caution that their dataset likely underrepresents sensitive uses because conversations were collected with user consent and may not include the most sensitive interactions. They emphasize that their findings are not comprehensive but do reveal substantial blind spots in company-published reports.

The AI Observatory team, led by researchers from MIT, Stanford, and the Data Provenance Initiative, plans to make its data available to other researchers and expand its datasets over time. They argue that independent, transparent analysis is necessary to inform policy and assess AI’s real-world impact beyond company narratives.