Can AI Chat Replace Traditional Search? Evidence from a RAG-based Live Events Experiment
Conversational AI tools are being increasingly integrated into online search and browsing interfaces. To understand how such tools affect user behavior and search outcomes, we run a pre-registered, incentivized online experiment in which participants search a catalog of several thousand events scraped from a major live-events platform. We compare three interfaces: a classic, attribute-based filter interface (Filter), a GenAI chat interface (Chat), and a combined interface offering both (Hybrid). In both AI conditions, AI output is restricted via retrieval-augmented generation (RAG). Our main result is that, relative to the other two conditions, the Chat condition performs the worst. The effect is robust across AI models (gpt-4.1-mini, gpt-5.2 and gpt-5.4) and is driven mainly by users who perform open-ended exploration (as opposed to users searching for a specific artist, venue or sports team). Furthermore, there are no significant differences on any metrics between Filter and Hybrid. We also document two mechanisms. First, Chat creates an echo chamber: without a visible menu, users construct queries based on what they think is contained in the catalog. As a result, they frequently search for event types that are not available in the catalog, and they search too narrowly. Second, Chat leads to prompting fatigue: it shifts effort towards prompting, and away from inspecting and evaluating the output. As a result, users often pick one of the first displayed suggestions rather than examining the list more carefully. Taken together, our results suggest that AI is best used as a complementary tool to existing filter-based search rather than as a replacement.