OpenAI Releases LifeVal Benchmark, Revealing AI Assistant Bottlenecks in Chat History Retrieval
Frontier AI models · AI assistants
OpenAI introduced the LifeVal benchmark covering 169 daily tasks across caregiving, education, and finance. The study revealed that when tasks require extracting personal information from chat history, models achieved an average success rate of only 33.33%. Placing background information inside past dialogues dropped the success rate from 55.8% to 41.5%, indicating persistent limitations in utilizing conversational records.
OpenAI announced the LifeVal benchmark, which encompasses 169 tasks across caregiving, education, shopping, personal finance, and government services. The evaluation pairs de-identified ChatGPT dialogues, memories, reference files, and background context to assess AI executing tasks via browser and computer operations. In evaluating personalization, criteria requiring retrieval from chat history reached an average success rate of only 33.33%, compared to 47.8% for ChatGPT memory and 55.8% for directly provided background information. Moving direct background information into chat history lowered the success rate from 55.8% to 41.5%, reflecting the difficulty of extracting critical details from past logs. Across overall LifeVal evaluations, GPT-6 Astra scored 80.6%, GPT-5.6 scored 80.0%, and GPT-5.2 scored 49.3%. OpenAI published the research paper and findings but has withheld the full dataset and evaluation scripts, and the personalization section does not specify the exact model tested.
Key takeaways
- OpenAI introduced the LifeVal benchmark covering 169 tasks to evaluate AI assistants across scenarios like caregiving and finance.
- Criteria requiring information extraction from chat history achieved an average success rate of only 33.33%, revealing retrieval bottlenecks.
- Relocating background information into past dialogues decreased success rates from 55.8% to 41.5%, highlighting the impact of context placement.
- Overall scores reached 80.6% for GPT-6 Astra, 80.0% for GPT-5.6, and 49.3% for GPT-5.2, while the full dataset remains unreleased.
Source：iThome Read the original ↗
Prepared, checked and automatically published by AI; not individually reviewed by a human. Translation checked by AI.