Study evaluates how language models interpret personal health records to answer patient questions

Researchers from Google and partner institutions conducted a controlled evaluation of how large language models use personal health records to respond to patient questions. The study employed Gemini 3.0 Flash and assessed its answers across three conditions: responses generated without any PHR context, with basic demographic and medication summaries, and with access to complete clinical notes.

The evaluation dataset comprised 2,257 queries representing three distinct question patterns: web-style searches, template-derived chatbot-style questions, and actual queries patients posed to healthcare providers. These queries were matched against de-identified records from a pool of 1,945 patients.

Results showed statistically significant improvements in answer helpfulness when the model had access to PHR data (p < 0.001), with benefits extending to safety, accuracy, relevance, and personalization across all question types. Clinician raters evaluated a subset of 95 responses alongside automated assessment.

Beyond aggregate improvement metrics, researchers developed a new evaluation framework to classify specific failure modes in PHR interpretation. The framework revealed gaps in how LLMs handle temporal reasoning in health records and documented instances of clinically meaningful confabulation — errors that seemed plausible but were factually incorrect.

The work frames PHR-augmented health AI as a potential tool for patient empowerment while establishing systematic methods for detecting when LLMs misinterpret complex clinical information.