arXiv:2607.08625 – The complexities of patient-centred conversational artificial intelligence
Note: This contribution summarizes a current, yet-to-be-peer-reviewed arXiv preprint. The results are preliminary.
Health chatbots based on large language models are increasingly used for the initial assessment of symptoms. However, their development and evaluation have so far mostly relied on cooperative, clearly articulating simulated patients – a setting that hardly reflects real communication. This is precisely where the arXiv preprint “The complexities of patient-centred conversational artificial intelligence” by João Matos, Olivia Buege, Donny Cheung, and colleagues comes in.
What is it about?
The authors initially analyzed 2,053 real conversations between patients and a chatbot from the Verily Me app (February to June 2026). The picture is heterogeneous: 77 percent of the sessions were prematurely terminated, 67 percent of patient messages contained fewer than six words, 37 percent showed recognizable emotional signals such as fear or frustration, and 79 percent exhibited atypical communication features – such as missing punctuation, typos, or unusual capitalization.
Based on this, the team developed a patient simulator that models four aspects of a conversation separately: which clinical facts are disclosed, which emotions are conveyed, which conversation strategy is pursued, and which communication style (formality, typos, vocabulary, language level) is employed. In total, 20 adjustable parameters are available. Technically, the simulator is based on the orchestration library LangGraph with differently sized language models for main and auxiliary functions.
Whether the simulator is realistic enough was tested in three steps: First, whether activated parameters in the conversation are actually recognizable (on average very reliable). Second, whether the simulated statements remain clinically plausible (94.3 percent of the checked statements were rated as completely accurate). Third – and this is the actual “Turing test” of the study – whether 15 human reviewers can distinguish real from simulated conversations. The result: The hit rate was only 55.0 percent, hardly better than chance. Two AI models, which were additionally used as reviewers, performed significantly better than humans with hit rates of 61.9 and 81.9 percent, respectively.
What is it good for / what does it bring?
The practical value of the work lies in uncovering a blind spot in the evaluation of health AI: If systems are only tested with cooperative, well-articulating test patients, their actual performance in the field is overestimated. To demonstrate this, the authors had four commercial language models (Gemini 3.5 Flash, GPT-5.5, GPT-5.4-mini, and Claude Opus 4.6) triage 1,164 clinically rated cases – each with the same medical facts but different communication styles, represented by five patient personas (“anxious,” “dismissive,” “informed self-advocacy,” “restricted communication,” and a neutral baseline).
The central finding: The communication style alone noticeably shifted the assigned urgency level – with identical medical facts. For example, with Gemini 3.5 Flash, the “anxious” persona led to a 13.5 percentage points higher overtriage rate compared to the “dismissive” persona (classified as more urgent than actually necessary). The direction of this effect was the same across all four tested models, although with varying degrees of intensity (between 5.4 and 8.7 percentage points difference in the other three models). Since communication style is closely related to health literacy, language skills, and socioeconomic status, the authors see this as a previously overlooked risk for health inequality in automated initial assessments.
How does it work (simplified)?
To understand the cause of the effect, the authors distinguish two abilities of a model: the ability to discriminate cases correctly and the ability to correctly set the threshold for an urgency level (calibration). The discrimination ability remained comparable across all five personas – the models could still fundamentally distinguish urgent from harmless cases. Instead, the calibration shifted: With the “anxious” persona, the models systematically overestimated the probability of an urgent case, while the “dismissive” persona had the least misestimation. The authors interpret this as the communication style shifting the decision threshold of the models rather than changing their fundamental medical judgment.
The 1,164 underlying cases come from two sources: 770 synthetic vignettes from established telephone triage protocols and 394 cases from anonymized electronic patient records, each independently assigned a reference urgency level by several specialists. It is also noteworthy that the pattern showed the same direction across gender and four age groups – an indication that the effect is not limited to a single demographic group.
Classification & Limitations
The authors themselves identify numerous limitations. Only a single clinical task has been investigated so far – the three-tier urgency classification; other tasks may behave differently. Several evaluation steps rely on language models as reviewers, which brings its own sources of error. The real conversation data come from an app whose users are presumably above average in digital literacy, which underrepresents people without reliable app or internet access. Additionally, only five personas from a significantly larger possible spectrum were tested, exclusively commercial models via APIs (no open model weights), and data, reviewers, and the author team are predominantly based in the English-speaking world (USA, Canada, UK). The study was not statistically strong enough to analyze interactions between persona and demographic characteristics.
As a non-peer-reviewed preprint, the results should be classified as preliminary but methodologically carefully documented. Those interested in the graphical presentation – such as the distribution of conversation features (Figure 2) or the pairwise over- and under-triage differences between the personas (Figure 4) – can find the corresponding figures in the preprint itself; embedding them here is not possible due to the paper’s licensing conditions.
