The CHART Statement: A Methodological Milestone for Improving the Reporting of Studies on Artificial Intelligence Health Chatbots

Author

Mathieu Repiquet

Published

July 7, 2026

During the summer of 2025, a new reporting guideline was published: the CHART Statement, which stands for Chatbot Assessment Reporting Tool (The CHART Collaborative, 2025a, 2025b).

The Need for Better Reporting of Studies

This initiative followed a concerning systematic review (Huo et al., 2025), which highlighted the poor quality of reporting (the way researchers describe the methodology and results of their studies) in research evaluating the performance of generative artificial intelligence chatbots (such as ChatGPT) used to provide health advice or synthesize scientific evidence. The review included 137 primary studies and reported the following key findings:

  • Regarding the models evaluated
    • 99% of the studies assessed a proprietary (closed-source) Large Language Model (LLM), such as ChatGPT.
    • None of the studies adequately reported the technical characteristics of the model (e.g., version, temperature, maximum token length, fine-tuning, number of layers, etc.).
  • Regarding the methodology
    • 99% of the studies failed to describe the prompt engineering process.
    • 40% did not report the date on which the queries were submitted.
    • Only 3% specified the geographical location from which the queries were performed.
    • 23% reported the number of chat conversations used.
    • 35% specified the total number of queries performed.
  • Regarding performance assessment
    • 13% described a standardized evaluation process.
    • 12% reported using a blinded assessment procedure.
    • 65% of investigators evaluated LLMs based solely on their own judgment, without reference to an established standard (e.g., clinical practice guidelines or Cochrane systematic reviews).

These studies belong to the emerging field of Chatbot Health Advice Studies (CHAS), whose objective is to evaluate the ability of chatbots to synthesize scientific evidence, provide advice on screening, diagnosis, treatment and prevention, and deliver general medical information.

Poor reporting compromises the reproducibility of research, undermines confidence in its findings and, ultimately, threatens the quality, effectiveness, and safety of healthcare.

Just as pilots rely on a checklist to ensure flight safety, the CHART Statement is intended as a practical and user-friendly tool designed to help authors avoid omitting essential information when reporting CHAS. Beyond this primary purpose, the checklist can also be used during the planning phase of a study and to support the development of a study protocol. However, it is intended to complement (and not replace) existing reporting guidelines such as STROBE, TRIPOD, STARD, and CONSORT when reporting observational studies, prediction models, diagnostic accuracy studies, or randomized controlled trials. This is, for example, the case in a study evaluating ChatGPT as a psychotherapist or health coach compared with a control group, with outcomes such as changes in depression scores or health-related behaviours.

A Collective Responsibility

The successful adoption of this new guideline now depends on all stakeholders, including clinicians, researchers, research institutions, journal editors, and publishers. Everyone has a role to play in promoting its use and improving the quality of reporting in studies evaluating health chatbots. The authors of the CHART Statement also emphasize that, given the rapid pace of progress in artificial intelligence for healthcare, the guideline will need to be updated regularly. A major next step will likely be the development of a dedicated risk-of-bias assessment tool for Chatbot Health Advice Studies.

References

  • Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., Sanger, S., Ramji, K., & Guyatt, G. (2025). Large Language Models for Chatbot Health Advice Studies: A Systematic Review. JAMA Network Open, 8(2), e2457879. https://doi.org/10.1001/jamanetworkopen.2024.57879

  • The CHART Collaborative. (2025a). Reporting guideline for chatbot health advice studies: The Chatbot Assessment Reporting Tool (CHART) statement. BMJ Medicine, 4(1), e001632. https://doi.org/10.1136/bmjmed-2025-001632

  • The CHART Collaborative. (2025b). Reporting guidelines for chatbot health advice studies: Explanation and elaboration for the Chatbot Assessment Reporting Tool (CHART). BMJ, 390, e083305. https://doi.org/10.1136/bmj-2024-083305