System Health's Differential Diagnosis Algorithm Achieves Industry-Leading Performance
Chethan Gangireddy, M.D., Ziquan Fu
7.22.2026
The complexity and high stakes of diagnostic reasoning make it one of the most difficult tasks that clinicians perform. Frontier LLM models have shown promise in tackling complex, multimodal diagnostic reasoning tasks, but are brittle, easily distracted [1], and can struggle to provide reliable reasoning due to their non-deterministic nature.
Graph-based methods offer the potential to address these shortcomings. Here, we report on the performance of System Diagnose, powered by the System Clinical Graph, on the MIMIC-IV-Ext-DiReCT dataset.
The MIMIC-IV-Ext-DiReCT (DiReCT) [2, 3] dataset is frequently used to evaluate the diagnostic reasoning capabilities of LLMs and to benchmark RAG based systems. DiReCT contains 511 cases derived from actual clinical notes with a primary discharge diagnosis serving as the ground truth.
Real-world clinical decision support (CDS) applications run on EHR records, not curated clinical vignettes. System has previously published a novel text-to-FHIR pipeline [4] that transforms unstructured input into clinically realistic EHR data. We used this pipeline to generate 132 FHIR-compatible patient bundles from DiReCT clinical notes (excluding notes with imaging data for this study). We then ran the Diagnose API on these bundles alongside frontier LLM models to produce top 5 diagnoses for each bundle. For evaluation, we used a GPT-5.4-mini judge following our approach in [4] to assess clinical equivalence between the model outputs and the ground truth and computed top-1 and top-5 accuracy.
System Diagnose achieved 68.2% top-1 accuracy, outperforming both general purpose frontier models such as GPT-5.6 Sol, Claude Opus 4.8, and Gemini 3.1 Pro, as well as Google’s health-specific model, MedGemma 27B.
System Diagnose performance on DiReCT (MIMIC IV) dataset
LLM-based CDS tools are subject to a range of potential limitations, particularly as benchmark datasets evolve beyond curated clinical vignettes towards more realistic raw EHR data, such as those encoded in FHIR records.
The System Clinical Graph addresses these limitations by providing a quantitative, deterministic substrate on which to construct diagnostic reasoning algorithms that comfortably engage complex FHIR records. We demonstrate category-leading performance on a widely used benchmark diagnostic dataset designed to recreate realistic EHR conditions using a quantitative graph-based solution.
Citations:
[1] Gu, Y., Fu, J., Liu, X. et al. Evaluating the robustness and readiness of large frontier models in health AI applications. Nat Med (2026). https://doi.org/10.1038/s41591-026-04501-8
[4] Muti, V. B., Dulout, E., & Fu, Z. (2026). MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings. arXiv preprint arXiv:2605.30295. https://arxiv.org/abs/2605.30295
[5] Wu, K., Wu, E., Thapa, R., Wei, K., Zhang, A., Suresh, A., ... & Zou, J. (2025). MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports. arXiv preprint arXiv:2505.11733. https://arxiv.org/abs/2505.11733
System Health's Differential Diagnosis Algorithm Achieves Industry-Leading Performance
Chethan Gangireddy, M.D., Ziquan Fu
July 22, 2026
The complexity and high stakes of diagnostic reasoning make it one of the most difficult tasks that clinicians perform. Frontier LLM models have shown promise in tackling complex, multimodal diagnostic reasoning tasks, but are brittle, easily distracted [1], and can struggle to provide reliable reasoning due to their non-deterministic nature.
Graph-based methods offer the potential to address these shortcomings. Here, we report on the performance of System Diagnose, powered by the System Clinical Graph, on the MIMIC-IV-Ext-DiReCT dataset.
The MIMIC-IV-Ext-DiReCT (DiReCT) [2, 3] dataset is frequently used to evaluate the diagnostic reasoning capabilities of LLMs and to benchmark RAG based systems. DiReCT contains 511 cases derived from actual clinical notes with a primary discharge diagnosis serving as the ground truth.
Real-world clinical decision support (CDS) applications run on EHR records, not curated clinical vignettes. System has previously published a novel text-to-FHIR pipeline [4] that transforms unstructured input into clinically realistic EHR data. We used this pipeline to generate 132 FHIR-compatible patient bundles from DiReCT clinical notes (excluding notes with imaging data for this study). We then ran the Diagnose API on these bundles alongside frontier LLM models to produce top 5 diagnoses for each bundle. For evaluation, we used a GPT-5.4-mini judge following our approach in [4] to assess clinical equivalence between the model outputs and the ground truth and computed top-1 and top-5 accuracy.
System Diagnose achieved 68.2% top-1 accuracy, outperforming both general purpose frontier models such as GPT-5.6 Sol, Claude Opus 4.8, and Gemini 3.1 Pro, as well as Google’s health-specific model, MedGemma 27B.
System Diagnose performance on DiReCT (MIMIC IV) dataset
LLM-based CDS tools are subject to a range of potential limitations, particularly as benchmark datasets evolve beyond curated clinical vignettes towards more realistic raw EHR data, such as those encoded in FHIR records.
The System Clinical Graph addresses these limitations by providing a quantitative, deterministic substrate on which to construct diagnostic reasoning algorithms that comfortably engage complex FHIR records. We demonstrate category-leading performance on a widely used benchmark diagnostic dataset designed to recreate realistic EHR conditions using a quantitative graph-based solution.
Citations:
[1] Gu, Y., Fu, J., Liu, X. et al. Evaluating the robustness and readiness of large frontier models in health AI applications. Nat Med (2026). https://doi.org/10.1038/s41591-026-04501-8
[4] Muti, V. B., Dulout, E., & Fu, Z. (2026). MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings. arXiv preprint arXiv:2605.30295. https://arxiv.org/abs/2605.30295
[5] Wu, K., Wu, E., Thapa, R., Wei, K., Zhang, A., Suresh, A., ... & Zou, J. (2025). MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports. arXiv preprint arXiv:2505.11733. https://arxiv.org/abs/2505.11733
.webp)
.webp)
