Eight perioperative tasks became public on 3 January 2024: balanced retrospective samples showed classification signal, not bedside validation at real-world prevalence.
Reading Library · Anesthesia Epochs · 中文版
Watch the companion film
2024: GPT-4 Turbo Reads Preoperative Notes · Watch on YouTube
Historical anchor: 2024. Date details and source qualifications are explained below.

The answer sounded clinical before anyone had established that it was clinically useful.
That is the peculiar power of a large language model. A conventional risk model gives a number and invites questions about discrimination, calibration, and validation. A language model can give the number, connect it to the medical record in fluent prose, and make those same questions feel less urgent.
On 3 January 2024, Philip Chung and colleagues posted a study that placed this problem squarely inside perioperative medicine. GPT-4 Turbo received procedure-booking information and selected clinical notes, then answered eight questions familiar to anesthesiologists: ASA Physical Status, postoperative hospital admission, ICU admission, unplanned admission, in-hospital mortality, and the durations of PACU, hospital, and ICU stays.
Five months later, the peer-reviewed report appeared in JAMA Surgery. The headline scores remained. What changed most was the boundary drawn around them.
This was not the arrival of an AI anesthesiologist. It was the arrival of a serious perioperative language-model benchmark, one that showed classification signal and numerical failure in the same experiment.
There was no 2023 preprint
The experiments were performed in November and December 2023, and the public code repository received its initial commit on 30 December. Neither event is a publication date.
The official arXiv history lists a single version, 2401.01620v1, submitted at 08:41:27 UTC on 3 January 2024. It shows no 2023 public version and no later v2. The journal article was accepted on 8 March, published online on 5 June, and assigned to the August 2024 issue.
The distinction matters because models, experiments, code, preprints, and journal articles move on separate clocks. Compressing them into “a 2023 preprint and a 2024 paper” creates a clean story at the cost of a false date. The historical anchor is 3 January 2024, when the benchmark first became public.
The model did not read the whole electronic record
The source data came from three University of Washington hospitals in Seattle: UW Medical Center-Montlake, UW Medical Center-Northwest, and Harborview Medical Center. From 1 April 2021 through 5 May 2023, the source cohort included 90,844 patients and 137,535 procedures performed with anesthesia care.
Every eligible case had an anesthesia preoperative evaluation and at least one other clinical note before the procedure. Yet the anesthesia preoperative evaluation itself was excluded from the experimental input.
GPT-4 Turbo received the procedure name, procedure description, diagnosis, procedural service, and up to the ten most recent other clinician-written notes. Notes shorter than 100 tokens were removed because many were administrative statements or attestations with little clinical content. The typical case supplied about four retained notes, not ten.
This was therefore not an experiment in which a model roamed through an entire chart. Nor was it a shortcut in which the model copied a preoperative anesthesiologist's completed risk assessment. The question was narrower and more interesting: could a general-purpose model extract useful perioperative signal from a bounded set of clinical prose and booking fields?
The investigators did not fine-tune a new model on these patients. They accessed GPT-4 Turbo through Microsoft Azure OpenAI Service during November and December 2023, composed prompts, and requested a structured JSON answer plus an explanation. The novelty lay in turning an existing general model toward several clinical prediction tasks through prompting.
Seven datasets answered eight questions
The investigators built seven task-specific datasets for eight outcomes because ICU admission and ICU duration used the same cases. Within each task, only one case per patient was retained. When enough cases existed, 1,250 were selected and divided into 1,000 inference cases and a separate pool of 250 cases used to construct few-shot demonstrations.
Rare outcomes could not fill that target. The unplanned-admission task contained 949 inference and 237 few-shot cases. The mortality task contained 576 and 144.
The datasets also overlapped across tasks. They cannot be summed into 7,525 independently evaluated patients. A patient could contribute to more than one task-specific set even though only one of that patient's procedures was retained within any single task.
Sampling changed the meaning of the headline results even more than overlap did.
Among all 137,535 source procedures, postoperative ICU admission occurred in 0.49%, unplanned admission in 0.43%, and in-hospital mortality in 0.3%. For the dedicated prediction datasets, the investigators sampled these outcomes to approximately equal positive and negative groups. ASA classes and duration ranges were also rebalanced by inverse-frequency sampling.
That choice made prompt strategies easier to compare. Without it, a model could look deceptively good by predicting the overwhelmingly common negative class. But a balanced experiment cannot tell a hospital how many false alerts to expect when the outcome prevalence returns to three or five cases per thousand.
An F1 score from a nearly 50% positive sample is not a bedside probability. It does not establish calibration, positive predictive value, or operational burden at the source prevalence.
Classification contained a real signal
The five classification tasks were scored with F1 for binary outcomes and F1-micro for the six-class ASA Physical Status task. Each result was compared with a random classifier, not with clinicians or a validated perioperative risk model.
The best task-specific results were:
- ASA Physical Status: F1-micro 0.50, 95% CI 0.47 to 0.53, versus a random baseline of 0.17.
- Hospital admission: F1 0.64, 95% CI 0.61 to 0.67, versus 0.52.
- ICU admission: F1 0.81, 95% CI 0.78 to 0.83, versus 0.52.
- Unplanned admission: F1 0.61, 95% CI 0.58 to 0.64, versus 0.45.
- Postoperative in-hospital mortality: F1 0.86, 95% CI 0.83 to 0.89, versus 0.49.
The ICU and mortality results were striking. They showed that procedure information and clinical narrative contained structure a general-purpose language model could exploit without training a new task-specific model on this cohort.
They did not show “86% accuracy.” F1 is the harmonic mean of precision and recall, not the proportion of all predictions that were correct. The mortality task was also built from a roughly balanced sample even though death occurred after only 0.3% of source procedures.
The ASA result captures the double meaning of capability especially well. Moving from a six-class random baseline of 0.17 to 0.50 is not trivial. Yet 0.50 still leaves substantial misclassification, often between adjacent ASA classes. Hospital and unplanned admission showed more modest gains.
The defensible conclusion is precise: GPT-4 Turbo outperformed random task-specific baselines on several internally constructed, balanced classification datasets. It was not a demonstration of real-world predictive accuracy.
Duration prediction removed the protection of fluent prose
The other three tasks asked for continuous numbers. Performance was measured with mean absolute error and compared with a dummy regressor that always returned the dataset mean.
For PACU phase 1 duration, the model's best error was 49 minutes, with a 95% confidence interval of 46 to 51. The mean baseline missed by only 36 minutes. The language model was worse.
For hospital duration, the best model error was 4.5 days versus 5.4 days for the baseline. That was a modest numerical improvement, not evidence of clinically adequate patient-level prediction.
For ICU duration, both the model and baseline produced an error of about 1.1 days.
The supplemental scatterplots explain the weakness better than a summary metric can. Predictions collect in horizontal bands at familiar integer values. Some strategies display visible ceilings. Actual stays continue across the horizontal axis while the model repeatedly returns a narrow vocabulary of plausible durations.
A language model is exceptionally good at producing a number that sounds like a reasonable hospital stay. That does not mean it has learned a stable continuous mapping between a patient's record and time. The same architecture that could separate some binary outcomes often treated duration as a collection of common verbal guesses.
Classification capability and duration failure are not competing interpretations of the paper. Together, they are its most durable result.
Prompting behaved like an experimental variable, not a magic dose
The study crossed several prompt choices. It compared original notes with GPT-4 Turbo-generated summaries, zero-shot with few-shot prompting, and answers produced with or without chain-of-thought instructions. Few-shot conditions used 5, 10, 20, or 50 demonstrations.
Demonstrations came from the separate 20% few-shot pool, not from the inference cases being scored. For chain-of-thought demonstrations, GPT-4 Turbo first received the correct label and generated a rationale. Non-chain-of-thought prompts requested the answer before the explanation. Chain-of-thought prompts reversed that order and asked for step-by-step reasoning before the answer.
No monotonic rule emerged.
Chain-of-thought improved some classification settings but did not benefit every task. Summaries could focus the context or remove useful detail. Increasing the number of examples helped some cells and worsened others, particularly among the numerical outcomes.
Fifty demonstrations were not automatically better than five. Asking for more reasoning did not reliably make the answer more accurate. Prompt design remained an intervention to test, not a universal prescription.
One case adds a useful footnote. In the unplanned-admission dataset, the only retained note for a patient contained no clinically relevant content. GPT-4 Turbo refused to predict across all five allowed attempts. That suggests the model could sometimes recognize inadequate context. It was one case, however, not a designed study of abstention, safety, or calibrated uncertainty.
An explanation was generated, not validated
Every prediction arrived with a natural-language explanation. The prose could connect diagnoses, history, and procedure to an answer in a way that felt familiar to a clinician.
The January preprint leaned into that property. It described the explanations in strongly positive terms and moved relatively quickly from benchmark performance toward possible clinical assistance.
The peer-reviewed paper added a crucial limit: there was no ground truth for explanation quality. Experts did not systematically score whether explanations were faithful, complete, biased, misleading, or clinically useful. Nor did the study test whether reading them improved a clinician's decision.
Readability is an interface property. It is not evidence that a rationale is correct.
The experiment also included no head-to-head comparison with anesthesiologists, other clinicians, established risk calculators, or dedicated perioperative machine-learning models. The discussion noted that specialized tabular models in prior work can achieve stronger strict F1 performance. That contextual observation was not a comparison conducted here.
Claims that GPT-4 “beat doctors” or “beat traditional risk models” have no experimental denominator in this study.
Six thousand dollars purchased an experiment, not a clinical system
The official workbook records about $6,002.01 in Azure GPT-4 Turbo API calls across the eight tasks, including input and output tokens. It is a useful measure of what repeated long-context experimentation cost at the end of 2023.
It is not a current cloud-services quotation. It excludes the much larger system around a clinical tool: data governance, integration, security, monitoring, human review, drift detection, implementation, and accountability.
The investigators released code for data processing, prompt construction, output validation, retry handling, metrics, and analysis. The archived public repository makes important parts of the method inspectable.
It does not contain patient data. The official data-sharing statement says the raw and generated datasets cannot be released publicly because they derive from electronic records containing protected health information. Access may be requested under a University of Washington data-use agreement and institutional policy.
Public code therefore improves transparency without making the cohort fully reproducible. It also does not establish that another health system can reproduce the results with different records, documentation practices, prevalence, infrastructure, or model service state.
Peer review added the conditions after the headline
The numerical centre of the paper survived from the January preprint to the June publication. The final report changed the force of the claim by making missing evidence explicit.
The peer-reviewed limitations addressed evaluation at real-world prevalence, the cost of long contexts, single-system generalizability, the absence of a ground truth for explanations, possible bias in clinician judgment, comparison with specialized models, and the need for prospective validation.
That was not a retreat from the experiment. It placed the experiment on the correct rung of the evidence ladder.
Retrospective internal performance comes first. External validation, prospective workflow evaluation, human-model interaction, calibration at natural prevalence, regulatory assessment, and patient outcomes come later. Fluency cannot be used to skip the intervening steps.
What entered perioperative medicine was a new set of questions
This study did not show that GPT-4 Turbo could decide who should enter an ICU. It did not demonstrate better triage, fewer complications, or improved survival. It did not evaluate or justify autonomous action.
It established something more basic and more likely to endure: perioperative clinical narrative had become a measurable input for a general-purpose language model. A single prompted model could extract classification signal across several tasks without a separate task-specific training program for each one.
At the same time, balanced sampling, overlapping datasets, mixed prompting effects, unvalidated explanations, and poor duration prediction prevented that signal from becoming a bedside claim.
The date 3 January 2024 belongs on the timeline not because a machine assumed responsibility for perioperative risk.
It belongs there because the field obtained a sufficiently complete demonstration of both sides of the technology. Clinical language contained signal. Fluent output did not abolish the need for validation.
Classification revealed the capability.
Duration prediction revealed the boundary.
Both are part of the history.
Sources and reading limits
The publication date is 3 January 2024.
Sources: Chung P, Fong CT, Walters AM, Aghaeepour N, Yetisgen M, O'Reilly-Shah VN. JAMA Surg. 2024;159(8):928-937. PMID 38837145. PMCID PMC11154375. DOI 10.1001/jamasurg.2024.1621; official JAMA Surgery eSupplements 1-3; arXiv:2401.01620v1, submitted 3 January 2024; archived PubMed, Crossref, PMC, arXiv, and GitHub records; public code archived at commit 5a3beeb3ca6c5ff4fbe9c222aef6d05834417a58. This is the history of a retrospective internal benchmark, not individualized medical advice or a deployed decision-support system.
Visual interpretations
These illustrations explore the setting and ideas. Depicted people, actions, instruments and documents are reconstructions, not evidence of their exact historical appearance.



Continue exploring
Related reading connects shared questions and contexts; it does not by itself establish historical influence.
Related reading
- 2008: STOP-Bang Makes OSA Risk Visible Before the Operating Room
- 2010: Makary's Five-Item Study Brings Frailty into Preoperative Assessment
- 2018: Prediction Enters the Monitor, Not the Treatment Order
- 2019: ENGAGES Cuts Anesthetic Exposure but Not Delirium
- 2020: COVIDSurg Turns 1,128 Patients into an International Surgical Warning
- 2023: The FDA Creates a Device Type for the NOL Index
The Source Room · About & Editorial Method
Sources and reading limits are stated above. Illustrations are labelled as original source images or AI-made. Historical accounts are educational; they are not current treatment instructions.