EvidencePredictive Performance

Predictive Performance

A developer-run synthetic analysis (not real patients). For each of 200 simulated ICU patients, Vancomyzer’s Bayesian model is fitted to two simulated vancomycin levels and then predicts a third concentration that was not used in the fit. The simulated patients’ “true” pharmacokinetics come from a different model than the one Vancomyzer uses.

Vancomyzer has not yet been validated in real patients. Its equations are checked against published values and synthetic test cases; external validation with patient data is planned. The results below come from a developer-run synthetic analysis (not real patients). They do not show how Vancomyzer performs in patients.

How the comparison was done

Analysis
Analysis by the Vancomyzer team using 200simulated patients, not patient records.
Calculator
Vancomyzer a posteriori (Bayesian) calculator with the Colin 2019prior. Calculator version 2026-09-25.1.
Synthetic population
200adults (age 26–95years, 129male). Age, sex, weight and serum creatinine are drawn from distributions calibrated to summary statistics of the Bai et al. 2025 ICU cohort (median weight 64.6kg, median serum creatinine 0.68mg/dL); height uses developer-chosen values. No dialysis, CRRT or ECMO. Augmented renal clearance is not excluded: Cockcroft–Gault CrCl is capped at 200 mL/min, 52patients are above 130 mL/min and 13of them are at the cap. 2patients have a BMI of 40 or more.
Simulation model
Goti 2018–based model, different from Vancomyzer’s prior: CL = 4.5 × (CrCl/120)0.8 × (WT/70)0.75 L/h, V1 = 58.4 × WT/70 L, Q = 6.5 × (WT/70)0.75 L/h, V2 = 38.4× WT/70 L, with log-normal between-subject variability (ωCL = 0.4, ωV1 = 0.3, ωQ = 0.5, ωV2 = 0.4). The weight terms and the variability values are developer choices, not Goti 2018 estimates.
Regimen
15mg/kg every 12 h, rounded to the nearest 250 mg, limited to 500–3000 mg, infused over 1.5h. No dose changes.
Fitted samples
Two levels in the dosing interval of dose 5: 3.0h after the start of the dose (1.5h after the end of the infusion) and 11.5 h (0.5h before the next dose). Each has simulated residual error (20% proportional + 1.0mg/L additive). The simulation and the fit both use steady-state equations.
What was measured
The concentration 6.0h after the start of the same dose (4.5h after the end of the infusion), predicted from the fitted calculator. This time is not used in the fit. The prediction is compared with (a) a synthetic observation at that time (truth plus residual error) and (b) the noise-free truth, reported separately.
Fits
200of 200posterior fits succeeded; the metrics use the successful fits.

Results · developer-run synthetic analysis (not real patients)

Error of the calculator’s prediction at the held-out time, one prediction per patient (calculator version 2026-09-25.1). Metric definitions are below.

Prediction compared withnBias (mg/L)rBias (%)RMSE (mg/L)rRMSE (%)
Simulated blood test at 6.0 h (including measurement error)200+0.48+8.126.1831.63
Simulated concentration at 6.0 h (without measurement error)200+0.77+4.733.5115.94

Error against the synthetic observation includes the residual error added to that observation, so it is larger than error against the noise-free truth; relative errors are inflated most when the observed concentration is low. No acceptance threshold was prespecified for this analysis. Each patient contributes one held-out concentration from one steady-state interval, which is a narrow test (see Limitations).

Metric definitions

Cpred is the calculator’s predicted concentration and Cref is the value it is compared with (the synthetic observation or the noise-free truth). Means are taken over patients.

Bias (mg/L)mean(Cpred − Cref)Signed mean error. Negative values mean predictions were lower than the reference on average.
rBias (%)mean((Cpred − Cref) / Cref) × 100Signed relative mean error.
RMSE (mg/L)√ mean((Cpred − Cref)²)Root mean squared error.
rRMSE (%)√ mean(((Cpred − Cref) / Cref)²) × 100Relative root mean squared error. Lower means less scatter.

Bias and precision as measures of predictive performance follow Sheiner and Beal (J Pharmacokinet Biopharm. 1981;9:503–512). Some later studies treat an rBias within ±20% as acceptable; that threshold is a convention, not a criterion set by Sheiner and Beal, and it was not prespecified here.

Published real-patient study · context only, not comparable

These numbers cannot be compared with the synthetic results above. Bai et al. evaluated three Bayesian dosing programs in real ICU patients. The cohort, sampling, dosing and definition of the true value (measured concentrations) all differ from the synthetic analysis, and Vancomyzer was not part of the study. Retrospective, single center (Beijing). 139 adult ICU patients with 284 measured vancomycin concentrations, mostly troughs; patients on blood purification or ECMO were excluded. Predictions were compared with measured concentrations from routine clinical dosing and monitoring.

Bai et al. 2025, Table 3, a posteriori predictions in real ICU patients
Program (model)Bias (mg/L)rBias (%)RMSE (mg/L)rRMSE (%)
SmartDose (He model)−1.54−8.734.7337.64
Pharmado (Yasuhara model)−1.15−6.603.8127.69
PrecisePK (Rodvold model)−2.03−16.034.4834.84
PrecisePK (Goti model)−0.55+0.103.8734.56

Source: Bai G, et al. Predictive performance of Bayesian dosing software for vancomycin in intensive care unit patients. Ther Drug Monit. 2025;47(5):594–602, Table 3 (a posteriori). doi:10.1097/FTD.0000000000001310

Limitations

  • Synthetic patients, not real patients

    The truth model contains only the variability that was simulated. It leaves out things that happen in real patients, such as changing renal function, fluid shifts, drug interactions, errors in dose or sample times, and assay problems. Error in real patients has not been measured and could be larger.

  • One held-out concentration in one steady-state interval

    Each patient contributes one held-out concentration, 6.0 h into the same steady-state interval as the two fitted levels (between the two fitted sample times). There are no dose changes, no levels before steady state and no change in physiology over time. This is held-out interpolation, not future forecasting: it does not show how well Vancomyzer forecasts later levels after a regimen change or as a patient's condition changes.

  • Two-level sampling only

    The calculator always receives two levels, one after the infusion and one before the next dose. Trough-only sampling and levels drawn before steady state were not simulated.

  • Truth model is partly developer-defined

    Typical values come from Goti 2018, but the weight scaling, the between-subject variability (ω = 0.4 / 0.3 / 0.5 / 0.4) and the residual error are developer choices. Different choices would change the results; the sensitivity has not been quantified.

  • Population coverage

    No dialysis, CRRT or ECMO patients are simulated. Augmented renal clearance is included (52 of 200 patients above 130 mL/min), but CrCl is capped at 200 mL/min. Only 2 patients have a BMI of 40 or more, so this analysis says little about high body weight.

  • Version

    Results reflect calculator version 2026-09-25.1, in which Colin 2019 is used at every BMI. These results apply to the stated software version and study design; they are not independent clinical validation.

Validation plan

Done

Developer-run synthetic analysis (this page)

Predictions checked against a different truth model in simulated patients. This does not validate Vancomyzer in real patients.

Planned

Retrospective evaluation with de-identified patient data

Compare Vancomyzer's predictions with measured concentrations in a de-identified ICU dataset from a partner institution, with the analysis plan and acceptance criteria written before the data are analyzed.

Planned

Prospective evaluation

A prospective study in ICU patients, similar in design to published prospective evaluations of other dosing programs (for example, ter Heine et al. 2020).