TL;DR: OpenAI Decisions API calibration cannot be assumed as of the Public Beta launch. You must backtest probability outputs yourself before using them in churn scoring, forecasting, or triage. Vendor documentation is silent on calibration methodology, so raw scores may be statistically overconfident or poorly ranked. A reliability-diagram and Brier-score workflow against held-out labeled data is the only safe path to production.
Key takeaways
- OpenAI's calibration silence is the risk: The Decisions API documentation never explains how its probability-like scores are produced. A 0.85 output does not guarantee 85% of those predictions will be correct in practice.
- Backtesting fills the vendor gap: Map API confidence scores against historical ground-truth outcomes before setting any threshold.
- Post-hoc scaling fixes bad probabilities: Temperature scaling, Platt scaling, and isotonic regression can correct miscalibrated scores without retraining anything.
- Structure is the API's real advantage: Evaluating complex, rule-based predicates against unstructured inputs suits tasks where a classical classifier would require far more labeled data.
- Calibrate before you deploy, not after: Hard thresholds set on unchecked scores will silently misroute customers or patients at scale.
Introduction
The OpenAI Decisions API, available as of Public Beta launch, is a structurally new endpoint that evaluates predicates, selects from fixed options, and scores text, images, and JSON against rubrics. OpenAI's documentation calls the predicate value "the model's estimate" and stops there: no calibration methodology, no reliability benchmarks. For teams running churn scoring, forecasting, or triage in production, that silence is the central risk.
What does the Decisions API actually return, and why is "the model's estimate" not the same as a probability?
Calibration means that when a model assigns a 0.85 confidence score, 85% of those predictions should be correct in practice. The Decisions API returns a structured score that OpenAI explicitly does not guarantee meets that standard, so a 0.85 output may not correspond to an 85% real-world outcome rate.
The API does three structurally distinct things: predicate evaluation, fixed-option selection, and rubric scoring. The underlying engine is GPT-6-Luna with structured and predetermined output. OpenAI makes no calibration claim, meaning there is no vendor guarantee that a score of 0.90 corresponds to a 90% correct prediction rate in your data.
Think of a weather forecaster who says "90% chance of rain" every day but is right only 70% of the time: not inaccurate on direction, but miscalibrated on confidence. As Hugging Face's guide frames it, confidence is evidence, not authorization.
What is the calibration gap in the Decisions API, and how does it compare to classical models and Jev?
The calibration gap is the absence of any vendor-documented methodology linking Decisions API confidence scores to empirically observed outcome frequencies, a gap that competing approaches address more explicitly.
OpenAI's documentation calls the predicate value "the model's estimate" while saying nothing about calibration methodology. Anth.us, which has been routing decisions on LLM confidence since 2023, identifies calibration as what they most hope OpenAI gets right, which signals how unresolved the question remains in practice.
| Model type | Calibration methodology | Calibration approach | Production track record |
|---|---|---|---|
| Logistic regression | Sigmoid output | Documented by design | Established |
| Gradient boosting | Platt scaling (post-hoc) | Documented, configurable | Established |
| OpenAI Decisions API | Undocumented ("model's estimate") | None stated | Public beta |
Training-objective calibration means calibration is built into what the model optimizes during training. Post-hoc calibration, used in gradient boosting, is applied after a model is already trained. These are architecturally distinct commitments, and neither constitutes an empirical guarantee without independent validation on your data.
You are not choosing between accurate and inaccurate. You are choosing between calibration you can audit and calibration you cannot.
How do you backtest Decisions API outputs against historical ground truth before setting any threshold?
Run historical labeled examples through the endpoint, bucket scores into confidence intervals, compare each bucket's predicted rate against actual outcomes, and plot the resulting reliability diagram. It shows you exactly where the scores diverge from observed reality.
Step 1: Assemble a historical dataset. Pull a meaningful sample of outcome-verified labeled records. Without clean ground truth, the harness measures nothing.
Step 2: Batch API scoring. Run each record through the Decisions API with the exact rubric you plan to deploy. Rubric drift between testing and deployment invalidates the calibration measurement.
Step 3: Bucket and measure. Group outputs into equal-width confidence buckets. Calculate observed outcome frequency per bucket and plot observed vs. predicted. A perfectly calibrated model produces a straight diagonal.
Step 4: Quantify with ECE. Expected Calibration Error (ECE) measures the weighted average gap between predicted and observed frequencies. A meaningful ECE gap in a high-stakes workflow warrants post-hoc correction before any threshold goes live.
Step 5: Apply post-hoc correction. Temperature scaling, Platt scaling, and isotonic regression can each bring the curve closer to the diagonal without retraining. Temperature scaling is the lightest option: a single learned scalar applied to raw scores. Isotonic regression handles non-monotonic miscalibration that simpler methods miss.
When should you use the Decisions API instead of a classical classifier for churn or triage?
Use the Decisions API over a classical classifier when your decision logic requires rubric-style evaluation of unstructured inputs and you lack the labeled data to fit a reliable gradient boosting model.
The API's structural advantage is that it evaluates natural-language predicates and scores rubrics without labeled training data, a real unlock for a new product line with no churn history. Where classical models have the edge: given a solid base of labeled outcomes, models with documented calibration histories and post-hoc scaling carry an auditability advantage the Decisions API cannot yet match.
The decision table below is a practical rule of thumb, not a vendor-published recommendation.
| Scenario | Recommended approach | Reason |
|---|---|---|
| New product, no labeled churn history | Decisions API + backtest harness | No data for classical model |
| Mature SaaS, years of labeled outcomes | Classical model + post-hoc scaling | Documented calibration history |
| Complex rubric-based support triage | Decisions API | Natural language predicate strength |
| Regulated fintech credit decision | Classical model or human review | Auditability requirement |
| Hybrid: structured features + text signals | Decisions API score as feature in ensemble | Captures both worlds |
FAQ
What does it mean that OpenAI calls the Decisions API score "the model's estimate," is it a real probability?
No: a 0.85 Decisions API score cannot be assumed to mean 85% of such predictions will be correct, because OpenAI's documentation discloses no calibration methodology. Validate against ground truth before using any threshold in production.
How do I apply temperature scaling on top of Decisions API outputs in production?
Fit a single scalar temperature parameter on a held-out validation set that minimizes log-loss between scaled scores and observed outcomes, then apply that scalar to all future scores before thresholding. This is one of the recommended application-level calibration techniques for the Decisions API.
When is the Decisions API the right tool for churn scoring?
When your logic involves unstructured text signals such as support tickets, NPS verbatims, or usage notes that a classical classifier cannot ingest without extensive feature engineering. Use the API score as one calibrated feature in an ensemble, not as the sole decision output.
Conclusion
OpenAI makes a consequential admission in its own documentation: the predicate value is "the model's estimate" with no calibration methodology attached. The backtesting framework described in this guide covers batch scoring, reliability diagrams, ECE measurement, and post-hoc scaling. That workflow will tell you more about the API's fitness for your specific use case than any benchmark OpenAI has yet published. Classical models hold a documented calibration advantage today. Jev holds an architectural calibration commitment built into its training objective. The Decisions API holds structural flexibility and an active development roadmap.
Confidence is evidence, not authorization.
Run your historical labeled outcomes through the Decisions API and plot your first reliability diagram before any threshold goes to production.
Learn from me
Agent Engineering Bootcamp: Developers Edition, my Maven cohort. Advanced agentic RAG, multi-agent orchestration, memory, evals, and guardrails. Take agents from prototype to production. Join the next cohort →
Hire us
Traversaal.ai. We're a team of forward deployed engineers solving the toughest AI problems for Fortune 100 companies: document intelligence, agentic data platforms, and real-time web intelligence, deployed in production. Work with our team to deploy your next agentic ecosystem. Talk to Traversaal.ai →
Join us
Want to solve these problems with us? We're always looking for forward deployed engineers who want to ship production AI. jobs@traversaal.ai
