Structured Output and Function Calling are often confused — what's the actual difference, and when should you use which?
The clearest distinction: Function Calling answers 'whether to call an external tool, which tool, and what parameters to pass,' while Structured Output answers 'regardless of whether a tool was called, what format should this result take.'
Example: an Agent receives 'check ETH's current price and judge whether this is a good time to buy.' Function Calling's role is deciding to call a get_eth_price() tool and passing its call parameters (which exchange's price to check) in a structured format; once the tool returns a price number, the Agent then generates a conclusion with three fields — current price, buy recommendation, confidence score — and the format specification for that conclusion is what Structured Output governs.
In practice the two are almost always used together: Function Calling needs structured format to describe which tool to call and with what parameters (itself an application of Structured Output), and the final conclusion after tool execution is also often wrapped in Structured Output for easy downstream processing. You can think of Function Calling as a specific application of Structured Output — when the structured data you need from the model is 'which tool to call, with what parameters,' that's Function Calling; when the structured data you need is any other fixed field format (analysis conclusion, classification label, risk score), that's Structured Output in the broader sense.
Why does 'asking the model to output JSON in the prompt' have such a different success rate from 'using the API's Structured Output feature to enforce format'?
The difference comes from operating at entirely different implementation layers.
Prompt-level: writing 'please respond using this JSON format: {...}' in the prompt is only a request — the model still generates text token by token, with no mechanism preventing it from drifting from the format. It might add extra text before or after the JSON ('Sure, here's the analysis:'), might skip a field, or might 'forget' the requested format midway through a complex reasoning process. These deviations happen less often with simple, clear prompts but become noticeably more frequent with complex tasks, long conversations, or when the model juggles multiple instructions simultaneously.
API-level: the Structured Output feature offered by most mainstream LLM APIs constrains at the model's underlying generation process — it doesn't check the format after the fact, but restricts, at the moment of generating each token, which tokens are even valid choices to comply with the JSON Schema syntax (a technique usually called constrained decoding). This means the model is technically incapable of generating malformed JSON, because malformed token options are excluded at generation time.
Practical implication: prompt-level format requests, in production, typically need extra fault-tolerance logic ('retry on format validation failure'), because even at 90%+ success, that 10% failure rate still triggers frequently in high-volume calling systems; API-level Structured Output needs almost none of that fault tolerance, with near-100% format correctness (the main exception being when the Schema itself is overly complex or logically contradictory). So production systems should generally use API-level enforcement, reserving prompt-level format requests for quick prototypes or low-reliability-requirement scenarios.
If Structured Output guarantees correct format, why does the 'format correct but content hallucinated' problem still happen? How should it be handled?
This is the most commonly misunderstood aspect of Structured Output: format correctness and content correctness are two entirely independent guarantee levels, and Structured Output only solves the former.
Technical reason: constrained decoding's mechanism restricts the model, at each token generation step, to choose only from candidate tokens that comply with the JSON Schema syntax — but this restriction doesn't touch the reasoning behind which valid-syntax token the model chooses. Concrete example: if the Schema requires a confidence_score field to be a number between 0 and 1, constrained decoding ensures the output is a legally formatted number in that range, but doesn't ensure that number was 'genuinely derived through reasoning' versus 'an arbitrary number that just looks plausible.' Similarly, if the Schema requires a source_url field, constrained decoding only ensures the output is a syntactically valid URL string, not that the URL actually exists or matches the claimed content.
Handling this requires layering additional verification on top of Structured Output: for verifiable fields (URLs, dates, IDs), add real-world verification in downstream code (actually fetching the URL to confirm it exists, checking the ID against a database); for fields not directly verifiable (analysis conclusions, confidence scores), a common approach is requiring the model to include an additional 'reasoning' or 'evidence' field within the same Structured Output, letting human review or another Agent cross-check whether the conclusion has reasonable support, rather than blindly trusting format-correct output; for high-risk decisions (e.g., involving financial trade execution), a common approach adds a second verification layer — an independent model call or rules engine checking the plausibility of the first structured result (e.g., 'does this recommended trade amount exceed the user's configured cap') — rather than mistaking Structured Output's format correctness for content trustworthiness.
In a multi-agent system where an Orchestrator reads multiple Sub-agents' outputs and integrates them into a final decision, what should Structured Output Schema design watch out for?
The core principle for multi-agent Schema design: each Sub-agent's output must carry enough metadata for the Orchestrator to make integration judgments without needing to re-understand each Sub-agent's internal logic. Three specifics matter.
Confidence scores must be on a uniform, comparable scale. If Sub-agent A's confidence is a text category ('high/medium/low') and Sub-agent B's is a 0–100 number, the Orchestrator can't directly compare them for integration ('high' maps to what number?). All Sub-agent Schemas should standardize the confidence scale (e.g., uniformly a 0–1 float), letting the Orchestrator directly perform integration operations like weighted averaging or taking the minimum.
Must include a traceable source field. Beyond the conclusion itself, each Sub-agent's output should include which input data or tool call the conclusion was based on (e.g., a source_refs field logging referenced trace_ids or tool call IDs). This is critical when the Orchestrator finds contradictory conclusions from multiple Sub-agents — without source traceability, the Orchestrator can't judge which to trust and can only guess or distrust everything; with it, the Orchestrator can further investigate which upstream data went wrong.
Must clearly distinguish 'conclusion' from 'uncertainty/exception state'. Sub-agent Schemas shouldn't only have a 'normal conclusion' field — they need an explicit 'unable to determine' or 'execution failed' status field (e.g., status: success / uncertain / failed, with a corresponding failure_reason). Without this status field designed in, when a Sub-agent encounters something it can't handle, it's often forced to 'manufacture a plausible-looking but actually guessed' conclusion just to satisfy the Schema's format requirements — a common trap where poor Structured Output design actually induces hallucination. It's better to let a Sub-agent honestly report 'uncertain' than force it to fill in a seemingly complete answer.
A Structured Output Schema design case for a news-analysis Agent
An Agent that tracks crypto news and judges its price impact on specific tokens uses this core analysis Schema: headline_summary (string, a summary within 20 words), affected_assets (array of token symbols), sentiment ('positive' | 'negative' | 'neutral'), confidence (0–1 float), reasoning (short explanation of the judgment basis), source_verified (boolean, whether cross-checked against at least two independent news sources), and status ('success' | 'insufficient_data'). How this Schema avoids common traps: the confidence field forces the model to give a quantified score rather than vague text like 'this news seems pretty important,' letting downstream systems set explicit thresholds (e.g., analysis below 0.4 confidence doesn't trigger auto-notification, only gets logged). The source_verified field directly addresses the 'format correct but content possibly hallucinated' defense — if this news source was only seen on one site with no corroboration, the model must honestly flag source_verified: false, and downstream systems seeing this flag automatically downgrade handling even with a high confidence score (e.g., logging only, not triggering an actual trade recommendation). The status field prevents the model from being forced to 'fabricate' an analysis conclusion when data is insufficient — if the news content is too vague to determine affected assets, the model can honestly report status: insufficient_data rather than forcing an affected_assets array. A problem discovered in actual operation: the source_verified field's own accuracy needed verification — the engineering team found the model occasionally mistook 'the same article syndicated across multiple sites' for 'multiple independent sources,' so they added a domain-deduplication layer downstream, counting only genuinely distinct news organizations (judged by different root domains) as independent sources — showing that even a verification field designed to defend against hallucination itself needs further downstream logical checking, not unconditional trust.
Structured Output's core tradeoff is constraint strictness versus model expressive flexibility. The stricter the Schema (fixed fields, strict types, no additional fields allowed), the more reliable downstream parsing, but it can also limit the model's room to express situations the Schema didn't anticipate, forcing it to jam things into ill-fitting fields (the 'forced fabrication' problem noted above). The looser the design (e.g., allowing an open-ended additional_notes field), the more flexibility, but downstream code handling that unstructured supplementary content runs back into the old fragile-parsing problem. Another tradeoff is real-time validation cost: more complex Schemas (nested structures, conditionally required fields) usually make generation harder for the model, potentially lowering format compliance rate or increasing generation latency; simpler Schemas may fail to cover all real situations, requiring more frequent revision. Recommendation: use strict Schema uniformly for core decision fields (confidence score, status, amount), and reserve room for model imperfection in a separate field explicitly labeled as 'supplementary notes' — design the two separately rather than asking one Schema to pursue both rigor and flexibility at once.