Circuit Breaker

Circuit Breaker and the 'per-task total cost cap' mentioned in the earlier retry-strategy article both sound like they're limiting loss — how do the two divide labor specifically?

The core of their division is different scopes: a per-task total cost cap is a task-level boundary, a Circuit Breaker is an Agent- or service-level boundary, and the two are layered together, not substitutes for each other.

The per-task total cost cap's scope: it limits only 'this one task' — a task's retry count plus accumulated cost can't exceed $5; once a task hits this cap, that task is terminated and flagged as failed, but the next new task starts a fresh count, unaffected by this task's cap being triggered, and the overall system (this Agent, this service) is still considered operating normally.

Circuit Breaker's scope: not a single task, but 'this Agent's overall task failure pattern over a recent period' — if multiple consecutive tasks all trigger a cost cap or retry cap for the same reason, this indicates the problem isn't 'this one task had bad luck' — it's a systemic anomaly in the Agent itself, or some underlying service it depends on (an MCP Server, an external API). A Circuit Breaker monitors this cross-task pattern; once detected, it doesn't just halt the current single task — it suspends this Agent's entire subsequent execution capability until human intervention confirms the problem is resolved.

Concrete division of labor logic: the per-task cost cap is responsible for intercepting 'this one task's own anomalous consumption,' preventing a single task from burning unreasonable cost; the Circuit Breaker is responsible for detecting 'the systemic problem reflected by multiple task-failure patterns,' preventing an already-broken Agent from continuously generating a stream of failed tasks that are each individually caught by the cost cap but still accumulate substantial loss and wasted attempts. You can think of the per-task cost cap as the first layer of protection (blocking a single task's anomaly), and Circuit Breaker as the second layer (blocking the larger systemic problem of 'the single-task protection keeps failing repeatedly') — layering both together is what handles the two different scales of risk, 'a single accident' and 'a systemic malfunction,' simultaneously.

After a Circuit Breaker enters the Open state and suspends the Agent's execution, what specifically does 'human intervention confirming the problem is resolved' involve? Can't it just automatically recover after waiting a while?

Plainly 'automatically recover after waiting a while' (without needing human intervention) actually corresponds to the Circuit Breaker's Half-Open state design, but Half-Open's automatic recovery applies to a scenario of 'the problem might be transient, worth an automatic probe' — not every scenario triggering the Open state is suited to relying entirely on automatic recovery, for the following reasons.

Scenarios where Half-Open automatic recovery applies: if the Open state's trigger cause is something like 'an external API had a brief instability causing the error rate to spike' — a problem that's fundamentally transient, caused by an external factor — using the Half-Open state for automated small-scale probing (wait a cooldown period, send a small number of test requests, transition automatically back to Closed on success) is reasonable, since this type of problem genuinely has a high chance of resolving itself after waiting, without needing human confirmation every time.

Scenarios where human intervention is essential, not solely automatic recovery: if the Open state's trigger cause involves something possibly being a bug in the code logic itself, or a fund-safety-related anomaly (an unusual fund flow pattern from a specific Sub-agent, as discussed in an earlier article), this type of problem won't automatically vanish just by 'waiting a while.' If you rely only on the Half-Open automatic-probing mechanism, a genuinely problematic system might instead repeatedly cycle between Open and Half-Open states, continuously attempting recovery without ever truly fixing the root cause — in this case, human intervention is essential for genuine repair.

What concrete human intervention should involve: reviewing the specific logs and data that triggered the Circuit Breaker (as discussed in the earlier retry-strategy article, the state object should retain sufficient debugging fields, which serve as the key diagnostic grounds here), judging whether the trigger cause is transient (can directly recover manually or enable Half-Open auto-probing) or structural (needs the code fixed or strategy adjusted first before it's safe to recover); if judged structural, the Agent's execution capability shouldn't be restored until the fix is actually deployed, even if the wait takes longer than expected.

A common practical tiered design: most mature systems classify trigger causes — 'known error types historically confirmed to usually be transient' (network timeouts) allow Half-Open auto-probing; 'unknown error types, or ones historically confirmed to usually need human diagnosis' (fund-flow anomalies, a large volume of format-error output) require mandatory human intervention to recover from Open, disallowing Half-Open auto-attempts. This classification itself is a continuously accumulated-experience, dynamically-adjusted process based on Circuit Breaker trigger history.

If my Agent system serves multiple different users simultaneously (e.g., a SaaS-ified Agent product), should the Circuit Breaker be globally shared as one, or independent per user?

It should be an independent Circuit Breaker per user (or at least per independent execution environment), not globally shared, because global sharing creates a serious 'collective punishment' problem, letting one user's anomaly drag down all other users.

The problem with a globally shared Circuit Breaker: if the Circuit Breaker monitors 'the entire system's error rate' rather than separately monitoring 'each user's own error rate,' a common failure scenario is — one particular user's usage scenario happens to trigger an edge-case bug (e.g., this user connected a problematic custom MCP Server), causing this user's tasks to continuously fail. If the Circuit Breaker is globally shared, this single user's anomalous error rate can pull up the entire system's average error rate, triggering a global trip — resulting in all other originally normally operating users also being forced to have their service stopped together. This is a classic 'one person gets sick, everyone gets quarantined together' over-punishment scenario, causing unnecessary service disruption for other innocent users.

Per-user independent Circuit Breaker design: each user (or more precisely, each independent execution environment or independent resource connection) maintains its own independent error rate and cost consumption statistics; only when this specific user's own statistics exceed the threshold does it trigger a trip that affects only this user, without spilling over to other users. This design echoes the earlier least-privilege entry's principle that 'the damage range should have an explicit ceiling' — except here, what's being limited isn't a single operation's fund-damage range, but 'the impact range of one anomalous user' — likewise using isolation design to bound the damage to a minimum.

An additional layered system-level Circuit Breaker: beyond each user's own independent circuit breaker, a system-level circuit breaker layer is usually also added, monitoring whether 'the entire system's underlying infrastructure' (database connections, core LLM API calls) shows anomalies. If it's the underlying infrastructure itself having a problem (not caused by a single user's individual scenario), this genuinely affects all users, and the system-level circuit breaker tripping to suspend the entire system is a reasonable response — this layer monitors an entirely different target and trigger scope from the user-level breaker; layering both together handles both 'an individual user's localized anomaly' and 'the entire system's global anomaly' at their different scales.

A concrete implementation reminder: the user-level Circuit Breaker's state (Closed/Open/Half-Open) must be stored and tracked separately by user ID or execution-environment ID — you can't record overall state with a single global variable. This is an implementation detail easily overlooked but directly determining 'whether the collective-punishment problem occurs.'

In a multi-agent system, if multiple Sub-agents called by an Orchestrator each have their own independent Circuit Breaker, and one Sub-agent's breaker trips into Open state, how should the Orchestrator handle this situation?

This extends principles discussed in the earlier Handoff entry and multi-agent error handling — the Orchestrator's handling logic facing a tripped Sub-agent shouldn't be 'the entire task directly fails' — it should make a tiered judgment based on this Sub-agent's role within the overall task.

Step one: judge whether this Sub-agent has a substitutable path. If the tripped Sub-agent's function has another available source (as in the earlier MCP Server entry's scenario — if the market-data-query function originally connected to both a primary and a backup MCP Server), the Orchestrator can first try switching to the backup path rather than directly deciding the entire task failed — this requires the architecture to have already prepared substitute paths for critical functions in advance, not discovering there's no backup only when it's too late.

Step two: if there's no substitute path, judge whether this Sub-agent's role is a necessary segment of the task. If the tripped Sub-agent handles a 'nice-to-have' supplementary function within the task (extra sentiment-analysis reference, not a core decision basis), the Orchestrator can choose to skip this segment, flag 'this part of the information is temporarily unavailable,' and continue completing the task's other core parts; if the tripped Sub-agent handles a core segment of the task (like the 'risk assessment' or 'execute transaction' Sub-agents in the earlier Handoff entry's case), the Orchestrator shouldn't skip it — it should terminate the entire task, flagging 'a critical segment is unavailable, task suspended, needs human intervention.'

Step three: the Orchestrator's own state record should clearly reflect 'which Sub-agent tripped its breaker, causing this outcome.' Echoing the debug-field design principle discussed in earlier articles, upon receiving a Sub-agent's trip notification, the Orchestrator should fully log this information (which Sub-agent, when it tripped, the trigger cause), rather than leaving only a vague 'task failed' record — this lets human intervention pinpoint directly which Sub-agent had the problem, without needing to investigate from scratch.

The core principle this design echoes: in a multi-agent system, a single Sub-agent's trip shouldn't automatically equal the entire system tripping — the Orchestrator's role is precisely to make this layer of judgment and decision about 'how much impact this localized anomaly has on the overall task' — this is also why earlier articles repeatedly emphasized that 'the Orchestrator shouldn't hold execution-type permission' but 'should hold coordination-decision permission.' Facing an anomaly scenario like a Sub-agent tripping is exactly where the Orchestrator's coordination role should deliver its value.