Property testing sounds reasonable, but practically speaking, how do you decide 'which properties are worth writing as assertions'? If too few properties are written, could important errors be missed; if too many, does it become another form of over-precise matching?
Deciding which properties property testing should cover, the core idea is asking 'if this output were wrong, what actual damage would result,' using severity of damage to prioritize, rather than trying to exhaustively enumerate every possible property.
Property types worth prioritizing as assertions: core fields directly affecting decision correctness (like the earlier-mentioned confidence score falling within a reasonable range, or the status field being one of the expected options) — errors in these properties directly cause downstream wrong decisions, worth prioritizing coverage; properties involving safety boundaries (a boundary condition like 'the Agent should absolutely never return execute: true when confidence score is below a certain threshold') — asserting these properties tests 'whether the defense mechanism genuinely works,' worth prioritizing coverage; format-and-content consistency properties (as mentioned earlier, if reasoning grounds claim to cite a specific number, that number should genuinely appear in the grounds' text, not be a vague description) — this type of assertion catches the 'format correct but content hollow' hidden risk discussed in an earlier article.
Parts that don't need assertions and can be loosened: purely phrasing differences (e.g., 'this opportunity is worth executing' versus 'recommend executing this trade' — semantically identical, differently worded) don't need asserting exact text content; non-core supplementary explanation fields, even with some expressive flexibility in content, have limited impact on overall decision quality, so testing priority can be lowered.
A practical signal for judging 'writing too much versus too little': if you find the test suite frequently fails due to LLM output phrasing differences (even when the actual decision logic is correct), this usually means assertions were written too precisely, too close to literal matching, and should be loosened into looser property assertions; if you find production shows errors the test suite completely failed to catch (e.g., a clearly unreasonable confidence score passing all tests), this indicates insufficient property-dimension coverage, and corresponding assertions should be added back. This is a dynamic process needing continuous adjustment based on actual operational experience, not a static checklist written once and never touched again.
If my Agent system frequently updates the LLM model version it uses (e.g., switching from one model to a newer one), does the testing framework described earlier need to be redesigned every time the model changes?
The entire framework doesn't need redesigning, but it does need re-execution, and testing after a model version update is actually one of the scenarios where the property-testing design approach delivers the most value, for the following reasons.
The testing framework itself (the four-layer split, the property-assertion design approach) is model-agnostic. Deterministic-logic-layer testing has nothing to do with which LLM is used, no need to adjust it for a model switch; LLM-reasoning-layer property assertions are designed against 'what reasonable range the output should satisfy' — this reasonable range itself, in theory, shouldn't change just because a model changed (if switching models genuinely changes the business logic requirement of 'what range the confidence score should fall in,' that's a business logic problem itself, not a testing framework problem).
What you should do is: after switching models, re-execute the entire test suite, rather than redesigning it. This is exactly where property testing's advantage over exact-match testing shows — if test assertions were written as 'exact match a certain string,' switching models and phrasing style changing would cause a large number of these assertions to fail, but the failure reason is only different phrasing, not the logic actually being wrong. This kind of testing generates a lot of 'false alarms' when switching models, wasting investigation time; if test assertions were written at the property level (as discussed earlier in Q1), re-executing after switching models — if the new model's reasoning quality hasn't degraded — these property assertions should still pass, since property assertions don't depend on a specific model's phrasing style to begin with.
A particularly worth-watching testing dimension after switching models: even though the testing framework doesn't need redesigning, after switching models, it's recommended to pay special attention to the results of 'edge case' scenarios within end-to-end testing — different models' behavioral differences when handling edge cases or ambiguous scenarios are usually larger than their differences handling clear scenarios. This is where a hidden degradation of 'the new model looks fine, but its edge-case handling got worse' is most likely to appear after switching models, worth prioritizing re-review.
Additional recommendation: retain the old model's test results as a baseline. If the testing framework logs statistical information like 'this test run's pass rate for each property assertion, the confidence score distribution,' after switching models, you can compare the new model's test results against the old model's baseline, rather than only looking at the binary result of 'did the new model pass the tests.' For example, the new model might pass every assertion, but its overall confidence-score distribution might have clearly shifted compared to the old model (becoming generally more conservative or more aggressive) — this kind of trend shift isn't easy to spot just looking at 'pass/fail,' but is visible through comparison against a baseline, worth incorporating into the evaluation before formally switching models.
The article mentions end-to-end testing should include 'anomalous or missing market data' scenarios — how specifically should this type of test case be designed? Can you just deliberately input a bunch of garbled fake data?
Deliberately inputting completely random garbage data has limited testing value, since this kind of data has a low probability of occurring in a real production environment and doesn't easily map to what specific defense logic needs verifying. A more valuable approach is designing test cases targeting 'anomaly patterns that genuinely could occur in the real world,' so test results can directly correspond to 'does this defense logic properly handle this genuinely encounterable scenario.'
Concrete anomaly pattern categories and corresponding test design: partial data missing (rather than completely random). For example, simulating 'querying a protocol's interest rate, the API returns success but a certain field is null or an empty string' — a genuinely possible real scenario (third-party APIs occasionally have missing fields) — testing whether the Agent has proper handling logic facing this partially missing data (e.g., flagging this data as incomplete, triggering the earlier-mentioned structural-failure handling, rather than directly treating a null value as 0 or a default value and forcing subsequent calculations, which could lead to decisions based on wrong data). Format-correct but numerically anomalous. For example, simulating 'the queried interest rate is -50%' or 'TVL suddenly becomes negative' — a value that's syntactically legal (it's a number) but clearly unreasonable business-logic-wise — testing whether the Agent does a plausibility check on this type of 'syntactically legal but semantically anomalous' value, rather than accepting it wholesale for calculation. Timestamp anomalies. For example, simulating 'queried market data's timestamp shows it's old data from long ago, not real-time data' — testing whether the Agent checks data timeliness, rather than treating data as real-time regardless of how stale it is — echoing the earlier slippage-tolerance article's discussion that time-sensitivity-critical information needs an expiration concept. Partial service unavailability. For example, simulating 'querying multiple protocols' rates, one protocol's API times out or returns an error, others normal' — testing whether the Agent can still make a reasonable judgment based on data it can still obtain when some data sources fail (or explicitly flag 'this judgment's basis is incomplete'), rather than the entire judgment logic crashing or getting stuck just because one source failed.
Design principle: each anomaly test case should ideally correspond to 'a scenario that genuinely happened in the real world, or that there's reasonable belief will happen,' rather than an imagined extreme case out of nowhere — you can reference past actual production issues encountered (if any), or common failure modes in similar systems, converting these genuinely possible anomaly patterns into concrete test cases, so test results carry real reference value, rather than testing for testing's sake.
In a multi-agent system, how should the testing framework adjust to cover Handoff logic between Sub-agents, especially the cycle-detection and global-cap mechanisms mentioned in the earlier Handoff entry?
Multi-agent system testing, beyond applying the earlier-mentioned four-layer testing framework to each individual Sub-agent, also needs dedicated test cases specifically designed for 'coordination-layer' logic — concretely broken into several directions.
Testing the cycle-detection mechanism: the earlier Handoff entry mentioned that if a task bounces back and forth between several Agents (A hands to B, B hands back to A), it should be intercepted by the cycle-detection mechanism. The method to test this mechanism is deliberately constructing a simulated scenario that triggers a cycle (e.g., having the test Sub-agent B deliberately judge 'I can't handle this task, hand back to A,' and A, upon receiving it, judges 'this task should go to B'), verifying whether the cycle-detection mechanism genuinely, upon detecting a repeated handoff target, correctly halts this cycle rather than letting it loop indefinitely. This type of testing needs the ability to precisely control the Sub-agent's judgment logic (e.g., through a test mock version, forcing the Sub-agent under test conditions to return a specific judgment result), rather than relying on real LLM reasoning's randomness to 'happen to' trigger a cycle scenario.
Testing the global-cap mechanism: simulate the scenario of 'a task handed off more than the cap allows,' verifying whether the system genuinely forcibly halts the task upon hitting the cap and flags it as needing human intervention, rather than the handoff-count counter itself having a bug (e.g., the counter not correctly accumulating across each layer — the earlier Handoff entry's mentioned risk that 'multi-layer retry counts get unintentionally amplified into a product rather than a sum' also applies to handoff-count testing, worth specifically designing a test scenario involving three or more Handoff layers to verify the global counter operates correctly).
Testing handoff information completeness: simulate the scenario of 'Sub-agent A hands off to Sub-agent B with a key field missing from the handoff package,' verifying whether Sub-agent B has proper handling logic facing incomplete handoff information (e.g., triggering the earlier-mentioned re-query or supplementary-information-request mechanism), rather than directly assuming the handoff package is always complete and forcing subsequent judgment with incomplete information.
Testing the pre-execution re-verification mechanism: simulate the scenario of 'market conditions have already shifted between Sub-agent A completing its judgment and Sub-agent B actually executing' (as in the earlier slippage-tolerance article's case), verifying whether Sub-agent B genuinely re-queries and compares, rather than blindly using A's stale data to execute directly.
The common trait across these tests: none of them test a single Sub-agent's reasoning quality (that's the responsibility of the earlier-mentioned LLM-reasoning-layer property testing) — instead, they test whether, 'when multiple Agents coordinate, the safety mechanisms repeatedly discussed in earlier articles (cycle detection, global caps, handoff completeness, pre-execution verification) genuinely take effect as designed.' This is a testing dimension specific to multi-agent systems, not covered by a single-Agent testing framework, worth treating as an independent testing category and investing dedicated design resources into.
Traditional software unit testing's core assumption is 'the same input should produce the same output' — this assumption makes testing predictable and automatable: write a test case, assert what result a function should return given a specific input, run the test once to confirm whether the logic is correct. This assumption directly breaks down for Agent systems, since an LLM's reasoning process inherently carries some uncertainty — the same input can, due to the model's randomness, follow a not-quite-identical reasoning path across different executions, arriving at slightly different results. This means Agent testing needs a testing framework that doesn't fully rely on the 'same input, same output' assumption, and this article breaks down what that framework should look like.
For an Agent system, splitting testing into three layers handled separately is more practical than trying to write 'one universal test': deterministic logic layer, LLM reasoning layer, end-to-end integration layer — each layer needs an entirely different suitable testing method.
Not everything in an Agent system involves uncertain LLM reasoning — a tool function's own logic (a 'query interest rate' tool, given a protocol name, returns the corresponding API query result), state object update logic, the retry-count calculation and cost-accumulation calculation mentioned in earlier articles — these are all deterministic program logic, where the same input inevitably produces the same output. This layer is entirely suited to traditional unit testing methods — write assertions, run tests, verify pass or fail, no special Agent-testing skill needed. This layer's testing value is easy to underestimate, since everyone's attention goes to the more eye-catching problem of 'is the LLM reasoning accurate,' but if even the deterministic logic itself has bugs (a retry counter forgetting to increment, cost accumulation calculated wrong), the most accurate LLM reasoning can't save a system with a flawed underlying logic layer.
For parts involving LLM reasoning, you can't assert 'input X must always produce output Y' (since output can vary due to the model's randomness), but you can assert 'the output must satisfy certain properties' — this is the core idea of property-based testing, applied to the Agent scenario. Concretely, rather than writing an exact-match assertion like 'given this market scenario, the Agent must return `execute: true`,' rewrite it as property assertions like 'given this market scenario (a clearly profitable arbitrage opportunity), the Agent's returned `confidence_score` must be above 0.7,' 'the Agent's reasoning grounds must mention specific interest rate numbers, not just a vague text description,' 'if the Agent judges not to execute, the `status` field must be `insufficient_opportunity` rather than left empty.' These assertions allow some room for the LLM output's phrasing and details to vary, while still catching the core question of 'is this output within a reasonable range.'
The first two layers each working well independently doesn't mean the whole system strung together is also problem-free — whether state passes correctly between nodes, whether Handoff information is complete, whether multi-agent coordination flow matches expectations — these can only be verified through end-to-end integration testing. The key design of end-to-end testing is 'using controlled simulated market scenarios, rather than directly connecting to real production environment data.' Concretely, build a set of representative test scenarios (a clearly profitable arbitrage opportunity, a small rate spread near the break-even boundary, an extremely poor liquidity scenario where transactions likely fail, a scenario where market data itself is anomalous or missing), and for each scenario, use simulated or historical-snapshot market data to run through the complete Agent flow once, verifying whether the entire system's behavior across these different scenarios matches expectation. This should specifically include the earlier-mentioned edge cases and anomalous scenarios, since most bugs don't show up in 'everything normal' scenarios — they show up in edge and anomalous-condition handling logic.
The first three testing layers assume input is 'normal' (even edge cases are naturally occurring ones), but the attack surface discussed in the earlier Prompt Injection entry and article needs dedicated adversarial testing to verify — deliberately constructing input containing malicious instructions (e.g., stuffing text attempting to manipulate Agent behavior into simulated on-chain metadata), verifying whether the Agent's defense mechanisms (input isolation, least privilege) genuinely intercept this type of attack, rather than assuming 'the defense mechanism was written, so it must be effective.' This layer's testing cost is higher (requires dedicated attack case design), but for an Agent handling actual funds, this is a step that shouldn't be skipped before launch.
If you're developing an Agent system preparing for launch, your testing strategy shouldn't stay at 'ran it manually a few times, looked normal, launched.' Splitting testing into deterministic logic, LLM reasoning properties, end-to-end integration, and adversarial testing — four layers, each with a suitable testing method (traditional unit testing, property testing, simulated-scenario integration testing, deliberately constructed attack testing) — can substantially improve the probability of catching problems before actual launch. Especially worth investing resources in are the edge cases within end-to-end testing and adversarial testing, since these two types of problems are the ones most easily missed in 'everything normal' manual testing, yet are the scenarios most likely to cause significant loss during actual operation.