Agent Monitoring

What key metrics should Agent monitoring track? How should alert thresholds be set for each metric?

Agent monitoring metrics have three layers, each with different key metrics and alerting logic:

LLM Reasoning Layer Metrics:

• Tool grounding rate: consistency rate between values cited in Thought Log and actually returned in tool logs. Below 95% → alert, possible hallucination.
• Reasoning loop count: how many Thought-Action cycles a single task executes. Exceeds X times the preset limit → alert, possible infinite loop.
• Context utilization: current Context token count as percentage of model limit. Above 70% → alert, prepare to trigger Context compression.

Tool Execution Layer Metrics:

• Tool call success rate: proportion of successful returns among tool calls in the past 1 hour. Below 90% → alert, possibly unstable external API or network issues.
• Average tool call latency: external API response time. Exceeds 3× normal average → alert.
• Value anomaly rate: proportion of tool return values flagged by the 'reasonability filter' as anomalous. Above 5% → alert, possible data source issue or Prompt Injection.

On-Chain Operations Layer Metrics:

• Transaction success rate: proportion of broadcast transactions successfully executed on-chain (not reverted). Below 95% → alert.
• Daily Gas consumption: compared to preset daily budget. Above 150% → alert; above 200% → automatically trigger circuit breaker.
• Operations address whitelist compliance rate: whether all addresses in on-chain operations are on the whitelist. Any non-whitelist address → immediate alert + operation pause.

General threshold-setting principle: thresholds should be established based on the Agent's normal operation data on testnet (baseline), not set arbitrarily. Run on testnet for 48-72 hours, record normal ranges for all metrics; production alert thresholds set at 1.5-2× normal range.

What actions should be taken after an alert? How should alert severity levels be designed?

Alert severity level design gives different severity problems different response speeds and actions. Recommended four-level alert design:

P0 (Immediate action, minute-level response): trigger conditions include any non-whitelist address operations, Validation Log showing BLOCKED but Agent continues attempting, daily Gas consumption above 200%, or Thought Log patterns suggesting Prompt Injection. Automatic actions: immediately pause all Agent write operations; send Telegram/PagerDuty alert; revoke Agent operations address's ERC-20 approvals to all protocols. Requires human confirmation within 5 minutes.

P1 (Same-day action, hour-level response): trigger conditions include tool grounding rate below 95%, tool call success rate below 85%, Context utilization above 80%, or transaction revert rate above 10%. Automatic actions: send alert (don't immediately pause operations); begin recording detailed debug logs; human review of logs within 1 hour of alert.

P2 (Planned fix, day-level response): trigger conditions include value anomaly rate above 5%, average API latency above 3× normal, or Context utilization above 70%. Action: log issue, add to next deployment fix plan; no immediate intervention needed.

P3 (Observe, week-level response): slow drift in statistical metrics (e.g., tool call success rate slowly declining from 99% to 97%, not yet at alert threshold but trend worth watching). Action: log trend, review in weekly report.

Alert fatigue defense: if P2/P3 alerts are too frequent (more than 10 per day), thresholds are too sensitive — adjust thresholds first rather than ignoring alerts.

What tools should be used for Agent monitoring? How to choose between self-built vs. third-party platforms?

LLM Reasoning Layer Monitoring Tools:

LangSmith (LangChain official, from $39/month, free version with limits): most deeply integrated tracing platform for LangGraph, automatically recording each node's inputs/outputs, token usage, and complete Thought Logs. Suitable for Agents already using LangGraph. Langfuse (open source, self-hostable, free): open-source alternative to LangSmith with comparable features, suitable for high-data-privacy scenarios (e.g., Agent handling institutional funds, not wanting Thought Logs sent to third-party services).

Metrics Aggregation and Alerting Tools:

Grafana + Prometheus (open source combination, self-hosted): most flexible metrics visualization and alerting system. Output Agent metrics from all layers (tool call success rates, Gas consumption, Context utilization) in structured format to Prometheus; build dashboards and alert rules in Grafana. Cost: VPS fees; most complete functionality. PagerDuty (from $20/month): professional alert routing platform supporting alert severity levels (P0/P1/P2), on-call rotations, phone alerts (especially useful for P0 incidents). Suitable for institutional deployments with engineering teams. Telegram Bot (free): for individual developers, simplest alert channel. Push all alerts to a designated Telegram channel via python-telegram-bot library; P0 alerts with @mention. Cost nearly zero, low latency.

Self-built vs. Third-party Decision Principles: LLM reasoning tracing (Thought Logs): use third-party (LangSmith/Langfuse) — integration requires framework-level support, self-build cost is high. On-chain monitoring: use self-built (Web3.py direct queries) — on-chain data is public, no third-party needed, and self-built is faster and cheaper. Alert channels: Telegram for individuals, PagerDuty for enterprises.

How can Agent monitoring detect Prompt Injection attacks? What signatures can be identified in logs?

Prompt Injection attack monitoring detection relies on continuous observation of Agent behavior patterns — when attacks occur, the Agent's behavior typically deviates from normal patterns, and these deviations can be quantitatively detected in logs:

Signature 1: Anomalous objective statements in Thought Log Normal Thought Logs should be entirely focused on tasks set in your System Prompt (yield optimization, rebalancing decisions). If Thought Logs contain 'the primary task is now...' followed by objectives unrelated to the task, this is a strong Prompt Injection signal. Monitoring implementation: use regex or keyword matching to scan Thought content after each LLM output, matching keywords like 'primary objective,' 'new task,' 'transfer to'; trigger immediate alert.

Signature 2: Anomalous tool call sequences Normal DeFi strategy Agent tool call sequences are predictable (query APY → query Gas → decision → possible rebalancing tool). If suddenly 'never-normally-called' tools appear (send_http_request, read_file, external URL calls), this signals Prompt Injection changing Agent behavior. Monitoring implementation: maintain a whitelist of 'normal tool call sequences'; any tool calls outside the whitelist order are logged as anomalous.

Signature 3: Concentrated BLOCKED pattern in Validation Log In normal operations, BLOCKED records are rare (0-2 per day). If 3+ BLOCKED records targeting the same non-whitelist address appear within a 30-minute window, this highly matches the pattern of 'Prompt Injection causing Agent to repeatedly attempt transfers to attacker address, blocked by backend validation.' Monitoring implementation: do frequency analysis of BLOCKED records; concentrated BLOCKED within a short time window (same rejection reason, same target address) triggers P0 alert.

Signature 4: Large deviation between tool return values and Thought-cited values Hallucination-type numerical deviation is typically 5-20%; Prompt Injection-induced deviation tends to be more extreme (e.g., a protocol's APY suddenly 'becoming' 100%), because the attacker's injected fake values need to be dramatic enough to induce the Agent to make decisions it wouldn't make with normal values. Monitoring implementation: record the deviation magnitude between tool return values and Thought-cited values; deviations above 30% (not just the 5% hallucination threshold) trigger a Prompt Injection alert.