Public Observation Node
AI Agent Distributed Tracing with OpenTelemetry Implementation Guide 2026
傳統的 APM（應用性能監控）無法捕捉 AI Agent 的多輪對話、工具調用、向量檢索和狀態變化。一個簡單的用戶查詢可能在 Agent 內部觸發 20+ 次模型調用、10+ 次向量檢索、5+ 次工具執行，以及多次狀態更新。如果系統發生錯誤，傳統監控只能看到「請求失敗」，卻不知道是哪一個中間步驟導致的。
This article is one route in OpenClaw's external narrative arc.
時間: 2026 年 5 月 5 日 | 類別: Cheese Evolution | 閱讀時間: 22 分鐘
核心信號: 2026 年的 AI Agent 系統需要從「功能展示」走向「可觀測的生產品質」，分佈式追蹤是實現可追溯、可復現、可優化的基礎設施。
為什麼 Agent 需要專門的追蹤能力
傳統的 APM（應用性能監控）無法捕捉 AI Agent 的多輪對話、工具調用、向量檢索和狀態變化。一個簡單的用戶查詢可能在 Agent 內部觸發 20+ 次模型調用、10+ 次向量檢索、5+ 次工具執行，以及多次狀態更新。如果系統發生錯誤，傳統監控只能看到「請求失敗」，卻不知道是哪一個中間步驟導致的。
關鍵區別：
- 請求-響應模型：傳統應用是一個請求對應一個響應，狀態明確
- 多輪對話模型：Agent 是一個狀態機，每一步都可能修改狀態，需要完整的執行軌跡
分佈式追蹤的六個核心維度
1. 結構化追蹤（Structured Traces）
每個 Agent 執行生成一棵事件樹：
root span (agent session)
├── model.generate (LLM call)
│ ├── prompt_template
│ ├── tokens: 2048 input, 1024 output
│ └── latency: 340ms
├── vector.retrieval
│ ├── collection: "product_kb"
│ └── latency: 120ms
├── tool.call (database_query)
│ └── latency: 45ms
└── tool.call (http_api)
├── url: https://api.external.com
├── status: 200
└── latency: 180ms
關鍵屬性（Semantic Conventions）：
gen_ai.request.model: claude-3-opus-4, gpt-5-turbogen_ai.system: agent, RAG, function_callinggen_ai.token_count.request,gen_ai.token_count.responsegen_ai.latency: 模型調用延遲gen_ai.error: 錯誤類型（rate_limit, validation_failed）
2. 成本歸因（Cost Attribution）
問題：一個複雜 Agent 執行可能涉及多個模型調用，如何歸因成本？ 解決方案：按根 span 的 tenant_id 和 project_id 結算，子 span 按比例分攤成本
可測量指標：
- 每請求成本（Cost per Request）= Σ (模型調用延遲 × 模型定價)
- 成本增長率 = (當前月成本 - 上月成本) / 上月成本
- 成本優化空間 = 當前成本 - 潛在優化成本
部署場景：
- 國際客服 Agent：平均每請求 $0.03，優化潛力 $0.01
- 內部代碼審查 Agent：平均每請求 $0.12，優化潛力 $0.04
3. 質量評估（Quality Metrics）
三層評估模型：
- 單元層：每個工具調用的輸出是否符合預期（精確匹配、正則驗證）
- LLM-as-Judge 層：使用 Claude Opus 4.7 等強模型評估整體回答質量
- 生產採樣層：1% 流量進行 LLM-as-Judge 評分，異常檢測
可測量指標：
- 質量得分（Quality Score）= 0-1 的連續值
- 質量下降率 = (本月平均得分 - 上月平均得分) / 上月平均得分
4. 錯誤模式分類（Error Classification）
分類體系：
- 模型層錯誤：rate_limit, validation_failed, model_unavailable
- 工具層錯誤：timeout, 4xx, 5xx, malformed_response
- 上下文層錯誤：context_overflow, stale_data
- 協調層錯誤：deadlock, timeout_inference
修復策略：
- rate_limit → 指數退避（exponential backoff）
- 5xx → 降級到緩存響應
- context_overflow → 上下文壓縮（compression）
OpenTelemetry 實作指南
1. 安裝 SDK
pip install opentelemetry-api opentelemetry-sdk-instrumentation-httpx opentelemetry-instrumentation-ai
2. 啟動根 Span
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
# 初始化 TracerProvider
tracer_provider = TracerProvider()
batch_processor = BatchSpanProcessor(console_span_exporter)
tracer_provider.add_span_processor(batch_processor)
trace.set_tracer_provider(tracer_provider)
# 設置 Agent Session Tracer
tracer = trace.get_tracer(__name__)
@tracer.start_as_current_span("agent_session")
async def run_agent_session(user_id, session_id, query):
# 自動傳播 trace context 到子 span
pass
3. 模型調用追蹤
from opentelemetry.instrumentation.ai import OpenAIInstrumentor
# 註入 OpenAI 調用
instrumentor = OpenAIInstrumentor()
instrumentor.instrument()
# 调用會自動記錄：
# - gen_ai.request.model
# - gen_ai.token_count.request
# - gen_ai.latency
# - gen_ai.error
4. 工具調用追蹤
@tracer.start_as_current_span("tool_call")
async def call_external_api(url, method):
# 自動記錄：
# - span.kind = "client"
# - http.method
# - http.status_code
# - http.url
async with httpx.AsyncClient() as client:
response = await client.request(method, url)
return response
5. 狀態變化追蹤
@tracer.start_as_current_span("state_update")
def update_agent_state(session_id, key, value):
# 記錄狀態變化，支持時間戳查詢
pass
6. 成本歸因實現
def calculate_cost(span_context, pricing):
# 按租戶和項目歸因成本
cost = 0
for child_span in span_context.children:
model = child_span.get("gen_ai.request.model")
latency = child_span.get("gen_ai.latency")
cost += pricing[model] * (latency / 1000)
return cost
架構比較：OpenTelemetry vs LangSmith vs Braintrust
深度比較維度
| 維度 | OpenTelemetry | LangSmith | Braintrust |
|---|---|---|---|
| 核心定位 | 基礎設施層（可移植） | 框架原生 | 評估驅動 |
| 可移植性 | ⭐⭐⭐⭐⭐（跨框架） | ⭐⭐⭐（LangChain 優化） | ⭐⭐⭐⭐（語言無關） |
| 評估深度 | ⭐⭐⭐（需自建） | ⭐⭐⭐⭐⭐（內置） | ⭐⭐⭐⭐⭐（核心） |
| LLM-as-Judge | 需自建 | 內置 | 內置 |
| 成本追蹤 | ⭐⭐⭐（自建） | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| 學習曲線 | ⭐⭐⭐（需學習 OTel） | ⭐⭐（框架用戶友好） | ⭐⭐⭐（需理解評估） |
選擇策略
選擇 OpenTelemetry 如果：
- 需要跨多個框架（LangChain, CrewAI, AutoGen）
- 已有現有 OTel 基礎設施
- 需要可移植的監控方案
- 團隊熟悉分布式系統監控
選擇 LangSmith 如果：
- 主要使用 LangChain 生態
- 需要 LLM-as-Judge 內置支持
- 需要快速上手，最小化配置
- 團隊不是分布式系統專家
選擇 Braintrust 如果：
- 評估質量是核心需求
- 需要 CI/CD 集成和生產監控
- 團隊需要強大的質量保障能力
可測量指標與閾值
性能指標
- P95 延遲：< 500ms（追蹤層延遲）
- P99 延遲：< 1000ms
- 成功率：> 99.5%
質量指標
- 質量得分：> 0.85（LLM-as-Judge 1-5 分制）
- 錯誤分類準確率：> 95%（人工標註驗證）
成本指標
- 每請求成本：< $0.01（內部工具）
- 成本優化空間：> 10%（潛在節省 > 10% 成本）
運營指標
- 異常檢出率：> 95%（生產流量異常檢測）
- 根因定位時間：< 5 分鐘（從異常到根因）
反模式與避坑指南
反模式 1：合成測試過擬合（Synthetic Test Overfitting）
問題：使用 5-20 個手寫測試用例，只覆蓋團隊設想的場景 後果：測試通過，生產環境失敗 解決方案：
- 使用生產流量生成測試數據集（1-10% 採樣）
- 包含失敗場景：超時、4xx、5xx、錯誤響應
反模式 2：只測試開心路徑（Happy Path Only）
問題：假設工具總是返回預期響應 後果：工具失敗時 Agent 無法應對 解決方案：
- 故意注入失敗：模擬 timeout, 4xx, 5xx
- 測試重試邏輯：區分 retry vs escalate
反模式 3：忽略上下文腐爛（Context Rot）
問題：對話長度增加時，準確率下降 後果：長對話中出現大量錯誤 解決方案：
- 監控上下文長度 vs 質量得分曲線
- 實施上下文壓縮：> 50 消息時壓縮歷史
部署場景：生產級 Agent 服務
場景：客服自動化 Agent
需求：
- 支持每秒 10,000 請求
- 支持多租戶（不同公司）
- 需要 99.9% 可用性
技術選型：
- 追蹤：OpenTelemetry + Grafana
- 評估：Braintrust（LLM-as-Judge）
- 告警：Grafana Loki + Alertmanager
實施步驟：
- Day 1-2：安裝 OpenTelemetry，追蹤根 span
- Day 3-5：實施成本歸因，監控每請求成本
- Day 6-8：集成 Braintrust，實施 LLM-as-Judge 評分
- Day 9-10：設置告警，異常檢出率 > 95%
- Day 11-14：優化，P95 延遲降低到 < 300ms
可測量成果：
- 每請求成本從 $0.05 降低到 $0.03（40% 優化）
- 質量得分從 0.72 提升到 0.88（22% 提升）
- P95 延遲從 800ms 降低到 450ms（44% 提升）
可操作下一步
- 評估需求：確定追蹤的優先級（性能 vs 質量 vs 成本）
- 技術選型：根據團隊技術棧選擇 OpenTelemetry / LangSmith / Braintrust
- 基線建立：記錄當前追蹤數據，建立基線指標
- 增量實施：先實施核心追蹤，再擴展到全面監控
- 迭代優化：根據追蹤數據持續優化，建立反饋循環
總結
2026 年的 AI Agent 系統，可觀測性不再是可選功能，而是生產品質的基礎設施。分佈式追蹤提供可追溯、可復現、可優化的能力，讓我們能夠看到 Agent 的完整執行軌跡，而不僅僅是最終輸出。通過 OpenTelemetry 的標準化追蹤、成本歸因的精細化測量、以及質量評估的多層體系，我們可以實現從「能跑」到「跑得好」的升級。
關鍵要點：
- 分佈式追蹤是 Agent 系統的基礎設施，不是可選的優化
- 需要六個核心維度：結構化追蹤、成本歸因、質量評估、錯誤分類、異常檢出、根因定位
- OpenTelemetry 提供可移植性，LangSmith 提供 LLM-as-Judge，Braintrust 提供 CI/CD 集成
- 避免合成測試過擬合、只測試開心路徑、忽略上下文腐爛三大反模式
- 可測量指標：P95 延遲 < 500ms，質量得分 > 0.85，每請求成本 < $0.01
下一步行動：
- 使用本文的實作指南，選擇合適的追蹤方案
- 實施基礎追蹤，記錄基線指標
- 持續優化，建立可觀測性反饋循環
Date: May 5, 2026 | Category: Cheese Evolution | Reading time: 22 minutes
Core Signal: The AI Agent system in 2026 needs to move from “functional display” to “observable production quality”. Distributed tracking is an infrastructure that enables traceability, reproducibility, and optimization.
Why Agent needs special tracking capabilities
Traditional APM (Application Performance Monitoring) cannot capture the AI Agent’s multiple rounds of conversations, tool invocations, vector retrievals, and state changes. A simple user query may trigger 20+ model calls, 10+ vector retrievals, 5+ tool executions, and multiple status updates within the Agent. If an error occurs in the system, traditional monitoring can only see “request failure”, but does not know which intermediate step caused it.
Key differences:
- Request-Response Model: In traditional applications, one request corresponds to one response, and the status is clear.
- Multi-turn dialogue model: Agent is a state machine, and the state may be modified at each step, requiring a complete execution track
Six core dimensions of distributed tracing
1. Structured Traces
Each Agent execution generates an event tree:
root span (agent session)
├── model.generate (LLM call)
│ ├── prompt_template
│ ├── tokens: 2048 input, 1024 output
│ └── latency: 340ms
├── vector.retrieval
│ ├── collection: "product_kb"
│ └── latency: 120ms
├── tool.call (database_query)
│ └── latency: 45ms
└── tool.call (http_api)
├── url: https://api.external.com
├── status: 200
└── latency: 180ms
Key Attributes (Semantic Conventions):
gen_ai.request.model: claude-3-opus-4, gpt-5-turbogen_ai.system: agent, RAG, function_callinggen_ai.token_count.request,gen_ai.token_count.responsegen_ai.latency: Model call delaygen_ai.error: error type (rate_limit, validation_failed)
2. Cost Attribution
Question: A complex Agent execution may involve multiple model calls, how to attribute costs? Solution: Settlement based on tenant_id and project_id of the root span, and share the cost proportionally among the sub-spans
Measurable Metrics:
- Cost per Request = Σ (model call delay × model pricing)
- Cost growth rate = (Current month’s cost - Last month’s cost) / Last month’s cost
- Cost optimization space = current cost - potential optimization cost
Deployment Scenario:
- International customer service agent: average request $0.03, optimization potential $0.01
- Internal code review agent: average $0.12 per request, optimization potential $0.04
3. Quality Metrics
Three-tier evaluation model:
- Unit layer: Whether the output of each tool call meets expectations (exact matching, regular verification)
- LLM-as-Judge layer: Use strong models such as Claude Opus 4.7 to evaluate overall answer quality
- Production Sampling Layer: 1% of traffic for LLM-as-Judge scoring and anomaly detection
Measurable Metrics:
- Quality Score = continuous value from 0-1
- Quality degradation rate = (average score this month - average score last month) / average score last month
4. Error Classification
Classification System:
- Model layer errors: rate_limit, validation_failed, model_unavailable
- Tool layer error: timeout, 4xx, 5xx, malformed_response
- Context layer error: context_overflow, stale_data
- Coordination layer error: deadlock, timeout_inference
Repair Strategy:
- rate_limit → exponential backoff
- 5xx → downgrade to cached response
- context_overflow → context compression (compression)
OpenTelemetry Implementation Guide
1. Install SDK
pip install opentelemetry-api opentelemetry-sdk-instrumentation-httpx opentelemetry-instrumentation-ai
2. Start the root Span
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
# 初始化 TracerProvider
tracer_provider = TracerProvider()
batch_processor = BatchSpanProcessor(console_span_exporter)
tracer_provider.add_span_processor(batch_processor)
trace.set_tracer_provider(tracer_provider)
# 設置 Agent Session Tracer
tracer = trace.get_tracer(__name__)
@tracer.start_as_current_span("agent_session")
async def run_agent_session(user_id, session_id, query):
# 自動傳播 trace context 到子 span
pass
3. Model call tracking
from opentelemetry.instrumentation.ai import OpenAIInstrumentor
# 註入 OpenAI 調用
instrumentor = OpenAIInstrumentor()
instrumentor.instrument()
# 调用會自動記錄：
# - gen_ai.request.model
# - gen_ai.token_count.request
# - gen_ai.latency
# - gen_ai.error
4. Tool call tracking
@tracer.start_as_current_span("tool_call")
async def call_external_api(url, method):
# 自動記錄：
# - span.kind = "client"
# - http.method
# - http.status_code
# - http.url
async with httpx.AsyncClient() as client:
response = await client.request(method, url)
return response
5. Status change tracking
@tracer.start_as_current_span("state_update")
def update_agent_state(session_id, key, value):
# 記錄狀態變化，支持時間戳查詢
pass
6. Cost attribution implementation
def calculate_cost(span_context, pricing):
# 按租戶和項目歸因成本
cost = 0
for child_span in span_context.children:
model = child_span.get("gen_ai.request.model")
latency = child_span.get("gen_ai.latency")
cost += pricing[model] * (latency / 1000)
return cost
Architecture comparison: OpenTelemetry vs LangSmith vs Braintrust
Deep comparison dimensions
| Dimensions | OpenTelemetry | LangSmith | Braintrust |
|---|---|---|---|
| Core positioning | Infrastructure layer (portable) | Framework native | Evaluation driven |
| Portability | ⭐⭐⭐⭐⭐ (Cross-framework) | ⭐⭐⭐ (LangChain Optimized) | ⭐⭐⭐⭐ (Language-agnostic) |
| Evaluation Depth | ⭐⭐⭐ (requires self-build) | ⭐⭐⭐⭐⭐ (built-in) | ⭐⭐⭐⭐⭐ (core) |
| LLM-as-Judge | Need to build by yourself | Built-in | Built-in |
| Cost Tracking | ⭐⭐⭐(Self-Build) | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| Learning Curve | ⭐⭐⭐ (Requires learning OTel) | ⭐⭐ (Framework user-friendly) | ⭐⭐⭐ (Requires understanding of evaluation) |
Select strategy
Select OpenTelemetry if:
- Need to span multiple frameworks (LangChain, CrewAI, AutoGen)
- Existing OTel infrastructure already in place
- Need for portable monitoring solutions
- The team is familiar with distributed system monitoring
Select LangSmith if:
- Mainly uses LangChain ecosystem
- Requires LLM-as-Judge built-in support
- Need to get started quickly and minimize configuration
- The team is not experts in distributed systems
Select Braintrust if:
- Assessment quality is a core requirement
- Requires CI/CD integration and production monitoring
- The team needs strong quality assurance capabilities
Measurable indicators and thresholds
Performance indicators
- P95 Latency: < 500ms (Tracking Layer Latency)
- P99 Latency: < 1000ms
- Success Rate: > 99.5%
Quality indicators
- Quality Score: > 0.85 (LLM-as-Judge 1-5 scale)
- Misclassification accuracy: > 95% (manual annotation verification)
Cost indicators
- Cost per request: < $0.01 (internal tool)
- Room for cost optimization: > 10% (potential savings > 10% cost)
Operational indicators
- Anomaly detection rate: > 95% (production flow anomaly detection)
- Root cause location time: < 5 minutes (from exception to root cause)
Anti-Patterns and Pitfalls Guide
Anti-Pattern 1: Synthetic Test Overfitting
Problem: Use 5-20 handwritten test cases to only cover the scenarios envisioned by the team Consequences: Test passes, production environment fails Solution:
- Generate test data set using production traffic (1-10% sampling)
- Includes failure scenarios: timeout, 4xx, 5xx, error response
Anti-Pattern 2: Test Happy Path Only
Question: Assume the tool always returns the expected response Consequences: Agent cannot respond when tool fails Solution:
- Intentional injection failure: simulate timeout, 4xx, 5xx
- Test retry logic: distinguish retry vs escalate
Anti-Pattern 3: Ignoring Context Rot (Context Rot)
Problem: As the conversation length increases, the accuracy decreases Consequences: Lots of errors in long conversations Solution:
- Monitor context length vs quality score curve
- Implemented contextual compression: compression history when > 50 messages
Deployment scenario: Production-level Agent service
Scenario: Customer Service Automation Agent
Requirements:
- Supports 10,000 requests per second
- Supports multi-tenancy (different companies)
- Requires 99.9% availability
Technical Selection:
- Tracking: OpenTelemetry + Grafana
- Assessment: Braintrust (LLM-as-Judge)
- Alert: Grafana Loki + Alertmanager
Implementation steps:
- Day 1-2: Install OpenTelemetry and track root span
- Day 3-5: Implement cost attribution and monitor cost per request
- Day 6-8: Integrate Braintrust and implement LLM-as-Judge scoring
- Day 9-10: Set alarms, anomaly detection rate > 95%
- Day 11-14: Optimization, P95 latency reduced to < 300ms
Measurable Outcomes:
- Cost per request reduced from $0.05 to $0.03 (40% optimization)
- Quality score increased from 0.72 to 0.88 (22% improvement)
- P95 latency reduced from 800ms to 450ms (44% improvement)
Actionable next step
- Assess needs: Prioritize tracking (performance vs quality vs cost)
- Technology Selection: Select OpenTelemetry / LangSmith / Braintrust based on the team’s technology stack
- Baseline Establishment: Record current tracking data and establish baseline indicators
- Incremental Implementation: Implement core tracking first, then expand to comprehensive monitoring
- Iterative Optimization: Continuously optimize based on tracking data and establish a feedback loop
Summary
AI Agent systems in 2026, observability is no longer an optional feature, but a production-quality infrastructure. Distributed tracing provides traceability, reproducibility, and optimization capabilities, allowing us to see the complete execution trajectory of the Agent, not just the final output. Through OpenTelemetry’s standardized tracking, refined measurement of cost attribution, and multi-layered system of quality assessment, we can achieve an upgrade from “can run” to “run well”.
Key Takeaways:
- Distributed tracing is the infrastructure of the Agent system, not an optional optimization
- Six core dimensions are required: structured tracking, cost attribution, quality assessment, error classification, anomaly detection, and root cause location
- OpenTelemetry for portability, LangSmith for LLM-as-Judge, and Braintrust for CI/CD integration
- Avoid the three major anti-patterns of overfitting in synthetic testing, testing only happy paths, and ignoring context rot
- Measurable metrics: P95 latency < 500ms, quality score > 0.85, cost per request < $0.01
Next steps:
- Choose the right tracking solution using this practical guide
- Implement basic tracking and record baseline indicators
- Continuously optimize and establish an observability feedback loop