Agent Monitoring¶

Agent Monitoring correlates Agent/LLM requests with the entire application trace, tracking the full flow of each conversation and precisely measuring the Token consumption for every generation task.

When using Agent Monitoring, you can:

• View the complete trace of a single request: clearly see the entire process from receiving the user question, processing (e.g., database query), to calling the LLM model and returning the answer.
• Analyze performance bottlenecks: precisely measure the time consumption of each step (e.g., model invocation, data retrieval) and detect latency promptly.
• Correlate upstream and downstream services: link Agent/LLM requests with related application and infrastructure metrics for comprehensive root cause analysis.

Core Capabilities¶

The core of Agent Monitoring is establishing a quantifiable relationship between the input (Prompt), output (Completion), and system behavior. Its capabilities are reflected in three dimensions:

1. Full Trace

In the Agent/LLM invocation framework, accurately trace the entire request chain using Traces and Spans, and pinpoint latency bottlenecks.

2. Quality Output Evaluation

Use Prompt Eval and Schema Eval to perform online or batch evaluation of model outputs, and view the scoring and rule-hit results.

3. Cost Measurement

Automatically collect and correlate the Token consumption (input/output breakdown), model type, and invocation parameters for each request, providing multi-business-dimension cost allocation.

Getting Started¶

Application List¶

Go to the Application List to create and manage monitoring applications. You can choose to create a new Agent Monitoring Application or LLM Monitoring Application. Agent Monitoring applications support Codex, Claude Code, OpenClaw, Hermes, Qoder, WorkBuddy, OpenCode, and AgentScope. AgentScope is manually instrumented via OpenTelemetry, while other types use the obs-agent-connector installer. LLM Monitoring applications can be integrated via Langfuse. After completing the parameter configuration in the page guide, data collection begins.

The Summary page allows you to customize the Agent types displayed using "Display Items" and adjust the display order of each type.

Explorer¶

After data is ingested, you can search and filter data by Session or Trace dimension in the Explorer:

• User List: Aggregates user activity by user_id, showing Agent type, instance, Session, request volume, Token or credit consumption, and last active time.
• Session List: Displays Session ID, input/output Token count, risk event count, and alert level. Drill down to the detail page to view the Trace waterfall chart, model/Skill/Tool call proportions, call details, and risk events.
• Trace List: Displays Trace ID, associated Session, duration, Token count, status, risky operation count, and alert level. Drill down to view Span details, risky operations, tool call records, and Skill call records.

Analysis Dashboard¶

Use the Analysis Dashboard to get an overview of application health through charts. You can filter by application type, and the dashboard content differs by type:

• LLM Monitoring Application: Displays overview metrics such as request count, Span count, request error rate, total Token consumption, average response time, as well as request trend chart, Token consumption trend chart, request count proportion by model, and Token usage ranking by model.

• Agent Monitoring Application: Displayed in four modules: Run Overview, Model & Token, Tool Module, and Skill Module.

  • Run Overview: Total operations, total Tool, LLM, and Skill invocations, total Token count, and distribution of model, Tool, and Skill calls.
  • Model & Token: LLM call trend, input/output Token trend, Token trend by model, and LLM call duration percentiles.
  • Tool Module: Active Tool count, max duration, call trend, average duration, and Top 10 anomalies.
  • Skill Module: Active Skill count, max duration, call trend, average duration, and Top 10 anomalies.