Skip to content

Agent Monitoring


Agent Monitoring connects Agent / LLM requests with the entire application trace, tracks the complete flow of each conversation, and accurately measures the number of Tokens consumed by each generation task.

When using Agent Monitoring, you can:

  • View the complete trace of a single request: clearly see the entire process of a user question, from receipt and processing (such as database queries) to calling the LLM and returning an answer
  • Analyze performance bottlenecks: accurately measure the time spent in each stage (such as model calls and data retrieval) and detect latency in time
  • Correlate upstream and downstream services: correlate Agent / LLM requests with related application and infrastructure metrics for comprehensive root cause analysis

Core Capabilities

The core of Agent Monitoring is to establish a quantifiable relationship between inputs (Prompt), outputs (Completion), and system behavior. Its core capabilities are reflected in three dimensions:

1. End-to-End Tracing

In Agent / LLM invocation frameworks, use Trace and Span to accurately trace the entire request path and identify latency bottlenecks.

2. Output Quality Evaluation

Use Prompt Eval and Schema Eval to evaluate model outputs online or in batches, and view scores and rule match results. Prompt Eval supports describing evaluation objectives to generate configurations, and selecting the context to pass in through custom evaluation inputs. It supports reviewing outputs through manual annotation and adding samples to datasets for centralized management.

3. Cost Measurement

Automatically collect and correlate the Token consumption (input/output breakdown), model type, and invocation parameters of each request, providing cost allocation based on multiple business dimensions.

Get Started

App List

Go to App List to create and manage monitoring apps. Agent Monitoring includes two integration types: Agent and LLM. The Agent type is further divided into Agent Apps and Agent Frameworks. The integration method for each type is as follows:

  • Agent Apps: Supports Codex, Claude Code, OpenClaw, Hermes, Qoder, WorkBuddy, OpenCode, CodeBuddy, Cursor, DeepSeek Harness, Opm, mimo, pi, and Qwen Work (desktop). For integration methods for each type, see Create an Agent Monitoring App;
  • Agent Frameworks: Supports AgentScope, Spring AI Alibaba, and Deep Agents, with manual integration through OpenTelemetry;
  • LLM Monitoring Apps: Supports integration through Langfuse.

After completing the parameter configuration in the page guide, you can start collecting data.

On the Summary page, you can use "Display Items" to customize the Agent types to display and adjust their display order.

If you need to view usage information for Codex, Claude Code, and WorkBuddy in a unified desktop interface, or manage basic collection and enhanced collection through a graphical interface, see the AgentMeter Usage Guide. The usage metrics supported by different Agents are subject to the instructions in the guide.

Explorer

After data is integrated, you can query and analyze Agent data from the User, Session, Trace, or Span view in the Explorer:

  • User List: Aggregates user usage by user_id, showing Agent type, instance, Session, request volume, Token or credit consumption, and last active time;
  • Session List: View Session ID, input/output Tokens, number of risk events, and alert level. Drill down to the details page to view the Trace Waterfall Chart, model/Skill/Tool call proportions, call details, and risk events;
  • Trace List: View Trace ID, associated Session, duration, status, number of risky operations, and alert level. Drill down to view Span details, risky operations, tool call records, and Skill call records.
  • Span List: View execution steps across Traces. Select Root Span to focus on execution entry points, or select All Spans to analyze model, tool, and skill calls.

Analysis Dashboard

Use the Analysis Dashboard to get a chart-based summary of application health. You can filter by application type; the content displayed on the Analysis Dashboard varies by application type:

  • LLM Monitoring Apps: Display overview metrics such as request count, Span count, request error rate, total Token consumption, and average response time, as well as request trends, Token consumption trends, request share by model, and Token usage rankings by model.

  • Agent Monitoring Apps: Displayed in four modules: Runtime Overview, Models & Tokens, Tool Module, and Skill Module.

    • Runtime Overview: Total operations, total Tool, LLM, and Skill calls, total Tokens, and the distribution of model, Tool, and Skill calls
    • Models & Tokens: LLM call trends, input/output Token trends, Token trends by model, LLM call duration percentiles, and average time to first Token (TTFT)
    • Tool Module: Number of active Tools, maximum duration, call trends, average duration, and Top 10 anomalies
    • Skill Module: Number of active Skills, maximum duration, call trends, average duration, and Top 10 anomalies

Feedback

Is this page helpful?