Agent Monitoring¶

Agent Monitoring associates Agent/LLM requests with the entire application trace, tracking the complete flow of each conversation and precisely measuring the Token consumption for each generation task.

In practical use of the Agent Monitoring service, you can:

• View the complete trace of a single request: Clearly see the entire process from receiving a user query, processing it (e.g., database query), to calling the LLM model and returning the answer.
• Analyze performance bottlenecks: Precisely measure the time consumption of each step (e.g., model calls, data retrieval) to promptly identify delays.
• Correlate upstream and downstream services: Associate Agent/LLM requests with related application and infrastructure metrics for comprehensive root cause analysis.

Core Capabilities¶

The most crucial aspect of Agent Monitoring is establishing a quantifiable link between inputs (Prompts), outputs (Completions), and system behavior. Its core capabilities are reflected in three dimensions:

1. Full Trace Tracking

Within the Agent/LLM invocation framework, precisely track the entire request trace using Traces and Spans to locate latency bottlenecks.

2. Quality Output Evaluation

Internally optimizes output content automatically based on rule engines and AI evaluation.

3. Cost Measurement

Automatically collects and associates the Token consumption (input/output breakdown), model type, and invocation parameters for each request, providing cost allocation capabilities based on multiple business dimensions.

Getting Started¶

Service List¶

Go to the Service List to create and manage monitoring applications. You can choose to create a new Agent Monitoring Application or LLM Monitoring Application. Currently, Langfuse and OpenClaw integration frameworks are supported by default. After defining the application name and ID, the system generates configuration parameters and a Client Token. Follow the instructions to complete the integration configuration for Python, JS/TS, or other frameworks, and data collection will begin.

Explorer¶

After data integration, you can search and filter data in the Explorer by Session or Trace dimensions:

• Session List: View Session ID, input/output Tokens, and drill down to the details page to see the Trace waterfall chart, model/Skill/Tool invocation proportions, and invocation details.
• Trace List: View Trace ID, associated Session, duration, Tokens, and status, and drill down to see Span details, tool invocation records, and Skill invocation records.

Analysis Dashboard¶

Use the Analysis Dashboard to get an overview of application operation status in chart form. Supports filtering by application type for viewing. The content displayed on the analysis dashboard differs for different application types:

• LLM Monitoring Application: Displays overview metrics such as request count, Span count, request error rate, total Token consumption, average response time, as well as request trend charts, Token consumption trend charts, request proportion by model, and Token usage ranking by model.

• Agent Monitoring Application: Displays data in four modules: Request, Model, Skill, and Tool.

  • Request Module: Total request trend, average request duration trend.
  • Model Module: Model usage ranking, average request duration trend, Token consumption ranking.
  • Skill Module: Skill usage ranking, average request duration trend.
  • Tool Module: Tool invocation ranking, average/maximum/minimum duration.