Trace Details¶
In the Trace Explorer, click on any trace to view its details, including relative time, duration, HTTP method, URL, status code, TraceId, flame graph, Span list, service call relationships, and associated logs, hosts, metrics, and network information. For frontend applications (such as browsers), you can also view the request time distribution, including the time proportions for queuing, first byte, and download.
Note
- The RUM SDK must be version 2.2.10 or higher to see this part of the data display. If cross-origin issues exist, header configuration adjustments are needed.
For more details, refer to Web Application Integration.
- The flame graph/ Span list/ waterfall chart can display up to 10,000 Spans. You can view unshown Spans by setting the Offset:
Flame Graph¶
The flame graph clearly shows the flow and execution time of each Span. The right side displays the service list and response time. Clicking on a Span allows you to view its JSON content in the trace details. Use the mouse wheel to zoom in and out for specific information.
For more details, refer to Leveraging Flame Graphs for Trace Performance Analysis.
From the flame graph above, it can be seen that this call trace involves two services: CloudCare and Mysql. The trace starts with a POST request to the CloudCare service, then executes ActionResource.executeAction, and finally executes Mysql statements. During the execution of ActionResource.executeAction, Mysql statements are executed multiple times. The execution times for CloudCare and Mysql are calculated as follows:
- CloudCare service execution time = D1+D2+D3+D4+D5+D6+D7+D8+D9+D10+D11
- Mysql service execution time = span2+span3+......+span11
Details of the specific executed statements and their execution times can be found in the Span list.
The service execution time percentage in the flame graph refers to the ratio of each service's time to the total time of this call trace. In the example below, this call trace involves two services, CloudCare and Mysql, with execution time percentages of 42.37% and 57.63% respectively. The CloudCare service has 2 Spans, and the Mysql service has 10 Spans.
- Mysql service execution time percentage calculation: Sum of all span execution times / Total duration of the current call trace.
Calculation explanation: The Mysql service in the figure below has a total of 10 Spans. Click on each Span to get its execution time. From the figure, this Span's execution time is 5.08ms. Similarly, get and sum the execution times of the remaining 9 Spans.
- CloudCare service execution time percentage calculation: (Total duration of the current call trace - Mysql service execution time) / Total duration of the current call trace
Calculation explanation: The CloudCare service in the figure below spans the entire call trace. The time remaining after subtracting the Mysql service execution time is the CloudCare service execution time (see the execution time marked by the red line). The execution time percentage can also be directly viewed in the Span list for each Span's execution time and its percentage.
In the flame graph, whether services are called synchronously or asynchronously, every detail of the trace performance data can be clearly tracked. For example, the flame graph can clearly show which requests are asynchronous, their start time, end time, and total time spent.
Span List¶
- Displays the resource name, count, duration, execution time, and percentage for all Spans in this trace.
- You can search by entering the resource name or Span ID.
- Click on any Span to view its JSON content in the trace details; switching to the flame graph will show it synchronously.
- If there are errors, error prompts will be displayed.
- Click Error Spans to directly display the filtered results.
Waterfall Chart¶
You can view the parent-child relationships between various resources.
The waterfall chart displays Span data in chronological order of their start times. The left list shows Span data, displaying the execution time percentage for each resource. The right side shows the waterfall chart in chronological order.
- Search by entering the resource name or Span ID corresponding to the Span;
- Collapse or expand resources on the left with one click;
-
Click to switch the format of the execution time;
-
Click Error Spans to directly display the filtered results.
Service Call Relationships¶
Provides an overview of the call relationships and call counts between different services, as well as the execution time, average duration, and call count for a single service.
Click on any service node to view its detailed information:
-
Service Operations and Resources: Lists the specific operations (such as API interfaces, methods, etc.) contained in this service and related resources.
-
Error Status Tracking:
-
If the service has errors, the error details will be displayed directly.
-
Click to view the parent Span of the service, used to trace the upstream call source.
-
Note
The system displays the service color here based on the error results corresponding to the service statistics in the trace details > Flame Graph.
Quick Actions¶
Action |
Description |
|---|---|
| Full Screen View/Restore Default Size | You can click the full-screen view icon in the upper right corner of the trace details to horizontally expand and view the trace flame graph; click the restore default size icon to return to the details page. |
| Locate to Current Span | Click to zoom the view and locate the current Span |
| Expand/Collapse Mini Map | You can click the expand/collapse mini map icon on the left side of the trace details. Use the mini map to select intervals, drag, and scroll to quickly view the flame graph. |
| View Global Trace | You can click the view global trace icon on the left side of the trace details to view the global trace in the flame graph. |
| Double-click Span | Zooms in on the Span in the middle of the flame graph, allowing you to quickly locate and view its contextually related Spans. |
| Click Service Name | Highlights the corresponding Span. Click the service name again to restore the default selection of all Spans. You can quickly filter and view the Spans corresponding to a service by clicking its name. |
Extended Attributes¶
-
In the search bar, you can enter a field name or value to quickly search and locate;
-
After checking the field alias, you can view it after the field name; you can select as needed.
-
On the trace details page, you can view the relevant field attributes of the current trace in Extended Attributes:
Field |
Attribute |
|---|---|
| Filter Field Value | Adds this field to the explorer to view all data related to this field. You can filter and view the list of traces related to this field in the trace explorer. See Figure 1. |
| Reverse Filter Field Value | Adds this field to the explorer to view data excluding this field. |
| Add to Display Columns | Adds this field to the explorer list for viewing. |
| Copy | Copies this field to the clipboard. |
Some fields do not support filtering, aggregation, and related logic
Guance has some fields that use full-text indexing. These fields do not support filtering, aggregation, and related logic. The scope of full-text indexed fields is as follows:
| Category | Field |
|---|---|
| Object, Resource Catalog | message |
| Log, Backup Log | message |
| Security | message / title |
| Network | message |
| Trace | error_message / error_stack |
| Event | message / title / df_message / df_title |
| RUM Error | error_message / error_stack |
| RUM Long Task | long_task_message / long_task_stack |
Figure 1
Error Details¶
On the trace details page, if there are errors in the current trace, you can check the error span filter option in the sidebar. The system will automatically highlight all error Spans. Click on any highlighted Span to view its associated error stack, logs, and other detailed information.
For more details, refer to Error Tracking.
Service Context¶
By obtaining object classifications under the infrastructure resource catalog and displaying the latest object selected by create_time, you can quickly view the current service's runtime information, service dependencies, and integration information.
Correlation Analysis¶
You can view logs correlated with the current trace (correlation field: trace_id) via Logs. You can customize the display columns. If you need to view more detailed log content, you can click on the log content to jump to the log details page, or click the jump button to open it in the Logs page.
If you are an administrator or have higher permissions, you can customize the correlation fields. Click the settings button to the right of the correlation field, select the fields to correlate in the pop-up dialog box, support manual input, drag-and-drop ordering, etc., and confirm to complete the configuration.
Note: Custom fields for correlated logs and custom fields for Service List correlation analysis configuration affect each other. If custom fields are configured in the Service List, they are displayed here as well.
When the application uses the ddtrace collector with both APM trace collection and Profile performance trace data collection enabled, the system provides Span-level correlation to view code hotspots, including execution time, methods, and execution time percentage.
Click View Profile Details to jump to the Profile details page to view more associated code.
On the trace details page, you can view related host metric views and attribute views (correlation field: host) via Hosts.
- Metric View: You can view the performance metric status of related hosts from 30 minutes before the trace ended to 30 minutes after the trace ended, including CPU, memory, and other performance metric views for the related hosts.
- Attribute View: Helps you回溯 the real situation of the host object when the trace was generated. It supports viewing the latest piece of object data generated by the related host within the corresponding time, including basic host information and integration运行 status. If cloud host collection is enabled, cloud vendor information can also be viewed.
Note: Guance by default saves host object historical data from the last 48 hours. If no host historical data corresponding to the current trace time is found, you will be unable to view the attribute view of the correlated host.
On the trace details page, you can view related container metric views and attribute views (correlation field: container_name) via Containers.
-
Metric View: Supports viewing the performance metric status of related containers from 30 minutes before the trace ended to 30 minutes after the trace ended, including container CPU, memory, and other performance metric views.
-
Attribute View: Helps you回溯 the real situation of the container object when the trace was generated. It supports viewing the latest piece of object data generated by the related container within the corresponding time, including basic container information and attribute information.
On the trace details page, you can view related Pod attribute views and metric views (correlation field: pod_name) via Pods.
-
Metric View: Supports viewing the performance metric status of the related container Pod from 30 minutes before the trace ended to 30 minutes after the trace ended, including container CPU, memory, and other performance metric views.
-
Attribute View: Helps you回溯 the real situation of the container Pod object when the trace was generated. It supports viewing the latest piece of object data generated by the related container Pod within the corresponding time, including basic container information and attribute information.
Guance supports viewing network topology maps and overview data across multiple dimensions including Host, Pod, Deployment, and Service in Network.
Matching Fields:
To view the related network in the details page, the corresponding correlation fields need to be matched, meaning the corresponding field tags need to be configured during data collection. Otherwise, the correlated network view cannot be matched and viewed in the details page.
-
Host: Match field
host. -
Pod:
| Matching Field Priority |
|---|
| namespace, pod_name |
| namespace, pod |
| pod_name |
| pod |
- Deployment:
| Matching Field Priority |
|---|
| namespace, deployment_name |
| namespace, deployment |
| deployment_name |
| deployment |
- Service:
| Matching Field Priority |
|---|
| namespace, service_name |
| namespace, service |
Note:
- If correlation fields for Host, Pod, Deployment, and Service are all queried simultaneously, the network data will be displayed in this order upon entering the details page;
- If no correlation fields are found, it will be displayed as gray at the end, with a prompt No network view matched.