Trace Explorer¶
After data collection is successful, you can view all related data on the tracing feature page. Using the powerful search function, you can easily filter and view trace data for any time period, quickly identifying and locating abnormal traces.
The flame graph provides an intuitive view to observe the flow and performance of each Span in the trace. Additionally, Guance's tagging feature allows you to automatically correlate infrastructure, logs, RUM PV, and other data with trace analysis, supporting code-level data association views to help you quickly locate and resolve issues. Tools like the Span list and waterfall chart further display related Span information for a specific trace.
Trace List¶
Guance provides three types of trace lists: All Spans, Service Entry Spans, and All Traces.
A Span represents a logical unit of work within a distributed system over a given time period, and multiple Spans form a trace trajectory (trace).
Displays all collected Span data within the currently selected time range.
Filters and displays all Spans generated by service entry calls within the currently selected time range.
Filters and displays all top-level entry Spans of trace calls within the currently selected time range.
Query and Analysis¶
-
Time Widget: The Trace Explorer defaults to displaying data from the last 15 minutes, but you can also customize the time range.
-
In the Trace Explorer search bar, various search methods and filtering options are supported.
-
Note: When switching between the Service or Trace Explorer, Guance retains your current filtering conditions and time range by default.
-
Analysis Mode: In the analysis panel of the Trace Explorer, you can perform multi-dimensional statistical analysis based on 1-3 labels and support various data chart analysis methods.
-
Quick Filters: Editing quick filters allows you to add new filter fields.
-
Filter History: Guance supports saving
key:valuesearch condition history for different Explorers within the current workspace via filter history. -
Custom display columns: By default, the trace list shows Time, Trace ID, Service, Resource, and Duration, which can be customized, edited, deleted, or dragged to adjust display columns.
-
Guance supports viewing historical snapshots directly by saving the current Explorer snapshot data. Through the snapshot feature, you can quickly reproduce instant copies of data and restore it to a specific point in time and data display logic.
Chart Statistics¶
In the Trace Explorer chart statistics, you can view the number of requests, Error request count, and response time for different states within the selected time range. You can also synchronize the display of chart statistics through filtering.
- Number of Requests/Error Requests: Divided into 60 time points based on the selected time range, displayed as a bar chart showing the number of requests and Error requests.
- Response Time: Divided into 60 time points based on the selected time range, displayed as a line chart showing four response metrics: average response time per second, P75 response time, P90 response time, and P99 response time.
Trace Data Export¶
In the Trace Explorer, after filtering and viewing the desired trace data for analysis, you can export it to a CSV file or scene dashboard and notes.
To export a particular data point, open the details page of that data point and click the icon in the upper right corner.
Trace Details¶
In the Trace Explorer, you can click on any trace to view its details, including relative time, duration, HTTP method, HTTP URL, HTTP status code, TraceId, flame graph, Span list, service call relationships, and associated logs, hosts, metrics, networks, etc.
If the current trace belongs to a frontend application (e.g., browser), you can view the request latency distribution in the trace details, including Queueing (queue), First Byte (first byte), Download (download) request latency ratios.
Note: Flame graphs/Span lists/waterfall charts display up to 10,000 Spans; you can use the offset setting to view un-displayed Spans:
Warning
The RUM SDK must be version 2.2.10 or higher to view this data. If there is a cross-origin situation, you need to adjust the header configuration.
For more details, refer to Web Application Integration.
Flame Graph¶
The flame graph clearly displays the flow and execution time of each Span in the entire trace. You can view the corresponding service list and response times on the right side of the flame graph. Clicking a Span in the flame graph allows you to view the corresponding JSON content in the Trace Details section. Zoom in and out using the mouse scroll wheel to view specific Span information.
For more details on using flame graphs for trace performance analysis, refer to Using Flame Graphs to Analyze Trace Performance.
From the flame graph above, we can see that this call chain includes two services: CloudCare and Mysql. The trace starts with a POST request initiated by the CloudCare service, then executes ActionResource.executeAction, and finally executes MySQL statements. During the execution of ActionResource.executeAction, MySQL statements are executed multiple times. The execution times for CloudCare and Mysql services are calculated as follows:
- CloudCare service execution time = D1 + D2 + D3 + D4 + D5 + D6 + D7 + D8 + D9 + D10 + D11
- Mysql service execution time = span2 + span3 +......+ span11
Specific execution statements and execution times can be referenced in the Span list.
The service execution time ratio in the flame graph refers to the proportion of each service's execution time in the total time of this call chain. As shown in the figure below, this call chain includes two services: CloudCare and Mysql, with execution time ratios of 42.37% and 57.63%, respectively. CloudCare service has 2 Spans, and Mysql service has 10 Spans.
- Mysql service execution time ratio calculation method: sum of all Span execution times / total duration of the current call chain.
Calculation explanation: In the figure below, Mysql service has a total of 10 Spans. You can click each Span to obtain the execution time of the current Span. From the figure, we can see that the execution time of this Span is 5.08ms. Then, similarly, obtain the execution time of the remaining 9 Spans and sum them up.
- CloudCare service execution time ratio calculation method: (total duration of the current call chain - Mysql service execution time) / total duration of the current call chain.
Calculation explanation: In the figure below, the CloudCare service spans the entire current call chain. Except for the execution time of the Mysql service, the remaining time is the execution time of the CloudCare service (see the red line part of the execution time). The execution time ratio can also be viewed directly through the Span list, which shows the execution time and execution time ratio of each Span.
In the flame graph, whether services are synchronous or asynchronous calls, each trace performance data detail can be clearly tracked. For example, the flame graph can clearly show which requests are asynchronous, their start and end times, and the total time spent.
Span List¶
Displays all Span lists and the total number of Spans in this trace, including resource name, Span count, duration, execution time, and execution time ratio.
You can search for matching Spans by entering the resource name or Span ID. Clicking any Span allows you to view the corresponding JSON content in the Trace Details section and switch to the flame graph to display the Span synchronously. If there is an error, an error message will be displayed.
Clicking Error Spans directly displays the filtered results.
Waterfall Chart¶
Shows parent-child relationships between resources.
The waterfall chart displays Span data in chronological order of start time. On the left side, Span data is listed along with the execution time ratio of each resource. On the right side, the waterfall chart is displayed chronologically.
-
You can input the resource name or Span ID corresponding to the Span for search matching;
-
Click to switch the format of execution time;
-
Clicking Error Spans directly displays the filtered results.
Service Call Relationships¶
Used to view call relationships between services and directly display call counts. You can search and filter related service call relationships through services, resources, and Span IDs.
Guance bases the color of services on the error results from the flame graph inside the trace details. If a service appears red, it indicates that the service has errors.
If you configure a service binding relationship in the user view, such as service:mysql, clicking the service card here will quickly view related user views associated with that service.
Clicking a view will redirect you to its details page.
Quick Actions¶
Action |
Description |
|---|---|
| Fullscreen View/Restore Default Size | You can click the fullscreen view icon in the upper-right corner of the trace details to expand the flame graph horizontally; clicking the restore default size icon restores the details page. |
| Expand/Collapse Minimap | You can click the expand/collapse minimap icon on the left side of the trace details to quickly view the flame graph by selecting intervals, dragging, or scrolling in the minimap. |
| View Global Trace | You can click the view global Trace icon on the left side of the trace details to view the global trace in the flame graph. |
| Double-click Span | Double-clicking a Span in the flame graph enlarges and displays the Span, allowing you to quickly locate and view contextually related Spans. |
| Click Service Name | Highlights the corresponding Span. Clicking the service name again restores the default selection of all Spans. You can quickly filter and view Spans corresponding to the service by clicking the service name. |
Extended Attributes¶
In the search bar, you can input field names or values for quick search and positioning;
After checking the alias of the field, you can view it after the field name. Choose as needed.
In the trace details page, you can view the relevant field attributes of the current trace under Extended Attributes:
Field |
Attribute |
|---|---|
| Filter Field Value | Adds this field to the Explorer to view all data related to this field, which can be filtered in the Trace Explorer. See Figure 1. |
| Reverse Filter Field Value | Adds this field to the Explorer to view data excluding this field. |
| Add to Display Columns | Adds this field to the Explorer list for viewing. |
| Copy | Copies this field to the clipboard. |
Some Fields Do Not Support Aggregation Logic
Guance has some fields indexed as full-text, which do not support filtering or aggregation logic. The full-text indexed fields include:
| Category | Field |
|---|---|
| Object, Resource Catalog | message |
| Logs, Backup Logs | message |
| Security | message / title |
| Network | message |
| Trace | error_message / error_stack |
| Events | message / title / df_message / df_title |
| RUM Errors | error_message / error_stack |
| RUM Long Tasks | long_task_message / long_task_stack |
Figure 1
Error Details¶
On the trace details page, if there is an error trace, you can view the related error details.
For more error trace analysis, refer to Error Tracking.
Service Context¶
By obtaining object categories from the infrastructure resource catalog and selecting the latest object based on create_time, you can quickly view the current service's runtime information, service dependencies, and integration information.
Correlation Analysis¶
You can view logs associated with the current trace (correlation field: trace_id). You can customize display columns, and if you need to view more detailed log content, you can click the log content to jump to the log details page or click the jump button to open the log page.
If you have administrator or higher permissions, you can customize correlation fields. Click the settings button next to the correlation field, choose the fields you need to correlate in the pop-up dialog box, and confirm the configuration. Operations such as manual input and drag-and-drop sorting are supported.
Note: Customizing correlation fields for correlated logs and customizing correlation fields for service list correlation analysis affect each other. If you configure custom fields in the service list, they will be displayed here.
When the application uses ddtrace collector and enables both APM trace tracking and Profile performance tracking data collection, Guance provides Span-level correlation views. On the trace details page, you can click Code Hotspots below the flame graph to view the code hotspots associated with the current trace, including execution latency, methods, and execution time ratios.
Clicking View Profile Details redirects you to the Profile details page to view more associated code.
On the trace details page, you can view the metric and attribute views of related hosts (correlation field: host).
- Metric View: View the performance metrics state of related hosts from 30 minutes before the trace ends to 30 minutes after the trace ends, including CPU, memory, and other performance metric views of the host.
- Attribute View: Helps you trace back the real-time situation of the host object when the trace was generated, supporting the viewing of the latest object data generated by the host within the corresponding time, including basic host information and integration runtime conditions. If cloud host collection is enabled, you can also view cloud provider information.
Note: Guance saves the most recent 48 hours of host object historical data by default. If no host historical data corresponding to the current trace time is found, you will not be able to view the attribute view of the associated host.
On the trace details page, you can view the metric and attribute views of related containers (correlation field: container_name).
-
Metric View: Supports viewing the performance metrics state of related containers from 30 minutes before the trace ends to 30 minutes after the trace ends, including container CPU, memory, and other performance metric views.
-
Attribute View: Helps you trace back the real-time situation of the container object when the trace was generated, supporting the viewing of the latest object data generated by the container within the corresponding time, including basic container information and attribute information.
On the trace details page, you can view the attribute and metric views of related Pods (correlation field: pod_name).
-
Metric View: Supports viewing the performance metrics state of related Pod containers from 30 minutes before the trace ends to 30 minutes after the trace ends, including container CPU, memory, and other performance metric views.
-
Attribute View: Helps you trace back the real-time situation of the Pod container object when the trace was generated, supporting the viewing of the latest object data generated by the Pod container within the corresponding time, including basic container information and attribute information.
Guance supports you in viewing multi-dimensional network topology and overview data including Host, Pod, Deployment, and Service.
Matching Fields:
To view related network data in the details page, you need to match the corresponding correlation fields, i.e., configure the corresponding field tags during data collection. Otherwise, you cannot match and view related network views in the details page.
-
Host: Match field
host. -
Pod:
| Priority of Matching Fields |
|---|
| namespace, pod_name |
| namespace, pod |
| pod_name |
| pod |
- Deployment:
| Priority of Matching Fields |
|---|
| namespace, deployment_name |
| namespace, deployment |
| deployment_name |
| deployment |
- Service:
| Priority of Matching Fields |
|---|
| namespace, service_name |
| namespace, service |
Note:
- If Host, Pod, Deployment, and Service correlation fields are found simultaneously, network data is displayed in this order upon entering the details page;
- If no correlation fields are found, they are displayed at the bottom in gray, and clicking prompts No network view matched.