Trace Explorer¶
After successful data collection, you can view all related data on the trace function page. Using the powerful search feature, you can easily filter and view trace data for any time period, quickly identifying and locating abnormal traces.
Flame Graphs provide you with an intuitive view to observe the flow and performance of each Span in a trace. Additionally, Guance's tagging feature allows you to automatically associate infrastructure, logs, user access monitoring, and other data with trace analysis, supporting code-level data association views that help you quickly locate and resolve abnormal issues. Tools like Span Lists, Waterfall Charts, etc., further display Span information related to a specific trace.
Trace List¶
Guance provides three types of trace viewing lists: All Spans, Service Entry Spans, and All Traces.
A Span represents a logical unit of work within a given time period in a distributed system, and multiple Spans form a trace trajectory (trace).
Displays all Span data collected within the currently selected time range.
Filters and displays all Spans generated by service entry calls within the currently selected time range.
Filters and displays all top-level entry Spans for trace calls within the currently selected time range.
Manage List¶
Chart Statistics¶
In the chart statistics of the trace explorer, you can view key Metrics for different states of traces within the selected time range:
- Number of Requests
- Error Request Count
- Response Time
These Metrics can be displayed synchronously in the charts via filtering features. Specific statistical methods are as follows:
- Number of Requests/Error Request Count: Based on the selected time range, the system divides it into 60 time points and displays the number of trace requests and error request counts through bar charts.
- Response Time: Similarly divided into 60 time points based on the selected time range, response time is displayed via line charts showing four response time Metrics, including average response time per second, P75 response time, P90 response time, and P99 response time.
Export¶
After filtering out the required trace data, you can export the data in the following formats for viewing and analysis:
- CSV Files
- JSONL Files
- Export to Use Cases > Dashboard/Notebook
If you need to export a particular piece of data, open the detailed page of that data and click the icon in the upper right corner.
Trace Details¶
In the trace explorer, by clicking any trace, you can view corresponding trace details, including relative time of the current trace occurrence, duration, HTTP method, HTTP URL, HTTP status code, TraceId, flame graph, Span list, service call relationships, and associated logs, hosts, Metrics, networks, etc.
If the current trace belongs to a frontend application (e.g., browser), you can view the distribution of request durations in the trace details, including Queueing (queue), First Byte (first packet), Download (download) request duration proportions.
Note: The User Analysis SDK must be version 2.2.10 or higher to see this data display, and if there are cross-origin situations, header configurations need to be adjusted.
For more details, refer to Web Application Integration.
Flame Graph¶
The flame graph clearly shows the flow and execution time of each Span in the entire trace. You can view the corresponding service list and response times on the right side of the flame graph. By clicking a Span in the flame graph, you can view the corresponding Json content in the Trace Details, and zoom in/out using the mouse scroll wheel to check specific Span information.
For more details about flame graphs, refer to Using Flame Graphs to Analyze Trace Performance.
From the flame graph above, we can see that this call trace includes two services, CloudCare and Mysql. The trace starts with a POST request initiated by the CloudCare service, then executes ActionResource.executeAction, and finally executes Mysql statements during the entire process of executing ActionResource.executeAction. The execution time calculation methods for CloudCare and Mysql are as follows:
- CloudCare service execution time = D1+D2+D3+D4+D5+D6+D7+D8+D9+D10+D11
- Mysql service execution time = span2+span3+......+span11
For specific execution statements and execution time details, refer to the Span list.
The service execution time proportion in the flame graph refers to the ratio of each service's execution time to the total time in this call trace. As shown in the figure below, this call trace includes two services, CloudCare and Mysql, with execution time proportions of 42.37% and 57.63%, respectively. CloudCare service has 2 Spans, and Mysql service has 10 Spans.
- Mysql service execution time proportion calculation method: sum of all span execution times / total duration of the current call trace.
Explanation of calculation method: In the figure below, Mysql service has a total of 10 Spans. You can click each Span to get its execution time, as shown in the figure where the execution time of this Span is 5.08ms. Then, use the same method to obtain and add up the execution times of the remaining 9 Spans.
- CloudCare service execution time proportion calculation method: (total duration of the current call trace - Mysql service execution time) / total duration of the current call trace.
Explanation of calculation method: In the figure below, CloudCare service spans throughout the entire current call trace. Except for the Mysql service execution time, the remaining time is the CloudCare service execution time (see the marked red line part of the execution time). The execution time proportion can also be directly viewed from the Span list for each Span’s execution time and proportion.
In the flame graph, whether services are synchronous or asynchronous calls, it can clearly track the performance data details of each trace. For example, through the flame graph, you can clearly see which requests were performed asynchronously, their start time, end time, and total time spent.
Span List¶
Displays the Span list and total Span count for this trace, including Resource Name, Span Count, Duration, Execution Time, and Execution Time Proportion.
You can input the resource name or Span ID corresponding to the Span for matching searches. Clicking any Span allows you to view the corresponding JSON content in the Trace Details and switch to the flame graph to simultaneously display the Span. If there are errors, an error message will be displayed.
Clicking Error Spans directly displays the results of the filtered hits.
Waterfall Chart¶
Used to view parent-child relationships between various resources.
The waterfall chart displays Span data according to the order of start times. On the left side, the Span data is displayed along with the execution time proportion of each resource. On the right side, the waterfall chart is displayed according to the chronological order.
-
You can input the resource name or Span ID corresponding to the Span for matching searches;
-
Clicking switches the format of the execution time;
-
Clicking Error Spans directly displays the results of the filtered hits.
Service Call Relationships¶
Used to view the call relationships between services and directly display the call counts. You can search and filter to view related service call relationships through services, resources, and Span IDs.
Guance displays the color of services here based on the error results statistics in the Flame Graph. If red appears, it indicates that the service has an error.
If you configure the binding relationship of service in the user view, such as service:mysql, clicking the service card here allows you to quickly view the related user views associated with that service.
Clicking a particular view redirects you to its detail page.
Quick Actions¶
Action |
Description |
|---|---|
| Fullscreen View/Restore Default Size | You can click the fullscreen view icon in the upper-right corner of the trace details to horizontally expand and view the trace flame graph; clicking the restore default size icon restores the detail page. |
| Expand/Collapse Minimap | You can click the expand/collapse minimap icon on the left side of the trace details, allowing you to quickly view the flame graph by selecting intervals, dragging, or scrolling on the minimap. |
| View Global Trace | You can click the view global Trace icon on the left side of the trace details to view the global trace in the flame graph. |
| Double-click Span | Amplifies the display of that Span in the middle of the flame graph, allowing you to quickly locate and view its context-related Spans. |
| Click Service Name | Highlights the corresponding Span; clicking the service name again selects all Spans by default. You can quickly filter and view Spans corresponding to the service by clicking the service name. |
Extended Attributes¶
In the search bar, you can input field names or values to quickly search and locate;
After checking field aliases, you can view them after the field names; select as needed.
In the trace details page, you can view related field attributes of the current trace under Extended Attributes:
Field |
Attribute |
|---|---|
| Filter Field Value | Adds the field to the Explorer to view all data related to that field, allowing you to filter and view related trace lists in the trace explorer. See Figure One. |
| Reverse Filter Field Value | Adds the field to the Explorer to view all data except for that field. |
| Add to Display Columns | Adds the field to the Explorer list for viewing. |
| Copy | Copies the field to the clipboard. |
Some fields do not support filtering and aggregation logic
Guance uses full-text indexing for some fields, and these fields do not support filtering and aggregation logic. The scope of full-text indexed fields is as follows:
| Category | Field |
|---|---|
| Objects, Resource Catalog | message |
| Logs, Backup Logs | message |
| Security | message / title |
| Networks | message |
| Traces | error_message / error_stack |
| Events | message / title / df_message / df_title |
| RUM Errors | error_message / error_stack |
| RUM Long Tasks | long_task_message / long_task_stack |
Figure One
Error Details¶
On the trace details page, if there are erroneous traces, you can view related error details.
For more error trace analysis, refer to Error Tracking.
Service Context¶
By obtaining object classifications from the infrastructure resource catalog and selecting the latest object based on create_time for display, you can quickly view the current service's runtime information, service dependencies, and integration information.
Correlation Analysis¶
You can view logs associated with the current trace (associated field: trace_id) through Logs. You can customize display columns, and if you need to view more detailed log content, you can click the log content to navigate to the log details page or click the jump button to open it on the log page.
If you have administrator-level or higher permissions, you can customize correlation fields. Click the settings button to the right of the correlation field, choose the fields you need to correlate in the pop-up dialog box, and support manual input, drag-and-drop ordering, etc. Confirm to complete configuration.
Note: Customized correlation fields for logs and customized correlation fields for service list analysis influence each other. If custom fields are configured in the service list, they will be displayed here synchronously.
When the application uses ddtrace collector and both APM trace tracking and Profile performance tracking data collection are enabled simultaneously, Guance provides Span-level correlation views. On the trace details page, you can click Code Hotspots below the flame graph to view code hotspots associated with the current trace, including execution time, methods, and execution time proportions.
Clicking View Profile Details navigates to the Profile details page to view more associated code.
On the trace details page, you can view metric views and property views of related hosts through Hosts (associated field: host).
- Metric View: Can view the performance metric status of related hosts 30 minutes before the end of the trace to 30 minutes after the trace ends, including CPU, memory, and other performance metric views of related hosts.
- Property View: Helps you trace back to the real situation of host objects when the trace was generated, supporting viewing of the latest object data produced by related hosts within the corresponding time, including basic host information, integration runtime conditions. If cloud host collection is enabled, you can also view cloud provider information.
Note: Guance defaults to saving the last 48 hours of historical data for host objects. If no historical data corresponding to the current trace time for the host is found, you will not be able to view the property view of the associated host.
On the trace details page, you can view metric views and property views of related containers through Containers (associated field: container_name).
-
Metric View: Supports viewing the performance metric status of related containers from 30 minutes before the end of the trace to 30 minutes after the trace ends, including container CPU, memory, and other performance metric views.
-
Property View: Helps you trace back to the real situation of container objects when the trace was generated, supporting viewing of the latest object data produced by related containers within the corresponding time, including basic container information, attribute information.
On the trace details page, you can view the property view and metric view of related Pods through Pods (associated field: pod_name).
-
Metric View: Supports viewing the performance metric status of related Pod containers from 30 minutes before the end of the trace to 30 minutes after the trace ends, including container CPU, memory, and other performance metric views.
-
Property View: Helps you trace back to the real situation of Pod container objects when the trace was generated, supporting viewing of the latest object data produced by related container Pods within the corresponding time, including basic container information, attribute information.
Guance supports you to view multi-dimensional network topology diagrams and summary data including Host, Pod, Deployment, and Service in Network.
Matching Fields:
To view related networks in the details page, matching corresponding association fields is required, i.e., configuring corresponding field tags during data collection. Otherwise, you cannot match and view related network views in the details page.
-
Host: Match field
host. -
Pod:
| Matching Field Priority |
|---|
| namespace、pod_name |
| namespace、pod |
| pod_name |
| pod |
- Deployment:
| Matching Field Priority |
|---|
| namespace、deployment_name |
| namespace、deployment |
| deployment_name |
| deployment |
- Service:
| Matching Field Priority |
|---|
| namespace、service_name |
| namespace、service |
Note:
- If associated fields for Host, Pod, Deployment, and Service are found simultaneously, network data is displayed in this order upon entering the details page;
- If no associated fields are found, they are displayed at the bottom in gray, and clicking prompts No network view matched.