Service Data Source Definitions and DQL Queries¶
Service Performance Data Source Definition¶
The TM index space stores service lists and performance metric data. The data for the Service view in APM > Performance Metrics is mainly queried from this index space. TM aggregates service metric data for each service at three granularities—minute, hour, and day—to improve query efficiency.
For example, to query all service metric data within 15 minutes from 2024-03-19 15:00:00 to 2024-03-19 15:15:00, you can use DQL:
The query returns results similar to the following:
Sample query result
[
{
"time": 1710835681000,
"time_us": 1710835681000000,
"__docid": "T_cnskg71jdosvib6m44s0",
"__source": "service_list_1m",
"source": "service_list_1m",
"__namespace": "tracing",
"r_env": "demo",
"r_error_count": 0,
"r_max_duration": 2293857,
"r_psketch": "Av1KgVq/UvA/AAAAAAAAAAAJAbgL",
"r_request_count": 1,
"r_resp_time": 2293857,
"r_service": "go-profiling-demo-1",
"r_service_sub": "go-profiling-demo-1:demo:v0.8.888",
"r_type": "custom",
"r_version": "v0.8.888",
"create_time": 1710835740447,
"date": 1710835681000,
"date_ns": 0
},
{
"time": 1710835201000,
"time_us": 1710835201000000,
"__docid": "T_cnskcf9jdosvib6jl4kg",
"__source": "service_list_1m",
"source": "service_list_1m",
"__namespace": "tracing",
"r_env": "demo",
"r_error_count": 0,
"r_max_duration": 2370648,
"r_psketch": "Av1KgVq/UvA/AAAAAAAAAAAJAboL",
"r_request_count": 1,
"r_resp_time": 2370648,
"r_service": "go-profiling-demo-1",
"r_service_sub": "go-profiling-demo-1:demo:v0.8.888",
"r_type": "custom",
"r_version": "v0.8.888",
"create_time": 1710835261477,
"date": 1710835201000,
"date_ns": 0
}
]
The main fields are described as follows:
| Field | Type | Description |
|---|---|---|
source |
string | Granularity of data aggregation: source="service_list_1m")source="service_list_1h")source="service_list_1d") |
r_env |
string | Environment where the service is deployed |
r_error_count |
int | Number of service errors |
r_max_duration |
int | Maximum response time within the time granularity, in microseconds |
r_request_count |
int | Number of requests |
r_resp_time |
int | Total response time aggregated within the time granularity |
r_service |
string | Service name |
r_service_sub |
string | |
r_type |
string | Service type, http/web/db/gateway... |
r_version |
string | Service version |
date |
int | Millisecond timestamp, corresponding to:hh:mm:00, when source="service_list_1m")hh:00, when source="service_list_1h")00:00:00, when source="service_list_1d") |
Similarly, to query two hours of data from 2024-03-19 15:00:00 to 2024-03-19 17:00:00, you can use DQL:
To query two days of data from 2024-03-19 00:00:00 to 2024-03-21 00:00:00, you can use DQL:
Different time granularities can be combined. For example, to query data for two and a half hours from 2024-03-19 15:00:00 to 2024-03-19 17:30:00, you can use DQL:
TM::`*`:(){ (source="service_list_1h" and date >= 1710831600000 and date < 1710838800000) or (source="service_list_1m" and date >= 1710838800000 and date <= 1710840600000) }
Abstract
Of course, you can also use only the minute-level granularity source="service_list_1m" to query data over an entire time range that spans hours or days, such as 2024-03-19 15:00:00 to 2024-03-19 17:30:00, but query efficiency will be greatly reduced and the returned data volume will multiply, so this query method is strongly discouraged.
By further processing the query results, you can calculate the corresponding service-level metrics. For example:
total_count = SUM(r_request_count)
error_count = SUM(r_error_count)
error_rate = SUM(r_error_count) / SUM(r_request_count)
max_duration = MAX(r_max_duration)
sum_resp_time = SUM(r_resp_time)
avg_per_second = SUM(r_request_count) / <query time range>
avg_resp_time = SUM(r_resp_time) / SUM(r_request_count)
p50: generate an array [(r_resp_time1/r_request_count1)...{repeat r_request_count1 times}, (r_resp_time2/r_request_count2)...{repeat r_request_count2 times}, (r_resp_time3/r_request_count3)...{repeat r_request_count3 times}, ...]; after sorting the array, take the index SUM(r_request_count)*0.5
For example, to query the QPS (requests per second) of each service over a time range, you can use:
TM::`*`:(r_service, sum(r_request_count) / (1737099000000 - 1737093600000) * 1000 as QPS){ (source="service_list_1h" and date >= 1737093600000 and date < 1737097200000) or (source="service_list_1m" and date >= 1737097200000 and date <= 1737099000000) } by r_service
Service Topology Data Source Definition¶
The TSM index space mainly stores call topology data between services. The data is pre-aggregated at minute-level granularity. For example, to query all service call relationships within a 15-minute range from 2024-03-19 15:00:00 to 2024-03-19 15:15:00, you can use DQL:
The query returns results similar to the following:
Sample query result
[
{
"time": 1710835700438,
"time_us": 1710835700438064,
"__docid": "6340252d-331c-6e1dd9338-0ed04e818c4d",
"__source": "relationship",
"source_service": "go-profiling-demo-1",
"source_wsuuid": "wksp_8d351d83bdf14b8b8270ab75fe29a990",
"source_env": "demo",
"source_project": "",
"source_version": "v0.8.888",
"source_type": "custom",
"source_organization": "",
"source_status": "ok",
"source_start": 1710835700059433,
"source_duration": 220210272,
"target_service": "go-profiling-demo-2",
"target_wsuuid": "wksp_8d351d83bdf14b8b8270ab75fe29a990",
"target_env": "demo",
"target_project": "",
"target_version": "v0.8.888",
"target_type": "custom",
"target_organization": "",
"target_status": "ok",
"target_start": 1710835700438064,
"target_duration": 886040,
"count": 96,
"unique_id": "XTzHH-jScNjXgBSXNIdFcSOVpHwWKyAZroh71ttyPnXK9nl3jW0re0hlKOeHj6PYgo-profiling-demo-1go-profiling-demo-2",
"unique_id_env_version": "Zp-9KKvEb4m9aU0OeUMon8MiH2isxqXU742YFlVtokgL2Vy73NwtykkG3vA3X0z1go-profiling-demo-1go-profiling-demo-2",
"error_count": 0
},
{
"time": 1710835243699,
"time_us": 1710835243699376,
"__docid": "f4c7de67-ca1b-6e17fd78e-e8bcb4d1d64d",
"__source": "relationship",
"source_service": "go-profiling-demo-1",
"source_wsuuid": "wksp_8d351d83bdf14b8b8270ab75fe29a990",
"source_env": "demo",
"source_project": "",
"source_version": "v0.8.888",
"source_type": "custom",
"source_organization": "",
"source_status": "ok",
"source_start": 1710835243379392,
"source_duration": 227582208,
"target_service": "go-profiling-demo-2",
"target_wsuuid": "wksp_8d351d83bdf14b8b8270ab75fe29a990",
"target_env": "demo",
"target_project": "",
"target_version": "v0.8.888",
"target_type": "custom",
"target_organization": "",
"target_status": "ok",
"target_start": 1710835243699376,
"target_duration": 856217,
"count": 96,
"unique_id": "XTzHH-jScNjXgBSXNIdFcSOVpHwWKyAZroh71ttyPnXK9nl3jW0re0hlKOeHj6PYgo-profiling-demo-1go-profiling-demo-2",
"unique_id_env_version": "Zp-9KKvEb4m9aU0OeUMon8MiH2isxqXU742YFlVtokgL2Vy73NwtykkG3vA3X0z1go-profiling-demo-1go-profiling-demo-2",
"error_count": 0
}
]
The main fields are described as follows:
| Field | Type | Description |
|---|---|---|
time |
int | Millisecond timestamp when the service call occurred |
time_us |
int | Time when the service call occurred, in microseconds |
source_service |
string | Calling service name |
source_wsuuid |
string | ID of the workspace to which the calling service reports data |
source_env |
string | Deployment environment of the calling service |
source_project |
string | Project name of the calling service |
source_version |
string | Version of the calling service |
source_type |
string | Type of the calling service |
source_organization |
string | Organization of the workspace to which the calling service reports data |
source_status |
string | Status of the calling service, ok/error |
source_start |
int | Start time of the calling service's Span, microsecond timestamp |
source_duration |
int | Aggregated duration of the calling service's Spans within each minute, in microseconds |
target_service |
string | Called service name |
target_wsuuid |
string | ID of the workspace to which the called service reports data |
target_env |
string | Deployment environment of the called service |
target_project |
string | Project name of the called service |
target_version |
string | Version of the called service |
target_type |
string | Type of the called service |
target_organization |
string | Organization of the workspace to which the called service reports data |
target_status |
string | Status of the called service, ok/error |
target_start |
int | Start time of the called service's Span (the time when the service call occurred), microsecond timestamp |
target_duration |
int | Aggregated duration of the called service's Spans within each minute, in microseconds |
count |
int | Number of calls within each minute |
unique_id |
string | Unique ID generated only from the calling service name and the called service name |
unique_id_env_version |
int | Unique ID generated to distinguish the calling service, environment, version and the called service, environment, version |
error_count |
int | Number of failed calls within each minute |
For service call relationships and service-call-level metrics, for example, you can use the following DQL to query call relationships between services: