Service Data Source Definitions and DQL Queries¶

Service Performance Data Source Definitions¶

The TM index space stores service list and performance metrics data. The data on APM > Performance Metrics is primarily queried from this index space. TM aggregates service metrics data for each service at three different granularities: minute, hour, and day, to improve query efficiency.

For example, to query all service metrics data for a 15-minute period from 2024-03-19 15:00:00 to 2024-03-19 15:15:00, you can use DQL:

TM::`*`:(){source="service_list_1m"} [1710831600000:1710832500000]

It will return results similar to the following:

Query Result Example

[
  {
    "time": 1710835681000,
    "time_us": 1710835681000000,
    "__docid": "T_cnskg71jdosvib6m44s0",
    "__source": "service_list_1m",
    "source": "service_list_1m",
    "__namespace": "tracing",
    "r_env": "demo",
    "r_error_count": 0,
    "r_max_duration": 2293857,
    "r_psketch": "Av1KgVq/UvA/AAAAAAAAAAAJAbgL",
    "r_request_count": 1,
    "r_resp_time": 2293857,
    "r_service": "go-profiling-demo-1",
    "r_service_sub": "go-profiling-demo-1:demo:v0.8.888",
    "r_type": "custom",
    "r_version": "v0.8.888",
    "create_time": 1710835740447,
    "date": 1710835681000,
    "date_ns": 0
  },
  {
    "time": 1710835201000,
    "time_us": 1710835201000000,
    "__docid": "T_cnskcf9jdosvib6jl4kg",
    "__source": "service_list_1m",
    "source": "service_list_1m",
    "__namespace": "tracing",
    "r_env": "demo",
    "r_error_count": 0,
    "r_max_duration": 2370648,
    "r_psketch": "Av1KgVq/UvA/AAAAAAAAAAAJAboL",
    "r_request_count": 1,
    "r_resp_time": 2370648,
    "r_service": "go-profiling-demo-1",
    "r_service_sub": "go-profiling-demo-1:demo:v0.8.888",
    "r_type": "custom",
    "r_version": "v0.8.888",
    "create_time": 1710835261477,
    "date": 1710835201000,
    "date_ns": 0
  }
]

The main fields are described as follows:

Field	Type	Description
`source`	string	Data aggregation granularity, divided into: Per minute (`source="service_list_1m"`) Per hour (`source="service_list_1h"`) Per day (`source="service_list_1d"`)
`r_env`	string	Service deployment environment
`r_error_count`	int	Number of service errors
`r_max_duration`	int	Maximum response time within the time granularity, unit: microseconds
`r_request_count`	int	Number of requests
`r_resp_time`	int	Sum of response times aggregated within the time granularity
`r_service`	string	Service name
`r_service_sub`	string	::
`r_type`	string	Service type, e.g., http/web/db/gateway...
`r_version`	string	Service version
`date`	int	Millisecond timestamp, corresponding to: The zero second of each minute (`hh:mm:00`, when `source="service_list_1m"`) The zero minute of each hour (`hh:00`, when `source="service_list_1h"`) The zero hour of each day (`00:00:00`, when `source="service_list_1d"`)

Similarly, to query data for two hours from 2024-03-19 15:00:00 to 2024-03-19 17:00:00, you can use DQL:

TM::`*`:(){source="service_list_1h"} [1710831600000:1710838800000]

To query data for two days from 2024-03-19 00:00:00 to 2024-03-21 00:00:00, you can use DQL:

TM::`*`:(){source="service_list_1d"} [1710777600000:1710950400000]

Different time granularities can be combined. For example, to query data for two and a half hours from 2024-03-19 15:00:00 to 2024-03-19 17:30:00, you can use DQL:

TM::`*`:(){ (source="service_list_1h" and date >= 1710831600000 and date < 1710838800000) or (source="service_list_1m" and date >= 1710838800000 and date <= 1710840600000) }

Abstract

Of course, you could also use only the minute granularity source="service_list_1m" to query data for a time range like 2024-03-19 15:00:00 to 2024-03-19 17:30:00 that spans hours or days. However, this would significantly reduce query efficiency and exponentially increase the amount of data returned, so it is highly discouraged.

Further processing of the query results can calculate relevant service-level metrics, for example:

total_count = SUM(r_request_count)
error_count = SUM(r_error_count)
error_rate = SUM(r_error_count) / SUM(r_request_count)
max_duration = MAX(r_max_duration)
sum_resp_time = SUM(r_resp_time)
avg_per_second = SUM(r_request_count) / <query time range>
avg_resp_time = SUM(r_resp_time) / SUM(r_request_count)
p50: Generate an array [(r_resp_time1/r_request_count1)...{repeat r_request_count1 times}, (r_resp_time2/r_request_count2)...{repeat r_request_count2 times}, (r_resp_time3/r_request_count3)...{repeat r_request_count3 times}, ...], sort the array, then take the element at index SUM(r_request_count)*0.5

For example, to query the QPS (Queries Per Second) for each service over a period of time, you can use:

TM::`*`:(r_service, sum(r_request_count) / (1737099000000 - 1737093600000) * 1000 as QPS){ (source="service_list_1h" and date >= 1737093600000 and date < 1737097200000) or (source="service_list_1m" and date >= 1737097200000 and date <= 1737099000000) } by r_service

Service Topology Data Source Definitions¶

The TSM index space primarily stores data on the calling topology relationships between services, pre-aggregated at a minute granularity. For example, to query all service call relationships for a 15-minute period from 2024-03-19 15:00:00 to 2024-03-19 15:15:00, you can use DQL:

TSM::`*`:(){} [1710831600000:1710832500000]

It returns query results similar to the following:

Query Result Example

[
    {
        "time": 1710835700438,
        "time_us": 1710835700438064,
        "__docid": "6340252d-331c-6e1dd9338-0ed04e818c4d",
        "__source": "relationship",
        "source_service": "go-profiling-demo-1",
        "source_wsuuid": "wksp_8d351d83bdf14b8b8270ab75fe29a990",
        "source_env": "demo",
        "source_project": "",
        "source_version": "v0.8.888",
        "source_type": "custom",
        "source_organization": "",
        "source_status": "ok",
        "source_start": 1710835700059433,
        "source_duration": 220210272,
        "target_service": "go-profiling-demo-2",
        "target_wsuuid": "wksp_8d351d83bdf14b8b8270ab75fe29a990",
        "target_env": "demo",
        "target_project": "",
        "target_version": "v0.8.888",
        "target_type": "custom",
        "target_organization": "",
        "target_status": "ok",
        "target_start": 1710835700438064,
        "target_duration": 886040,
        "count": 96,
        "unique_id": "XTzHH-jScNjXgBSXNIdFcSOVpHwWKyAZroh71ttyPnXK9nl3jW0re0hlKOeHj6PYgo-profiling-demo-1go-profiling-demo-2",
        "unique_id_env_version": "Zp-9KKvEb4m9aU0OeUMon8MiH2isxqXU742YFlVtokgL2Vy73NwtykkG3vA3X0z1go-profiling-demo-1go-profiling-demo-2",
        "error_count": 0
    },
    {
        "time": 1710835243699,
        "time_us": 1710835243699376,
        "__docid": "f4c7de67-ca1b-6e17fd78e-e8bcb4d1d64d",
        "__source": "relationship",
        "source_service": "go-profiling-demo-1",
        "source_wsuuid": "wksp_8d351d83bdf14b8b8270ab75fe29a990",
        "source_env": "demo",
        "source_project": "",
        "source_version": "v0.8.888",
        "source_type": "custom",
        "source_organization": "",
        "source_status": "ok",
        "source_start": 1710835243379392,
        "source_duration": 227582208,
        "target_service": "go-profiling-demo-2",
        "target_wsuuid": "wksp_8d351d83bdf14b8b8270ab75fe29a990",
        "target_env": "demo",
        "target_project": "",
        "target_version": "v0.8.888",
        "target_type": "custom",
        "target_organization": "",
        "target_status": "ok",
        "target_start": 1710835243699376,
        "target_duration": 856217,
        "count": 96,
        "unique_id": "XTzHH-jScNjXgBSXNIdFcSOVpHwWKyAZroh71ttyPnXK9nl3jW0re0hlKOeHj6PYgo-profiling-demo-1go-profiling-demo-2",
        "unique_id_env_version": "Zp-9KKvEb4m9aU0OeUMon8MiH2isxqXU742YFlVtokgL2Vy73NwtykkG3vA3X0z1go-profiling-demo-1go-profiling-demo-2",
        "error_count": 0
    }
]

The main fields are described as follows:

Field	Type	Description
`time`	int	Millisecond timestamp, the time when the service call occurred
`time_us`	int	The time when the service call occurred, microsecond level
`source_service`	string	Caller service name
`source_wsuuid`	string	Caller service reporting workspace ID
`source_env`	string	Caller service deployment environment
`source_project`	string	Caller service project name
`source_version`	string	Caller service version
`source_type`	string	Caller service type
`source_organization`	string	Organization to which the caller service's reporting workspace belongs
`source_status`	string	Caller service status, ok/error
`source_start`	int	Caller service Span start time, microsecond timestamp
`source_duration`	int	Sum of caller service Span durations aggregated per minute, unit: microseconds
`target_service`	string	Callee service name
`target_wsuuid`	string	Callee service reporting workspace ID
`target_env`	string	Callee service deployment environment
1. 专用名词翻译：
- 使用以下预置词典进行专用名词的翻译：
- 观测云: Guance
- 应用性能监测: APM
- 用户访问监测: RUM
- 体验版: Free Plan
- 商业版: Commercial Plan
- 部署版: Deployment Plan
- 专属版: Exclusive Plan
- 指标: Metrics
- 指标集: Measurement
- 资源目录: Resource Catalog
- 资源分类: Resource Class
- 时间线: Time Series
- 排行榜: Top List
- 查看器: Explorer
- 异常追踪: Incident
- 总览: Summary
- 静默: Mute
- 服务清单: Service List
- 服务拓扑: Service Map
- 顶层 Span: Top Span
- 服务顶层 Span: Service Entry Span
- 页面: View
- 操作: Action
- 用户洞察: User Analysis
- 可用性监测: Synthetic Tests
- 自建节点: Self-built Nodes
- 安全巡检: Security Check
- 可用性数据检测: Synthetic Testing Anomaly Detection
- 通知对象管理: Notification Targets
- DataFlux Func 托管版: DataFlux Func (Automata)
- 作战室: Warroom
- 个人设置: User Settings
- 属性声明: Attribute Claims
- 危险操作: Risky Operations
- 任务调用: Triggers
- API 拨测: API Tests
- 多步拨测: Multistep Tests
- 服务费: Service Charges
- 快捷入口: Shortcut
- 空间管理: Workspace Management
- 安全断言标记语言: SAML
- 存在: Exist
- 不存在: Not exist
- 智能巡检: Intelligent Inspection
- 概念先解: Concepts
- 开始新建: Create
- 新建规则: Create
- 新建通知策略: Create
- 新建日程: Create
- 新建频道: Create
- 新建 Issue: Create
- 新建索引: Create
- 新建标签: Create
- 新建查看器: Create
- 新增字段: Create
- 新建追踪: Create
- 新建节点: Create
- 管理策略: Manage
- 管理 Issue: Manage
- 管理索引: Manage
- 管理标签: Manage
- 管理查看器: Manage
- 节点管理: Manage
- 管理节点: Manage
- 管理规则: Manage Rules
- 规则管理: Manage Rules
- 管理策略列表: Manage Rules
- 功能介绍: Features
- 功能模块: Features
- 聚类分析: Pattern
- 开始配置: Configure
- 概览: Overview
- 版本说明: Plans
- 付费计划与账单 Billing
- 费用中心账号: Billing Center account
- 观测云费用中心 Guance Billing Center
- 用户访问 PV RUM PV
- 相关配置/配置步骤: Configuration
- 列表操作/相关操作: Options
- 时间控件: Time Widget
- 使用场景: Use Cases
- 适用场景: Use Cases
- 使用范围: Use Cases
- 飞书: Lark
- 告警策略管理: Alert Strategies
- 通知对象管理: Notification Targets
- 企业微信: WeCom
- 批量操作: Batch operations
- 阿里云: Alibaba Cloud
- 腾讯云: Tencent Cloud
- 华为云: Huawei Cloud
- 火山引擎: Volcengine
- 谷歌云: GCP
- 中间件: MIDDLEWARE
- 主机: HOST
- 容器: CONTAINERS
- 网络: NETWORK
- 缓存: CACHING
- 消息队列: MESSAGE QUEUES
- 数据库: DATABASE
- 语言: LANGUAGE
- 链路追踪: APM
- 日志: LOG
- 拨测: TESTING
- 移动端: MOBILE
- 会话重放: SESSION REPLAY
- 确保这些专用名词在翻译时严格按照词典处理。

Service Data Source Definitions and DQL Queries¶

Service Performance Data Source Definitions¶

Service Topology Data Source Definitions¶

Is this page helpful? ×