Prometheus Remote Write¶
监听 Prometheus Remote Write 数据，上报到观测云。
配置¶
前置条件¶
注意，对于 vmalert 的一些早期版本，需要在采集器的配置文件中打开设置 default_content_encoding = "snappy"。
开启 Prometheus Remote Write 功能，在 prometheus.yml 添加如下配置：
remote_write:
- url: "http://<datakit-ip>:9529/prom_remote_write"
# If want add some tag, ( __source will not in tag, only show in Datakit expose metrics)
# remote_write:
# - url: "http://<datakit-ip>:9529/prom_remote_write?host=1.2.3.4&foo=bar&__source=<your_source>"
采集器配置¶
进入 DataKit 安装目录下的 conf.d/prom 目录，复制 prom_remote_write.conf.sample 并命名为 prom_remote_write.conf。示例如下：
[[inputs.prom_remote_write]]
## Path to listen to.
path = "/prom_remote_write"
## accepted methods
methods = ["PUT", "POST"]
## If the data is decoded incorrectly, you need to set the default HTTP body encoding;
## this usually occurs when the sender does not correctly pass the encoding in the HTTP header.
#
# default_content_encoding = "snappy"
## Part of the request to consume. Available options are "body" and "query".
# data_source = "body"
## output source
# specify this to output collected metrics to local file
# if not specified, metrics is sent to datakit io
# if specified, you can use 'datakit --prom-conf /path/to/this/conf' to debug collected data
# output = "/abs/path/file"
## Metric name filter
# Regex is supported.
# Only metric matches one of the regex can pass through. No filter if left empty.
# metric_name_filter = ["gc", "go"]
## Measurement name filter
# Regex is supported.
# Only measurement matches one of the regex can pass through. No filter if left empty.
# This filtering is done before any prefixing rule or renaming rule is applied.
# measurement_name_filter = ["kubernetes", "container"]
## metric name prefix
## prefix will be added to metric name
# measurement_prefix = "prefix_"
## metric name
## metric name will be divided by "_" by default.
## metric is named by the first divided field, the remaining field is used as the current metric name
## metric name will not be divided if measurement_name is configured
## measurement_prefix will be added to the start of measurement_name
# measurement_name = "prom_remote_write"
## max body size in bytes, default set to 500MB
# max_body_size = 0
## Optional username and password to accept for HTTP basic authentication.
## You probably want to make sure you have TLS configured above for this.
# basic_username = ""
# basic_password = ""
## If both blacklist and whitelist, all list will cancel.
## tags to ignore (blacklist)
# tags_ignore = ["xxxx"]
## tags to ignore with regex (blacklist)
# tags_ignore_regex = ["xxxx"]
## tags whitelist
# tags_only = ["xxxx"]
## tags whitelist with regex
# tags_only_regex = ["xxxx"]
## Indicate whether tags_rename overwrites existing key if tag with the new key name already exists.
overwrite = false
## tags to rename
[inputs.prom_remote_write.tags_rename]
# old_tag_name = "new_tag_name"
# more_old_tag_name = "other_new_tag_name"
## Optional setting to map http headers into tags
## If the http header is not present on the request, no corresponding tag will be added
## If multiple instances of the http header are present, only the first value will be used
[inputs.prom_remote_write.http_header_tags]
# HTTP_HEADER = "TAG_NAME"
## Customize measurement set name.
## Treat those metrics with prefix as one set.
## Prioritier over 'measurement_name' configuration.
## Must measurement_name = ""
[[inputs.prom_remote_write.measurements]]
prefix = "etcd_network_"
name = "etcd_network"
[[inputs.prom_remote_write.measurements]]
prefix = "etcd_server_"
name = "etcd_server"
## custom tags
[inputs.prom_remote_write.tags]
# some_tag = "some_value"
# more_tag = "some_other_value"
配置好后，重启 DataKit 即可。
可通过 ConfigMap 方式注入采集器配置 或 配置 ENV_DATAKIT_INPUTS 开启采集器。
tags 的处理¶
可以通过配置 tags 为采集到的指标加上标签，如下：
## custom tags
[inputs.prom_remote_write.tags]
some_tag = "some_value"
more_tag = "some_other_value"
注意：黑名单和白名单同时配置，黑白名单会全部失效。
可以通过配置 tags_ignore 忽略指标上的某些标签（黑名单），如下：
可以通过配置 tags_ignore_regex 正则匹配并忽略指标上的标签（黑名单），如下：
可以通过配置 tags_only 配置指标上的标签白名单，如下：
可以通过配置 tags_only_regex 正则匹配指标上的标签白名单，如下：
可以通过配置 tags_rename 重命名指标已有的某些标签名，如下：
## tags to rename
[inputs.prom_remote_write.tags_rename]
old_tag_name = "new_tag_name"
more_old_tag_name = "other_new_tag_name"
另外，当重命名后的 tag key 与已有 tag key 相同时，可以通过 overwrite 配置是否覆盖掉已有的 tag key。
注意：对于 DataKit 全局 tag key，此处不支持将它们重命名。
指标¶
指标集以 Prometheus 发送过来的指标集为准。
命令行调试指标集¶
DataKit 提供一个简单的调试 prom.conf 的工具，如果不断调整 prom.conf 的配置，可以实现只采集符合一定名称规则的 Prometheus 指标的目的。
Datakit 支持命令行直接调试本采集器的配置文件。在配置 conf.d/prom 下 prom_remote_write.conf 的 output 项，将其配置为一个本地文件路径，之后 prom_remote_write.conf 会将采集到的数据写到文件中，数据就不会上传到中心。
重启 Datakit，让配置文件生效：
这时 prom_remote_write 采集器将把采集的数据写到 output 指明的本地文件中。
这时执行如下命令，即可调试 prom_remote_write.conf
参数说明：
prom-conf: 指定配置文件，默认在当前目录下寻找prom_remote_write.conf文件，如果未找到，会去 <datakit-install-dir>/conf.d/prom 目录下查找相应文件。
输出示例：
================= Line Protocol Points ==================
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor target_scrapes_sample_out_of_order_total=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=node target_sync_failed_total=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=prometheus target_sync_failed_total=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=node target_sync_length_seconds_sum=0.000070352 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=node target_sync_length_seconds_count=1 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=prometheus target_sync_length_seconds_sum=0.000089457 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=prometheus target_sync_length_seconds_count=1 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor template_text_expansion_failures_total=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor template_text_expansions_total=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor treecache_watcher_goroutines=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor treecache_zookeeper_failures_total=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_blocks_loaded=1 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_checkpoint_creations_failed_total=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_checkpoint_creations_total=1 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_checkpoint_deletions_failed_total=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_checkpoint_deletions_total=1 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_clean_start=1 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=100,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=400,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=1600,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=6400,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=25600,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=102400,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=409600,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=1.6384e+06,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=6.5536e+06,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=2.62144e+07,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=+Inf,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_sum=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_count=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=4,monitor=codelab-monitor tsdb_compaction_chunk_samples_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=6,monitor=codelab-monitor tsdb_compaction_chunk_samples_bucket=0 1634548272855000000
prometheus,instance=localhost:9090,job=prometheus,le=9,monitor=codelab-monitor tsdb_compaction_chunk_samples_bucket=0 1634548272855000000
...
================= Summary ==================
Total time series: 155
Total line protocol points: 487
Total measurements: 6 (prometheus, promhttp, up, scrape, go, node)
输出说明：
- Line Protocol Points： 产生的行协议点
- Summary： 汇总结果
- Total time series: 时间线数量
- Total line protocol points: 行协议点数
- Total measurements: 指标集个数及其名称。