跳转至

Prometheus Remote Write

监听 Prometheus Remote Write 数据，上报到观测云。

配置

前置条件

注意，对于 vmalert 的一些早期版本，需要在采集器的配置文件中打开设置 default_content_encoding = "snappy"。

开启 Prometheus Remote Write 功能，在 prometheus.yml 添加如下配置：

remote_write:
 - url: "http://<datakit-ip>:9529/prom_remote_write"

# If want add some tag, ( __source will not in tag, only show in Datakit expose metrics)
# remote_write:
# - url: "http://<datakit-ip>:9529/prom_remote_write?host=1.2.3.4&foo=bar&__source=<your_source>"

采集器配置

进入 DataKit 安装目录下的 conf.d/prom 目录，复制 prom_remote_write.conf.sample 并命名为 prom_remote_write.conf。示例如下：

[[inputs.prom_remote_write]]
  ## Path to listen to.
  path = "/prom_remote_write"

  ## accepted methods
  methods = ["PUT", "POST"]

  ## If the data is decoded incorrectly, you need to set the default HTTP body encoding;
  ## this usually occurs when the sender does not correctly pass the encoding in the HTTP header.
  #
  # default_content_encoding = "snappy"

  ## Part of the request to consume.  Available options are "body" and "query".
  # data_source = "body"

  ## output source
  # specify this to output collected metrics to local file
  # if not specified, metrics is sent to datakit io
  # if specified, you can use 'datakit --prom-conf /path/to/this/conf' to debug collected data
  # output = "/abs/path/file"

  ## If job_as_measurement is true, use the job field for the measurement name.
  ## The measurement_name configuration takes precedence.
  job_as_measurement = false

  ## Metric name filter
  # Regex is supported.
  # Only metric matches one of the regex can pass through. No filter if left empty.
  # metric_name_filter = ["gc", "go"]

  ## Measurement name filter
  # Regex is supported.
  # Only measurement matches one of the regex can pass through. No filter if left empty.
  # This filtering is done before any prefixing rule or renaming rule is applied.
  # measurement_name_filter = ["kubernetes", "container"]

  ## metric name prefix
  ## prefix will be added to metric name
  # measurement_prefix = "prefix_"

  ## metric name
  ## metric name will be divided by "_" by default.
  ## metric is named by the first divided field, the remaining field is used as the current metric name
  ## metric name will not be divided if measurement_name is configured
  ## measurement_prefix will be added to the start of measurement_name
  # measurement_name = "prom_remote_write"

  ## max body size in bytes, default set to 500MB
  # max_body_size = 0

  ## Optional username and password to accept for HTTP basic authentication.
  ## You probably want to make sure you have TLS configured above for this.
  # basic_username = ""
  # basic_password = ""

  ## If both blacklist and whitelist, all list will cancel.
  ## tags to ignore (blacklist)
  # tags_ignore = ["xxxx"]

  ## tags to ignore with regex (blacklist)
  # tags_ignore_regex = ["xxxx"]

  ## tags whitelist
  # tags_only = ["xxxx"]

  ## tags whitelist with regex
  # tags_only_regex = ["xxxx"]

  ## Indicate whether tags_rename overwrites existing key if tag with the new key name already exists.
  overwrite = false

  ## tags to rename
  [inputs.prom_remote_write.tags_rename]
  # old_tag_name = "new_tag_name"
  # more_old_tag_name = "other_new_tag_name"

  ## Optional setting to map http headers into tags
  ## If the http header is not present on the request, no corresponding tag will be added
  ## If multiple instances of the http header are present, only the first value will be used
  [inputs.prom_remote_write.http_header_tags]
  # HTTP_HEADER = "TAG_NAME"

  ## Customize measurement set name.
  ## Treat those metrics with prefix as one set.
  ## Prioritier over 'measurement_name' configuration.
  ## Must measurement_name = ""
  [[inputs.prom_remote_write.measurements]]
    prefix = "etcd_network_"
    name = "etcd_network"

  [[inputs.prom_remote_write.measurements]]
    prefix = "etcd_server_"
    name = "etcd_server"

  ## custom tags
  [inputs.prom_remote_write.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"

配置好后，重启 DataKit 即可。

可通过 ConfigMap 方式注入采集器配置 或 配置 ENV_DATAKIT_INPUTS 开启采集器。

tags 的处理

可以通过配置 tags 为采集到的指标加上标签，如下：

  ## custom tags
  [inputs.prom_remote_write.tags]
  some_tag = "some_value"
  more_tag = "some_other_value"

注意：黑名单和白名单同时配置，黑白名单会全部失效。

可以通过配置 tags_ignore 忽略指标上的某些标签（黑名单），如下：

  ## tags to ignore
  tags_ignore = ["xxxx"]

可以通过配置 tags_ignore_regex 正则匹配并忽略指标上的标签（黑名单），如下：

  ## tags to ignore with regex
  tags_ignore_regex = ["xxxx"]

可以通过配置 tags_only 配置指标上的标签白名单，如下：

  ## tags white list
  # tags_only = ["xxxx"]

可以通过配置 tags_only_regex 正则匹配指标上的标签白名单，如下：

  ## tags white list with regex
  # tags_only_regex = ["xxxx"]

可以通过配置 tags_rename 重命名指标已有的某些标签名，如下：

  ## tags to rename
  [inputs.prom_remote_write.tags_rename]
  old_tag_name = "new_tag_name"
  more_old_tag_name = "other_new_tag_name"

另外，当重命名后的 tag key 与已有 tag key 相同时，可以通过 overwrite 配置是否覆盖掉已有的 tag key。

注意：对于 DataKit 全局 tag key，此处不支持将它们重命名。

指标

指标集以 Prometheus 发送过来的指标集为准。

配置 Prometheus Remote Write 指标过滤

当使用 Prometheus 以 remote write 方式往 Datakit 推送指标时，如果指标太多，可能导致 存储中的数据暴增。此时我们可以通过 Prometheus 自身的 relabel 功能来选取特定的指标。

在 Prometheus 中，要配置 remote_write 到另一个服务，并且只发送指定的指标列表，我们需要在 Prometheus 的配置文件（通常是 prometheus.yml）中设置 remote_write 部分，并指定 match[] 参数来定义要发送的指标。

以下是一个配置示例，它展示了如何将特定的指标列表发送到远程写入端点：

remote_write:
  - url: "http://remote-write-service:9090/api/v1/write"
    write_relabel_configs:
      - source_labels: ["__name__"]
        regex: "my_metric|another_metric|yet_another_metric"
        action: keep

在这个配置中：

  • url: 远程写入服务的 URL
  • write_relabel_configs: 一个列表，用于重新标记和过滤要发送的指标
    • source_labels: 指定用于匹配和重新标记的源标签
    • regex: 一个正则表达式，用于匹配要保留的指标名称
    • action: 指定匹配正则表达式的指标是被保留（keep）还是被丢弃（drop）

在上面的示例中，只有名称匹配 my_metric、another_metric 或 yet_another_metric 的指标 会被发送到远程写入端点。其他所有指标都会被忽略。

最后，重新加载或重启 Prometheus 服务以应用更改。

命令行调试指标集

DataKit 提供一个简单的调试 prom.conf 的工具，如果不断调整 prom.conf 的配置，可以实现只采集符合一定名称规则的 Prometheus 指标的目的。

Datakit 支持命令行直接调试本采集器的配置文件。在配置 conf.d/prom 下 prom_remote_write.conf 的 output 项，将其配置为一个本地文件路径，之后 prom_remote_write.conf 会将采集到的数据写到文件中，数据就不会上传到中心。

重启 Datakit，让配置文件生效：

datakit service -R

这时 prom_remote_write 采集器将把采集的数据写到 output 指明的本地文件中。

这时执行如下命令，即可调试 prom_remote_write.conf

datakit debug --prom-conf prom_remote_write.conf

参数说明：

  • prom-conf: 指定配置文件，默认在当前目录下寻找 prom_remote_write.conf 文件，如果未找到，会去 <datakit-install-dir>/conf.d/prom 目录下查找相应文件。

输出示例：

================= Line Protocol Points ==================

 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor target_scrapes_sample_out_of_order_total=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=node target_sync_failed_total=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=prometheus target_sync_failed_total=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=node target_sync_length_seconds_sum=0.000070352 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=node target_sync_length_seconds_count=1 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=prometheus target_sync_length_seconds_sum=0.000089457 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor,scrape_job=prometheus target_sync_length_seconds_count=1 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor template_text_expansion_failures_total=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor template_text_expansions_total=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor treecache_watcher_goroutines=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor treecache_zookeeper_failures_total=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_blocks_loaded=1 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_checkpoint_creations_failed_total=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_checkpoint_creations_total=1 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_checkpoint_deletions_failed_total=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_checkpoint_deletions_total=1 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_clean_start=1 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=100,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=400,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=1600,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=6400,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=25600,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=102400,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=409600,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=1.6384e+06,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=6.5536e+06,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=2.62144e+07,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=+Inf,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_sum=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,monitor=codelab-monitor tsdb_compaction_chunk_range_seconds_count=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=4,monitor=codelab-monitor tsdb_compaction_chunk_samples_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=6,monitor=codelab-monitor tsdb_compaction_chunk_samples_bucket=0 1634548272855000000
 prometheus,instance=localhost:9090,job=prometheus,le=9,monitor=codelab-monitor tsdb_compaction_chunk_samples_bucket=0 1634548272855000000
...
================= Summary ==================

Total time series: 155
Total line protocol points: 487
Total measurements: 6 (prometheus, promhttp, up, scrape, go, node)

输出说明：

  • Line Protocol Points： 产生的行协议点
  • Summary： 汇总结果
    • Total time series: 时间线数量
    • Total line protocol points: 行协议点数
    • Total measurements: 指标集个数及其名称。

文档评价

文档内容是否对您有帮助？ ×