Consul

·

Consul collector is used to collect metric data related to Consul, and currently it only supports data in Prometheus format.

Configuration¶

Preconditions¶

Installing consul-exporter

Download consul_exporter package

sudo wget https://github.com/prometheus/consul_exporter/releases/download/v0.7.1/consul_exporter-0.7.1.linux-amd64.tar.gz

Unzip consul_exporter package

sudo tar -zxvf consul_exporter-0.7.1.linux-amd64.tar.gz

Go to the consul_exporter-0.7.1.linux-amd64 directory and run the consul_exporter script

./consul_exporter

Collector Configuration¶

host installationKubernetes

Go to the conf.d/samples directory under the DataKit installation directory, copy consul.conf.sample and name it consul.conf. Examples are as follows:

[[inputs.prom]]
  url = "http://127.0.0.1:9107/metrics"
  source = "consul"
  metric_types = ["counter", "gauge"]
  metric_name_filter = ["consul_raft_leader", "consul_raft_peers", "consul_serf_lan_members", "consul_catalog_service", "consul_catalog_service_node_healthy", "consul_health_node_status", "consul_serf_lan_member_status"]
  measurement_prefix = ""
  tags_ignore = ["check"]
  interval = "10s"

[[inputs.prom.measurements]]
  prefix = "consul_"
  name = "consul"

After configuration, restart DataKit.

The collector can now be turned on by ConfigMap Injection Collector Configuration.

Metric¶

`consul`¶

Tags & Fields	Description
check (`tag`)	Consul health check label emitted by consul_exporter.
check_id (`tag`)	Consul health check ID.
check_name (`tag`)	Consul health check name.
host (`tag`)	Host name associated with the scraped Consul exporter target.
instance (`tag`)	Scraped exporter instance endpoint.
key (`tag`)	Consul KV key.
member (`tag`)	Consul Serf member name.
node (`tag`)	Consul node name.
service_id (`tag`)	Consul service ID.
service_name (`tag`)	Consul service name.
status (`tag`)	Consul health status label, such as critical, maintenance, passing, or warning.
tag (`tag`)	Consul service tag value.
catalog_kv	Numeric value stored at the tagged Consul key in the key/value catalog. Non-numeric keys are omitted. Type: float \| (gauge) Unit: N/A
catalog_service_node_healthy	Whether the tagged Consul service is healthy on the tagged node. Type: float \| (gauge) Unit: bool
catalog_services	Number of services registered in the Consul catalog. Type: float \| (gauge) Unit: count
health_node_status	Whether the tagged Consul node health check is currently in the tagged status. Type: float \| (gauge) Unit: bool
health_service_status	Whether the tagged Consul service health check is currently in the tagged status. Type: float \| (gauge) Unit: bool
raft_leader	Whether this Consul node reports that the Raft cluster currently has a leader. Type: float \| (gauge) Unit: bool
raft_peers	Number of Consul servers participating as peers in the Raft cluster. Type: float \| (gauge) Unit: count
serf_lan_member_status	Numeric Serf LAN member status code for the tagged member. 1=alive, 2=leaving, 3=left, 4=failed. Type: float \| (gauge) Unit: N/A
serf_lan_members	Number of members in the Consul Serf LAN pool. Type: float \| (gauge) Unit: count
serf_wan_member_status	Numeric Serf WAN member status code for the tagged member. 1=alive, 2=leaving, 3=left, 4=failed. Type: float \| (gauge) Unit: N/A
service_checks	Whether the tagged Consul service is linked with the tagged check_id and check_name. Type: float \| (gauge) Unit: bool
service_tag	Whether the tagged Consul service tag exists for the service_id, node, and tag label set. Type: float \| (gauge) Unit: bool
up	Whether the consul_exporter successfully queried Consul during the last scrape. Type: float \| (gauge) Unit: bool

Logs¶

If you need to collect the log of Consul, you need to use the-syslog parameter when opening Consul, for example:

consul agent -dev -syslog

To use the logging collector to collect logs, you need to configure the logging collector. Go to the conf.d/log directory under the DataKit installation directory, copy logging.conf.sample and name it logging.conf. The configuration is as follows:

[[inputs.logging]]
  ## required
  logfiles = [
    "/var/log/syslog",
  ]

  ## glob filteer
  ignore = [""]

  ## your logging source, if it's empty, use 'default'
  source = "consul"

  ## add service tag, if it's empty, use $source.
  service = ""

  ## grok pipeline script path
  pipeline = "consul.p"

  ## optional status:
  ##   "emerg","alert","critical","error","warning","info","debug","OK"
  ignore_status = []

  ## optional encodings:
  ##    "utf-8", "utf-16le", "utf-16le", "gbk", "gb18030" or ""
  character_encoding = ""

  ## The pattern should be a regexp. Note the use of '''this regexp'''
  ## regexp link: https://golang.org/pkg/regexp/syntax/#hdr-Syntax
  multiline_match = '''^\S'''

  [inputs.logging.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"

Original log:

Sep 18 19:30:23 derrick-ThinkPad-X230 consul[11803]: 2021-09-18T19:30:23.522+0800 [INFO]  agent.server.connect: initialized primary datacenter CA with provider: provider=consul

The list of cut fields is as follows:

Field name	Field value	Description
`date`	`2021-09-18T19:30:23.522+0800`	log date
`level`	`INFO`	log level
`character`	`agent.server.connect`	role
`msg`	`initialized primary datacenter CA with provider: provider=consul`	log content