Skip to content

TiDB

The TiDB view displays the relevant Metrics information, including the Summary (such as: start time, storage information, node information, etc.), cluster (related cluster information), TiDB, Etcd, Region, etc.

Installation and Configuration

Version Support

Version support depends on the support of the TiDB system itself.

Note: The example TiDB version is 6.3+

(Linux / Windows environments are the same)

If you need to install TiDB, refer to the document Deploy a Local Test Cluster for TiDB

The following collection process is based on the deployment method described in Deploy a Local Test Cluster for TiDB for collection.

Metrics Collection Configuration

Each component of TiDB (a total of 4 components) has exposed metrics, with the protocol being http

Component metrics port
TiDB 10080
pd 2379
TiKV 20180
TiFlash 8234
20292

The above are the metrics ports for single-node cluster component deployments; if it's a multi-node cluster, the port configuration is similar.

Enable DataKit Collector

  1. Copy the sample file 

      cd /usr/local/datakit/conf.d/prom/
  cp prom.conf.sample tidb_prom.conf

  2. Adjust tidb_prom.conf

tidb_prom.conf 
[[inputs.prom]]
  # Exporter URLs.
  # urls = ["http://127.0.0.1:9100/metrics", "http://127.0.0.1:9200/metrics"]
  urls = ["http://0.0.0.0:10080/metrics"]
  # Unix Domain Socket URL. Using socket to request data when not empty.
  uds_path = ""

  # Ignore URL request errors.
  ignore_req_err = false

  ## Collector alias.
  source = "tidb_prom"

  ## Metrics type whitelist. Optional: counter, gauge, histogram, summary
  # Default only collect 'counter' and 'gauge'.
  # Collect all if empty.
  metric_types = ["counter", "gauge","histogram"]

  ## Measurement prefix.
  # Add prefix to measurement set name.
  measurement_prefix = "tidb_"

  ## TLS configuration.
  tls_open = false
  # tls_ca = "/tmp/ca.crt"
  # tls_cert = "/tmp/peer.crt"
  # tls_key = "/tmp/peer.key"

  ## Set to 'true' to enable election.
  election = true

  # Ignore tags. Multi supported.
  # The matched tags would be dropped, but the item would still be sent.
  # tags_ignore = ["xxxx"]

  ## Customize authentication. For now supports Bearer Token only.
  # Filling in 'token' or 'token_file' is acceptable.
  # [inputs.prom.auth]
  # type = "bearer_token"
  # token = "xxxxxxxx"
  # token_file = "/tmp/token"

  ## Customize measurement set name.
  # Treat those metrics with prefix as one set.
  # Prioritier over 'measurement_name' configuration.
  #[[inputs.prom.measurements]]
  #  prefix = "cpu_"
  #  name = "cpu"

  # [[inputs.prom.measurements]]
  # prefix = "mem_"
  # name = "mem"

  # Not collecting those data when tag matched.
  [inputs.prom.ignore_tag_kv_match]
  # key1 = [ "val1.*", "val2.*"]
  # key2 = [ "val1.*", "val2.*"]

  # Add HTTP headers to data pulling.
  [inputs.prom.http_headers]
  # Root = "passwd"
  # Michael = "1234"

  # Rename tag key in prom data.
  [inputs.prom.tags_rename]
    overwrite_exist_tags = false
    [inputs.prom.tags_rename.mapping]
    # tag1 = "new-name-1"
    # tag2 = "new-name-2"
    # tag3 = "new-name-3"

  # Send collected metrics to center as log.
  # When 'service' field is empty, using 'service tag' as measurement set name.
  [inputs.prom.as_logging]
    enable = false
    service = "service_name"

  ## Customize tags.
  [inputs.prom.tags]
    tidb_cluster="test"

  1. Copy the sample file 

    cd /usr/local/datakit/conf.d/prom/
cp prom.conf.sample tidb_pd_prom.conf

  2. Adjust tidb_pd_prom.conf

tidb_pd_prom.conf 
[[inputs.prom]]
  # Exporter URLs.
  # urls = ["http://127.0.0.1:9100/metrics", "http://127.0.0.1:9200/metrics"]
  urls = ["http://0.0.0.0:2379/metrics"]
  # Unix Domain Socket URL. Using socket to request data when not empty.
  uds_path = ""

  # Ignore URL request errors.
  ignore_req_err = false

  ## Collector alias.
  source = "tidb_pd_prom"

  ## Collect data output.
  # Fill this when want to collect the data to local file nor center.
  # After filling, could use 'datakit --prom-conf /path/to/this/conf' to debug local storage measurement set.
  # Using '--prom-conf' when priority debugging data in 'output' path.
  # output = "/abs/path/to/file"

  ## Collect data upper limit as bytes.
  # Only available when set output to local file.
  # If collect data exceeded the limit, the data would be dropped.
  # Default is 32MB.
  # max_file_size = 0

  ## Metrics type whitelist. Optional: counter, gauge, histogram, summary
  # Default only collect 'counter' and 'gauge'.
  # Collect all if empty.
  metric_types = ["counter", "gauge"]

  ## Metrics name whitelist.
  # Regex supported. Multi supported, conditions met when one matched.
  # Collect all if empty.
  # metric_name_filter = ["cpu"]

  ## Measurement prefix.
  # Add prefix to measurement set name.
  measurement_prefix = "tidb_pd_"

  ## Measurement name.
  # If measurement_name is empty, split metric name by '_', the first field after split as measurement set name, the rest as current metric name.
  # If measurement_name is not empty, using this as measurement set name.
  # Always add 'measurement_prefix' prefix at last.
  # measurement_name = "prom"

  ## TLS configuration.
  tls_open = false
  # tls_ca = "/tmp/ca.crt"
  # tls_cert = "/tmp/peer.crt"
  # tls_key = "/tmp/peer.key"

  ## Set to 'true' to enable election.
  election = true

  # Ignore tags. Multi supported.
  # The matched tags would be dropped, but the item would still be sent.
  # tags_ignore = ["xxxx"]

  ## Customize authentication. For now supports Bearer Token only.
  # Filling in 'token' or 'token_file' is acceptable.
  # [inputs.prom.auth]
  # type = "bearer_token"
  # token = "xxxxxxxx"
  # token_file = "/tmp/token"

  ## Customize measurement set name.
  # Treat those metrics with prefix as one set.
  # Prioritier over 'measurement_name' configuration.
  #[[inputs.prom.measurements]]
  #  prefix = "cpu_"
  #  name = "cpu"

  # [[inputs.prom.measurements]]
  # prefix = "mem_"
  # name = "mem"

  # Not collecting those data when tag matched.
  [inputs.prom.ignore_tag_kv_match]
  # key1 = [ "val1.*", "val2.*"]
  # key2 = [ "val1.*", "val2.*"]

  # Add HTTP headers to data pulling.
  [inputs.prom.http_headers]
  # Root = "passwd"
  # Michael = "1234"

  # Rename tag key in prom data.
  [inputs.prom.tags_rename]
    #overwrite_exist_tags = false
    overwrite_exist_tags = false
    [inputs.prom.tags_rename.mapping]
      cluster_version = "version"
      server_id = "s_id"
      server_version = "version"
      server_go_version = "version"
    # tag1 = "new-name-1"
    # tag2 = "new-name-2"
    # tag3 = "new-name-3"

  # Send collected metrics to center as log.
  # When 'service' field is empty, using 'service tag' as measurement set name.
  [inputs.prom.as_logging]
    enable = false
    service = "service_name"

  ## Customize tags.
  [inputs.prom.tags]
    tidb_cluster="test"

  1. Copy the sample file 

    cd /usr/local/datakit/conf.d/prom/
cp prom.conf.sample tidb_tikv_prom.conf

  2. Adjust tidb_tikv_prom.conf

tidb_tikv_prom.conf 
[[inputs.prom]]
  # Exporter URLs.
  # urls = ["http://127.0.0.1:9100/metrics", "http://127.0.0.1:9200/metrics"]
  urls = ["http://0.0.0.0:20180/metrics"]
  # Unix Domain Socket URL. Using socket to request data when not empty.
  uds_path = ""

  # Ignore URL request errors.
  ignore_req_err = false

  ## Collector alias.
  source = "tidb_tikv_prom"

  ## Collect data output.
  # Fill this when want to collect the data to local file nor center.
  # After filling, could use 'datakit --prom-conf /path/to/this/conf' to debug local storage measurement set.
  # Using '--prom-conf' when priority debugging data in 'output' path.
  # output = "/abs/path/to/file"

  ## Collect data upper limit as bytes.
  # Only available when set output to local file.
  # If collect data exceeded the limit, the data would be dropped.
  # Default is 32MB.
  # max_file_size = 0

  ## Metrics type whitelist. Optional: counter, gauge, histogram, summary
  # Default only collect 'counter' and 'gauge'.
  # Collect all if empty.
  metric_types = ["counter", "gauge"]

  ## Metrics name whitelist.
  # Regex supported. Multi supported, conditions met when one matched.
  # Collect all if empty.
  # metric_name_filter = ["cpu"]

  ## Measurement prefix.
  # Add prefix to measurement set name.
  measurement_prefix = "tidb_tikv_"

  ## Measurement name.
  # If measurement_name is empty, split metric name by '_', the first field after split as measurement set name, the rest as current metric name.
  # If measurement_name is not empty, using this as measurement set name.
  # Always add 'measurement_prefix' prefix at last.
  # measurement_name = "prom"

  ## TLS configuration.
  tls_open = false
  # tls_ca = "/tmp/ca.crt"
  # tls_cert = "/tmp/peer.crt"
  # tls_key = "/tmp/peer.key"

  ## Set to 'true' to enable election.
  election = true

  # Ignore tags. Multi supported.
  # The matched tags would be dropped, but the item would still be sent.
  # tags_ignore = ["xxxx"]

  ## Customize authentication. For now supports Bearer Token only.
  # Filling in 'token' or 'token_file' is acceptable.
  # [inputs.prom.auth]
  # type = "bearer_token"
  # token = "xxxxxxxx"
  # token_file = "/tmp/token"

  ## Customize measurement set name.
  # Treat those metrics with prefix as one set.
  # Prioritier over 'measurement_name' configuration.
  #[[inputs.prom.measurements]]
  #  prefix = "cpu_"
  #  name = "cpu"

  # [[inputs.prom.measurements]]
  # prefix = "mem_"
  # name = "mem"

  # Not collecting those data when tag matched.
  [inputs.prom.ignore_tag_kv_match]
  # key1 = [ "val1.*", "val2.*"]
  # key2 = [ "val1.*", "val2.*"]

  # Add HTTP headers to data pulling.
  [inputs.prom.http_headers]
  # Root = "passwd"
  # Michael = "1234"

  # Rename tag key in prom data.
  [inputs.prom.tags_rename]
    overwrite_exist_tags = false
    [inputs.prom.tags_rename.mapping]
    # tag1 = "new-name-1"
    # tag2 = "new-name-2"
    # tag3 = "new-name-3"

  # Send collected metrics to center as log.
  # When 'service' field is empty, using 'service tag' as measurement set name.
  [inputs.prom.as_logging]
    enable = false
    service = "service_name"

  ## Customize tags.
  [inputs.prom.tags]
    tidb_cluster="test"

  1. Copy the sample file 

    cd /usr/local/datakit/conf.d/prom/
cp prom.conf.sample tidb_tiflash_prom.conf

  2. Adjust tidb_tiflash_prom.conf

tidb_tiflash_prom.conf 
[[inputs.prom]]
  # Exporter URLs.
  # urls = ["http://127.0.0.1:9100/metrics", "http://127.0.0.1:9200/metrics"]
  urls = ["http://0.0.0.0:8234/metrics","http://0.0.0.0:20292/metrics"]
  # Unix Domain Socket URL. Using socket to request data when not empty.
  uds_path = ""

  # Ignore URL request errors.
  ignore_req_err = false

  ## Collector alias.
  source = "tidb_tiflash"

  ## Collect data output.
  # Fill this when want to collect the data to local file nor center.
  # After filling, could use 'datakit --prom-conf /path/to/this/conf' to debug local storage measurement set.
  # Using '--prom-conf' when priority debugging data in 'output' path.
  # output = "/abs/path/to/file"

  ## Collect data upper limit as bytes.
  # Only available when set output to local file.
  # If collect data exceeded the limit, the data would be dropped.
  # Default is 32MB.
  # max_file_size = 0

  ## Metrics type whitelist. Optional: counter, gauge, histogram, summary
  # Default only collect 'counter' and 'gauge'.
  # Collect all if empty.
  metric_types = ["counter", "gauge"]

  ## Metrics name whitelist.
  # Regex supported. Multi supported, conditions met when one matched.
  # Collect all if empty.
  # metric_name_filter = ["cpu"]

  ## Measurement prefix.
  # Add prefix to measurement set name.
  measurement_prefix = "tidb_tiflash_"

  ## Measurement name.
  # If measurement_name is empty, split metric name by '_', the first field after split as measurement set name, the rest as current metric name.
  # If measurement_name is not empty, using this as measurement set name.
  # Always add 'measurement_prefix' prefix at last.
  # measurement_name = "prom"

  ## TLS configuration.
  tls_open = false
  # tls_ca = "/tmp/ca.crt"
  # tls_cert = "/tmp/peer.crt"
  # tls_key = "/tmp/peer.key"

  ## Set to 'true' to enable election.
  election = true

  # Ignore tags. Multi supported.
  # The matched tags would be dropped, but the item would still be sent.
  # tags_ignore = ["xxxx"]

  ## Customize authentication. For now supports Bearer Token only.
  # Filling in 'token' or 'token_file' is acceptable.
  # [inputs.prom.auth]
  # type = "bearer_token"
  # token = "xxxxxxxx"
  # token_file = "/tmp/token"

  ## Customize measurement set name.
  # Treat those metrics with prefix as one set.
  # Prioritier over 'measurement_name' configuration.
  #[[inputs.prom.measurements]]
  #  prefix = "cpu_"
  #  name = "cpu"

  # [[inputs.prom.measurements]]
  # prefix = "mem_"
  # name = "mem"

  # Not collecting those data when tag matched.
  [inputs.prom.ignore_tag_kv_match]
  # key1 = [ "val1.*", "val2.*"]
  # key2 = [ "val1.*", "val2.*"]

  # Add HTTP headers to data pulling.
  [inputs.prom.http_headers]
  # Root = "passwd"
  # Michael = "1234"

  # Rename tag key in prom data.
  [inputs.prom.tags_rename]
    overwrite_exist_tags = false
    [inputs.prom.tags_rename.mapping]
    # tag1 = "new-name-1"
    # tag2 = "new-name-2"
    # tag3 = "new-name-3"

  # Send collected metrics to center as log.
  # When 'service' field is empty, using 'service tag' as measurement set name.
  [inputs.prom.as_logging]
    enable = false
    service = "service_name"

  ## Customize tags.
  [inputs.prom.tags]
    tidb_cluster="test"

Note that adjustments are required for marked places

Main parameter explanations:

  • urls: prometheus Metrics address, fill in the corresponding component exposed Metrics url here
  • source: Collector alias, recommended to differentiate
  • interval: Collection interval
  • measurement_prefix: Metrics set prefix, convenient for management and classification
  • tls_open: TLS configuration
  • metric_types: Metrics types, leaving it blank means collecting all Metrics, it's recommended to fill it according to needs, involving Time Series
  • tags_ignore: Ignore unnecessary tags
  • [inputs.prom.tags_rename.mapping]: Tag renaming, if there's a conflict between a tag and a field, the tag should be renamed; otherwise, the entire Metrics cannot be collected.
  • [inputs.prom.tags] : Set tags, applied to all Metrics of the current Metrics

Restart DataKit

systemctl restart datakit

Feedback

Is this page helpful? ×