Skip to content

TiDB

The TiDB view displays information including the Summary (such as: start time, storage information, node information, etc.), cluster (cluster-related information), TiDB, Etcd, Region, and other related Metrics information.

Installation and Configuration

Version Support

Version support depends on TiDB's own system support.

Note: The example TiDB version is 6.3+

(Linux / Windows environment is the same)

If you need to install TiDB, refer to the documentation Deploying a Local Test Cluster for TiDB

The following collection process follows the deployment method outlined in Deploying a Local Test Cluster for TiDB for data collection.

Metrics Collection Configuration

All components of TiDB (a total of 4 components) have exposed metrics, with the protocol being http

Component metrics port
TiDB 10080
pd 2379
TiKV 20180
TiFlash 8234
20292

The above are the metrics ports related to single-node cluster component deployments. If it's a multi-node cluster, the port configuration is similar.

Enable DataKit Collector

  1. Copy the sample file 

      cd /usr/local/datakit/conf.d/prom/
  cp prom.conf.sample tidb_prom.conf

  2. Adjust tidb_prom.conf

tidb_prom.conf 
[[inputs.prom]]
  # Exporter URLs.
  # urls = ["http://127.0.0.1:9100/metrics", "http://127.0.0.1:9200/metrics"]
  urls = ["http://0.0.0.0:10080/metrics"]
  # Unix Domain Socket URL. Using socket to request data when not empty.
  uds_path = ""

  # Ignore URL request errors.
  ignore_req_err = false

  ## Collector alias.
  source = "tidb_prom"

  ## Metrics type whitelist. Optional: counter, gauge, histogram, summary
  # Default only collect 'counter' and 'gauge'.
  # Collect all if empty.
  metric_types = ["counter", "gauge","histogram"]

  ## Measurement prefix.
  # Add prefix to measurement set name.
  measurement_prefix = "tidb_"

  ## TLS configuration.
  tls_open = false
  # tls_ca = "/tmp/ca.crt"
  # tls_cert = "/tmp/peer.crt"
  # tls_key = "/tmp/peer.key"

  ## Set to 'true' to enable election.
  election = true

  # Ignore tags. Multi supported.
  # The matched tags would be dropped, but the item would still be sent.
  # tags_ignore = ["xxxx"]

  ## Customize authentication. For now supports Bearer Token only.
  # Filling in 'token' or 'token_file' is acceptable.
  # [inputs.prom.auth]
  # type = "bearer_token"
  # token = "xxxxxxxx"
  # token_file = "/tmp/token"

  ## Customize measurement set name.
  # Treat those metrics with prefix as one set.
  # Prioritier over 'measurement_name' configuration.
  #[[inputs.prom.measurements]]
  #  prefix = "cpu_"
  #  name = "cpu"

  # [[inputs.prom.measurements]]
  # prefix = "mem_"
  # name = "mem"

  # Not collecting those data when tag matched.
  [inputs.prom.ignore_tag_kv_match]
  # key1 = [ "val1.*", "val2.*"]
  # key2 = [ "val1.*", "val2.*"]

  # Add HTTP headers to data pulling.
  [inputs.prom.http_headers]
  # Root = "passwd"
  # Michael = "1234"

  # Rename tag key in prom data.
  [inputs.prom.tags_rename]
    overwrite_exist_tags = false
    [inputs.prom.tags_rename.mapping]
    # tag1 = "new-name-1"
    # tag2 = "new-name-2"
    # tag3 = "new-name-3"

  # Send collected metrics to center as log.
  # When 'service' field is empty, using 'service tag' as measurement set name.
  [inputs.prom.as_logging]
    enable = false
    service = "service_name"

  ## Customize tags.
  [inputs.prom.tags]
    tidb_cluster="test"

  1. Copy the sample file 

    cd /usr/local/datakit/conf.d/prom/
cp prom.conf.sample tidb_pd_prom.conf

  2. Adjust tidb_pd_prom.conf

tidb_pd_prom.conf 
[[inputs.prom]]
  # Exporter URLs.
  # urls = ["http://127.0.0.1:9100/metrics", "http://127.0.0.1:9200/metrics"]
  urls = ["http://0.0.0.0:2379/metrics"]
  # Unix Domain Socket URL. Using socket to request data when not empty.
  uds_path = ""

  # Ignore URL request errors.
  ignore_req_err = false

  ## Collector alias.
  source = "tidb_pd_prom"

  ## Collect data output.
  # Fill this when want to collect the data to local file nor center.
  # After filling, could use 'datakit --prom-conf /path/to/this/conf' to debug local storage measurement set.
  # Using '--prom-conf' when priority debugging data in 'output' path.
  # output = "/abs/path/to/file"

  ## Collect data upper limit as bytes.
  # Only available when set output to local file.
  # If collect data exceeded the limit, the data would be dropped.
  # Default is 32MB.
  # max_file_size = 0

  ## Metrics type whitelist. Optional: counter, gauge, histogram, summary
  # Default only collect 'counter' and 'gauge'.
  # Collect all if empty.
  metric_types = ["counter", "gauge"]

  ## Metrics name whitelist.
  # Regex supported. Multi supported, conditions met when one matched.
  # Collect all if empty.
  # metric_name_filter = ["cpu"]

  ## Measurement prefix.
  # Add prefix to measurement set name.
  measurement_prefix = "tidb_pd_"

  ## Measurement name.
  # If measurement_name is empty, split metric name by '_', the first field after split as measurement set name, the rest as current metric name.
  # If measurement_name is not empty, using this as measurement set name.
  # Always add 'measurement_prefix' prefix at last.
  # measurement_name = "prom"

  ## TLS configuration.
  tls_open = false
  # tls_ca = "/tmp/ca.crt"
  # tls_cert = "/tmp/peer.crt"
  # tls_key = "/tmp/peer.key"

  ## Set to 'true' to enable election.
  election = true

  # Ignore tags. Multi supported.
  # The matched tags would be dropped, but the item would still be sent.
  # tags_ignore = ["xxxx"]

  ## Customize authentication. For now supports Bearer Token only.
  # Filling in 'token' or 'token_file' is acceptable.
  # [inputs.prom.auth]
  # type = "bearer_token"
  # token = "xxxxxxxx"
  # token_file = "/tmp/token"

  ## Customize measurement set name.
  # Treat those metrics with prefix as one set.
  # Prioritier over 'measurement_name' configuration.
  #[[inputs.prom.measurements]]
  #  prefix = "cpu_"
  #  name = "cpu"

  # [[inputs.prom.measurements]]
  # prefix = "mem_"
  # name = "mem"

  # Not collecting those data when tag matched.
  [inputs.prom.ignore_tag_kv_match]
  # key1 = [ "val1.*", "val2.*"]
  # key2 = [ "val1.*", "val2.*"]

  # Add HTTP headers to data pulling.
  [inputs.prom.http_headers]
  # Root = "passwd"
  # Michael = "1234"

  # Rename tag key in prom data.
  [inputs.prom.tags_rename]
    #overwrite_exist_tags = false
    overwrite_exist_tags = false
    [inputs.prom.tags_rename.mapping]
      cluster_version = "version"
      server_id = "s_id"
      server_version = "version"
      server_go_version = "version"
    # tag1 = "new-name-1"
    # tag2 = "new-name-2"
    # tag3 = "new-name-3"

  # Send collected metrics to center as log.
  # When 'service' field is empty, using 'service tag' as measurement set name.
  [inputs.prom.as_logging]
    enable = false
    service = "service_name"

  ## Customize tags.
  [inputs.prom.tags]
    tidb_cluster="test"

  1. Copy the sample file 

    cd /usr/local/datakit/conf.d/prom/
cp prom.conf.sample tidb_tikv_prom.conf

  2. Adjust tidb_tikv_prom.conf

tidb_tikv_prom.conf 
[[inputs.prom]]
  # Exporter URLs.
  # urls = ["http://127.0.0.1:9100/metrics", "http://127.0.0.1:9200/metrics"]
  urls = ["http://0.0.0.0:20180/metrics"]
  # Unix Domain Socket URL. Using socket to request data when not empty.
  uds_path = ""

  # Ignore URL request errors.
  ignore_req_err = false

  ## Collector alias.
  source = "tidb_tikv_prom"

  ## Collect data output.
  # Fill this when want to collect the data to local file nor center.
  # After filling, could use 'datakit --prom-conf /path/to/this/conf' to debug local storage measurement set.
  # Using '--prom-conf' when priority debugging data in 'output' path.
  # output = "/abs/path/to/file"

  ## Collect data upper limit as bytes.
  # Only available when set output to local file.
  # If collect data exceeded the limit, the data would be dropped.
  # Default is 32MB.
  # max_file_size = 0

  ## Metrics type whitelist. Optional: counter, gauge, histogram, summary
  # Default only collect 'counter' and 'gauge'.
  # Collect all if empty.
  metric_types = ["counter", "gauge"]

  ## Metrics name whitelist.
  # Regex supported. Multi supported, conditions met when one matched.
  # Collect all if empty.
  # metric_name_filter = ["cpu"]

  ## Measurement prefix.
  # Add prefix to measurement set name.
  measurement_prefix = "tidb_tikv_"

  ## Measurement name.
  # If measurement_name is empty, split metric name by '_', the first field after split as measurement set name, the rest as current metric name.
  # If measurement_name is not empty, using this as measurement set name.
  # Always add 'measurement_prefix' prefix at last.
  # measurement_name = "prom"

  ## TLS configuration.
  tls_open = false
  # tls_ca = "/tmp/ca.crt"
  # tls_cert = "/tmp/peer.crt"
  # tls_key = "/tmp/peer.key"

  ## Set to 'true' to enable election.
  election = true

  # Ignore tags. Multi supported.
  # The matched tags would be dropped, but the item would still be sent.
  # tags_ignore = ["xxxx"]

  ## Customize authentication. For now supports Bearer Token only.
  # Filling in 'token' or 'token_file' is acceptable.
  # [inputs.prom.auth]
  # type = "bearer_token"
  # token = "xxxxxxxx"
  # token_file = "/tmp/token"

  ## Customize measurement set name.
  # Treat those metrics with prefix as one set.
  # Prioritier over 'measurement_name' configuration.
  #[[inputs.prom.measurements]]
  #  prefix = "cpu_"
  #  name = "cpu"

  # [[inputs.prom.measurements]]
  # prefix = "mem_"
  # name = "mem"

  # Not collecting those data when tag matched.
  [inputs.prom.ignore_tag_kv_match]
  # key1 = [ "val1.*", "val2.*"]
  # key2 = [ "val1.*", "val2.*"]

  # Add HTTP headers to data pulling.
  [inputs.prom.http_headers]
  # Root = "passwd"
  # Michael = "1234"

  # Rename tag key in prom data.
  [inputs.prom.tags_rename]
    overwrite_exist_tags = false
    [inputs.prom.tags_rename.mapping]
    # tag1 = "new-name-1"
    # tag2 = "new-name-2"
    # tag3 = "new-name-3"

  # Send collected metrics to center as log.
  # When 'service' field is empty, using 'service tag' as measurement set name.
  [inputs.prom.as_logging]
    enable = false
    service = "service_name"

  ## Customize tags.
  [inputs.prom.tags]
    tidb_cluster="test"

  1. Copy the sample file 

    cd /usr/local/datakit/conf.d/prom/
cp prom.conf.sample tidb_tiflash_prom.conf

  2. Adjust tidb_tiflash_prom.conf

tidb_tiflash_prom.conf 
[[inputs.prom]]
  # Exporter URLs.
  # urls = ["http://127.0.0.1:9100/metrics", "http://127.0.0.1:9200/metrics"]
  urls = ["http://0.0.0.0:8234/metrics","http://0.0.0.0:20292/metrics"]
  # Unix Domain Socket URL. Using socket to request data when not empty.
  uds_path = ""

  # Ignore URL request errors.
  ignore_req_err = false

  ## Collector alias.
  source = "tidb_tiflash"

  ## Collect data output.
  # Fill this when want to collect the data to local file nor center.
  # After filling, could use 'datakit --prom-conf /path/to/this/conf' to debug local storage measurement set.
  # Using '--prom-conf' when priority debugging data in 'output' path.
  # output = "/abs/path/to/file"

  ## Collect data upper limit as bytes.
  # Only available when set output to local file.
  # If collect data exceeded the limit, the data would be dropped.
  # Default is 32MB.
  # max_file_size = 0

  ## Metrics type whitelist. Optional: counter, gauge, histogram, summary
  # Default only collect 'counter' and 'gauge'.
  # Collect all if empty.
  metric_types = ["counter", "gauge"]

  ## Metrics name whitelist.
  # Regex supported. Multi supported, conditions met when one matched.
  # Collect all if empty.
  # metric_name_filter = ["cpu"]

  ## Measurement prefix.
  # Add prefix to measurement set name.
  measurement_prefix = "tidb_tiflash_"

  ## Measurement name.
  # If measurement_name is empty, split metric name by '_', the first field after split as measurement set name, the rest as current metric name.
  # If measurement_name is not empty, using this as measurement set name.
  # Always add 'measurement_prefix' prefix at last.
  # measurement_name = "prom"

  ## TLS configuration.
  tls_open = false
  # tls_ca = "/tmp/ca.crt"
  # tls_cert = "/tmp/peer.crt"
  # tls_key = "/tmp/peer.key"

  ## Set to 'true' to enable election.
  election = true

  # Ignore tags. Multi supported.
  # The matched tags would be dropped, but the item would still be sent.
  # tags_ignore = ["xxxx"]

  ## Customize authentication. For now supports Bearer Token only.
  # Filling in 'token' or 'token_file' is acceptable.
  # [inputs.prom.auth]
  # type = "bearer_token"
  # token = "xxxxxxxx"
  # token_file = "/tmp/token"

  ## Customize measurement set name.
  # Treat those metrics with prefix as one set.
  # Prioritier over 'measurement_name' configuration.
  #[[inputs.prom.measurements]]
  #  prefix = "cpu_"
  #  name = "cpu"

  # [[inputs.prom.measurements]]
  # prefix = "mem_"
  # name = "mem"

  # Not collecting those data when tag matched.
  [inputs.prom.ignore_tag_kv_match]
  # key1 = [ "val1.*", "val2.*"]
  # key2 = [ "val1.*", "val2.*"]

  # Add HTTP headers to data pulling.
  [inputs.prom.http_headers]
  # Root = "passwd"
  # Michael = "1234"

  # Rename tag key in prom data.
  [inputs.prom.tags_rename]
    overwrite_exist_tags = false
    [inputs.prom.tags_rename.mapping]
    # tag1 = "new-name-1"
    # tag2 = "new-name-2"
    # tag3 = "new-name-3"

  # Send collected metrics to center as log.
  # When 'service' field is empty, using 'service tag' as measurement set name.
  [inputs.prom.as_logging]
    enable = false
    service = "service_name"

  ## Customize tags.
  [inputs.prom.tags]
    tidb_cluster="test"

Note that adjustments need to be made where marked

Main parameter descriptions:

  • urls: prometheus Metrics address, fill in the Metrics url exposed by the corresponding component here.
  • source: Collector alias, it's recommended to make distinctions.
  • interval: Collection interval.
  • measurement_prefix: Measurement prefix, convenient for management classification.
  • tls_open: TLS configuration.
  • metric_types: Metrics types, leaving it blank represents collecting all Metrics, it's recommended to fill in as needed, involving Time Series.
  • tags_ignore: Ignore unnecessary tags.
  • [inputs.prom.tags_rename.mapping]: Tag renaming, if a tag has the same name as a field, then the tag needs to be renamed, otherwise the entire Metric cannot be collected.
  • [inputs.prom.tags]: Set tags, applied to all Metrics under the current metrics.

Restart DataKit

systemctl restart datakit

Feedback

Is this page helpful? ×