MongoDB

·

MongoDb database, Collection, MongoDb database cluster running status data Collection.

Config¶

Preconditions¶

• Already tested version:

  • 6.0
  • 5.0
  • 4.0
  • 3.0
  • 2.8.0

• Developed and used MongoDB version 4.4.5;

• Write the configuration file in the corresponding directory and then start DataKit to complete the configuration;
• For secure connections using TLS, please configure the response certificate file path and configuration under ## TLS connection config in the configuration file;
• If MongoDB has access control enabled, you need to configure the necessary user rights to establish an authorized connection:

# Run MongoDB shell.
$ mongo

# Authenticate as the admin/root user.
> use admin
> db.auth("<admin OR root>", "<YOUR_MONGODB_ADMIN_PASSWORD>")

# Create the user for the Datakit.
> db.createUser({
  "user": "datakit",
  "pwd": "<YOUR_COLLECT_PASSWORD>",
  "roles": [
    { role: "read", db: "admin" },
    { role: "clusterMonitor", db: "admin" },
    { role: "backup", db: "admin" },
    { role: "read", db: "local" }
  ]
})

More authorization information can refer to official documentation Built-In Roles。

After done with commands above, filling the user and pwd to Datakit configuration file conf.d/db/mongodb.conf.

Collector Configuration¶

[[inputs.mongodb]]
  ## Gathering interval
  interval = "10s"

  ## Specify one single Mongodb server. These server related fields will be ignored when the 'servers' field is not empty.
  ## connection_format is a string in the standard connection format (mongodb://) or SRV connection format (mongodb+srv://).
  connection_format = "mongodb://"

  ## The host and port. 
  host_port = "127.0.0.1:27017"

  ## Username
  username = "datakit"

  ## Password
  password = "<PASS>"

  ## The authentication database to use.
  # default_db = "admin"

  ## A query string that specifies connection specific options as <name>=<value> pairs.
  # query_string = "authSource=admin&authMechanism=SCRAM-SHA-256"

  ## A list of Mongodb servers URL
  ## Note: must escape special characters in password before connect to Mongodb server, otherwise parse will failed.
  ## Form: "mongodb://[user ":" pass "@"] host [ ":" port]"
  ## Some examples:
  ## mongodb://user:pswd@localhost:27017/?authMechanism=SCRAM-SHA-256&authSource=admin
  ## mongodb://user:pswd@127.0.0.1:27017,
  ## mongodb://10.10.3.33:18832,
  # servers = ["mongodb://127.0.0.1:27017"]

  ## When true, collect replica set stats
  gather_replica_set_stats = false

  ## When true, collect cluster stats
  ## Note that the query that counts jumbo chunks triggers a COLLSCAN, which may have an impact on performance.
  gather_cluster_stats = false

  ## When true, collect per database stats
  gather_per_db_stats = true

  ## When true, collect per collection stats
  gather_per_col_stats = true

  ## List of db where collections stats are collected, If empty, all dbs are concerned.
  col_stats_dbs = []

  ## When true, collect top command stats.
  gather_top_stat = true

  ## Set true to enable election
  election = true

  ## TLS connection config
  # ca_certs = ["/etc/ssl/certs/mongod.cert.pem"]
  # cert = "/etc/ssl/certs/mongo.cert.pem"
  # cert_key = "/etc/ssl/certs/mongo.key.pem"
  # insecure_skip_verify = true
  # server_name = ""

  ## Mongodb log files and Grok Pipeline files configuration
  # [inputs.mongodb.log]
    # files = ["/var/log/mongodb/mongod.log"]
    # pipeline = "mongod.p"

  ## Customer tags, if set will be seen with every metric.
  # [inputs.mongodb.tags]
    # "key1" = "value1"
    # "key2" = "value2"
    # ...

TLS config (self-signed)¶

Use OpenSSL to generate a certificate file for MongoDB TLS configuration to enable server-side encryption and client-side authentication.

• Configure TLS certificates

Install OpenSSL and run the following command:

sudo apt install openssl -y

• Configure MongoDB server-side encryption

Use OpenSSL to generate a certificate-level key file, run the following command and enter the corresponding authentication block information at the command prompt:

sudo openssl req -x509 -newkey rsa:<bits> -days <days> -keyout <mongod.key.pem> -out <mongod.cert.pem> -nodes

• bits: rsa key digits, for example, 2048
• days: expired date
• mongod.key.pem: key file
• mongod.cert.pem: CA certificate file

Running the above command generates the cert.pem file and the key.pem file, and we need to merge the block inside the two files to run the following command:

sudo bash -c "cat mongod.cert.pem mongod.key.pem >>mongod.pem"

Configure the TLS subentry in the /etc/mongod.config file after merging

# TLS config
net:
  tls:
    mode: requireTLS
    certificateKeyFile: </etc/ssl/mongod.pem>

Start MongoDB with the configuration file and run the following command:

mongod --config /etc/mongod.conf

Start MongoDB from the command line and run the following command:

mongod --tlsMode requireTLS --tlsCertificateKeyFile </etc/ssl/mongod.pem> --dbpath <.db/mongodb>

Copy mongod.cert.pem as mongo.cert.pem to MongoDB client and enable TLS:

mongo --tls --host <mongod_url> --tlsCAFile </etc/ssl/mongo.cert.pem>

• Configuring MongoDB Client Authentication

Use OpenSSL to generate a certificate-level key file and run the following command:

sudo openssl req -x509 -newkey rsa:<bits> -days <days> -keyout <mongod.key.pem> -out <mongod.cert.pem> -nodes

• bits: rsa key digits, for example, 2048
• days: expired date
• mongo.key.pem: key file
• mongo.cert.pem: CA certificate file

Merging the block in the mongod.cert.pem and mongod.key.pem files runs the following command:

sudo bash -c "cat mongod.cert.pem mongod.key.pem >>mongod.pem"

Copy the mongod.cert.pem file to the MongoDB server and configure the TLS entry in the /etc/mongod.config file.

# Tls config
net:
  tls:
    mode: requireTLS
    certificateKeyFile: </etc/ssl/mongod.pem>
    CAFile: </etc/ssl/mongod.cert.pem>

Start MongoDB and run the following command:

mongod --config /etc/mongod.conf

Copy mongod.cert.pem for mongo.cert.pem; Copy mongod.pem for mongo.pem to MongoDB client and enable TLS:

mongo --tls --host <mongod_url> --tlsCAFile </etc/ssl/mongo.cert.pem> --tlsCertificateKeyFile </etc/ssl/mongo.pem>

Note:insecure_skip_verify must be true in mongodb.conf configuration when using self-signed certificates.

Metric¶

For all of the following data collections, the global election tags will added automatically, we can add extra tags in [inputs.mongodb.tags] if needed:

 [inputs.mongodb.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"
  # ...

mongodb_db_stats¶

• explain

MongoDB stats measurement. Some metrics may not appear depending on the MongoDB version or DB running status.

• Tags

• Metrics

mongodb_col_stats¶

• explain

MongoDB collection measurement. Some metrics may not appear depending on the MongoDB version or DB running status.

• Tags

• Metrics

mongodb_shard_stats¶

• explain

MongoDB shard measurement. Some metrics may not appear depending on the MongoDB version or DB running status.

• Tags

• Metrics

mongodb_top_stats¶

• explain

MongoDB top measurement. Some metrics may not appear depending on the MongoDB version or DB running status.

• Tags

• Metrics

Custom Object¶

database¶

• Tags

• Metrics

Mongod Log Collection¶

Annotate the configuration file # enable_mongod_log = false and change false to true. Other configuration options for mongod log are in [inputs.mongodb.log], and the commented configuration is very default. If the path correspondence is correct, no configuration is needed. After starting Datakit, you will see a collection measurement named mongod_log.

Log raw data sample

{"t":{"$date":"2021-06-03T09:12:19.977+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"WTCheckpointThread","msg":"WiredTiger message","attr":{"message":"[1622711539:977142][1:0x7f1b9f159700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 653, snapshot max: 653 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0)"}}

Log cut field