Ingress Nginx (Prometheus)¶

Collect Ingress Nginx (Prometheus) related Metrics, APM traces, and LOG information

Configuration¶

Prerequisites¶

DataKit has been deployed. Refer to Kubernetes cluster <Install Datakit>
K8s environment has been deployed
Ingress-nginx version: 1.9.6

Configure Ingress-nginx Service ConfigMap Resource¶

Add the following configurations in the Yaml file where Ingress-nginx is installed:

apiVersion: v1
data:
  allow-snippet-annotations: "false"
  enable-opentracing: "true"
  opentracing-trust-incoming-span: "true"
  datadog-collector-host: "datakit-service.datakit.svc.cluster.local"
  datadog-collector-port: "9529"
  datadog-service-name: "ingress-nginx"
  datadog-environment: "testing"
  datadog-operation-name-override: "HTTP $request_method $service_name $uri $opentelemetry_trace_id"
  datadog-priority-sampling: "false"
  datadog-sample-rate: "1.0"
kind: ConfigMap
metadata:
  labels:
    app.Kubernetes.io/component: controller
    app.Kubernetes.io/instance: ingress-nginx
    app.Kubernetes.io/name: ingress-nginx
    app.Kubernetes.io/part-of: ingress-nginx
    app.Kubernetes.io/version: 1.9.6
  name: ingress-nginx-controller
  namespace: ingress-nginx

Apply this configuration using the following command:

kubectl apply -f ingress-nginx.yaml

Collector Configuration¶

Metrics Collection¶

Enable the KubernetesPrometheus collector by injecting its configuration via ConfigMap method

Add kubernetesprometheus.conf under the ConfigMap section of the datakit.yaml file

apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:
    Kubernetesprometheus.conf: |-
      [inputs.Kubernetesprometheus]
        [[inputs.Kubernetesprometheus.instances]]
          role       = "pod"
          #namespaces = ["ingress-nginx"]
          scheme     = "http"
          #selector   = "app=ingress-nginx"
          port       = "10254"
          path       = "/metrics"
          node_local = "true"
          [inputs.Kubernetesprometheus.instances.custom]
            measurement = "ingress-nginx"
            [inputs.Kubernetesprometheus.instances.custom.tags]
              instance         = "__Kubernetes_mate_instance"
              host             = "__Kubernetes_mate_host"
              pod_name         = "__Kubernetes_pod_name"
              pod_namespace    = "__Kubernetes_pod_namespace"

Mount kubernetesprometheus.conf to /usr/local/datakit/conf.d/kubernetesprometheus/kubernetesprometheus.conf under DataKit

        - mountPath: /usr/local/datakit/conf.d/Kubernetesprometheus/Kubernetesprometheus.conf
          name: datakit-conf
          subPath: Kubernetesprometheus.conf

Restart DataKit with the following commands

kubectl delete -f datakit.yaml
kubectl apply -f datakit.yaml

In the configuration file Ingress-nginx.yaml for installing Ingress-nginx-controller, set the type of the Service to NodePort, and expose port 10254 externally

spec:
  type: NodePort
......
    - name: prometheus
      port: 10254
      targetPort: prometheus

Run the following command to restart the Ingress-nginx service

kubectl apply -f ingress-nginx.yaml

Trace Collection¶

Enable DDTrace collector

Modify the datakit.yaml file and append DDtrace to the default enabled collectors' configuration

 - name: ENV_DEFAULT_ENABLED_INPUTS
   value: cpu,disk,diskio,mem,swap,system,hostobject,net,host_processes,container,ddtrace

Restart DataKit with the following commands

kubectl delete -f datakit.yaml
kubectl apply -f datakit.yaml

Download the DataKit Operator Yaml file

wget https://static.guance.com/datakit-operator/datakit-operator.yaml

Run the installation command

kubectl apply -f datakit-operator.yaml

Ingress Access Test

The type of the Ingress-nginx-controller Service is NodePort, mapping port 80 to port 30796 (Note: 192.168.0.5 is the IP address of the node running the Ingress-nginx Pod)

curl -H "Host:service.com" http://192.168.0.5:30796/service-a/api/a

Service-A received: Hello from Service-B

In the Guance APM section, you can see that the Ingress-nginx trace data is reported normally.

Log Collection¶

DataKit collects container standard output logs by default, no additional configuration is required.

Metrics¶

Ingress Nginx metrics are located under the ingress-nginx Measurement set. Below is an introduction to the relevant metrics for Ingress Nginx.

Metric	Description	Data Type	Unit
nginx_ingress_controller_requests	The total number of client requests	int	count
nginx_ingress_controller_nginx_process_connections	Current number of client connections with states	int	count
nginx_ingress_controller_success	Cumulative number of Ingress controller reload operations	int	count
nginx_ingress_controller_config_last_reload_successful	Whether the last configuration reload attempt was successful	int	count
nginx_ingress_controller_nginx_process_resident_memory_bytes	Number of bytes of memory in use	float	B
nginx_ingress_controller_nginx_process_cpu_seconds_total	CPU usage in seconds	float	B
nginx_process_resident_memory_bytes	Number of bytes of memory in use	int	B
nginx_ingress_controller_request_duration_seconds_bucket	Request processing time in milliseconds	int	count
nginx_ingress_controller_request_size_sum	Request length (including request line, header, and request body)	int	count
nginx_ingress_controller_response_size_sum	Response length (including request line, header, and request body)	int	count
nginx_ingress_controller_ssl_expire_time_seconds	Number of seconds since 1970 to SSL Certificate expiration	int	count

Best Practices¶