eBPF

eBPF collector, collecting host network TCP, UDP connection information, Bash execution log, etc. This collector mainly includes ebpf-net, ebpf-conntrack and ebpf-bash three plugins:

• ebpf-net:

  • Data category: Network
  • It is composed of netflow, httpflow and dnsflow, which are used to collect host TCP/UDP connection statistics and host DNS resolution information respectively;

• ebpf-bash:

  • Data category: Logging
  • Collect Bash execution log, including Bash process number, user name, executed command and time, etc.;

• ebpf-conntrack: Version-1.8.0 · Experimental

  • Add two tags dst_nat_ip and dst_nat_port to the network flow data.

• ebpf-trace:

  • Application call relationship tracking.

• bpf-netlog:

  • Data categories: Logging, Network
  • This plugin implements the collection of network logs bpf_net_l4_log/bpf_net_l7_log, and can also replace ebpf-net's netflow/httpflow data collection when the kernel does not support eBPF;

Configuration¶

Preconditions¶

For DataKit before v1.5.6, you need to execute the installation command to install:

• v1.2.13 ~ v1.2.18
  • Install time specify environment variable：DK_INSTALL_EXTERNALS="datakit-ebpf"
  • After the DataKit is installed, manually install the eBPF collector: datakit install --datakit-ebpf
• v1.2.19+
  • specify environment variable：DK_INSTALL_EXTERNALS="ebpf" when installing
  • After the DataKit is installed, manually install the eBPF collector: datakit install --ebpf
• v1.5.6+
  • No manual installation required

When deploying in Kubernetes environment, you must mount the host's' /sys/kernel/debug directory into pod, refer to the latest datakit.yaml;

Linux Kernel Version Requirement¶

In addition to CentOS 7.6+ and Ubuntu 16.04, other distributions recommend that the Linux kernel version is higher than 4.9, otherwise the eBPF collector may not start.

If you want to enable the eBPF-conntrack plugin, usually requires a higher kernel version, such as v5.4.0 etc., please confirm whether the symbols in the kernel contain nf_ct_delete and __nf_conntrack_hash_insert, you can execute the following command to view:

cat /proc/kallsyms | awk '{print $3}' | grep "^nf_ct_delete$\|^__nf_conntrack_hash_insert$"

SELinux-enabled System¶

For SELinux-enabled systems, you need to shut them down (pending subsequent optimization), and execute the following command to shut them down:

setenforce 0

Collector Configuration¶

[[inputs.ebpf]]
  daemon = true
  name = 'ebpf'
  cmd = "/usr/local/datakit/externals/datakit-ebpf"
  args = [
    "--datakit-apiserver", "0.0.0.0:9529",
  ]
  envs = []

  ## Resource limits.
  ## The collector automatically exits when the limit is exceeded.
  ## Can configure the number of cpu cores, memory size and network bandwidth.
  ##
  # cpu_limit = "2.0"
  # mem_limit = "4GiB"
  # net_limit = "100MiB/s"

  ## automatically takes effect when running DataKit in 
  ## Kubernetes daemonset mode
  ##
  # kubernetes_url = "https://kubernetes.default:443"
  # bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
  ##
  ## or 
  # bearer_token_string = "<your-token-string>"

  ## all supported plugins:
  ## - "ebpf-net"  :
  ##     contains L4-network(netflow), L7-network(httpflow, dnsflow) collection
  ## - "ebpf-bash" :
  ##     log bash
  ## - "ebpf-conntrack":
  ##     add two tags "dst_nat_ip" and "dst_nat_port" to the network flow data
  ## - "ebpf-trace":
  ##     param trace_server must be set simultaneously.
  ## - "bpf-netlog":
  ##     contains L4-network log (bpf_net_l4_log), L7-network log (bpf_net_l7_log), 
  ##              L4-network(netflow), L7-network(httpflow, dnsflow) collection
  enabled_plugins = [
    "ebpf-net",
  ]

  ## If you enable the ebpf-net plugin, you can configure:
  ##  - "httpflow" (* enabled by default)
  ##  - "httpflow-tls"
  ##
  l7net_enabled = [
    "httpflow",
    # "httpflow-tls"
  ]


  ## datakit-ebpf pprof service
  pprof_host = "127.0.0.1"
  pprof_port = "6061"

  ## netlog blacklist
  ##
  # netlog_blacklist = "ip_saddr=='127.0.0.1' || ip_daddr=='127.0.0.1'"

  ## bpf-netlog plugin collection metric and log
  ##
  netlog_metric = true
  netlog_log = false

  ## eBPF trace generation server center address.
  trace_server = ""

  ## trace all processes directly
  ##
  trace_all_process = false

  ## trace all processes containing any specified environment variable
  trace_env_list = [
    "DK_BPFTRACE_SERVICE",
    "DD_SERVICE",
    "OTEL_SERVICE_NAME",
  ]

  ## deny tracking any process containing any specified environment variable
  trace_env_blacklist = []

  ## trace all processes containing any specified process names,
  ## can be used with trace_namedenyset
  ##
  trace_name_list = []

  ## deny tracking any process containing any specified process names
  ##
  trace_name_blacklist = [

    ## The following two processes are hard-coded to never be traced,
    ## and do not need to be set:
    ##
    # "datakit",
    # "datakit-ebpf",
  ]

  ## conv other trace id to datadog trace id (base 10, 64-bit) 
  conv_to_ddtrace = false

  ## If the system does not enable ipv6, it needs to be changed to true
  ##
  ipv6_disabled = false

  ## ephemeral port strart from <ephemeral_port>
  ##
  # ephemeral_port = 10001

  # interval = "60s"

  # sampling_rate = "0.50"
  # sampling_rate_pts_per_min = "1500"

  [inputs.ebpf.tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"

#############################
## Parameter description (if marked * is required)
#############################
##  --hostname               : Host name, this parameter can change the value of the host tag when the collector uploads data, the priority is: specify this parameter >
##                             ENV_HOSTNAME value in datakit.conf (if it is not empty, this parameter will be added automatically at startup) >
##                             collector Get it yourself (the default).
##  --datakit-apiserver      : DataKit API Server address, default value 0.0.0.0:9529 .
##  --log                    : Log output path, default <DataKitInstallDir>/externals/datakit-ebpf.log.
##  --log-level              : Log level, the default value is 'info'.
##  --service                : The default value is 'ebpf'.

- name: ENV_ENABLE_INPUTS
       value: cpu,disk,diskio,mem,swap,system,hostobject,net,host_processes,container,ebpf

Environment variables and configuration items¶

The following environment variables can be used to adjust the eBPF collection configuration in Kubernetes:

Configuration items:

• enabled_plugins:

  • Description: Used to configure the built-in plugins for the collector
  • Environment variable: ENV_INPUT_EBPF_ENABLED_PLUGINS
  • Example: ebpf-net,ebpf-trace

• l7net_enabled

  • Description: Enable http protocol data collection
  • Environment variable: ENV_INPUT_EBPF_L7NET_ENABLED
  • Example: httpflow

• interval

  • Description: Set the sampling time interval
  • Environment variable: ENV_INPUT_EBPF_INTERVAL
  • Example: 1m30s

• ipv6_disabled

  • Description: Whether the system does not support IPv6
  • Environment variable: ENV_INPUT_EBPF_IPV6_DISABLED
  • Example: false

• ephemeral_port

  • Description: Ephemeral port start location
  • Environment variable: ENV_INPUT_EBPF_EPHEMERAL_PORT
  • Example: 32768

• pprof_host

  • Description: pprof host
  • Environment variable: ENV_INPUT_EBPF_PPROF_HOST
  • Example: 127.0.0.1

• pprof_port

  • Description: pprof port
  • Environment variable: ENV_INPUT_EBPF_PPROF_PORT
  • Example: 6061

• trace_server

  • Description: The address of DataKit ELinker/Datakit to enable the ebpftrace collector
  • Environment variable: ENV_INPUT_EBPF_TRACE_SERVER
  • Example: <ip>:<port>

• trace_all_process

  • Description: Trace all processes in the system
  • Environment variable: ENV_INPUT_EBPF_TRACE_ALL_PROCESS
  • Example: false

• trace_name_blacklist

  • Description: The process with the specified process name will be disabled from collecting trace data
  • Environment variable: ENV_INPUT_EBPF_TRACE_NAME_BLACKLIST
  • Example:

• trace_env_blacklist

  • Description: Any process containing any of the specified environment variable names will be disabled from collecting trace data
  • Environment variable: ENV_INPUT_EBPF_TRACE_ENV_BLACKLIST
  • Example: DKE_DISABLE_ETRACE

• trace_env_list

  • Description: Link data for processes with any specified environment variables will be traced and reported
  • Environment variable: ENV_INPUT_EBPF_TRACE_ENV_LIST
  • Example: DK_BPFTRACE_SERVICE,DD_SERVICE,OTEL_SERVICE_NAME

• trace_name_list

  • Description: Processes whose names are in the specified set will be traced and reported
  • Environment variable: ENV_INPUT_EBPF_TRACE_NAME_LIST
  • Example: chrome,firefox

• conv_to_ddtrace

  • Description: Convert all application side link IDs to decimal strings for compatibility purposes, not used unless necessary
  • Environment variable: ENV_INPUT_EBPF_CONV_TO_DDTRACE
  • Example: false

• netlog_blacklist

  • Description: Used to filter packets after packet capture
  • Environment variable: ENV_INPUT_EBPF_NETLOG_BLACKLIST
  • Example: ip_saddr=='127.0.0.1' \|\| ip_daddr=='127.0.0.1'

• netlog_metric

  • Description: Collect network metrics from network packet analysis
  • Environment variable: ENV_INPUT_EBPF_NETLOG_METRIC
  • Example: true

• netlog_log

  • Description: Collect network logs from network packet analysis
  • Environment variable: ENV_INPUT_EBPF_NETLOG_LOG
  • Example: false

• cpu_limit

  • Description: The maximum number of CPU cores used per unit time. When the upper limit is reached, the collector exits.
  • Environment variable: ENV_INPUT_EBPF_CPU_LIMIT
  • Example: "2.0"`

• mem_limit

  • Description: Memory size usage limit
  • Environment variable: ENV_INPUT_EBPF_MEM_LIMIT
  • Example: "4GiB"

• net_limit

  • Description: Network bandwidth (any network card) limit
  • Environment variable: ENV_INPUT_EBPF_NET_LIMIT
  • Example: "100MiB/s"

• sampling_rate

  • Description: The sampling rate when the eBPF collector reports data, ranging from 0.01 to 1.00; Mutually exclusive with the samping_rate_pts_per_min setting
  • Environment variable: ENV_INPUT_EBPF_SAMPLING_RATE
  • Example: 0.50

• sampling_rate_pts_per_min

  • Description: Set the data volume threshold per minute when the eBPF collector reports data, and dynamically adjust the sampling rate
  • Environment variable: ENV_INPUT_EBPF_SAMPLING_RATE_PTSPERMIN
  • Example: 1500

eBPF Tracing function¶

ebpf-trace collects and analyzes the network data read and written by the process on the host, and tracks the kernel-level threads/user-level threads (such as golang goroutine) of the process to generate link eBPF Span. This data needs to be collected by ebpftrace for further processing.

When using, you need to deploy the eBPF collector with link data collection enabled on multiple nodes, then you need to send all eBPF Span data to the same DataKit ELinker/DataKit with the ebpftrace collector plug-in enabled. For more configuration details, see the eBPF link document

The blacklist function of the bpf-netlog plug-in¶

Filter rule example:

Single rule:

The following rules filter network data with ip 1.1.1.1 and port 80. (Line breaks allowed after operator)

(ip_saddr == "1.1.1.1" || ip_saddr == "1.1.1.1") &&
      (src_port == 80 || dst_port == 80)

Multiple rules:

Use ; or \n to separate the rules. If any rule is met, the data will be filtered.

udp
ip_saddr == "1.1.1.1" && (src_port == 80 || dst_port == 80);
ip_saddr == "10.10.0.1" && (src_port == 80 || dst_port == 80)

ipnet_contains("127.0.0.0/8", ip_saddr); ipv6

Data available for filtering:

This filter is used to filter network data. Comparable data is as follows:

Operator:

Operators from highest to lowest:

function:

1. ipnet_contains

   Function signature: fn ipnet_contains(ipnet: str, ipaddr: str) bool

   Description: Determine whether the address is within the specified network segment

   Example:

   ipnet_contains("127.0.0.0/8", ip_saddr)
   

   If the ip_saddr value is "127.0.0.1", then this rule returns true and the TCP connection packet/UDP packet will be filtered.

2. has_prefix

   Function signature: fn has_prefix(s: str, prefix: str) bool

   Description: Specifies whether the field contains a certain prefix

   Example:

   has_prefix(k8s_src_pod, "datakit-") || has_prefix(k8s_dst_pod, "datakit-")
   

   This rule returns true if the pod name is datakit-kfez321.

Network aggregation data¶

For all of the following data collections, a global tag named host is appended by default (the tag value is the host name of the DataKit), or other tags can be specified in the configuration by [inputs.ebpf.tags]:

 [inputs.ebpf.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"
  # ...

netflow¶

• tag list

• field list

dnsflow¶

• tag list

• field list

httpflow¶

• tag list

• field list

Logging¶

bash¶

• tag list

• field list

bpf_net_l4_log¶

• tag list

• field list

bpf_net_l7_log¶

• tag list

• field list

Tracing¶

dketrace¶

• tag list

• field list