跳转至

eBPF


eBPF 采集器,采集主机网络 TCP、UDP 连接信息,Bash 执行日志等。采集器包含以下几个插件:

  • ebpf-net:

    • 数据类别: Network
    • 由 netflow/httpflow/dnsflow 构成,分别用于采集主机 TCP/UDP 连接统计信息,HTTP 请求信息和主机 DNS 解析信息;
  • ebpf-bash:

    • 数据类别: Logging
    • 采集 Bash 的执行日志,包含 Bash 进程号、用户名、执行的命令和时间等;
  • ebpf-conntrack: Version-1.8.0

    • 往网络流数据上添加两个标签 dst_nat_ip 和 dst_nat_port,记录经 DNAT 后的目标 ip 和 port;当内核加载 nf_conntrack 时可选择开启该插件;
  • ebpf-trace: Version-1.17.0 · Experimental

    • 数据类别: Tracing
    • 用于跟踪应用网络请求调用关系,基于 ebpf-net 的 httpflow 数据和 eBPF 探针实现;
  • bpf-netlog:

    • 数据类别: Logging, Network
    • 该插件实现网络日志 bpf_net_l4_log/bpf_net_l7_log 采集,也可以在内核不支持 eBPF 的情况下替代 ebpf-net 的 netflow/httpflow 数据采集;

配置

前置条件

在 Kubernetes 环境下部署时,需要将以下目录挂在到容器中:

  • /sys/kernel/debug
  • 将主机的 / 目录挂载为容器的 /rootfs 目录,并设置环境变量 HOST_ROOT="/rootfs" 和 HOST_PROC="/rootfs/proc"

可参考 datakit.yaml;

如果为低于 v1.5.6 版本的 DataKit ,则需手动安装 datakit-ebpf 外部采集器。

Linux 内核版本要求

支持原生 Linux i386/i686 内核,使用 linux/386 的 DataKit 和 datakit-ebpf。32 位采集器启动时会检查内核架构;在 x86_64 内核上应使用 linux/amd64 版本。内核必须启用 eBPF、kprobe 和 perf event,采集器需要 root 权限。

i386 的网络统计覆盖 IPv4/IPv6 TCP、UDP、RTT、重传、连接失败、进程、网络命名空间和 DNS 关联。UDP 默认关闭,在 [[inputs.ebpf]] 中设置 enable_udp = true,或设置 ENV_INPUT_EBPF_ENABLE_UDP=true。DNAT 关联还需要启用 ebpf-conntrack 并加载对应的内核模块。

启动时,来自当前内核 /sys/kernel/btf/vmlinux 且字段完整的 BTF 直接提供结构偏移,不额外依赖私有 socket 的探测事件。外置 BTF、旧偏移缓存或缺失字段仍通过实际连接验证;偏移缓存绑定完整内核版本和架构,内核更新后自动重新解析。基础采集所需偏移无法确认时,采集器报告启动错误,并同时写入采集器日志。NAT 与 HTTP 偏移复用同一次 BTF 解析;完整的当前内核 BTF 不再触发额外的偏移猜测。无 BTF 时,L7 序号使用已有的有界私有 socket 验证,不假设它与 RTT 字段的相对位置;已验证的诊断字段直接复用,不重复运行同一组测试。可选的 L7 序号无法确认时,只停用对应七层能力,已确认的 netflow 继续采集。私有测试 socket、抓包 ring 和临时探针在启动验证后释放。

目前 Linux 3.10 内核的项目生命周期已经结束,建议您升级至 Linux 4.9 及以上 LTS 版内核。

除 CentOS 7.6+ 和 Ubuntu 16.04 以外,其他发行版本推荐 Linux 内核版本高于 4.9,否则可能无法启动 eBPF 采集器。

若 Linux 内核版本低于 4.4 时可能无法开启 ebpf-trace 插件。

如果要启用 ebpf-conntrack ,请确认内核中的符号是否包含 nf_ct_delete 和 __nf_conntrack_hash_insert,可执行以下命令查看:

cat /proc/kallsyms | awk '{print $3}' | grep "^nf_ct_delete$\|^__nf_conntrack_hash_insert$"

或内核是否加载 nf_conntrack 模块:

lsmod | grep nf_conntrack

已启用 SELinux 的系统

对于启用了 SELinux 的系统,无法开启 eBPF 采集器,需要关闭其,执行以下命令进行关闭:

setenforce 0

采集器配置

进入 DataKit 安装目录下的 conf.d/samples 目录,复制 ebpf.conf.sample 并命名为 ebpf.conf。示例如下:

[[inputs.ebpf]]
  daemon = true
  name = 'ebpf'
  cmd = "/usr/local/datakit/externals/datakit-ebpf"
  args = [
    "--datakit-apiserver", "0.0.0.0:9529",
  ]
  envs = []

  ## Recover excessive historical candidates before registering new probes.
  ## Only DK-compatible definitions count; duplicate definitions are not required.
  ## Default total threshold: 200. Zero disables heuristic recovery.
  ## Requires upstream Linux 3.10.10+ or standard EL7 x86_64 3.10.0-123+,
  ## complete HOST_PROC and no other DK eBPF collector. See KERNEL_COMPATIBILITY.md.
  ## Enabled/perf-busy events are preserved. Small inventories stay conservative.
  legacy_probe_cleanup_threshold = 200

  ## Resource limits.
  ## The collector automatically exits when the limit is exceeded.
  ## Can configure the number of cpu cores, memory size and network bandwidth.
  ##
  # cpu_limit = "2.0"
  # mem_limit = "4GiB"
  # net_limit = "100MiB/s"

  ## automatically takes effect when running DataKit in 
  ## Kubernetes daemonset mode
  ##
  # kubernetes_url = "https://kubernetes.default:443"
  # bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
  ##
  ## or 
  # bearer_token_string = "<your-token-string>"

  ## k8s workload labels
  ##
  # workload_labels = ["app"]  
  # workload_label_prefix = ""

  ## operator url
  ##
  # operator_url = "https://datakit-operator.datakit.svc:443"

  ## all supported plugins:
  ## - "ebpf-net"  :
  ##     contains L4-network(netflow), L7-network(httpflow, dnsflow) collection
  ## - "ebpf-bash" :
  ##     log bash
  ## - "ebpf-conntrack":
  ##     add two tags "dst_nat_ip" and "dst_nat_port" to the network flow data
  ## - "ebpf-trace":
  ##     param trace_server must be set simultaneously.
  ## - "bpf-netlog":
  ##     contains L4-network log (bpf_net_l4_log), L7-network log (bpf_net_l7_log), 
  ##              L4-network(netflow), L7-network(httpflow, dnsflow) collection
  enabled_plugins = [
    "ebpf-net",
  ]

  ## Collect UDP flows in addition to TCP.
  enable_udp = false

  ## Network diagnostic metrics and bounded structured problem logs.
  ## These use their own budgets, independent of netflow sampling.
  network_diagnostics = true
  ## Default: 40 node metric timelines with stable global tags.
  ## Full metrics add collector fields and per-netns/interface timelines.
  network_diagnostics_full_metrics = false
  # network_diagnostics_max_groups = 4096
  ## Both limits apply; allow four minutes of burst tokens.
  # network_diagnostics_log_bytes_per_minute = 524288
  # network_diagnostics_log_points_per_minute = 240

  ## If you enable the ebpf-net plugin, you can configure:
  ##  - "httpflow" (* enabled by default)
  ##  - "httpflow-tls"
  ##
  l7net_enabled = [
    "httpflow",
    # "httpflow-tls"
  ]


  ## datakit-ebpf pprof service
  pprof_host = "127.0.0.1"
  pprof_port = "6061"

  ## netlog blacklist
  ##
  # netlog_blacklist = "ip_saddr=='127.0.0.1' || ip_daddr=='127.0.0.1'"

  ## bpf-netlog plugin collection metric and log
  ##
  netlog_metric = true
  netlog_log = false

  ## bpf-netlog L7 log protocols.
  ## "http" enables both HTTP/1.x and HTTP/2. Use "http1" only when HTTP/1.x is needed alone.
  ## Available values: "http", "http1", "http2", "grpc".
  ##
  # netlog_l7log_protocols = ["http"]

  ## bpf-netlog L7 log HTTP headers.
  ## Empty uses the built-in recommended allowlist, ["none"] disables header capture.
  ##
  # netlog_l7log_headers = [
  #   "host",
  #   "x-request-id",
  #   "traceparent",
  # ]

  ## bpf-netlog AF_PACKET capture ring limits.
  ## In Kubernetes, datakit-ebpf uses smaller defaults to reduce resident memory.
  ##
  # netlog_fallback_sockets = 0
  # netlog_fallback_blocks = 0
  # netlog_shared_blocks = 0

  ## Advanced datakit-ebpf capacity tuning can be added to envs above.
  ## Prefer changing these only when self-monitoring shows drops caused by limits.
  ##
  ##   "DK_EBPF_NETFLOW_MAP_MAX_ENTRIES=131072",
  ##   "DK_EBPF_L4LOG_CONN_CACHE_LIMIT=131072",
  ##   "DK_EBPF_L7FLOW_CONN_MAP_LIMIT=131072",

  ## eBPF trace generation server center address.
  trace_server = ""

  ## trace all processes containing any specified environment variable
  trace_env_list = [
    # "DK_BPFTRACE_SERVICE",
    # "DD_SERVICE",
    # "OTEL_SERVICE_NAME",
  ]

  ## deny tracking any process containing any specified environment variable
  trace_env_blacklist = []

  ## trace all processes containing any specified process names,
  ## can be used with trace_namedenyset
  ##
  trace_name_list = []

  ## deny tracking any process containing any specified process names
  ##
  trace_name_blacklist = [

    ## The following two processes are hard-coded to never be traced,
    ## and do not need to be set:
    ##
    # "datakit",
    # "datakit-ebpf",
  ]

  ## conv other trace id to datadog trace id (base 10, 64-bit) 
  conv_to_ddtrace = false

  ## If the system does not enable ipv6, it needs to be changed to true
  ##
  ipv6_disabled = false

  ## ephemeral port strart from <ephemeral_port>
  ##
  # ephemeral_port = 10001

  ## Dynamic network path discovery.
  ## datakit-ebpf only reports candidates; DataKit netpath runs traceroute
  ## and uploads netpath points.
  ##
  # network_path_enabled = false
  # network_path_api = "http://127.0.0.1:9529/v1/netpath/candidates"
  # network_path_token = ""
  # network_path_flush_interval = "10s"
  # network_path_batch_size = 100
  # network_path_http_timeout = "3s"
  # network_path_queue_size = 4096

  # interval = "60s"

  # sampling_rate = "0.50"
  # sampling_rate_pts_per_min = "1500"

  [inputs.ebpf.tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"

#############################
## Parameter description (if marked * is required)
#############################
##  --hostname               : Host name, this parameter can change the value of the host tag when the collector uploads data, the priority is: specify this parameter >
##                             ENV_HOSTNAME value in datakit.conf (if it is not empty, this parameter will be added automatically at startup) >
##                             collector Get it yourself (the default).
##  --datakit-apiserver      : DataKit API Server address, default value 0.0.0.0:9529 .
##  --log                    : Log output path, default <DataKitInstallDir>/externals/datakit-ebpf.log.
##  --log-level              : Log level, the default value is 'info'.
##  --service                : The default value is 'ebpf'.

配置好后,重启 DataKit 即可。

Kubernetes 中可以通过 ConfigMap 或者直接默认启用 eBPF 采集器两种方式来开启采集:

  1. ConfigMap 方式参照通用的安装示例。
  2. 在 datakit.yaml 中的环境变量 ENV_ENABLE_INPUTS 中追加 ebpf,此时使用默认配置,即仅开启 ebpf-net 网络数据采集
- name: ENV_ENABLE_INPUTS
       value: cpu,disk,diskio,mem,swap,system,hostobject,net,host_processes,container,ebpf

环境变量与配置项

通过以下环境变量可以调整 Kubernetes 中 eBPF 采集配置:

配置项:

  • enabled_plugins:

    • 描述:用于配置开启采集器的内置插件
    • 环境变量:ENV_INPUT_EBPF_ENABLED_PLUGINS
    • 示例:ebpf-net,ebpf-trace
  • l7net_enabled

    • 描述:开启 http 协议数据采集
    • 环境变量:ENV_INPUT_EBPF_L7NET_ENABLED
    • 示例:httpflow
  • interval

    • 描述:设置采样时间间隔
    • 环境变量:ENV_INPUT_EBPF_INTERVAL
    • 示例:1m30s

聚合周期由 interval 统一控制,默认及最小值为 60s,适用于 ebpf-net 的 netflow/httpflow/dnsflow 和 bpf-netlog 的 netflow/httpflow。DNS 请求匹配超时与聚合周期独立;窗口内的次数和总延迟直接累加,以请求次数计算平均延迟,避免对分组平均值再次求平均。

DNSFlow 在采集器所在的 network namespace 中使用一个抓包 socket,统计主机侧可见的明文 DNS 流量,不逐个进入 Pod namespace 抓包,也不按 Kubernetes namespace 筛选。不同业务 namespace 的 Pod 流量可能在主机侧被观测到;容器内部未经过可见接口的流量不保证覆盖。DNS 聚合键保持客户端/服务端顺序,本地观察侧在导出时确定,避免地址信息在窗口中途补齐造成同一组统计拆分。

DNS 域名关联优先使用同一作用域内的客户端 IP 与应答地址;主机侧抓到的应答也可按原始客户端 IP 关联到 Pod socket,抓包所在 netns 不等同于 socket 所在 netns。不会退回任意 namespace 或全局 IP 匹配;跨作用域关联不接受回环、未指定或 link-local 客户端地址。TCP 连接按创建或首次接管的观测时间关联 DNS 候选,并随该 socket 代次保留;TTL 从 DNS 抓包时间起算。首次未命中时仅随原有上报周期重试,允许补入早已发生但处理较晚的应答,不采用晚于连接观测时间的解析。已有候选不会因 TTL 过期而消失。缓存仅为历史时点查询多保留一个清理周期(1 分钟)的过期元数据,当前时点查询仍严格检查 TTL,不保存完整 DNS 历史。更强的应用层域名关联仍沿用原有逻辑。入站 server_domain 使用本地服务端证据;dst_domain 继续表示原始远端地址的域名,不能直接当成入站服务端域名。连接回收后释放候选,新连接重新查有效缓存。UDP 仍按当前观测关联。同一聚合桶内 DNS 候选不一致时不填 DNS 域名,不增加聚合维度或上报点数。此关联属于 DNS 推断,并不能证明应用使用了该域名;采集器启动前的解析、不可见或加密 DNS 无法据此补回。

DNSFlow 将正常查询与诊断明细分开聚合,沿用 dnsflow measurement 和原有字段类型:

DNS 客户端 namespace 或 Pod 为 <nil> 时,表示没有匹配到 Kubernetes 元数据,不能直接解释为 Node 流量。客户端为节点 IP 时,也可能是 hostNetwork 或经 SNAT 的 DNS 转发。排查时结合 client_ip、server_ip 和可用的 Pod 标签区分各段链路;同一次解析的客户端、转发和上游记录不能相加作为唯一业务查询数。

aggregation_scope 聚合口径 保留的信息
service 正常查询按客户端 namespace、workload 类型及名称、DNS 服务器 IP/端口、查询类型、返回码、传输协议及地址族汇总 次数、平均/最大延迟;客户端 IP、Pod 名和 query_domain 为 *,不使用某个副本作为整组的代表
endpoint 客户端缺少可信工作负载归属时,正常查询按客户端 IP 与 DNS 端点汇总 保留客户端 IP,query_domain=*;不同未知客户端不会混成一个服务
query 错误响应、响应时间达到 1 秒、或超过匹配超时仍未观测到响应 保留客户端 IP、可用的 Pod 标签及完整查询域名

上述三类计数互斥,异常请求不会再计入正常汇总。快速 NXDOMAIN 沿用原有规则,不直接视为故障,但 rcode=3 独立汇总,仍可观察名称不存在的次数;慢 NXDOMAIN 保留诊断明细。异常同时继续进入已启用的 ebpf_network_event 通道,受原有日志预算约束;关闭该通道不影响 query 类 DNSFlow。此调整不增加抓包、proc 扫描或 namespace 切换,服务归属只查现有内存中的 Kubernetes 元数据,并在聚合窗口内缓存,缓存上限沿用 DNS 聚合上限。反向域名缓存仍消费原始 DNS 应答。

  • ipv6_disabled

    • 描述:系统是否不支持 IPv6
    • 环境变量:ENV_INPUT_EBPF_IPV6_DISABLED
    • 示例:false
  • ephemeral_port

    • 描述:仅用于 UDP 方向的临时端口辅助推断。TCP 方向由连接或协议证据确定,不按端口大小猜测。netflow/httpflow 在角色明确时将客户端端口归并为 *,保留服务端端口;角色未知时两端端口归并为 *,保留地址对及收发总量,不生成客户端/服务端派生字段。连接明细和本地诊断保留原始端口。
    • 环境变量:ENV_INPUT_EBPF_EPHEMERAL_PORT
    • 示例:32768
  • pprof_host

    • 描述:pprof host
    • 环境变量:ENV_INPUT_EBPF_PPROF_HOST
    • 示例:127.0.0.1
  • pprof_port

    • 描述:pprof port
    • 环境变量:ENV_INPUT_EBPF_PPROF_PORT
    • 示例:6061
  • trace_server

    • 描述:开启 ebpftrace 采集器的 DataKit ELinker/ DataKit 的地址
    • 环境变量:ENV_INPUT_EBPF_TRACE_SERVER
    • 示例:<ip>:<port>
  • trace_name_blacklist

    • 描述:指定进程名的进程将被禁止采集链路数据
    • 环境变量:ENV_INPUT_EBPF_TRACE_NAME_BLACKLIST
    • 示例:
  • trace_env_blacklist

    • 描述:包含任意一个指定环境变量名的进程将被禁止采集链路数据
    • 环境变量:ENV_INPUT_EBPF_TRACE_ENV_BLACKLIST
    • 示例:DKE_DISABLE_ETRACE
  • trace_env_list

    • 描述:含有任意指定环境变量的进程的链路数据将被跟踪和上报
    • 环境变量:ENV_INPUT_EBPF_TRACE_ENV_LIST
    • 示例:DK_BPFTRACE_SERVICE,DD_SERVICE,OTEL_SERVICE_NAME
  • trace_name_list

    • 描述:进程名在指定集合内的的进程将被跟踪和上报
    • 环境变量:ENV_INPUT_EBPF_TRACE_NAME_LIST
    • 示例:chrome,firefox
  • conv_to_ddtrace

    • 描述:将所有的应用侧链路 id 转换为 10 进制表示的字符串,兼容用途,非必要不使用
    • 环境变量:ENV_INPUT_EBPF_CONV_TO_DDTRACE
    • 示例:false
  • netlog_blacklist

    • 描述:用于实现在抓包之后的数据包的过滤
    • 环境变量:ENV_INPUT_EBPF_NETLOG_BLACKLIST
    • 示例:ip_saddr=='127.0.0.1' \|\| ip_daddr=='127.0.0.1'
  • netlog_metric

    • 描述:从网络数据包分析采集网络指标
    • 环境变量:ENV_INPUT_EBPF_NETLOG_METRIC
    • 示例:true
  • netlog_log

    • 描述:从网络数据包分析采集网络日志
    • 环境变量:ENV_INPUT_EBPF_NETLOG_LOG
    • 示例:false
  • netlog_l7log_protocols

    • 描述:设置 bpf-netlog 七层日志解析协议。http 会同时开启 HTTP/1.x 和 HTTP/2;仅需 HTTP/1.x 时再使用 http1。可选值:http、http1、http2、grpc
    • 环境变量:ENV_INPUT_EBPF_NETLOG_L7LOG_PROTOCOLS
    • 示例:http,http2,grpc
  • netlog_l7log_headers

    • 描述:设置 bpf-netlog 七层日志记录的 HTTP Header。空值使用内置推荐列表;设置为 none 可关闭 Header 采集。敏感 Header 会被强制忽略
    • 环境变量:ENV_INPUT_EBPF_NETLOG_L7LOG_HEADERS
    • 示例:host,x-request-id,traceparent
  • netlog_fallback_sockets

    • 描述:设置 bpf-netlog fallback 网络命名空间抓包 socket 数量上限。Kubernetes 中默认关闭 fallback 抓包,可按需调大
    • 环境变量:ENV_INPUT_EBPF_NETLOG_FALLBACK_SOCKETS
    • 示例:8
  • netlog_fallback_blocks

    • 描述:设置 bpf-netlog fallback 抓包 AF_PACKET ring block 数量
    • 环境变量:ENV_INPUT_EBPF_NETLOG_FALLBACK_BLOCKS
    • 示例:4
  • netlog_shared_blocks

    • 描述:设置 bpf-netlog 共享 host-peer 抓包 AF_PACKET ring block 数量
    • 环境变量:ENV_INPUT_EBPF_NETLOG_SHARED_BLOCKS
    • 示例:64
  • cpu_limit

    • 描述:单位时间内 CPU 最大核心数使用限制,到达上限,采集器退出
    • 环境变量:ENV_INPUT_EBPF_CPU_LIMIT
    • 示例:"2.0"`
  • mem_limit

    • 描述:内存大小使用限制
    • 环境变量:ENV_INPUT_EBPF_MEM_LIMIT
    • 示例:"4GiB"
  • net_limit

    • 描述:网络带宽(任意网卡)限制
    • 环境变量:ENV_INPUT_EBPF_NET_LIMIT
    • 示例:"100MiB/s"
  • sampling_rate

    • 描述:设置 eBPF 采集器上报数据时的采样率,范围 0.01 ~ 1.00;与设置 samping_rate_pts_per_min 互斥
    • 环境变量:ENV_INPUT_EBPF_SAMPLING_RATE
    • 示例:0.50
  • sampling_rate_pts_per_min

    • 描述:设置 eBPF 采集器上报数据时的每分钟数据量阈值,动态调整采样率
    • 环境变量:ENV_INPUT_EBPF_SAMPLING_RATE_PTSPERMIN
    • 示例:1500
  • workload_labels

    • 描述:设置需要添加到数据的 K8s workload 所有指定的标签
    • 环境变量:ENV_INPUT_EBPF_WORKLOAD_LABELS
    • 示例:app,project_id
  • workload_label_prefix

    • 描述:为 k8s 的 workload 标签添加前缀
    • 环境变量:ENV_INPUT_EBPF_WORKLOAD_LABEL_PREFIX
    • 示例:k8s_workload_label_
  • operator_url

    • 描述:设置 operator 地址,用于采集 k8s 集群的 workload 标签
    • 环境变量:ENV_INPUT_EBPF_OPERATOR_URL
    • 示例:https://datakit-operator.datakit.svc:443

高负载采集上限调优

以下为 datakit-ebpf 进程内部的高级容量参数,不建议作为常规配置项调大。只有在自监控指标或摘要日志中看到 limit、queue_full、pending_query_limit、bpf_map_fill_ratio 接近 1 等压力信号时,再通过 inputs.ebpf.envs 或 DaemonSet 环境变量设置。

环境变量 默认值 最大值 作用范围
DK_EBPF_NETFLOW_MAP_MAX_ENTRIES 65536 1048576 ebpf-net 内核连接 map 容量
DK_EBPF_NETFLOW_CLOSED_CONN_CACHE_LIMIT 65536 1000000 ebpf-net 已关闭连接缓存
DK_EBPF_DNSFLOW_PENDING_QUERY_LIMIT 65536 1000000 DNS 未响应 query 缓存
DK_EBPF_DNSFLOW_AGG_LIMIT 32768 1000000 DNS 聚合 key 数
DK_EBPF_DNSFLOW_ANSWER_RECORD_LIMIT 65536 1000000 DNS 反查记录缓存
DK_EBPF_L4LOG_CONN_CACHE_LIMIT 65536 1000000 bpf-netlog L4 连接缓存
DK_EBPF_L4LOG_NETFLOW_AGG_LIMIT 32768 1000000 bpf-netlog L4 指标聚合 key 数
DK_EBPF_L4LOG_HTTP_AGG_LIMIT 32768 1000000 bpf-netlog HTTP 指标聚合 key 数
DK_EBPF_L4LOG_HTTP_ELEM_LIMIT 1024 65536 bpf-netlog 单连接尚未完成的 HTTP 请求状态数
DK_EBPF_L4LOG_HTTP2_STREAM_LIMIT 1024 65536 bpf-netlog 单连接尚未完成的 HTTP/2 stream 状态数
DK_EBPF_L7FLOW_CONN_MAP_LIMIT 65536 1000000 ebpf-trace/httpflow 用户态连接状态数
DK_EBPF_L7FLOW_HTTP_AGG_LIMIT 32768 1000000 httpflow HTTP 聚合 key 数
DK_EBPF_L7FLOW_HTTP2_STREAM_LIMIT 1024 65536 httpflow 单连接尚未完成的 HTTP/2 stream 状态数
DK_EBPF_L7FLOW_MYSQL_PENDING_BUFFER_LIMIT 65536 4194304 MySQL 协议半包缓冲字节数
DK_EBPF_CONNTRACK_TUPLE_MAP_MAX_ENTRIES 65535 1048576 连接跟踪 tuple map 容量
DK_EBPF_PROCWATCH_MAP_MAX_ENTRIES 12800 1048576 进程 watch 内核 map 容量

DK_EBPF_* 环境变量会从 DataKit 传入 datakit-ebpf;同名参数以 inputs.ebpf.envs 中的显式配置为准。连接 map 容量在采集器启动时生效,默认仍为 65536。配置扩容后应检查自监控中对应 map 的 capacity / max_entries 实际值;仅支持扩容或修改配置文件不表示运行中的 map 已扩容。

对于 bpf-netlog,上述 HTTP/HTTP2 上限约束尚未完成的事务。已完成事务达到一批时会主动唤醒消费,不必等待 8 秒维护周期;等待生成日志或聚合的数据由每个抓包实例内所有连接共享 16 MiB 的暂存预算,消费完成后归还。该预算按结构体、字符串和请求/响应头的保留量估算,不是整个进程 RSS 上限。输出持续阻塞并耗尽预算时,完成记录的淘汰会计入采集质量计数,原因是 completed_budget。

HTTP/2 支持协商后 HPACK 动态表大小的变化,并以编码流中的表大小更新驱动解码与字典淘汰,避免跨方向 syscall 完成顺序影响解码,参见 RFC 9113 和 RFC 7541。每个方向的动态表解码上限为 64 KiB;仅宣告更大的允许值不会分配同等大小的表,编码流要求超过该上限时会停止该连接的解码并记录采集质量计数。httpflow 在请求和响应两侧都结束后导出完整事务,取消或被 GOAWAY 拒绝的未完成流会释放状态,不生成成功事务。

HTTP/2 对先于请求事件到达的 RST_STREAM,在现有连接索引中保留终止标记,直到观测到对应请求;标记不占用正常未完成事务配额,其数量受同一个 stream 配置上限约束。终止标记超限时停止该连接解码,并记录 terminal_limit 采集质量计数。服务端 push 暂不导出事务,但仍解压相关头部以维护 HPACK 字典,避免已完成 push 占用正常请求容量。

DNS over TCP 的请求/响应匹配包含本地连接代际,避免重连复用端口和 DNS ID 后串配;该身份不增加上报字段、标签或聚合维度。SYN-ACK 校验包含已观测到的 SYN 数据范围,支持 Fast Open;RST 需要匹配当前方向的连续序号,握手拒绝则校验 SYN 确认范围,避免旧连接报文清理新连接状态。

FIN 也会先校验重组序号范围和已观测到的对端确认范围,再更新关闭状态;无法匹配的控制报文不会改变当前连接。空闲时只释放过期的未完成片段,干净的连接身份仍在原有 4096 条方向状态上限内保留,以支持空闲后继续拆包通信;容量不足时优先回收关闭或双向空闲的身份。片段过期导致的重组缺口会使相关待匹配查询失效,不作为业务 DNS 超时上报。

eBPF 链路功能

ebpf-trace 采集分析主机上的进程读写的网络数据,并对进程的内核级线程/用户级线程(如 golang goroutine)进行跟踪,生成链路 eBPF Span 该数据需要被 ebpftrace 采集进行进一步的加工处理。

使用时,需要在多个节点部署了该开启链路数据采集的 eBPF 采集器,则需要将所有 eBPF Span 数据发往同一个开启了 ebpftrace 采集器插件的 DataKit ELinker/DataKit。更多配置细节见 eBPF 链路文档

bpf-netlog 插件的黑名单功能

过滤器规则示例:

单条规则:

以下规则过滤 ip 为 1.1.1.1 且端口为 80 的网络数据。(运算符后允许换行)

(ip_saddr == "1.1.1.1" || ip_saddr == "1.1.1.1") &&
     (src_port == 80 || dst_port == 80)

多条规则:

规则间使用 ; 或 \n 分隔,满足任意一条规则就进行数据过滤

udp
ip_saddr == "1.1.1.1" && (src_port == 80 || dst_port == 80);
ip_saddr == "10.10.0.1" && (src_port == 80 || dst_port == 80)

ipnet_contains("127.0.0.0/8", ip_saddr); ipv6

可用于过滤的数据:

该过滤器用于对网络数据进行过滤,可比较的数据如下:

key 名 类型 描述
tcp bool 是否为 TCP 协议
udp bool 是否为 UDP 协议
ipv4 bool 是否为 IPv4 协议
ipv6 bool 是否为 IPv6 协议
src_port int 源端口(以被观测网卡/主机/容器为参考系)
dst_port int 目标端口
ip_saddr str 源 IPv4 网络地址
ip_saddr str 目标 IPv4 网络地址
ip6_saddr str 源 IPv6 网络地址
ip6_daddr str 目标 IPv6 网络地址
k8s_src_pod str 源 pod 名
k8s_dst_pod str 目标 pod 名

运算符:

运算符从高往低:

优先级 Op 名称 结合方向
1 () 圆括号 左
2 ! 逻辑非,一元运算符 右
3 != 不等于 左
3 >= 大于等于 左
3 > 大于 左
3 == 等于 左
3 <= 小于等于 左
3 < 小于 左
4 && 逻辑与 左
4 \|\| 逻辑或 左

函数:

  1. ipnet_contains

    函数签名: fn ipnet_contains(ipnet: str, ipaddr: str) bool

    描述: 判断地址是否在指定的网段内

    示例:

    ipnet_contains("127.0.0.0/8", ip_saddr)
    

    如果 ip_saddr 值为 "127.0.0.1",则该规则返回 true,该 TCP 连接数据包/ UDP 数据包将被过滤。

  2. has_prefix

    函数签名: fn has_prefix(s: str, prefix: str) bool

    描述: 指定字段是否包含某一前缀

    示例:

    has_prefix(k8s_src_pod, "datakit-") || has_prefix(k8s_dst_pod, "datakit-")
    

    如果 pod 名为 datakit-kfez321,该规则返回 true。

网络问题诊断

启用 ebpf-net 后,网络诊断默认开启。数据直接上报到 DataKit,无需访问主机上的 /debug/netflow:

TCP 优先将实际 NAT 报文验证的目标保存在对应 socket 指针和出生时间的连接快照中,随连接统计保留到用户态消费,避免 conntrack 删除或源端口复用抹掉尚未消费的归属。同一 TCP 世代出现冲突的目标时保持 N/A。

IPVS 的 MASQ/NAT 路径在启用 IPVS conntrack 支持、相关内核钩子可用时,通过实际 skb 和 conntrack reply 发布关联;存量连接在 IPVS 更新函数返回后读取已确认的映射。无 BTF 时,至少两个不同 conntrack owner 对同一 skb 指针字段达成唯一验证后才读取该字段;校准前或证据冲突时,缺少证明的存量连接保持 N/A。常规路径不遍历 IPVS 连接表,也不保存报文历史。

UDP 和缺少上述 socket 证明的连接,使用地址族、TCP/UDP 协议、来源网络命名空间和完整地址端口回退查询。同一 tuple 出现多个 owner 时,只暂停该 tuple 的回退关联;源版本和成员数稳定且归属唯一后可自动恢复,不需要重启采集器。UDP 不给整个 socket 固定一个后端。跨命名空间 DNAT 需要实际报文入口和经过双向核验的 veth 关系;无法确认对应关系时保持 N/A,不能保证覆盖所有 CNI 和流量卸载路径。map 容量拒绝会记录为采集缺口。

单独启用 httpflow-tls 会启动其依赖的七层采集。七层偏移或探针初始化失败时,已验证的基础网络采集继续运行;启动日志及开启的网络诊断会记录七层能力不可用。DNS 重组容量丢弃会使受影响的待响应请求失效,不会将已知漏采直接判为 DNS 无响应。

数据 用途 频率与范围
netflow(Network) 保留流量聚合,追加连接结果、实际观测的角色、I/O 错误、重传相关调用和 SRTT 快照统计 沿用原有聚合周期、标签和采样;已有字段的类型、单位和语义保持不变
ebpf_network(Metric) 查看节点异常趋势、来源新鲜度、采集能力和丢失情况 每分钟;默认固定 40 个节点级字段,完整指标模式追加命名空间及发生错误增量的网卡
ebpf_network_event(Logging) 保留有时间、进程、端点、阶段和证据范围的问题样本 首次满足日志条件后通常在 5 秒内进入发送队列;同组明确错误有新观测时每分钟更新,普通连接线索最多每 5 分钟更新一次
# 在 [[inputs.ebpf]] 内配置
network_diagnostics = true
network_diagnostics_full_metrics = false
network_diagnostics_max_groups = 4096
network_diagnostics_log_bytes_per_minute = 524288
network_diagnostics_log_points_per_minute = 240

对应环境变量为 ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS、ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_FULL_METRICS、ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_MAX_GROUPS、ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_LOG_BYTES_PER_MINUTE、ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_LOG_POINTS_PER_MINUTE。日志预算的有效范围为 4096~1048576 字节/分钟、1~4096 条/分钟,分组数量最多 4096。以上预算按节点生效。UDP 的启用配置仍由 enable_udp 控制;DNS 明文抓包和命名空间计数有各自的能力字段。

netflow 的 tcp_role_observed 是字符串 field,表示本指标窗口首次观测到的 TCP 角色类型:client、server、unknown;同一聚合点包含多种角色时为 mixed。没有新的角色观测时省略该字段。它不是连接数量。direction 和 conn_side 根据可用的连接或协议证据确定;无法确认的 TCP 角色保持 unknown,聚合不按端口大小猜测服务端。

对启动前已建立的 TCP 连接,ebpf-net 与 bpf-netlog 共用按网络命名空间刷新的监听索引。优先使用 connect/accept 或抓包握手证据,其次使用可验证的协议角色,再使用本地监听端点推断服务端;没有匹配监听不等于客户端。监听匹配核对地址族、地址与端口,失败刷新不覆盖完整快照,超过 30 秒未成功刷新即不再使用该推断。监听推断不会永久写成握手事实,后续直接角色证据可以纠正;ebpf-net 在当前窗口内迁移已计入的贡献;bpf-netlog 对存活连接在上报窗口结束时使用最新角色聚合,提前移除的连接在移除时结算,不增加每连接迁移账本。两路均保持流量总量不变,已上报窗口不回写。存量发现不补造历史连接建立次数、握手耗时或启动前流量。网络诊断中的 role_source=listener 表示监听推断;tcp_role_observed 仍只描述内核实际观测角色,因此可以与推断出的 direction 不同。监听刷新共享现有 namespace 清单与 INET_DIAG 查询,不遍历进程 FD;该逻辑不依赖 BTF。

bpf-netlog 在漏掉 TCP 握手时,可在汇总阶段将 NAT 后的报文端点关联到原始 socket,再使用实际观测到的连接角色或单独标识的监听证据确定服务端口。关联需要唯一的 NAT 映射、已确认的源命名空间和匹配的连接生命周期;NAT 的 original/reply 方向本身不代表应用的客户端/服务端角色。映射冲突、已清理或缺少原始角色证据时保持 unknown,包括可能在关闭时单独汇总的未知角色计数;收发总量仍保留,连接明细仍使用抓包看到的原始端点。

bpf-netlog 按已捕获的 TCP 唯一序号范围统计字节,不等待协议重组补齐缺口。新的初始序号可区分元组复用与 SYN 重传;换代时先结算旧代际,分别保留新旧连接的状态,不补造未观测到的关闭事件。去重仅保存有限的序号区间,不保留额外报文;超出历史范围时,通过本地采集质量计数 dkebpf_exporter_bpf_events_dropped_total{component="l4log",event="tcp_accounting",reason="range_history"} 标识不完整证据。

L7 已知的入队丢失标记绑定具体 socket 代际,内核仍持有该代际时不会仅因时间到期而清除,也不会按出生时间封禁其他老连接。标记受容量约束;确实无法保留时记录 event="admission_marker",reason="limit" 并使用已有丢失代次保护,避免静默错配。监听刷新使用固定四个 worker,独立调度到期 namespace,优先续期已确认的监听;namespace 清单刷新与监听读取分别执行。

为减少上报体积,netflow 的追加计数字段仅在增量非零时输出;成功建立耗时及 SRTT 快照字段仅在对应样本数非零时输出,有样本的合法零值仍保留。原有字段不采用此省略规则。查询应允许追加字段缺失,并结合采集能力与证据完整性判断,不能把缺失直接解释为没有网络故障。

默认指标与时间线数量

默认 network_diagnostics_full_metrics = false,每节点只输出一组稳定全局标签下的 41 个指标字段,不添加 netns 或 interface 标签。命名空间和网卡的来源计数、增量、重置与过期处理仍用于异常判断,异常日志继续包含相应归属与样本;netflow 字段和语义不变。

用途 默认指标字段
采集能力与完整性(9) diagnostics_enabled、connection_evidence_complete、tcp_socket_errno_available、tcp_queue_snapshots_available、tcp_retransmit_counter_available、listener_queue_snapshots_available、dns_capture_available、udp_collection_available、namespace_full_scan_available
来源新鲜度与覆盖(4) namespace_counter_sample_at_ns、namespace_candidates、namespace_samples、namespace_counter_limit_total
上报队列(4) diagnostic_queue_bytes、diagnostic_queue_oldest_age_seconds、netflow_pending_bytes、netflow_queue_oldest_age_seconds
预算与输出损失(7) group_admission_failures_total、samples_omitted_total、logs_limited_total、output_dropped_points_total、output_rejected_points_total、netflow_queue_full_points_total、netflow_rejected_points_total
采集异常(3) capture_loss_total、capture_unavailable_total、map_pressure_total
连接失败(3) connect_refused_total、connect_timeout_total、connect_unreachable_total
TCP 与监听压力(4) tcp_retransmit_total、tcp_rto_total、listener_overflow_total、tcp_receive_drop_total
UDP、IP 与网卡(5) udp_receive_error_total、udp_receive_buffer_total、ip_discard_total、icmp_unreachable_total、interface_error_total
DNS(2) dns_no_response_total、dns_error_response_total

connection_evidence_complete 描述节点侧连接证据的采集完整性,不表示平台已经入库。上报时,DataKit 接收失败会返回 HTTP 503,采集器保留已入队的数据重试;HTTP 200 表示 DataKit 接收成功,后续仍经过 Pipeline、过滤及发送链路。netflow_queue_full_points_total 统计入队失败而丢弃的点,这部分不会重试,需结合队列及输出损失的增量判断统计是否完整。

在单采集器实例、主机及全局标签稳定时,100、1000、10000 个节点分别约有 4100、41000、410000 条时间线,不随连接、进程、Pod 或命名空间数量增加;全局标签变化仍会生成新时间线。上报间隔影响数据点数量,不能消除不同标签组合产生的时间线。

需要逐来源连续指标时,设置 network_diagnostics_full_metrics = true,或独立运行时使用 --network-diagnostics-full-metrics / DKE_NETWORK_DIAGNOSTICS_FULL_METRICS=true。完整模式额外输出采集器字段与按 host + netns 区分的来源累计值、增量、来源时间和网卡计数,时间线数量会随命名空间数及重建增长。下面的完整指标定义包含可选字段,默认不会全部输出。

完整模式的网络栈与网卡指标使用小写下划线命名,明确协议和收发方向;*_total 表示内核来源累计值,*_delta 表示上一个指标窗口以来观测到的增量。首次采样或来源计数重置时,该次样本只建立新基线、不追加增量;同一指标窗口内此前已观测的增量仍保留。IPv4 使用 ipv4_*,IPv6 使用 ipv6_*、icmpv6_* 和 udpv6_*;icmp_*、udp_* 分别表示 ICMPv4 和 IPv4 UDP。计数可能重叠,不能相加作为唯一故障数或线路丢包率。

节点级 udp_no_port_total 表示本采集器会话中观测到的 udp_no_port 诊断增量累计值;完整模式下带 scope=namespace 的同名字段表示内核 Udp.NoPorts 累计值,可能包含采集器启动前的历史。两种 scope 是不同口径的证据视图,不应相加。

示例字段 含义
icmpv6_receive_destination_unreachable_delta 窗口内收到的 ICMPv6 目标不可达消息数
icmpv6_receive_packet_too_big_delta 窗口内收到的 ICMPv6 报文过大消息数,可辅助排查 MTU
tcp_listen_overflows_delta 窗口内内核监听队列溢出计数
udp_receive_buffer_errors_delta 窗口内 IPv4 UDP 接收缓冲区错误计数
interface_receive_drops_delta 窗口内该网卡的接收丢弃计数

TCP 的 retransmits、tcp_retransmit_total 及 tcp_retransmit 日志计数采用 socket 内核 total_retrans 的增量口径,包含 TLP;不再累加重传函数入口请求的段数。它们不是唯一丢包数、失败请求数或端到端丢包率;内核在完成重传记账后仍可能遇到本地发送错误。采集器首次通过 I/O 接管的既有连接以首次观测值为基线,避免把启动前的重传归到当前窗口。

tcp_retransmit_counter_available=false 表示无法获得经过验证的计数偏移,此时零值不能证明没有重传。无 BTF 时使用有界的私有 socket 自检验证该字段;自检只在启动偏移解析阶段运行。关闭网络诊断功能不影响 netflow 原有重传计数;此时无 BTF 的回退仅验证重传字段,不创建抓包 ring 或运行其他诊断自检。日志的 capabilities.tcp_retransmit_events 表示成功重传 tracepoint 是否可用;可用时在重传事件刷新队列快照,否则在后续重传尝试、I/O 或连接结束时补齐累计计数。查看样本时同时比较 at_ns 与 snapshot_at_ns,旧快照不能作为已恢复的证据。

诊断日志的每个连接样本还包含 retransmits_since_observed、tcp_retransmit_counter_available、socket_errno 和 socket_error_observed_at_ns。重传数是该 socket 自采集器接管以来的累计增量,不是该日志聚合窗口内的次数;可用性为 false 时零值不代表没有重传。socket 错误及其观测时间独立保留,避免在后续重传线索中与应用 I/O 返回码混淆。已关闭连接不再提供当前队列值,但仍保留已观测的重传、RTO 和 socket 错误。普通线索日志按服务及问题类型聚合,每组最多保留 3 个连接样本;严重异常的逐连接保留方式见下文。

ack_progress_observed_at_ns 是无进展判断的观测基准:ACK 推进、未确认队列清空或首次观测到新的未确认数据时更新。它不是首个丢包的精确时间;连接此前的空闲时间不计入发送停滞。

ACK 停滞只覆盖本端仍有未确认发送数据的情况。请求已经被对端 TCP 确认、但响应在返回路径丢失时,客户端可以同时表现为重传为 0、未确认量为 0、没有 RST 或 socket 错误。因此,这些值为 0 不能排除网络故障;connection_evidence_complete 也不表示已覆盖所有故障类型。httpflow 的 HTTP 事务不等于 MySQL 请求等待观测。排查此类数据库连接需要结合客户端请求/读超时、服务端日志,或对端的 TCP 重传证据。TCP_USER_TIMEOUT 约束未确认/未发送数据的传输等待,不能替代应用的响应读取超时。

已建立的 TCP 连接出现有效快照下的 ACK 停滞(仍有未确认量且至少 10 秒无进展)或观测到 socket/I/O 硬错误时,额外记录 problem_type=tcp_connection_issue。日志仍使用 ebpf_network_event/v1,以字段 connection_id 关联;连接 ID 不作为指标标签。每组只对应一条连接,samples 最多保留首次异常、最新关键变化、结束三个快照,分别用 phase=started/updated/closed 和 reason 说明。首次和结束在现有发送循环内尽快排队,中间变化最多每 5 分钟上报一次;同一 ACK 停滞不再重复输出服务级样本日志,其统计计数仍保留。occurrences 在此类日志中表示证据更新次数,重传数应读取样本的 retransmits_since_observed。

严重连接异常在首次观测时,还会从已有 PID→Pod 缓存提取 k8s_pod_name、k8s_namespace 和 k8s_workload_name,作为日志字段和 message.entity 内容保留到结束。缓存无法对应到该 PID 时留空;不使用后来复用该 IP 的 Pod 补写旧连接,也不为关联元数据增加文件或 namespace 扫描。这些字段不作为指标标签,不改变正常 netflow 的聚合维度。

event_status=closed 表示已观测到 TCP CLOSE 或 socket 最终回收,不一定已到应用关闭调用。first_seen_ns 是首次异常被观测的时间,不是首个丢包的精确时间;连接年龄也不是异常持续时间。只捕获到结束、采集覆盖出现缺口时标记 evidence_partial=true;单个超时错误不能区分重传耗尽、TCP_USER_TIMEOUT 或 TCP 保活 超时。沿用原有分组、内存、队列和日志预算,结束日志入队后释放组状态,非活动组沿用 30 分钟保留期。预算不足、上报拒绝或采集丢失时不保证每条异常都完整保留,应同时检查已有丢弃和完整性指标。正常连接和单次重传不启用该记录。

判断规则与证据范围

观测 输出与判断 排查方向
connect 的最终错误、socket 错误、I/O 硬错误 按拒绝、超时、不可达、资源不足、权限、消息过大或 reset 分类;EINPROGRESS、EALREADY、EAGAIN、中断及内核重启返回码不算硬错误 对端服务是否监听、路由、防火墙、端口或内存资源;UDP EMSGSIZE 同时检查消息长度和路径 MTU
连接超过 10 秒仍未建立,或成功建立耗时至少 3 秒 connect_pending / connect_slow,信息级线索;应用先关闭的 connect_canceled 及最终结果不可用的 connect_unknown 仅保留计数和 netflow 结果 区分握手等待、应用取消和已确认的连接失败
TCP 重传、RTO 定时器、零窗口探测、主动/收到 reset 分别计数,满足下述收敛条件后保留日志样本,信息级线索 结合连接阶段、窗口、队列、SRTT 及两端数据判断;调用次数不能当作线路丢包数
有效且不超过 10 秒的队列快照 已建立或 CLOSE_WAIT 连接仍有未确认序列空间,至少 10 秒未观测到 ACK 推进时记录 tcp_ack_stall;未读数据至少 64 KiB 时计数为 tcp_receive_backlog,满足持续观测条件后记录日志 对端或路径响应、接收窗口、本地应用消费速度;快照可能包含 SYN/FIN 的序列空间
SRTT 至少 200 ms;CLOSE_WAIT 至少 60 秒 tcp_srtt_elevated / tcp_close_wait,每连接每分钟最多一次 RTT 是内核采样值;半关闭合法,不能直接判为网络故障
监听队列压力、网络栈丢弃、UDP 缓冲区错误、ICMP、网卡错误 监听快照归属监听端点;其余按命名空间或网卡输出增量 找到本地服务或网络栈压力。ListenDrops 与 ListenOverflows 等可能重叠,不能相加作为独立丢包量
DNS 错误响应、响应至少 1 秒、6 秒未匹配到响应 dns_error_response / dns_slow / dns_no_response,信息级线索;NXDOMAIN 本身不算故障 检查 DNS 服务、传输与解析结果。抓包丢失会清除待匹配项并记录采集损失,不制造 DNS 超时
map 占用至少 90%、map 写入失败、probe/perf 丢失、发送积压 map_pressure、capture_loss、output_backlog,另有累计指标 先判断采集证据是否完整;map 压力和唤醒通知丢失本身不等于流量数据已丢失

普通连接线索在生成日志前会进一步收敛,原始累计计数和 netflow 数据继续保留:

日志线索 出日志的条件
tcp_retransmit、tcp_rto、tcp_zero_window_probe 同组从首个新观测开始的 60 秒计数窗口内至少 5 个计数单位(重传为内核段数增量,RTO/零窗口探测为调用次数);或当前有效快照显示握手等待至少 10 秒且 RTO 至少 3 次,或已建立/CLOSE_WAIT 连接仍有未确认量且至少 10 秒没有 ACK 推进
tcp_active_reset、tcp_received_reset 同组 60 秒计数窗口内至少 10 次;次数门槛只是保留排障样本,不等于确认故障
listener_queue_full 同组 60 秒计数窗口内至少 3 次队列压力观测;不等于确认 SYN 丢弃
tcp_receive_backlog、tcp_srtt_elevated 同一代表连接的有效快照反复达到阈值,观测跨度至少 30 秒,相邻观测间隔不超过 90 秒;单次接收突发或 RTT 抬升仅计数
connect_pending、connect_slow、tcp_ack_stall、tcp_close_wait 达到上述持续握手、建立耗时、ACK 停滞或半关闭规则即保留日志
connect_canceled、connect_unknown 单独的应用取消或未知结果只保留计数和 netflow,不生成问题日志

普通连接线索的首次合格观测通常在 5 秒内进入发送队列,同组后续合格观测最多每 5 分钟汇总一次。明确连接/I/O 错误、网络栈丢弃、DNS 和采集质量事件沿用每分钟更新。收敛不会把 info 自动升级成告警,也不会改变指标字段、netflow 字段或原始观测次数。occurrences 是自上次上报以来的观测增量,可包含此前未达到日志门槛的观测;它不是唯一连接数。输出预算耗尽时,已经合格的证据继续等待发送,未合格线索不会在退出时强制转为日志。

这些阈值是保守的初始排障规则,信息级线索不自动升级为故障告警。连接和 I/O 返回码遵循 connect(2);UDP 异步错误可能来自之前的数据报,因此日志仅归属 socket/进程并省略无法确认的远端,参见 udp(7)。计数器重叠及 offload 的影响参见 Linux SNMP counters。零窗口、半关闭和序列空间参见 TCP RFC 9293;DNS 返回码参见 RFC 1035。

数据量、保留与可靠性

异常按问题类型及进程/服务/命名空间/服务端点归组,每组最多保留 3 个代表样本,不采集业务报文载荷;明文 DNS 诊断可能包含查询名。日志同时受编码字节与条数预算限制:默认持续 512 KiB/分钟、240 条/分钟,任一额度不足都会暂缓;允许最多 2 MiB、960 条的短时突发。单条编码体积最多 16 KiB,超过上限计入 output_rejected_points_total,同一组相同累计观测不会重复编码拒绝,新观测可再次尝试。首次连接和问题事实仍按原有计数保留。每节点持续运行的采集器全天入队的新日志,持续额度分别最多约 720 MiB、345600 条,另加初始突发、重试和传输开销;两项上限不保证同时打满。这个预算不包含原有 netflow、新增 netflow 字段和诊断指标,无法仅据此估算全部网络采集量。

问题分组记账预算 16 MiB,发送队列预算 8 MiB / 2048 批;来源计数表最多 16384 个键。分组和队列按实际观测及待发送数据增长,不在启动时预分配全部预算。内核 TCP 故障数据与现有连接按同一生命周期保存,每条增加 144 字节的值及 map 键/管理开销。这些预算不等于进程总 RSS 上限。指标不使用 PID、连接 ID 或远端地址作为标签;默认不输出命名空间或网卡指标点,完整模式下无错误增量的网卡也不单独上报点。

监听端点在 30 分钟无新 SYN 观测后释放本地槽位,清理次数见 listener_idle_evictions_total;监听快照是采样线索,清理与新 SYN 并发时可能遗漏边界样本。无新观测的问题组和过期来源计数在 30 分钟后清理,重启后不回放,也不要求持久化目录。平台日志的保留时长由平台存储策略决定。安静不表示恢复:仅已确认的 connect 硬错误,在同一分组连续两个完整的安静窗口都有成功连接时输出 recovered;其他线索只报告观测及持续更新。

超限、队列满、采样省略、永久拒收和退出时未送达均有显式计数。仅新日志成功进入发送队列后扣减字节与条数额度;队列拒绝不消耗额度,指标不使用日志额度。暂时发送失败保留同一编码批次及 event_id 重试,不重复扣减额度;平台可用 collector_session + event_id 去重,但不保证端到端 exactly-once。发送阻塞时首条日志会延后。

使用顺序与能力边界

场景化排障、查询示例与实测数据见 eBPF 网络排障最佳实践。其中分别介绍连接与监听、TCP 传输、DNS/UDP 和采集可信度的判断及验证方法。

并发写入导致快照暂缓读取时,flow_snapshot_deferred_total 增加且该窗口的连接证据标为不完整,累计流量等待下一次稳定读取;这不表示已丢失。

  1. 先检查 ebpf_network 的 connection_evidence_complete、*_available、namespace_counter_sample_at_ns、覆盖及采集损失计数;完整模式还提供逐来源 sample_at_ns / sample_age_seconds。缺失能力或过期数据不能解释为“没有问题”。
  2. 用 netflow 的流量、连接结果及实际观测角色缩小进程/服务/对端范围。启动前已存在的连接可能缺少角色证据,此时 direction 和 conn_side 为 unknown,聚合保留地址对流量总量,原始端口可从连接明细或本地诊断查询。
  3. 按时间、problem_type、scope 检索 ebpf_network_event,读取 message 中的 ebpf_network_event/v1、entity、samples、evidence_partial 和能力字段。PID、端点属于字段,监听端点及命名空间事件不虚构进程归属。
  4. 结合两端和网络栈证据定位到连接阶段、服务队列、本地消费、DNS 或采集链路;单端数据通常不能确定具体哪一跳丢包。

socket errno、TCP 队列、发送窗口和监听队列优先采用验证过的内核 BTF 字段;缺少 BTF 或字段时,启动时通过本地 TCP socket 自检验证偏移。自检的 socket 等待与采样最多持续 8 秒,临时探针和抓包 socket 随后释放,不增加运行期采样频率。验证失败、有歧义、缺少所需内核函数或 loopback 抓包权限时,对应可选能力显式为 false,流量、连接结果、I/O 错误及网络栈计数继续采集。终止状态的队列快照标为不可用,socket 错误和 reset 证据继续保留。

命名空间来源仅覆盖已观测连接的候选命名空间及采集器自身,每 10 秒最多轮转 128 个命名空间、每个最多 32 个网卡;覆盖数量单独上报。加密 DNS、抓包不可达的命名空间和两次扫描间的短暂队列压力不能保证被捕获。

网络诊断指标

ebpf_network

Tags & Fields Description
host
(tag)
System hostname
interface
(tag)
Device name within netns, only present on full-mode interface points
netns
(tag)
Observed namespace in full metric mode; absent on node points
scope
(tag)
Namespace or interface scope in full metric mode; absent on node points
capture_loss_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
capture_unavailable_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_canceled_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_failed_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_pending_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_permission_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_refused_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_resource_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_slow_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_success_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_timeout_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_unknown_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connect_unreachable_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
connection_evidence_complete Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: N/A
diagnostic_admission_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
diagnostic_log_bytes_per_minute Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: N/A
diagnostic_log_points_per_minute Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: count
diagnostic_max_groups Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: count
diagnostic_queue_bytes Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: digital,B
diagnostic_queue_oldest_age_seconds Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: time,s
diagnostic_state_bytes Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: digital,B
diagnostics_enabled Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
dns_answer_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
dns_capture_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
dns_capture_loss_epoch_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
dns_error_response_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
dns_no_response_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
dns_query_unknown_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
dns_slow_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
early_connect_failures_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
flow_admission_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
flow_snapshot_deferred_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
flow_snapshot_scan_errors_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
group_admission_failures_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
icmp_packet_too_big_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
icmp_receive_destination_unreachable_delta ICMPv4 destination-unreachable messages received (Linux source: Icmp.InDestUnreachs). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
icmp_receive_destination_unreachable_total ICMPv4 destination-unreachable messages received (Linux source: Icmp.InDestUnreachs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
icmp_receive_errors_delta ICMPv4 receive errors (Linux source: Icmp.InErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
icmp_receive_errors_total ICMPv4 receive errors (Linux source: Icmp.InErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
icmp_receive_time_exceeded_delta ICMPv4 time-exceeded messages received (Linux source: Icmp.InTimeExcds). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
icmp_receive_time_exceeded_total ICMPv4 time-exceeded messages received (Linux source: Icmp.InTimeExcds). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
icmp_send_destination_unreachable_delta ICMPv4 destination-unreachable messages sent (Linux source: Icmp.OutDestUnreachs). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
icmp_send_destination_unreachable_total ICMPv4 destination-unreachable messages sent (Linux source: Icmp.OutDestUnreachs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
icmp_time_exceeded_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
icmp_unreachable_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
icmpv6_receive_destination_unreachable_delta ICMPv6 destination-unreachable messages received (Linux source: Icmp6.InDestUnreachs). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
icmpv6_receive_destination_unreachable_total ICMPv6 destination-unreachable messages received (Linux source: Icmp6.InDestUnreachs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
icmpv6_receive_packet_too_big_delta ICMPv6 packet-too-big messages received (Linux source: Icmp6.InPktTooBigs). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
icmpv6_receive_packet_too_big_total ICMPv6 packet-too-big messages received (Linux source: Icmp6.InPktTooBigs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
icmpv6_receive_time_exceeded_delta ICMPv6 time-exceeded messages received (Linux source: Icmp6.InTimeExcds). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
icmpv6_receive_time_exceeded_total ICMPv6 time-exceeded messages received (Linux source: Icmp6.InTimeExcds). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
icmpv6_send_destination_unreachable_delta ICMPv6 destination-unreachable messages sent (Linux source: Icmp6.OutDestUnreachs). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
icmpv6_send_destination_unreachable_total ICMPv6 destination-unreachable messages sent (Linux source: Icmp6.OutDestUnreachs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
icmpv6_send_packet_too_big_delta ICMPv6 packet-too-big messages sent (Linux source: Icmp6.OutPktTooBigs). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
icmpv6_send_packet_too_big_total ICMPv6 packet-too-big messages sent (Linux source: Icmp6.OutPktTooBigs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
interface_counters_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
interface_error_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
interface_full_scan_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
interface_namespaces_sampled Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: count
interface_receive_drops_delta Network interface receive drops (Linux source: rx_dropped). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
interface_receive_drops_total Network interface receive drops (Linux source: rx_dropped). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
interface_receive_errors_delta Network interface receive errors (Linux source: rx_errors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
interface_receive_errors_total Network interface receive errors (Linux source: rx_errors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
interface_send_drops_delta Network interface transmit drops (Linux source: tx_dropped). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
interface_send_drops_total Network interface transmit drops (Linux source: tx_dropped). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
interface_send_errors_delta Network interface transmit errors (Linux source: tx_errors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
interface_send_errors_total Network interface transmit errors (Linux source: tx_errors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
interfaces_expected Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: count
interfaces_sampled Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: count
io_correlation_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
io_message_size_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
io_other_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
io_permission_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
io_refused_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
io_reset_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
io_resource_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
io_timeout_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
io_unreachable_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
ip_discard_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
ipv4_receive_discards_delta IPv4 packets discarded in the receive path (Linux source: Ip.InDiscards). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
ipv4_receive_discards_total IPv4 packets discarded in the receive path (Linux source: Ip.InDiscards). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
ipv4_send_discards_delta IPv4 packets discarded in the send path (Linux source: Ip.OutDiscards). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
ipv4_send_discards_total IPv4 packets discarded in the send path (Linux source: Ip.OutDiscards). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
ipv6_receive_discards_delta IPv6 packets discarded in the receive path (Linux source: Ip6.InDiscards). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
ipv6_receive_discards_total IPv6 packets discarded in the receive path (Linux source: Ip6.InDiscards). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
ipv6_send_discards_delta IPv6 packets discarded in the send path (Linux source: Ip6.OutDiscards). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
ipv6_send_discards_total IPv6 packets discarded in the send path (Linux source: Ip6.OutDiscards). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
kprobe_profile_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
lifecycle_lost_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
lifecycle_read_errors_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
listener_child_failures_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
listener_drop_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
listener_idle_evictions_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
listener_overflow_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
listener_queue_full_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
listener_queue_snapshots_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
listener_syn_requests_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
logs_limited_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
map_pressure_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
namespace_candidates Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: count
namespace_counter_limit_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
namespace_counter_sample_at_ns Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: timeStamp,nsec
namespace_full_scan_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
namespace_samples Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: count
netflow_pending_batches Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: count
netflow_pending_bytes Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: digital,B
netflow_queue_full_points_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
netflow_queue_oldest_age_seconds Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: time,s
netflow_rejected_points_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
netflow_shutdown_frames_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
netflow_shutdown_points_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
notification_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
output_backlog_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
output_dropped_batches_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
output_dropped_points_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
output_rejected_points_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
problem_groups Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: count
sample_age_seconds Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: time,s
sample_at_ns Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds.
Type: int | (gauge)
Unit: timeStamp,nsec
samples_omitted_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
socket_admission_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_abort_close_delta TCP connections reset on close with unread data (Linux source: TcpExt.TCPAbortOnClose). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_abort_close_total TCP connections reset on close with unread data (Linux source: TcpExt.TCPAbortOnClose). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_abort_data_delta TCP connections reset by an immediate abortive close (Linux source: TcpExt.TCPAbortOnData). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_abort_data_total TCP connections reset by an immediate abortive close (Linux source: TcpExt.TCPAbortOnData). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_abort_memory_delta TCP connections aborted under memory pressure (Linux source: TcpExt.TCPAbortOnMemory). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_abort_memory_total TCP connections aborted under memory pressure (Linux source: TcpExt.TCPAbortOnMemory). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_abort_timeout_delta TCP connections aborted on timeout (Linux source: TcpExt.TCPAbortOnTimeout). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_abort_timeout_total TCP connections aborted on timeout (Linux source: TcpExt.TCPAbortOnTimeout). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_ack_stall_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_active_reset_calls_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_active_reset_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_backlog_drops_delta Packets dropped from the TCP socket backlog (Linux source: TcpExt.TCPBacklogDrop). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_backlog_drops_total Packets dropped from the TCP socket backlog (Linux source: TcpExt.TCPBacklogDrop). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_close_wait_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_lifecycle_cpu_coverage_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_lifecycle_lost_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
tcp_lifecycle_read_errors_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
tcp_listen_drops_delta Packets dropped by listening TCP sockets (Linux source: TcpExt.ListenDrops). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_listen_drops_total Packets dropped by listening TCP sockets (Linux source: TcpExt.ListenDrops). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_listen_overflows_delta TCP listen queue overflows (Linux source: TcpExt.ListenOverflows). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_listen_overflows_total TCP listen queue overflows (Linux source: TcpExt.ListenOverflows). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_memory_abort_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_original_data_segments_sent_delta TCP segments sent with original data, excluding retransmissions (Linux source: TcpExt.TCPOrigDataSent). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_original_data_segments_sent_total TCP segments sent with original data, excluding retransmissions (Linux source: TcpExt.TCPOrigDataSent). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_probe0_calls_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_queue_snapshots_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_receive_backlog_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_receive_drop_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_receive_errors_delta TCP receive errors, including malformed headers or checksum errors (Linux source: Tcp.InErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_receive_errors_total TCP receive errors, including malformed headers or checksum errors (Linux source: Tcp.InErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_receive_queue_drops_delta Packets dropped from the TCP receive queue (Linux source: TcpExt.TCPRcvQDrop). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_receive_queue_drops_total Packets dropped from the TCP receive queue (Linux source: TcpExt.TCPRcvQDrop). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_receive_segments_delta TCP segments received; offload can affect segment counts (Linux source: Tcp.InSegs). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_receive_segments_total TCP segments received; offload can affect segment counts (Linux source: Tcp.InSegs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_received_reset_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_received_resets_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_retransmit_counter_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_retransmit_events_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_retransmit_failures_delta Failed TCP retransmission attempts (Linux source: TcpExt.TCPRetransFail). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_retransmit_failures_total Failed TCP retransmission attempts (Linux source: TcpExt.TCPRetransFail). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_retransmit_segments_delta TCP segments retransmitted (Linux source: Tcp.RetransSegs). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_retransmit_segments_total TCP segments retransmitted (Linux source: Tcp.RetransSegs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_retransmit_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_rto_timer_fires_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_rto_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_send_segments_delta TCP segments sent, excluding retransmitted segments (Linux source: Tcp.OutSegs). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_send_segments_total TCP segments sent, excluding retransmitted segments (Linux source: Tcp.OutSegs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_socket_destroy_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_socket_errno_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_socket_error_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_spurious_retransmits_host_queue_delta Spurious TCP retransmissions caused by data retained in host queues (Linux source: TcpExt.TCPSpuriousRtxHostQueues). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_spurious_retransmits_host_queue_total Spurious TCP retransmissions caused by data retained in host queues (Linux source: TcpExt.TCPSpuriousRtxHostQueues). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_srtt_elevated_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
tcp_state_changes_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
tcp_syn_cookies_failed_delta Invalid TCP SYN cookies received (Linux source: TcpExt.SyncookiesFailed). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_syn_cookies_failed_total Invalid TCP SYN cookies received (Linux source: TcpExt.SyncookiesFailed). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_syn_cookies_sent_delta TCP SYN cookies sent (Linux source: TcpExt.SyncookiesSent). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_syn_cookies_sent_total TCP SYN cookies sent (Linux source: TcpExt.SyncookiesSent). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_syn_queue_full_cookies_delta TCP SYN queue full events handled with SYN cookies (Linux source: TcpExt.TCPReqQFullDoCookies). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_syn_queue_full_cookies_total TCP SYN queue full events handled with SYN cookies (Linux source: TcpExt.TCPReqQFullDoCookies). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_syn_queue_full_drops_delta TCP SYN queue full events that dropped a request (Linux source: TcpExt.TCPReqQFullDrop). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_syn_queue_full_drops_total TCP SYN queue full events that dropped a request (Linux source: TcpExt.TCPReqQFullDrop). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_syn_retransmits_delta TCP SYN or SYN-ACK retransmissions (Linux source: TcpExt.TCPSynRetrans). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
tcp_syn_retransmits_total TCP SYN or SYN-ACK retransmissions (Linux source: TcpExt.TCPSynRetrans). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
tcp_zero_window_probe_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp6_counters_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
udp_collection_available Availability of this evidence channel; missing or false does not mean no network faults.
Type: bool | (gauge)
Unit: bool
udp_errors_message_size_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_errors_other_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_errors_permission_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_errors_refused_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_errors_reset_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_errors_resource_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_errors_timeout_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_errors_unreachable_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_errqueue_reads_total Cumulative observed collector count since this collector session began.
Type: int | (count)
Unit: count
udp_lite_no_port_delta UDP-Lite datagrams received for a port without a listener (Linux source: UdpLite.NoPorts). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udp_lite_no_port_total UDP-Lite datagrams received for a port without a listener (Linux source: UdpLite.NoPorts). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
udp_lite_receive_buffer_errors_delta UDP-Lite receive buffer errors (Linux source: UdpLite.RcvbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udp_lite_receive_buffer_errors_total UDP-Lite receive buffer errors (Linux source: UdpLite.RcvbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
udp_lite_receive_errors_delta UDP-Lite receive errors (Linux source: UdpLite.InErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udp_lite_receive_errors_total UDP-Lite receive errors (Linux source: UdpLite.InErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
udp_lite_send_buffer_errors_delta UDP-Lite send buffer errors (Linux source: UdpLite.SndbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udp_lite_send_buffer_errors_total UDP-Lite send buffer errors (Linux source: UdpLite.SndbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
udp_no_port_delta IPv4 UDP datagrams received for a port without a listener (Linux source: Udp.NoPorts). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udp_no_port_total Node scope: cumulative observed udp_no_port diagnostic increments during this collector session. Namespace scope (full metrics): Linux Udp.NoPorts cumulative count, including pre-attachment history. These scopes are different views of evidence, not independent faults; do not sum them.
Type: int | (count)
Unit: count
udp_receive_buffer_errors_delta IPv4 UDP receive buffer errors (Linux source: Udp.RcvbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udp_receive_buffer_errors_total IPv4 UDP receive buffer errors (Linux source: Udp.RcvbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
udp_receive_buffer_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_receive_error_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_receive_errors_delta IPv4 UDP receive errors; overlaps with buffer errors (Linux source: Udp.InErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udp_receive_errors_total IPv4 UDP receive errors; overlaps with buffer errors (Linux source: Udp.InErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
udp_send_buffer_errors_delta IPv4 UDP send buffer errors (Linux source: Udp.SndbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udp_send_buffer_errors_total IPv4 UDP send buffer errors (Linux source: Udp.SndbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
udp_send_buffer_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_tuple_total Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections.
Type: int | (count)
Unit: count
udp_unbound_io_errors_total UDP negative I/O returns without a complete tuple, including would-block/interrupted returns; not a hard-error or datagram-loss count.
Type: int | (count)
Unit: count
udpv6_no_port_delta IPv6 UDP datagrams received for a port without a listener (Linux source: Udp6.NoPorts). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udpv6_no_port_total IPv6 UDP datagrams received for a port without a listener (Linux source: Udp6.NoPorts). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
udpv6_receive_buffer_errors_delta IPv6 UDP receive buffer errors (Linux source: Udp6.RcvbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udpv6_receive_buffer_errors_total IPv6 UDP receive buffer errors (Linux source: Udp6.RcvbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
udpv6_receive_errors_delta IPv6 UDP receive errors; overlaps with buffer errors (Linux source: Udp6.InErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udpv6_receive_errors_total IPv6 UDP receive errors; overlaps with buffer errors (Linux source: Udp6.InErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count
udpv6_send_buffer_errors_delta IPv6 UDP send buffer errors (Linux source: Udp6.SndbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline.
Type: int | (gauge)
Unit: count
udpv6_send_buffer_errors_total IPv6 UDP send buffer errors (Linux source: Udp6.SndbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds.
Type: int | (count)
Unit: count

网络聚合数据

以下所有数据采集,默认会追加名为 host 的全局 tag(tag 值为 DataKit 所在主机名),也可以在配置中通过 [inputs.ebpf.tags] 指定其它标签:

 [inputs.ebpf.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"
  # ...

netflow

Tags & Fields Description
client_ip
(tag)
Client IP address
client_ip_type
(tag)
Client IP type. (other/private/multicast)
client_k8s_deployment_name
(tag)
Client K8s deployment name
client_k8s_namespace
(tag)
Client K8s namespace
client_k8s_pod_name
(tag)
Client K8s pod name
client_k8s_service_name
(tag)
Client K8s service name
client_port
(tag)
Client port, rolled up to * in aggregates; omitted when the connection role is unknown
conn_side
(tag)
The side of the connection: client/server/unknown
direction
(tag)
Use the source (src_ip:src_port) as a frame of reference to identify the connection initiator. (incoming/outgoing/unknown; unknown TCP roles are not inferred from port magnitude)
dst_domain
(tag)
Destination domain
dst_ip
(tag)
Destination IP address
dst_ip_type
(tag)
Destination IP type. (other/private/multicast)
dst_k8s_deployment_name
(tag)
Destination K8s deployment name
dst_k8s_namespace
(tag)
Destination K8s namespace
dst_k8s_pod_name
(tag)
Destination K8s pod name
dst_k8s_service_name
(tag)
Destination K8s service name
dst_nat_ip
(tag)
For data containing the outgoing tag, this value is the ip after the DNAT operation
dst_nat_port
(tag)
For data containing the outgoing tag, this value is the port after the DNAT operation
dst_port
(tag)
Destination port; * when it is the client port or the connection role is unknown
family
(tag)
Network layer protocol. (IPv4/IPv6)
host
(tag)
System hostname
pid
(tag)
Process identification number
process_name
(tag)
Process name
server_domain
(tag)
Server domain
server_ip
(tag)
Server IP
server_ip_type
(tag)
Server IP type. (other/private/multicast)
server_k8s_deployment_name
(tag)
Server K8s deployment name
server_k8s_namespace
(tag)
Server K8s namespace
server_k8s_pod_name
(tag)
Server K8s pod name
server_k8s_service_name
(tag)
Server K8s service name
server_port
(tag)
Server port; omitted when the connection role is unknown
source
(tag)
Fixed value: netflow.
src_ip
(tag)
Source IP
src_ip_type
(tag)
Source IP type. (other/private/multicast)
src_k8s_deployment_name
(tag)
Source K8s deployment name
src_k8s_namespace
(tag)
Source K8s namespace
src_k8s_pod_name
(tag)
Source K8s pod name
src_k8s_service_name
(tag)
Source K8s service name
src_port
(tag)
Source port; * when it is the client port or the connection role is unknown
sub_source
(tag)
Some specific connection classifications, such as the sub_source value for Kubernetes network traffic is K8s
transport
(tag)
Transport layer protocol. (udp/tcp)
bytes_read The number of bytes read
Type: int | (gauge)
Unit: digital,B
bytes_written The number of bytes written
Type: int | (gauge)
Unit: digital,B
io_errors_message_size Interval count of hard TCP I/O errors classified as message_size. Nonblocking and interrupted I/O are excluded.
Type: int | (gauge)
Unit: count
io_errors_other Interval count of hard TCP I/O errors classified as other. Nonblocking and interrupted I/O are excluded.
Type: int | (gauge)
Unit: count
io_errors_permission Interval count of hard TCP I/O errors classified as permission. Nonblocking and interrupted I/O are excluded.
Type: int | (gauge)
Unit: count
io_errors_refused Interval count of hard TCP I/O errors classified as refused. Nonblocking and interrupted I/O are excluded.
Type: int | (gauge)
Unit: count
io_errors_reset Interval count of hard TCP I/O errors classified as reset. Nonblocking and interrupted I/O are excluded.
Type: int | (gauge)
Unit: count
io_errors_resource Interval count of hard TCP I/O errors classified as resource. Nonblocking and interrupted I/O are excluded.
Type: int | (gauge)
Unit: count
io_errors_timeout Interval count of hard TCP I/O errors classified as timeout. Nonblocking and interrupted I/O are excluded.
Type: int | (gauge)
Unit: count
io_errors_unreachable Interval count of hard TCP I/O errors classified as unreachable. Nonblocking and interrupted I/O are excluded.
Type: int | (gauge)
Unit: count
io_recv_errors Interval count of hard TCP receive errors.
Type: int | (gauge)
Unit: count
io_send_errors Interval count of hard TCP send errors.
Type: int | (gauge)
Unit: count
packets_read The number of packets read
Type: int | (gauge)
Unit: count
packets_written The number of packets written
Type: int | (gauge)
Unit: count
retransmits Increments of the socket kernel total_retrans counter, including TLP; not a wire packet-loss count
Type: int | (gauge)
Unit: count
rtt TCP Latency
Type: int | (gauge)
Unit: time,μs
rtt_var TCP Jitter
Type: int | (gauge)
Unit: time,μs
tcp_active_reset_calls Interval count of observed kernel hook invocations; not proof of wire delivery. Availability is reported in ebpf_network.
Type: int | (gauge)
Unit: count
tcp_close_wait The number of TCP_CLOSE_WAIT state transitions
Type: int | (gauge)
Unit: count
tcp_closed The number of TCP connection closed
Type: int | (gauge)
Unit: count
tcp_connect_attempts The number of TCP connection attempts
Type: int | (gauge)
Unit: count
tcp_connect_failures The number of TCP connection failures
Type: int | (gauge)
Unit: count
tcp_connect_result_canceled Outgoing connection outcomes completed in this interval: canceled. Pending attempts are excluded.
Type: int | (gauge)
Unit: count
tcp_connect_result_failure Outgoing connection outcomes completed in this interval: failure. Pending attempts are excluded.
Type: int | (gauge)
Unit: count
tcp_connect_result_success Outgoing connection outcomes completed in this interval: success. Pending attempts are excluded.
Type: int | (gauge)
Unit: count
tcp_connect_result_unknown Outgoing connection outcomes completed in this interval: unknown. Pending attempts are excluded.
Type: int | (gauge)
Unit: count
tcp_connect_success_duration_count Completed outgoing handshakes with a known start timestamp.
Type: int | (gauge)
Unit: count
tcp_connect_success_duration_max_us Completed outgoing handshake duration in microseconds: maximum.
Type: int | (gauge)
Unit: time,μs
tcp_connect_success_duration_sum_us Completed outgoing handshake duration in microseconds: sum.
Type: int | (gauge)
Unit: time,μs
tcp_established The number of TCP connection established
Type: int | (gauge)
Unit: count
tcp_last_ack The number of TCP_LAST_ACK state transitions
Type: int | (gauge)
Unit: count
tcp_probe0_calls Interval count of observed kernel hook invocations; not proof of wire delivery. Availability is reported in ebpf_network.
Type: int | (gauge)
Unit: count
tcp_resets_received Interval count of observed kernel hook invocations; not proof of wire delivery. Availability is reported in ebpf_network.
Type: int | (gauge)
Unit: count
tcp_role_observed TCP roles first observed in this interval: client, server, unknown, or mixed. Omitted without new observations; not a connection count. TCP aggregation keeps an unknown direction until role evidence is available.
Type: string | (string)
Unit: N/A
tcp_rto_timer_fires Interval count of observed kernel hook invocations; not proof of wire delivery. Availability is reported in ebpf_network.
Type: int | (gauge)
Unit: count
tcp_socket_errors Newly observed terminal kernel socket errors in this interval.
Type: int | (gauge)
Unit: count
tcp_srtt_snapshot_count Collected kernel SRTT snapshots; repeated userspace map reads are excluded.
Type: int | (gauge)
Unit: count
tcp_srtt_snapshot_max_us Maximum sampled kernel SRTT; not the true RTT peak or application latency.
Type: int | (gauge)
Unit: time,μs
tcp_time_wait The number of TCP_TIME_WAIT state transitions
Type: int | (gauge)
Unit: count

dnsflow

Tags & Fields Description
aggregation_scope
(tag)
service: routine queries grouped by client namespace and workload; endpoint: routine queries with no reliable workload attribution; query: errors, slow queries and timeouts retaining client and domain detail. Counts across scopes are disjoint.
client_ip
(tag)
DNS client IP; * for a workload summary spanning replicas.
client_k8s_namespace
(tag)
Client Kubernetes namespace; part of the workload summary identity.
client_k8s_pod_name
(tag)
Client Pod name when available for query detail; * for a workload summary.
client_k8s_workload_name
(tag)
Client workload name for service summaries.
client_k8s_workload_type
(tag)
Client workload kind; distinguishes same-named workloads of different kinds.
client_port
(tag)
DNS client port; ephemeral ports are aggregated as *.
direction
(tag)
Use the source (src_ip:src_port) as a frame of reference to identify the connection initiator. (incoming/outgoing)
dst_domain
(tag)
Destination domain
dst_ip
(tag)
Destination IP address; * when the destination represents multiple client replicas
dst_ip_type
(tag)
Destination IP type. (other/private/multicast)
dst_k8s_deployment_name
(tag)
Destination K8s deployment name
dst_k8s_namespace
(tag)
Destination K8s namespace
dst_k8s_pod_name
(tag)
Destination K8s pod name
dst_k8s_service_name
(tag)
Destination K8s service name
dst_port
(tag)
Destination port
family
(tag)
Network layer protocol. (IPv4/IPv6)
host
(tag)
System hostname
query_type
(tag)
DNS query type. (A/AAAA/NS/...)
server_ip
(tag)
DNS server IP, retained in routine summaries to distinguish resolvers.
server_port
(tag)
DNS server port.
source
(tag)
Fixed value: dnsflow.
src_ip
(tag)
Source IP; * when the source represents multiple client replicas
src_ip_type
(tag)
Source IP type. (other/private/multicast)
src_k8s_deployment_name
(tag)
Source K8s deployment name
src_k8s_namespace
(tag)
Source K8s namespace
src_k8s_pod_name
(tag)
Source K8s pod name
src_k8s_service_name
(tag)
Source K8s service name
src_port
(tag)
Source port
sub_source
(tag)
Some specific connection classifications, such as the sub_source value for Kubernetes network traffic is K8s
transport
(tag)
Transport layer protocol. (udp/tcp)
count The number of DNS requests in a collection cycle
Type: int | (gauge)
Unit: count
latency Average response time for DNS requests
Type: int | (gauge)
Unit: time,ns
latency_max Maximum response time for DNS requests
Type: int | (gauge)
Unit: time,ns
query_domain The queried domain for errors, slow queries and timeouts; * for routine query summaries
Type: string | (string)
Unit: N/A
rcode DNS response code: 0 - NoError, 1 - FormErr, 2 - ServFail, 3 - NXDomain, 4 - NotImp, 5 - Refused, ...; A value of -1 means the request timed out
Type: int | (gauge)
Unit: N/A

httpflow

Tags & Fields Description
client_ip
(tag)
Client IP address
client_ip_type
(tag)
Client IP type. (other/private/multicast)
client_k8s_deployment_name
(tag)
Client K8s deployment name
client_k8s_namespace
(tag)
Client K8s namespace
client_k8s_pod_name
(tag)
Client K8s pod name
client_k8s_service_name
(tag)
Client K8s service name
client_port
(tag)
Client port, rolled up to * in aggregates; omitted when the connection role is unknown
conn_side
(tag)
The side of the connection: client/server/unknown
direction
(tag)
Use the source (src_ip:src_port) as a frame of reference to identify the connection initiator. (incoming/outgoing/unknown; unknown TCP roles are not inferred from port magnitude)
dst_domain
(tag)
Destination domain
dst_ip
(tag)
Destination IP address
dst_ip_type
(tag)
Destination IP type. (other/private/multicast)
dst_k8s_deployment_name
(tag)
Destination K8s deployment name
dst_k8s_namespace
(tag)
Destination K8s namespace
dst_k8s_pod_name
(tag)
Destination K8s pod name
dst_k8s_service_name
(tag)
Destination K8s service name
dst_nat_ip
(tag)
For data containing the outgoing tag, this value is the ip after the DNAT operation
dst_nat_port
(tag)
For data containing the outgoing tag, this value is the port after the DNAT operation
dst_port
(tag)
Destination port; * when it is the client port or the connection role is unknown
family
(tag)
Network layer protocol. (IPv4/IPv6)
host
(tag)
System hostname
pid
(tag)
Process identification number
process_name
(tag)
Process name
server_domain
(tag)
Server domain
server_ip
(tag)
Server IP
server_ip_type
(tag)
Server IP type. (other/private/multicast)
server_k8s_deployment_name
(tag)
Server K8s deployment name
server_k8s_namespace
(tag)
Server K8s namespace
server_k8s_pod_name
(tag)
Server K8s pod name
server_k8s_service_name
(tag)
Server K8s service name
server_port
(tag)
Server port; omitted when the connection role is unknown
source
(tag)
Fixed value: httpflow.
src_ip
(tag)
Source IP
src_ip_type
(tag)
Source IP type. (other/private/multicast)
src_k8s_deployment_name
(tag)
Source K8s deployment name
src_k8s_namespace
(tag)
Source K8s namespace
src_k8s_pod_name
(tag)
Source K8s pod name
src_k8s_service_name
(tag)
Source K8s service name
src_port
(tag)
Source port; * when it is the client port or the connection role is unknown
sub_source
(tag)
Some specific connection classifications, such as the sub_source value for Kubernetes network traffic is K8s
transport
(tag)
Transport layer protocol. (udp/tcp)
bytes_read The number of bytes read
Type: int | (gauge)
Unit: digital,B
bytes_written The number of bytes written
Type: int | (gauge)
Unit: digital,B
count The total number of HTTP requests in a collection cycle
Type: int | (gauge)
Unit: count
http_version 1.1 / 1.0 ...
Type: string | (string)
Unit: N/A
latency TTFB
Type: int | (gauge)
Unit: time,ns
method GET/POST/...
Type: string | (string)
Unit: N/A
path Request path
Type: string | (string)
Unit: N/A
status_code HTTP status code
Type: int | (gauge)
Unit: N/A
truncated The length of the request path has reached the upper limit of the number of bytes collected, and the request path may be truncated
Type: bool | (gauge)
Unit: bool

日志

ebpf_network_event

Tags & Fields Description
host
(tag)
System hostname
level
(tag)
info for factual clues; warn for hard errors or observed drops
problem_type
(tag)
Bounded network problem classification
source
(tag)
Fixed value ebpf_network_event
collector_session Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
conn_side Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
connection_id Socket-generation identity for tcp_connection_issue logs; a field, never a metric tag.
Type: string | (string)
Unit: N/A
event_id Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
event_status Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
evidence_partial Incomplete evidence or a UDP error without original datagram attribution.
Type: bool | (gauge)
Unit: bool
first_seen_ns Context or observed event count, not unique connections. Timestamps are Unix nanoseconds.
Type: int | (gauge)
Unit: N/A
interface Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
k8s_namespace Local Pod ownership from the cached PID mapping when a severe connection issue first appears. Retained through closure; omitted when no mapping is available. Log fields, never metric tags.
Type: string | (string)
Unit: N/A
k8s_pod_name Local Pod ownership from the cached PID mapping when a severe connection issue first appears. Retained through closure; omitted when no mapping is available. Log fields, never metric tags.
Type: string | (string)
Unit: N/A
k8s_workload_name Local Pod ownership from the cached PID mapping when a severe connection issue first appears. Retained through closure; omitted when no mapping is available. Log fields, never metric tags.
Type: string | (string)
Unit: N/A
last_seen_ns Context or observed event count, not unique connections. Timestamps are Unix nanoseconds.
Type: int | (gauge)
Unit: N/A
local_ip Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
message Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
netns Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
occurrences Context or observed event count, not unique connections. Timestamps are Unix nanoseconds.
Type: int | (gauge)
Unit: N/A
occurrences_total Context or observed event count, not unique connections. Timestamps are Unix nanoseconds.
Type: int | (gauge)
Unit: N/A
pid Context or observed event count, not unique connections. Timestamps are Unix nanoseconds.
Type: int | (gauge)
Unit: N/A
process_name Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
remote_ip Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
scope Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
service Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
service_port Context or observed event count, not unique connections. Timestamps are Unix nanoseconds.
Type: int | (gauge)
Unit: N/A
status Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A
transport Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection.
Type: string | (string)
Unit: N/A

bash

Tags & Fields Description
host
(tag)
host name
cmd Command
Type: string | (string)
Unit: N/A
message The bash execution record generated by the collector
Type: string | (string)
Unit: N/A
pid Process identification number
Type: string | (string)
Unit: N/A
user The user who executes the bash command
Type: string | (string)
Unit: N/A

bpf_net_l4_log

Tags & Fields Description
client_ip
(tag)
The IP address of the client
client_port
(tag)
Client port
conn_side
(tag)
The side of the connection: client/server/unknown
direction
(tag)
Use the source (src_ip:src_port) as a frame of reference to identify the connection initiator. (incoming/outgoing)
dst_ip
(tag)
The IP address of the foreign network interface
dst_k8s_deployment_name
(tag)
Destination K8s deployment name
dst_k8s_namespace
(tag)
Destination K8s namespace
dst_k8s_pod_name
(tag)
Destination K8s pod name
dst_k8s_service_name
(tag)
Destination K8s service name
dst_port
(tag)
Foreign port
host
(tag)
Host name
host_network
(tag)
Whether the network log data is collected on the host network
inner_traceid
(tag)
Correlate the layer 4 and layer 7 network log data of a TCP connection on the collected network interface
k8s_container_name
(tag)
Kubernetes container name
k8s_namespace
(tag)
Kubernetes namespace
k8s_pod_name
(tag)
Kubernetes pod name
l4_proto
(tag)
Transport protocol
l7_proto
(tag)
Application protocol
netns
(tag)
Network namespace, format: NS(<device id>:<inode number>)
nic_mac
(tag)
MAC address of the collected network interface
nic_name
(tag)
Name of the collected network interface
server_ip
(tag)
The IP address of the server
server_port
(tag)
Server port
src_ip
(tag)
The IP address of the collected local network interface
src_k8s_deployment_name
(tag)
Source K8s deployment name
src_k8s_namespace
(tag)
Source K8s namespace
src_k8s_pod_name
(tag)
Source K8s pod name
src_k8s_service_name
(tag)
Source K8s service name
src_port
(tag)
Local port
sub_source
(tag)
Some specific connection classifications, such as the sub_source value for Kubernetes network traffic is K8s
virtual_nic
(tag)
Whether the network log data is collected on the virtual network interface
vni_id
(tag)
Virtual Network Identifier
vxlan_packet
(tag)
Whether it is a VXLAN packet
chunk_id A connection may be divided into several chunks for upload based on time interval or TCP segment number
Type: int | (gauge)
Unit: count
rx_bytes The number of bytes received by the network interface
Type: int | (gauge)
Unit: digital,B
rx_packets The number of packets received by the network interface
Type: int | (gauge)
Unit: count
rx_retrans The number of retransmitted packets received by the network interface
Type: int | (gauge)
Unit: count
rx_seq_max The maximum value of the TCP sequence number of the data packet received by the network interface, which is a 32-bit unsigned integer
Type: int | (gauge)
Unit: N/A
rx_seq_min The minimum value of the TCP sequence number of the data packet received by the network interface, which is a 32-bit unsigned integer
Type: int | (gauge)
Unit: N/A
tcp_syn_retrans The number of retransmitted SYN packets sent by the network interface
Type: int | (gauge)
Unit: count
tx_bytes The number of bytes sent by the network interface
Type: int | (gauge)
Unit: digital,B
tx_packets The number of packets sent by the network interface
Type: int | (gauge)
Unit: count
tx_retrans The number of retransmitted packets sent by the network interface
Type: int | (gauge)
Unit: count
tx_seq_max The maximum value of the TCP sequence number of the data packet sent by the network interface, which is a 32-bit unsigned integer
Type: int | (gauge)
Unit: N/A
tx_seq_min The minimum value of the TCP sequence number of the data packet sent by the network interface, which is a 32-bit unsigned integer
Type: int | (gauge)
Unit: N/A

bpf_net_l7_log

Tags & Fields Description
client_ip
(tag)
The IP address of the client
client_port
(tag)
Client port
conn_side
(tag)
The side of the connection: client/server/unknown
direction
(tag)
Use the source (src_ip:src_port) as a frame of reference to identify the connection initiator. (incoming/outgoing)
dst_ip
(tag)
The IP address of the foreign network interface
dst_k8s_deployment_name
(tag)
Destination K8s deployment name
dst_k8s_namespace
(tag)
Destination K8s namespace
dst_k8s_pod_name
(tag)
Destination K8s pod name
dst_k8s_service_name
(tag)
Destination K8s service name
dst_port
(tag)
Foreign port
host
(tag)
Host name
host_network
(tag)
Whether the network log data is collected on the host network
inner_traceid
(tag)
Correlate the layer 4 and layer 7 network log data of a TCP connection on the collected network interface
k8s_container_name
(tag)
Kubernetes container name
k8s_namespace
(tag)
Kubernetes namespace
k8s_pod_name
(tag)
Kubernetes pod name
l4_proto
(tag)
Transport protocol
l7_proto
(tag)
Application protocol
l7_traceid
(tag)
Correlate the layer 7 network log data of a TCP connection on the all collected network interface
netns
(tag)
Network namespace, format: NS(<device id>:<inode number>)
nic_mac
(tag)
MAC address of the collected network interface
nic_name
(tag)
Name of the collected network interface
parent_id
(tag)
The span id of the APM span corresponding to this network request
server_ip
(tag)
The IP address of the server
server_port
(tag)
Server port
src_ip
(tag)
The IP address of the collected local network interface
src_k8s_deployment_name
(tag)
Source K8s deployment name
src_k8s_namespace
(tag)
Source K8s namespace
src_k8s_pod_name
(tag)
Source K8s pod name
src_k8s_service_name
(tag)
Source K8s service name
src_port
(tag)
Local port
sub_source
(tag)
Some specific connection classifications, such as the sub_source value for Kubernetes network traffic is K8s
trace_id
(tag)
APM trace id
virtual_nic
(tag)
Whether the network log data is collected on the virtual network interface
vni_id
(tag)
Virtual Network Identifier
vxlan_packet
(tag)
Whether it is a VXLAN packet
http_method HTTP method
Type: string | (string)
Unit: N/A
http_path HTTP path
Type: string | (string)
Unit: N/A
http_status_code HTTP status code
Type: int | (gauge)
Unit: N/A
rx_seq The TCP sequence number of the first request or response byte received by the network interface
Type: int | (gauge)
Unit: N/A
tx_seq The TCP sequence number of the first request or response byte sent by the network interface
Type: int | (gauge)
Unit: N/A

链路

dketrace

Tags & Fields Description
dst_ip
(tag)
Destination IP address
dst_port
(tag)
Destination port
host
(tag)
System hostname
service
(tag)
Service name
src_ip
(tag)
Source IP
src_port
(tag)
Source port
app_parent_id Parent span id carried by the application in the request
Type: string | (string)
Unit: N/A
app_trace_id Trace id carried by the application in the request
Type: string | (string)
Unit: N/A
bytes_read Bytes read
Type: int | (gauge)
Unit: digital,B
bytes_written Bytes written
Type: int | (gauge)
Unit: digital,B
duration Duration
Type: int | (gauge)
Unit: time,μs
ebpf_parent_id eBPF parent span id, generated by the ebpftrace collector
Type: string | (string)
Unit: N/A
ebpf_trace_id eBPF trace id, generated by the ebpftrace collector
Type: string | (string)
Unit: N/A
err_msg Redis error message
Type: string | (string)
Unit: N/A
grpc_status_code gRPC status code
Type: string | (string)
Unit: N/A
http_method HTTP method
Type: string | (string)
Unit: N/A
http_route HTTP route
Type: string | (string)
Unit: N/A
http_status_code HTTP status code
Type: string | (string)
Unit: N/A
mysql_err_msg MySQL error message
Type: string | (string)
Unit: N/A
mysql_status_code MySQL request status code
Type: int | (gauge)
Unit: N/A
operation Operation
Type: string | (string)
Unit: N/A
parent_id APM parent span id, set by the ebpftrace collector
Type: string | (string)
Unit: N/A
pid Process identification number
Type: string | (string)
Unit: N/A
process_name Process name
Type: string | (string)
Unit: N/A
resource_type Redis resource type
Type: string | (string)
Unit: N/A
source_type Source type, value is ebpf
Type: string | (string)
Unit: N/A
span_id APM span id, generated by the ebpftrace collector
Type: string | (string)
Unit: N/A
span_type Span type
Type: string | (string)
Unit: N/A
start Start time
Type: int | (gauge)
Unit: timeStamp,usec
status Status
Type: string | (string)
Unit: N/A
status_msg Redis status message
Type: string | (string)
Unit: N/A
thread_name Thread name
Type: string | (string)
Unit: N/A
trace_id APM trace id, can choose between existing app trace id and ebpf generation,set by the ebpftrace collector
Type: string | (string)
Unit: N/A

探针与系统开销

仅在配置追踪目标时启用 Go 关联;优先使用开销较低的探针方式,不支持时自动回退。最后一个目标及其约 30 秒的复用窗口结束后卸载相关探针,后续目标到来时重新启用。

TLS 探针在目标间复用,最后一个目标消失后保留最多约 30 秒,减少短时重启带来的反复挂卸。资源不足时提前回收空闲探针,采集器停止时清理。这些优化不改变上报字段或数据格式。

文件读取预算不等同于磁盘 I/O 限速或超时。实际资源占用请结合节点指标判断。

Prometheus 自监控

配置 pprof_port 后,datakit-ebpf 会在同一监听地址上同时暴露:

  • /debug/pprof/
  • /metrics

建议仅在内网或本机地址上暴露,例如:

pprof_host = "127.0.0.1"
pprof_port = "6061"

OOM 排查重点指标

未显式设置采集器的内存限制或 GOMEMLIMIT 时,采集器启动时读取 cgroup v1/v2 的有效内存上限(包括可见祖先和 v1 层级限制)。由 DataKit 启动的子进程默认将 Go runtime 软预算设为该上限的 35%,独立运行时为 70%;无法读取父进程信息时使用 35%。例如共享 2000 MiB 上限时,eBPF 子进程的 Go 软预算约为 700 MiB。检测只在启动时执行;调整容器限制后需要重启采集器。

此预算通过 Go 的软内存限制促使 GC 回收,不能限制 AF_PACKET/perf mmap、内核 BPF map 或 DataKit 父进程,也不保证容器总 RSS 不越界。已有 GOMEMLIMIT 优先;显式 mem_limit 仍使用原有资源限制和退出行为,并按 DK_EBPF_GO_MEMORY_LIMIT_RATIO(默认 0.8)设置 Go 预算。未配置有限 cgroup 上限的主机不会自动限制 Go 内存。

L7 空闲事件池分为 7 个大小档位,每档记账容量不超过约 512 KiB,总计不超过 3.5 MiB;正在处理的事件仍由异步队列和解码器预算记账。TCP 缺口重试复用 syscall 事件的排序状态。网络字段和连接、字节核账口径不变。

L7 异步工作队列满或字节预算不足时立即拒绝当前事件,按 socket 生命周期标记缺口;perf 回调不等待队列腾空,也不获取该连接的解码器锁。工作线程负责释放失去同步的排队数据和解码状态,并丢弃与过载同时处理的可疑结果。尚未建立用户态连接状态时,以有界的生命周期记录阻止从流中间开始解析。这不会扩大队列或解码内存预算;持续过载仍可能丢样本,应结合 async_event、perf lost 和解码预算计数判断覆盖情况。

支持返回 perf 溢出错误的内核会使用现有有界恢复队列保存 L7 事件,并在恢复完成后继续发送;恢复后的事件仍保持原顺序。dkebpf_exporter_perf_lost_samples_total 记录 perf 传输通知,增长本身不代表相同数量的 L7 事务丢失,应同时查看 dkebpf_exporter_l7_event_recovery_total{result="delivered"} 和实际序号、解码缺口。部分老内核(例如 Linux 4.15)在 perf 溢出时仍返回成功,无法通过该返回值追回事件;未恢复的序号缺口仍会使受影响的采集证据失去完整性,不能按完整事务解释。恢复队列持续过载也可能产生不可恢复缺口。

已经确认无法恢复同步的连接会及时释放待排序 payload 和解码状态,避免这些数据继续占用健康连接的预算;队列中已收到的关闭通知仍用于回收连接。已知 HTTP body 边界且可以安全恢复的缺口保留原有处理。

httpflow 通过 eBPF syscall/socket 探针采集应用读写数据,TLS 明文通过支持的 uprobe 采集;不再启动额外的 L7 AF_PACKET 报文补齐、namespace 监听器或 TCP payload 缓存。bpf-netlog 和 DNS 的独立抓包功能保持原有行为。

sendfile / splice 可能仅提供实际完成的字节范围,不提供对应报文内容。已观测到 HTTP 头且 body 长度已知时可推进字节统计;HTTP 头或 chunk framing 不可见时记录采集缺口,不构造完整事务,也不保证该连接后续的 L7 事务可恢复。TCP 连接和字节统计独立保留。

应用读写数据按有界长度捕获。已知 HTTP body 长度时,解析器按实际完成的 I/O 字节数推进,允许跳过未捕获的 body;若请求头、chunk framing 或其他协议边界缺失,则标记解码缺口并停止不可靠的事务配对。主动截断、用户内存读取失败和 perf/队列事件丢失应分别核对,不能将任一计数直接当作事务丢失率。

以下指标适合优先用于定位 OOM 和内存异常增长:

  • dkebpf_runtime_memory_bytes{type="rss"}
  • 进程实际驻留内存,最接近 OOM 压力
  • dkebpf_runtime_memory_bytes{type="hwm"}
  • 进程 RSS 历史峰值
  • dkebpf_runtime_memory_bytes{type="go_limit"}
  • 当前有限的 Go runtime 软预算;不代表总 RSS 上限
  • dkebpf_runtime_memory_bytes{type="go_total"}
  • Go runtime 已向系统申请且仍保留的主要内存
  • dkebpf_runtime_memory_bytes{type="heap_alloc"}
  • Go 堆当前已分配且仍在使用的内存
  • dkebpf_runtime_memory_bytes{type="non_go_estimate"}
  • 近似值,计算方式为 rss - go_total
  • 用于判断 RSS 增长是否主要来自非 Go 内存,例如 cgo、native buffer、mmap 或其它运行时外分配
  • dkebpf_runtime_memory_ratio{type="non_go_vs_rss"}
  • 非 Go 内存占 RSS 的比例

除上述指标外,还会暴露更细粒度的 Go 内存分量,例如:

  • heap_inuse
  • heap_idle
  • heap_released
  • heap_unused
  • stack_inuse
  • stack_sys
  • mspan_inuse
  • mspan_sys
  • mcache_inuse
  • mcache_sys
  • gc_sys
  • other_sys
  • buck_hash_sys

以及以下辅助指标:

  • dkebpf_runtime_memory_objects
  • dkebpf_runtime_memory_ops_total
  • dkebpf_runtime_gc
  • dkebpf_runtime_goroutines

聚合与缓存压力指标

为避免直接输出高基数业务标签,eBPF 自监控只暴露低基数的聚合压力指标:

  • dkebpf_exporter_agg_entries{component}
  • 当前聚合池中的 key 数量
  • dkebpf_exporter_agg_flush_points_total{component,result}
  • 每次 flush 输出的点数量累计
  • dkebpf_exporter_agg_flush_duration_seconds{component,result}
  • flush 耗时
  • dkebpf_exporter_cache_entries{component,cache}
  • 中间缓存或队列长度

当前已覆盖的组件包括:

  • netflow
  • dnsflow
  • l4log_netflow
  • l4log_httpflow
  • l7flow_http
  • l4log
  • l4log_host_peer_shared

以及部分中间态缓存,例如:

  • dnsflow pending query / packet queue / flush queue
  • l4log connection pool / two-msl pool / blacklist cache
  • l7flow conn map / closed conn map / span buffer

perf 和 TPacket 指标

当前还会暴露与 perf 事件通道以及 afpacket.TPacket 抓包 ring 相关的累计指标:

  • dkebpf_exporter_perf_lost_samples_total{component,stream}
  • perf reader 在用户态观察到的丢样本数量
  • dkebpf_exporter_perf_read_errors_total{component,stream}
  • perf reader 读取错误次数
  • dkebpf_exporter_tpacket_stats_total{component,type}
  • TPacket 累计统计信息
  • type 包括:
    • packets
    • drops
    • queue_freezes

这些指标更适合暴露为累计计数器,而不是单独暴露“速率”指标。 如需查看速率,请直接在 Prometheus 中使用 rate() 或 irate() 计算。

发送链路指标

如果内存增长来自发送堵塞或上游写入变慢,可重点关注:

发送器的队列及在途编码数据合计上限仍为 16 MiB。每批最多 64 点,通常最多 4 MiB;单个较大的点保留完整字段,无法装入其准入预算时记为 batch_too_large。原始网络日志和详细 tracing 不占用最后 2 MiB,为其他输入留出容量。已从抓包状态分离的网络日志和详细 tracing 最多等待 5 秒入队;DNS 等普通输入仍保持 1 秒等待上限。DK_EBPF_EXPORTER_ENQUEUE_TIMEOUT 可同时覆盖两个默认值。下游持续过载仍会显式计数并丢弃;预留容量不保证送达,也不预留 HTTP 工作线程。

诊断 map 准入失败通过本地自监控指标 dkebpf_exporter_diagnostic_map_update_failures_total{map,reason} 细分:map 仅为 flow_faults、connect_failures、udp_errors、listener_stats,reason 仅为 capacity、memory、exists、missing、other。这些计数及 /debug/netflow 的 health.diagnostic_map_failures 补充原有总失败数,不增加平台 measurement。map 占用观测也包含错误样本和监听队列 map。

连接失败和 UDP 硬错误样本的内部 map 键使用结束时间与 PID/TID 组合,避免多个任务在同一时间戳下相互冲突。样本中的开始、结束时间和已有上报字段保持原口径。

错误样本读取在数据探针挂载前启动,独立于整张连接表扫描和 namespace 文件读取,每 100 ms 处理一次有界批次。支持时使用原子批量取出并删除;旧内核使用复用缓冲区的逐项方式。两条路径均不新增无界队列,固定 map 容量不变。

  • dkebpf_exporter_sender_queue_length
  • exporter 当前待发送任务数
  • dkebpf_exporter_sender_batch_points
  • 每个发送批次包含的点数分布
  • dkebpf_exporter_sender_batch_bytes
  • 每个发送批次编码后的字节大小分布
  • dkebpf_exporter_sender_requests_total{result}
  • 发送请求次数
  • dkebpf_exporter_sender_request_duration_seconds{result}
  • 发送请求耗时

其中 result 为低基数枚举值,例如:

  • ok
  • request_error
  • http_4xx
  • http_5xx

周期性摘要日志

除 Prometheus 指标外,datakit-ebpf 还会按采集周期额外发送一条低频摘要日志到 DataKit:

  • 数据类别:Logging
  • measurement:ebpf_monitor
  • source:ebpf_monitor
  • input:ebpf-monitor

这条日志用于补充指标查看,单条日志会汇总:

  • runtime 内存摘要
  • 聚合池和缓存长度
  • 网卡分类数量与 route 数量摘要
  • exporter 队列与发送结果增量
  • perf lost / read error 增量
  • TPacket packets / drops / queue_freezes 增量
  • BPF map 条数、容量、fill ratio 和写入失败增量
  • 输出队列实际丢弃点数增量;不代表临时排队或重试次数
  • perf buffer / socket / filter 的配置与状态摘要
  • 例如 perf reader 大小、共享 host-peer socket 模式、filter attach 结果、socket 侧 drops/freezes 摘要

map 写入失败、输出队列丢弃和 perf 读取错误有新增时,摘要状态为 warn。历史累计失败值保持不变时不会重复记作新增失败。原有 cache.entries 明细保留;cache_entries_total 和 cache.total_entries 不再累加其中的失败计数、累计计数及丢失 epoch。

日志中会带一个较大的 message 字段,内容为固定 schema 的 JSON 摘要,便于直接解析和检索,同时保留少量关键数值 field 便于筛选。

对于 perf/ringbuf 配置、socket buffer 侧状态、共享 socket/filter 模式这类信息,建议只作为摘要日志补充,不再继续扩成独立 Prometheus 指标,以避免额外时间线和低价值告警噪声。

排查建议

可按以下顺序判断问题来源:

  1. rss 持续增长,但 heap_alloc 和 go_total 增长不明显 这通常意味着问题更偏向非 Go 内存,应重点查看 non_go_estimate、non_go_vs_rss
  2. agg_entries、cache_entries 持续增长且不回落 这通常意味着聚合池或中间缓存堆积
  3. sender_queue_length 持续增长,且 sender_request_duration_seconds 变高 这通常意味着发送链路变慢,导致聚合结果无法及时出队
  4. bpf_map_fill_ratio 持续接近 1 这通常意味着内核态 map 压力过高,可能同时伴随事件丢弃

DDTrace 是由 Datadog 发起并开源的链路追踪项目,相关著作权归 Datadog 及其他贡献者所有。观测云支持 DDTrace 的数据上报,具体使用遵循相应项目的开源许可证。

文档评价

文档内容是否对您有帮助?