eBPF
eBPF 采集器,采集主机网络 TCP、UDP 连接信息,Bash 执行日志等。采集器包含以下几个插件:
-
ebpf-net:- 数据类别:
Network - 由
netflow/httpflow/dnsflow构成,分别用于采集主机 TCP/UDP 连接统计信息,HTTP 请求信息和主机 DNS 解析信息;
- 数据类别:
-
ebpf-bash:- 数据类别:
Logging - 采集 Bash 的执行日志,包含 Bash 进程号、用户名、执行的命令和时间等;
- 数据类别:
-
ebpf-conntrack: Version-1.8.0- 往网络流数据上添加两个标签
dst_nat_ip和dst_nat_port,记录经DNAT后的目标 ip 和 port;当内核加载nf_conntrack时可选择开启该插件;
- 往网络流数据上添加两个标签
-
ebpf-trace: Version-1.17.0 · Experimental- 数据类别:
Tracing - 用于跟踪应用网络请求调用关系,基于
ebpf-net的httpflow数据和 eBPF 探针实现;
- 数据类别:
-
bpf-netlog:- 数据类别:
Logging,Network - 该插件实现网络日志
bpf_net_l4_log/bpf_net_l7_log采集,也可以在内核不支持 eBPF 的情况下替代ebpf-net的netflow/httpflow数据采集;
- 数据类别:
配置¶
前置条件¶
在 Kubernetes 环境下部署时,需要将以下目录挂在到容器中:
/sys/kernel/debug- 将主机的
/目录挂载为容器的/rootfs目录,并设置环境变量HOST_ROOT="/rootfs"和HOST_PROC="/rootfs/proc"
可参考 datakit.yaml;
如果为低于 v1.5.6 版本的 DataKit ,则需手动安装 datakit-ebpf 外部采集器。
Linux 内核版本要求¶
支持原生 Linux i386/i686 内核,使用 linux/386 的 DataKit 和 datakit-ebpf。32 位采集器启动时会检查内核架构;在 x86_64 内核上应使用 linux/amd64 版本。内核必须启用 eBPF、kprobe 和 perf event,采集器需要 root 权限。
i386 的网络统计覆盖 IPv4/IPv6 TCP、UDP、RTT、重传、连接失败、进程、网络命名空间和 DNS 关联。UDP 默认关闭,在 [[inputs.ebpf]] 中设置 enable_udp = true,或设置 ENV_INPUT_EBPF_ENABLE_UDP=true。DNAT 关联还需要启用 ebpf-conntrack 并加载对应的内核模块。
启动时,来自当前内核 /sys/kernel/btf/vmlinux 且字段完整的 BTF 直接提供结构偏移,不额外依赖私有 socket 的探测事件。外置 BTF、旧偏移缓存或缺失字段仍通过实际连接验证;偏移缓存绑定完整内核版本和架构,内核更新后自动重新解析。基础采集所需偏移无法确认时,采集器报告启动错误,并同时写入采集器日志。NAT 与 HTTP 偏移复用同一次 BTF 解析;完整的当前内核 BTF 不再触发额外的偏移猜测。无 BTF 时,L7 序号使用已有的有界私有 socket 验证,不假设它与 RTT 字段的相对位置;已验证的诊断字段直接复用,不重复运行同一组测试。可选的 L7 序号无法确认时,只停用对应七层能力,已确认的 netflow 继续采集。私有测试 socket、抓包 ring 和临时探针在启动验证后释放。
目前 Linux 3.10 内核的项目生命周期已经结束,建议您升级至 Linux 4.9 及以上 LTS 版内核。
除 CentOS 7.6+ 和 Ubuntu 16.04 以外,其他发行版本推荐 Linux 内核版本高于 4.9,否则可能无法启动 eBPF 采集器。
若 Linux 内核版本低于 4.4 时可能无法开启 ebpf-trace 插件。
如果要启用 ebpf-conntrack ,请确认内核中的符号是否包含 nf_ct_delete 和 __nf_conntrack_hash_insert,可执行以下命令查看:
或内核是否加载 nf_conntrack 模块:
已启用 SELinux 的系统¶
对于启用了 SELinux 的系统,无法开启 eBPF 采集器,需要关闭其,执行以下命令进行关闭:
采集器配置¶
进入 DataKit 安装目录下的 conf.d/samples 目录,复制 ebpf.conf.sample 并命名为 ebpf.conf。示例如下:
[[inputs.ebpf]]
daemon = true
name = 'ebpf'
cmd = "/usr/local/datakit/externals/datakit-ebpf"
args = [
"--datakit-apiserver", "0.0.0.0:9529",
]
envs = []
## Recover excessive historical candidates before registering new probes.
## Only DK-compatible definitions count; duplicate definitions are not required.
## Default total threshold: 200. Zero disables heuristic recovery.
## Requires upstream Linux 3.10.10+ or standard EL7 x86_64 3.10.0-123+,
## complete HOST_PROC and no other DK eBPF collector. See KERNEL_COMPATIBILITY.md.
## Enabled/perf-busy events are preserved. Small inventories stay conservative.
legacy_probe_cleanup_threshold = 200
## Resource limits.
## The collector automatically exits when the limit is exceeded.
## Can configure the number of cpu cores, memory size and network bandwidth.
##
# cpu_limit = "2.0"
# mem_limit = "4GiB"
# net_limit = "100MiB/s"
## automatically takes effect when running DataKit in
## Kubernetes daemonset mode
##
# kubernetes_url = "https://kubernetes.default:443"
# bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
##
## or
# bearer_token_string = "<your-token-string>"
## k8s workload labels
##
# workload_labels = ["app"]
# workload_label_prefix = ""
## operator url
##
# operator_url = "https://datakit-operator.datakit.svc:443"
## all supported plugins:
## - "ebpf-net" :
## contains L4-network(netflow), L7-network(httpflow, dnsflow) collection
## - "ebpf-bash" :
## log bash
## - "ebpf-conntrack":
## add two tags "dst_nat_ip" and "dst_nat_port" to the network flow data
## - "ebpf-trace":
## param trace_server must be set simultaneously.
## - "bpf-netlog":
## contains L4-network log (bpf_net_l4_log), L7-network log (bpf_net_l7_log),
## L4-network(netflow), L7-network(httpflow, dnsflow) collection
enabled_plugins = [
"ebpf-net",
]
## Collect UDP flows in addition to TCP.
enable_udp = false
## Network diagnostic metrics and bounded structured problem logs.
## These use their own budgets, independent of netflow sampling.
network_diagnostics = true
## Default: 40 node metric timelines with stable global tags.
## Full metrics add collector fields and per-netns/interface timelines.
network_diagnostics_full_metrics = false
# network_diagnostics_max_groups = 4096
## Both limits apply; allow four minutes of burst tokens.
# network_diagnostics_log_bytes_per_minute = 524288
# network_diagnostics_log_points_per_minute = 240
## If you enable the ebpf-net plugin, you can configure:
## - "httpflow" (* enabled by default)
## - "httpflow-tls"
##
l7net_enabled = [
"httpflow",
# "httpflow-tls"
]
## datakit-ebpf pprof service
pprof_host = "127.0.0.1"
pprof_port = "6061"
## netlog blacklist
##
# netlog_blacklist = "ip_saddr=='127.0.0.1' || ip_daddr=='127.0.0.1'"
## bpf-netlog plugin collection metric and log
##
netlog_metric = true
netlog_log = false
## bpf-netlog L7 log protocols.
## "http" enables both HTTP/1.x and HTTP/2. Use "http1" only when HTTP/1.x is needed alone.
## Available values: "http", "http1", "http2", "grpc".
##
# netlog_l7log_protocols = ["http"]
## bpf-netlog L7 log HTTP headers.
## Empty uses the built-in recommended allowlist, ["none"] disables header capture.
##
# netlog_l7log_headers = [
# "host",
# "x-request-id",
# "traceparent",
# ]
## bpf-netlog AF_PACKET capture ring limits.
## In Kubernetes, datakit-ebpf uses smaller defaults to reduce resident memory.
##
# netlog_fallback_sockets = 0
# netlog_fallback_blocks = 0
# netlog_shared_blocks = 0
## Advanced datakit-ebpf capacity tuning can be added to envs above.
## Prefer changing these only when self-monitoring shows drops caused by limits.
##
## "DK_EBPF_NETFLOW_MAP_MAX_ENTRIES=131072",
## "DK_EBPF_L4LOG_CONN_CACHE_LIMIT=131072",
## "DK_EBPF_L7FLOW_CONN_MAP_LIMIT=131072",
## eBPF trace generation server center address.
trace_server = ""
## trace all processes containing any specified environment variable
trace_env_list = [
# "DK_BPFTRACE_SERVICE",
# "DD_SERVICE",
# "OTEL_SERVICE_NAME",
]
## deny tracking any process containing any specified environment variable
trace_env_blacklist = []
## trace all processes containing any specified process names,
## can be used with trace_namedenyset
##
trace_name_list = []
## deny tracking any process containing any specified process names
##
trace_name_blacklist = [
## The following two processes are hard-coded to never be traced,
## and do not need to be set:
##
# "datakit",
# "datakit-ebpf",
]
## conv other trace id to datadog trace id (base 10, 64-bit)
conv_to_ddtrace = false
## If the system does not enable ipv6, it needs to be changed to true
##
ipv6_disabled = false
## ephemeral port strart from <ephemeral_port>
##
# ephemeral_port = 10001
## Dynamic network path discovery.
## datakit-ebpf only reports candidates; DataKit netpath runs traceroute
## and uploads netpath points.
##
# network_path_enabled = false
# network_path_api = "http://127.0.0.1:9529/v1/netpath/candidates"
# network_path_token = ""
# network_path_flush_interval = "10s"
# network_path_batch_size = 100
# network_path_http_timeout = "3s"
# network_path_queue_size = 4096
# interval = "60s"
# sampling_rate = "0.50"
# sampling_rate_pts_per_min = "1500"
[inputs.ebpf.tags]
# some_tag = "some_value"
# more_tag = "some_other_value"
#############################
## Parameter description (if marked * is required)
#############################
## --hostname : Host name, this parameter can change the value of the host tag when the collector uploads data, the priority is: specify this parameter >
## ENV_HOSTNAME value in datakit.conf (if it is not empty, this parameter will be added automatically at startup) >
## collector Get it yourself (the default).
## --datakit-apiserver : DataKit API Server address, default value 0.0.0.0:9529 .
## --log : Log output path, default <DataKitInstallDir>/externals/datakit-ebpf.log.
## --log-level : Log level, the default value is 'info'.
## --service : The default value is 'ebpf'.
配置好后,重启 DataKit 即可。
Kubernetes 中可以通过 ConfigMap 或者直接默认启用 eBPF 采集器两种方式来开启采集:
- ConfigMap 方式参照通用的安装示例。
- 在 datakit.yaml 中的环境变量
ENV_ENABLE_INPUTS中追加ebpf,此时使用默认配置,即仅开启ebpf-net网络数据采集
环境变量与配置项¶
通过以下环境变量可以调整 Kubernetes 中 eBPF 采集配置:
配置项:
-
enabled_plugins:- 描述:用于配置开启采集器的内置插件
- 环境变量:
ENV_INPUT_EBPF_ENABLED_PLUGINS - 示例:
ebpf-net,ebpf-trace
-
l7net_enabled- 描述:开启 http 协议数据采集
- 环境变量:
ENV_INPUT_EBPF_L7NET_ENABLED - 示例:
httpflow
-
interval- 描述:设置采样时间间隔
- 环境变量:
ENV_INPUT_EBPF_INTERVAL - 示例:
1m30s
聚合周期由 interval 统一控制,默认及最小值为 60s,适用于 ebpf-net 的 netflow/httpflow/dnsflow 和 bpf-netlog 的 netflow/httpflow。DNS 请求匹配超时与聚合周期独立;窗口内的次数和总延迟直接累加,以请求次数计算平均延迟,避免对分组平均值再次求平均。
DNSFlow 在采集器所在的 network namespace 中使用一个抓包 socket,统计主机侧可见的明文 DNS 流量,不逐个进入 Pod namespace 抓包,也不按 Kubernetes namespace 筛选。不同业务 namespace 的 Pod 流量可能在主机侧被观测到;容器内部未经过可见接口的流量不保证覆盖。DNS 聚合键保持客户端/服务端顺序,本地观察侧在导出时确定,避免地址信息在窗口中途补齐造成同一组统计拆分。
DNS 域名关联优先使用同一作用域内的客户端 IP 与应答地址;主机侧抓到的应答也可按原始客户端 IP 关联到 Pod socket,抓包所在 netns 不等同于 socket 所在 netns。不会退回任意 namespace 或全局 IP 匹配;跨作用域关联不接受回环、未指定或 link-local 客户端地址。TCP 连接按创建或首次接管的观测时间关联 DNS 候选,并随该 socket 代次保留;TTL 从 DNS 抓包时间起算。首次未命中时仅随原有上报周期重试,允许补入早已发生但处理较晚的应答,不采用晚于连接观测时间的解析。已有候选不会因 TTL 过期而消失。缓存仅为历史时点查询多保留一个清理周期(1 分钟)的过期元数据,当前时点查询仍严格检查 TTL,不保存完整 DNS 历史。更强的应用层域名关联仍沿用原有逻辑。入站 server_domain 使用本地服务端证据;dst_domain 继续表示原始远端地址的域名,不能直接当成入站服务端域名。连接回收后释放候选,新连接重新查有效缓存。UDP 仍按当前观测关联。同一聚合桶内 DNS 候选不一致时不填 DNS 域名,不增加聚合维度或上报点数。此关联属于 DNS 推断,并不能证明应用使用了该域名;采集器启动前的解析、不可见或加密 DNS 无法据此补回。
DNSFlow 将正常查询与诊断明细分开聚合,沿用 dnsflow measurement 和原有字段类型:
DNS 客户端 namespace 或 Pod 为 <nil> 时,表示没有匹配到 Kubernetes 元数据,不能直接解释为 Node 流量。客户端为节点 IP 时,也可能是 hostNetwork 或经 SNAT 的 DNS 转发。排查时结合 client_ip、server_ip 和可用的 Pod 标签区分各段链路;同一次解析的客户端、转发和上游记录不能相加作为唯一业务查询数。
aggregation_scope |
聚合口径 | 保留的信息 |
|---|---|---|
service |
正常查询按客户端 namespace、workload 类型及名称、DNS 服务器 IP/端口、查询类型、返回码、传输协议及地址族汇总 | 次数、平均/最大延迟;客户端 IP、Pod 名和 query_domain 为 *,不使用某个副本作为整组的代表 |
endpoint |
客户端缺少可信工作负载归属时,正常查询按客户端 IP 与 DNS 端点汇总 | 保留客户端 IP,query_domain=*;不同未知客户端不会混成一个服务 |
query |
错误响应、响应时间达到 1 秒、或超过匹配超时仍未观测到响应 | 保留客户端 IP、可用的 Pod 标签及完整查询域名 |
上述三类计数互斥,异常请求不会再计入正常汇总。快速 NXDOMAIN 沿用原有规则,不直接视为故障,但 rcode=3 独立汇总,仍可观察名称不存在的次数;慢 NXDOMAIN 保留诊断明细。异常同时继续进入已启用的 ebpf_network_event 通道,受原有日志预算约束;关闭该通道不影响 query 类 DNSFlow。此调整不增加抓包、proc 扫描或 namespace 切换,服务归属只查现有内存中的 Kubernetes 元数据,并在聚合窗口内缓存,缓存上限沿用 DNS 聚合上限。反向域名缓存仍消费原始 DNS 应答。
-
ipv6_disabled- 描述:系统是否不支持 IPv6
- 环境变量:
ENV_INPUT_EBPF_IPV6_DISABLED - 示例:
false
-
ephemeral_port- 描述:仅用于 UDP 方向的临时端口辅助推断。TCP 方向由连接或协议证据确定,不按端口大小猜测。netflow/httpflow 在角色明确时将客户端端口归并为
*,保留服务端端口;角色未知时两端端口归并为*,保留地址对及收发总量,不生成客户端/服务端派生字段。连接明细和本地诊断保留原始端口。 - 环境变量:
ENV_INPUT_EBPF_EPHEMERAL_PORT - 示例:
32768
- 描述:仅用于 UDP 方向的临时端口辅助推断。TCP 方向由连接或协议证据确定,不按端口大小猜测。netflow/httpflow 在角色明确时将客户端端口归并为
-
pprof_host- 描述:pprof host
- 环境变量:
ENV_INPUT_EBPF_PPROF_HOST - 示例:
127.0.0.1
-
pprof_port- 描述:pprof port
- 环境变量:
ENV_INPUT_EBPF_PPROF_PORT - 示例:
6061
-
trace_server- 描述:开启
ebpftrace采集器的 DataKit ELinker/ DataKit 的地址 - 环境变量:
ENV_INPUT_EBPF_TRACE_SERVER - 示例:
<ip>:<port>
- 描述:开启
-
trace_name_blacklist- 描述:指定进程名的进程将被禁止采集链路数据
- 环境变量:
ENV_INPUT_EBPF_TRACE_NAME_BLACKLIST - 示例:
-
trace_env_blacklist- 描述:包含任意一个指定环境变量名的进程将被禁止采集链路数据
- 环境变量:
ENV_INPUT_EBPF_TRACE_ENV_BLACKLIST - 示例:
DKE_DISABLE_ETRACE
-
trace_env_list- 描述:含有任意指定环境变量的进程的链路数据将被跟踪和上报
- 环境变量:
ENV_INPUT_EBPF_TRACE_ENV_LIST - 示例:
DK_BPFTRACE_SERVICE,DD_SERVICE,OTEL_SERVICE_NAME
-
trace_name_list- 描述:进程名在指定集合内的的进程将被跟踪和上报
- 环境变量:
ENV_INPUT_EBPF_TRACE_NAME_LIST - 示例:
chrome,firefox
-
conv_to_ddtrace- 描述:将所有的应用侧链路 id 转换为 10 进制表示的字符串,兼容用途,非必要不使用
- 环境变量:
ENV_INPUT_EBPF_CONV_TO_DDTRACE - 示例:
false
-
netlog_blacklist- 描述:用于实现在抓包之后的数据包的过滤
- 环境变量:
ENV_INPUT_EBPF_NETLOG_BLACKLIST - 示例:
ip_saddr=='127.0.0.1' \|\| ip_daddr=='127.0.0.1'
-
netlog_metric- 描述:从网络数据包分析采集网络指标
- 环境变量:
ENV_INPUT_EBPF_NETLOG_METRIC - 示例:
true
-
netlog_log- 描述:从网络数据包分析采集网络日志
- 环境变量:
ENV_INPUT_EBPF_NETLOG_LOG - 示例:
false
-
netlog_l7log_protocols- 描述:设置
bpf-netlog七层日志解析协议。http会同时开启 HTTP/1.x 和 HTTP/2;仅需 HTTP/1.x 时再使用http1。可选值:http、http1、http2、grpc - 环境变量:
ENV_INPUT_EBPF_NETLOG_L7LOG_PROTOCOLS - 示例:
http,http2,grpc
- 描述:设置
-
netlog_l7log_headers- 描述:设置
bpf-netlog七层日志记录的 HTTP Header。空值使用内置推荐列表;设置为none可关闭 Header 采集。敏感 Header 会被强制忽略 - 环境变量:
ENV_INPUT_EBPF_NETLOG_L7LOG_HEADERS - 示例:
host,x-request-id,traceparent
- 描述:设置
-
netlog_fallback_sockets- 描述:设置
bpf-netlogfallback 网络命名空间抓包 socket 数量上限。Kubernetes 中默认关闭 fallback 抓包,可按需调大 - 环境变量:
ENV_INPUT_EBPF_NETLOG_FALLBACK_SOCKETS - 示例:
8
- 描述:设置
-
netlog_fallback_blocks- 描述:设置
bpf-netlogfallback 抓包 AF_PACKET ring block 数量 - 环境变量:
ENV_INPUT_EBPF_NETLOG_FALLBACK_BLOCKS - 示例:
4
- 描述:设置
-
netlog_shared_blocks- 描述:设置
bpf-netlog共享 host-peer 抓包 AF_PACKET ring block 数量 - 环境变量:
ENV_INPUT_EBPF_NETLOG_SHARED_BLOCKS - 示例:
64
- 描述:设置
-
cpu_limit- 描述:单位时间内 CPU 最大核心数使用限制,到达上限,采集器退出
- 环境变量:
ENV_INPUT_EBPF_CPU_LIMIT - 示例:"2.0"`
-
mem_limit- 描述:内存大小使用限制
- 环境变量:
ENV_INPUT_EBPF_MEM_LIMIT - 示例:
"4GiB"
-
net_limit- 描述:网络带宽(任意网卡)限制
- 环境变量:
ENV_INPUT_EBPF_NET_LIMIT - 示例:
"100MiB/s"
-
sampling_rate- 描述:设置 eBPF 采集器上报数据时的采样率,范围
0.01 ~ 1.00;与设置samping_rate_pts_per_min互斥 - 环境变量:
ENV_INPUT_EBPF_SAMPLING_RATE - 示例:
0.50
- 描述:设置 eBPF 采集器上报数据时的采样率,范围
-
sampling_rate_pts_per_min- 描述:设置 eBPF 采集器上报数据时的每分钟数据量阈值,动态调整采样率
- 环境变量:
ENV_INPUT_EBPF_SAMPLING_RATE_PTSPERMIN - 示例:
1500
-
workload_labels- 描述:设置需要添加到数据的 K8s workload 所有指定的标签
- 环境变量:
ENV_INPUT_EBPF_WORKLOAD_LABELS - 示例:
app,project_id
-
workload_label_prefix- 描述:为 k8s 的 workload 标签添加前缀
- 环境变量:
ENV_INPUT_EBPF_WORKLOAD_LABEL_PREFIX - 示例:
k8s_workload_label_
-
operator_url- 描述:设置 operator 地址,用于采集 k8s 集群的 workload 标签
- 环境变量:
ENV_INPUT_EBPF_OPERATOR_URL - 示例:
https://datakit-operator.datakit.svc:443
高负载采集上限调优¶
以下为 datakit-ebpf 进程内部的高级容量参数,不建议作为常规配置项调大。只有在自监控指标或摘要日志中看到 limit、queue_full、pending_query_limit、bpf_map_fill_ratio 接近 1 等压力信号时,再通过 inputs.ebpf.envs 或 DaemonSet 环境变量设置。
| 环境变量 | 默认值 | 最大值 | 作用范围 |
|---|---|---|---|
DK_EBPF_NETFLOW_MAP_MAX_ENTRIES |
65536 |
1048576 |
ebpf-net 内核连接 map 容量 |
DK_EBPF_NETFLOW_CLOSED_CONN_CACHE_LIMIT |
65536 |
1000000 |
ebpf-net 已关闭连接缓存 |
DK_EBPF_DNSFLOW_PENDING_QUERY_LIMIT |
65536 |
1000000 |
DNS 未响应 query 缓存 |
DK_EBPF_DNSFLOW_AGG_LIMIT |
32768 |
1000000 |
DNS 聚合 key 数 |
DK_EBPF_DNSFLOW_ANSWER_RECORD_LIMIT |
65536 |
1000000 |
DNS 反查记录缓存 |
DK_EBPF_L4LOG_CONN_CACHE_LIMIT |
65536 |
1000000 |
bpf-netlog L4 连接缓存 |
DK_EBPF_L4LOG_NETFLOW_AGG_LIMIT |
32768 |
1000000 |
bpf-netlog L4 指标聚合 key 数 |
DK_EBPF_L4LOG_HTTP_AGG_LIMIT |
32768 |
1000000 |
bpf-netlog HTTP 指标聚合 key 数 |
DK_EBPF_L4LOG_HTTP_ELEM_LIMIT |
1024 |
65536 |
bpf-netlog 单连接尚未完成的 HTTP 请求状态数 |
DK_EBPF_L4LOG_HTTP2_STREAM_LIMIT |
1024 |
65536 |
bpf-netlog 单连接尚未完成的 HTTP/2 stream 状态数 |
DK_EBPF_L7FLOW_CONN_MAP_LIMIT |
65536 |
1000000 |
ebpf-trace/httpflow 用户态连接状态数 |
DK_EBPF_L7FLOW_HTTP_AGG_LIMIT |
32768 |
1000000 |
httpflow HTTP 聚合 key 数 |
DK_EBPF_L7FLOW_HTTP2_STREAM_LIMIT |
1024 |
65536 |
httpflow 单连接尚未完成的 HTTP/2 stream 状态数 |
DK_EBPF_L7FLOW_MYSQL_PENDING_BUFFER_LIMIT |
65536 |
4194304 |
MySQL 协议半包缓冲字节数 |
DK_EBPF_CONNTRACK_TUPLE_MAP_MAX_ENTRIES |
65535 |
1048576 |
连接跟踪 tuple map 容量 |
DK_EBPF_PROCWATCH_MAP_MAX_ENTRIES |
12800 |
1048576 |
进程 watch 内核 map 容量 |
DK_EBPF_* 环境变量会从 DataKit 传入 datakit-ebpf;同名参数以 inputs.ebpf.envs 中的显式配置为准。连接 map 容量在采集器启动时生效,默认仍为 65536。配置扩容后应检查自监控中对应 map 的 capacity / max_entries 实际值;仅支持扩容或修改配置文件不表示运行中的 map 已扩容。
对于 bpf-netlog,上述 HTTP/HTTP2 上限约束尚未完成的事务。已完成事务达到一批时会主动唤醒消费,不必等待 8 秒维护周期;等待生成日志或聚合的数据由每个抓包实例内所有连接共享 16 MiB 的暂存预算,消费完成后归还。该预算按结构体、字符串和请求/响应头的保留量估算,不是整个进程 RSS 上限。输出持续阻塞并耗尽预算时,完成记录的淘汰会计入采集质量计数,原因是 completed_budget。
HTTP/2 支持协商后 HPACK 动态表大小的变化,并以编码流中的表大小更新驱动解码与字典淘汰,避免跨方向 syscall 完成顺序影响解码,参见 RFC 9113 和 RFC 7541。每个方向的动态表解码上限为 64 KiB;仅宣告更大的允许值不会分配同等大小的表,编码流要求超过该上限时会停止该连接的解码并记录采集质量计数。httpflow 在请求和响应两侧都结束后导出完整事务,取消或被 GOAWAY 拒绝的未完成流会释放状态,不生成成功事务。
HTTP/2 对先于请求事件到达的 RST_STREAM,在现有连接索引中保留终止标记,直到观测到对应请求;标记不占用正常未完成事务配额,其数量受同一个 stream 配置上限约束。终止标记超限时停止该连接解码,并记录 terminal_limit 采集质量计数。服务端 push 暂不导出事务,但仍解压相关头部以维护 HPACK 字典,避免已完成 push 占用正常请求容量。
DNS over TCP 的请求/响应匹配包含本地连接代际,避免重连复用端口和 DNS ID 后串配;该身份不增加上报字段、标签或聚合维度。SYN-ACK 校验包含已观测到的 SYN 数据范围,支持 Fast Open;RST 需要匹配当前方向的连续序号,握手拒绝则校验 SYN 确认范围,避免旧连接报文清理新连接状态。
FIN 也会先校验重组序号范围和已观测到的对端确认范围,再更新关闭状态;无法匹配的控制报文不会改变当前连接。空闲时只释放过期的未完成片段,干净的连接身份仍在原有 4096 条方向状态上限内保留,以支持空闲后继续拆包通信;容量不足时优先回收关闭或双向空闲的身份。片段过期导致的重组缺口会使相关待匹配查询失效,不作为业务 DNS 超时上报。
eBPF 链路功能¶
ebpf-trace 采集分析主机上的进程读写的网络数据,并对进程的内核级线程/用户级线程(如 golang goroutine)进行跟踪,生成链路 eBPF Span 该数据需要被 ebpftrace 采集进行进一步的加工处理。
使用时,需要在多个节点部署了该开启链路数据采集的 eBPF 采集器,则需要将所有 eBPF Span 数据发往同一个开启了 ebpftrace 采集器插件的 DataKit ELinker/DataKit。更多配置细节见 eBPF 链路文档
bpf-netlog 插件的黑名单功能¶
过滤器规则示例:
单条规则:
以下规则过滤 ip 为 1.1.1.1 且端口为 80 的网络数据。(运算符后允许换行)
多条规则:
规则间使用 ; 或 \n 分隔,满足任意一条规则就进行数据过滤
udp
ip_saddr == "1.1.1.1" && (src_port == 80 || dst_port == 80);
ip_saddr == "10.10.0.1" && (src_port == 80 || dst_port == 80)
ipnet_contains("127.0.0.0/8", ip_saddr); ipv6
可用于过滤的数据:
该过滤器用于对网络数据进行过滤,可比较的数据如下:
| key 名 | 类型 | 描述 |
|---|---|---|
tcp |
bool | 是否为 TCP 协议 |
udp |
bool | 是否为 UDP 协议 |
ipv4 |
bool | 是否为 IPv4 协议 |
ipv6 |
bool | 是否为 IPv6 协议 |
src_port |
int | 源端口(以被观测网卡/主机/容器为参考系) |
dst_port |
int | 目标端口 |
ip_saddr |
str | 源 IPv4 网络地址 |
ip_saddr |
str | 目标 IPv4 网络地址 |
ip6_saddr |
str | 源 IPv6 网络地址 |
ip6_daddr |
str | 目标 IPv6 网络地址 |
k8s_src_pod |
str | 源 pod 名 |
k8s_dst_pod |
str | 目标 pod 名 |
运算符:
运算符从高往低:
| 优先级 | Op | 名称 | 结合方向 |
|---|---|---|---|
| 1 | () |
圆括号 | 左 |
| 2 | ! |
逻辑非,一元运算符 | 右 |
| 3 | != |
不等于 | 左 |
| 3 | >= |
大于等于 | 左 |
| 3 | > |
大于 | 左 |
| 3 | == |
等于 | 左 |
| 3 | <= |
小于等于 | 左 |
| 3 | < |
小于 | 左 |
| 4 | && |
逻辑与 | 左 |
| 4 | \|\| |
逻辑或 | 左 |
函数:
-
ipnet_contains
函数签名:
fn ipnet_contains(ipnet: str, ipaddr: str) bool描述: 判断地址是否在指定的网段内
示例:
如果
ip_saddr值为 "127.0.0.1",则该规则返回true,该 TCP 连接数据包/ UDP 数据包将被过滤。 -
has_prefix
函数签名:
fn has_prefix(s: str, prefix: str) bool描述: 指定字段是否包含某一前缀
示例:
如果 pod 名为
datakit-kfez321,该规则返回true。
网络问题诊断¶
启用 ebpf-net 后,网络诊断默认开启。数据直接上报到 DataKit,无需访问主机上的 /debug/netflow:
TCP 优先将实际 NAT 报文验证的目标保存在对应 socket 指针和出生时间的连接快照中,随连接统计保留到用户态消费,避免 conntrack 删除或源端口复用抹掉尚未消费的归属。同一 TCP 世代出现冲突的目标时保持 N/A。
IPVS 的 MASQ/NAT 路径在启用 IPVS conntrack 支持、相关内核钩子可用时,通过实际 skb 和 conntrack reply 发布关联;存量连接在 IPVS 更新函数返回后读取已确认的映射。无 BTF 时,至少两个不同 conntrack owner 对同一 skb 指针字段达成唯一验证后才读取该字段;校准前或证据冲突时,缺少证明的存量连接保持 N/A。常规路径不遍历 IPVS 连接表,也不保存报文历史。
UDP 和缺少上述 socket 证明的连接,使用地址族、TCP/UDP 协议、来源网络命名空间和完整地址端口回退查询。同一 tuple 出现多个 owner 时,只暂停该 tuple 的回退关联;源版本和成员数稳定且归属唯一后可自动恢复,不需要重启采集器。UDP 不给整个 socket 固定一个后端。跨命名空间 DNAT 需要实际报文入口和经过双向核验的 veth 关系;无法确认对应关系时保持 N/A,不能保证覆盖所有 CNI 和流量卸载路径。map 容量拒绝会记录为采集缺口。
单独启用 httpflow-tls 会启动其依赖的七层采集。七层偏移或探针初始化失败时,已验证的基础网络采集继续运行;启动日志及开启的网络诊断会记录七层能力不可用。DNS 重组容量丢弃会使受影响的待响应请求失效,不会将已知漏采直接判为 DNS 无响应。
| 数据 | 用途 | 频率与范围 |
|---|---|---|
netflow(Network) |
保留流量聚合,追加连接结果、实际观测的角色、I/O 错误、重传相关调用和 SRTT 快照统计 | 沿用原有聚合周期、标签和采样;已有字段的类型、单位和语义保持不变 |
ebpf_network(Metric) |
查看节点异常趋势、来源新鲜度、采集能力和丢失情况 | 每分钟;默认固定 40 个节点级字段,完整指标模式追加命名空间及发生错误增量的网卡 |
ebpf_network_event(Logging) |
保留有时间、进程、端点、阶段和证据范围的问题样本 | 首次满足日志条件后通常在 5 秒内进入发送队列;同组明确错误有新观测时每分钟更新,普通连接线索最多每 5 分钟更新一次 |
# 在 [[inputs.ebpf]] 内配置
network_diagnostics = true
network_diagnostics_full_metrics = false
network_diagnostics_max_groups = 4096
network_diagnostics_log_bytes_per_minute = 524288
network_diagnostics_log_points_per_minute = 240
对应环境变量为 ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS、ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_FULL_METRICS、ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_MAX_GROUPS、ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_LOG_BYTES_PER_MINUTE、ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_LOG_POINTS_PER_MINUTE。日志预算的有效范围为 4096~1048576 字节/分钟、1~4096 条/分钟,分组数量最多 4096。以上预算按节点生效。UDP 的启用配置仍由 enable_udp 控制;DNS 明文抓包和命名空间计数有各自的能力字段。
netflow 的 tcp_role_observed 是字符串 field,表示本指标窗口首次观测到的 TCP 角色类型:client、server、unknown;同一聚合点包含多种角色时为 mixed。没有新的角色观测时省略该字段。它不是连接数量。direction 和 conn_side 根据可用的连接或协议证据确定;无法确认的 TCP 角色保持 unknown,聚合不按端口大小猜测服务端。
对启动前已建立的 TCP 连接,ebpf-net 与 bpf-netlog 共用按网络命名空间刷新的监听索引。优先使用 connect/accept 或抓包握手证据,其次使用可验证的协议角色,再使用本地监听端点推断服务端;没有匹配监听不等于客户端。监听匹配核对地址族、地址与端口,失败刷新不覆盖完整快照,超过 30 秒未成功刷新即不再使用该推断。监听推断不会永久写成握手事实,后续直接角色证据可以纠正;ebpf-net 在当前窗口内迁移已计入的贡献;bpf-netlog 对存活连接在上报窗口结束时使用最新角色聚合,提前移除的连接在移除时结算,不增加每连接迁移账本。两路均保持流量总量不变,已上报窗口不回写。存量发现不补造历史连接建立次数、握手耗时或启动前流量。网络诊断中的 role_source=listener 表示监听推断;tcp_role_observed 仍只描述内核实际观测角色,因此可以与推断出的 direction 不同。监听刷新共享现有 namespace 清单与 INET_DIAG 查询,不遍历进程 FD;该逻辑不依赖 BTF。
bpf-netlog 在漏掉 TCP 握手时,可在汇总阶段将 NAT 后的报文端点关联到原始 socket,再使用实际观测到的连接角色或单独标识的监听证据确定服务端口。关联需要唯一的 NAT 映射、已确认的源命名空间和匹配的连接生命周期;NAT 的 original/reply 方向本身不代表应用的客户端/服务端角色。映射冲突、已清理或缺少原始角色证据时保持 unknown,包括可能在关闭时单独汇总的未知角色计数;收发总量仍保留,连接明细仍使用抓包看到的原始端点。
bpf-netlog 按已捕获的 TCP 唯一序号范围统计字节,不等待协议重组补齐缺口。新的初始序号可区分元组复用与 SYN 重传;换代时先结算旧代际,分别保留新旧连接的状态,不补造未观测到的关闭事件。去重仅保存有限的序号区间,不保留额外报文;超出历史范围时,通过本地采集质量计数 dkebpf_exporter_bpf_events_dropped_total{component="l4log",event="tcp_accounting",reason="range_history"} 标识不完整证据。
L7 已知的入队丢失标记绑定具体 socket 代际,内核仍持有该代际时不会仅因时间到期而清除,也不会按出生时间封禁其他老连接。标记受容量约束;确实无法保留时记录 event="admission_marker",reason="limit" 并使用已有丢失代次保护,避免静默错配。监听刷新使用固定四个 worker,独立调度到期 namespace,优先续期已确认的监听;namespace 清单刷新与监听读取分别执行。
为减少上报体积,netflow 的追加计数字段仅在增量非零时输出;成功建立耗时及 SRTT 快照字段仅在对应样本数非零时输出,有样本的合法零值仍保留。原有字段不采用此省略规则。查询应允许追加字段缺失,并结合采集能力与证据完整性判断,不能把缺失直接解释为没有网络故障。
默认指标与时间线数量¶
默认 network_diagnostics_full_metrics = false,每节点只输出一组稳定全局标签下的 41 个指标字段,不添加 netns 或 interface 标签。命名空间和网卡的来源计数、增量、重置与过期处理仍用于异常判断,异常日志继续包含相应归属与样本;netflow 字段和语义不变。
| 用途 | 默认指标字段 |
|---|---|
| 采集能力与完整性(9) | diagnostics_enabled、connection_evidence_complete、tcp_socket_errno_available、tcp_queue_snapshots_available、tcp_retransmit_counter_available、listener_queue_snapshots_available、dns_capture_available、udp_collection_available、namespace_full_scan_available |
| 来源新鲜度与覆盖(4) | namespace_counter_sample_at_ns、namespace_candidates、namespace_samples、namespace_counter_limit_total |
| 上报队列(4) | diagnostic_queue_bytes、diagnostic_queue_oldest_age_seconds、netflow_pending_bytes、netflow_queue_oldest_age_seconds |
| 预算与输出损失(7) | group_admission_failures_total、samples_omitted_total、logs_limited_total、output_dropped_points_total、output_rejected_points_total、netflow_queue_full_points_total、netflow_rejected_points_total |
| 采集异常(3) | capture_loss_total、capture_unavailable_total、map_pressure_total |
| 连接失败(3) | connect_refused_total、connect_timeout_total、connect_unreachable_total |
| TCP 与监听压力(4) | tcp_retransmit_total、tcp_rto_total、listener_overflow_total、tcp_receive_drop_total |
| UDP、IP 与网卡(5) | udp_receive_error_total、udp_receive_buffer_total、ip_discard_total、icmp_unreachable_total、interface_error_total |
| DNS(2) | dns_no_response_total、dns_error_response_total |
connection_evidence_complete 描述节点侧连接证据的采集完整性,不表示平台已经入库。上报时,DataKit 接收失败会返回 HTTP 503,采集器保留已入队的数据重试;HTTP 200 表示 DataKit 接收成功,后续仍经过 Pipeline、过滤及发送链路。netflow_queue_full_points_total 统计入队失败而丢弃的点,这部分不会重试,需结合队列及输出损失的增量判断统计是否完整。
在单采集器实例、主机及全局标签稳定时,100、1000、10000 个节点分别约有 4100、41000、410000 条时间线,不随连接、进程、Pod 或命名空间数量增加;全局标签变化仍会生成新时间线。上报间隔影响数据点数量,不能消除不同标签组合产生的时间线。
需要逐来源连续指标时,设置 network_diagnostics_full_metrics = true,或独立运行时使用 --network-diagnostics-full-metrics / DKE_NETWORK_DIAGNOSTICS_FULL_METRICS=true。完整模式额外输出采集器字段与按 host + netns 区分的来源累计值、增量、来源时间和网卡计数,时间线数量会随命名空间数及重建增长。下面的完整指标定义包含可选字段,默认不会全部输出。
完整模式的网络栈与网卡指标使用小写下划线命名,明确协议和收发方向;*_total 表示内核来源累计值,*_delta 表示上一个指标窗口以来观测到的增量。首次采样或来源计数重置时,该次样本只建立新基线、不追加增量;同一指标窗口内此前已观测的增量仍保留。IPv4 使用 ipv4_*,IPv6 使用 ipv6_*、icmpv6_* 和 udpv6_*;icmp_*、udp_* 分别表示 ICMPv4 和 IPv4 UDP。计数可能重叠,不能相加作为唯一故障数或线路丢包率。
节点级 udp_no_port_total 表示本采集器会话中观测到的 udp_no_port 诊断增量累计值;完整模式下带 scope=namespace 的同名字段表示内核 Udp.NoPorts 累计值,可能包含采集器启动前的历史。两种 scope 是不同口径的证据视图,不应相加。
| 示例字段 | 含义 |
|---|---|
icmpv6_receive_destination_unreachable_delta |
窗口内收到的 ICMPv6 目标不可达消息数 |
icmpv6_receive_packet_too_big_delta |
窗口内收到的 ICMPv6 报文过大消息数,可辅助排查 MTU |
tcp_listen_overflows_delta |
窗口内内核监听队列溢出计数 |
udp_receive_buffer_errors_delta |
窗口内 IPv4 UDP 接收缓冲区错误计数 |
interface_receive_drops_delta |
窗口内该网卡的接收丢弃计数 |
TCP 的 retransmits、tcp_retransmit_total 及 tcp_retransmit 日志计数采用 socket 内核 total_retrans 的增量口径,包含 TLP;不再累加重传函数入口请求的段数。它们不是唯一丢包数、失败请求数或端到端丢包率;内核在完成重传记账后仍可能遇到本地发送错误。采集器首次通过 I/O 接管的既有连接以首次观测值为基线,避免把启动前的重传归到当前窗口。
tcp_retransmit_counter_available=false 表示无法获得经过验证的计数偏移,此时零值不能证明没有重传。无 BTF 时使用有界的私有 socket 自检验证该字段;自检只在启动偏移解析阶段运行。关闭网络诊断功能不影响 netflow 原有重传计数;此时无 BTF 的回退仅验证重传字段,不创建抓包 ring 或运行其他诊断自检。日志的 capabilities.tcp_retransmit_events 表示成功重传 tracepoint 是否可用;可用时在重传事件刷新队列快照,否则在后续重传尝试、I/O 或连接结束时补齐累计计数。查看样本时同时比较 at_ns 与 snapshot_at_ns,旧快照不能作为已恢复的证据。
诊断日志的每个连接样本还包含 retransmits_since_observed、tcp_retransmit_counter_available、socket_errno 和 socket_error_observed_at_ns。重传数是该 socket 自采集器接管以来的累计增量,不是该日志聚合窗口内的次数;可用性为 false 时零值不代表没有重传。socket 错误及其观测时间独立保留,避免在后续重传线索中与应用 I/O 返回码混淆。已关闭连接不再提供当前队列值,但仍保留已观测的重传、RTO 和 socket 错误。普通线索日志按服务及问题类型聚合,每组最多保留 3 个连接样本;严重异常的逐连接保留方式见下文。
ack_progress_observed_at_ns 是无进展判断的观测基准:ACK 推进、未确认队列清空或首次观测到新的未确认数据时更新。它不是首个丢包的精确时间;连接此前的空闲时间不计入发送停滞。
ACK 停滞只覆盖本端仍有未确认发送数据的情况。请求已经被对端 TCP 确认、但响应在返回路径丢失时,客户端可以同时表现为重传为 0、未确认量为 0、没有 RST 或 socket 错误。因此,这些值为 0 不能排除网络故障;connection_evidence_complete 也不表示已覆盖所有故障类型。httpflow 的 HTTP 事务不等于 MySQL 请求等待观测。排查此类数据库连接需要结合客户端请求/读超时、服务端日志,或对端的 TCP 重传证据。TCP_USER_TIMEOUT 约束未确认/未发送数据的传输等待,不能替代应用的响应读取超时。
已建立的 TCP 连接出现有效快照下的 ACK 停滞(仍有未确认量且至少 10 秒无进展)或观测到 socket/I/O 硬错误时,额外记录 problem_type=tcp_connection_issue。日志仍使用 ebpf_network_event/v1,以字段 connection_id 关联;连接 ID 不作为指标标签。每组只对应一条连接,samples 最多保留首次异常、最新关键变化、结束三个快照,分别用 phase=started/updated/closed 和 reason 说明。首次和结束在现有发送循环内尽快排队,中间变化最多每 5 分钟上报一次;同一 ACK 停滞不再重复输出服务级样本日志,其统计计数仍保留。occurrences 在此类日志中表示证据更新次数,重传数应读取样本的 retransmits_since_observed。
严重连接异常在首次观测时,还会从已有 PID→Pod 缓存提取 k8s_pod_name、k8s_namespace 和 k8s_workload_name,作为日志字段和 message.entity 内容保留到结束。缓存无法对应到该 PID 时留空;不使用后来复用该 IP 的 Pod 补写旧连接,也不为关联元数据增加文件或 namespace 扫描。这些字段不作为指标标签,不改变正常 netflow 的聚合维度。
event_status=closed 表示已观测到 TCP CLOSE 或 socket 最终回收,不一定已到应用关闭调用。first_seen_ns 是首次异常被观测的时间,不是首个丢包的精确时间;连接年龄也不是异常持续时间。只捕获到结束、采集覆盖出现缺口时标记 evidence_partial=true;单个超时错误不能区分重传耗尽、TCP_USER_TIMEOUT 或 TCP 保活 超时。沿用原有分组、内存、队列和日志预算,结束日志入队后释放组状态,非活动组沿用 30 分钟保留期。预算不足、上报拒绝或采集丢失时不保证每条异常都完整保留,应同时检查已有丢弃和完整性指标。正常连接和单次重传不启用该记录。
判断规则与证据范围¶
| 观测 | 输出与判断 | 排查方向 |
|---|---|---|
| connect 的最终错误、socket 错误、I/O 硬错误 | 按拒绝、超时、不可达、资源不足、权限、消息过大或 reset 分类;EINPROGRESS、EALREADY、EAGAIN、中断及内核重启返回码不算硬错误 |
对端服务是否监听、路由、防火墙、端口或内存资源;UDP EMSGSIZE 同时检查消息长度和路径 MTU |
| 连接超过 10 秒仍未建立,或成功建立耗时至少 3 秒 | connect_pending / connect_slow,信息级线索;应用先关闭的 connect_canceled 及最终结果不可用的 connect_unknown 仅保留计数和 netflow 结果 |
区分握手等待、应用取消和已确认的连接失败 |
| TCP 重传、RTO 定时器、零窗口探测、主动/收到 reset | 分别计数,满足下述收敛条件后保留日志样本,信息级线索 | 结合连接阶段、窗口、队列、SRTT 及两端数据判断;调用次数不能当作线路丢包数 |
| 有效且不超过 10 秒的队列快照 | 已建立或 CLOSE_WAIT 连接仍有未确认序列空间,至少 10 秒未观测到 ACK 推进时记录 tcp_ack_stall;未读数据至少 64 KiB 时计数为 tcp_receive_backlog,满足持续观测条件后记录日志 |
对端或路径响应、接收窗口、本地应用消费速度;快照可能包含 SYN/FIN 的序列空间 |
| SRTT 至少 200 ms;CLOSE_WAIT 至少 60 秒 | tcp_srtt_elevated / tcp_close_wait,每连接每分钟最多一次 |
RTT 是内核采样值;半关闭合法,不能直接判为网络故障 |
| 监听队列压力、网络栈丢弃、UDP 缓冲区错误、ICMP、网卡错误 | 监听快照归属监听端点;其余按命名空间或网卡输出增量 | 找到本地服务或网络栈压力。ListenDrops 与 ListenOverflows 等可能重叠,不能相加作为独立丢包量 |
| DNS 错误响应、响应至少 1 秒、6 秒未匹配到响应 | dns_error_response / dns_slow / dns_no_response,信息级线索;NXDOMAIN 本身不算故障 |
检查 DNS 服务、传输与解析结果。抓包丢失会清除待匹配项并记录采集损失,不制造 DNS 超时 |
| map 占用至少 90%、map 写入失败、probe/perf 丢失、发送积压 | map_pressure、capture_loss、output_backlog,另有累计指标 |
先判断采集证据是否完整;map 压力和唤醒通知丢失本身不等于流量数据已丢失 |
普通连接线索在生成日志前会进一步收敛,原始累计计数和 netflow 数据继续保留:
| 日志线索 | 出日志的条件 |
|---|---|
tcp_retransmit、tcp_rto、tcp_zero_window_probe |
同组从首个新观测开始的 60 秒计数窗口内至少 5 个计数单位(重传为内核段数增量,RTO/零窗口探测为调用次数);或当前有效快照显示握手等待至少 10 秒且 RTO 至少 3 次,或已建立/CLOSE_WAIT 连接仍有未确认量且至少 10 秒没有 ACK 推进 |
tcp_active_reset、tcp_received_reset |
同组 60 秒计数窗口内至少 10 次;次数门槛只是保留排障样本,不等于确认故障 |
listener_queue_full |
同组 60 秒计数窗口内至少 3 次队列压力观测;不等于确认 SYN 丢弃 |
tcp_receive_backlog、tcp_srtt_elevated |
同一代表连接的有效快照反复达到阈值,观测跨度至少 30 秒,相邻观测间隔不超过 90 秒;单次接收突发或 RTT 抬升仅计数 |
connect_pending、connect_slow、tcp_ack_stall、tcp_close_wait |
达到上述持续握手、建立耗时、ACK 停滞或半关闭规则即保留日志 |
connect_canceled、connect_unknown |
单独的应用取消或未知结果只保留计数和 netflow,不生成问题日志 |
普通连接线索的首次合格观测通常在 5 秒内进入发送队列,同组后续合格观测最多每 5 分钟汇总一次。明确连接/I/O 错误、网络栈丢弃、DNS 和采集质量事件沿用每分钟更新。收敛不会把 info 自动升级成告警,也不会改变指标字段、netflow 字段或原始观测次数。occurrences 是自上次上报以来的观测增量,可包含此前未达到日志门槛的观测;它不是唯一连接数。输出预算耗尽时,已经合格的证据继续等待发送,未合格线索不会在退出时强制转为日志。
这些阈值是保守的初始排障规则,信息级线索不自动升级为故障告警。连接和 I/O 返回码遵循 connect(2);UDP 异步错误可能来自之前的数据报,因此日志仅归属 socket/进程并省略无法确认的远端,参见 udp(7)。计数器重叠及 offload 的影响参见 Linux SNMP counters。零窗口、半关闭和序列空间参见 TCP RFC 9293;DNS 返回码参见 RFC 1035。
数据量、保留与可靠性¶
异常按问题类型及进程/服务/命名空间/服务端点归组,每组最多保留 3 个代表样本,不采集业务报文载荷;明文 DNS 诊断可能包含查询名。日志同时受编码字节与条数预算限制:默认持续 512 KiB/分钟、240 条/分钟,任一额度不足都会暂缓;允许最多 2 MiB、960 条的短时突发。单条编码体积最多 16 KiB,超过上限计入 output_rejected_points_total,同一组相同累计观测不会重复编码拒绝,新观测可再次尝试。首次连接和问题事实仍按原有计数保留。每节点持续运行的采集器全天入队的新日志,持续额度分别最多约 720 MiB、345600 条,另加初始突发、重试和传输开销;两项上限不保证同时打满。这个预算不包含原有 netflow、新增 netflow 字段和诊断指标,无法仅据此估算全部网络采集量。
问题分组记账预算 16 MiB,发送队列预算 8 MiB / 2048 批;来源计数表最多 16384 个键。分组和队列按实际观测及待发送数据增长,不在启动时预分配全部预算。内核 TCP 故障数据与现有连接按同一生命周期保存,每条增加 144 字节的值及 map 键/管理开销。这些预算不等于进程总 RSS 上限。指标不使用 PID、连接 ID 或远端地址作为标签;默认不输出命名空间或网卡指标点,完整模式下无错误增量的网卡也不单独上报点。
监听端点在 30 分钟无新 SYN 观测后释放本地槽位,清理次数见 listener_idle_evictions_total;监听快照是采样线索,清理与新 SYN 并发时可能遗漏边界样本。无新观测的问题组和过期来源计数在 30 分钟后清理,重启后不回放,也不要求持久化目录。平台日志的保留时长由平台存储策略决定。安静不表示恢复:仅已确认的 connect 硬错误,在同一分组连续两个完整的安静窗口都有成功连接时输出 recovered;其他线索只报告观测及持续更新。
超限、队列满、采样省略、永久拒收和退出时未送达均有显式计数。仅新日志成功进入发送队列后扣减字节与条数额度;队列拒绝不消耗额度,指标不使用日志额度。暂时发送失败保留同一编码批次及 event_id 重试,不重复扣减额度;平台可用 collector_session + event_id 去重,但不保证端到端 exactly-once。发送阻塞时首条日志会延后。
使用顺序与能力边界¶
场景化排障、查询示例与实测数据见 eBPF 网络排障最佳实践。其中分别介绍连接与监听、TCP 传输、DNS/UDP 和采集可信度的判断及验证方法。
并发写入导致快照暂缓读取时,flow_snapshot_deferred_total 增加且该窗口的连接证据标为不完整,累计流量等待下一次稳定读取;这不表示已丢失。
- 先检查
ebpf_network的connection_evidence_complete、*_available、namespace_counter_sample_at_ns、覆盖及采集损失计数;完整模式还提供逐来源sample_at_ns/sample_age_seconds。缺失能力或过期数据不能解释为“没有问题”。 - 用 netflow 的流量、连接结果及实际观测角色缩小进程/服务/对端范围。启动前已存在的连接可能缺少角色证据,此时
direction和conn_side为unknown,聚合保留地址对流量总量,原始端口可从连接明细或本地诊断查询。 - 按时间、
problem_type、scope检索ebpf_network_event,读取message中的ebpf_network_event/v1、entity、samples、evidence_partial和能力字段。PID、端点属于字段,监听端点及命名空间事件不虚构进程归属。 - 结合两端和网络栈证据定位到连接阶段、服务队列、本地消费、DNS 或采集链路;单端数据通常不能确定具体哪一跳丢包。
socket errno、TCP 队列、发送窗口和监听队列优先采用验证过的内核 BTF 字段;缺少 BTF 或字段时,启动时通过本地 TCP socket 自检验证偏移。自检的 socket 等待与采样最多持续 8 秒,临时探针和抓包 socket 随后释放,不增加运行期采样频率。验证失败、有歧义、缺少所需内核函数或 loopback 抓包权限时,对应可选能力显式为 false,流量、连接结果、I/O 错误及网络栈计数继续采集。终止状态的队列快照标为不可用,socket 错误和 reset 证据继续保留。
命名空间来源仅覆盖已观测连接的候选命名空间及采集器自身,每 10 秒最多轮转 128 个命名空间、每个最多 32 个网卡;覆盖数量单独上报。加密 DNS、抓包不可达的命名空间和两次扫描间的短暂队列压力不能保证被捕获。
网络诊断指标¶
ebpf_network¶
| Tags & Fields | Description |
|---|---|
| host ( tag) |
System hostname |
| interface ( tag) |
Device name within netns, only present on full-mode interface points |
| netns ( tag) |
Observed namespace in full metric mode; absent on node points |
| scope ( tag) |
Namespace or interface scope in full metric mode; absent on node points |
| capture_loss_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| capture_unavailable_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_canceled_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_failed_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_pending_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_permission_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_refused_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_resource_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_slow_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_success_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_timeout_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_unknown_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connect_unreachable_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| connection_evidence_complete | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: N/A |
| diagnostic_admission_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| diagnostic_log_bytes_per_minute | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: N/A |
| diagnostic_log_points_per_minute | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: count |
| diagnostic_max_groups | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: count |
| diagnostic_queue_bytes | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: digital,B |
| diagnostic_queue_oldest_age_seconds | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: time,s |
| diagnostic_state_bytes | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: digital,B |
| diagnostics_enabled | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| dns_answer_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| dns_capture_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| dns_capture_loss_epoch_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| dns_error_response_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| dns_no_response_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| dns_query_unknown_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| dns_slow_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| early_connect_failures_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| flow_admission_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| flow_snapshot_deferred_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| flow_snapshot_scan_errors_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| group_admission_failures_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| icmp_packet_too_big_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| icmp_receive_destination_unreachable_delta | ICMPv4 destination-unreachable messages received (Linux source: Icmp.InDestUnreachs). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| icmp_receive_destination_unreachable_total | ICMPv4 destination-unreachable messages received (Linux source: Icmp.InDestUnreachs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| icmp_receive_errors_delta | ICMPv4 receive errors (Linux source: Icmp.InErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| icmp_receive_errors_total | ICMPv4 receive errors (Linux source: Icmp.InErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| icmp_receive_time_exceeded_delta | ICMPv4 time-exceeded messages received (Linux source: Icmp.InTimeExcds). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| icmp_receive_time_exceeded_total | ICMPv4 time-exceeded messages received (Linux source: Icmp.InTimeExcds). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| icmp_send_destination_unreachable_delta | ICMPv4 destination-unreachable messages sent (Linux source: Icmp.OutDestUnreachs). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| icmp_send_destination_unreachable_total | ICMPv4 destination-unreachable messages sent (Linux source: Icmp.OutDestUnreachs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| icmp_time_exceeded_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| icmp_unreachable_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| icmpv6_receive_destination_unreachable_delta | ICMPv6 destination-unreachable messages received (Linux source: Icmp6.InDestUnreachs). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| icmpv6_receive_destination_unreachable_total | ICMPv6 destination-unreachable messages received (Linux source: Icmp6.InDestUnreachs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| icmpv6_receive_packet_too_big_delta | ICMPv6 packet-too-big messages received (Linux source: Icmp6.InPktTooBigs). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| icmpv6_receive_packet_too_big_total | ICMPv6 packet-too-big messages received (Linux source: Icmp6.InPktTooBigs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| icmpv6_receive_time_exceeded_delta | ICMPv6 time-exceeded messages received (Linux source: Icmp6.InTimeExcds). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| icmpv6_receive_time_exceeded_total | ICMPv6 time-exceeded messages received (Linux source: Icmp6.InTimeExcds). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| icmpv6_send_destination_unreachable_delta | ICMPv6 destination-unreachable messages sent (Linux source: Icmp6.OutDestUnreachs). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| icmpv6_send_destination_unreachable_total | ICMPv6 destination-unreachable messages sent (Linux source: Icmp6.OutDestUnreachs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| icmpv6_send_packet_too_big_delta | ICMPv6 packet-too-big messages sent (Linux source: Icmp6.OutPktTooBigs). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| icmpv6_send_packet_too_big_total | ICMPv6 packet-too-big messages sent (Linux source: Icmp6.OutPktTooBigs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| interface_counters_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| interface_error_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| interface_full_scan_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| interface_namespaces_sampled | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: count |
| interface_receive_drops_delta | Network interface receive drops (Linux source: rx_dropped). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| interface_receive_drops_total | Network interface receive drops (Linux source: rx_dropped). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| interface_receive_errors_delta | Network interface receive errors (Linux source: rx_errors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| interface_receive_errors_total | Network interface receive errors (Linux source: rx_errors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| interface_send_drops_delta | Network interface transmit drops (Linux source: tx_dropped). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| interface_send_drops_total | Network interface transmit drops (Linux source: tx_dropped). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| interface_send_errors_delta | Network interface transmit errors (Linux source: tx_errors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| interface_send_errors_total | Network interface transmit errors (Linux source: tx_errors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| interfaces_expected | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: count |
| interfaces_sampled | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: count |
| io_correlation_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| io_message_size_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| io_other_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| io_permission_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| io_refused_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| io_reset_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| io_resource_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| io_timeout_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| io_unreachable_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| ip_discard_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| ipv4_receive_discards_delta | IPv4 packets discarded in the receive path (Linux source: Ip.InDiscards). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| ipv4_receive_discards_total | IPv4 packets discarded in the receive path (Linux source: Ip.InDiscards). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| ipv4_send_discards_delta | IPv4 packets discarded in the send path (Linux source: Ip.OutDiscards). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| ipv4_send_discards_total | IPv4 packets discarded in the send path (Linux source: Ip.OutDiscards). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| ipv6_receive_discards_delta | IPv6 packets discarded in the receive path (Linux source: Ip6.InDiscards). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| ipv6_receive_discards_total | IPv6 packets discarded in the receive path (Linux source: Ip6.InDiscards). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| ipv6_send_discards_delta | IPv6 packets discarded in the send path (Linux source: Ip6.OutDiscards). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| ipv6_send_discards_total | IPv6 packets discarded in the send path (Linux source: Ip6.OutDiscards). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| kprobe_profile_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| lifecycle_lost_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| lifecycle_read_errors_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| listener_child_failures_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| listener_drop_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| listener_idle_evictions_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| listener_overflow_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| listener_queue_full_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| listener_queue_snapshots_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| listener_syn_requests_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| logs_limited_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| map_pressure_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| namespace_candidates | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: count |
| namespace_counter_limit_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| namespace_counter_sample_at_ns | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: timeStamp,nsec |
| namespace_full_scan_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| namespace_samples | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: count |
| netflow_pending_batches | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: count |
| netflow_pending_bytes | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: digital,B |
| netflow_queue_full_points_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| netflow_queue_oldest_age_seconds | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: time,s |
| netflow_rejected_points_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| netflow_shutdown_frames_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| netflow_shutdown_points_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| notification_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| output_backlog_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| output_dropped_batches_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| output_dropped_points_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| output_rejected_points_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| problem_groups | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: count |
| sample_age_seconds | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: time,s |
| sample_at_ns | Diagnostic state, output queue, or namespace coverage. Names ending in bytes use bytes; age_seconds uses seconds. Type: int | (gauge) Unit: timeStamp,nsec |
| samples_omitted_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| socket_admission_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_abort_close_delta | TCP connections reset on close with unread data (Linux source: TcpExt.TCPAbortOnClose). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_abort_close_total | TCP connections reset on close with unread data (Linux source: TcpExt.TCPAbortOnClose). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_abort_data_delta | TCP connections reset by an immediate abortive close (Linux source: TcpExt.TCPAbortOnData). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_abort_data_total | TCP connections reset by an immediate abortive close (Linux source: TcpExt.TCPAbortOnData). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_abort_memory_delta | TCP connections aborted under memory pressure (Linux source: TcpExt.TCPAbortOnMemory). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_abort_memory_total | TCP connections aborted under memory pressure (Linux source: TcpExt.TCPAbortOnMemory). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_abort_timeout_delta | TCP connections aborted on timeout (Linux source: TcpExt.TCPAbortOnTimeout). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_abort_timeout_total | TCP connections aborted on timeout (Linux source: TcpExt.TCPAbortOnTimeout). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_ack_stall_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_active_reset_calls_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_active_reset_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_backlog_drops_delta | Packets dropped from the TCP socket backlog (Linux source: TcpExt.TCPBacklogDrop). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_backlog_drops_total | Packets dropped from the TCP socket backlog (Linux source: TcpExt.TCPBacklogDrop). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_close_wait_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_lifecycle_cpu_coverage_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_lifecycle_lost_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| tcp_lifecycle_read_errors_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| tcp_listen_drops_delta | Packets dropped by listening TCP sockets (Linux source: TcpExt.ListenDrops). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_listen_drops_total | Packets dropped by listening TCP sockets (Linux source: TcpExt.ListenDrops). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_listen_overflows_delta | TCP listen queue overflows (Linux source: TcpExt.ListenOverflows). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_listen_overflows_total | TCP listen queue overflows (Linux source: TcpExt.ListenOverflows). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_memory_abort_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_original_data_segments_sent_delta | TCP segments sent with original data, excluding retransmissions (Linux source: TcpExt.TCPOrigDataSent). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_original_data_segments_sent_total | TCP segments sent with original data, excluding retransmissions (Linux source: TcpExt.TCPOrigDataSent). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_probe0_calls_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_queue_snapshots_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_receive_backlog_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_receive_drop_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_receive_errors_delta | TCP receive errors, including malformed headers or checksum errors (Linux source: Tcp.InErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_receive_errors_total | TCP receive errors, including malformed headers or checksum errors (Linux source: Tcp.InErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_receive_queue_drops_delta | Packets dropped from the TCP receive queue (Linux source: TcpExt.TCPRcvQDrop). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_receive_queue_drops_total | Packets dropped from the TCP receive queue (Linux source: TcpExt.TCPRcvQDrop). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_receive_segments_delta | TCP segments received; offload can affect segment counts (Linux source: Tcp.InSegs). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_receive_segments_total | TCP segments received; offload can affect segment counts (Linux source: Tcp.InSegs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_received_reset_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_received_resets_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_retransmit_counter_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_retransmit_events_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_retransmit_failures_delta | Failed TCP retransmission attempts (Linux source: TcpExt.TCPRetransFail). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_retransmit_failures_total | Failed TCP retransmission attempts (Linux source: TcpExt.TCPRetransFail). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_retransmit_segments_delta | TCP segments retransmitted (Linux source: Tcp.RetransSegs). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_retransmit_segments_total | TCP segments retransmitted (Linux source: Tcp.RetransSegs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_retransmit_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_rto_timer_fires_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_rto_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_send_segments_delta | TCP segments sent, excluding retransmitted segments (Linux source: Tcp.OutSegs). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_send_segments_total | TCP segments sent, excluding retransmitted segments (Linux source: Tcp.OutSegs). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_socket_destroy_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_socket_errno_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_socket_error_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_spurious_retransmits_host_queue_delta | Spurious TCP retransmissions caused by data retained in host queues (Linux source: TcpExt.TCPSpuriousRtxHostQueues). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_spurious_retransmits_host_queue_total | Spurious TCP retransmissions caused by data retained in host queues (Linux source: TcpExt.TCPSpuriousRtxHostQueues). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_srtt_elevated_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| tcp_state_changes_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| tcp_syn_cookies_failed_delta | Invalid TCP SYN cookies received (Linux source: TcpExt.SyncookiesFailed). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_syn_cookies_failed_total | Invalid TCP SYN cookies received (Linux source: TcpExt.SyncookiesFailed). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_syn_cookies_sent_delta | TCP SYN cookies sent (Linux source: TcpExt.SyncookiesSent). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_syn_cookies_sent_total | TCP SYN cookies sent (Linux source: TcpExt.SyncookiesSent). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_syn_queue_full_cookies_delta | TCP SYN queue full events handled with SYN cookies (Linux source: TcpExt.TCPReqQFullDoCookies). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_syn_queue_full_cookies_total | TCP SYN queue full events handled with SYN cookies (Linux source: TcpExt.TCPReqQFullDoCookies). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_syn_queue_full_drops_delta | TCP SYN queue full events that dropped a request (Linux source: TcpExt.TCPReqQFullDrop). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_syn_queue_full_drops_total | TCP SYN queue full events that dropped a request (Linux source: TcpExt.TCPReqQFullDrop). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_syn_retransmits_delta | TCP SYN or SYN-ACK retransmissions (Linux source: TcpExt.TCPSynRetrans). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| tcp_syn_retransmits_total | TCP SYN or SYN-ACK retransmissions (Linux source: TcpExt.TCPSynRetrans). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| tcp_zero_window_probe_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp6_counters_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| udp_collection_available | Availability of this evidence channel; missing or false does not mean no network faults. Type: bool | (gauge) Unit: bool |
| udp_errors_message_size_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_errors_other_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_errors_permission_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_errors_refused_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_errors_reset_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_errors_resource_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_errors_timeout_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_errors_unreachable_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_errqueue_reads_total | Cumulative observed collector count since this collector session began. Type: int | (count) Unit: count |
| udp_lite_no_port_delta | UDP-Lite datagrams received for a port without a listener (Linux source: UdpLite.NoPorts). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udp_lite_no_port_total | UDP-Lite datagrams received for a port without a listener (Linux source: UdpLite.NoPorts). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| udp_lite_receive_buffer_errors_delta | UDP-Lite receive buffer errors (Linux source: UdpLite.RcvbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udp_lite_receive_buffer_errors_total | UDP-Lite receive buffer errors (Linux source: UdpLite.RcvbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| udp_lite_receive_errors_delta | UDP-Lite receive errors (Linux source: UdpLite.InErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udp_lite_receive_errors_total | UDP-Lite receive errors (Linux source: UdpLite.InErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| udp_lite_send_buffer_errors_delta | UDP-Lite send buffer errors (Linux source: UdpLite.SndbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udp_lite_send_buffer_errors_total | UDP-Lite send buffer errors (Linux source: UdpLite.SndbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| udp_no_port_delta | IPv4 UDP datagrams received for a port without a listener (Linux source: Udp.NoPorts). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udp_no_port_total | Node scope: cumulative observed udp_no_port diagnostic increments during this collector session. Namespace scope (full metrics): Linux Udp.NoPorts cumulative count, including pre-attachment history. These scopes are different views of evidence, not independent faults; do not sum them. Type: int | (count) Unit: count |
| udp_receive_buffer_errors_delta | IPv4 UDP receive buffer errors (Linux source: Udp.RcvbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udp_receive_buffer_errors_total | IPv4 UDP receive buffer errors (Linux source: Udp.RcvbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| udp_receive_buffer_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_receive_error_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_receive_errors_delta | IPv4 UDP receive errors; overlaps with buffer errors (Linux source: Udp.InErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udp_receive_errors_total | IPv4 UDP receive errors; overlaps with buffer errors (Linux source: Udp.InErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| udp_send_buffer_errors_delta | IPv4 UDP send buffer errors (Linux source: Udp.SndbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udp_send_buffer_errors_total | IPv4 UDP send buffer errors (Linux source: Udp.SndbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| udp_send_buffer_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_tuple_total | Cumulative observed evidence count since collector start; hook calls and overlapping error counters are not a wire loss rate or unique connections. Type: int | (count) Unit: count |
| udp_unbound_io_errors_total | UDP negative I/O returns without a complete tuple, including would-block/interrupted returns; not a hard-error or datagram-loss count. Type: int | (count) Unit: count |
| udpv6_no_port_delta | IPv6 UDP datagrams received for a port without a listener (Linux source: Udp6.NoPorts). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udpv6_no_port_total | IPv6 UDP datagrams received for a port without a listener (Linux source: Udp6.NoPorts). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| udpv6_receive_buffer_errors_delta | IPv6 UDP receive buffer errors (Linux source: Udp6.RcvbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udpv6_receive_buffer_errors_total | IPv6 UDP receive buffer errors (Linux source: Udp6.RcvbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| udpv6_receive_errors_delta | IPv6 UDP receive errors; overlaps with buffer errors (Linux source: Udp6.InErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udpv6_receive_errors_total | IPv6 UDP receive errors; overlaps with buffer errors (Linux source: Udp6.InErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
| udpv6_send_buffer_errors_delta | IPv6 UDP send buffer errors (Linux source: Udp6.SndbufErrors). Observed increments since the preceding metric window; initial or reset source values establish a baseline. Type: int | (gauge) Unit: count |
| udpv6_send_buffer_errors_total | IPv6 UDP send buffer errors (Linux source: Udp6.SndbufErrors). Kernel cumulative count; may include history before attachment and overlap with other counters. Check sample_at_ns and sample_age_seconds. Type: int | (count) Unit: count |
网络聚合数据¶
以下所有数据采集,默认会追加名为 host 的全局 tag(tag 值为 DataKit 所在主机名),也可以在配置中通过 [inputs.ebpf.tags] 指定其它标签:
netflow¶
| Tags & Fields | Description |
|---|---|
| client_ip ( tag) |
Client IP address |
| client_ip_type ( tag) |
Client IP type. (other/private/multicast) |
| client_k8s_deployment_name ( tag) |
Client K8s deployment name |
| client_k8s_namespace ( tag) |
Client K8s namespace |
| client_k8s_pod_name ( tag) |
Client K8s pod name |
| client_k8s_service_name ( tag) |
Client K8s service name |
| client_port ( tag) |
Client port, rolled up to * in aggregates; omitted when the connection role is unknown |
| conn_side ( tag) |
The side of the connection: client/server/unknown |
| direction ( tag) |
Use the source (src_ip:src_port) as a frame of reference to identify the connection initiator. (incoming/outgoing/unknown; unknown TCP roles are not inferred from port magnitude) |
| dst_domain ( tag) |
Destination domain |
| dst_ip ( tag) |
Destination IP address |
| dst_ip_type ( tag) |
Destination IP type. (other/private/multicast) |
| dst_k8s_deployment_name ( tag) |
Destination K8s deployment name |
| dst_k8s_namespace ( tag) |
Destination K8s namespace |
| dst_k8s_pod_name ( tag) |
Destination K8s pod name |
| dst_k8s_service_name ( tag) |
Destination K8s service name |
| dst_nat_ip ( tag) |
For data containing the outgoing tag, this value is the ip after the DNAT operation |
| dst_nat_port ( tag) |
For data containing the outgoing tag, this value is the port after the DNAT operation |
| dst_port ( tag) |
Destination port; * when it is the client port or the connection role is unknown |
| family ( tag) |
Network layer protocol. (IPv4/IPv6) |
| host ( tag) |
System hostname |
| pid ( tag) |
Process identification number |
| process_name ( tag) |
Process name |
| server_domain ( tag) |
Server domain |
| server_ip ( tag) |
Server IP |
| server_ip_type ( tag) |
Server IP type. (other/private/multicast) |
| server_k8s_deployment_name ( tag) |
Server K8s deployment name |
| server_k8s_namespace ( tag) |
Server K8s namespace |
| server_k8s_pod_name ( tag) |
Server K8s pod name |
| server_k8s_service_name ( tag) |
Server K8s service name |
| server_port ( tag) |
Server port; omitted when the connection role is unknown |
| source ( tag) |
Fixed value: netflow. |
| src_ip ( tag) |
Source IP |
| src_ip_type ( tag) |
Source IP type. (other/private/multicast) |
| src_k8s_deployment_name ( tag) |
Source K8s deployment name |
| src_k8s_namespace ( tag) |
Source K8s namespace |
| src_k8s_pod_name ( tag) |
Source K8s pod name |
| src_k8s_service_name ( tag) |
Source K8s service name |
| src_port ( tag) |
Source port; * when it is the client port or the connection role is unknown |
| sub_source ( tag) |
Some specific connection classifications, such as the sub_source value for Kubernetes network traffic is K8s |
| transport ( tag) |
Transport layer protocol. (udp/tcp) |
| bytes_read | The number of bytes read Type: int | (gauge) Unit: digital,B |
| bytes_written | The number of bytes written Type: int | (gauge) Unit: digital,B |
| io_errors_message_size | Interval count of hard TCP I/O errors classified as message_size. Nonblocking and interrupted I/O are excluded. Type: int | (gauge) Unit: count |
| io_errors_other | Interval count of hard TCP I/O errors classified as other. Nonblocking and interrupted I/O are excluded. Type: int | (gauge) Unit: count |
| io_errors_permission | Interval count of hard TCP I/O errors classified as permission. Nonblocking and interrupted I/O are excluded. Type: int | (gauge) Unit: count |
| io_errors_refused | Interval count of hard TCP I/O errors classified as refused. Nonblocking and interrupted I/O are excluded. Type: int | (gauge) Unit: count |
| io_errors_reset | Interval count of hard TCP I/O errors classified as reset. Nonblocking and interrupted I/O are excluded. Type: int | (gauge) Unit: count |
| io_errors_resource | Interval count of hard TCP I/O errors classified as resource. Nonblocking and interrupted I/O are excluded. Type: int | (gauge) Unit: count |
| io_errors_timeout | Interval count of hard TCP I/O errors classified as timeout. Nonblocking and interrupted I/O are excluded. Type: int | (gauge) Unit: count |
| io_errors_unreachable | Interval count of hard TCP I/O errors classified as unreachable. Nonblocking and interrupted I/O are excluded. Type: int | (gauge) Unit: count |
| io_recv_errors | Interval count of hard TCP receive errors. Type: int | (gauge) Unit: count |
| io_send_errors | Interval count of hard TCP send errors. Type: int | (gauge) Unit: count |
| packets_read | The number of packets read Type: int | (gauge) Unit: count |
| packets_written | The number of packets written Type: int | (gauge) Unit: count |
| retransmits | Increments of the socket kernel total_retrans counter, including TLP; not a wire packet-loss count Type: int | (gauge) Unit: count |
| rtt | TCP Latency Type: int | (gauge) Unit: time,μs |
| rtt_var | TCP Jitter Type: int | (gauge) Unit: time,μs |
| tcp_active_reset_calls | Interval count of observed kernel hook invocations; not proof of wire delivery. Availability is reported in ebpf_network. Type: int | (gauge) Unit: count |
| tcp_close_wait | The number of TCP_CLOSE_WAIT state transitions Type: int | (gauge) Unit: count |
| tcp_closed | The number of TCP connection closed Type: int | (gauge) Unit: count |
| tcp_connect_attempts | The number of TCP connection attempts Type: int | (gauge) Unit: count |
| tcp_connect_failures | The number of TCP connection failures Type: int | (gauge) Unit: count |
| tcp_connect_result_canceled | Outgoing connection outcomes completed in this interval: canceled. Pending attempts are excluded. Type: int | (gauge) Unit: count |
| tcp_connect_result_failure | Outgoing connection outcomes completed in this interval: failure. Pending attempts are excluded. Type: int | (gauge) Unit: count |
| tcp_connect_result_success | Outgoing connection outcomes completed in this interval: success. Pending attempts are excluded. Type: int | (gauge) Unit: count |
| tcp_connect_result_unknown | Outgoing connection outcomes completed in this interval: unknown. Pending attempts are excluded. Type: int | (gauge) Unit: count |
| tcp_connect_success_duration_count | Completed outgoing handshakes with a known start timestamp. Type: int | (gauge) Unit: count |
| tcp_connect_success_duration_max_us | Completed outgoing handshake duration in microseconds: maximum. Type: int | (gauge) Unit: time,μs |
| tcp_connect_success_duration_sum_us | Completed outgoing handshake duration in microseconds: sum. Type: int | (gauge) Unit: time,μs |
| tcp_established | The number of TCP connection established Type: int | (gauge) Unit: count |
| tcp_last_ack | The number of TCP_LAST_ACK state transitions Type: int | (gauge) Unit: count |
| tcp_probe0_calls | Interval count of observed kernel hook invocations; not proof of wire delivery. Availability is reported in ebpf_network. Type: int | (gauge) Unit: count |
| tcp_resets_received | Interval count of observed kernel hook invocations; not proof of wire delivery. Availability is reported in ebpf_network. Type: int | (gauge) Unit: count |
| tcp_role_observed | TCP roles first observed in this interval: client, server, unknown, or mixed. Omitted without new observations; not a connection count. TCP aggregation keeps an unknown direction until role evidence is available. Type: string | (string) Unit: N/A |
| tcp_rto_timer_fires | Interval count of observed kernel hook invocations; not proof of wire delivery. Availability is reported in ebpf_network. Type: int | (gauge) Unit: count |
| tcp_socket_errors | Newly observed terminal kernel socket errors in this interval. Type: int | (gauge) Unit: count |
| tcp_srtt_snapshot_count | Collected kernel SRTT snapshots; repeated userspace map reads are excluded. Type: int | (gauge) Unit: count |
| tcp_srtt_snapshot_max_us | Maximum sampled kernel SRTT; not the true RTT peak or application latency. Type: int | (gauge) Unit: time,μs |
| tcp_time_wait | The number of TCP_TIME_WAIT state transitions Type: int | (gauge) Unit: count |
dnsflow¶
| Tags & Fields | Description |
|---|---|
| aggregation_scope ( tag) |
service: routine queries grouped by client namespace and workload; endpoint: routine queries with no reliable workload attribution; query: errors, slow queries and timeouts retaining client and domain detail. Counts across scopes are disjoint. |
| client_ip ( tag) |
DNS client IP; * for a workload summary spanning replicas. |
| client_k8s_namespace ( tag) |
Client Kubernetes namespace; part of the workload summary identity. |
| client_k8s_pod_name ( tag) |
Client Pod name when available for query detail; * for a workload summary. |
| client_k8s_workload_name ( tag) |
Client workload name for service summaries. |
| client_k8s_workload_type ( tag) |
Client workload kind; distinguishes same-named workloads of different kinds. |
| client_port ( tag) |
DNS client port; ephemeral ports are aggregated as *. |
| direction ( tag) |
Use the source (src_ip:src_port) as a frame of reference to identify the connection initiator. (incoming/outgoing) |
| dst_domain ( tag) |
Destination domain |
| dst_ip ( tag) |
Destination IP address; * when the destination represents multiple client replicas |
| dst_ip_type ( tag) |
Destination IP type. (other/private/multicast) |
| dst_k8s_deployment_name ( tag) |
Destination K8s deployment name |
| dst_k8s_namespace ( tag) |
Destination K8s namespace |
| dst_k8s_pod_name ( tag) |
Destination K8s pod name |
| dst_k8s_service_name ( tag) |
Destination K8s service name |
| dst_port ( tag) |
Destination port |
| family ( tag) |
Network layer protocol. (IPv4/IPv6) |
| host ( tag) |
System hostname |
| query_type ( tag) |
DNS query type. (A/AAAA/NS/...) |
| server_ip ( tag) |
DNS server IP, retained in routine summaries to distinguish resolvers. |
| server_port ( tag) |
DNS server port. |
| source ( tag) |
Fixed value: dnsflow. |
| src_ip ( tag) |
Source IP; * when the source represents multiple client replicas |
| src_ip_type ( tag) |
Source IP type. (other/private/multicast) |
| src_k8s_deployment_name ( tag) |
Source K8s deployment name |
| src_k8s_namespace ( tag) |
Source K8s namespace |
| src_k8s_pod_name ( tag) |
Source K8s pod name |
| src_k8s_service_name ( tag) |
Source K8s service name |
| src_port ( tag) |
Source port |
| sub_source ( tag) |
Some specific connection classifications, such as the sub_source value for Kubernetes network traffic is K8s |
| transport ( tag) |
Transport layer protocol. (udp/tcp) |
| count | The number of DNS requests in a collection cycle Type: int | (gauge) Unit: count |
| latency | Average response time for DNS requests Type: int | (gauge) Unit: time,ns |
| latency_max | Maximum response time for DNS requests Type: int | (gauge) Unit: time,ns |
| query_domain | The queried domain for errors, slow queries and timeouts; * for routine query summariesType: string | (string) Unit: N/A |
| rcode | DNS response code: 0 - NoError, 1 - FormErr, 2 - ServFail, 3 - NXDomain, 4 - NotImp, 5 - Refused, ...; A value of -1 means the request timed outType: int | (gauge) Unit: N/A |
httpflow¶
| Tags & Fields | Description |
|---|---|
| client_ip ( tag) |
Client IP address |
| client_ip_type ( tag) |
Client IP type. (other/private/multicast) |
| client_k8s_deployment_name ( tag) |
Client K8s deployment name |
| client_k8s_namespace ( tag) |
Client K8s namespace |
| client_k8s_pod_name ( tag) |
Client K8s pod name |
| client_k8s_service_name ( tag) |
Client K8s service name |
| client_port ( tag) |
Client port, rolled up to * in aggregates; omitted when the connection role is unknown |
| conn_side ( tag) |
The side of the connection: client/server/unknown |
| direction ( tag) |
Use the source (src_ip:src_port) as a frame of reference to identify the connection initiator. (incoming/outgoing/unknown; unknown TCP roles are not inferred from port magnitude) |
| dst_domain ( tag) |
Destination domain |
| dst_ip ( tag) |
Destination IP address |
| dst_ip_type ( tag) |
Destination IP type. (other/private/multicast) |
| dst_k8s_deployment_name ( tag) |
Destination K8s deployment name |
| dst_k8s_namespace ( tag) |
Destination K8s namespace |
| dst_k8s_pod_name ( tag) |
Destination K8s pod name |
| dst_k8s_service_name ( tag) |
Destination K8s service name |
| dst_nat_ip ( tag) |
For data containing the outgoing tag, this value is the ip after the DNAT operation |
| dst_nat_port ( tag) |
For data containing the outgoing tag, this value is the port after the DNAT operation |
| dst_port ( tag) |
Destination port; * when it is the client port or the connection role is unknown |
| family ( tag) |
Network layer protocol. (IPv4/IPv6) |
| host ( tag) |
System hostname |
| pid ( tag) |
Process identification number |
| process_name ( tag) |
Process name |
| server_domain ( tag) |
Server domain |
| server_ip ( tag) |
Server IP |
| server_ip_type ( tag) |
Server IP type. (other/private/multicast) |
| server_k8s_deployment_name ( tag) |
Server K8s deployment name |
| server_k8s_namespace ( tag) |
Server K8s namespace |
| server_k8s_pod_name ( tag) |
Server K8s pod name |
| server_k8s_service_name ( tag) |
Server K8s service name |
| server_port ( tag) |
Server port; omitted when the connection role is unknown |
| source ( tag) |
Fixed value: httpflow. |
| src_ip ( tag) |
Source IP |
| src_ip_type ( tag) |
Source IP type. (other/private/multicast) |
| src_k8s_deployment_name ( tag) |
Source K8s deployment name |
| src_k8s_namespace ( tag) |
Source K8s namespace |
| src_k8s_pod_name ( tag) |
Source K8s pod name |
| src_k8s_service_name ( tag) |
Source K8s service name |
| src_port ( tag) |
Source port; * when it is the client port or the connection role is unknown |
| sub_source ( tag) |
Some specific connection classifications, such as the sub_source value for Kubernetes network traffic is K8s |
| transport ( tag) |
Transport layer protocol. (udp/tcp) |
| bytes_read | The number of bytes read Type: int | (gauge) Unit: digital,B |
| bytes_written | The number of bytes written Type: int | (gauge) Unit: digital,B |
| count | The total number of HTTP requests in a collection cycle Type: int | (gauge) Unit: count |
| http_version | 1.1 / 1.0 ... Type: string | (string) Unit: N/A |
| latency | TTFB Type: int | (gauge) Unit: time,ns |
| method | GET/POST/... Type: string | (string) Unit: N/A |
| path | Request path Type: string | (string) Unit: N/A |
| status_code | HTTP status code Type: int | (gauge) Unit: N/A |
| truncated | The length of the request path has reached the upper limit of the number of bytes collected, and the request path may be truncated Type: bool | (gauge) Unit: bool |
日志¶
ebpf_network_event¶
| Tags & Fields | Description |
|---|---|
| host ( tag) |
System hostname |
| level ( tag) |
info for factual clues; warn for hard errors or observed drops |
| problem_type ( tag) |
Bounded network problem classification |
| source ( tag) |
Fixed value ebpf_network_event |
| collector_session | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| conn_side | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| connection_id | Socket-generation identity for tcp_connection_issue logs; a field, never a metric tag. Type: string | (string) Unit: N/A |
| event_id | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| event_status | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| evidence_partial | Incomplete evidence or a UDP error without original datagram attribution. Type: bool | (gauge) Unit: bool |
| first_seen_ns | Context or observed event count, not unique connections. Timestamps are Unix nanoseconds. Type: int | (gauge) Unit: N/A |
| interface | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| k8s_namespace | Local Pod ownership from the cached PID mapping when a severe connection issue first appears. Retained through closure; omitted when no mapping is available. Log fields, never metric tags. Type: string | (string) Unit: N/A |
| k8s_pod_name | Local Pod ownership from the cached PID mapping when a severe connection issue first appears. Retained through closure; omitted when no mapping is available. Log fields, never metric tags. Type: string | (string) Unit: N/A |
| k8s_workload_name | Local Pod ownership from the cached PID mapping when a severe connection issue first appears. Retained through closure; omitted when no mapping is available. Log fields, never metric tags. Type: string | (string) Unit: N/A |
| last_seen_ns | Context or observed event count, not unique connections. Timestamps are Unix nanoseconds. Type: int | (gauge) Unit: N/A |
| local_ip | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| message | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| netns | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| occurrences | Context or observed event count, not unique connections. Timestamps are Unix nanoseconds. Type: int | (gauge) Unit: N/A |
| occurrences_total | Context or observed event count, not unique connections. Timestamps are Unix nanoseconds. Type: int | (gauge) Unit: N/A |
| pid | Context or observed event count, not unique connections. Timestamps are Unix nanoseconds. Type: int | (gauge) Unit: N/A |
| process_name | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| remote_ip | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| scope | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| service | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| service_port | Context or observed event count, not unique connections. Timestamps are Unix nanoseconds. Type: int | (gauge) Unit: N/A |
| status | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
| transport | Network evidence context. message contains ebpf_network_event/v1 and up to three representative samples. event_status is observed, ongoing, recovered, or closed. Severe connection issues retain up to three snapshots for one connection. Type: string | (string) Unit: N/A |
bash¶
| Tags & Fields | Description |
|---|---|
| host ( tag) |
host name |
| cmd | Command Type: string | (string) Unit: N/A |
| message | The bash execution record generated by the collector Type: string | (string) Unit: N/A |
| pid | Process identification number Type: string | (string) Unit: N/A |
| user | The user who executes the bash command Type: string | (string) Unit: N/A |
bpf_net_l4_log¶
| Tags & Fields | Description |
|---|---|
| client_ip ( tag) |
The IP address of the client |
| client_port ( tag) |
Client port |
| conn_side ( tag) |
The side of the connection: client/server/unknown |
| direction ( tag) |
Use the source (src_ip:src_port) as a frame of reference to identify the connection initiator. (incoming/outgoing) |
| dst_ip ( tag) |
The IP address of the foreign network interface |
| dst_k8s_deployment_name ( tag) |
Destination K8s deployment name |
| dst_k8s_namespace ( tag) |
Destination K8s namespace |
| dst_k8s_pod_name ( tag) |
Destination K8s pod name |
| dst_k8s_service_name ( tag) |
Destination K8s service name |
| dst_port ( tag) |
Foreign port |
| host ( tag) |
Host name |
| host_network ( tag) |
Whether the network log data is collected on the host network |
| inner_traceid ( tag) |
Correlate the layer 4 and layer 7 network log data of a TCP connection on the collected network interface |
| k8s_container_name ( tag) |
Kubernetes container name |
| k8s_namespace ( tag) |
Kubernetes namespace |
| k8s_pod_name ( tag) |
Kubernetes pod name |
| l4_proto ( tag) |
Transport protocol |
| l7_proto ( tag) |
Application protocol |
| netns ( tag) |
Network namespace, format: NS(<device id>:<inode number>) |
| nic_mac ( tag) |
MAC address of the collected network interface |
| nic_name ( tag) |
Name of the collected network interface |
| server_ip ( tag) |
The IP address of the server |
| server_port ( tag) |
Server port |
| src_ip ( tag) |
The IP address of the collected local network interface |
| src_k8s_deployment_name ( tag) |
Source K8s deployment name |
| src_k8s_namespace ( tag) |
Source K8s namespace |
| src_k8s_pod_name ( tag) |
Source K8s pod name |
| src_k8s_service_name ( tag) |
Source K8s service name |
| src_port ( tag) |
Local port |
| sub_source ( tag) |
Some specific connection classifications, such as the sub_source value for Kubernetes network traffic is K8s |
| virtual_nic ( tag) |
Whether the network log data is collected on the virtual network interface |
| vni_id ( tag) |
Virtual Network Identifier |
| vxlan_packet ( tag) |
Whether it is a VXLAN packet |
| chunk_id | A connection may be divided into several chunks for upload based on time interval or TCP segment number Type: int | (gauge) Unit: count |
| rx_bytes | The number of bytes received by the network interface Type: int | (gauge) Unit: digital,B |
| rx_packets | The number of packets received by the network interface Type: int | (gauge) Unit: count |
| rx_retrans | The number of retransmitted packets received by the network interface Type: int | (gauge) Unit: count |
| rx_seq_max | The maximum value of the TCP sequence number of the data packet received by the network interface, which is a 32-bit unsigned integer Type: int | (gauge) Unit: N/A |
| rx_seq_min | The minimum value of the TCP sequence number of the data packet received by the network interface, which is a 32-bit unsigned integer Type: int | (gauge) Unit: N/A |
| tcp_syn_retrans | The number of retransmitted SYN packets sent by the network interface Type: int | (gauge) Unit: count |
| tx_bytes | The number of bytes sent by the network interface Type: int | (gauge) Unit: digital,B |
| tx_packets | The number of packets sent by the network interface Type: int | (gauge) Unit: count |
| tx_retrans | The number of retransmitted packets sent by the network interface Type: int | (gauge) Unit: count |
| tx_seq_max | The maximum value of the TCP sequence number of the data packet sent by the network interface, which is a 32-bit unsigned integer Type: int | (gauge) Unit: N/A |
| tx_seq_min | The minimum value of the TCP sequence number of the data packet sent by the network interface, which is a 32-bit unsigned integer Type: int | (gauge) Unit: N/A |
bpf_net_l7_log¶
| Tags & Fields | Description |
|---|---|
| client_ip ( tag) |
The IP address of the client |
| client_port ( tag) |
Client port |
| conn_side ( tag) |
The side of the connection: client/server/unknown |
| direction ( tag) |
Use the source (src_ip:src_port) as a frame of reference to identify the connection initiator. (incoming/outgoing) |
| dst_ip ( tag) |
The IP address of the foreign network interface |
| dst_k8s_deployment_name ( tag) |
Destination K8s deployment name |
| dst_k8s_namespace ( tag) |
Destination K8s namespace |
| dst_k8s_pod_name ( tag) |
Destination K8s pod name |
| dst_k8s_service_name ( tag) |
Destination K8s service name |
| dst_port ( tag) |
Foreign port |
| host ( tag) |
Host name |
| host_network ( tag) |
Whether the network log data is collected on the host network |
| inner_traceid ( tag) |
Correlate the layer 4 and layer 7 network log data of a TCP connection on the collected network interface |
| k8s_container_name ( tag) |
Kubernetes container name |
| k8s_namespace ( tag) |
Kubernetes namespace |
| k8s_pod_name ( tag) |
Kubernetes pod name |
| l4_proto ( tag) |
Transport protocol |
| l7_proto ( tag) |
Application protocol |
| l7_traceid ( tag) |
Correlate the layer 7 network log data of a TCP connection on the all collected network interface |
| netns ( tag) |
Network namespace, format: NS(<device id>:<inode number>) |
| nic_mac ( tag) |
MAC address of the collected network interface |
| nic_name ( tag) |
Name of the collected network interface |
| parent_id ( tag) |
The span id of the APM span corresponding to this network request |
| server_ip ( tag) |
The IP address of the server |
| server_port ( tag) |
Server port |
| src_ip ( tag) |
The IP address of the collected local network interface |
| src_k8s_deployment_name ( tag) |
Source K8s deployment name |
| src_k8s_namespace ( tag) |
Source K8s namespace |
| src_k8s_pod_name ( tag) |
Source K8s pod name |
| src_k8s_service_name ( tag) |
Source K8s service name |
| src_port ( tag) |
Local port |
| sub_source ( tag) |
Some specific connection classifications, such as the sub_source value for Kubernetes network traffic is K8s |
| trace_id ( tag) |
APM trace id |
| virtual_nic ( tag) |
Whether the network log data is collected on the virtual network interface |
| vni_id ( tag) |
Virtual Network Identifier |
| vxlan_packet ( tag) |
Whether it is a VXLAN packet |
| http_method | HTTP method Type: string | (string) Unit: N/A |
| http_path | HTTP path Type: string | (string) Unit: N/A |
| http_status_code | HTTP status code Type: int | (gauge) Unit: N/A |
| rx_seq | The TCP sequence number of the first request or response byte received by the network interface Type: int | (gauge) Unit: N/A |
| tx_seq | The TCP sequence number of the first request or response byte sent by the network interface Type: int | (gauge) Unit: N/A |
链路¶
dketrace¶
| Tags & Fields | Description |
|---|---|
| dst_ip ( tag) |
Destination IP address |
| dst_port ( tag) |
Destination port |
| host ( tag) |
System hostname |
| service ( tag) |
Service name |
| src_ip ( tag) |
Source IP |
| src_port ( tag) |
Source port |
| app_parent_id | Parent span id carried by the application in the request Type: string | (string) Unit: N/A |
| app_trace_id | Trace id carried by the application in the request Type: string | (string) Unit: N/A |
| bytes_read | Bytes read Type: int | (gauge) Unit: digital,B |
| bytes_written | Bytes written Type: int | (gauge) Unit: digital,B |
| duration | Duration Type: int | (gauge) Unit: time,μs |
| ebpf_parent_id | eBPF parent span id, generated by the ebpftrace collectorType: string | (string) Unit: N/A |
| ebpf_trace_id | eBPF trace id, generated by the ebpftrace collectorType: string | (string) Unit: N/A |
| err_msg | Redis error message Type: string | (string) Unit: N/A |
| grpc_status_code | gRPC status code Type: string | (string) Unit: N/A |
| http_method | HTTP method Type: string | (string) Unit: N/A |
| http_route | HTTP route Type: string | (string) Unit: N/A |
| http_status_code | HTTP status code Type: string | (string) Unit: N/A |
| mysql_err_msg | MySQL error message Type: string | (string) Unit: N/A |
| mysql_status_code | MySQL request status code Type: int | (gauge) Unit: N/A |
| operation | Operation Type: string | (string) Unit: N/A |
| parent_id | APM parent span id, set by the ebpftrace collectorType: string | (string) Unit: N/A |
| pid | Process identification number Type: string | (string) Unit: N/A |
| process_name | Process name Type: string | (string) Unit: N/A |
| resource_type | Redis resource type Type: string | (string) Unit: N/A |
| source_type | Source type, value is ebpfType: string | (string) Unit: N/A |
| span_id | APM span id, generated by the ebpftrace collectorType: string | (string) Unit: N/A |
| span_type | Span type Type: string | (string) Unit: N/A |
| start | Start time Type: int | (gauge) Unit: timeStamp,usec |
| status | Status Type: string | (string) Unit: N/A |
| status_msg | Redis status message Type: string | (string) Unit: N/A |
| thread_name | Thread name Type: string | (string) Unit: N/A |
| trace_id | APM trace id, can choose between existing app trace id and ebpf generation,set by the ebpftrace collectorType: string | (string) Unit: N/A |
探针与系统开销¶
仅在配置追踪目标时启用 Go 关联;优先使用开销较低的探针方式,不支持时自动回退。最后一个目标及其约 30 秒的复用窗口结束后卸载相关探针,后续目标到来时重新启用。
TLS 探针在目标间复用,最后一个目标消失后保留最多约 30 秒,减少短时重启带来的反复挂卸。资源不足时提前回收空闲探针,采集器停止时清理。这些优化不改变上报字段或数据格式。
文件读取预算不等同于磁盘 I/O 限速或超时。实际资源占用请结合节点指标判断。
Prometheus 自监控¶
配置 pprof_port 后,datakit-ebpf 会在同一监听地址上同时暴露:
/debug/pprof//metrics
建议仅在内网或本机地址上暴露,例如:
OOM 排查重点指标¶
未显式设置采集器的内存限制或 GOMEMLIMIT 时,采集器启动时读取 cgroup v1/v2 的有效内存上限(包括可见祖先和 v1 层级限制)。由 DataKit 启动的子进程默认将 Go runtime 软预算设为该上限的 35%,独立运行时为 70%;无法读取父进程信息时使用 35%。例如共享 2000 MiB 上限时,eBPF 子进程的 Go 软预算约为 700 MiB。检测只在启动时执行;调整容器限制后需要重启采集器。
此预算通过 Go 的软内存限制促使 GC 回收,不能限制 AF_PACKET/perf mmap、内核 BPF map 或 DataKit 父进程,也不保证容器总 RSS 不越界。已有 GOMEMLIMIT 优先;显式 mem_limit 仍使用原有资源限制和退出行为,并按 DK_EBPF_GO_MEMORY_LIMIT_RATIO(默认 0.8)设置 Go 预算。未配置有限 cgroup 上限的主机不会自动限制 Go 内存。
L7 空闲事件池分为 7 个大小档位,每档记账容量不超过约 512 KiB,总计不超过 3.5 MiB;正在处理的事件仍由异步队列和解码器预算记账。TCP 缺口重试复用 syscall 事件的排序状态。网络字段和连接、字节核账口径不变。
L7 异步工作队列满或字节预算不足时立即拒绝当前事件,按 socket 生命周期标记缺口;perf 回调不等待队列腾空,也不获取该连接的解码器锁。工作线程负责释放失去同步的排队数据和解码状态,并丢弃与过载同时处理的可疑结果。尚未建立用户态连接状态时,以有界的生命周期记录阻止从流中间开始解析。这不会扩大队列或解码内存预算;持续过载仍可能丢样本,应结合 async_event、perf lost 和解码预算计数判断覆盖情况。
支持返回 perf 溢出错误的内核会使用现有有界恢复队列保存 L7 事件,并在恢复完成后继续发送;恢复后的事件仍保持原顺序。dkebpf_exporter_perf_lost_samples_total 记录 perf 传输通知,增长本身不代表相同数量的 L7 事务丢失,应同时查看 dkebpf_exporter_l7_event_recovery_total{result="delivered"} 和实际序号、解码缺口。部分老内核(例如 Linux 4.15)在 perf 溢出时仍返回成功,无法通过该返回值追回事件;未恢复的序号缺口仍会使受影响的采集证据失去完整性,不能按完整事务解释。恢复队列持续过载也可能产生不可恢复缺口。
已经确认无法恢复同步的连接会及时释放待排序 payload 和解码状态,避免这些数据继续占用健康连接的预算;队列中已收到的关闭通知仍用于回收连接。已知 HTTP body 边界且可以安全恢复的缺口保留原有处理。
httpflow 通过 eBPF syscall/socket 探针采集应用读写数据,TLS 明文通过支持的 uprobe 采集;不再启动额外的 L7 AF_PACKET 报文补齐、namespace 监听器或 TCP payload 缓存。bpf-netlog 和 DNS 的独立抓包功能保持原有行为。
sendfile / splice 可能仅提供实际完成的字节范围,不提供对应报文内容。已观测到 HTTP 头且 body 长度已知时可推进字节统计;HTTP 头或 chunk framing 不可见时记录采集缺口,不构造完整事务,也不保证该连接后续的 L7 事务可恢复。TCP 连接和字节统计独立保留。
应用读写数据按有界长度捕获。已知 HTTP body 长度时,解析器按实际完成的 I/O 字节数推进,允许跳过未捕获的 body;若请求头、chunk framing 或其他协议边界缺失,则标记解码缺口并停止不可靠的事务配对。主动截断、用户内存读取失败和 perf/队列事件丢失应分别核对,不能将任一计数直接当作事务丢失率。
以下指标适合优先用于定位 OOM 和内存异常增长:
dkebpf_runtime_memory_bytes{type="rss"}- 进程实际驻留内存,最接近 OOM 压力
dkebpf_runtime_memory_bytes{type="hwm"}- 进程 RSS 历史峰值
dkebpf_runtime_memory_bytes{type="go_limit"}- 当前有限的 Go runtime 软预算;不代表总 RSS 上限
dkebpf_runtime_memory_bytes{type="go_total"}- Go runtime 已向系统申请且仍保留的主要内存
dkebpf_runtime_memory_bytes{type="heap_alloc"}- Go 堆当前已分配且仍在使用的内存
dkebpf_runtime_memory_bytes{type="non_go_estimate"}- 近似值,计算方式为
rss - go_total - 用于判断 RSS 增长是否主要来自非 Go 内存,例如 cgo、native buffer、mmap 或其它运行时外分配
dkebpf_runtime_memory_ratio{type="non_go_vs_rss"}- 非 Go 内存占 RSS 的比例
除上述指标外,还会暴露更细粒度的 Go 内存分量,例如:
heap_inuseheap_idleheap_releasedheap_unusedstack_inusestack_sysmspan_inusemspan_sysmcache_inusemcache_sysgc_sysother_sysbuck_hash_sys
以及以下辅助指标:
dkebpf_runtime_memory_objectsdkebpf_runtime_memory_ops_totaldkebpf_runtime_gcdkebpf_runtime_goroutines
聚合与缓存压力指标¶
为避免直接输出高基数业务标签,eBPF 自监控只暴露低基数的聚合压力指标:
dkebpf_exporter_agg_entries{component}- 当前聚合池中的 key 数量
dkebpf_exporter_agg_flush_points_total{component,result}- 每次 flush 输出的点数量累计
dkebpf_exporter_agg_flush_duration_seconds{component,result}- flush 耗时
dkebpf_exporter_cache_entries{component,cache}- 中间缓存或队列长度
当前已覆盖的组件包括:
netflowdnsflowl4log_netflowl4log_httpflowl7flow_httpl4logl4log_host_peer_shared
以及部分中间态缓存,例如:
dnsflowpending query / packet queue / flush queuel4logconnection pool / two-msl pool / blacklist cachel7flowconn map / closed conn map / span buffer
perf 和 TPacket 指标¶
当前还会暴露与 perf 事件通道以及 afpacket.TPacket 抓包 ring 相关的累计指标:
dkebpf_exporter_perf_lost_samples_total{component,stream}- perf reader 在用户态观察到的丢样本数量
dkebpf_exporter_perf_read_errors_total{component,stream}- perf reader 读取错误次数
dkebpf_exporter_tpacket_stats_total{component,type}TPacket累计统计信息type包括:packetsdropsqueue_freezes
这些指标更适合暴露为累计计数器,而不是单独暴露“速率”指标。
如需查看速率,请直接在 Prometheus 中使用 rate() 或 irate() 计算。
发送链路指标¶
如果内存增长来自发送堵塞或上游写入变慢,可重点关注:
发送器的队列及在途编码数据合计上限仍为 16 MiB。每批最多 64 点,通常最多 4 MiB;单个较大的点保留完整字段,无法装入其准入预算时记为 batch_too_large。原始网络日志和详细 tracing 不占用最后 2 MiB,为其他输入留出容量。已从抓包状态分离的网络日志和详细 tracing 最多等待 5 秒入队;DNS 等普通输入仍保持 1 秒等待上限。DK_EBPF_EXPORTER_ENQUEUE_TIMEOUT 可同时覆盖两个默认值。下游持续过载仍会显式计数并丢弃;预留容量不保证送达,也不预留 HTTP 工作线程。
诊断 map 准入失败通过本地自监控指标 dkebpf_exporter_diagnostic_map_update_failures_total{map,reason} 细分:map 仅为 flow_faults、connect_failures、udp_errors、listener_stats,reason 仅为 capacity、memory、exists、missing、other。这些计数及 /debug/netflow 的 health.diagnostic_map_failures 补充原有总失败数,不增加平台 measurement。map 占用观测也包含错误样本和监听队列 map。
连接失败和 UDP 硬错误样本的内部 map 键使用结束时间与 PID/TID 组合,避免多个任务在同一时间戳下相互冲突。样本中的开始、结束时间和已有上报字段保持原口径。
错误样本读取在数据探针挂载前启动,独立于整张连接表扫描和 namespace 文件读取,每 100 ms 处理一次有界批次。支持时使用原子批量取出并删除;旧内核使用复用缓冲区的逐项方式。两条路径均不新增无界队列,固定 map 容量不变。
dkebpf_exporter_sender_queue_length- exporter 当前待发送任务数
dkebpf_exporter_sender_batch_points- 每个发送批次包含的点数分布
dkebpf_exporter_sender_batch_bytes- 每个发送批次编码后的字节大小分布
dkebpf_exporter_sender_requests_total{result}- 发送请求次数
dkebpf_exporter_sender_request_duration_seconds{result}- 发送请求耗时
其中 result 为低基数枚举值,例如:
okrequest_errorhttp_4xxhttp_5xx
周期性摘要日志¶
除 Prometheus 指标外,datakit-ebpf 还会按采集周期额外发送一条低频摘要日志到 DataKit:
- 数据类别:
Logging - measurement:
ebpf_monitor - source:
ebpf_monitor - input:
ebpf-monitor
这条日志用于补充指标查看,单条日志会汇总:
- runtime 内存摘要
- 聚合池和缓存长度
- 网卡分类数量与 route 数量摘要
- exporter 队列与发送结果增量
- perf lost / read error 增量
- TPacket packets / drops / queue_freezes 增量
- BPF map 条数、容量、fill ratio 和写入失败增量
- 输出队列实际丢弃点数增量;不代表临时排队或重试次数
- perf buffer / socket / filter 的配置与状态摘要
- 例如 perf reader 大小、共享 host-peer socket 模式、filter attach 结果、socket 侧 drops/freezes 摘要
map 写入失败、输出队列丢弃和 perf 读取错误有新增时,摘要状态为 warn。历史累计失败值保持不变时不会重复记作新增失败。原有 cache.entries 明细保留;cache_entries_total 和 cache.total_entries 不再累加其中的失败计数、累计计数及丢失 epoch。
日志中会带一个较大的 message 字段,内容为固定 schema 的 JSON 摘要,便于直接解析和检索,同时保留少量关键数值 field 便于筛选。
对于 perf/ringbuf 配置、socket buffer 侧状态、共享 socket/filter 模式这类信息,建议只作为摘要日志补充,不再继续扩成独立 Prometheus 指标,以避免额外时间线和低价值告警噪声。
排查建议¶
可按以下顺序判断问题来源:
rss持续增长,但heap_alloc和go_total增长不明显 这通常意味着问题更偏向非 Go 内存,应重点查看non_go_estimate、non_go_vs_rssagg_entries、cache_entries持续增长且不回落 这通常意味着聚合池或中间缓存堆积sender_queue_length持续增长,且sender_request_duration_seconds变高 这通常意味着发送链路变慢,导致聚合结果无法及时出队bpf_map_fill_ratio持续接近1这通常意味着内核态 map 压力过高,可能同时伴随事件丢弃
DDTrace 是由 Datadog 发起并开源的链路追踪项目,相关著作权归 Datadog 及其他贡献者所有。观测云支持 DDTrace 的数据上报,具体使用遵循相应项目的开源许可证。