Skip to content

Proxy

Proxy collector used to proxy HTTP request.

Configuration

Collector Configuration

Go to the conf.d/samples directory under the DataKit installation directory, copy proxy.conf.sample and name it proxy.conf. Examples are as follows:

[[inputs.proxy]]
  ## choose some inner IP address
  bind = "127.0.0.1"
  ## default bind port
  port = 9530

  # allowed client IP address(in CIDR format)
  allowed_client_cidrs = []

  # verbose mode will show more info about during proxying.
  verbose = false

  # mitm: man-in-the-middle mode
  mitm = false

After configuration, restart DataKit.

The collector can now be turned on by ConfigMap Injection Collector Configuration.

Network Topology

If all local DataKit Proxied there HTTP(s) requests to some proxy input:

# /usr/local/datakit/conf.d/datakit.conf
[dataway]
  http_proxy = "http://some-datakit-with-proxy-ip:port"
  # some other configures...

The topology seems like this(here proxy server bind on some IP's 9530 port):

flowchart LR;
dk_A(DataKit A);
dk_B(DataKit B);
dk_C(DataKit C);
dk_X_proxy("DataKit X's Proxy(some-ip:9530)");
dw(DataWay/Openway);

subgraph "Local network"
dk_A --> dk_X_proxy;
dk_B --> dk_X_proxy;
dk_C --> dk_X_proxy;
end

subgraph "Public network"
dk_X_proxy ==> |https://openway.guance.com|dw;
end

About MITM mode

MITM: Man In The Middle.

We can enable MITM mode to observe more details about the proxy input:

  • All local DataKit instances that connect to the proxy must enable tls_insecure:
# /usr/local/datakit/conf.d/datakit.conf
[dataway]
  tls_insecure = true # Don't worry about the insecure settings, see below.
  # some other configures...

Here, tls_insecure allows local DataKit instances to accept the self-signed TLS certificate used by the proxy collector. See the certificate source.

  • Once DataKit accepts the certificate, the proxy can inspect the HTTP(S) request and export additional Prometheus metrics.
  • The proxy forwards the request to DataWay with a valid TLS certificate.
Warning

Enabling MITM substantially reduces proxy collector performance because the proxy must read and copy each incoming request. See the benchmark below for details.

Metrics

The proxy collector exports the following Prometheus metrics:

POSITION TYPE NAME LABELS HELP
internal/plugins/inputs/proxy COUNTER datakit_input_proxy_connect client_ip Proxied connect(method CONNECT)
internal/plugins/inputs/proxy COUNTER datakit_input_proxy_api_total api,method Proxied API total
internal/plugins/inputs/proxy SUMMARY datakit_input_proxy_api_latency_seconds api,method,status Proxied API latency

If some DataKit enabled Proxy input, there will be some metrics in dashboard of DataKit.

Attention

Without MITM, datakit_input_proxy_api_total and datakit_input_proxy_api_latency_seconds will be null.

Benchmark

We got a simple HTTP(s) server & client to benchmark the proxy input. Basic settings:

  • Machine: Apple M1 Pro/16GB
  • OS: macOS Ventura 13
  • HTTP(s) server: A simple HTTP(s) server that route on POST /v1/write/:category, and response 200 immediately.
  • Client: POST a text file about 170KB(metric.data) to the server.
  • Proxy: Started a Proxy input(on http://localhost:19530) within a local DataKit
  • Jobs: 16 clients, each POST 100 requests

The command seems like this:

$./cli -c 16 -r 100 -f metric.data -proxy http://localhost:19530

We got following result(in Prometheus metrics):

  • Without MITM:
Benchmark metrics:
# HELP api_elapsed_seconds Proxied API elapsed seconds
# TYPE api_elapsed_seconds gauge
api_elapsed_seconds 0.249329709
# HELP api_latency_seconds Proxied API latency
# TYPE api_latency_seconds summary
api_latency_seconds{api="/v1/write/xxx",status="200 OK",quantile="0.5"} 0.002227916
api_latency_seconds{api="/v1/write/xxx",status="200 OK",quantile="0.9"} 0.002964042
api_latency_seconds{api="/v1/write/xxx",status="200 OK",quantile="0.99"} 0.008195959
api_latency_seconds_sum{api="/v1/write/xxx",status="200 OK"} 3.9450724669999992
api_latency_seconds_count{api="/v1/write/xxx",status="200 OK"} 1600
# HELP api_post_bytes_total Proxied API post bytes total
# TYPE api_post_bytes_total counter
api_post_bytes_total{api="/v1/write/xxx",status="200 OK"} 2.764592e+08
  • With MITM, performance decrease dramatically(~100X):
Benchmark metrics:
# HELP api_elapsed_seconds Proxied API elapsed seconds
# TYPE api_elapsed_seconds gauge
api_elapsed_seconds 29.454341333
# HELP api_latency_seconds Proxied API latency
# TYPE api_latency_seconds summary
api_latency_seconds{api="/v1/write/xxx",status="200 OK",quantile="0.5"} 0.29453425
api_latency_seconds{api="/v1/write/xxx",status="200 OK",quantile="0.9"} 0.405621917
api_latency_seconds{api="/v1/write/xxx",status="200 OK",quantile="0.99"} 0.479301875
api_latency_seconds_sum{api="/v1/write/xxx",status="200 OK"} 461.3323555329998
api_latency_seconds_count{api="/v1/write/xxx",status="200 OK"} 1600
# HELP api_post_bytes_total Proxied API post bytes total
# TYPE api_post_bytes_total counter
api_post_bytes_total{api="/v1/write/xxx",status="200 OK"} 2.764592e+08

Conclusion:

  • Without MITM, the TPS is 1600/0.249329709 = 6417/Sec
  • With MITM, the TPS decrease to 1600/29.454341333 = 54/sec

So we do NOT recommend to enable MITM, it's a settings for debugging or testing.

Feedback

Is this page helpful?