Dataway¶

Introduction¶

DataWay is the data gateway of the Guance, and the collector needs to pass through the DataWay gateway to report data to the Guance.

Dataway Installation¶

• New Dataway

On the Data Gateways page in the Guance management console, click Create Dataway. Enter a name and binding address, and then click Create.

After the creation is successful, a new Dataway is automatically created and the installation script for the Dataway is generated.

• Install Dataway

DW_KODO=http://kodo_ip:port \
   DW_TOKEN=<tkn_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
   DW_UUID=<YOUR_UUID> \
   bash -c "$(curl https://static.guance.com/dataway/install.sh)"

# ============= DATAWAY CONFIG =============

# Dataway UUID, we can get it on during create a new dataway
uuid:

# It's the workspace token, most of the time, it's
# system worker space's token.
token:

# secret_token used under sinker mode, and to check if incomming datakit
# requests are valid.
secret_token:

# If __internal__ token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_internal_token: false

# is empty token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_empty_token: false

# Is dataway cascaded? For cascaded Dataway, it's remote_host is
# another Dataway and not Kodo.
cascaded: false

# kodo(next dataway) related configures
remote_host:
http_timeout: 3s

http_max_idle_conn_perhost: 0 # default to CPU cores
http_max_conn_perhost: 0      # default no limit
kodo_queue:
  enabled: true
  workers: 256
  queue_size: 1024
  queue_max_bytes: 1073741824 # 1GB
  enqueue_timeout: 100ms

insecure_skip_verify: false
http_client_trace: false
sni: ""

# dataway API configures
# IPv4 and IPv6 use independent listeners to avoid OS bindv6only differences.
bind: 0.0.0.0:9528
# IPv6 is disabled by default. Set this to "[::]:9528" to enable it.
bind_ipv6: ""

# Forwarded IP headers are trusted for whitelist checks only from these proxies.
# trusted_proxies:
#   - 10.0.0.0/8
#   - 2001:db8:ffff::/48

# disable 404 page
disable_404page: false

# dataway TLS file path
tls_crt:
tls_key:

# pprof is disabled by default and accepts loopback listeners only
pprof_bind: ""

api_limit_rate : 100000         # 100K
max_http_body_bytes : 67108864  # 64MB
copy_buffer_drop_size : 262144  # 256KB, if copy buffer memory larger than this, this memory released
reserved_pool_size: 4096        # reserved pool size for better GC

within_docker: false

log_level: info
log: log
gin_log: gin.log

ip_black_list:
  ttl: "1m"
  clean_interval: "1h"

cache_cfg:
  # cache disk path
  dir: "disk_cache"

  # disable cache
  disabled: false

  clean_interval: "1s"

  # in MB, max single data package size in disk cache, such as HTTP body
  max_data_size: 100

  # in MB, single disk-batch(single file) size
  batch_size: 128

  # in MB, max disk size allowed to cache data
  max_disk_size: 65535

  # expire duration, default 7 days
  expire_duration: "168h"

prometheus:
  listen: "localhost:9090"
  url: "/metrics"
  enable: true

#sinker:
#  cache_options:
#    prealloc: true
#    reserved_capacity: 10000000 # max cached items
#    buckets: 64
#    ttl: 10m # clear unactive matches
#  etcd:
#    urls:
#    - http://localhost:2379 # one or multiple etcd host
#    dial_timeout: 30s
#    key_space: "/dw_sinker" # subscribe to the etcd key
#    username: "dataway"
#    password: "<PASSWORD>"
#  file:
#    path: /path/to/sinker.json

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: sts-utils-dataway
  name: dataway
  namespace: utils
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sts-utils-dataway
  serviceName: dataway
  template:
    metadata:
      annotations:
        datakit/logs: |
          [
            {
              "disable": false,
              "source": "dataway",
              "service": "dataway",
              "multiline_match": "^\\d{4}|^\\[GIN\\]"
            }
          ]
        datakit/prom.instances: |
          [[inputs.prom]]
            url = "http://$IP:9090/metrics"

            source = "dataway"
            measurement_name = "dw"
            interval = "10s"
            disable_instance_tag = true
          [inputs.prom.tags]
            service = "dataway"
            instance = "$PODNAME" # we can set as "xxx-$PODNAME"
      labels:
        app: sts-utils-dataway
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - sts-utils-dataway
              topologyKey: kubernetes.io/hostname
      containers:
        - env:
            - name: DW_REMOTE_HOST
              value: http://kodo.forethought-kodo:9527
            - name: DW_BIND
              value: 0.0.0.0:9528
            # Uncomment the next two lines to listen on IPv6.
            # - name: DW_BIND_IPV6
            #   value: "[::]:9528"
            - name: DW_UUID
              value: agnt_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx   # Dataway UUID
            - name: DW_TOKEN
              value: tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  # Dataway token
            - name: DW_PROM_LISTEN
              value: 0.0.0.0:9090
            - name: DW_LOG
              value: stdout
            - name: DW_LOG_LEVEL
              value: info
            - name: DW_GIN_LOG
              value: stdout
            - name: DW_DISKCACHE_DIR
              value: cache
            - name: DW_HTTP_TIMEOUT
              value: '3s'
            - name: DW_ENABLE_INTERNAL_TOKEN
              value: 'false'
            - name: DW_MAX_HTTP_BODY_BYTES
              value: '67108864'
            - name: DW_HTTP_CLIENT_TRACE
              value: 'on'
            - name: DW_RESERVED_POOL_SIZE
              value: '0'
            - name: DW_COPY_BUFFER_DROP_SIZE
              value: '262144'
            - name: DW_DISKCACHE_CAPACITY_MB
              value: 102400
          image: pubrepo.guance.com/dataflux/dataway:1.19.0
          imagePullPolicy: IfNotPresent
          name: dataway
          ports:
            - containerPort: 9528
              name: 9528tcp01
              protocol: TCP
          resources:
            limits:
              cpu: '4'
              memory: 4Gi
            requests:
              cpu: 100m
              memory: 512Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /usr/local/cloudcare/dataflux/dataway/cache
              name: dataway-cache
      dnsPolicy: ClusterFirst
      imagePullSecrets: []
      #nodeSelector:
      #  nodepool: dataway
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      #tolerations:
      #  - effect: NoSchedule
      #    key: nodepool
      #    operator: Equal
      #    value: dataway
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: dataway-cache
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: xxxxxx  # High-Performance Storage StorageClass
        volumeMode: Filesystem
      status:
        phase: Pending

---

apiVersion: v1
kind: Service
metadata:
  name: dataway
  namespace: utils
spec:
  ipFamilyPolicy: SingleStack # Use PreferDualStack for dual-stack mode
  # ipFamilies:
  #   - IPv4
  #   - IPv6
  ports:
    - name: 9528tcp02
      nodePort: 30928
      port: 9528
      protocol: TCP
      targetPort: 9528
  selector:
    app: sts-utils-dataway
  type: NodePort
---

apiVersion: v1
kind: Service
metadata:
  name: dataway
  namespace: utils
spec:
  ipFamilyPolicy: SingleStack # Use PreferDualStack for dual-stack mode
  # ipFamilies:
  #   - IPv4
  #   - IPv6
  ports:
  - name: 9528tcp02
    port: 9528
    protocol: TCP
    targetPort: 9528
    nodePort: 30928
  selector:
    app: deployment-utils-dataway
  type: NodePort

docker run -d \
    --name <YOUR-DW-IN-DOCKER> \
    -p 19528:9528 -p 19090:9090 \
    --mount type=bind,source=<host/path/for/diskcache>,target=/usr/local/cloudcare/dataflux/dataway/cache \
    --memory=2g --memory-reservation=256m \
    --cpus="2" \
    -e DW_UUID=<YOUR-AGNT_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
    -e DW_TOKEN=<YOUR-TKN_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
    -e DW_REMOTE_HOST=http://kodo.forethought-kodo:9527 \
    -e DW_BIND=0.0.0.0:9528 \
    -e DW_PROM_LISTEN=0.0.0.0:9090 \
    -e DW_HTTP_CLIENT_TRACE=true \
    -e DW_LOG_LEVEL=info \
    -e DW_LOG=stdout \
    -e DW_GIN_LOG=stdout \
    -e DW_DISKCACHE_CAPACITY_MB=65536 \
    pubrepo.guance.com/dataflux/dataway:1.19.0

• Verify Dataway installation

After installation, wait for a while to refresh the "Data Gateway" page, if you see the version number in the "Version Information" column of the data gateway you just added, it means that the Dataway has been successfully connected to the Guance center, and front-end users can access data through it.

After Dataway is successfully connected to the Guance center, log in to the Guance console, view all Dataway addresses on the "Integration" / DataKit page, select the required Dataway gateway address, and obtain the DataKit installation command to execute on the server to start collecting data.

Manage DataWay¶

Delete DataWay¶

On the "Data Gateway" page of the Guance management background, select the DataWay to be deleted, click "Configure", and click the "Delete" button in the lower left corner of the pop-up Edit DataWay dialog box.

Upgrade DataWay¶

On the Data Gateways page in the Guance management background, if an upgradeable version exists for DataWay, an upgrade prompt appears in the version information.

DW_UPGRADE=1 bash -c "$(curl https://static.guance.com/dataway/install.sh)"

- image: pubrepo.guance.com/dataflux/dataway:1.19.0

Dataway Service Management¶

When the host installs Dataway, you can use the following command to manage the Dataway service:

# Start
$ systemctl start dataway

# Reboot
$ systemctl restart dataway

# Stop
$ systemctl stop dataway

Kubernetes can restart the corresponding pod.

Environment variable¶

Docker image environment variable¶

When Dataway runs in a Kubernetes environment, it supports the following environment variables.

HTTP Server Settings¶

HTTP TLS Settings¶

To generate a TLS certificate with a one-year validity period, you can use the following OpenSSL command:

# Generate a TLS certificate with a one-year validity
$ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out tls.crt -keyout tls.key
...

After executing this command, the system will prompt you to enter some necessary information, including your country, region, city, organization name, department name, and your email address. This information will be included in your certificate.

After completing the information input, you will generate two files: tls.crt (certificate file) and tls.key (private key file). Please keep your private key file secure and safe.

In order for the application to use these TLS certificates, you need to set the absolute paths of these two files to the application's environment variables. Here is an example of setting environment variables:

DW_ENABLE_TLS must be turned on first, and then the other two ENVs (DW_TLS_CRT/DW_TLS_KEY) will take effect. Version-1.4.1

env:
- name: DW_ENABLE_TLS
  value: "true"
- name: DW_TLS_CRT
  value: "/path/to/your/tls.crt"
- name: DW_TLS_KEY
  value: "/path/to/your/tls.key"

Replace /path/to/your/tls.crt and /path/to/your/tls.key with the actual paths to your tls.crt and tls.key files, respectively.

After setting up the environment variables, you can test whether TLS is functioning correctly with the following command:

curl -k https://localhost:9528

If successful, you should see an ASCII art message indicating "It's working!" If the certificates are missing, you might encounter an error in the Dataway logs similar to:

server listen(TLS) failed: open /path/to/your/tls.{crt,key}: no such file or directory

In this case, Dataway would not start, and the curl command would also result in an error:

curl: (7) Failed to connect to localhost port 9528 after 6 ms: Couldn't connect to server

Ensure that the paths to the TLS certificate and key are correctly specified and that the files have the appropriate permissions for the application to read them.

Log Settings¶

Token/UUID Settings¶

Sinker Settings¶

Prometheus Metrics Exposure¶

Disk Cache Settings¶

Performance Settings¶

Version-1.6.0

Dataway API List¶

Except for diagnostic and locally handled APIs, forwarding APIs validate tokens according to their own rules and forward requests to Kodo or the next Dataway. See the Langfuse compatibility API section below for its authentication behavior.

GET /v1/ping¶

Version-1.11.0

• API description: Get current dataway version and release date, also return the public IP address of your request

If 404 page disabled(disable_404page), this API not working.

GET /v1/ntp¶

Version-1.6.0

• API description: Get current Dataway unix timestamp(unit: second)

POST /v1/write/:category¶

• API description: Receive various collection data uploaded by Datakit

POST /v1/aggregate¶

• API description: Receives aggregated metric data. This route is registered only when the aggregate feature is initialized successfully.
• For the operating modes, configuration, and request requirements, see Dataway Metric Aggregation.

POST /v1/tail_sampling, POST /v1/tail_sampling_v2, POST /v2/tail_sampling, and POST /v1/tail_sampling_config¶

• API description: /v1/tail_sampling accepts raw tail-sampling packets and also accepts headerless zstd packets sent by Datakit 2.10; the legacy /v1/tail_sampling_v2 path accepts raw packets only; /v2/tail_sampling accepts zstd packets with the compression-negotiation header; /v1/tail_sampling_config accepts tail-sampling configuration. These routes are registered only when the tail-sampling feature is initialized successfully.
• For the operating modes, configuration, and request requirements, see Dataway Tail Sampling.

POST /otel/v1/{traces,metrics,logs}¶

Version-1.17.0

• API description: Receive OpenTelemetry OTLP HTTP/protobuf data and forward it to the matching Kodo path without rewriting the payload.
• Supported paths:
  • POST /otel/v1/traces
  • POST /otel/v1/metrics
  • POST /otel/v1/logs

• Request requirements:

  • Content-Type must be application/x-protobuf; media type parameters are accepted, for example application/x-protobuf; charset=utf-8
  • The workspace token can be provided through the token query, the X-Token header, or Authorization: Bearer <token>; Bearer tokens are converted to X-Token before forwarding
  • When using OpenTelemetry SDK/Agent environment variables, explicitly configure OTLP HTTP/protobuf and set the base endpoint to the Dataway /otel prefix; OpenTelemetry appends /v1/traces, /v1/metrics, and /v1/logs for each signal:

  export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
  export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://dataway.example.com/otel/v1/logs
  export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://dataway.example.com/otel/v1/traces
  export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://dataway.example.com/otel/v1/metrics
  export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20{token}"
  

  • If per-signal endpoints are used, configure the full Dataway paths, for example OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://dataway.example.com/otel/v1/traces, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://dataway.example.com/otel/v1/metrics, and OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://dataway.example.com/otel/v1/logs
  • Startup options have the same meaning as the environment variables. For example, with the Java Agent, configure -Dotel.exporter.otlp.protocol=http/protobuf, -Dotel.exporter.otlp.endpoint=https://dataway.example.com/otel, and -Dotel.exporter.otlp.headers=Authorization=Bearer%20{token}. If an implementation handles the encoded space in the Authorization header differently, use OTEL_EXPORTER_OTLP_HEADERS="X-Token={token}" or the matching startup option instead.
  • Dataway does not parse or rewrite the OTLP body; /otel/v1/metrics is not converted into /v1/write/metric line protocol. Kodo parses it with DataKit's default OTLP metrics rules: the measurement is otel_service, and metric field names keep the original OTLP metric names, for example runtime.jvm.memory is not rewritten to runtime_jvm_memory
  • Kodo's raw OTLP response status, response body, and required response headers are passed through; when forwarding fails and matches the disk-cache policy, the request enters Dataway disk-cache replay

GET /v1/datakit/pull¶

• API description: Handles Datakit pull center configuration (blacklist/pipeline) requests

POST /v1/datakit/usage_trace¶

• API description: Receives Datakit usage tracing data.

POST /v1/write/rum/replay¶

• API description: Receive Session Replay data uploaded by Datakit

POST /v1/write/rum/replay_assets¶

• API description: Receives Session Replay asset upload requests.

POST /v1/check/rum/replay_assets¶

• API description: Checks whether a Session Replay asset already exists.

POST /v1/upload/profiling¶

• API description: Receive profiling data uploaded by Datakit

POST /v1/input/firehose¶

• API description: Receives Firehose input data.

POST /v1/election¶

• API description: Handles election requests for Datakit

POST /v1/election/heartbeat¶

• API description: Handles election heartbeat requests for Datakit

POST /v1/query/raw¶

• API description: Handles DQL query requests initiated by the Datakit side

POST /v1/workspace¶

• API description: Handles workspace query requests initiated by Datakit

GET /v1/env_variable¶

• API description: Retrieves workspace environment variables.

POST /v1/dataway/heartbeat¶

• API description: Receives Dataway heartbeat reports.

POST /v1/object/labels¶

• API description: Handles requests to modify object labels

DELETE /v1/object/labels¶

• API description: Handles delete object Label requests

GET /v1/check/token/:token¶

• API description: Checks whether a token is valid.

Langfuse Compatibility APIs¶

The following APIs are provided for Langfuse client compatibility. Dataway forwards these requests upstream without token validation:

• POST /api/public/otel/v1/traces: Receives Langfuse OTLP trace data.
• GET /api/public/projects: Retrieves Langfuse project information.
• POST /api/public/ingestion: Receives Langfuse ingestion data.

Dataway metrics collection¶

[[inputs.prom]]
  ## Exporter URLs.
  urls = [ "http://localhost:9090/metrics", ]

  source = "dataway"

  election = true

  ## Dataway metric set fixed to dw, do not change
  measurement_name = "dw"

annotations:
   datakit/prom.instances: |
     [[inputs.prom]]
       url = "http://$IP:9090/metrics" # Here the port (default 9090) is as appropriate
       source = "dataway"
       measurement_name = "dw" # pinned to this metric set
       interval = "30s"

       [inputs.prom.tags]
         namespace = "$NAMESPACE"
         pod_name = "$PODNAME"
         node_name = "$NODENAME"

If the collection is successful, search for dataway in the "Scene"/"Built-in View" of the Guance to see the corresponding monitoring view.

Cache and Kodo Queue Troubleshooting¶

When Kodo returns 5xx, Dataway fails to send a request to Kodo, or the Kodo dispatch queue is full, Dataway tries to write eligible requests into the disk cache. Kodo 4xx responses are not cached.

Use these metrics first when troubleshooting:

• dataway_http_api_cached_bytes{reason=...}: the request was written to cache. Common reason values include Kodo 5xx status text, httpCliDoFailed, and kodoQueueFull.
• dataway_http_api_dropped_cache{reason=...}: Dataway tried to cache the request but did not write it. Common reason values include feat-disabled, expected-api-drop, body-md5-not-matched, body-length-not-matched, expired, put-failed, and sendopt-nil.
• dataway_kodo_queue_full_total{action=...}: how Dataway handled a full Kodo queue. action="cache" means the request was moved to cache; action="reject" means Dataway returned 503 after cache was unavailable or cache write failed.

Dataway Metric List¶

The following are the indicators exposed by Dataway, which can be obtained by requesting http://localhost:9090/metrics, and you can view (3s) a specific indicator in real time by following the following command:

If some metrics cannot be queried, it may be caused by the relevant business module not running.

watch -n 3 'curl -s http://localhost:9090/metrics | grep -a <METRIC-NAME>'

Metrics under Docker¶

There are two modes for non-Kubernetes, host mode and Docker mode. This section will specifically discuss the differences in metrics collection when installing in Docker.

When installed in docker, the HTTP port that exposes metrics will be mapped to port 19090 on the host machine (by default). In this case, the metrics collection address is http://localhost:19090/metrics.

If a different port is specified, the installer will add 10000 to the specified port during installation. Therefore, the specified port should not exceed 45535.

pprof is disabled by default, and Docker installations do not publish a profiling port to the host. For temporary diagnosis, explicitly set pprof_bind or DW_PPROF_BIND to localhost:6060, then collect through docker exec, kubectl exec, or a controlled port forward. Non-loopback addresses are rejected.

Dataway's Own Log Collection and Processing¶

Dataway's own logs are divided into two categories: one is the gin log, and the other is the Dataway's own log. The following Pipeline can separate them:

# Pipeline for dataway logging

# Testing sample loggin
'''
2023-12-14T11:27:06.744+0800    DEBUG   apis    apis/api_upload_profile.go:272  save profile file to disk [ok] /v1/upload/profiling?token=****************a4e3db8481c345a94fe5a
[GIN] 2021/10/25 - 06:48:07 | 200 |   30.890624ms |  114.215.200.73 | POST     "/v1/write/logging?token=tkn_5c862a11111111111111111111111"
'''

add_pattern("TOKEN", "tkn_\\w+")
add_pattern("GINTIME", "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}-%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}")
grok(_,"\\[GIN\\]%{SPACE}%{GINTIME:timestamp}%{SPACE}\\|%{SPACE}%{NUMBER:dataway_code}%{SPACE}\\|%{SPACE}%{NOTSPACE:cost_time}%{SPACE}\\|%{SPACE}%{NOTSPACE:client_ip}%{SPACE}\\|%{SPACE}%{NOTSPACE:method}%{SPACE}%{GREEDYDATA:http_url}")

# gin logging
if cost_time != nil {
  if http_url != nil  {
    grok(http_url, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(http_url, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  group_between(dataway_code, [200,299], "info", status)
  group_between(dataway_code, [300,399], "notice", status)
  group_between(dataway_code, [400,499], "warning", status)
  group_between(dataway_code, [500,599], "error", status)

  if sample(0.1) { # drop 90% debug log
    drop()
    exit()
  } else {
    set_tag(sample_rate, "0.1")
  }

  parse_duration(cost_time)
  duration_precision(cost_time, "ns", "ms")

  set_measurement('gin', true)
  set_tag(service,"dataway")
  exit()
}

# app logging
if cost_time == nil {
  grok(_,"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NOTSPACE:status}%{SPACE}%{NOTSPACE:module}%{SPACE}%{NOTSPACE:code}%{SPACE}%{GREEDYDATA:msg}")
  if level == nil {
    grok(message,"Error%{SPACE}%{DATA:errormsg}")
    if errormsg != nil {
      add_key(status,"error")
      drop_key(errormsg)
    }
  }
  lowercase(level)

  # if debug level enabled, drop most of them
  if status == 'debug' {
    if sample(0.1) { # drop 90% debug log
      drop()
      exit()
    } else {
      set_tag(sample_rate, "0.1")
    }
  }

  group_in(status, ["error", "panic", "dpanic", "fatal","err","fat"], "error", status) # mark them as 'error'

  if msg != nil {
    grok(msg, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(msg, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  set_measurement("dataway-log", true)
  set_tag(service,"dataway")
}

Dataway Bug Report¶

Dataway exposes metrics and can temporarily enable a profiling endpoint for troubleshooting. pprof is disabled by default; set pprof_bind to a loopback address before collection and disable it again afterward.

Use the actual configured addresses and ports when collecting the following information.

br_dir="dw-br-$(date +%s)"
mkdir -p $br_dir

echo "Save bug report to ${br_dir}"

# Modify the following configurations according to your actual situation
dw_ip="localhost" # The IP address where Dataway's metrics/profile is exposed
metric_port=9090  # The port where metrics are exposed
profile_port=6060 # Explicitly enable the localhost:6060 pprof listener first
dw_yaml_conf="/usr/local/cloudcare/dataflux/dataway/dataway.yaml"
dw_dot_yaml_conf="/usr/local/cloudcare/dataflux/dataway/.dataway.yaml"

# Collect runtime metrics
curl -v "http://${dw_ip}:${metric_port}/metrics" -o $br_dir/metrics

# Collect profiling information
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/allocs" -o $br_dir/allocs
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/heap" -o $br_dir/heap
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/profile" -o $br_dir/profile # This command will take about 30 seconds to run

cp $dw_yaml_conf $br_dir/dataway.yaml.copy
cp $dw_dot_yaml_conf $br_dir/.dataway.yaml.copy

tar czvf ${br_dir}.tar.gz ${br_dir}
rm -rf ${br_dir}

Run the script:

$ sh dw-bug-report.sh
...

After execution, a file similar to dw-br-1721188604.tar.gz will be generated. You can then retrieve this file for further use.

FAQ¶

Request Entity Too Large Issue¶

Version-1.3.7

Dataway has a default setting for the size of the request body (default is 64MB), but when the request body is too large, the client will receive an HTTP 413 error (Request Entity Too Large). If the request body is within a reasonable range, you can appropriately increase this value (unit is bytes):

• Set the environment variable DW_MAX_HTTP_BODY_BYTES for Kubernetes Pod install
• In dataway.yaml, set max_http_body_bytes for host install

If there is a request that is too large during runtime, it is reflected in both metrics and logs:

• The metric dataway_http_too_large_dropped_total exposes the number of discarded large requests
• Search the Dataway logs with cat log | grep 'drop too large request'. The logs will output the details of the HTTP request Header, which is helpful for further understanding the client situation