Dataway¶

Introduction¶

Dataway is the data gateway of Guance. All data reported by collectors to Guance must pass through the Dataway gateway.

Dataway Installation¶

• Create a New Dataway

In the Guance Management Console, go to the "Data Gateway" page and click Create Dataway. Enter the name and binding address, then click Create.

After creation, a new Dataway is automatically created and an installation script is generated.

• Install Dataway

DW_KODO=http://kodo_ip:port \
   DW_TOKEN=<tkn_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
   DW_UUID=<YOUR_UUID> \
   bash -c "$(curl https://static.guance.com/dataway/install.sh)"

# ============= DATAWAY CONFIG =============

# Dataway UUID, we can get it on during create a new dataway
uuid:

# It's the workspace token, most of the time, it's
# system worker space's token.
token:

# secret_token used under sinker mode, and to check if incomming datakit
# requests are valid.
secret_token:

# If __internal__ token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_internal_token: false

# is empty token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_empty_token: false

# Is dataway cascaded? For cascaded Dataway, it's remote_host is
# another Dataway and not Kodo.
cascaded: false

# kodo(next dataway) related configures
remote_host:
http_timeout: 3s

http_max_idle_conn_perhost: 0 # default to CPU cores
http_max_conn_perhost: 0      # default no limit
kodo_queue:
  enabled: true
  workers: 256
  queue_size: 1024
  queue_max_bytes: 1073741824 # 1GB
  enqueue_timeout: 100ms

insecure_skip_verify: false
http_client_trace: false
sni: ""

# dataway API configures
# IPv4 and IPv6 use separate listeners to avoid OS bindv6only behavior
bind: 0.0.0.0:9528
# IPv6 is disabled by default; explicitly set to "[::]:9528" when needed
bind_ipv6: ""

# Only requests from trusted proxies use forwarded headers for IP whitelist evaluation
# trusted_proxies:
#   - 10.0.0.0/8
#   - 2001:db8:ffff::/48

# disable 404 page
disable_404page: false

# dataway TLS file path
tls_crt:
tls_key:

# pprof is disabled by default; only loopback addresses are allowed
pprof_bind: ""

api_limit_rate : 100000         # 100K
max_http_body_bytes : 67108864  # 64MB
copy_buffer_drop_size : 262144  # 256KB, if copy buffer memory larger than this, this memory released
reserved_pool_size: 4096        # reserved pool size for better GC

within_docker: false

log_level: info
log: log
gin_log: gin.log

ip_black_list:
  ttl: "1m"
  clean_interval: "1h"

cache_cfg:
  # cache disk path
  dir: "disk_cache"

  # disable cache
  disabled: false

  clean_interval: "1s"

  # in MB, max single data package size in disk cache, such as HTTP body
  max_data_size: 100

  # in MB, single disk-batch(single file) size
  batch_size: 128

  # in MB, max disk size allowed to cache data
  max_disk_size: 65535

  # expire duration, default 7 days
  expire_duration: "168h"

prometheus:
  listen: "localhost:9090"
  url: "/metrics"
  enable: true

#sinker:
#  cache_options:
#    prealloc: true
#    reserved_capacity: 10000000 # max cached items
#    buckets: 64
#    ttl: 10m # clear unactive matches
#  etcd:
#    urls:
#    - http://localhost:2379 # one or multiple etcd host
#    dial_timeout: 30s
#    key_space: "/dw_sinker" # subscribe to the etcd key
#    username: "dataway"
#    password: "<PASSWORD>"
#  file:
#    path: /path/to/sinker.json

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: sts-utils-dataway
  name: dataway
  namespace: utils
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sts-utils-dataway
  serviceName: dataway
  template:
    metadata:
      annotations:
        datakit/logs: |
          [
            {
              "disable": false,
              "source": "dataway",
              "service": "dataway",
              "multiline_match": "^\\d{4}|^\\[GIN\\]"
            }
          ]
        datakit/prom.instances: |
          [[inputs.prom]]
            url = "http://$IP:9090/metrics"

            source = "dataway"
            measurement_name = "dw"
            interval = "10s"
            disable_instance_tag = true
          [inputs.prom.tags]
            service = "dataway"
            instance = "$PODNAME" # we can set as "xxx-$PODNAME"
      labels:
        app: sts-utils-dataway
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - sts-utils-dataway
              topologyKey: kubernetes.io/hostname
      containers:
        - env:
            - name: DW_REMOTE_HOST
              value: http://kodo.forethought-kodo:9527
            - name: DW_BIND
              value: 0.0.0.0:9528
            # To enable IPv6, uncomment the following two lines
            # - name: DW_BIND_IPV6
            #   value: "[::]:9528"
            - name: DW_UUID
              value: agnt_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx   # Dataway UUID
            - name: DW_TOKEN
              value: tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  # Dataway token
            - name: DW_PROM_LISTEN
              value: 0.0.0.0:9090
            - name: DW_LOG
              value: stdout
            - name: DW_LOG_LEVEL
              value: info
            - name: DW_GIN_LOG
              value: stdout
            - name: DW_DISKCACHE_DIR
              value: cache
            - name: DW_HTTP_TIMEOUT
              value: '3s'
            - name: DW_ENABLE_INTERNAL_TOKEN
              value: 'false'
            - name: DW_MAX_HTTP_BODY_BYTES
              value: '67108864'
            - name: DW_HTTP_CLIENT_TRACE
              value: 'on'
            - name: DW_RESERVED_POOL_SIZE
              value: '0'
            - name: DW_COPY_BUFFER_DROP_SIZE
              value: '262144'
            - name: DW_DISKCACHE_CAPACITY_MB
              value: 102400
          image: pubrepo.guance.com/dataflux/dataway:1.19.0
          imagePullPolicy: IfNotPresent
          name: dataway
          ports:
            - containerPort: 9528
              name: 9528tcp01
              protocol: TCP
          resources:
            limits:
              cpu: '4'
              memory: 4Gi
            requests:
              cpu: 100m
              memory: 512Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /usr/local/cloudcare/dataflux/dataway/cache
              name: dataway-cache
      dnsPolicy: ClusterFirst
      imagePullSecrets: []
      #nodeSelector:
      #  nodepool: dataway
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      #tolerations:
      #  - effect: NoSchedule
      #    key: nodepool
      #    operator: Equal
      #    value: dataway
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: dataway-cache
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: xxxxxx  # High-Performance Storage StorageClass
        volumeMode: Filesystem
      status:
        phase: Pending

---

apiVersion: v1
kind: Service
metadata:
  name: dataway
  namespace: utils
spec:
  ipFamilyPolicy: SingleStack # Change to PreferDualStack for dual-stack
  # ipFamilies:
  #   - IPv4
  #   - IPv6
  ports:
    - name: 9528tcp02
      nodePort: 30928
      port: 9528
      protocol: TCP
      targetPort: 9528
  selector:
    app: sts-utils-dataway
  type: NodePort

containers:
  volumeMounts:
    - name: dataway-config
      mountPath: /usr/local/cloudcare/dataflux/dataway/dataway.yaml
      subPath: config.yaml
volumes:
- configMap:
    defaultMode: 256
    name: dataway-config
    optional: false
  name: dataway-config

docker run -d \
    --name <YOUR-DW-IN-DOCKER> \
    -p 19528:9528 -p 19090:9090 \
    --mount type=bind,source=<host/path/for/diskcache>,target=/usr/local/cloudcare/dataflux/dataway/cache \
    --memory=2g --memory-reservation=256m \
    --cpus="2" \
    -e DW_UUID=<YOUR-AGNT_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
    -e DW_TOKEN=<YOUR-TKN_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
    -e DW_REMOTE_HOST=http://kodo.forethought-kodo:9527 \
    -e DW_BIND=0.0.0.0:9528 \
    -e DW_PROM_LISTEN=0.0.0.0:9090 \
    -e DW_HTTP_CLIENT_TRACE=true \
    -e DW_LOG_LEVEL=info \
    -e DW_LOG=stdout \
    -e DW_GIN_LOG=stdout \
    -e DW_DISKCACHE_CAPACITY_MB=65536 \
    pubrepo.guance.com/dataflux/dataway:1.19.0

• Verify Dataway Installation

After installation, wait a moment and refresh the "Data Gateway" page. If you see a version number in the "Version Information" column of the newly added Dataway, it means the Dataway has successfully connected to the Guance center. Front-end users can then use it to ingest data.

Once Dataway successfully connects to the Guance center, log in to the Guance console. On the "Integrations" / "DataKit" page, you will see all Dataway addresses. Select the desired Dataway gateway address, obtain the DataKit installation command, and execute it on the server to start data collection.

Manage Dataway¶

Delete Dataway¶

On the Guance Management Console "Data Gateway" page, select the Dataway to delete, click Configure, and in the Edit Dataway dialog, click the Delete button at the bottom left.

Upgrade Dataway¶

On the Guance Management Console "Data Gateway" page, if an upgrade is available for a Dataway, an upgrade prompt will appear in the version information area.

DW_UPGRADE=1 bash -c "$(curl https://static.guance.com/dataway/install.sh)"

- image: pubrepo.guance.com/dataflux/dataway:1.19.0

Dataway Service Management¶

For host-installed Dataway, use the following commands to manage the Dataway service.

# Start
$ systemctl start dataway

# Restart
$ systemctl restart dataway

# Stop
$ systemctl stop dataway

For Kubernetes, restart the corresponding Pod.

Environment Variables¶

Image Environment Variables¶

When Dataway runs in a Kubernetes environment, it supports the following environment variables.

HTTP Server Settings¶

HTTP TLS Settings¶

To generate a TLS certificate valid for one year, use the following OpenSSL command:

# Generate a TLS certificate valid for one year
$ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out tls.crt -keyout tls.key
...

After running the command, you will be prompted to enter necessary information, including your country, region, city, organization name, department name, and email address. This information will be included in your certificate.

After completing the input, two files will be generated: tls.crt (certificate file) and tls.key (private key file). Keep your private key file secure.

To make the application use these TLS certificates, set the absolute paths of these two files to the application's environment variables. Example:

DW_ENABLE_TLS must be enabled first for the other two ENVs (DW_TLS_CRT/DW_TLS_KEY) to take effect. Version-1.4.1

env:
- name: DW_ENABLE_TLS
  value: "true"
- name: DW_TLS_CRT
  value: "/path/to/your/tls.crt"
- name: DW_TLS_KEY
  value: "/path/to/your/tls.key"

Replace /path/to/your/tls.crt and /path/to/your/tls.key with the actual paths to your tls.crt and tls.key files.

After configuration, test TLS with the following command:

$ curl -k http://localhost:9528

If successful, an ASCII Art message "It's working!" will be displayed. If the certificate does not exist, Dataway logs will show an error similar to:

server listen(TLS) failed: open /path/to/your/tls.{crt,key}: no such file or directory

In this case, Dataway will not start, and the curl command above will also fail:

$ curl -vvv -k http://localhost:9528
curl: (7) Failed to connect to localhost port 9528 after 6 ms: Couldn't connect to server

Log Settings¶

Token/UUID Settings¶

Sinker Settings¶

Prometheus Metrics Exposure¶

Disk Cache Settings¶

Performance-Related Settings¶

Version-1.6.0

Dataway API List¶

Except for diagnostic and local processing interfaces, forwarding interfaces forward requests to Kodo or the next-hop Dataway according to their respective token validation rules. See below for Langfuse-compatible interface authentication behavior.

GET /v1/ping¶

Version-1.11.0

• API Description: Retrieve the current Dataway version number, release date, and the client's outgoing IP address.

If the Dataway has disabled the 404 page (disable_404page), this endpoint will not be available.

GET /v1/ntp¶

Version-1.6.0

• API Description: Retrieve the current Dataway Unix timestamp (in seconds).

POST /v1/write/:category¶

• API Description: Receive various telemetry data uploaded by Datakit.

POST /v1/aggregate¶

• API Description: Receive metric aggregation data. Registered only when the aggregation function is successfully initialized.
• For detailed working modes, configuration, and request requirements, see Dataway Metric Aggregation.

POST /v1/tail_sampling, POST /v1/tail_sampling_v2, POST /v2/tail_sampling and POST /v1/tail_sampling_config¶

• API Description: /v1/tail_sampling receives raw tail sampling data packets and is compatible with zstd data packets sent by Datakit 2.10 without negotiation headers; the legacy path /v1/tail_sampling_v2 only receives raw data packets; /v2/tail_sampling receives zstd data packets with compression negotiation headers; /v1/tail_sampling_config receives tail sampling configuration. Registered only when the tail sampling function is successfully initialized.
• For detailed working modes, configuration, and request requirements, see Dataway Tail Sampling.

POST /otel/v1/{traces,metrics,logs}¶

Version-1.17.0

• API Description: Receive OpenTelemetry OTLP HTTP/protobuf data and forward it unchanged to the corresponding Kodo path.
• Supported paths:
  • POST /otel/v1/traces
  • POST /otel/v1/metrics
  • POST /otel/v1/logs

• Request requirements:

  • Content-Type must be application/x-protobuf, with media type parameters allowed (e.g., application/x-protobuf; charset=utf-8)
  • The workspace token can be passed via the token query parameter, X-Token header, or Authorization: Bearer <token>. Bearer tokens are converted to X-Token before forwarding.
  • When using OpenTelemetry SDK/Agent environment variables, it is recommended to explicitly configure OTLP HTTP/protobuf and set the base endpoint to the Dataway's /otel prefix. OpenTelemetry automatically appends /v1/traces, /v1/metrics, and /v1/logs by signal:

  export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
  export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://dataway.example.com/otel/v1/logs
  export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://dataway.example.com/otel/v1/traces
  export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://dataway.example.com/otel/v1/metrics
  export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20{token}"
  

  • The startup parameters and environment variables have the same meaning. For example, with the Java Agent, you can configure -Dotel.exporter.otlp.protocol=http/protobuf, -Dotel.exporter.otlp.endpoint=https://dataway.example.com/otel, and -Dotel.exporter.otlp.headers=Authorization=Bearer%20{token}. If the implementation has inconsistent encoding compatibility for spaces in the Authorization header, you can also use OTEL_EXPORTER_OTLP_HEADERS="X-Token={token}" or the corresponding startup parameter.
  • Dataway does not parse or rewrite the OTLP body. /otel/v1/metrics is not converted to /v1/write/metric line protocol. Kodo parses it according to the default DataKit OTLP metrics rules, with the measurement being otel_service. Metric field names retain the original OTLP metric name (e.g., runtime.jvm.memory is not rewritten to runtime_jvm_memory).
  • The original OTLP response status, response body, and necessary response headers from Kodo are passed through. If forwarding fails and the disk cache policy is satisfied, the request enters the Dataway disk cache replay chain.

GET /v1/datakit/pull¶

• API Description: Handle Datakit requests to pull center configuration (blacklist/Pipeline).

POST /v1/datakit/usage_trace¶

• API Description: Receive Datakit usage trace data.

POST /v1/write/rum/replay¶

• API Description: Receive Session Replay data uploaded by Datakit.

POST /v1/write/rum/replay_assets¶

• API Description: Receive upload requests for Session Replay-related assets.

POST /v1/check/rum/replay_assets¶

• API Description: Check if Session Replay-related assets already exist.

POST /v1/upload/profiling¶

• API Description: Receive Profiling data uploaded by Datakit.

POST /v1/input/firehose¶

• API Description: Receive Firehose input data.

POST /v1/election¶

• API Description: Handle Datakit election requests.

POST /v1/election/heartbeat¶

• API Description: Handle Datakit election heartbeat requests.

POST /v1/query/raw¶

Handle DQL query requests. Example:

POST /v1/query/raw?token=<workspace-token> HTTP/1.1
Content-Type: application/json

{
    "token": "workspace-token",
    "queries": [
        {
            "query": "M::cpu LIMIT 1"
        }
    ],
    "echo_explain": <true/false>
}

Response example:

{
  "content": [
    {
      "series": [
        {
          "name": "cpu",
          "columns": [
            "time",
            "usage_iowait",
            "usage_total",
            "usage_user",
            "usage_guest",
            "usage_system",
            "usage_steal",
            "usage_guest_nice",
            "usage_irq",
            "load5s",
            "usage_idle",
            "usage_nice",
            "usage_softirq",
            "global_tag1",
            "global_tag2",
            "host",
            "cpu"
          ],
          "values": [
            [
              1709782208662,
              0,
              7.421875,
              3.359375,
              0,
              4.0625,
              0,
              0,
              0,
              1,
              92.578125,
              0,
              0,
              null,
              null,
              "WIN-JCHUL92N9IP",
              "cpu-total"
            ]
          ]
        }
      ],
      "points": null,
      "cost": "24.558375ms",
      "is_running": false,
      "async_id": "",
      "query_parse": {
        "namespace": "metric",
        "sources": {
          "cpu": "exact"
        },
        "fields": {},
        "funcs": {}
      },
      "index_name": "",
      "index_store_type": "",
      "query_type": "guancedb",
      "complete": false,
      "index_names": "",
      "scan_completed": false,
      "scan_index": "",
      "next_cursor_time": -1,
      "sample": 1,
      "interval": 0,
      "window": 0
    }
  ]
}

Response description:

• The actual data is in the inner series field.
• name indicates the measurement name (here, the query is for CPU metrics; for log data, this field is absent).
• columns lists the names of the result columns.
• values contains the corresponding column results from columns.

POST /v1/workspace¶

• API Description: Handle workspace query requests initiated by Datakit.

GET /v1/env_variable¶

• API Description: Retrieve workspace environment variables.

POST /v1/dataway/heartbeat¶

• API Description: Receive Dataway heartbeat reporting requests.

POST /v1/object/labels¶

• API Description: Handle requests to modify object labels.

DELETE /v1/object/labels¶

• API Description: Handle requests to delete object labels.

GET /v1/check/token/:token¶

• API Description: Check if a token is valid.

Langfuse-Compatible Interfaces¶

The following interfaces are for Langfuse client compatibility. Requests are forwarded directly upstream; Dataway does not perform token validation:

• POST /api/public/otel/v1/traces: Receive Langfuse OTLP Trace data.
• GET /api/public/projects: Retrieve Langfuse project information.
• POST /api/public/ingestion: Receive Langfuse ingestion data.

Dataway Metrics Collection¶

[[inputs.prom]]
  ## Exporter URLs.
  urls = [ "http://localhost:9090/metrics", ]
  source = "dataway"
  election = true
  measurement_name = "dw" # dataway measurement is fixed to dw; do not change
[inputs.prom.tags]
  service = "dataway"

annotations: # The following annotations are added by default
   datakit/prom.instances: |
     [[inputs.prom]]
       url = "http://$IP:9090/metrics" # The port (default 9090) depends on the actual setup
       source = "dataway"
       measurement_name = "dw" # Fixed to this measurement
       interval = "10s"
       disable_instance_tag = true

     [inputs.prom.tags]
       service = "dataway"
       instance = "$PODNAME"

...
env:
- name: DW_PROM_LISTEN
  value: "0.0.0.0:9090" # The port here must match the port in the url above

If collection is successful, you can search for dataway in the Guance "Scenarios" / "Built-in Views" to see the corresponding monitoring dashboard.

Cache and Kodo Queue Troubleshooting¶

When Kodo returns a 5xx, Dataway fails to request Kodo, or the Kodo dispatch queue is full, Dataway attempts to write cacheable requests to disk cache. Kodo 4xx responses are not cached.

For troubleshooting, first check the following metrics:

• dataway_http_api_cached_bytes{reason=...}: Request has been written to cache. Common reason values include Kodo 5xx status text, httpCliDoFailed, and kodoQueueFull.
• dataway_http_api_dropped_cache{reason=...}: Request attempted to cache but was not written. Common reason values include feat-disabled, expected-api-drop, body-md5-not-matched, body-length-not-matched, expired, put-failed, and sendopt-nil.
• dataway_kodo_queue_full_total{action=...}: Result after the Kodo queue is full. action="cache" indicates a transfer to cache; action="reject" indicates cache is unavailable or write failed, returning 503.

Dataway Metrics List¶

The following are the metrics exposed by Dataway. You can retrieve them by requesting http://localhost:9090/metrics. Use the following command to watch a specific metric in real time (3s interval):

If some metrics are not found, it may be because the relevant business module has not yet started. Some new metrics only exist in the latest version. The version information for each metric is not individually marked here; refer to the metric list returned by the /metrics endpoint.

watch -n 3 'curl -s http://localhost:9090/metrics | grep -a <METRIC-NAME>'

Metrics Collection in Docker Mode¶

Host installation has two modes: host-based installation and Docker installation. This section explains the differences for Docker installation.

When installed via Docker, the metrics exposure HTTP port is mapped to host port 19090 (by default). The metrics collection address is http://localhost:19090/metrics.

If a different port is specified, Docker will add 10000 to that port. Therefore, do not specify a port number exceeding 45535.

pprof is disabled by default. Docker installation does not publish the profile port to the host. For temporary troubleshooting, you can explicitly set pprof_bind or DW_PPROF_BIND to localhost:6060 and collect via docker exec, kubectl exec, or controlled port forwarding. Configuring a non-loopback address will be rejected.

Dataway Log Collection and Processing¶

Dataway logs are divided into two types: gin logs and application logs. The following Pipeline can be used to separate them:

# Pipeline for dataway logging

# Testing sample loggin
'''
2023-12-14T11:27:06.744+0800    DEBUG   apis    apis/api_upload_profile.go:272  save profile file to disk [ok] /v1/upload/profiling?token=****************a4e3db8481c345a94fe5a
[GIN] 2021/10/25 - 06:48:07 | 200 |   30.890624ms |  114.215.200.73 | POST     "/v1/write/logging?token=tkn_5c862a11111111111111111111111111"
'''

add_pattern("TOKEN", "tkn_\\w+")
add_pattern("GINTIME", "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}-%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}")
grok(_,"\\[GIN\\]%{SPACE}%{GINTIME:timestamp}%{SPACE}\\|%{SPACE}%{NUMBER:dataway_code}%{SPACE}\\|%{SPACE}%{NOTSPACE:cost_time}%{SPACE}\\|%{SPACE}%{NOTSPACE:client_ip}%{SPACE}\\|%{SPACE}%{NOTSPACE:method}%{SPACE}%{GREEDYDATA:http_url}")

# gin logging
if cost_time != nil {
  if http_url != nil  {
    grok(http_url, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(http_url, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  group_between(dataway_code, [200,299], "info", status)
  group_between(dataway_code, [300,399], "notice", status)
  group_between(dataway_code, [400,499], "warning", status)
  group_between(dataway_code, [500,599], "error", status)

  if sample(0.1) { # drop 90% debug log
    drop()
    exit()
  } else {
    set_tag(sample_rate, "0.1")
  }

  parse_duration(cost_time)
  duration_precision(cost_time, "ns", "ms")

  set_measurement('gin', true)
  set_tag(service,"dataway")
  exit()
}

# app logging
if cost_time == nil {
  grok(_,"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NOTSPACE:status}%{SPACE}%{NOTSPACE:module}%{SPACE}%{NOTSPACE:code}%{SPACE}%{GREEDYDATA:msg}")
  if level == nil {
    grok(message,"Error%{SPACE}%{DATA:errormsg}")
    if errormsg != nil {
      add_key(status,"error")
      drop_key(errormsg)
    }
  }
  lowercase(level)

  # if debug level enabled, drop most of them
  if status == 'debug' {
    if sample(0.1) { # drop 90% debug log
      drop()
      exit()
    } else {
      set_tag(sample_rate, "0.1")
    }
  }

  group_in(status, ["error", "panic", "dpanic", "fatal","err","fat"], "error", status) # mark them as 'error'

  if msg != nil {
    grok(msg, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(msg, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  set_measurement("dataway-log", true)
  set_tag(service,"dataway")
}

Dataway Bug Report¶

Dataway exposes metrics and can temporarily enable profiling collection endpoints for troubleshooting. pprof is disabled by default; set pprof_bind explicitly to a loopback address before collection, and disable it again after collection.

The following information collection uses the actual configured port and address.

br_dir="dw-br-$(date +%s)"
mkdir -p $br_dir

echo "save bug report to ${br_dir}"

# Modify these settings according to the actual situation
dw_ip="localhost" # IP address where Dataway metrics/profile are exposed
metric_port=9090  # Port for metrics exposure
profile_port=6060 # Must first explicitly enable localhost:6060 pprof listener
dw_yaml_conf="/usr/local/cloudcare/dataflux/dataway/dataway.yaml"
dw_dot_yaml_conf="/usr/local/cloudcare/dataflux/dataway/.dataway.yaml" # Exists in container installation

# Collect runtime metrics
curl -v "http://${dw_ip}:${metric_port}/metrics" -o $br_dir/metrics

# Collect profiling information
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/allocs" -o $br_dir/allocs
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/heap" -o $br_dir/heap
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/profile" -o $br_dir/profile # This command runs for about 30s

cp $dw_yaml_conf $br_dir/dataway.yaml.copy
cp $dw_dot_yaml_conf $br_dir/.dataway.yaml.copy

tar czvf ${br_dir}.tar.gz ${br_dir}
rm -rf ${br_dir}

Run the script:

$ sh dw-bug-report.sh
...

After execution, a file like dw-br-1721188604.tar.gz will be generated. Extract that file.

FAQ¶

Request Body Too Large¶

Version-1.3.7

Dataway has a default limit on request body size (default 64MB). When the request body is too large, the client receives an HTTP 413 error (Request Entity Too Large). If the request body is within a reasonable range, you can increase this value (in bytes):

• Set the environment variable DW_MAX_HTTP_BODY_BYTES
• Set max_http_body_bytes in dataway.yaml

If unusually large request packets occur during runtime, this is reflected in both metrics and logs:

• The metric dataway_http_too_large_dropped_total exposes the count of dropped large requests.
• Search the Dataway log cat log | grep 'drop too large request'; the log will output the HTTP request header details for further understanding of the client situation.