Dataway¶

Introduction¶

DataWay is the data gateway of the Guance, and the collector needs to pass through the DataWay gateway to report data to the Guance.

Dataway Installation¶

• New Dataway

On the Data Gateways page in the Guance management console, click Create Dataway. Enter a name and binding address, and then click Create.

After the creation is successful, a new Dataway is automatically created and the installation script for the Dataway is generated.

• Install Dataway

DW_KODO=http://kodo_ip:port \
   DW_TOKEN=<tkn_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
   DW_UUID=<YOUR_UUID> \
   bash -c "$(curl https://static.guance.com/dataway/install.sh)"

# ============= DATAWAY CONFIG =============

# Dataway UUID, we can get it on during create a new dataway
uuid:

# It's the workspace token, most of the time, it's
# system worker space's token.
token:

# secret_token used under sinker mode, and to check if incomming datakit
# requests are valid.
secret_token:

# If __internal__ token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_internal_token: false

# is empty token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_empty_token: false

# Is dataway cascaded? For cascaded Dataway, it's remote_host is
# another Dataway and not Kodo.
cascaded: false

# kodo(next dataway) related configures
remote_host:
http_timeout: 3s
# Inbound HTTP deadlines, independent of downstream request timeout
http_read_timeout: 5m
http_write_timeout: 10m
http_idle_timeout: 2m

http_max_idle_conn_perhost: 0 # default to CPU cores
http_max_conn_perhost: 0      # default no limit
kodo_queue:
  enabled: true
  # Also limits concurrent HTTP requests before body reading.
  workers: 256
  queue_size: 1024
  queue_max_bytes: 1073741824 # 1GB
  enqueue_timeout: 100ms

insecure_skip_verify: false
http_client_trace: false
sni: ""

# dataway API configures
# IPv4 and IPv6 use independent listeners to avoid OS bindv6only differences.
bind: 0.0.0.0:9528
# IPv6 is disabled by default. Set this to "[::]:9528" to enable it.
bind_ipv6: ""

# Forwarded IP headers are trusted for whitelist checks only from these proxies.
# trusted_proxies:
#   - 10.0.0.0/8
#   - 2001:db8:ffff::/48

# disable 404 page
disable_404page: false

# dataway TLS file path
tls_crt:
tls_key:

# pprof is disabled by default and accepts loopback listeners only
pprof_bind: ""

api_limit_rate : 100000         # 100K
max_http_body_bytes : 67108864  # 64MB
copy_buffer_drop_size : 262144  # 256KB, if copy buffer memory larger than this, this memory released
reserved_pool_size: 4096        # reserved pool size for better GC

within_docker: false

log_level: info
log: log
gin_log: gin.log

ip_black_list:
  ttl: "1m"
  clean_interval: "1h"

cache_cfg:
  # cache disk path
  dir: "disk_cache"

  # disable cache
  disabled: false

  clean_interval: "1s"

  # in MB, max single data package size in disk cache, such as HTTP body
  max_data_size: 100

  # in MB, single disk-batch(single file) size
  batch_size: 128

  # in MB, max disk size allowed to cache data
  max_disk_size: 65535

  # expire duration, default 7 days
  expire_duration: "168h"

prometheus:
  listen: "localhost:9090"
  url: "/metrics"
  enable: true

#sinker:
#  cache_options:
#    prealloc: true
#    reserved_capacity: 10000000 # max cached items
#    buckets: 64
#    ttl: 10m # clear unactive matches
#  etcd:
#    urls:
#    - http://localhost:2379 # one or multiple etcd host
#    dial_timeout: 30s
#    key_space: "/dw_sinker" # subscribe to the etcd key
#    username: "dataway"
#    password: "<PASSWORD>"
#  file:
#    path: /path/to/sinker.json

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: sts-utils-dataway
  name: dataway
  namespace: utils
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sts-utils-dataway
  serviceName: dataway
  template:
    metadata:
      annotations:
        datakit/logs: |
          [
            {
              "disable": false,
              "source": "dataway",
              "service": "dataway",
              "multiline_match": "^\\d{4}|^\\[GIN\\]"
            }
          ]
        datakit/prom.instances: |
          [[inputs.prom]]
            url = "http://$IP:9090/metrics"

            source = "dataway"
            measurement_name = "dw"
            interval = "10s"
            disable_instance_tag = true
          [inputs.prom.tags]
            service = "dataway"
            instance = "$PODNAME" # we can set as "xxx-$PODNAME"
      labels:
        app: sts-utils-dataway
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - sts-utils-dataway
              topologyKey: kubernetes.io/hostname
      containers:
        - env:
            - name: DW_REMOTE_HOST
              value: http://kodo.forethought-kodo:9527
            - name: DW_BIND
              value: 0.0.0.0:9528
            # Uncomment the next two lines to listen on IPv6.
            # - name: DW_BIND_IPV6
            #   value: "[::]:9528"
            - name: DW_UUID
              value: agnt_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx   # Dataway UUID
            - name: DW_TOKEN
              value: tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  # Dataway token
            - name: DW_PROM_LISTEN
              value: 0.0.0.0:9090
            - name: DW_LOG
              value: stdout
            - name: DW_LOG_LEVEL
              value: info
            - name: DW_GIN_LOG
              value: stdout
            - name: DW_DISKCACHE_DIR
              value: cache
            - name: DW_HTTP_TIMEOUT
              value: '3s'
            - name: DW_ENABLE_INTERNAL_TOKEN
              value: 'false'
            - name: DW_MAX_HTTP_BODY_BYTES
              value: '67108864'
            - name: DW_HTTP_CLIENT_TRACE
              value: 'on'
            - name: DW_RESERVED_POOL_SIZE
              value: '0'
            - name: DW_COPY_BUFFER_DROP_SIZE
              value: '262144'
            - name: DW_DISKCACHE_CAPACITY_MB
              value: 102400
          image: pubrepo.guance.com/dataflux/dataway:1.21.0
          imagePullPolicy: IfNotPresent
          name: dataway
          ports:
            - containerPort: 9528
              name: 9528tcp01
              protocol: TCP
          resources:
            limits:
              cpu: '4'
              memory: 4Gi
            requests:
              cpu: 100m
              memory: 512Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /usr/local/cloudcare/dataflux/dataway/cache
              name: dataway-cache
      dnsPolicy: ClusterFirst
      imagePullSecrets: []
      #nodeSelector:
      #  nodepool: dataway
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      #tolerations:
      #  - effect: NoSchedule
      #    key: nodepool
      #    operator: Equal
      #    value: dataway
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: dataway-cache
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: xxxxxx  # High-Performance Storage StorageClass
        volumeMode: Filesystem
      status:
        phase: Pending

---

apiVersion: v1
kind: Service
metadata:
  name: dataway
  namespace: utils
spec:
  ipFamilyPolicy: SingleStack # Use PreferDualStack for dual-stack mode
  # ipFamilies:
  #   - IPv4
  #   - IPv6
  ports:
    - name: 9528tcp02
      nodePort: 30928
      port: 9528
      protocol: TCP
      targetPort: 9528
  selector:
    app: sts-utils-dataway
  type: NodePort
---

apiVersion: v1
kind: Service
metadata:
  name: dataway
  namespace: utils
spec:
  ipFamilyPolicy: SingleStack # Use PreferDualStack for dual-stack mode
  # ipFamilies:
  #   - IPv4
  #   - IPv6
  ports:
  - name: 9528tcp02
    port: 9528
    protocol: TCP
    targetPort: 9528
    nodePort: 30928
  selector:
    app: deployment-utils-dataway
  type: NodePort

docker run -d \
    --name <YOUR-DW-IN-DOCKER> \
    -p 19528:9528 -p 19090:9090 \
    --mount type=bind,source=<host/path/for/diskcache>,target=/usr/local/cloudcare/dataflux/dataway/cache \
    --memory=2g --memory-reservation=256m \
    --cpus="2" \
    -e DW_UUID=<YOUR-AGNT_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
    -e DW_TOKEN=<YOUR-TKN_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
    -e DW_REMOTE_HOST=http://kodo.forethought-kodo:9527 \
    -e DW_BIND=0.0.0.0:9528 \
    -e DW_PROM_LISTEN=0.0.0.0:9090 \
    -e DW_HTTP_CLIENT_TRACE=true \
    -e DW_LOG_LEVEL=info \
    -e DW_LOG=stdout \
    -e DW_GIN_LOG=stdout \
    -e DW_DISKCACHE_CAPACITY_MB=65536 \
    pubrepo.guance.com/dataflux/dataway:1.21.0

• Verify Dataway installation

After installation, wait for a while to refresh the "Data Gateway" page, if you see the version number in the "Version Information" column of the data gateway you just added, it means that the Dataway has been successfully connected to the Guance center, and front-end users can access data through it.

After Dataway is successfully connected to the Guance center, log in to the Guance console, view all Dataway addresses on the "Integration" / DataKit page, select the required Dataway gateway address, and obtain the DataKit installation command to execute on the server to start collecting data.

Manage DataWay¶

Delete DataWay¶

On the "Data Gateway" page of the Guance management background, select the DataWay to be deleted, click "Configure", and click the "Delete" button in the lower left corner of the pop-up Edit DataWay dialog box.

Upgrade DataWay¶

On the Data Gateways page in the Guance management background, if an upgradeable version exists for DataWay, an upgrade prompt appears in the version information.

DW_UPGRADE=1 bash -c "$(curl https://static.guance.com/dataway/install.sh)"

- image: pubrepo.guance.com/dataflux/dataway:1.21.0

Dataway Service Management¶

When the host installs Dataway, you can use the following command to manage the Dataway service:

# Start
$ systemctl start dataway

# Reboot
$ systemctl restart dataway

# Stop
$ systemctl stop dataway

Kubernetes can restart the corresponding pod.

Environment variable¶

Docker image environment variable¶

When Dataway runs in a Kubernetes environment, it supports the following environment variables.

HTTP Server Settings¶

HTTP concurrency is limited before request bodies are read, using kodo_queue.workers (environment variable DW_KODO_QUEUE_WORKERS), which defaults to 256; missing or non-positive values use this default. When the limit is reached, up to 128 additional requests may wait for at most 1s without reading their bodies. A position is released when request handling completes. When the waiting queue is full or a wait times out, registered DK-compatible endpoints return 503, and all other endpoints return 429. Both responses include Retry-After: 1, and these requests are not written to disk cache. This status-code distinction applies only to HTTP concurrency admission rejections; errors from subsequent forwarding and caching are unchanged. All requests with a body share the same concurrency limit; requests without a body do not count toward it. HTTP concurrency and Kodo workers share the configured count but are not bound to individual workers. Disabling the Kodo dispatch queue does not disable the HTTP concurrency limit. This limit does not guarantee a bound on total Pod memory.

When HTTP concurrency admission rejects an HTTP/1.x request, Dataway stops reading the remaining body and closes the connection after sending the response. Clients must establish a new connection when retrying.

Inbound HTTP requests use the following deadlines, independently of the downstream http_timeout:

Request body read timeouts return retryable 503 responses (dataway.httpBodyReadTimeout), release admission slots, and do not cache incomplete data. A streaming downstream may already have received part of the data; clients must still retry.

The request header read deadline remains 10s. The write deadline should exceed the read deadline plus expected processing time. These are network I/O deadlines and do not forcibly stop CPU computation.

RUM performs local token validation before reading the first body byte. Missing tokens or tokens rejected by local configuration immediately receive the corresponding authentication error. For HTTP/1.x, Dataway stops reading the remaining body and closes the connection without waiting for the upload deadline. Existing compatibility rules for allowed empty tokens, internal tokens, and secret tokens on non-sink requests remain unchanged. Aggregation and tail-sampling proxies return 502 (dataway.proxyFailed) when the backend connection fails; responses received from the backend retain their status codes, including empty responses.

All three settings must be positive. Non-positive YAML values and invalid or non-positive ENV values produce warnings and retain default deadlines; invalid YAML durations fail configuration loading. Increase read/write deadlines for legitimate large uploads or slow networks based on observed upload durations. See the Go HTTP Server documentation for deadline semantics.

DK-compatible endpoints are matched against the following list, not identified by token, query parameters, or User-Agent. Third-party clients calling the same endpoints also receive 503. The list must be maintained when DK adds endpoints; not every /v1/write/* endpoint is a DK endpoint.

Cloudflare's /v1/write/cf-logpush, AWS Firehose's /v1/input/firehose, OTLP's /otel/v1/*, Langfuse's /api/public/*, and other unlisted endpoints return 429 when admission is rejected. Health checks, configuration pulls, and other requests without a body still bypass the concurrency limit.

HTTP TLS Settings¶

To generate a TLS certificate with a one-year validity period, you can use the following OpenSSL command:

# Generate a TLS certificate with a one-year validity
$ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out tls.crt -keyout tls.key
...

After executing this command, the system will prompt you to enter some necessary information, including your country, region, city, organization name, department name, and your email address. This information will be included in your certificate.

After completing the information input, you will generate two files: tls.crt (certificate file) and tls.key (private key file). Please keep your private key file secure and safe.

In order for the application to use these TLS certificates, you need to set the absolute paths of these two files to the application's environment variables. Here is an example of setting environment variables:

DW_ENABLE_TLS must be turned on first, and then the other two ENVs (DW_TLS_CRT/DW_TLS_KEY) will take effect. Version-1.4.1

env:
- name: DW_ENABLE_TLS
  value: "true"
- name: DW_TLS_CRT
  value: "/path/to/your/tls.crt"
- name: DW_TLS_KEY
  value: "/path/to/your/tls.key"

Replace /path/to/your/tls.crt and /path/to/your/tls.key with the actual paths to your tls.crt and tls.key files, respectively.

After setting up the environment variables, you can test whether TLS is functioning correctly with the following command:

curl -k https://localhost:9528

If successful, you should see an ASCII art message indicating "It's working!" If the certificates are missing, you might encounter an error in the Dataway logs similar to:

server listen(TLS) failed: open /path/to/your/tls.{crt,key}: no such file or directory

In this case, Dataway would not start, and the curl command would also result in an error:

curl: (7) Failed to connect to localhost port 9528 after 6 ms: Couldn't connect to server

Ensure that the paths to the TLS certificate and key are correctly specified and that the files have the appropriate permissions for the application to read them.

Log Settings¶

Token/UUID Settings¶

Sinker Settings¶

Prometheus Metrics Exposure¶

Disk Cache Settings¶

Performance Settings¶

Version-1.6.0

Dataway API List¶

Except for diagnostic and locally handled APIs, forwarding APIs validate tokens according to their own rules and forward requests to Kodo or the next Dataway. See the Langfuse compatibility API section below for its authentication behavior.

GET /v1/ping¶

Version-1.11.0

• API description: Get current dataway version and release date, also return the public IP address of your request

If 404 page disabled(disable_404page), this API not working.

GET /v1/ntp¶

Version-1.6.0

• API description: Get current Dataway unix timestamp(unit: second)

POST /v1/write/:category¶

• API description: Receive various collection data uploaded by Datakit
• Downstream 4xx: Ordinary writes and OCSF forwarding share error handling that preserves the HTTP status. A 404 response returns {}. For other 4xx responses, Dataway extracts error_code from a JSON error body it can decode and omits other error details. An empty body, a non-JSON body, or an undecodable error_code results in {}; failure to decode an error body no longer changes a 4xx response to 500. Downstream 4xx responses do not trigger disk caching.

POST /v1/ocsf_ingest¶

Accepts OCSF JSON, converts events into logging points on the current Dataway node, and writes them to the fixed ocsf index through /v1/write/logging, without requiring aggregation or tail sampling to be enabled.

• Requests use Content-Type: application/json; charset parameters are accepted. Uncompressed bodies, Content-Encoding: identity, and gzip are supported. NDJSON and other compression encodings are not supported.
• Supports a single event object, an event array, or an {"events":[...]} bundle. A top-level object containing class_uid is treated as a single event; otherwise it is treated as a bundle. Outer bundle metadata does not produce points; the point count reflects the actual events.
• Authentication runs before reading the body. Token precedence is the token query parameter, X-Token, then X-Amz-Firehose-Access-Key. Empty tokens, internal tokens, and secret tokens follow existing configuration and sink compatibility rules.
• Each event must contain a SemVer 1.x metadata.version, a non-negative integer class_uid, and an integer millisecond time. The timestamp must fit in int64 after conversion to nanoseconds. A batch may mix 1.x versions and event classes. Full OCSF Schema validation and external Schema service requests are not performed.
• The request body, decompressed gzip JSON, and converted protobuf batch are each subject to max_http_body_bytes (DW_MAX_HTTP_BODY_BYTES), default 64 MiB. Exceeding a limit rejects the entire batch with 413; the sender must reduce the batch size. Dataway does not split batches automatically.

Point mapping and field limits¶

Each event produces one point with measurement/source ocsf, its timestamp taken from the event's time, and logging status fixed to info. The original OCSF severity and event outcome do not change the logging level.

Each point supports at most 1024 fields, including the generated message and status. Final field names are limited to 256 bytes, and individual string values to 32 MiB, including message. Exceeding a limit rejects the entire batch with 413; fields and values are not truncated automatically.

Requests and responses¶

Example OCSF 1.3.0 Authentication event:

curl -X POST 'https://dataway.example.com/v1/ocsf_ingest?token=<token>' \
  -H 'Content-Type: application/json' \
  --data-binary '{"metadata":{"version":"1.3.0","product":{"name":"example","vendor_name":"example"}},"class_uid":3002,"category_uid":3,"type_uid":300201,"activity_id":1,"severity_id":1,"time":1713632701123,"user":{"name":"alice"},"service":{"name":"ssh"}}'

This request produces one point named ocsf with timestamp 1713632701123000000, fields user.name=alice and service.name=ssh, and the complete message. Any caller-specified X-Storage-Index-Name is overridden with ocsf; cache replay retains that index.

Dataway validates and converts the entire batch before sending it synchronously through the existing Kodo, sink, signing, cache, and replay paths:

Authentication errors and delivery failures that cannot be cached follow the existing paths. This endpoint adds no asynchronous ingestion queue and does not guarantee deduplication or atomic downstream batch writes. Existing HTTP metrics report ingress status and latency; send and cache metrics use /v1/write/logging.

Duplicate JSON keys and field conflicts¶

When a key occurs more than once within the same JSON object, the value of its last occurrence in the original request is retained. Duplicate keys alone do not cause a 400 response. This rule applies to the outer bundle, event objects, nested objects, and objects inside arrays; separate objects do not overwrite each other. Keys are compared after decoding JSON escapes; for example, class_uid and class\u005fuid are treated as the same name. If a repeated key has an object or array value, the later value replaces the earlier value entirely; values are not merged.

Field validation and point mapping use the final retained values. The point's message preserves the complete original event JSON, including duplicate keys and their original values; it is not replaced with deduplicated JSON. If a final retained value violates a field requirement, such as a negative class_uid, the entire batch still returns 400 for an invalid event.

The following single-event request passes conversion validation and produces one point with class_uid=4001 and user.name=bob. Its message retains the complete original JSON below:

{
  "metadata": {"version": "1.3.0"},
  "class_uid": 3002,
  "class_uid": 4001,
  "time": 1713632701123,
  "user": {"name": "alice", "name": "bob"}
}

Field conflicts caused by renaming or flattening into dotted paths still reject the entire batch with 400, and no points are sent. For example, in {"a.b":1,"a":{"b":2}}, two distinct paths both become a.b after flattening, causing a field mapping conflict. The error description identifies the conflicting field path and the event's position in the batch (starting at 1).

POST /v1/aggregate¶

• API description: Receives aggregated metric data. This route is registered only when the aggregate feature is initialized successfully.
• For the operating modes, configuration, and request requirements, see Dataway Metric Aggregation.

POST /v1/tail_sampling, POST /v1/tail_sampling_v2, POST /v2/tail_sampling, and POST /v1/tail_sampling_config¶

• API description: /v1/tail_sampling accepts raw tail-sampling packets and also accepts headerless zstd packets sent by Datakit 2.10; the legacy /v1/tail_sampling_v2 path accepts raw packets only; /v2/tail_sampling accepts zstd packets with the compression-negotiation header; /v1/tail_sampling_config accepts tail-sampling configuration. These routes are registered only when the tail-sampling feature is initialized successfully.
• For the operating modes, configuration, and request requirements, see Dataway Tail Sampling.

POST /otel/v1/{traces,metrics,logs}¶

Version-1.17.0

• API description: Receive OpenTelemetry OTLP HTTP/protobuf data and forward it to the matching Kodo path without rewriting the payload.
• Supported paths:
  • POST /otel/v1/traces
  • POST /otel/v1/metrics
  • POST /otel/v1/logs

• Request requirements:

  • Content-Type must be application/x-protobuf; media type parameters are accepted, for example application/x-protobuf; charset=utf-8
  • The workspace token can be provided through the token query, the X-Token header, or Authorization: Bearer <token>; Bearer tokens are converted to X-Token before forwarding
  • When using OpenTelemetry SDK/Agent environment variables, explicitly configure OTLP HTTP/protobuf and set the base endpoint to the Dataway /otel prefix; OpenTelemetry appends /v1/traces, /v1/metrics, and /v1/logs for each signal:

  export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
  export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://dataway.example.com/otel/v1/logs
  export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://dataway.example.com/otel/v1/traces
  export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://dataway.example.com/otel/v1/metrics
  export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20{token}"
  

  • If per-signal endpoints are used, configure the full Dataway paths, for example OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://dataway.example.com/otel/v1/traces, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://dataway.example.com/otel/v1/metrics, and OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://dataway.example.com/otel/v1/logs
  • Startup options have the same meaning as the environment variables. For example, with the Java Agent, configure -Dotel.exporter.otlp.protocol=http/protobuf, -Dotel.exporter.otlp.endpoint=https://dataway.example.com/otel, and -Dotel.exporter.otlp.headers=Authorization=Bearer%20{token}. If an implementation handles the encoded space in the Authorization header differently, use OTEL_EXPORTER_OTLP_HEADERS="X-Token={token}" or the matching startup option instead.
  • Dataway does not parse or rewrite the OTLP body; /otel/v1/metrics is not converted into /v1/write/metric line protocol. Kodo parses it with DataKit's default OTLP metrics rules: the measurement is otel_service, and metric field names keep the original OTLP metric names, for example runtime.jvm.memory is not rewritten to runtime_jvm_memory
  • Kodo's raw OTLP response status, response body, and required response headers are passed through; when forwarding fails and matches the disk-cache policy, the request enters Dataway disk-cache replay

GET /v1/datakit/pull¶

• API description: Handles Datakit pull center configuration (blacklist/pipeline) requests

POST /v1/datakit/usage_trace¶

• API description: Receives Datakit usage tracing data.

POST /v1/write/rum/replay¶

• API description: Receive Session Replay data uploaded by Datakit

POST /v1/write/rum/replay_assets¶

• API description: Receives Session Replay asset upload requests.

POST /v1/check/rum/replay_assets¶

• API description: Checks whether a Session Replay asset already exists.

POST /v1/upload/profiling¶

• API description: Receive profiling data uploaded by Datakit

POST /v1/input/firehose¶

• API description: Receives Firehose input data.

POST /v1/election¶

• API description: Handles election requests for Datakit

POST /v1/election/heartbeat¶

• API description: Handles election heartbeat requests for Datakit

POST /v1/query/raw¶

• API description: Handles DQL query requests initiated by the Datakit side

POST /v1/workspace¶

• API description: Handles workspace query requests initiated by Datakit

GET /v1/env_variable¶

• API description: Retrieves workspace environment variables.

POST /v1/dataway/heartbeat¶

• API description: Receives Dataway heartbeat reports.

POST /v1/object/labels¶

• API description: Handles requests to modify object labels

DELETE /v1/object/labels¶

• API description: Handles delete object Label requests

GET /v1/check/token/:token¶

• API description: Checks whether a token is valid.

Langfuse Compatibility APIs¶

The following APIs are provided for Langfuse client compatibility. Dataway forwards these requests upstream without token validation:

• POST /api/public/otel/v1/traces: Receives Langfuse OTLP trace data.
• GET /api/public/projects: Retrieves Langfuse project information.
• POST /api/public/ingestion: Receives Langfuse ingestion data. When the upstream accepts a batch but returns an empty body, Dataway builds a Langfuse SDK 4.x-compatible successes/errors JSON response from the event IDs in the request batch.

Dataway metrics collection¶

[[inputs.prom]]
  ## Exporter URLs.
  urls = [ "http://localhost:9090/metrics", ]

  source = "dataway"

  election = true

  ## Dataway metric set fixed to dw, do not change
  measurement_name = "dw"

annotations:
   datakit/prom.instances: |
     [[inputs.prom]]
       url = "http://$IP:9090/metrics" # Here the port (default 9090) is as appropriate
       source = "dataway"
       measurement_name = "dw" # pinned to this metric set
       interval = "30s"

       [inputs.prom.tags]
         namespace = "$NAMESPACE"
         pod_name = "$PODNAME"
         node_name = "$NODENAME"

If the collection is successful, search for dataway in the "Scene"/"Built-in View" of the Guance to see the corresponding monitoring view.

Cache and Kodo Queue Troubleshooting¶

When Kodo returns 5xx, Dataway fails to send a request to Kodo, or the Kodo dispatch queue is full, Dataway tries to write eligible requests into the disk cache. Kodo 4xx responses are not cached.

Use these metrics first when troubleshooting:

• dataway_http_api_cached_bytes{reason=...}: the request was written to cache. Common reason values include Kodo 5xx status text, httpCliDoFailed, and kodoQueueFull.
• dataway_http_api_dropped_cache{reason=...}: Dataway tried to cache the request but did not write it. Common reason values include feat-disabled, expected-api-drop, body-md5-not-matched, body-length-not-matched, expired, put-failed, and sendopt-nil.
• dataway_kodo_queue_full_total{action=...}: how Dataway handled a full Kodo queue. action="cache" means the request was moved to cache; action="reject" means Dataway returned 503 after cache was unavailable or cache write failed.
• dataway_http_concurrency_in_flight / dataway_http_concurrency_waiting: current requests admitted for processing and requests still waiting before their bodies are read.
• dataway_http_concurrency_requests_total{result=...}: HTTP concurrency outcomes, including admitted, waiting_full, wait_timeout, and canceled.

Dataway Metric List¶

The following are the indicators exposed by Dataway, which can be obtained by requesting http://localhost:9090/metrics, and you can view (3s) a specific indicator in real time by following the following command:

If some metrics cannot be queried, it may be caused by the relevant business module not running.

watch -n 3 'curl -s http://localhost:9090/metrics | grep -a <METRIC-NAME>'

Metrics under Docker¶

There are two modes for non-Kubernetes, host mode and Docker mode. This section will specifically discuss the differences in metrics collection when installing in Docker.

When installed in docker, the HTTP port that exposes metrics will be mapped to port 19090 on the host machine (by default). In this case, the metrics collection address is http://localhost:19090/metrics.

If a different port is specified, the installer will add 10000 to the specified port during installation. Therefore, the specified port should not exceed 45535.

pprof is disabled by default, and Docker installations do not publish a profiling port to the host. For temporary diagnosis, explicitly set pprof_bind or DW_PPROF_BIND to localhost:6060, then collect through docker exec, kubectl exec, or a controlled port forward. Non-loopback addresses are rejected.

Dataway's Own Log Collection and Processing¶

Dataway's own logs are divided into two categories: one is the gin log, and the other is the Dataway's own log. The following Pipeline can separate them:

# Pipeline for dataway logging

# Testing sample loggin
'''
2023-12-14T11:27:06.744+0800    DEBUG   apis    apis/api_upload_profile.go:272  save profile file to disk [ok] /v1/upload/profiling?token=****************a4e3db8481c345a94fe5a
[GIN] 2021/10/25 - 06:48:07 | 200 |   30.890624ms |  114.215.200.73 | POST     "/v1/write/logging?token=tkn_5c862a11111111111111111111111"
'''

add_pattern("TOKEN", "tkn_\\w+")
add_pattern("GINTIME", "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}-%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}")
grok(_,"\\[GIN\\]%{SPACE}%{GINTIME:timestamp}%{SPACE}\\|%{SPACE}%{NUMBER:dataway_code}%{SPACE}\\|%{SPACE}%{NOTSPACE:cost_time}%{SPACE}\\|%{SPACE}%{NOTSPACE:client_ip}%{SPACE}\\|%{SPACE}%{NOTSPACE:method}%{SPACE}%{GREEDYDATA:http_url}")

# gin logging
if cost_time != nil {
  if http_url != nil  {
    grok(http_url, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(http_url, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  group_between(dataway_code, [200,299], "info", status)
  group_between(dataway_code, [300,399], "notice", status)
  group_between(dataway_code, [400,499], "warning", status)
  group_between(dataway_code, [500,599], "error", status)

  if sample(0.1) { # drop 90% debug log
    drop()
    exit()
  } else {
    set_tag(sample_rate, "0.1")
  }

  parse_duration(cost_time)
  duration_precision(cost_time, "ns", "ms")

  set_measurement('gin', true)
  set_tag(service,"dataway")
  exit()
}

# app logging
if cost_time == nil {
  grok(_,"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NOTSPACE:status}%{SPACE}%{NOTSPACE:module}%{SPACE}%{NOTSPACE:code}%{SPACE}%{GREEDYDATA:msg}")
  if level == nil {
    grok(message,"Error%{SPACE}%{DATA:errormsg}")
    if errormsg != nil {
      add_key(status,"error")
      drop_key(errormsg)
    }
  }
  lowercase(level)

  # if debug level enabled, drop most of them
  if status == 'debug' {
    if sample(0.1) { # drop 90% debug log
      drop()
      exit()
    } else {
      set_tag(sample_rate, "0.1")
    }
  }

  group_in(status, ["error", "panic", "dpanic", "fatal","err","fat"], "error", status) # mark them as 'error'

  if msg != nil {
    grok(msg, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(msg, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  set_measurement("dataway-log", true)
  set_tag(service,"dataway")
}

Dataway Bug Report¶

Dataway exposes metrics and can temporarily enable a profiling endpoint for troubleshooting. pprof is disabled by default; set pprof_bind to a loopback address before collection and disable it again afterward.

Use the actual configured addresses and ports when collecting the following information.

br_dir="dw-br-$(date +%s)"
mkdir -p $br_dir

echo "Save bug report to ${br_dir}"

# Modify the following configurations according to your actual situation
dw_ip="localhost" # The IP address where Dataway's metrics/profile is exposed
metric_port=9090  # The port where metrics are exposed
profile_port=6060 # Explicitly enable the localhost:6060 pprof listener first
dw_yaml_conf="/usr/local/cloudcare/dataflux/dataway/dataway.yaml"
dw_dot_yaml_conf="/usr/local/cloudcare/dataflux/dataway/.dataway.yaml"

# Collect runtime metrics
curl -v "http://${dw_ip}:${metric_port}/metrics" -o $br_dir/metrics

# Collect profiling information
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/allocs" -o $br_dir/allocs
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/heap" -o $br_dir/heap
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/profile" -o $br_dir/profile # This command will take about 30 seconds to run

cp $dw_yaml_conf $br_dir/dataway.yaml.copy
cp $dw_dot_yaml_conf $br_dir/.dataway.yaml.copy

tar czvf ${br_dir}.tar.gz ${br_dir}
rm -rf ${br_dir}

Run the script:

$ sh dw-bug-report.sh
...

After execution, a file similar to dw-br-1721188604.tar.gz will be generated. You can then retrieve this file for further use.

FAQ¶

Request Entity Too Large Issue¶

Version-1.3.7

Dataway has a default setting for the size of the request body (default is 64MB), but when the request body is too large, the client will receive an HTTP 413 error (Request Entity Too Large). If the request body is within a reasonable range, you can appropriately increase this value (unit is bytes):

• Set the environment variable DW_MAX_HTTP_BODY_BYTES for Kubernetes Pod install
• In dataway.yaml, set max_http_body_bytes for host install

If there is a request that is too large during runtime, it is reflected in both metrics and logs:

• The metric dataway_http_too_large_dropped_total exposes the number of discarded large requests
• Search the Dataway logs with cat log | grep 'drop too large request'. The logs will output the details of the HTTP request Header, which is helpful for further understanding the client situation