Dataway¶

Introduction¶

DataWay is the data gateway of Guance. All data reported by collectors to Guance must pass through the DataWay gateway.

Dataway Installation¶

• Create Dataway

In the Guance management console, navigate to the "Data Gateway" page, click "Create Dataway". Enter the name and binding address, then click "Create".

After successful creation, a new Dataway will be created and the installation script for Dataway will be generated.

• Install Dataway

DW_KODO=http://kodo_ip:port \
   DW_TOKEN=<tkn_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
   DW_UUID=<YOUR_UUID> \
   bash -c "$(curl https://static.guance.com/dataway/install.sh)"

# ============= DATAWAY CONFIG =============

# Dataway UUID, we can get it on during create a new dataway
uuid:

# It's the workspace token, most of the time, it's
# system worker space's token.
token:

# secret_token used under sinker mode, and to check if incomming datakit
# requests are valid.
secret_token:

# If __internal__ token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_internal_token: false

# is empty token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_empty_token: false

# Is dataway cascaded? For cascaded Dataway, it's remote_host is
# another Dataway and not Kodo.
cascaded: false

# kodo(next dataway) related configures
remote_host:
http_timeout: 3s

http_max_idle_conn_perhost: 0 # default to CPU cores
http_max_conn_perhost: 0      # default no limit
kodo_queue:
  enabled: true
  workers: 256
  queue_size: 1024
  queue_max_bytes: 1073741824 # 1GB
  enqueue_timeout: 100ms

insecure_skip_verify: false
http_client_trace: false
sni: ""

# dataway API configures
bind: 0.0.0.0:9528

# disable 404 page
disable_404page: false

# dataway TLS file path
tls_crt:
tls_key:

# enable pprof
pprof_bind: localhost:6060

api_limit_rate : 100000         # 100K
max_http_body_bytes : 67108864  # 64MB
copy_buffer_drop_size : 262144  # 256KB, if copy buffer memory larger than this, this memory released
reserved_pool_size: 4096        # reserved pool size for better GC

within_docker: false

log_level: info
log: log
gin_log: gin.log

ip_blacklist:
  ttl = "1m"
  clean_interval = "1h"

cache_cfg:
  # cache disk path
  dir: "disk_cache"

  # disable cache
  disabled: false

  clean_interval: "1s"

  # in MB, max single data package size in disk cache, such as HTTP body
  max_data_size: 100

  # in MB, single disk-batch(single file) size
  batch_size: 128

  # in MB, max disk size allowed to cache data
  max_disk_size: 65535

  # expire duration, default 7 days
  expire_duration: "168h"

prometheus:
  listen: "localhost:9090"
  url: "/metrics"
  enable: true

#sinker:
#  cache_options:
#    prealloc: true
#    reserved_capacity: 10000000 # max cached items
#    buckets: 64
#    ttl: 10m # clear unactive matches
#  etcd:
#    urls:
#    - http://localhost:2379 # one or multiple etcd host
#    dial_timeout: 30s
#    key_space: "/dw_sinker" # subscribe to the etcd key
#    username: "dataway"
#    password: "<PASSWORD>"
#  file:
#    path: /path/to/sinker.json

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: sts-utils-dataway
  name: dataway
  namespace: utils
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sts-utils-dataway
  serviceName: dataway
  template:
    metadata:
      annotations:
        datakit/logs: |
          [
            {
              "disable": false,
              "source": "dataway",
              "service": "dataway",
              "multiline_match": "^\\d{4}|^\\[GIN\\]"
            }
          ]
        datakit/prom.instances: |
          [[inputs.prom]]
            url = "http://$IP:9090/metrics"

            source = "dataway"
            measurement_name = "dw"
            interval = "10s"
            disable_instance_tag = true
          [inputs.prom.tags]
            service = "dataway"
            instance = "$PODNAME" # we can set as "xxx-$PODNAME"
      labels:
        app: sts-utils-dataway
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - sts-utils-dataway
              topologyKey: kubernetes.io/hostname
      containers:
        - env:
            - name: DW_REMOTE_HOST
              value: http://kodo.forethought-kodo:9527
            - name: DW_BIND
              value: 0.0.0.0:9528
            - name: DW_UUID
              value: agnt_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx   # Dataway UUID
            - name: DW_TOKEN
              value: tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  # Dataway token
            - name: DW_PROM_LISTEN
              value: 0.0.0.0:9090
            - name: DW_LOG
              value: stdout
            - name: DW_LOG_LEVEL
              value: info
            - name: DW_GIN_LOG
              value: stdout
            - name: DW_DISKCACHE_DIR
              value: cache
            - name: DW_HTTP_TIMEOUT
              value: '3s'
            - name: DW_ENABLE_INTERNAL_TOKEN
              value: 'false'
            - name: DW_MAX_HTTP_BODY_BYTES
              value: '67108864'
            - name: DW_HTTP_CLIENT_TRACE
              value: 'on'
            - name: DW_RESERVED_POOL_SIZE
              value: '0'
            - name: DW_COPY_BUFFER_DROP_SIZE
              value: '262144'
            - name: DW_DISKCACHE_CAPACITY_MB
              value: 102400
          image: pubrepo.guance.com/dataflux/dataway:1.15.0
          imagePullPolicy: IfNotPresent
          name: dataway
          ports:
            - containerPort: 9528
              name: 9528tcp01
              protocol: TCP
          resources:
            limits:
              cpu: '4'
              memory: 4Gi
            requests:
              cpu: 100m
              memory: 512Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /usr/local/cloudcare/dataflux/dataway/cache
              name: dataway-cache
      dnsPolicy: ClusterFirst
      imagePullSecrets: []
      #nodeSelector:
      #  nodepool: dataway
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      #tolerations:
      #  - effect: NoSchedule
      #    key: nodepool
      #    operator: Equal
      #    value: dataway
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: dataway-cache
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: xxxxxx  # High-Performance Storage StorageClass
        volumeMode: Filesystem
      status:
        phase: Pending

---

apiVersion: v1
kind: Service
metadata:
  name: dataway
  namespace: utils
spec:
  ports:
    - name: 9528tcp02
      nodePort: 30928
      port: 9528
      protocol: TCP
      targetPort: 9528
  selector:
    app: sts-utils-dataway
  type: NodePort

containers:
  volumeMounts:
    - name: dataway-config
      mountPath: /usr/local/cloudcare/dataflux/dataway/dataway.yaml
      subPath: config.yaml
volumes:
- configMap:
    defaultMode: 256
    name: dataway-config
    optional: false
  name: dataway-config

docker run -d \
    --name <YOUR-DW-IN-DOCKER> \
    -p 19528:9528 -p 19090:9090 \
    --mount type=bind,source=<host/path/for/diskcache>,target=/usr/local/cloudcare/dataflux/dataway/cache \
    --memory=2g --memory-reservation=256m \
    --cpus="2" \
    -e DW_UUID=<YOUR-AGNT_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
    -e DW_TOKEN=<YOUR-TKN_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
    -e DW_REMOTE_HOST=http://kodo.forethought-kodo:9527 \
    -e DW_BIND=0.0.0.0:9528 \
    -e DW_PROM_LISTEN=0.0.0.0:9090 \
    -e DW_HTTP_CLIENT_TRACE=true \
    -e DW_LOG_LEVEL=info \
    -e DW_LOG=stdout \
    -e DW_GIN_LOG=stdout \
    -e DW_DISKCACHE_CAPACITY_MB=65536 \
    pubrepo.guance.com/dataflux/dataway:1.15.0

• Verify Dataway Installation

After installation, wait a moment and refresh the "Data Gateway" page. If you see a version number in the "Version Information" column of the newly added data gateway, it means this Dataway has successfully connected to the Guance center. Frontend users can then use it to ingest data.

After Dataway successfully connects to the Guance center, log in to the Guance console. On the "Integration" / "DataKit" page, you can view all Dataway addresses. Select the required Dataway gateway address, obtain the DataKit installation command, and execute it on the server to start collecting data.

Manage DataWay¶

Delete DataWay¶

In the Guance management console "Data Gateway" page, select the DataWay to be deleted, click "Configure". In the pop-up edit DataWay dialog, click the "Delete" button at the bottom left.

Upgrade DataWay¶

In the Guance management console "Data Gateway" page, if an upgrade is available for DataWay, an upgrade prompt will appear in the version information.

DW_UPGRADE=1 bash -c "$(curl https://static.guance.com/dataway/install.sh)"

- image: pubrepo.guance.com/dataflux/dataway:1.15.0

Dataway Service Management¶

When Dataway is installed on a host, use the following commands to manage the Dataway service.

# Start
$ systemctl start dataway

# Restart
$ systemctl restart dataway

# Stop
$ systemctl stop dataway

For Kubernetes, restart the corresponding Pod.

Environment Variables¶

Image Environment Variables¶

When Dataway runs in a Kubernetes environment, the following environment variables are supported.

HTTP Server Settings¶

HTTP TLS Settings¶

To generate a TLS certificate valid for one year, you can use the following OpenSSL command:

# Generate a TLS certificate valid for one year
$ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out tls.crt -keyout tls.key
...

After executing this command, you will be prompted to enter some necessary information, including your country, region, city, organization name, department name, and your email address. This information will be included in your certificate.

After completing the information input, you will generate two files: tls.crt (certificate file) and tls.key (private key file). Please keep your private key file secure and ensure its safety.

To enable the application to use these TLS certificates, you need to set the absolute paths of these two files into the application's environment variables. Here is an example of setting environment variables:

DW_ENABLE_TLS must be enabled first, then the other two ENVs (DW_TLS_CRT/DW_TLS_KEY) will take effect. Version-1.4.1

env:
- name: DW_ENABLE_TLS
  value: "true"
- name: DW_TLS_CRT
  value: "/path/to/your/tls.crt"
- name: DW_TLS_KEY
  value: "/path/to/your/tls.key"

Replace /path/to/your/tls.crt and /path/to/your/tls.key with the actual paths where your tls.crt and tls.key files are stored.

After setting, you can test if TLS is effective with the following command:

$ curl -k http://localhost:9528

If successful, an ASCII Art message It's working! will be displayed. If the certificate does not exist, Dataway logs will have an error similar to the following:

server listen(TLS) failed: open /path/to/your/tls.{crt,key}: no such file or directory

At this point, Dataway cannot start, and the above curl command will also report an error:

$ curl -vvv -k http://localhost:9528
curl: (7) Failed to connect to localhost port 9528 after 6 ms: Couldn't connect to server

Logging Settings¶

Token/UUID Settings¶

Sinker Settings¶

Prometheus Metrics Exposure¶

Disk Cache Settings¶

Performance Related Settings¶

Version-1.6.0

Dataway API List¶

Details for each API below are to be supplemented.

GET /v1/ping¶

Version-1.11.0

• API Description: Get the current version number and release date of Dataway, and return the egress IP of the client request.

If DataWay disables the 404 page (disable_404page), this interface will not be able to provide service.

GET /v1/ntp¶

Version-1.6.0

• API Description: Get the current Unix timestamp (in seconds) of Dataway

POST /v1/write/:category¶

• API Description: Receive various collection data uploaded by Datakit

GET /v1/datakit/pull¶

• API Description: Handle Datakit requests to pull center configuration (blacklist/Pipeline)

POST /v1/write/rum/replay¶

• API Description: Receive Session Replay data uploaded by Datakit

POST /v1/upload/profiling¶

• API Description: Receive Profiling data uploaded by Datakit

POST /v1/election¶

• API Description: Handle Datakit election requests

POST /v1/election/heartbeat¶

• API Description: Handle Datakit election heartbeat requests

POST /v1/query/raw¶

Handle DQL query requests, simple example as follows:

POST /v1/query/raw?token=<workspace-token> HTTP/1.1
Content-Type: application/json

{
    "token": "workspace-token",
    "queries": [
        {
            "query": "M::cpu LIMIT 1"
        }
    ],
    "echo_explain": <true/false>
}

Return example:

{
  "content": [
    {
      "series": [
        {
          "name": "cpu",
          "columns": [
            "time",
            "usage_iowait",
            "usage_total",
            "usage_user",
            "usage_guest",
            "usage_system",
            "usage_steal",
            "usage_guest_nice",
            "usage_irq",
            "load5s",
            "usage_idle",
            "usage_nice",
            "usage_softirq",
            "global_tag1",
            "global_tag2",
            "host",
            "cpu"
          ],
          "values": [
            [
              1709782208662,
              0,
              7.421875,
              3.359375,
              0,
              4.0625,
              0,
              0,
              0,
              1,
              92.578125,
              0,
              0,
              null,
              null,
              "WIN-JCHUL92N9IP",
              "cpu-total"
            ]
          ]
        }
      ],
      "points": null,
      "cost": "24.558375ms",
      "is_running": false,
      "async_id": "",
      "query_parse": {
        "namespace": "metric",
        "sources": {
          "cpu": "exact"
        },
        "fields": {},
        "funcs": {}
      },
      "index_name": "",
      "index_store_type": "",
      "query_type": "guancedb",
      "complete": false,
      "index_names": "",
      "scan_completed": false,
      "scan_index": "",
      "next_cursor_time": -1,
      "sample": 1,
      "interval": 0,
      "window": 0
    }
  ]
}

Return result description:

• Real data is located in the inner series field
• name indicates the measurement name (here querying CPU metrics, if it's log data, this field is not present)
• columns indicates the names of the returned result columns
• values contains the corresponding column results for columns

POST /v1/workspace¶

• API Description: Handle workspace query requests initiated by Datakit

POST /v1/object/labels¶

• API Description: Handle requests to modify object Labels

DELETE /v1/object/labels¶

• API Description: Handle requests to delete object Labels

GET /v1/check/:token¶

• API Description: Check if token is valid

Dataway Metrics Collection¶

[[inputs.prom]]
  ## Exporter URLs.
  urls = [ "http://localhost:9090/metrics", ]
  source = "dataway"
  election = true
  measurement_name = "dw" # dataway measurement is fixed as dw, do not change
[inputs.prom.tags]
  service = "dataway"

annotations: # The following annotations are added by default
   datakit/prom.instances: |
     [[inputs.prom]]
       url = "http://$IP:9090/metrics" # This port (default 9090) depends on the situation
       source = "dataway"
       measurement_name = "dw" # Fixed as this measurement
       interval = "10s"
       disable_instance_tag = true

     [inputs.prom.tags]
       service = "dataway"
       instance = "$PODNAME"

...
env:
- name: DW_PROM_LISTEN
  value: "0.0.0.0:9090" # Keep this port consistent with the port in the url above

If collection is successful, search for dataway in Guance "Scenarios" / "Built-in Views" to see the corresponding monitoring view.

Dataway Metrics List¶

The following are the metrics exposed by Dataway. These metrics can be obtained by requesting http://localhost:9090/metrics. You can view a specific metric in real-time (3s) with the following command:

If some metrics cannot be queried, it may be because the relevant business module is not running yet. Some new metrics only exist in the latest version. The version information for each metric is not indicated here one by one. Refer to the metric list returned by the /metrics interface.

watch -n 3 'curl -s http://localhost:9090/metrics | grep -a <METRIC-NAME>'

Metrics Collection in Docker Mode¶

There are two modes for host installation: one is installation on the host machine, the other is installation via Docker. Here we specifically explain the differences in metrics collection when installed via Docker.

When installed via Docker, the HTTP port for metrics exposure will be mapped to port 19090 on the host machine (by default). At this point, its metrics collection address is http://localhost:19090/metrics.

If a different port is specified separately, then during Docker installation, 10000 will be added to that port. Therefore, the port specified here should not exceed 45535.

Additionally, during Docker installation, the profile collection port will also be exposed, defaulting to port 16060 on the host machine. Its mechanism is also to add 10000 to the specified port.

Dataway Self Log Collection and Processing¶

Dataway's own Logs are divided into two categories: gin logs and its own program logs. They can be separated using the following Pipeline:

# Pipeline for dataway logging

# Testing sample loggin
'''
2023-12-14T11:27:06.744+0800    DEBUG   apis    apis/api_upload_profile.go:272  save profile file to disk [ok] /v1/upload/profiling?token=****************a4e3db8481c345a94fe5a
[GIN] 2021/10/25 - 06:48:07 | 200 |   30.890624ms |  114.215.200.73 | POST     "/v1/write/logging?token=tkn_5c862a11111111111111111111111111"
'''

add_pattern("TOKEN", "tkn_\\w+")
add_pattern("GINTIME", "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}-%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}")
grok(_,"\\[GIN\\]%{SPACE}%{GINTIME:timestamp}%{SPACE}\\|%{SPACE}%{NUMBER:dataway_code}%{SPACE}\\|%{SPACE}%{NOTSPACE:cost_time}%{SPACE}\\|%{SPACE}%{NOTSPACE:client_ip}%{SPACE}\\|%{SPACE}%{NOTSPACE:method}%{SPACE}%{GREEDYDATA:http_url}")

# gin logging
if cost_time != nil:
  if http_url != nil:
    grok(http_url, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(http_url, "tkn_\\w{0,5}\\w{6}", "****************$4")

  group_between(dataway_code, [200,299], "info", status)
  group_between(dataway_code, [300,399], "notice", status)
  group_between(dataway_code, [400,499], "warning", status)
  group_between(dataway_code, [500,599], "error", status)

  if sample(0.1): # drop 90% debug log
    drop()
    exit()
  else:
    set_tag(sample_rate, "0.1")

  parse_duration(cost_time)
  duration_precision(cost_time, "ns", "ms")

  set_measurement('gin', true)
  set_tag(service,"dataway")
  exit()

# app logging
if cost_time == nil:
  grok(_,"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NOTSPACE:status}%{SPACE}%{NOTSPACE:module}%{SPACE}%{NOTSPACE:code}%{SPACE}%{GREEDYDATA:msg}")
  if level == nil:
    grok(message,"Error%{SPACE}%{DATA:errormsg}")
    if errormsg != nil:
      add_key(status,"error")
      drop_key(errormsg)
  lowercase(level)

  # if debug level enabled, drop most of them
  if status == 'debug':
    if sample(0.1): # drop 90% debug log
      drop()
      exit()
    else:
      set_tag(sample_rate, "0.1")

  group_in(status, ["error", "panic", "dpanic", "fatal","err","fat"], "error", status) # mark them as 'error'

  if msg != nil:
    grok(msg, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(msg, "tkn_\\w{0,5}\\w{6}", "****************$4")

  set_measurement("dataway-log", true)
  set_tag(service,"dataway")

Dataway bug report¶

Dataway itself exposes metrics and profiling collection endpoints. We can collect this information for troubleshooting.

The following information collection is based on the actual configured ports and addresses. The listed commands follow default parameters.

br_dir="dw-br-$(date +%s)"
mkdir -p $br_dir

echo "save bug report to ${br_dir}"

# Modify configurations here according to the actual situation
dw_ip="localhost" # IP address where Dataway metrics/profile are exposed
metric_port=9090  # Port for metrics exposure
profile_port=6060 # Port for profile exposure
dw_yaml_conf="/usr/local/cloudcare/dataflux/dataway/dataway.yaml"
dw_dot_yaml_conf="/usr/local/cloudcare/dataflux/dataway/.dataway.yaml" # This file exists for container installation

# Collect runtime metrics
curl -v "http://${dw_ip}:${metric_port}/metrics" -o $br_dir/metrics

# Collect profiling information
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/allocs" -o $br_dir/allocs
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/heap" -o $br_dir/heap
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/profile" -o $br_dir/profile # This command will run for about 30s

cp $dw_yaml_conf $br_dir/dataway.yaml.copy
cp $dw_dot_yaml_conf $br_dir/.dataway.yaml.copy

tar czvf ${br_dir}.tar.gz ${br_dir}
rm -rf ${br_dir}

Run the script:

$ sh dw-bug-report.sh
...

After execution, a file similar to dw-br-1721188604.tar.gz will be generated. Extract this file.

FAQ¶

Request Body Too Large Issue¶

Version-1.3.7

Dataway has a default setting for request body size (default 64MB). When the request body is too large, the client will receive an HTTP 413 error (Request Entity Too Large). If the request body is within a reasonable range, this value can be appropriately increased (in bytes):

• Set environment variable DW_MAX_HTTP_BODY_BYTES
• Set max_http_body_bytes in dataway.yaml

If excessively large request packets occur during operation, they are reflected in both metrics and logs:

• The metric dataway_http_too_large_dropped_total exposes the number of dropped large requests
• Search Dataway logs cat log | grep 'drop too large request'. The logs will output details of the HTTP request headers, facilitating further understanding of the client situation.