Dataway¶

Introduction¶

DataWay is the data gateway of Guance. All data reported by collectors to Guance must go through the DataWay gateway.

Dataway Installation¶

• Create Dataway

In the Guance management backend under the "Data Gateway" page, click "Create Dataway". Input a name and binding address, then click "Create".

After successful creation, a new Dataway will be automatically created along with the installation script for Dataway.

• Install Dataway

DW_KODO=http://kodo_ip:port \
   DW_TOKEN=<tkn_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
   DW_UUID=<YOUR_UUID> \
   bash -c "$(curl https://static.guance.com/dataway/install.sh)"

# ============= DATAWAY CONFIG =============

# Dataway UUID, we can get it during the creation of a new dataway
uuid:

# It's the workspace token, most of the time, it's
# system worker space's token.
token:

# secret_token used under sinker mode, and to check if incoming datakit
# requests are valid.
secret_token:

# If __internal__ token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_internal_token: false

# Is empty token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_empty_token: false

# Is dataway cascaded? For cascaded Dataway, it's remote_host is
# another Dataway and not Kodo.
cascaded: false

# kodo(next dataway) related configurations
remote_host:
http_timeout: 30s

http_max_idle_conn_perhost: 0 # default to CPU cores
http_max_conn_perhost: 0      # default no limit

insecure_skip_verify: false
http_client_trace: false
max_conns_per_host: 0
sni: ""

# dataway API configurations
bind: 0.0.0.0:9528

# disable 404 page
disable_404page: false

# dataway TLS file path
tls_crt:
tls_key:

# enable pprof
pprof_bind: localhost:6060

api_limit_rate : 100000         # 100K
max_http_body_bytes : 67108864  # 64MB
copy_buffer_drop_size : 262144  # 256KB, if copy buffer memory larger than this, this memory released
reserved_pool_size: 4096        # reserved pool size for better GC

within_docker: false

log_level: info
log: log
gin_log: gin.log

cache_cfg:
  # cache disk path
  dir: "disk_cache"

  # disable cache
  disabled: false

  clean_interval: "10s"

  # in MB, max single data package size in disk cache, such as HTTP body
  max_data_size: 100

  # in MB, single disk-batch(single file) size
  batch_size: 128

  # in MB, max disk size allowed to cache data
  max_disk_size: 65535

  # expire duration, default 7 days
  expire_duration: "168h"

prometheus:
  listen: "localhost:9090"
  url: "/metrics"
  enable: true

#sinker:
#  etcd:
#    urls:
#    - http://localhost:2379 # one or multiple etcd host
#    dial_timeout: 30s
#    key_space: "/dw_sinker" # subscribe to the etcd key
#    username: "dataway"
#    password: "<PASSWORD>"
#  #file:
#  #  path: /path/to/sinker.json

---

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: deployment-utils-dataway
  name: dataway
  namespace: utils
spec:
  replicas: 2
  selector:
    matchLabels:
      app: deployment-utils-dataway
  template:
    metadata:
      labels:
        app: deployment-utils-dataway
      annotations:
        datakit/logs: |
          [
            {
              "disable": false,
              "source": "dataway",
              "service": "dataway",
              "multiline_match": "^\\d{4}|^\\[GIN\\]"
            }
          ]
        datakit/prom.instances: |
          [[inputs.prom]]
            url = "http://$IP:9090/metrics"

            source = "dataway"
            measurement_name = "dw"
            interval = "10s"
            disable_instance_tag = true
          [inputs.prom.tags]
            service = "dataway"
            instance = "$PODNAME" # we can set as "guangzhou-$PODNAME"
    spec:
      affinity:
        podAffinity: {}
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - deployment-utils-dataway
              topologyKey: kubernetes.io/hostname

      containers:
      - image: pubrepo.guance.com/dataflux/dataway:1.9.0
        imagePullPolicy: IfNotPresent
        name: dataway
        env:
        - name: DW_REMOTE_HOST
          value: "http://kodo.forethought-kodo:9527"
        - name: DW_BIND
          value: "0.0.0.0:9528"
        - name: DW_UUID
          value: "agnt_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" # Fill in the real Dataway UUID here
        - name: DW_TOKEN
          value: "tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" # Fill in the real Dataway token here, usually the system workspace token
        - name: DW_PROM_LISTEN
          value: "0.0.0.0:9090"

        ports:
        - containerPort: 9528
          name: 9528tcp01
          protocol: TCP
        volumeMounts:
          - mountPath: /usr/local/cloudcare/dataflux/dataway/cache
            name: dataway-cache
        resources:
          limits:
            cpu: '4'
            memory: 4Gi
          requests:
            cpu: 100m
            memory: 512Mi
      # nodeSelector:
      #   key: string              
      imagePullSecrets:
      - name: registry-key
      restartPolicy: Always
      volumes:
      - hostPath:
          path: /root/dataway_cache
          type: DirectoryOrCreate
        name: dataway-cache
---

apiVersion: v1
kind: Service
metadata:
  name: dataway
  namespace: utils
spec:
  ports:
  - name: 9528tcp02
    port: 9528
    protocol: TCP
    targetPort: 9528
    nodePort: 30928
  selector:
    app: deployment-utils-dataway
  type: NodePort

containers:
  volumeMounts:
    - name: dataway-config
      mountPath: /usr/local/cloudcare/dataflux/dataway/dataway.yaml
      subPath: config.yaml
volumes:
- configMap:
    defaultMode: 256
    name: dataway-config
    optional: false
  name: dataway-config

• Verify Dataway Installation

After installation, wait a moment and refresh the "Data Gateway" page. If you see a version number in the "Version Information" column of the newly added data gateway, it indicates that this Dataway has successfully connected to the Guance center, and front-end users can connect data through it.

After Dataway successfully connects to the Guance center, log in to the Guance console, navigate to the "Integration" / "DataKit" page, view all Dataway addresses, select the needed Dataway gateway address, obtain the DataKit installation command, and execute it on the server to start collecting data.

Manage DataWay¶

Delete DataWay¶

In the Guance management backend under the "Data Gateway" page, select the DataWay to delete, click "Configuration", in the popped-up Edit DataWay dialog box, click the "Delete" button at the bottom left corner.

Upgrade DataWay¶

In the Guance management backend under the "Data Gateway" page, if there is an upgradable version for DataWay, there will be an upgrade prompt in the version information section.

DW_UPGRADE=1 bash -c "$(curl https://static.guance.com/dataway/install.sh)"

- image: pubrepo.guance.com/dataflux/dataway:<VERSION>

Dataway Service Management¶

When installing Dataway on a host, you can manage the Dataway service with the following commands.

# Start
$ systemctl start dataway

# Restart
$ systemctl restart dataway

# Stop
$ systemctl stop dataway

For Kubernetes, simply restart the corresponding Pod.

Environment Variables¶

Host Installation Supported Environment Variables¶

Host installation is no longer recommended, and new configuration items do not support configuration via command-line parameters. If you cannot change the deployment method, it is suggested to manually modify the corresponding configuration after installation (upgrade). Default configurations are shown in the default configuration example above.

When installing on a host, the following environment variables can be injected into the installation command:

Image Environment Variables¶

When Dataway runs in a Kubernetes environment, it supports the following environment variables.

HTTP Server Settings¶

HTTP TLS Settings¶

To generate a TLS certificate valid for one year, you can use the following OpenSSL command:

# Generate a TLS certificate valid for one year
$ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out tls.crt -keyout tls.key
...

After executing this command, the system will prompt you to input necessary information, including your country, region, city, organization name, department name, and email address. This information will be included in your certificate.

After completing the information input, you will generate two files: tls.crt (certificate file) and tls.key (private key file). Please keep your private key file secure.

To allow the application to use these TLS certificates, you need to set the absolute paths of these two files to the application's environment variables. Below is an example of setting environment variables:

You must first enable DW_ENABLE_TLS for the other two ENVs (DW_TLS_CRT/DW_TLS_KEY) to take effect. Version-1.4.1

env:
- name: DW_ENABLE_TLS
  value: "true"
- name: DW_TLS_CRT
  value: "/path/to/your/tls.crt"
- name: DW_TLS_KEY
  value: "/path/to/your/tls.key"

Replace /path/to/your/tls.crt and /path/to/your/tls.key with the actual paths where your tls.crt and tls.key files are stored.

After setting, you can test whether TLS is effective using the following command:

$ curl -k http://localhost:9528

If successful, it will display an ASCII Art message saying It's working!. If the certificate does not exist, the Dataway logs will have an error similar to the following:

server listen(TLS) failed: open /path/to/your/tls.{crt,key}: no such file or directory

At this point, Dataway cannot start, and the above curl command will also result in an error:

$ curl -vvv -k http://localhost:9528
curl: (7) Failed to connect to localhost port 9528 after 6 ms: Couldn't connect to server

Logging Settings¶

Token/UUID Settings¶

Sinker Settings¶

Prometheus Metrics Exposure¶

Disk Cache Settings¶

Performance-Related Settings¶

Version-1.6.0

Dataway API List¶

Details of the following APIs will be supplemented later.

GET /v1/ntp/¶

Version-1.6.0

• API Description: Get the current Unix timestamp (in seconds) of Dataway.

POST /v1/write/:category¶

• API Description: Receive various collected data uploaded by Datakit.

GET /v1/datakit/pull¶

• API Description: Handle Datakit pulling central configuration (blacklist/Pipeline) requests.

POST /v1/write/rum/replay¶

• API Description: Receive Session Replay data uploaded by Datakit.

POST /v1/upload/profiling¶

• API Description: Receive Profiling data uploaded by Datakit.

POST /v1/election¶

• API Description: Handle Datakit election requests.

POST /v1/election/heartbeat¶

• API Description: Handle Datakit election heartbeat requests.

POST /v1/query/raw¶

Handles DQL query requests, simple example as follows:

POST /v1/query/raw?token=<workspace-token> HTTP/1.1
Content-Type: application/json

{
    "token": "workspace-token",
    "queries": [
        {
            "query": "M::cpu LIMIT 1"
        }
    ],
    "echo_explain": <true/false>
}

Example response:

{
  "content": [
    {
      "series": [
        {
          "name": "cpu",
          "columns": [
            "time",
            "usage_iowait",
            "usage_total",
            "usage_user",
            "usage_guest",
            "usage_system",
            "usage_steal",
            "usage_guest_nice",
            "usage_irq",
            "load5s",
            "usage_idle",
            "usage_nice",
            "usage_softirq",
            "global_tag1",
            "global_tag2",
            "host",
            "cpu"
          ],
          "values": [
            [
              1709782208662,
              0,
              7.421875,
              3.359375,
              0,
              4.0625,
              0,
              0,
              0,
              1,
              92.578125,
              0,
              0,
              null,
              null,
              "WIN-JCHUL92N9IP",
              "cpu-total"
            ]
          ]
        }
      ],
      "points": null,
      "cost": "24.558375ms",
      "is_running": false,
      "async_id": "",
      "query_parse": {
        "namespace": "metric",
        "sources": {
          "cpu": "exact"
        },
        "fields": {},
        "funcs": {}
      },
      "index_name": "",
      "index_store_type": "",
      "query_type": "guancedb",
      "complete": false,
      "index_names": "",
      "scan_completed": false,
      "scan_index": "",
      "next_cursor_time": -1,
      "sample": 1,
      "interval": 0,
      "window": 0
    }
  ]
}

Explanation of the return results:

• Real data is located in the inner series field.
• name indicates the name of the Measurement (here querying CPU metrics; if it’s log-type data, this field does not exist).
• columns indicate the names of the returned result columns.
• values contain the corresponding column results for columns.

POST /v1/workspace¶

• API Description: Handle workspace query requests initiated by Datakit.

POST /v1/object/labels¶

• API Description: Handle object Label modification requests.

DELETE /v1/object/labels¶

• API Description: Handle object Label deletion requests.

GET /v1/check/:token¶

• API Description: Check if the token is valid.

Dataway Metric Collection¶

[[inputs.prom]]
  ## Exporter URLs.
  urls = [ "http://localhost:9090/metrics", ]
  source = "dataway"
  election = true
  measurement_name = "dw" # Dataway metrics set fixed as dw, do not change
[inputs.prom.tags]
  service = "dataway"

annotations: # The following annotation is added by default
   datakit/prom.instances: |
     [[inputs.prom]]
       url = "http://$IP:9090/metrics" # This port (default 9090) depends on the situation
       source = "dataway"
       measurement_name = "dw" # Fixed as this metrics set
       interval = "10s"
       disable_instance_tag = true

     [inputs.prom.tags]
       service = "dataway"
       instance = "$PODNAME"

...
env:
- name: DW_PROM_LISTEN
  value: "0.0.0.0:9090" # This port should match the port in the URL above

If the collection is successful, search for dataway in the Guance "Scenarios" / "Built-in Views" to see the corresponding monitoring views.

Dataway Metrics List¶

Below are the metrics exposed by Dataway. You can obtain these metrics by requesting http://localhost:9090/metrics, and you can monitor a specific metric in real-time (every 3s) using the following command:

Some metrics might not be found because the relevant business modules have not yet run. Some new metrics are only present in the latest versions; version information for each metric is not listed here. Refer to the /metrics interface for the final list of metrics.

watch -n 3 'curl -s http://localhost:9090/metrics | grep -a <METRIC-NAME>'

Metric Collection in Docker Mode¶

Host installation has two modes: one is host machine installation, and the other is via Docker installation. Here we specifically explain the differences in metric collection when installed via Docker.

When installed via Docker, the exposed HTTP port for metrics is mapped to port 19090 on the host (by default), so the metric collection address is http://localhost:19090/metrics.

If a different port is specified separately, then during Docker installation, 10000 is added to that port, so the specified port should not exceed 45535.

In addition, during Docker installation, the profile collection port is also exposed, which by default is mapped to port 16060 on the host. The mechanism is also to add 10000 to the specified port.

Dataway Self-Logging Collection and Processing¶

Dataway's own logs are divided into two categories: one is Gin logs, and the other is application logs. These can be separated using the following Pipeline:

# Pipeline for dataway logging

# Testing sample logging
'''
2023-12-14T11:27:06.744+0800    DEBUG   apis    apis/api_upload_profile.go:272  save profile file to disk [ok] /v1/upload/profiling?token=****************a4e3db8481c345a94fe5a
[GIN] 2021/10/25 - 06:48:07 | 200 |   30.890624ms |  114.215.200.73 | POST     "/v1/write/logging?token=tkn_5c862a11111111111111111111111111"
'''

add_pattern("TOKEN", "tkn_\\w+")
add_pattern("GINTIME", "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}-%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}")
grok(_,"\\[GIN\\]%{SPACE}%{GINTIME:timestamp}%{SPACE}\\|%{SPACE}%{NUMBER:dataway_code}%{SPACE}\\|%{SPACE}%{NOTSPACE:cost_time}%{SPACE}\\|%{SPACE}%{NOTSPACE:client_ip}%{SPACE}\\|%{SPACE}%{NOTSPACE:method}%{SPACE}%{GREEDYDATA:http_url}")

# Gin logging
if cost_time != nil {
  if http_url != nil  {
    grok(http_url, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(http_url, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  group_between(dataway_code, [200,299], "info", status)
  group_between(dataway_code, [300,399], "notice", status)
  group_between(dataway_code, [400,499], "warning", status)
  group_between(dataway_code, [500,599], "error", status)

  if sample(0.1) { # drop 90% debug log
    drop()
    exit()
  } else {
    set_tag(sample_rate, "0.1")
  }

  parse_duration(cost_time)
  duration_precision(cost_time, "ns", "ms")

  set_measurement('gin', true)
  set_tag(service,"dataway")
  exit()
}

# App logging
if cost_time == nil {
  grok(_,"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NOTSPACE:status}%{SPACE}%{NOTSPACE:module}%{SPACE}%{NOTSPACE:code}%{SPACE}%{GREEDYDATA:msg}")
  if level == nil {
    grok(message,"Error%{SPACE}%{DATA:errormsg}")
    if errormsg != nil {
      add_key(status,"error")
      drop_key(errormsg)
    }
  }
  lowercase(level)

  # If debug level is enabled, drop most of them
  if status == 'debug' {
    if sample(0.1) { # drop 90% debug log
      drop()
      exit()
    } else {
      set_tag(sample_rate, "0.1")
    }
  }

  group_in(status, ["error", "panic", "dpanic", "fatal","err","fat"], "error", status) # mark them as 'error'

  if msg != nil {
    grok(msg, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(msg, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  set_measurement("dataway-log", true)
  set_tag(service,"dataway")
}

Dataway Bug Report¶

Dataway exposes its own metrics and profiling collection entry points, and we can collect this information to assist with troubleshooting.

The following information collection uses actual configured ports and addresses; the commands below are listed according to default parameters.

br_dir="dw-br-$(date +%s)"
mkdir -p $br_dir

echo "save bug report to ${br_dir}"

# Modify these configurations based on actual conditions
dw_ip="localhost" # IP address where Dataway metrics/profile are exposed
metric_port=9090  # Port exposing metrics
profile_port=6060 # Port exposing profiles
dw_yaml_conf="/usr/local/cloudcare/dataflux/dataway/dataway.yaml"
dw_dot_yaml_conf="/usr/local/cloudcare/dataflux/dataway/.dataway.yaml" # This file exists when installed in a container

# Collect runtime metrics
curl -v "http://${dw_ip}:${metric_port}/metrics" -o $br_dir/metrics

# Collect profiling information
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/allocs" -o $br_dir/allocs
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/heap" -o $br_dir/heap
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/profile" -o $br_dir/profile # This command runs for about 30s

cp $dw_yaml_conf $br_dir/dataway.yaml.copy
cp $dw_dot_yaml_conf $br_dir/.dataway.yaml.copy

tar czvf ${br_dir}.tar.gz ${br_dir}
rm -rf ${br_dir}

Run the script:

$ sh dw-bug-report.sh
...

After execution, a file similar to dw-br-1721188604.tar.gz will be generated, which can be extracted for analysis.

FAQ¶

Too Large Request Body Issue¶

Version-1.3.7

Dataway has a default setting for the request body size (default 64MB). When the request body is too large, the client will receive an HTTP 413 error (Request Entity Too Large). If the request body is within a reasonable range, you can appropriately increase this value (unit bytes):

• Set the environment variable DW_MAX_HTTP_BODY_BYTES
• In dataway.yaml, set max_http_body_bytes

If there are very large request packets during operation, they will be reflected in both metrics and logs:

• The metric dataway_http_too_large_dropped_total exposes the number of large requests dropped
• Search Dataway logs cat log | grep 'drop too large request', the logs will output the HTTP request Header details, facilitating further understanding of the client situation