Dataway¶

Introduction¶

DataWay is the data gateway of Guance. All data reported by collectors to Guance must go through the DataWay gateway.

Dataway Installation¶

• Create Dataway

In the Guance management console on the "Data Gateway" page, click “Create Dataway”. Enter a name and bind address, then click “Create”.

After successful creation, a new Dataway will be automatically created along with the installation script for Dataway.

• Install Dataway

DW_KODO=http://kodo_ip:port \
   DW_TOKEN=<tkn_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
   DW_UUID=<YOUR_UUID> \
   bash -c "$(curl https://static.guance.com/dataway/install.sh)"

# ============= DATAWAY CONFIG =============

# Dataway UUID, we can get it during the creation of a new dataway
uuid:

# It's the workspace token, most of the time, it's
# system worker space's token.
token:

# secret_token used under sinker mode, and to check if incoming datakit
# requests are valid.
secret_token:

# If __internal__ token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_internal_token: false

# Is empty token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_empty_token: false

# Is dataway cascaded? For cascaded Dataway, its remote_host is
# another Dataway and not Kodo.
cascaded: false

# kodo(next dataway) related configurations
remote_host:
http_timeout: 30s

http_max_idle_conn_perhost: 0 # default to CPU cores
http_max_conn_perhost: 0      # default no limit

insecure_skip_verify: false
http_client_trace: false
max_conns_per_host: 0
sni: ""

# dataway API configurations
bind: 0.0.0.0:9528

# disable 404 page
disable_404page: false

# dataway TLS file path
tls_crt:
tls_key:

# enable pprof
pprof_bind: localhost:6060

api_limit_rate : 100000         # 100K
max_http_body_bytes : 67108864  # 64MB
copy_buffer_drop_size : 262144  # 256KB, if copy buffer memory larger than this, this memory released
reserved_pool_size: 4096        # reserved pool size for better GC

within_docker: false

log_level: info
log: log
gin_log: gin.log

cache_cfg:
  # cache disk path
  dir: "disk_cache"

  # disable cache
  disabled: false

  clean_interval: "10s"

  # in MB, max single data package size in disk cache, such as HTTP body
  max_data_size: 100

  # in MB, single disk-batch(single file) size
  batch_size: 128

  # in MB, max disk size allowed to cache data
  max_disk_size: 65535

  # expire duration, default 7 days
  expire_duration: "168h"

prometheus:
  listen: "localhost:9090"
  url: "/metrics"
  enable: true

#sinker:
#  etcd:
#    urls:
#    - http://localhost:2379 # one or multiple etcd hosts
#    dial_timeout: 30s
#    key_space: "/dw_sinker" # subscribe to the etcd key
#    username: "dataway"
#    password: "<PASSWORD>"
#  #file:
#  #  path: /path/to/sinker.json

$ wget https://static.guance.com/dataway/dataway.yaml -O dw-deployment.yaml
$ kubectl apply -f dw-deployment.yaml

containers:
  volumeMounts:
    - name: dataway-config
      mountPath: /usr/local/cloudcare/dataflux/dataway/dataway.yaml
      subPath: config.yaml
volumes:
- configMap:
    defaultMode: 256
    name: dataway-config
    optional: false
  name: dataway-config

• Verify Dataway Installation

After installation, wait a moment and refresh the "Data Gateway" page. If you see the version number in the "Version Information" column of the newly added data gateway, it indicates that this Dataway has successfully connected to the Guance center, and front-end users can start connecting data through it.

Once Dataway successfully connects to the Guance center, log in to the Guance console, go to the "Integration"/"DataKit" page, view all Dataway addresses, select the required Dataway gateway address, obtain the DataKit installation instructions and execute them on the server to start collecting data.

Manage DataWay¶

Delete DataWay¶

In the Guance management console on the "Data Gateway" page, choose the DataWay you need to delete, click “Configure”, in the popped-up DataWay edit dialog box, click the “Delete” button at the bottom left.

Upgrade DataWay¶

On the Guance management console "Data Gateway" page, if there is an upgradable version for DataWay, an upgrade prompt will appear in the version information section.

DW_UPGRADE=1 bash -c "$(curl https://static.guance.com/dataway/install.sh)"

- image: pubrepo.guance.com/dataflux/dataway:<VERSION>

Dataway Service Management¶

For host-installed Dataway, use the following commands to manage the Dataway service.

# Start
$ systemctl start dataway

# Restart
$ systemctl restart dataway

# Stop
$ systemctl stop dataway

For Kubernetes, simply restart the corresponding Pod.

Environment Variables¶

Host Installation Supported Environment Variables¶

We no longer recommend host installation, and new configuration items are no longer supported via command-line parameters. If you cannot change the deployment method, it is suggested to manually modify the corresponding configurations after installation (upgrade). Default configurations are shown in the default configuration example above.

During host installation, you can inject the following environment variables into the installation command:

Image Environment Variables¶

When Dataway runs in a Kubernetes environment, it supports the following environment variables.

HTTP Server Settings¶

HTTP TLS Settings¶

To generate a TLS certificate valid for one year, you can use the following OpenSSL command:

# Generate a TLS certificate valid for one year
$ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out tls.crt -keyout tls.key
...

After executing this command, the system will prompt you to enter some necessary information, including your country, region, city, organization name, department name, and email address. This information will be included in your certificate.

After completing the information input, two files will be generated: tls.crt (certificate file) and tls.key (private key file). Safely store your private key file and ensure its security.

To make the application use these TLS certificates, you need to set the absolute paths of these two files in the application's environment variables. Below is an example of setting environment variables:

Must first enable DW_ENABLE_TLS, then the other two ENV (DW_TLS_CRT/DW_TLS_KEY) will take effect. Version-1.4.1

env:
- name: DW_ENABLE_TLS
  value: "true"
- name: DW_TLS_CRT
  value: "/path/to/your/tls.crt"
- name: DW_TLS_KEY
  value: "/path/to/your/tls.key"

Replace /path/to/your/tls.crt and /path/to/your/tls.key with the actual paths where you store tls.crt and tls.key.

After setting, you can use the following command to test whether TLS is effective:

$ curl -k http://localhost:9528

If successful, it will display an ASCII Art message saying It's working!. If the certificate does not exist, there will be an error in the Dataway logs similar to the following:

server listen(TLS) failed: open /path/to/your/tls.{crt,key}: no such file or directory

At this point, Dataway cannot start, and the above curl command will also report an error:

$ curl -vvv -k http://localhost:9528
curl: (7) Failed to connect to localhost port 9528 after 6 ms: Couldn't connect to server

Log Settings¶

Token/UUID Settings¶

Sinker Settings¶

Prometheus Metrics Exposure¶

Disk Cache Settings¶

Performance-Related Settings¶

Version-1.6.0

Dataway API List¶

Details of the following APIs are pending.

GET /v1/ntp/¶

Version-1.6.0

• API Description: Get the current Unix timestamp (in seconds) of Dataway

POST /v1/write/:category¶

• API Description: Receive various types of collected data uploaded by Datakit

GET /v1/datakit/pull¶

• API Description: Handle Datakit's pull request for central configurations (blacklist/Pipeline)

POST /v1/write/rum/replay¶

• API Description: Receive Session Replay data uploaded by Datakit

POST /v1/upload/profiling¶

• API Description: Receive Profiling data uploaded by Datakit

POST /v1/election¶

• API Description: Handle Datakit's election request

POST /v1/election/heartbeat¶

• API Description: Handle Datakit's election heartbeat request

POST /v1/query/raw¶

Handle DQL query requests. A simple example is as follows:

POST /v1/query/raw?token=<workspace-token> HTTP/1.1
Content-Type: application/json

{
    "token": "workspace-token",
    "queries": [
        {
            "query": "M::cpu LIMIT 1"
        }
    ],
    "echo_explain": <true/false>
}

Response example:

{
  "content": [
    {
      "series": [
        {
          "name": "cpu",
          "columns": [
            "time",
            "usage_iowait",
            "usage_total",
            "usage_user",
            "usage_guest",
            "usage_system",
            "usage_steal",
            "usage_guest_nice",
            "usage_irq",
            "load5s",
            "usage_idle",
            "usage_nice",
            "usage_softirq",
            "global_tag1",
            "global_tag2",
            "host",
            "cpu"
          ],
          "values": [
            [
              1709782208662,
              0,
              7.421875,
              3.359375,
              0,
              4.0625,
              0,
              0,
              0,
              1,
              92.578125,
              0,
              0,
              null,
              null,
              "WIN-JCHUL92N9IP",
              "cpu-total"
            ]
          ]
        }
      ],
      "points": null,
      "cost": "24.558375ms",
      "is_running": false,
      "async_id": "",
      "query_parse": {
        "namespace": "metric",
        "sources": {
          "cpu": "exact"
        },
        "fields": {},
        "funcs": {}
      },
      "index_name": "",
      "index_store_type": "",
      "query_type": "guancedb",
      "complete": false,
      "index_names": "",
      "scan_completed": false,
      "scan_index": "",
      "next_cursor_time": -1,
      "sample": 1,
      "interval": 0,
      "window": 0
    }
  ]
}

Explanation of the response results:

• Real data is located in the inner series field
• name indicates the Measurement name (for this query, it is the CPU metric; if it is log data, this field does not exist)
• columns indicate the names of the returned result columns
• values contain the column results corresponding to columns

POST /v1/workspace¶

• API Description: Handle workspace query requests initiated by Datakit

POST /v1/object/labels¶

• API Description: Handle object Label modification requests

DELETE /v1/object/labels¶

• API Description: Handle object Label deletion requests

GET /v1/check/:token¶

• API Description: Check if the token is valid

Dataway Metric Collection¶

[[inputs.prom]]
  ## Exporter URLs.
  urls = [ "http://localhost:9090/metrics", ]
  source = "dataway"
  election = true
  measurement_name = "dw" # The metric set for Dataway is fixed as "dw", do not change it
[inputs.prom.tags]
  service = "dataway"

annotations: # The following annotation is added by default
   datakit/prom.instances: |
     [[inputs.prom]]
       url = "http://$IP:9090/metrics" # This port (default 9090) depends on the situation
       source = "dataway"
       measurement_name = "dw" # Fixed as this metric set
       interval = "10s"
       disable_instance_tag = true

     [inputs.prom.tags]
       service = "dataway"
       instance = "$PODNAME"

...
env:
- name: DW_PROM_LISTEN
  value: "0.0.0.0:9090" # This port should match the port in the above URL

If collection is successful, search for dataway in Guance's "Scenarios"/"Built-in Views" to see the corresponding monitoring view.

Dataway Metric List¶

Below are the metrics exposed by Dataway. By requesting http://localhost:9090/metrics, you can obtain these metrics. You can use the following command to monitor (every 3 seconds) a specific metric in real-time:

Some metrics may not be found because the relevant business modules have not yet started. Some new metrics only exist in the latest versions, so the version information of each metric is not listed here. Refer to the metrics list returned by the /metrics interface for accurate information.

watch -n 3 'curl -s http://localhost:9090/metrics | grep -a <METRIC-NAME>'

Metrics Collection in Docker Mode¶

Host installation has two modes: one is host installation, and the other is through Docker. Here we specifically explain the differences in metrics collection when installed via Docker.

When installed via Docker, the exposed HTTP port for metrics will be mapped to the host's port 19090 (by default). At this point, the metrics collection address is http://localhost:19090/metrics.

If a different port is specified separately, then during Docker installation, it will add 10000 to the specified port, so ensure the specified port does not exceed 45535.

Additionally, when installing via Docker, the profile collection port will also be exposed, defaulting to port 16060 on the host. This mechanism also adds 10000 to the specified port.

Dataway Self-Logging Collection and Processing¶

Dataway's own logs are divided into two categories: one is Gin logs, and the other is application logs. The following Pipeline can separate them:

# Pipeline for dataway logging

# Testing sample logging
'''
2023-12-14T11:27:06.744+0800    DEBUG   apis     apis/api_upload_profile.go:272   save profile file to disk [ok] /v1/upload/profiling?token=****************a4e3db8481c345a94fe5a
[GIN] 2021/10/25 - 06:48:07 | 200 |   30.890624ms |  114.215.200.73 | POST     "/v1/write/logging?token=tkn_5c862a11111111111111111111111111"
'''

add_pattern("TOKEN", "tkn_\\w+")
add_pattern("GINTIME", "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}-%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}")
grok(_,"\\[GIN\\]%{SPACE}%{GINTIME:timestamp}%{SPACE}\\|%{SPACE}%{NUMBER:dataway_code}%{SPACE}\\|%{SPACE}%{NOTSPACE:cost_time}%{SPACE}\\|%{SPACE}%{NOTSPACE:client_ip}%{SPACE}\\|%{SPACE}%{NOTSPACE:method}%{SPACE}%{GREEDYDATA:http_url}")

# Gin logging
if cost_time != nil {
  if http_url != nil  {
    grok(http_url, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(http_url, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  group_between(dataway_code, [200,299], "info", status)
  group_between(dataway_code, [300,399], "notice", status)
  group_between(dataway_code, [400,499], "warning", status)
  group_between(dataway_code, [500,599], "error", status)

  if sample(0.1) { # drop 90% debug log
    drop()
    exit()
  } else {
    set_tag(sample_rate, "0.1")
  }

  parse_duration(cost_time)
  duration_precision(cost_time, "ns", "ms")

  set_measurement('gin', true)
  set_tag(service,"dataway")
  exit()
}

# App logging
if cost_time == nil {
  grok(_,"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NOTSPACE:status}%{SPACE}%{NOTSPACE:module}%{SPACE}%{NOTSPACE:code}%{SPACE}%{GREEDYDATA:msg}")
  if level == nil {
    grok(message,"Error%{SPACE}%{DATA:errormsg}")
    if errormsg != nil {
      add_key(status,"error")
      drop_key(errormsg)
    }
  }
  lowercase(level)

  # If debug level is enabled, drop most of them
  if status == 'debug' {
    if sample(0.1) { # drop 90% debug log
      drop()
      exit()
    } else {
      set_tag(sample_rate, "0.1")
    }
  }

  group_in(status, ["error", "panic", "dpanic", "fatal","err","fat"], "error", status) # mark them as 'error'

  if msg != nil {
    grok(msg, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(msg, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  set_measurement("dataway-log", true)
  set_tag(service,"dataway")
}

Dataway Bug Report¶

Dataway exposes metrics and profiling collection endpoints, which we can collect to facilitate issue troubleshooting.

The following information collection uses actual configured ports and addresses. The commands below use default parameters.

br_dir="dw-br-$(date +%s)"
mkdir -p $br_dir

echo "save bug report to ${br_dir}"

# Modify these configurations based on actual conditions
dw_ip="localhost" # IP address where Dataway exposes metrics/profile
metric_port=9090  # Port for exposing metrics
profile_port=6060 # Port for exposing profiles
dw_yaml_conf="/usr/local/cloudcare/dataflux/dataway/dataway.yaml"
dw_dot_yaml_conf="/usr/local/cloudcare/dataflux/dataway/.dataway.yaml" # Present in container installations

# Collect runtime metrics
curl -v "http://${dw_ip}:${metric_port}/metrics" -o $br_dir/metrics

# Collect profiling information
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/allocs" -o $br_dir/allocs
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/heap" -o $br_dir/heap
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/profile" -o $br_dir/profile # This command runs for about 30 seconds

cp $dw_yaml_conf $br_dir/dataway.yaml.copy
cp $dw_dot_yaml_conf $br_dir/.dataway.yaml.copy

tar czvf ${br_dir}.tar.gz ${br_dir}
rm -rf ${br_dir}

Run the script:

$ sh dw-bug-report.sh
...

After execution, a file similar to dw-br-1721188604.tar.gz will be generated. You can retrieve this file for further analysis.

FAQ¶

Too Large Request Body Issue¶

Version-1.3.7

Dataway has a default setting for request body size (default 64MB). When the request body is too large, the client receives an HTTP 413 error (Request Entity Too Large). If the request body is within a reasonable range, you can appropriately increase this value (unit bytes):

• Set the environment variable DW_MAX_HTTP_BODY_BYTES
• In dataway.yaml, set max_http_body_bytes

If a large request packet appears during operation, it will be reflected in both metrics and logs:

• The metric dataway_http_too_large_dropped_total exposes the number of dropped large requests
• Search Dataway logs cat log | grep 'drop too large request', which outputs HTTP request headers for further understanding of the client situation