Dataway¶

Introduction¶

DataWay is Guance's data gateway; all collectors reporting data to Guance must go through the DataWay gateway.

Dataway Installation¶

• Create Dataway

In the Guance management backend under the "Data Gateway" page, click "Create Dataway". Input a name and binding address, then click "Create".

After successful creation, a new Dataway will be automatically created along with the installation script for Dataway.

• Install Dataway

DW_KODO=http://kodo_ip:port \
   DW_TOKEN=<tkn_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
   DW_UUID=<YOUR_UUID> \
   bash -c "$(curl https://static.guance.com/dataway/install.sh)"

# ============= DATAWAY CONFIG =============

# Dataway UUID, we can get it during the creation of a new dataway
uuid:

# It's the workspace token, most of the time, it's
# system worker space's token.
token:

# secret_token used under sinker mode, and to check if incomming datakit
# requests are valid.
secret_token:

# If __internal__ token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_internal_token: false

# is empty token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_empty_token: false

# Is dataway cascaded? For cascaded Dataway, its remote_host is
# another Dataway and not Kodo.
cascaded: false

# kodo(next dataway) related configurations
remote_host:
http_timeout: 30s

http_max_idle_conn_perhost: 0 # default to CPU cores
http_max_conn_perhost: 0      # default no limit

insecure_skip_verify: false
http_client_trace: false
max_conns_per_host: 0
sni: ""

# dataway API configurations
bind: 0.0.0.0:9528

# disable 404 page
disable_404page: false

# dataway TLS file path
tls_crt:
tls_key:

# enable pprof
pprof_bind: localhost:6060

api_limit_rate : 100000         # 100K
max_http_body_bytes : 67108864  # 64MB
copy_buffer_drop_size : 262144  # 256KB, if copy buffer memory larger than this, this memory released
reserved_pool_size: 4096        # reserved pool size for better GC

within_docker: false

log_level: info
log: log
gin_log: gin.log

cache_cfg:
  # cache disk path
  dir: "disk_cache"

  # disable cache
  disabled: false

  clean_interval: "10s"

  # in MB, max single data package size in disk cache, such as HTTP body
  max_data_size: 100

  # in MB, single disk-batch(single file) size
  batch_size: 128

  # in MB, max disk size allowed to cache data
  max_disk_size: 65535

  # expire duration, default 7 days
  expire_duration: "168h"

prometheus:
  listen: "localhost:9090"
  url: "/metrics"
  enable: true

#sinker:
#  etcd:
#    urls:
#    - http://localhost:2379 # one or multiple etcd hosts
#    dial_timeout: 30s
#    key_space: "/dw_sinker" # subscribe to the etcd key
#    username: "dataway"
#    password: "<PASSWORD>"
#  #file:
#  #  path: /path/to/sinker.json

$ wget https://static.guance.com/dataway/dataway.yaml -O dw-deployment.yaml
$ kubectl apply -f dw-deployment.yaml

containers:
  volumeMounts:
    - name: dataway-config
      mountPath: /usr/local/cloudcare/dataflux/dataway/dataway.yaml
      subPath: config.yaml
volumes:
- configMap:
    defaultMode: 256
    name: dataway-config
    optional: false
  name: dataway-config

• Verify Dataway Installation

After installation, wait a moment and refresh the "Data Gateway" page. If you see the version number in the "Version Information" column of the newly added data gateway, it indicates that this Dataway has successfully connected to the Guance center, and front-end users can start accessing data through it.

After Dataway successfully connects to the Guance center, log into the Guance console. On the "Integration" / "DataKit" page, you can view all Dataway addresses, select the required Dataway gateway address, obtain the DataKit installation command, and execute it on the server to start collecting data.

Manage DataWay¶

Delete DataWay¶

In the Guance management backend under the "Data Gateway" page, select the DataWay you want to delete, click "Configuration", in the pop-up edit DataWay dialog box, click the "Delete" button at the bottom left corner.

Upgrade DataWay¶

On the "Data Gateway" page in the Guance management backend, if there is an upgradable version for DataWay, there will be an upgrade prompt in the version information section.

DW_UPGRADE=1 bash -c "$(curl https://static.guance.com/dataway/install.sh)"

- image: pubrepo.guance.com/dataflux/dataway:<VERSION>

Dataway Service Management¶

When installing Dataway on a host, you can manage the Dataway service using the following commands.

# Start
$ systemctl start dataway

# Restart
$ systemctl restart dataway

# Stop
$ systemctl stop dataway

For Kubernetes, simply restart the corresponding Pod.

Environment Variables¶

Host Installation Supported Environment Variables¶

We no longer recommend host-based installations, and new configuration items are no longer supported via command-line parameters. If you cannot change the deployment method, it is suggested to manually modify the corresponding configurations after installation (upgrade). Default configurations are shown in the default configuration example above.

When installing on a host, the following environment variables can be injected into the installation command:

Image Environment Variables¶

Dataway supports the following environment variables when running in a Kubernetes environment.

HTTP Server Settings¶

HTTP TLS Settings¶

To generate a TLS certificate valid for one year, you can use the following OpenSSL command:

# Generate a TLS certificate valid for one year
$ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out tls.crt -keyout tls.key
...

Executing this command will prompt you to input necessary information, including your country, region, city, organization name, department name, and email address. This information will be included in your certificate.

After completing the information input, two files will be generated: tls.crt (certificate file) and tls.key (private key file). Safeguard your private key file and ensure its security.

To allow applications to use these TLS certificates, you need to set the absolute paths of these two files in the application's environment variables. Below is an example of setting environment variables:

DW_ENABLE_TLS must be enabled first, and the other two ENV (DW_TLS_CRT/DW_TLS_KEY) will only take effect after this. Version-1.4.1

env:
- name: DW_ENABLE_TLS
  value: "true"
- name: DW_TLS_CRT
  value: "/path/to/your/tls.crt"
- name: DW_TLS_KEY
  value: "/path/to/your/tls.key"

Replace /path/to/your/tls.crt and /path/to/your/tls.key with the actual paths where tls.crt and tls.key files are stored.

After setting, you can use the following command to test if TLS is effective:

$ curl -k http://localhost:9528

If successful, an It's working! ASCII Art message will be displayed. If the certificate does not exist, the Dataway logs will show an error similar to the following:

server listen(TLS) failed: open /path/to/your/tls.{crt,key}: no such file or directory

At this point, Dataway will fail to start, and the above curl command will also produce an error:

$ curl -vvv -k http://localhost:9528
curl: (7) Failed to connect to localhost port 9528 after 6 ms: Couldn't connect to server

Logging Settings¶

Token/UUID Settings¶

Sinker Settings¶

Prometheus Metrics Exposure¶

Disk Cache Settings¶

Performance-Related Settings¶

Version-1.6.0

Dataway API List¶

Details of the following APIs are to be supplemented.

GET /v1/ntp/¶

Version-1.6.0

• API Description: Get the current Unix timestamp (in seconds) of Dataway

POST /v1/write/:category¶

• API Description: Receive various types of collected data uploaded by Datakit

GET /v1/datakit/pull¶

• API Description: Handle Datakit pull requests for central configurations (blacklist/Pipeline)

POST /v1/write/rum/replay¶

• API Description: Receive Session Replay data uploaded by Datakit

POST /v1/upload/profiling¶

• API Description: Receive Profiling data uploaded by Datakit

POST /v1/election¶

• API Description: Handle Datakit election requests

POST /v1/election/heartbeat¶

• API Description: Handle Datakit election heartbeat requests

POST /v1/query/raw¶

Handles DQL query requests. A simple example is as follows:

POST /v1/query/raw?token=<workspace-token> HTTP/1.1
Content-Type: application/json

{
    "token": "workspace-token",
    "queries": [
        {
            "query": "M::cpu LIMIT 1"
        }
    ],
    "echo_explain": <true/false>
}

Return example:

{
  "content": [
    {
      "series": [
        {
          "name": "cpu",
          "columns": [
            "time",
            "usage_iowait",
            "usage_total",
            "usage_user",
            "usage_guest",
            "usage_system",
            "usage_steal",
            "usage_guest_nice",
            "usage_irq",
            "load5s",
            "usage_idle",
            "usage_nice",
            "usage_softirq",
            "global_tag1",
            "global_tag2",
            "host",
            "cpu"
          ],
          "values": [
            [
              1709782208662,
              0,
              7.421875,
              3.359375,
              0,
              4.0625,
              0,
              0,
              0,
              1,
              92.578125,
              0,
              0,
              null,
              null,
              "WIN-JCHUL92N9IP",
              "cpu-total"
            ]
          ]
        }
      ],
      "points": null,
      "cost": "24.558375ms",
      "is_running": false,
      "async_id": "",
      "query_parse": {
        "namespace": "metric",
        "sources": {
          "cpu": "exact"
        },
        "fields": {},
        "funcs": {}
      },
      "index_name": "",
      "index_store_type": "",
      "query_type": "guancedb",
      "complete": false,
      "index_names": "",
      "scan_completed": false,
      "scan_index": "",
      "next_cursor_time": -1,
      "sample": 1,
      "interval": 0,
      "window": 0
    }
  ]
}

Return result description:

• The real data is located in the inner series field
• name represents the name of the measurement (in this case querying CPU metrics, if it is log-type data, this field does not exist)
• columns represent the names of the returned result columns
• values contain the corresponding column results from columns

POST /v1/workspace¶

• API Description: Handle workspace query requests initiated by Datakit

POST /v1/object/labels¶

• API Description: Handle object Label modification requests

DELETE /v1/object/labels¶

• API Description: Handle object Label deletion requests

GET /v1/check/:token¶

• API Description: Check if the token is valid

Dataway Metric Collection¶

[[inputs.prom]]
  ## Exporter URLs.
  urls = [ "http://localhost:9090/metrics", ]
  source = "dataway"
  election = true
  measurement_name = "dw" # Dataway metrics set is fixed as dw, do not change
[inputs.prom.tags]
  service = "dataway"

annotations: # The following annotation is added by default
   datakit/prom.instances: |
     [[inputs.prom]]
       url = "http://$IP:9090/metrics" # This port (default 9090) depends on the situation
       source = "dataway"
       measurement_name = "dw" # Fixed as this metrics set
       interval = "10s"
       disable_instance_tag = true

     [inputs.prom.tags]
       service = "dataway"
       instance = "$PODNAME"

...
env:
- name: DW_PROM_LISTEN
  value: "0.0.0.0:9090" # This port should match the port in the above url

If the collection is successful, search for dataway in the Guance "Scenarios" / "Built-in Views" to see the corresponding monitoring views.

Dataway Metric List¶

Below are the metrics exposed by Dataway. You can retrieve these metrics by requesting http://localhost:9090/metrics. You can use the following command to monitor (every 3 seconds) a specific metric in real-time:

If certain metrics cannot be queried, it might be because the related business modules have not yet run. Some new metrics only exist in the latest version, so the version information for each metric is not listed here. Refer to the metrics list returned by the /metrics interface.

watch -n 3 'curl -s http://localhost:9090/metrics | grep -a <METRIC-NAME>'

Metric Collection in Docker Mode¶

Host installation has two modes: one is native host installation, and the other is through Docker installation. Here we explain the differences in metric collection when installed via Docker.

When installed via Docker, the exposed HTTP port for metrics will be mapped to port 19090 on the host machine (by default). At this point, the metric collection address is http://localhost:19090/metrics.

If a different port is specified separately, then during Docker installation, 10000 will be added to that port, so the specified port should not exceed 45535.

Additionally, during Docker installation, the profile collection port is also exposed, which by default maps to port 16060 on the host machine. The mechanism here is also to add 10000 to the specified port.

Dataway Log Collection and Processing¶

Dataway's logs are divided into two categories: one is gin logs, and the other is application logs. The following Pipeline can separate them:

# Pipeline for dataway logging

# Testing sample logging
'''
2023-12-14T11:27:06.744+0800    DEBUG   apis     apis/api_upload_profile.go:272   save profile file to disk [ok] /v1/upload/profiling?token=****************a4e3db8481c345a94fe5a
[GIN] 2021/10/25 - 06:48:07 | 200 |   30.890624ms |  114.215.200.73 | POST     "/v1/write/logging?token=tkn_5c862a11111111111111111111111111"
'''

add_pattern("TOKEN", "tkn_\\w+")
add_pattern("GINTIME", "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}-%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}")
grok(_,"\\[GIN\\]%{SPACE}%{GINTIME:timestamp}%{SPACE}\\|%{SPACE}%{NUMBER:dataway_code}%{SPACE}\\|%{SPACE}%{NOTSPACE:cost_time}%{SPACE}\\|%{SPACE}%{NOTSPACE:client_ip}%{SPACE}\\|%{SPACE}%{NOTSPACE:method}%{SPACE}%{GREEDYDATA:http_url}")

# Gin logging
if cost_time != nil {
  if http_url != nil  {
    grok(http_url, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(http_url, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  group_between(dataway_code, [200,299], "info", status)
  group_between(dataway_code, [300,399], "notice", status)
  group_between(dataway_code, [400,499], "warning", status)
  group_between(dataway_code, [500,599], "error", status)

  if sample(0.1) { # drop 90% debug log
    drop()
    exit()
  } else {
    set_tag(sample_rate, "0.1")
  }

  parse_duration(cost_time)
  duration_precision(cost_time, "ns", "ms")

  set_measurement('gin', true)
  set_tag(service,"dataway")
  exit()
}

# App logging
if cost_time == nil {
  grok(_,"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NOTSPACE:status}%{SPACE}%{NOTSPACE:module}%{SPACE}%{NOTSPACE:code}%{SPACE}%{GREEDYDATA:msg}")
  if level == nil {
    grok(message,"Error%{SPACE}%{DATA:errormsg}")
    if errormsg != nil {
      add_key(status,"error")
      drop_key(errormsg)
    }
  }
  lowercase(level)

  # If debug level is enabled, drop most of them
  if status == 'debug' {
    if sample(0.1) { # drop 90% debug log
      drop()
      exit()
    } else {
      set_tag(sample_rate, "0.1")
    }
  }

  group_in(status, ["error", "panic", "dpanic", "fatal","err","fat"], "error", status) # mark them as 'error'

  if msg != nil {
    grok(msg, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(msg, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  set_measurement("dataway-log", true)
  set_tag(service,"dataway")
}

## Dataway Bug Report {#bug-report}

Dataway exposes its own metrics and profiling collection entry points, and we can collect this information for troubleshooting.

> The following information collection is based on the actual configured ports and addresses, with the commands listed according to the default parameters.

```shell title="dw-bug-report.sh"
br_dir="dw-br-$(date +%s)"
mkdir -p $br_dir

echo "save bug report to ${br_dir}"

# Modify the configuration here according to the actual situation
dw_ip="localhost" # IP address where Dataway metrics/profile are exposed
metric_port=9090  # Port exposing metrics
profile_port=6060 # Port exposing profiles
dw_yaml_conf="/usr/local/cloudcare/dataflux/dataway/dataway.yaml"
dw_dot_yaml_conf="/usr/local/cloudcare/dataflux/dataway/.dataway.yaml" # This file exists for container installations

# Collect runtime metrics
curl -v "http://${dw_ip}:${metric_port}/metrics" -o $br_dir/metrics

# Collect profiling information
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/allocs" -o $br_dir/allocs
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/heap" -o $br_dir/heap
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/profile" -o $br_dir/profile # This command runs for about 30s

cp $dw_yaml_conf $br_dir/dataway.yaml.copy
cp $dw_dot_yaml_conf $br_dir/.dataway.yaml.copy

tar czvf ${br_dir}.tar.gz ${br_dir}
rm -rf ${br_dir}

Run the script:

$ sh dw-bug-report.sh
...

After execution, a file similar to dw-br-1721188604.tar.gz will be generated, and you can extract it.

FAQ¶

Too Large Request Body Issue¶

Version-1.3.7

Dataway has a default setting for request body size (default 64MB). However, when the request body is too large, the client will receive an HTTP 413 error (Request Entity Too Large). If the request body is within a reasonable range, you can appropriately increase this value (unit bytes):

• Set the environment variable DW_MAX_HTTP_BODY_BYTES
• In dataway.yaml, set max_http_body_bytes

If too large requests appear during runtime, they will be reflected in both metrics and logs:

• Metric dataway_http_too_large_dropped_total exposes the number of dropped large requests
• Search Dataway logs cat log | grep 'drop too large request', which will output the HTTP request Header details to further understand the client situation