Dataway¶

Introduction¶

DataWay is the data gateway of Guance. All data reported by collectors to Guance must pass through the DataWay gateway.

Dataway Installation¶

• Create a new Dataway

On the "Data Gateway" page in the Guance management console, click "Create Dataway". Enter a name and binding address, then click "Create".

After creation is successful, a new Dataway will be created automatically and the installation script for the Dataway will be generated.

• Install Dataway

DW_KODO=http://kodo_ip:port \
   DW_TOKEN=<tkn_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
   DW_UUID=<YOUR_UUID> \
   bash -c "$(curl https://static.guance.com/dataway/install.sh)"

# ============= DATAWAY CONFIG =============

# Dataway UUID, we can get it on during create a new dataway
uuid:

# It's the workspace token, most of the time, it's
# system worker space's token.
token:

# secret_token used under sinker mode, and to check if incomming datakit
# requests are valid.
secret_token:

# If __internal__ token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_internal_token: false

# is empty token allowed? If ok, the data/request will direct to
# the workspace with the token above
enable_empty_token: false

# Is dataway cascaded? For cascaded Dataway, it's remote_host is
# another Dataway and not Kodo.
cascaded: false

# kodo(next dataway) related configures
remote_host:
http_timeout: 3s

http_max_idle_conn_perhost: 0 # default to CPU cores
http_max_conn_perhost: 0      # default no limit

insecure_skip_verify: false
http_client_trace: false
max_conns_perhost: 0
sni: ""

# dataway API configures
bind: 0.0.0.0:9528

# disable 404 page
disable_404page: false

# dataway TLS file path
tls_crt:
tls_key:

# enable pprof
pprof_bind: localhost:6060

api_limit_rate : 100000         # 100K
max_http_body_bytes : 67108864  # 64MB
copy_buffer_drop_size : 262144  # 256KB, if copy buffer memory larger than this, this memory released
reserved_pool_size: 4096        # reserved pool size for better GC

within_docker: false

log_level: info
log: log
gin_log: gin.log

ip_blacklist:
  ttl = "1m"
  clean_interval = "1h"

cache_cfg:
  # cache disk path
  dir: "disk_cache"

  # disable cache
  disabled: false

  clean_interval: "10s"

  # in MB, max single data package size in disk cache, such as HTTP body
  max_data_size: 100

  # in MB, single disk-batch(single file) size
  batch_size: 128

  # in MB, max disk size allowed to cache data
  max_disk_size: 65535

  # expire duration, default 7 days
  expire_duration: "168h"

prometheus:
  listen: "localhost:9090"
  url: "/metrics"
  enable: true

#sinker:
#  etcd:
#    urls:
#    - http://localhost:2379 # one or multiple etcd host
#    dial_timeout: 30s
#    key_space: "/dw_sinker" # subscribe to the etcd key
#    username: "dataway"
#    password: "<PASSWORD>"
#  #file:
#  #  path: /path/to/sinker.json

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: sts-utils-dataway
  name: dataway
  namespace: utils
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sts-utils-dataway
  serviceName: dataway
  template:
    metadata:
      annotations:
        datakit/logs: |
          [
            {
              "disable": false,
              "source": "dataway",
              "service": "dataway",
              "multiline_match": "^\\d{4}|^\\[GIN\\]"
            }
          ]
        datakit/prom.instances: |
          [[inputs.prom]]
            url = "http://$IP:9090/metrics"

            source = "dataway"
            measurement_name = "dw"
            interval = "10s"
            disable_instance_tag = true
          [inputs.prom.tags]
            service = "dataway"
            instance = "$PODNAME" # we can set as "xxx-$PODNAME"
      labels:
        app: sts-utils-dataway
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - sts-utils-dataway
              topologyKey: kubernetes.io/hostname
      containers:
        - env:
            - name: DW_REMOTE_HOST
              value: http://kodo.forethought-kodo:9527
            - name: DW_BIND
              value: 0.0.0.0:9528
            - name: DW_UUID
              value: agnt_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx   # Dataway UUID
            - name: DW_TOKEN
              value: tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  # Dataway token
            - name: DW_PROM_LISTEN
              value: 0.0.0.0:9090
            - name: DW_LOG
              value: stdout
            - name: DW_LOG_LEVEL
              value: info
            - name: DW_GIN_LOG
              value: stdout
            - name: DW_DISKCACHE_DIR
              value: cache
            - name: DW_HTTP_TIMEOUT
              value: '3s'
            - name: DW_ENABLE_INTERNAL_TOKEN
              value: 'false'
            - name: DW_MAX_HTTP_BODY_BYTES
              value: '67108864'
            - name: DW_HTTP_CLIENT_TRACE
              value: 'on'
            - name: DW_RESERVED_POOL_SIZE
              value: '0'
            - name: DW_COPY_BUFFER_DROP_SIZE
              value: '262144'
            - name: DW_DISKCACHE_CAPACITY_MB
              value: 102400
          image: pubrepo.guance.com/dataflux/dataway:1.12.1
          imagePullPolicy: IfNotPresent
          name: dataway
          ports:
            - containerPort: 9528
              name: 9528tcp01
              protocol: TCP
          resources:
            limits:
              cpu: '4'
              memory: 4Gi
            requests:
              cpu: 100m
              memory: 512Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /usr/local/cloudcare/dataflux/dataway/cache
              name: dataway-cache
      dnsPolicy: ClusterFirst
      imagePullSecrets: []
      #nodeSelector:
      #  nodepool: dataway
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      #tolerations:
      #  - effect: NoSchedule
      #    key: nodepool
      #    operator: Equal
      #    value: dataway
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: dataway-cache
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: xxxxxx  # High-Performance Storage StorageClass
        volumeMode: Filesystem
      status:
        phase: Pending

---

apiVersion: v1
kind: Service
metadata:
  name: dataway
  namespace: utils
spec:
  ports:
    - name: 9528tcp02
      nodePort: 30928
      port: 9528
      protocol: TCP
      targetPort: 9528
  selector:
    app: sts-utils-dataway
  type: NodePort

containers:
  volumeMounts:
    - name: dataway-config
      mountPath: /usr/local/cloudcare/dataflux/dataway/dataway.yaml
      subPath: config.yaml
volumes:
- configMap:
    defaultMode: 256
    name: dataway-config
    optional: false
  name: dataway-config

docker run -d \
    --name <YOUR-DW-IN-DOCKER> \
    -p 19528:9528 -p 19090:9090 \
    --mount type=bind,source=<host/path/for/diskcache>,target=/usr/local/cloudcare/dataflux/dataway/cache \
    --memory=2g --memory-reservation=256m \
    --cpus="2" \
    -e DW_UUID=<YOUR-AGNT_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
    -e DW_TOKEN=<YOUR-TKN_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
    -e DW_REMOTE_HOST=http://kodo.forethought-kodo:9527 \
    -e DW_BIND=0.0.0.0:9528 \
    -e DW_PROM_LISTEN=0.0.0.0:9090 \
    -e DW_HTTP_CLIENT_TRACE=true \
    -e DW_LOG_LEVEL=info \
    -e DW_LOG=stdout \
    -e DW_GIN_LOG=stdout \
    -e DW_DISKCACHE_CAPACITY_MB=65536 \
    pubrepo.guance.com/dataflux/dataway:1.12.1

• Verify Dataway Installation

After installation is complete, wait a moment and refresh the "Data Gateway" page. If you see a version number in the "Version Information" column for the newly added data gateway, it means this Dataway has successfully connected to the Guance center. Frontend users can then use it to ingest data.

Once Dataway successfully connects to the Guance center, log in to the Guance console. On the "Integration" / "DataKit" page, you can view all Dataway addresses. Select the required Dataway gateway address, obtain the DataKit installation command, and execute it on the server to start collecting data.

Manage DataWay¶

Delete DataWay¶

On the "Data Gateway" page in the Guance management console, select the DataWay you want to delete, click "Configure", and then click the "Delete" button in the bottom left corner of the Edit DataWay dialog box that pops up.

Upgrade DataWay¶

On the "Data Gateway" page in the Guance management console, if an upgrade is available for the DataWay, an upgrade prompt will appear in the version information.

DW_UPGRADE=1 bash -c "$(curl https://static.guance.com/dataway/install.sh)"

- image: pubrepo.guance.com/dataflux/dataway:1.12.1

Dataway Service Management¶

When Dataway is installed on a host, use the following commands to manage the Dataway service.

# Start
$ systemctl start dataway

# Restart
$ systemctl restart dataway

# Stop
$ systemctl stop dataway

For Kubernetes, restart the corresponding Pod.

Environment Variables¶

Image Environment Variables¶

The following environment variables are supported when Dataway runs in a Kubernetes environment.

HTTP Server Settings¶

HTTP TLS Settings¶

To generate a TLS certificate valid for one year, you can use the following OpenSSL command:

# Generate a TLS certificate valid for one year
$ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out tls.crt -keyout tls.key
...

After executing this command, you will be prompted to enter some necessary information, including your country, region, city, organization name, department name, and your email address. This information will be included in your certificate.

After completing the information entry, you will generate two files: tls.crt (the certificate file) and tls.key (the private key file). Please keep your private key file safe and ensure its security.

To enable the application to use these TLS certificates, you need to set the absolute paths of these two files into the application's environment variables. Here is an example of setting environment variables:

DW_ENABLE_TLS must be enabled first for the other two ENVs (DW_TLS_CRT/DW_TLS_KEY) to take effect. Version-1.4.1

env:
- name: DW_ENABLE_TLS
  value: "true"
- name: DW_TLS_CRT
  value: "/path/to/your/tls.crt"
- name: DW_TLS_KEY
  value: "/path/to/your/tls.key"

Replace /path/to/your/tls.crt and /path/to/your/tls.key with the actual paths where your tls.crt and tls.key files are stored.

After setting, you can test if TLS is working with the following command:

$ curl -k http://localhost:9528

If successful, an ASCII Art message saying It's working! will be displayed. If the certificate does not exist, the Dataway log will have an error similar to the following:

server listen(TLS) failed: open /path/to/your/tls.{crt,key}: no such file or directory

At this point, Dataway cannot start, and the curl command above will also report an error:

$ curl -vvv -k http://localhost:9528
curl: (7) Failed to connect to localhost port 9528 after 6 ms: Couldn't connect to server

Logging Settings¶

Token/UUID Settings¶

Sinker Settings¶

Prometheus Metrics Exposure¶

Disk Cache Settings¶

Performance Related Settings¶

Version-1.6.0

Dataway API List¶

Details for each API below are to be added.

GET /v1/ping¶

Version-1.11.0

• API Description: Get the current version number and release date of Dataway, and also return the egress IP of the client request.

If DataWay has the 404 page disabled (disable_404page), this interface will not be available.

GET /v1/ntp¶

Version-1.6.0

• API Description: Get the current Unix timestamp (in seconds) of Dataway.

POST /v1/write/:category¶

• API Description: Receive various collected data uploaded by Datakit.

GET /v1/datakit/pull¶

• API Description: Handle Datakit's request to pull central configuration (blacklist/Pipeline).

POST /v1/write/rum/replay¶

• API Description: Receive Session Replay data uploaded by Datakit.

POST /v1/upload/profiling¶

• API Description: Receive Profiling data uploaded by Datakit.

POST /v1/election¶

• API Description: Handle Datakit's election request.

POST /v1/election/heartbeat¶

• API Description: Handle Datakit's election heartbeat request.

POST /v1/query/raw¶

Handle DQL query requests. A simple example is as follows:

POST /v1/query/raw?token=<workspace-token> HTTP/1.1
Content-Type: application/json

{
    "token": "workspace-token",
    "queries": [
        {
            "query": "M::cpu LIMIT 1"
        }
    ],
    "echo_explain": <true/false>
}

Return example:

{
  "content": [
    {
      "series": [
        {
          "name": "cpu",
          "columns": [
            "time",
            "usage_iowait",
            "usage_total",
            "usage_user",
            "usage_guest",
            "usage_system",
            "usage_steal",
            "usage_guest_nice",
            "usage_irq",
            "load5s",
            "usage_idle",
            "usage_nice",
            "usage_softirq",
            "global_tag1",
            "global_tag2",
            "host",
            "cpu"
          ],
          "values": [
            [
              1709782208662,
              0,
              7.421875,
              3.359375,
              0,
              4.0625,
              0,
              0,
              0,
              1,
              92.578125,
              0,
              0,
              null,
              null,
              "WIN-JCHUL92N9IP",
              "cpu-total"
            ]
          ]
        }
      ],
      "points": null,
      "cost": "24.558375ms",
      "is_running": false,
      "async_id": "",
      "query_parse": {
        "namespace": "metric",
        "sources": {
          "cpu": "exact"
        },
        "fields": {},
        "funcs": {}
      },
      "index_name": "",
      "index_store_type": "",
      "query_type": "guancedb",
      "complete": false,
      "index_names": "",
      "scan_completed": false,
      "scan_index": "",
      "next_cursor_time": -1,
      "sample": 1,
      "interval": 0,
      "window": 0
    }
  ]
}

Return result description:

• The real data is located in the inner series field.
• name indicates the measurement name (here the CPU metric is queried; for log data, this field is absent).
• columns indicates the names of the returned result columns.
• values contains the corresponding column results for columns.

POST /v1/workspace¶

• API Description: Handle workspace query requests initiated by the Datakit end.

POST /v1/object/labels¶

• API Description: Handle requests to modify object Labels.

DELETE /v1/object/labels¶

• API Description: Handle requests to delete object Labels.

GET /v1/check/:token¶

• API Description: Check if the token is valid.

Dataway Metrics Collection¶

[[inputs.prom]]
  ## Exporter URLs.
  urls = [ "http://localhost:9090/metrics", ]
  source = "dataway"
  election = true
  measurement_name = "dw" # The dataway measurement set is fixed as dw, do not change it.
[inputs.prom.tags]
  service = "dataway"

annotations: # The following annotations are added by default.
   datakit/prom.instances: |
     [[inputs.prom]]
       url = "http://$IP:9090/metrics" # The port here (default 9090) depends on the situation.
       source = "dataway"
       measurement_name = "dw" # Fixed as this measurement set.
       interval = "10s"
       disable_instance_tag = true

     [inputs.prom.tags]
       service = "dataway"
       instance = "$PODNAME"

...
env:
- name: DW_PROM_LISTEN
  value: "0.0.0.0:9090" # Keep the port here consistent with the port in the url above.

If collection is successful, you can see the corresponding monitoring view by searching for dataway in the Guance "Scenes" / "Built-in Views".

Dataway Metrics List¶

The following are the metrics exposed by Dataway. These metrics can be obtained by requesting http://localhost:9090/metrics. You can use the following command to view a specific metric in real time (3s):

If some metrics cannot be queried, it may be because the relevant business module is not yet running. Some new metrics only exist in the latest version. The version information for each metric is not listed here one by one. Please refer to the metric list returned by the /metrics interface.

watch -n 3 'curl -s http://localhost:9090/metrics | grep -a <METRIC-NAME>'

Metrics Collection in Docker Mode¶

There are two modes for host installation: one is installation on the host machine, and the other is installation via Docker. Here we specifically explain the differences in metrics collection when installed via Docker.

When installed via Docker, the HTTP port exposed for metrics is mapped to port 19090 on the host machine (by default). In this case, the metrics collection address is http://localhost:19090/metrics.

If a different port is specified, during Docker installation, 10000 will be added to that port. Therefore, the specified port should not exceed 45535.

Additionally, when installed via Docker, the profile collection port is also exposed. By default, it is mapped to port 16060 on the host machine. Its mechanism is also to add 10000 to the specified port.

Dataway's Own Log Collection and Processing¶

Dataway's own Log is divided into two types: gin logs and its own program logs. They can be separated using the following Pipeline:

# Pipeline for dataway logging

# Testing sample loggin
'''
2023-12-14T11:27:06.744+0800    DEBUG   apis    apis/api_upload_profile.go:272  save profile file to disk [ok] /v1/upload/profiling?token=****************a4e3db8481c345a94fe5a
[GIN] 2021/10/25 - 06:48:07 | 200 |   30.890624ms |  114.215.200.73 | POST     "/v1/write/logging?token=tkn_5c862a11111111111111111111111111"
'''

add_pattern("TOKEN", "tkn_\\w+")
add_pattern("GINTIME", "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}%{SPACE}-%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}")
grok(_,"\\[GIN\\]%{SPACE}%{GINTIME:timestamp}%{SPACE}\\|%{SPACE}%{NUMBER:dataway_code}%{SPACE}\\|%{SPACE}%{NOTSPACE:cost_time}%{SPACE}\\|%{SPACE}%{NOTSPACE:client_ip}%{SPACE}\\|%{SPACE}%{NOTSPACE:method}%{SPACE}%{GREEDYDATA:http_url}")

# gin logging
if cost_time != nil {
  if http_url != nil  {
    grok(http_url, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(http_url, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  group_between(dataway_code, [200,299], "info", status)
  group_between(dataway_code, [300,399], "notice", status)
  group_between(dataway_code, [400,499], "warning", status)
  group_between(dataway_code, [500,599], "error", status)

  if sample(0.1) { # drop 90% debug log
    drop()
    exit()
  } else {
    set_tag(sample_rate, "0.1")
  }

  parse_duration(cost_time)
  duration_precision(cost_time, "ns", "ms")

  set_measurement('gin', true)
  set_tag(service,"dataway")
  exit()
}

# app logging
if cost_time == nil {
  grok(_,"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NOTSPACE:status}%{SPACE}%{NOTSPACE:module}%{SPACE}%{NOTSPACE:code}%{SPACE}%{GREEDYDATA:msg}")
  if level == nil {
    grok(message,"Error%{SPACE}%{DATA:errormsg}")
    if errormsg != nil {
      add_key(status,"error")
      drop_key(errormsg)
    }
  }
  lowercase(level)

  # if debug level enabled, drop most of them
  if status == 'debug' {
    if sample(0.1) { # drop 90% debug log
      drop()
      exit()
    } else {
      set_tag(sample_rate, "0.1")
    }
  }

  group_in(status, ["error", "panic", "dpanic", "fatal","err","fat"], "error", status) # mark them as 'error'

  if msg != nil {
    grok(msg, "%{TOKEN:token}")
    cover(token, [5, 15])
    replace(message, "tkn_\\w{0,5}\\w{6}", "****************$4")
    replace(msg, "tkn_\\w{0,5}\\w{6}", "****************$4")
  }

  set_measurement("dataway-log", true)
  set_tag(service,"dataway")
}

Dataway bug report¶

Dataway itself exposes metrics and profiling collection endpoints. We can collect this information for troubleshooting.

The following information collection is based on the actual configured ports and addresses. The existing commands are listed according to the default parameters.

br_dir="dw-br-$(date +%s)"
mkdir -p $br_dir

echo "save bug report to ${br_dir}"

# Modify the configuration here according to the actual situation.
dw_ip="localhost" # IP address where Dataway metrics/profile are exposed.
metric_port=9090  # Port for metrics exposure.
profile_port=6060 # Port for profile exposure.
dw_yaml_conf="/usr/local/cloudcare/dataflux/dataway/dataway.yaml"
dw_dot_yaml_conf="/usr/local/cloudcare/dataflux/dataway/.dataway.yaml" # This file exists for container installation.

# Collect runtime metrics.
curl -v "http://${dw_ip}:${metric_port}/metrics" -o $br_dir/metrics

# Collect profiling information.
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/allocs" -o $br_dir/allocs
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/heap" -o $br_dir/heap
curl -v "http://${dw_ip}:${profile_port}/debug/pprof/profile" -o $br_dir/profile # This command will run for about 30s.

cp $dw_yaml_conf $br_dir/dataway.yaml.copy
cp $dw_dot_yaml_conf $br_dir/.dataway.yaml.copy

tar czvf ${br_dir}.tar.gz ${br_dir}
rm -rf ${br_dir}

Run the script:

$ sh dw-bug-report.sh
...

After execution, a file similar to dw-br-1721188604.tar.gz will be generated. Extract this file.

FAQ¶

Request Body Too Large Issue¶

Version-1.3.7

Dataway has a default setting for the request body size (default 64MB). When the request body is too large, the client will receive an HTTP 413 error (Request Entity Too Large). If the request body is within a reasonable range, you can appropriately increase this value (in bytes):

• Set the environment variable DW_MAX_HTTP_BODY_BYTES
• Set max_http_body_bytes in dataway.yaml

If excessively large request packets occur during operation, they are reflected in both metrics and logs:

• The metric dataway_http_too_large_dropped_total exposes the number of dropped large requests.
• Search the Dataway log cat log | grep 'drop too large request'. The log will output details of the HTTP request headers, making it easier to understand the client situation.

```yaml - image: pubrepo.guance.com/dataflux/dataway:1.12.1

<!-- markdownlint-enable -->

### Dataway Service Management {#manage-service}

When Dataway is installed on a host, use the following commands to manage the Dataway service.

``` shell
# Start
$ systemctl start dataway

# Restart
$ systemctl restart dataway

# Stop
$ systemctl stop dataway

For Kubernetes, restart the corresponding Pod.

Environment Variables¶

Image Environment Variables¶

The following environment variables are supported when Dataway runs in a Kubernetes environment.

HTTP Server Settings¶

HTTP TLS Settings¶

To generate a TLS certificate valid for one year, you can use the following OpenSSL command:

# Generate a TLS certificate valid for one year
$ openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out tls.crt -keyout tls.key
...

After executing this command, you will be prompted to enter some necessary information, including your country, region, city, organization name, department name, and your email address. This information will be included in your certificate.

After completing the information entry, you will generate two files: tls.crt (the certificate file) and tls.key (the private key file). Please keep your private key file safe and ensure its security.

To enable the application to use these TLS certificates, you need to set the absolute paths of these two files into the application's environment variables. Here is an example of setting environment variables:

DW_ENABLE_TLS must be enabled first for the other two ENVs (DW_TLS_CRT/DW_TLS_KEY) to take effect. Version-1.4.1

env:
- name: DW_ENABLE_TLS
  value: "true"
- name: DW_TLS_CRT
  value: "/path/to/your/tls.crt"
- name: DW_TLS_KEY
  value: "/path/to/your/tls.key"

Replace /path/to/your/tls.crt and /path/to/your/tls.key with the actual paths where your tls.crt and tls.key files are stored.

After setting, you can test if TLS is working with the following command:

$ curl -k http://localhost:9528

If successful, an ASCII Art message saying It's working! will be displayed. If the certificate does not exist, the Dataway log will have an error similar to the following:

server listen(TLS) failed: open /path/to/your/tls.{crt,key}: no such file or directory

At this point, Dataway cannot start, and the curl command above will also report an error:

$ curl -vvv -k http://localhost:9528
curl: (7) Failed to connect to localhost port 9528 after 6 ms: Couldn't connect to server

Logging Settings¶

Token/UUID Settings¶

Sinker Settings¶

Prometheus Metrics Exposure¶

Disk Cache Settings¶

Performance Related Settings¶

Version-1.6.0

Dataway API List¶

Details for each API below are to be added.

GET /v1/ping¶

Version-1.11.0

• API Description: Get the current version number and release date of Dataway, and also return the egress IP of the client request.

If DataWay has the 404 page disabled (disable_404page), this interface will not be available.

GET /v1/ntp¶

Version-1.6.0

• API Description: Get the current Unix timestamp (in seconds) of Dataway.

POST /v1/write/:category¶

• API Description: Receive various collected data uploaded by Datakit.

GET /v1/datakit/pull¶

• API Description: Handle Datakit's request to pull central configuration (blacklist/Pipeline).

POST /v1/write/rum/replay¶

• API Description: Receive Session Replay data uploaded by Datakit.

POST /v1/upload/profiling¶

• API Description: Receive Profiling data uploaded by Datakit.

POST /v1/election¶

• API Description: Handle Datakit's election request.

POST /v1/election/heartbeat¶

• API Description: Handle Datakit's election heartbeat request.

POST /v1/query/raw¶

Handle DQL query requests. A simple example is as follows:

POST /v1/query/raw?token=<workspace-token> HTTP/1.1
Content-Type: application/json

{
    "token": "workspace-token",
    "queries": [
        {
            "query": "M::cpu LIMIT 1"
        }
    ],
    "echo_explain": <true/false>
}

Return example:

{
  "content": [
    {
      "series": [
        {
          "name": "cpu",
          "columns": [
            "time",
            "usage_iowait",
            "usage_total",
            "usage_user",
            "usage_guest",
            "usage_system",
            "usage_steal",
            "usage_guest_nice",
            "usage_irq",
            "load5s",
            "usage_idle",
            "usage_nice",
            "usage_softirq",
            "global_tag1",
            "global_tag2",
            "host",
            "cpu"
          ],
          "values": [
            [
              1709782208662,
              0,
              7.421875,
              3.359375,
              0,
              4.0625,
              0,
              0,
              0,
              1,
              92.578125,
              0,
              0,
              null,
              null,
              "WIN-JCHUL92N9IP",
              "cpu-total"
            ]
          ]
        }
      ],
      "points": null,
      "cost": "24.558375ms",
      "is_running": false,
      "async_id": "",
      "query_parse": {
        "namespace": "metric",
        "sources": {
          "cpu": "exact"
        },
        "fields": {},
        "funcs": {}
      },
      "index_name": "",
      "index_store_type": "",
      "query_type": "guancedb",
      "complete": false,
      "index_names": "",
      "scan_completed": false,
      "scan_index": "",
      "next_cursor_time": -1,
      "sample": 1,
      "interval": 0,
      "window": 0
    }
  ]
}

Return result description:

• The real data is located in the inner series field.
• name indicates the measurement name (here the CPU metric is queried; for log data, this field is absent).
• columns indicates the names of the returned result columns.
• values contains the corresponding column results for columns.

POST /v1/workspace¶

• API Description: Handle workspace query requests initiated by the Datakit end.

POST /v1/object/labels¶

• API Description: Handle requests to modify object Labels.

DELETE /v1/object/labels¶

• API Description: Handle requests to delete object Labels.

GET /v1/check/:token¶

• API Description: Check if the token is valid.

Dataway Metrics Collection¶

[[inputs.prom]]
  ## Exporter URLs.
  urls = [ "http://localhost:9090/metrics", ]
  source = "dataway"
  election = true
  measurement_name = "dw" # The dataway measurement set is fixed as dw, do not change it.
[inputs.prom.tags]
  service = "dataway"