Dataway¶

Introduction¶

DataWay is the data gateway of the observation cloud, and the collector needs to pass through the DataWay gateway to report data to the observation cloud

Dataway Installation¶

• New Dataway

On the Data Gateways page in the observation cloud management console, click Create Dataway. Enter a name and binding address, and then click Create.

After the creation is successful, a new Dataway is automatically created and the installation script for the Dataway is generated.

• Install Dataway

DW_KODO=http://kodo_ip:port \
   DW_TOKEN=<tkn_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX> \
   DW_UUID=<YOUR_UUID> \
   bash -c "$(curl https://static.guance.com/dataway/install.sh)"

uuid                  : not-set
token                 : tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
secret_token          : ""
enable_internal_token : false
enable_empty_token    : false
http_timeout          : 30s
insecure_skip_verify  : false
remote_host           : https://kodo.guance.com:443
cascaded              : false
bind                  : 0.0.0.0:9528
api_limit_rate        : 100000
max_http_body_bytes   : 67108864
heartbeat_second      : 60
http_client_trace     : true
white_list            : []

log : log
log_level : debug
gin_log   : gin.log

cache_cfg:
  disabled            : false
  max_disk_size       : 20480
  max_data_size       : 67108864
  batch_size          : 67108864
  dir : disk_cache
  clean_interval      : 30s
  expire_duration: 24h

sinker: null

prometheus:
  url                 : /metrics
  listen              : 0.0.0.0:9091
  enable              : true
  disablegometrics    : false

$ wget https://static.guance.com/dataway/dataway.yaml -O dw-deployment.yaml
$ kubectl apply -f dw-deployment.yaml

• Verify Dataway installation

After installation, wait for a while to refresh the "Data Gateway" page, if you see the version number in the "Version Information" column of the data gateway you just added, it means that the Dataway has been successfully connected to the observation cloud center, and front-end users can access data through it.

After Dataway is successfully connected to the observation cloud center, log in to the observation cloud console, view all Dataway addresses on the "Integration" / DataKit page, select the required Dataway gateway address, and obtain the DataKit installation command to execute on the server to start collecting data.

Manage DataWay¶

Delete DataWay¶

On the "Data Gateway" page of the observation cloud management background, select the DataWay to be deleted, click "Configure", and click the "Delete" button in the lower left corner of the pop-up Edit DataWay dialog box.

Upgrade DataWay¶

On the Data Gateways page in the observation cloud management background, if an upgradeable version exists for DataWay, an upgrade prompt appears in the version information.

DW_UPGRADE=1 bash -c "$(curl https://static.guance.com/dataway/install.sh)"

- image: pubrepo.jiagouyun.com/dataflux/dataway:<VERSION>

Dataway Service Management¶

When the host installs Dataway, you can use the following command to manage the Dataway service:

# Start
$ systemctl start dataway

# Reboot
$ systemctl restart dataway

# Stop
$ systemctl stop dataway

Kubernetes can restart the corresponding pod.

Environment variable¶

Host installation supports environment variable¶

When installing a host, you can inject the following environment variables into the installation command:

Docker image environment variable¶

When Dataway runs in a Kubernetes environment, it supports the following environment variables.

API about¶

logabout¶

Token/UUID Settings¶

Sinker About setting¶

Prometheus metrics expose¶

Disk cache settings¶

Dataway API List¶

Details of each API below are to be added.

POST /v1/write/:category¶

• API description: Receive various collection data uploaded by Datakit

GET /v1/datakit/pull¶

• API description: Handles Datakit pull center configuration (blacklist/pipeline) requests

POST /v1/write/rum/replay¶

• API description: Receive Session Replay data uploaded by Datakit

POST /v1/upload/profiling¶

• API description: Receive profiling data uploaded by Datakit

POST /v1/election¶

• API description: Handles election requests for Datakit

POST /v1/election/heartbeat¶

• API description: Handles election heartbeat requests for Datakit

POST /v1/query/raw¶

• API description: Handles DQL query requests initiated by the Datakit side

POST /v1/workspace¶

• API description: Handles workspace query requests initiated by Datakit

POST /v1/object/labels¶

• API description: Handles requests to modify object labels

DELETE /v1/object/labels¶

• API description: Handles delete object Label requests

GET /v1/check/:token¶

• API description: Detect if tokken is legitimate

Dataway metrics collection¶

[[inputs.prom]]
  ## Exporter URLs.
  urls = [ "http://localhost:9090/metrics", ]

  source = "dataway"

  election = true

  ## Dataway metric set fixed to dw, do not change
  measurement_name = "dw"

annotations:
   datakit/prom.instances: |
     [[inputs.prom]]
       url = "http://$IP:9090/metrics" # Here the port (default 9090) is as appropriate
       source = "dataway"
       measurement_name = "dw" # pinned to this metric set
       interval = "30s"

       [inputs.prom.tags]
         namespace = "$NAMESPACE"
         pod_name = "$PODNAME"
         node_name = "$NODENAME"

If the collection is successful, search for dataway in the "Scene"/"Built-in View" of the observation cloud to see the corresponding monitoring view.

Dataway Metric List¶

The following are the indicators exposed by Dataway, which can be obtained by requesting http://localhost:9090/metrics, and you can view (3s) a specific indicator in real time by following the following command:

If some metrics cannot be queried, it may be caused by the relevant business module not running.

watch -n 3 'curl -s http://localhost:9090/metrics | grep -a <METRIC-NAME>'