Dataway Sink¶

Version-1.14.0 The Sinker feature in this section can only be used with Datakit version 1.14.0 or later.

Dataway Sinker Feature Introduction¶

In daily data collection, due to the existence of multiple different workspaces, we may need to send different data to different workspaces. For example, in a shared Kubernetes cluster, the collected data may involve different teams or business departments. In this case, we can send data with specific attributes to different workspaces, achieving fine-grained collection in a shared infrastructure scenario.

The processing flow of a Sink request is as follows:

sequenceDiagram
autonumber

participant dk as DataKit

box Dataway server
participant etcd
participant dw as DataWay
participant rmatch as Rule matching
participant drop as Drop
end

box Workspaces
participant wksp1 as Workspace
participant wkspx as Default workspace
end

etcd ->> dw: pull sinker rules

activate dw
dk ->> dw: upload
deactivate dw

alt non-sink request
    dw ->> wksp1: write
else sink request
    dw ->> rmatch: matching
end

alt match ok(at lease 1 workspace)
    rmatch ->> wksp1: write
else match failed but enabled default workspace
    rmatch ->> wkspx: write
else no default workspace
    rmatch ->> drop: drop
end

Dataway 1.8.0 supports receiving both Sinker and non-Sinker requests. Only one Dataway needs to be deployed.

Dataway Cascaded Mode¶

For SaaS users, you can deploy a Dataway locally (in your k8s cluster) dedicated to traffic splitting, and then forward data to Openway:

sequenceDiagram
autonumber

participant dk as DataKit

box local Dataway server
participant etcd
participant dw1 as DataWay
end

box SAAS Dataway server
participant dw2 as DataWay
end

box Workspaces
participant wksp1 as Workspace
end

etcd ->> dw1: pull sinker rules
dk ->> dw1: upload data
dw1 ->> dw1: sink rule matching
dw1 ->> dw2: deliver request
dw2 ->> wksp1: write

Impacts of cascading:

• Some API behaviors will differ. Due to historical reasons, the request URLs sent by Datakit differ from those on Kodo. Dataway plays an API translation role here. In cascaded mode, the API translation function is disabled.
• The cascaded Dataway will not send heartbeat requests to the center, because the downstream Dataway does not handle this request (resulting in a 404).
• When the cascaded Dataway sends a received request to the next Dataway, it does not sign the API.

Dataway Installation¶

See here

Dataway Configuration¶

In addition to the regular Dataway configuration, several additional settings need to be configured (located in /usr/local/cloudcare/dataflux/dataway/dataway.yaml):

# Set the upload address for Dataway, usually Kodo, but can also be another Dataway
remote_host: https://kodo.guance.com

# If the upload address is another Dataway, set this to true to indicate cascaded Dataway
cascaded: false

# This token is arbitrarily set on the Dataway. It needs to be filled in the
# Datakit configuration (datakit.conf). It must have a certain length and format.
secret_token: tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Sinker rule settings
sinker:
  etcd: # Supports etcd
    urls:
    - http://localhost:2379
    dial_timeout: 30s
    key_space: /dw_sinker
    username: "dataway"
    password: "<PASSWORD>"

  #file: # Also supports local file mode, commonly used for debugging
  #  path: /path/to/sinker.json

Sinker Rules Configuration¶

Dataway Sinker rules are a set of JSON configurations. The matching rule syntax is the same as the blocklist format. See here.

Currently, two configuration sources are supported:

• Specify a local JSON file, mainly used for debugging Sinker rules. In this case, after updating the Sinker rules in the JSON file, Dataway needs to be restarted for the changes to take effect.
• etcd: Store the debugged rule file in etcd. When fine-tuning rules later, just update etcd directly, no need to restart Dataway.

In fact, the JSON stored in etcd is the same as the local JSON file. Below only introduces the etcd hosting method.

etcd Configuration¶

The following commands are all operated on Linux.

Dataway, as an etcd client, can set the following username and role in etcd (etcd 3.5+). See here

Create the dataway account and corresponding role:

# Add username, will prompt for password
$ etcdctl user add dataway

# Add the sinker role
$ etcdctl role add sinker

# Add dataway to the role
$ etcdctl user grant-role dataway sinker

# Restrict key permissions for the role (/dw_sinker and /ping are the two keys used by default)
$ etcdctl role grant-permission sinker readwrite /dw_sinker
$ etcdctl role grant-permission sinker readwrite /ping       # Used for connectivity check

$ etcdctl --user name:password ...

Writing Sinker Rules¶

Newer versions (1.3.6) of Dataway support using the dataway command to operate Sinker rules in etcd.

Assume the sinker.json rules are defined as follows:

{
    "strict":true,
    "rules": [
        {
            "rules": [
                "{ host = 'my-host'}"
            ],
            "url": "https://kodo.guance.com?token=tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        },
        {
            "rules": [
                "{ host = 'my-host' OR cluster = 'cluster-A' }"
            ],
            "url": "https://kodo.guance.com?token=tkn_yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
        }
     ]
}

To write the Sinker rules configuration, use the following command:

$ etcdctl --user dataway:PASSWORD put /dw_sinker "$(<sinker.json)"
OK

{
    "rules": [
        "{ host = 'my-host' OR cluster = 'cluster-A' }"
    ],
    "info": "This is the yyy workspace",
    "url": "https://kodo.guance.com?token=tkn_yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
}

Default Rule¶

Version-1.6.0

In a specific rule entry, add the as_default flag to set that rule as the default fallback rule. The fallback rule can be set without any matching conditions (no rules field configured); also, it should not participate in regular rule matching. A suggested fallback rule is as follows:

{
    "as_default": true,
    "info": "This is the default fallback workspace",
    "url": "https://kodo.guance.com?token=tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

Note: Only one fallback rule should be set. If there are multiple fallback rules in the sinker configuration, the last one will take effect.

Token Rules¶

Since Datakit checks the token on Dataway, the token set here (including secret_token) must meet the following conditions:

Starts with token_ or tkn_, followed by 32 characters.

If the token does not meet these conditions, Datakit will fail to install.

Datakit Side Configuration¶

In Datakit, we need to make several settings to allow it to tag the collected data with specific labels for grouping.

• Configure the global custom key list

Datakit will look for fields with these keys (only string-type fields) in the collected data, extract them, and use them as the basis for grouping and sending.

• Configure "Global Host Tags" and "Global Election Tags"

All data uploaded by Datakit will carry these configured global tags (including tag key and tag value) as the basis for grouping and sending.

Datakit Customer Key Settings¶

If you want the data collected by a specific Datakit to meet the traffic splitting requirements, ensure the following:

• Datakit has the Sinker feature enabled.
• Effective Global Customer Keys are configured in Datakit.

The two configurations are as follows:

# /usr/local/datakit/conf.d/datakit.conf
[dataway]

  # Specify a set of customer keys
  global_customer_keys = [
    # Example: add two keys: category and class
    # Do not configure too many keys here, generally 2 to 3 are sufficient.
    "category",
    "class",
  ]

  # Enable the sinker feature
  enable_sinker = true

In addition to synthetic test data and regular data categories, it also supports binary data such as Session Replay and Profiling. Therefore, all field names can be selected here. One important note is do not configure non-string type fields. Normal keys usually come from Tags (all Tag values are string types). Datakit will not use non-string type fields as the basis for traffic splitting.

Impact of Global Tags on Sink¶

In addition to global_customer_keys, the global tags configured on Datakit (including global election tags and global host tags) also affect the traffic splitting markers. That is, if a data point contains fields that appear in the global tags (these field values must be string types), they will also be included in the traffic splitting. Assume the global election tags are as follows:

# datakit.conf
[election.tags]
    cluster = "my-cluster"

For the following data point:

pi,cluster=cluster_A,app=math,other_tag=other_value value=3.14 1712796013000000000

Since the global election tag contains cluster (regardless of the configured value), and the data point itself also has the cluster tag, the final X-Global-Tags will append the key-value pair cluster=cluster_A:

X-Global-Tags: cluster=cluster_A

If global_customer_keys also configures the app key, the final traffic splitting header will be (the order of the two key-value pairs does not matter):

X-Global-Tags: cluster=cluster_A,app=math

Dataway Sink Command¶

Starting from Dataway version Version-1.3.6, the sinker configuration can be managed via the command line. The usage is as follows:

$ ./dataway sink --help

Usage of sink:
  -add string
        single rule json file
  -cfg-file string
        configure file (default "/usr/local/cloudcare/dataflux/dataway/dataway.yaml")
  -file string
        file path of the rule json, only used for command put and get
  -get
        get the rule json
  -list
        list rules
  -log string
        log file path (default "/dev/null")
  -put
        save the rule json
  -token string
        rules filtered by token, eg: xx,yy

Specify configuration file

When executing the command, the default configuration file loaded is /usr/local/cloudcare/dataflux/dataway/dataway.yaml. If you need to load a different configuration, you can specify it using --cfg-file.

$ ./dataway sink --cfg-file dataway.yaml [--list...]

Command log settings

By default, the command's output log is disabled. If you need to view it, you can set the --log parameter.

# output log to stdout
$ ./dataway sink --list --log stdout

# output log to file
$ ./dataway sink --list --log /tmp/log

View rule list

# list all rules
$ ./dataway sink --list

# list all rules filtered by token
$ ./dataway sink --list --token=token1,token2

CreateRevision: 2
ModRevision: 41
Version: 40
Rules:
[
    {
        "rules": [
            "{ workspace = 'zhengb-test'}"
        ],
        "url": "https://openway.guance.com?token=token1"
    }
]

Add rule

Create a rule file rule.json with the following content:

[
  {
    "rules": [
      "{ host = 'HOST1'}"
    ],
    "url": "https://openway.guance.com?token=tkn_xxxxxxxxxxxxx"
  },
  {
    "rules": [
      "{ host = 'HOST2'}"
    ],
    "url": "https://openway.guance.com?token=tkn_yyyyyyyyyyyyy"
  }
]

Add the rule:

$ ./dataway sink --add rule.json

add 2 rules ok!

Export configuration

Export the sinker configuration to a local file.

$ ./dataway sink --get --file sink-get.json

rules json was saved to sink-get.json!

Write configuration

Write the local rule file to the sinker configuration.

Create a rule file sink-put.json with the following content:

{
    "rules": [
        {
            "rules": [
                "{ workspace = 'test'}"
            ],
            "url": "https://openway.guance.com?token=tkn_xxxxxxxxxxxxxx"
        }
    ],
    "strict": true
}

Write the configuration:

$ ./dataway sink --put --file sink-put.json

Configuration Examples¶

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: deployment-utils-dataway
  name: dataway
  namespace: utils
spec:
  replicas: 1
  selector:
    matchLabels:
      app: deployment-utils-dataway
  template:
    metadata:
      labels:
        app: deployment-utils-dataway
      annotations:
        datakit/logs: |
          [{"disable": true}]
        datakit/prom.instances: |
          [[inputs.prom]]
            url = "http://$IP:9090/metrics" # Port (default 9090) depends on the situation
            source = "dataway"
            measurement_name = "dw" # Fixed to this measurement
            interval = "10s"

            [inputs.prom.tags]
              namespace = "$NAMESPACE"
              pod_name = "$PODNAME"
              node_name = "$NODENAME"
    spec:
      affinity:
        podAffinity: {}
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - deployment-utils-dataway
              topologyKey: kubernetes.io/hostname

      containers:
      - image: registry.jiagouyun.com/dataway/dataway:1.3.6 # Choose the appropriate version here
        #imagePullPolicy: IfNotPresent
        imagePullPolicy: Always
        name: dataway
        env:
        - name: DW_REMOTE_HOST
          value: "http://kodo.forethought-kodo:9527" # Fill in the actual Kodo address or the next Dataway address
        - name: DW_BIND
          value: "0.0.0.0:9528"
        - name: DW_UUID
          value: "agnt_xxxxx" # Fill in the actual Dataway UUID
        - name: DW_TOKEN
          value: "tkn_oooooooooooooooooooooooooooooooo" # Fill in the actual Dataway token, usually the token of the system workspace
        - name: DW_PROM_LISTEN
          value: "0.0.0.0:9090"
        - name: DW_SECRET_TOKEN
          value: "tkn_zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"
        - name: DW_SINKER_FILE_PATH
          value: "/usr/local/cloudcare/dataflux/dataway/sinker.json"
        ports:
        - containerPort: 9528
          name: 9528tcp01
          protocol: TCP
        volumeMounts:
          - mountPath: /usr/local/cloudcare/dataflux/dataway/cache
            name: dataway-cache
          - mountPath: /usr/local/cloudcare/dataflux/dataway/sinker.json
            name: sinker
            subPath: sinker.json
        resources:
          limits:
            cpu: '4'
            memory: 4Gi
          requests:
            cpu: 100m
            memory: 512Mi
      # nodeSelector:
      #   key: string
      imagePullSecrets:
      - name: registry-key
      restartPolicy: Always
      volumes:
      - hostPath:
          path: /root/dataway_cache
        name: dataway-cache
      - configMap:
          name: sinker
        name: sinker
---

apiVersion: v1
kind: Service
metadata:
  name: dataway
  namespace: utils
spec:
  ports:
  - name: 9528tcp02
    port: 9528
    protocol: TCP
    targetPort: 9528
    nodePort: 30928
  selector:
    app: deployment-utils-dataway
  type: NodePort

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: sinker
  namespace: utils
data:
  sinker.json: |
    {
        "strict":true,
        "rules": [
            {
                "rules": [
                    "{ project = 'xxxxx'}"
                ],
                "url": "http://kodo.forethought-kodo:9527?token=tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
            },
            {
                "rules": [
                    "{ project = 'xxxxx'}"
                ],
                "url": "http://kodo.forethought-kodo:9527?token=tkn_yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
            }
         ]
    }

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dataway-sinker
  namespace: utils
spec:
  ingressClassName: nginx
  rules:
  - host: datawaysinker-xxxx.com
    http:
      paths:
      - backend:
          service:
            name: dataway
            port:
              number: 9528
        path: /
        pathType: ImplementationSpecific

FAQ¶

View Details of Dropped Requests¶

Version-1.3.7

When a request does not match the Sinker rules, Dataway will drop it and increment a drop counter in the metrics. However, during debugging, we need to know the specifics of a dropped request, especially the X-Global-Tags information carried in the request header.

We can search the Dataway log using the following command:

$ cat <path/to/dataway/log> | grep dropped

In the output, we can see similar output as follows:

for API /v1/write/logging with X-Global-Tags <some-X-Global-Tags...> dropped

Troubleshooting Datakit Requests Being Dropped¶

Version-1.3.9

When a Datakit request is dropped by Dataway, Dataway returns a corresponding HTTP error. In the Datakit log, there will be similar errors as follows:

post 3641 to http://dataway-ip:9528/v1/write/metric failed(HTTP: 406 Not Acceptable):
{"error_code":"dataway.sinkRulesNotMatched","message":"X-Global-Tags: `host=my-host',
URL: `/v1/write/metric'"}, data dropped

This error indicates that the request /v1/write/metric was dropped because its X-Global-Tags did not satisfy any rule on Dataway.

At the same time, in the DataWay APIs panel in the lower right corner of Datakit monitor (datakit monitor -V), the Status column will show Not Acceptable, indicating that the corresponding Dataway API request was dropped.

Viewing Datakit's own metrics also shows the corresponding metrics:

$ curl -s http://localhost:9529/metrics | grep datakit_io_dataway_api_latency_seconds_count

datakit_io_dataway_api_latency_seconds_count{api="/v1/datakit/pull",status="Not Acceptable"} 50
datakit_io_dataway_api_latency_seconds_count{api="/v1/write/metric",status="Not Acceptable"} 301

Datakit Error 403¶

If the Sinker configuration on Dataway is incorrect, causing all Datakit requests to use secret_token, and this token is not recognized by the center (Kodo), a 403 error kodo.tokenNotFound will be reported.

This problem may be caused by incorrect etcd username/password, causing Dataway to fail to retrieve the Sinker configuration, so Dataway considers the current Sinker invalid and all data is sent directly to the center.

etcd Permission Configuration Issues¶

If the following error appears in the Dataway log, the permission settings may be problematic:

sinker ping: etcdserver: permission denied, retrying(97th)

If the permissions are configured incorrectly, you can delete all existing permissions based on Dataway and reconfigure them. See here

Key Override Relationships on Datakit¶

When configuring the "Global Custom Key List", if the "Global Host Tags" and "Global Election Tags" also have the same key, the corresponding Key-Value pair from the collected data will be used.

For example, if the configured "Global Custom Key List" includes key1,key2,key3, and the "Global Host Tags" or "Global Election Tags" also configure these keys with specified values, e.g., key1=value-1, and in a certain data collection, a field key1=value-from-data is also present, then the final grouping basis will use key1=value-from-data from the data, ignoring the corresponding Key's Value from the "Global Host Tags" or "Global Election Tags".

If there is a key with the same name in both "Global Host Tags" and "Global Election Tags", the key in "Global Election Tags" takes precedence. In summary, the priority of the Key's value source (descending) is:

• Collected data
• Global Election Tags
• Global Host Tags

Built-in "Global Custom Keys"¶

Datakit has several built-in custom keys that can be used. They generally do not appear in the collected data, but Datakit can use these keys to group data. If there is a need for traffic splitting based on these key dimensions, they can be added to the "Global Custom Key List" (these keys are not configured by default). We can use the following built-in custom keys to achieve data traffic splitting.

• class For object data, after enabling, traffic splitting will be based on the object's class. For example, a Pod's object class is kubelet_pod, so you can create a traffic splitting rule for pods:

{
    "strict": true,
    "rules": [
        {
            "rules": [
                "{ class = 'kubelet_pod' AND other_conditon = 'some-value' }",
            ],
            "url": "https://kodo.guance.com?token=<YOUR-TOKEN>"
        },
        {
            ... # other rules
        }
    ]
}

• measurement For metrics data, you can send specific measurements to a specific workspace. For example, the measurement name for disk is disk. You can write the rule like this:

{
    "strict": true,
    "rules": [
        {
           "rules": [
               "{ measurement = 'disk' AND other_conditon = 'some-value' }",
           ],
           "url": "https://kodo.guance.com?token=<YOUR-TOKEN>"
        },
        {
            ... # other rules
        }
    ]
}

• source For logs (L), eBPF network metrics (N), events (E), and RUM data
• service For Tracing, Scheck, and Profiling
• category For all regular data categories. Its value corresponds to the "Name" column of the data category (e.g., time series is metric, object is object). Taking logs as an example, you can create a traffic splitting rule specifically for logs:

{
    "strict": true,
    "rules": [
        {
            "rules": [
                "{ category = 'logging' AND other_conditon = 'some-value' }",
            ],
            "url": "https://kodo.guance.com?token=<YOUR-TOKEN>"
        },
        {
            ... # other rules
        }
    ]
}

Special Traffic Splitting Behavior¶

Starting from Version-2.0.0 and later, Datakit will only add the Dataway Sinker header to point write requests (/v1/write/*). Other Dataway APIs that pull resources from the center, perform identity recognition, or synchronize configurations will no longer carry X-Global-Tags/X-Global-Tags-V2. Therefore, there is no need to add special traffic splitting rules for these APIs.

For versions before Datakit 2.0.0, some requests initiated by Datakit are intended to pull resources from the center or perform self-identity recognition. Their behavior is inherently atomic and cannot be further divided. These requests cannot be distributed to multiple workspaces (because Datakit needs to process the responses of these API requests and determine its subsequent behavior). Therefore, these APIs can only be split to at most one workspace.

If multiple conditions are met in the traffic splitting rule, these APIs will only be split to the workspace pointed to by the first rule that matches.

Below is an example of a compatibility rule available for versions before Datakit 2.0.0:

Only needed when an older version of Datakit needs to fix these APIs to a specific workspace. Datakit 2.0.0 and later do not need to add this.

{
    "strict": true,
    "info": "Special workspace (only used for data pull APIs, etc.)",
    "rules": [
        {
            "rules": [
                "{ __dataway_api in ['/v1/datakit/pull', '/v1/election', '/v1/election/heartbeat', '/v1/query/raw', '/v1/workspace', '/v1/object/labels', '/v1/check/token'] }"
            ],
            "url": "https://kodo.guance.com?token=<SOME-SPECIAL-WORKSPACE-TOKEN>"
        }
    ]
}

Before Datakit 2.0.0, the key __dataway_api does not need to be configured in global_customer_keys in datakit.conf. Dataway will treat it as a key for traffic splitting by default, and its value will be the current request's API route. That is, for a certain API:

POST /v1/some/api
X-Global-Tags: cluster=cluster_A,app=math

The final effect in traffic splitting is equivalent to the following:

POST /v1/some/api
X-Global-Tags: cluster=cluster_A,app=math,__dataway_api=/v1/write/metrics

Therefore, in the compatibility scenario for older versions, you can directly use the KV pair __dataway_api in the Sink rule for matching. This also reminds us that in this special rule, do not include other important data upload APIs, such as /v1/write/... interfaces, otherwise the data will end up in an undefined workspace. Starting from Datakit 2.0.0 and later, these non-point write APIs no longer carry the Sinker header, so there is no need to rely on the __dataway_api rule for traffic splitting.