Dataway Sink¶

Version-1.14.0 Only Datakit versions from this release can use the Sinker feature described here.

Dataway Sinker Features¶

During daily data collection, due to the existence of multiple different workspaces, we may need to send different data to different workspaces. For example, in a shared Kubernetes cluster, the collected data may involve different teams or business departments. In this case, we can send data with specific attributes to their respective workspaces to achieve fine-grained collection in shared infrastructure scenarios.

The Sink request processing flow is as follows:

sequenceDiagram
autonumber

participant dk as DataKit

box Dataway server
participant etcd
participant dw as DataWay
participant rmatch as Rule matching
participant drop as Drop
end

box Workspaces
participant wksp1 as Workspace
participant wkspx as Default workspace
end

etcd ->> dw: pull sinker rules

activate dw
dk ->> dw: upload
deactivate dw

alt non-sink request
    dw ->> wksp1: write
else sink request
    dw ->> rmatch: matching
end

alt match ok(at lease 1 workspace)
    rmatch ->> wksp1: write
else match failed but enabled default workspace
    rmatch ->> wkspx: write
else no default workspace
    rmatch ->> drop: drop
end

Dataway 1.8.0 supports receiving both Sinker and non-Sinker requests, allowing deployment of a single Dataway.

Dataway Cascaded Mode¶

For SaaS users, you can deploy a Dataway locally (in your k8s Cluster) specifically for traffic splitting, and then forward the data to Openway:

sequenceDiagram
autonumber

participant dk as DataKit

box local Dataway server
participant etcd
participant dw1 as DataWay
end

box SAAS Dataway server
participant dw2 as DataWay
end

box Workspaces
participant wksp1 as Workspace
end

etcd ->> dw1: pull sinker rules
dk ->> dw1: upload data
dw1 ->> dw1: sink rule matching
dw1 ->> dw2: deliver request
dw2 ->> wksp1: write

Impact of cascading:

• Some API behaviors will differ. Due to historical reasons, there are differences between the requests sent by Datakit and the request URLs on Kodo. Dataway plays a role in API translation here. In cascaded scenarios, the API translation feature is disabled.
• A cascaded Dataway will not send heartbeat requests to the center. Because the next-level Dataway does not handle this request (resulting in a 404).
• Requests received by a cascaded Dataway will not have their API signed when forwarded to the next Dataway.

Dataway Installation¶

Refer to here

Dataway Settings¶

In addition to the regular Dataway settings, several extra configurations need to be set (located in /usr/local/cloudcare/dataflux/dataway/dataway.yaml):

# Set the upload address for Dataway here, usually Kodo, but can also be another Dataway
remote_host: https://kodo.guance.com

# If the upload address is a Dataway, set this to true, indicating Dataway cascading
cascaded: false

# This token is a randomly set token on the dataway. We need to fill it into
# the datakit.conf configuration of Datakit. It should maintain a certain length and format.
secret_token: tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# sinker rule settings
sinker:
  etcd: # supports etcd
    urls:
    - http://localhost:2379
    dial_timeout: 30s
    key_space: /dw_sinker
    username: "dataway"
    password: "<PASSWORD>"

  #file: # also supports local file mode, commonly used for debugging
  #  path: /path/to/sinker.json

Sinker Rule Settings¶

Dataway Sinker rules are a set of configurations in JSON format. The matching rule syntax is consistent with the blacklist syntax. Refer to here.

Currently, two configuration sources are supported:

• Specify a local JSON file, mainly used for debugging Sinker rules. In this case, after updating the Sinker rules in the JSON file, you need to restart Dataway for the changes to take effect.
• etcd: Store the debugged rule file in etcd. Later, when fine-tuning rules, you can directly update etcd, without restarting Dataway.

In fact, the JSON stored in etcd is the same as the JSON in the local file. Only the etcd hosting method is introduced below.

etcd Settings¶

The following commands are operated in Linux.

As an etcd client, Dataway can set the following username and role in etcd (etcd 3.5+). Refer to here

Create the dataway account and corresponding role:

# Add a username, you will be prompted to enter a password here
$ etcdctl user add dataway

# Add the sinker role
$ etcdctl role add sinker

# Add dataway to the role
$ etcdctl user grant-role dataway sinker

# Restrict the key permissions for the role (here /dw_sinker and /ping are the two keys used by default)
$ etcdctl role grant-permission sinker readwrite /dw_sinker
$ etcdctl role grant-permission sinker readwrite /ping       # Used for connectivity detection

$ etcdctl --user name:password ...

Writing Sinker Rules¶

Newer versions (1.3.6) of Dataway support managing Sinker rules in etcd via the dataway command.

Assume the sinker.json rule definition is as follows:

{
    "strict":true,
    "rules": [
        {
            "rules": [
                "{ host = 'my-host'}"
            ],
            "url": "https://kodo.guance.com?token=tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        },
        {
            "rules": [
                "{ host = 'my-host' OR cluster = 'cluster-A' }"
            ],
            "url": "https://kodo.guance.com?token=tkn_yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
        }
     ]
}

You can write the Sinker rule configuration with the following command:

$ etcdctl --user dataway:PASSWORD put /dw_sinker "$(<sinker.json)"
OK

{
    "rules": [
        "{ host = 'my-host' OR cluster = 'cluster-A' }"
    ],
    "info": "This is the yyy workspace",
    "url": "https://kodo.guance.com?token=tkn_yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
}

Default Rule¶

Version-1.6.0

Add the as_default flag to a specific rule entry to set it as the default fallback rule. The fallback rule can have no matching conditions set (no rules field configured); at the same time, it should not participate in regular rule matching. A suggested fallback rule is as follows:

{
    "as_default": true,
    "info": "This is the default fallback workspace",
    "url": "https://kodo.guance.com?token=tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

Note: Only one fallback rule should be set. If multiple fallback rules exist in the sinker configuration, the last one will be used.

Token Rules¶

Since Datakit will check the token on Dataway, the token (including secret_token) set here must meet the following conditions:

Starts with token_ or tkn_, followed by 32 characters.

For tokens that do not meet this condition, Datakit installation will fail.

Datakit End Settings¶

In Datakit, we need to make several settings to allow the collected data to be tagged with specific labels for grouping.

• Configure the global custom Key list

Datakit will search for fields with these Keys in the data it collects (only looking for string-type fields), extract them, and use them as the basis for grouping and sending.

• Configure "Global Host Tags" and "Global Election Tags"

All data uploaded by Datakit will carry these configured global tags (including tag key and tag value) as the basis for grouping and sending.

Datakit End Customer Key Settings¶

If you want the data collected by a specific Datakit to meet the requirements for traffic splitting, ensure the following:

• Datakit has the Sinker feature enabled
• Datakit is configured with valid Global Customer Keys

These two configurations are as follows:

# /usr/local/datakit/conf.d/datakit.conf
[dataway]

  # Specify a set of customer keys
  global_customer_keys = [
    # Example: Add two keys, category and class
    # It is not advisable to configure too many keys here; generally 2 ~ 3 are sufficient.
    "category",
    "class",
  ]

  # Enable the sinker feature
  enable_sinker = true

In addition to synthetic testing data and regular data categories, it also supports Session Replay and Profiling binary file data. Therefore, all field names can be selected here. It is important to note: do not configure non-string type fields. Normal Keys generally come from Tags (all Tag values are string type). Datakit will not use non-string type fields as the basis for traffic splitting.

Impact of Global Tags¶

In addition to global_customer_keys affecting traffic splitting markers, the global Tags configured on Datakit (including global election tags and global host tags) will also affect traffic splitting markers. That is, if a data point contains fields that appear in the global Tags (the values of these fields must be string type), they will also be counted for traffic splitting. Assume the global election tags are as follows:

# datakit.conf
[election.tags]
    cluster = "my-cluster"

For the following data point:

pi,cluster=cluster_A,app=math,other_tag=other_value value=3.14 1712796013000000000

Since the global election tags contain cluster (regardless of the value configured for this Tag), and the point itself also has the cluster Tag, in the final X-Global-Tags, the key-value pair cluster=cluster_A will be appended:

X-Global-Tags: cluster=cluster_A

If global_customer_keys also configures the app key, then the final traffic splitting Header is (the order of the two key-value pairs is not important):

X-Global-Tags: cluster=cluster_A,app=math

Dataway sink Command¶

Starting from Dataway version Version-1.3.6, command-line management of sinker configuration is supported. The specific usage is as follows:

$ ./dataway sink --help

Usage of sink:
  -add string
        single rule json file
  -cfg-file string
        configure file (default "/usr/local/cloudcare/dataflux/dataway/dataway.yaml")
  -file string
        file path of the rule json, only used for command put and get
  -get
        get the rule json
  -list
        list rules
  -log string
        log file path (default "/dev/null")
  -put
        save the rule json
  -token string
        rules filtered by token, eg: xx,yy

Specify Configuration File

When executing the command, the default loaded configuration file is /usr/local/cloudcare/dataflux/dataway/dataway.yaml. If you need to load other configurations, you can specify it via --cfg-file.

$ ./dataway sink --cfg-file dataway.yaml [--list...]

Command Log Settings

By default, command output logs are disabled. If you need to view them, you can set the --log parameter.

# output log to stdout
$ ./dataway sink --list --log stdout

# output log to file
$ ./dataway sink --list --log /tmp/log

View Rule List

# list all rules
$ ./dataway sink --list

# list all rules filtered by token
$ ./dataway sink --list --token=token1,token2

CreateRevision: 2
ModRevision: 41
Version: 40
Rules:
[
    {
        "rules": [
            "{ workspace = 'zhengb-test'}"
        ],
        "url": "https://openway.guance.com?token=token1"
    }
]

Add Rules

Create a rule file rule.json, with content similar to the following:

[
  {
    "rules": [
      "{ host = 'HOST1'}"
    ],
    "url": "https://openway.guance.com?token=tkn_xxxxxxxxxxxxx"
  },
  {
    "rules": [
      "{ host = 'HOST2'}"
    ],
    "url": "https://openway.guance.com?token=tkn_yyyyyyyyyyyyy"
  }
]

Add rules

$ ./dataway sink --add rule.json

add 2 rules ok!

Export Configuration

Exporting configuration can save the sinker configuration to a local file.

$ ./dataway sink --get --file sink-get.json

rules json was saved to sink-get.json!

Write Configuration

Writing rules synchronizes the local rule file to sinker.

Create a rule file sink-put.json, with content similar to the following:

{
    "rules": [
        {
            "rules": [
                "{ workspace = 'test'}"
            ],
            "url": "https://openway.guance.com?token=tkn_xxxxxxxxxxxxxx"
        }
    ],
    "strict": true
}

Write configuration

$ ./dataway sink --put --file sink-put.json

Configuration Examples¶

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: deployment-utils-dataway
  name: dataway
  namespace: utils
spec:
  replicas: 1
  selector:
    matchLabels:
      app: deployment-utils-dataway
  template:
    metadata:
      labels:
        app: deployment-utils-dataway
      annotations:
        datakit/logs: |
          [{"disable": true}]
        datakit/prom.instances: |
          [[inputs.prom]]
            url = "http://$IP:9090/metrics" # This port (default 9090) depends on the situation
            source = "dataway"
            measurement_name = "dw" # Fixed as this measurement
            interval = "10s"

            [inputs.prom.tags]
              namespace = "$NAMESPACE"
              pod_name = "$PODNAME"
              node_name = "$NODENAME"
    spec:
      affinity:
        podAffinity: {}
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - deployment-utils-dataway
              topologyKey: kubernetes.io/hostname

      containers:
      - image: registry.jiagouyun.com/dataway/dataway:1.3.6 # Choose the appropriate version number here
        #imagePullPolicy: IfNotPresent
        imagePullPolicy: Always
        name: dataway
        env:
        - name: DW_REMOTE_HOST
          value: "http://kodo.forethought-kodo:9527" # Fill in the real Kodo address here, or the next Dataway address
        - name: DW_BIND
          value: "0.0.0.0:9528"
        - name: DW_UUID
          value: "agnt_xxxxx" # Fill in the real Dataway UUID here
        - name: DW_TOKEN
          value: "tkn_oooooooooooooooooooooooooooooooo" # Fill in the real Dataway token here, usually the token of the system workspace
        - name: DW_PROM_LISTEN
          value: "0.0.0.0:9090"
        - name: DW_SECRET_TOKEN
          value: "tkn_zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"
        - name: DW_SINKER_FILE_PATH
          value: "/usr/local/cloudcare/dataflux/dataway/sinker.json"
        ports:
        - containerPort: 9528
          name: 9528tcp01
          protocol: TCP
        volumeMounts:
          - mountPath: /usr/local/cloudcare/dataflux/dataway/cache
            name: dataway-cache
          - mountPath: /usr/local/cloudcare/dataflux/dataway/sinker.json
            name: sinker
            subPath: sinker.json
        resources:
          limits:
            cpu: '4'
            memory: 4Gi
          requests:
            cpu: 100m
            memory: 512Mi
      # nodeSelector:
      #   key: string
      imagePullSecrets:
      - name: registry-key
      restartPolicy: Always
      volumes:
      - hostPath:
          path: /root/dataway_cache
        name: dataway-cache
      - configMap:
          name: sinker
        name: sinker
---

apiVersion: v1
kind: Service
metadata:
  name: dataway
  namespace: utils
spec:
  ports:
  - name: 9528tcp02
    port: 9528
    protocol: TCP
    targetPort: 9528
    nodePort: 30928
  selector:
    app: deployment-utils-dataway
  type: NodePort

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: sinker
  namespace: utils
data:
  sinker.json: |
    {
        "strict":true,
        "rules": [
            {
                "rules": [
                    "{ project = 'xxxxx'}"
                ],
                "url": "http://kodo.forethought-kodo:9527?token=tkn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
            },
            {
                "rules": [
                    "{ project = 'xxxxx'}"
                ],
                "url": "http://kodo.forethought-kodo:9527?token=tkn_yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
            }
         ]
    }

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dataway-sinker
  namespace: utils
spec:
  ingressClassName: nginx
  rules:
  - host: datawaysinker-xxxx.com
    http:
      paths:
      - backend:
          service:
            name: dataway
            port:
              number: 9528
        path: /
        pathType: ImplementationSpecific

FAQ¶

Viewing Details of Dropped Requests¶

Version-1.3.7

When a request does not meet the Sinker rules, Dataway will drop the request and increment a drop count in the metrics. However, during debugging, we need to know the specific details of a dropped request, especially the X-Global-Tags information carried in the request Header.

We can search Dataway logs with the following command:

$ cat <path/to/dataway/log> | grep dropped

In the output, we can see output similar to the following:

for API /v1/write/logging with X-Global-Tags <some-X-Global-Tags...> dropped

Datakit Request Drop Troubleshooting¶

Version-1.3.9

When a Datakit request is dropped by Dataway, Dataway will return a corresponding HTTP error. In the Datakit logs, there will be an error similar to the following:

post 3641 to http://dataway-ip:9528/v1/write/metric failed(HTTP: 406 Not Acceptable):
{"error_code":"dataway.sinkRulesNotMatched","message":"X-Global-Tags: `host=my-host',
URL: `/v1/write/metric'"}, data dropped

This error indicates that the request /v1/write/metric was dropped because its own X-Global-Tags did not satisfy all the rules on Dataway.

At the same time, in the Datakit monitor (datakit monitor -V) in the bottom right DataWay APIs panel, the Status column will show Not Acceptable output, indicating that the corresponding Dataway API request was dropped.

Checking Datakit's own metrics, you can also see the corresponding metrics:

$ curl -s http://localhost:9529/metrics | grep datakit_io_dataway_api_latency_seconds_count

datakit_io_dataway_api_latency_seconds_count{api="/v1/datakit/pull",status="Not Acceptable"} 50
datakit_io_dataway_api_latency_seconds_count{api="/v1/write/metric",status="Not Acceptable"} 301

Datakit Reports Error 403¶

If the sinker configuration on Dataway is incorrect, causing all Datakit requests to use secret_token, and this token is not recognized by the center (Kodo), a 403 error kodo.tokenNotFound is reported.

The cause of this problem may be incorrect etcd username/password, causing Dataway to fail to obtain the Sinker configuration. Consequently, Dataway considers the current sinker invalid, and all data is sent directly to the center.

etcd Permission Configuration Issues¶

If there is the following error in the Dataway logs, it may indicate a permission issue:

sinker ping: etcdserver: permission denied, retrying(97th)

If permissions are not configured properly, you can delete all existing Dataway-based permissions and reconfigure them. Refer specifically to here

Datakit End Key Override Relationship¶

When configuring the "Global Custom Key List", if the "Global Host Tags" and "Global Election Tags" also contain keys with the same name, then the Key-Value pairs from the collected data are used.

For example, if the configured "Global Custom Key List" includes key1,key2,key3, and the "Global Host Tags" or "Global Election Tags" also configure these keys with specified values, e.g., key1=value-1. In a certain data collection, there is also a field key1=value-from-data. Then, the final grouping basis uses key1=value-from-data from the data, ignoring the Value configured for the corresponding Key in "Global Host Tags" and "Global Election Tags".

If there are keys with the same name between "Global Host Tags" and "Global Election Tags", the Key in "Global Election Tags" takes priority. In summary, the priority of Key value sources is as follows (decreasing):

• Collected data
• Global election tags
• Global host tags

Built-in "Global Custom Keys"¶

Datakit has several built-in available custom Keys. They generally do not appear in the collected data, but Datakit can use these Keys to group data. If there is a need for traffic splitting based on these Key dimensions, they can be added to the "Global Custom Key" list (these Keys are not configured by default). We can use some of these built-in custom Keys to implement data splitting.

• class For object data, when enabled, it splits traffic based on the object's classification. For example, the object classification for Pod is kubelet_pod. Then you can formulate splitting rules specifically for pods:

{
    "strict": true,
    "rules": [
        {
            "rules": [
                "{ class = 'kubelet_pod' AND other_conditon = 'some-value' }",
            ],
            "url": "https://kodo.guance.com?token=<YOUR-TOKEN>"
        },
        {
            ... # other rules
        }
    ]
}

• measurement For metric data, we can send specific measurements to specific workspaces. For example, the measurement name for disk is disk. We can write rules like this:

{
    "strict": true,
    "rules": [
        {
           "rules": [
               "{ measurement = 'disk' AND other_conditon = 'some-value' }",
           ],
           "url": "https://kodo.guance.com?token=<YOUR-TOKEN>"
        },
        {
            ... # other rules
        }
    ]
}

• source For logs (L), eBPF network metrics (N), events (E), and RUM data
• service For Tracing, Scheck, and Profiling
• category For all regular data categories, its value is the "Name" column of the corresponding data category (e.g., time series is metric, object is object, etc.). Taking logs as an example, we can formulate splitting rules specifically for logs:

{
    "strict": true,
    "rules": [
        {
            "rules": [
                "{ category = 'logging' AND other_conditon = 'some-value' }",
            ],
            "url": "https://kodo.guance.com?token=<YOUR-TOKEN>"
        },
        {
            ... # other rules
        }
    ]
}

Special Traffic Splitting Behavior¶

Starting from Version-2.0.0, Datakit will only add the Dataway Sinker header to point write class requests (/v1/write/*). Other Dataway APIs that pull resources from the center, perform identity recognition, or configuration synchronization will no longer carry X-Global-Tags/X-Global-Tags-V2. Therefore, there is no need to add special traffic splitting rules for these APIs.

For Datakit versions before 2.0.0, some requests initiated by Datakit are for pulling resources from the center or performing self-identity recognition. Their behavior is already atomic, cannot be further split, and cannot be distributed to multiple workspaces (because Datakit needs to process the returns of these API requests and decide its subsequent actions). Therefore, these APIs can only be split to one workspace at most.

If multiple conditions are met in the splitting rules, these APIs will only be split to the workspace pointed to by the first rule that satisfies the condition.

Below is a compatible rule example for Datakit versions before 2.0.0:

This rule only needs to be added when older versions of Datakit need to fix these APIs to a specific workspace. Datakit 2.0.0 and later do not need to add it.

{
    "strict": true,
    "info": "Special workspace (only for data pulling APIs, etc.)",
    "rules": [
        {
            "rules": [
                "{ __dataway_api in ['/v1/datakit/pull', '/v1/election', '/v1/election/heartbeat', '/v1/query/raw', '/v1/workspace', '/v1/object/labels', '/v1/check/token'] }"
            ],
            "url": "https://kodo.guance.com?token=<SOME-SPECIAL-WORKSPACE-TOKEN>"
        }
    ]
}

Before Datakit 2.0.0, the __dataway_api key does not need to be configured in global_customer_keys in datakit.conf. Dataway will default to using this as the splitting Key, with the current request's API route as its value. That is, for a certain API:

POST /v1/some/api
X-Global-Tags: cluster=cluster_A,app=math

The final effect of splitting participation is the same as below:

POST /v1/some/api
X-Global-Tags: cluster=cluster_A,app=math,__dataway_api=/v1/write/metrics

Therefore, in compatibility scenarios for older versions, you can directly use the __dataway_api KV pair for matching in Sink rules. This also reminds us that in this special rule, do not include other important data upload APIs, such as /v1/write/... interfaces. Otherwise, which workspace the data ultimately lands in is undefined. Starting from Datakit 2.0.0, these non-point write class APIs no longer carry the Sinker header, eliminating the need to rely on __dataway_api rules for splitting.