Customize Dialtesting

In some cases, you may not be able to connect to SAAS's dialing task service. In this case, we can define the dialing task through the local JSON file.

Configuration¶

Configure Collector¶

Host InstallationKubernetes

Go to the conf.d/network directory under the DataKit installation directory, copy dialtesting.conf.sample and name it dialtesting.conf. Examples are as follows:

[[inputs.dialtesting]]
  server = "file://</path/to/your/local.json>"

  # Note: Taking Linux as an example, assuming your json directory is /some/path/my.json, then the
  # server should be written as file:///some/path/my.json

  # Note that the following tag suggestions are filled in one by one (do not modify the tag key here), so that the complete dialing test results can be displayed on the page.
  [inputs.dialtesting.tags] 
    country  = "<specify-datakit-country>"  # Countries where DataKit is deployed
    province = "<specify-datakit-province>" # Provices where DataKit is deployed
    city     = "<specify-datakit-city>"     # Cities where DataKit is deployed
    isp      = "<specify-datakit-ISP>"      # Specify the network service provider where DataKit is located
    region   = "<your-region>"              # You can specify a region name at will

The collector can now be turned on by ConfigMap injection collector configuration.

The specific country/region and ISP selection can be selected as shown in the following figure (note that you don't really create a new "self-built node", just provide an alternative source here):

Configure the Dial Test Task¶

At present, the dialing test task supports four dialing test types, namely HTTP, TCP, ICMP and WEBSOCKET services. The JSON format is as follows:

{
  "<Dial Test Type>": [
    {Dial test task 1},
    {Dial test task 2},
       ...
    {Dial test task n},
  ]
}

The following is a specific dialing test example:

{
  "HTTP": [
    {
      "name": "baidu-json-test",
      "method": "GET",
      "url": "http://baidu.com",
      "post_url": "https://<your-dataway-host>?token=<your-token>",
      "status": "OK",
      "schedule_type": "frequency",
      "frequency": "10s",
      "success_when_logic": "and",
      "success_when": [
        {
          "response_time": "1000ms",
          "header": {
            "Content-Type": [
              {
                "contains": "html"
              }
            ]
          },
          "status_code": [
            {
              "is": "200"
            }
          ]
        }
      ],
      "advance_options": {
        "request_options": {
          "auth": {}
        },
        "request_body": {}
      },
      "update_time": 1645065786362746
    }
  ],
  "TCP": [
    {
      "name": "tcp-test",
      "host": "www.baidu.com",
      "port": "80",
      "status": "OK",
      "frequency": "10s",
      "success_when_logic": "or",
      "success_when": [
        {
          "response_time": [
            {
              "is_contain_dns": true,
              "target": "10ms"
            }
          ]
        }
      ],
      "update_time": 1641440314550918
    }
  ]
}

After editing this JSON, it is recommended to find some（online tools or this tool）to verify that the JSON format is correct. If the JSON format is incorrect, the dialing test will not take effect.

After configuration, restart DataKit.

Test Task Field Definition¶

The dialing task fields include "public fields" and "additional fields" for specific dialing tasks.

Public Field¶

The public fields of dialing test tasks are defined as follows:

Field	Type	Whether Required	Description
`name`	string	Y	Dial test service name
`status`	string	Y	Dial test service status, such as "OK"/"stop"
`frequency`	string	Y	Dial frequency
`schedule_type`	string	Y	The execution type of the monitoring task, such as `frequency` or `crontab`, default is `frequency`
`crontab`	string	N	The `crontab` expression, such as `/10 * * *`, only valid when `schedule_type` is `crontab`
`success_when_logic`	string	N	The logical relationship between success_when conditions, such as "and"/"or", defaults to "and"
`success_when`	object	Y	See below for details
`advance_options`	object	N	See below for details
`post_url`	string	N	Send the dialing test result to the workspace pointed by the Token, and if it is not filled in, send it to the workspace where the current DataKit is located
`tags_info`	string	N	Custom tags, such as `t1,t2`
`workspace_language`	string	N	Workspace language, such as `zh`，`en`

HTTP Dial Test¶

Extra field:

Field	Type	Whether Required	Description
`method`	string	Y	HTTP request method
`url`	string	Y	Complete HTTP request address

The overall JSON structure is as follows:

{
  "HTTP": [
    {
      "name": "baidu-json-test",
      "method": "GET",
      "url": "http://baidu.com",
      "post_url": "https://<your-dataway-host>?token=<your-token>",
      "status": "OK",
      "frequency": "10s",
      "success_when_logic": "and",
      "success_when": ...,
      "advance_options": ...
    },
    {
      ... another HTTP dialtesting
    }
  ]
}

`success_when` Definition¶

The judging conditions used to define the success of dialing test mainly include the following aspects:

HTTP request returns body judgment（body）

Field	Type	Whether Required	Description
`is`	string	N	Whether the returned body is equal to the specified field
`is_not`	string	N	Whether the returned body is not equal to the specified field
`match_regex`	string	N	Whether the returned body contains a substring of the matching regular expression
`not_match_regex`	string	N	Whether the returned body does not contain a substring of the matching regular expression
`contains`	string	N	Whether the returned body contains the specified substring
`not_contains`	string	N	Whether the returned body does not contain the specified substring

eg.

"success_when": [
  {
    "body": [
      {
        "match_regex": "\\d\\d.*",
      }
    ]
  }
]

Here, body can configure multiple verification rules, and the relationship between them is determined by "success_when_logic". When it is configured as and, if any rule is verified, it will be considered that the current dialing test failed; When it is configured to or, if any rule is verified, it will be considered that the current dialing test is successful. The default is an and relationship. The following verification rules all follow this judgment principle.

Note that the regular is escaped correctly here, and the actual regular expression in the example is \d\d.*.

HTTP request returns header judgment (header)

Field	Type	Whether Required	Description
`is`	string	N	The header returned specifies whether the field is equal to the specified value
`is_not`	string	N	The header returned specifies whether the field is not equal to the specified value.
`match_regex`	string	N	The header returned specifies whether the field contains a substring of the matching regular expression.
`not_match_regex`	string	N	The header returned specifies whether the field does not contain a substring of the matching regular expression.
`contains`	string	N	The header returned specifies whether the field contains the specified substring.
`not_contains`	string	N	The header returned specifies whether the field does not contain the specified substring.

for example:

"success_when": [
  {
    "header": {
       "Content-Type": [
         {
           "contains": "html"
         }
       ]
    }
  }
]

Because there may be decisions for multiple types of headers, validation for multiple headers can also be configured here:

"success_when": [
  {
    "header": {
       "Content-Type": [
         {
           "contains": "some-header-value"
         }
       ],

       "Etag": [
         {
           "match_regex": "\\d\\d-.*"
         }
       ]
    }
  }
]

HTTP request returns status code (status_code)

Field	Type	Whether Required	Description
`is`	string	N	Whether the status code returned is equal to the specified field
`is_not`	string	N	Whether the status code returned is not equal to the specified field
`match_regex`	string	N	Whether the status code returned contains a substring of the matching regular expression
`not_match_regex`	string	N	Whether the status code returned does not contain a substring of the matching regular expression
`contains`	string	N	Whether the status code returned contains the specified substring
`not_contains`	string	N	Whether the status code returned does not contain the specified substring

for example:

"success_when": [
  {
    "status_code": [
      {
        "is": "200"
      }
    ]
  }
]

For a certain URL dial test, its HTTP return is usually only one, so only one validation rule is generally configured here (although multiple array configurations are supported).

HTTP request response time (response_time)

Only one time value can be filled in here. If the response time of the request is less than the specified value, the dialing test is judged to be successful, such as:

"success_when": [
  {
    "response_time": "100ms"
  }
]

Note that the time units specified here are ns (nanoseconds)/us (microseconds) /ms (milliseconds) /s (seconds) /m (minutes) /h (hours). For HTTP dial testing, ms units are generally used.

Several kinds of judgment basis listed above can be used in combination, and the relationship between them is determined by "success_when_logic". When it is configured as and, if any rule is verified, it is considered that the current dialing test fails; When it is configured to or, if any rule is verified, it will be considered that the current dialing test is successful; The default is an and relationship. Such as:

"success_when": [
  {
    "response_time": "1000ms",
    "header": { HTTP header 相关判定 },
    "status_code": [ HTTP 状态码相关判定 ],
    "body": [ HTTP body 相关判定 ]
  }
]

`advance_options` Definition¶

Advanced options are mainly used to adjust specific dialing behavior, mainly in the following aspects:

HTTP Request Option (request_options）

Field	Type	Whether Required	Description
`follow_redirect`	bool	N	Whether redirect jump is supported
`headers`	map[string]string	N	Specify a set of headers on an HTTP request
`cookies`	string	N	Specify the requested Cookie
`auth`	object	N	Specify the authentication method of the request

Among them, auth only supports ordinary username and password authentication, which is defined as follows:

Field	Type	Whether Required	Description
`username`	string	Y	User name
`password`	string	Y	User name and password

request_options example:

"advance_options": {
  "request_options": {
    "auth": {
        "username": "zhangsan",
        "password": "fawaikuangtu"
      },
    "headers": {
      "X-Prison-Breaker": "zhangsan",
      "X-Prison-Break-Password": "fawaikuangtu"
    },
    "follow_redirect": false
  },
}

HTTP Request Body（request_body）

Field	Type	Whether Required	Description
`body_type`	string	N	Body type, that is, the value of the request header `Content-Type`
`body`	string	N	Request Body
`form`	map[string]string	N	When the Content-Type is `multipart/form-data`, it is the form data of the request Body

request_body example:

"advance_options": {
  "request_body": {
    "body_type": "multipart/form-data",
    "body": "Fill in the request body, and pay attention to various complicated escapes here",
    "form": {
      "name": "file",
    }
  }
}

HTTP Request Timeout (request_timeout)

HTTP request timeout is mainly used to adjust the timeout of the HTTP request. The default timeout is 60 seconds, and the timeout can be adjusted here.

request_timeout example:

"advance_options": {
  "request_timeout": "60s",
}

HTTP Request a Certificate (certificate)

Field	Type	Whether Required	Description
`ignore_server_certificate_error`	bool	N	Whether to ignore certificate errors
`private_key`	string	N	key
`certificate`	string	N	Certificate
`ca`	string	N	Temporarily unused

certificate example:

"advance_options": {
  "certificate": {
    "ignore_server_certificate_error": false,
    "private_key": "<your-private-key>",
    "certificate": "<your-certificate-key>"
  },
}

private_key example:

Here is an example of certificate:

Under Linux, this pair of keys can be generated by the following command:

openssl req -newkey rsa:2048 -x509 -sha256 -days 3650 -nodes -out example.crt -keyout example.key

HTTP Request broker (proxy)

Field	Type	Whether Required	Description
`url`	string	N	The URL of the proxy, such as `http://1.2.3.4:4321`
`headers`	map[string]string	N	Specify a set of headers on an HTTP request

proxy example:

"advance_options": {
  "request_options": {
    "proxy": {
      "url": "http://1.2.3.4:4321",
      "headers": {
        "X-proxy-header": "my-proxy-foobar"
      }
    }
  },
}

`post_script` Definition¶

post_script is a Pipeline script used to evaluate the result of the test and extract variables from it.

inject variables¶

To facilitate the processing of HTTP responses by post_script and to enable the determination of test results and the extraction of variables, certain predefined variables can be utilized when composing the script. These are detailed as follows:

response: the response object of the HTTP request

Field	Type	Description
`status_code`	number	status code
`header`	json	response header `{"header1": ["value1", "value2"]}`
`body`	string	response body

result： the test result

Field	Type	Description
`is_failed`	bool	failed or not
`error_message`	string	error message

vars: the variable object to store the extracted variables

A JSON object where the key represents the variable name and the value represents the variable value. For example: vars["token"] = "123"

Example¶

body = load_json(response["body"])

if body["code"] == "200" {
  result["is_failed"] = false
  vars["token"] = body["token"]
} else {
  result["is_failed"] = true
  result["error_message"] = body["message"]
}

In the above example, the response content is initially parsed into a JSON object using load_json. Subsequently, it checks whether the response status code is 200. If it is, the token from the response content is extracted and assigned to the variable vars. If the status code is not 200, the is_failed attribute of result is set to true, and the error_message is assigned the message from the response content.

TCP Dial Test¶

Extra Field¶

Field	Type	Whether Required	Description
`host`	string	Y	TCP Host address
`port`	string	Y	TCP Port
`timeout`	string	N	TCP connection timeout
`message`	string	N	TCP message sent

The complete JSON structure is as follows:

{
  "TCP": [
    {
      "name": "tcp-test",
      "host": "www.baidu.com",
      "port": "80",
      "message": "hello",
      "timeout": "10ms",
      "enable_traceroute": true,
      "post_url": "https://<your-dataway-host>?token=<your-token>",
      "status": "OK",
      "frequency": "60s",
      "success_when_logic": "and",
      "success_when": [
        {
          "response_time":[ 
            {
              "is_contain_dns": true,
              "target": "10ms"
            }
          ],
          "response_message": [
              {
                  "is": "hello"
              }
          ],
          "hops": [
            {
              "op": "eq",
              "target": 20
            }
          ]
        }
      ]
    }
  ]
}

`success_when` Definition¶

TCP Response Time Determination (response_time)

response_time is an array object with the following parameters for each object:

Field	Type	Whether Required	Description
`target`	string	Y	Determining whether the response time is less than the value
`is_contain_dns`	bool	N	Indicates whether the response time includes DNS resolution time

"success_when": [
  {
    "response_time": [
      {
        "is_contain_dns": true,
        "target": "10ms"
      }
    ]
  }
]

Return a message decision (response_message）

response_message is an array object with the following parameters for each object:

Field	Type	Whether Required	Description
`is`	string	N	Whether the returned message is equal to the specified field
`is_not`	string	N	Whether the returned message is not equal to the specified field
`match_regex`	string	N	Whether the returned message contains a substring of the matching regular expression
`not_match_regex`	string	N	Whether the returned message does not contain a substring of the matching regular expression
`contains`	string	N	Whether the returned message contains the specified substring
`not_contains`	string	N	Whether the returned message does not contain the specified substring

for example:

"success_when": [
  {
    "response_message": [
      {
        "is": "reply",
      }
    ]
  }
]

Network hop count (hops)

hops is an array object with the following parameters for each object:

Field	Type	Whether Required	Description
`op`	string	Y	Compare relation, retrievable `eq(=),lt(<),leq(<=),gt(>),geq(>=)`
`target`	float	Y	Decision value

"success_when": [
  {
    "hops": [
      {
        "op": "eq",
        "target": 20
      }
    ]
  }
]

ICMP Dial Test¶

Extra Field¶

Field	Type	Whether Required	Description
`host`	string	Y	Host address
`packet_count`	int	N	Number of ICMP packets sent
`timeout`	string	N	Connection timeout

The complete JSON structure is as follows:

{
  "ICMP": [
    {
      "name": "icmp-test",
      "host": "www.baidu.com",
      "timeout": "10ms",
      "packet_count": 3,
      "enable_traceroute": true,
      "post_url": "https://<your-dataway-host>?token=<your-token>",
      "status": "OK",
      "frequency": "10s",
      "success_when_logic": "and",
      "success_when": [
        {
          "response_time": [
            {
              "func": "avg",
              "op": "leq",
              "target": "50ms"
            }
          ],
          "packet_loss_percent": [
            {
              "op": "leq",
              "target": 20
            }
          ],
          "hops": [
            {
              "op": "eq",
              "target": 20
            }
          ],
          "packets": [
            {
              "op": "geq",
              "target": 1
            }
          ]
        }
      ]
    }
  ]
}

`success_when` Definition¶

ICMP packet loss rate (packet_loss_percent)

Fill in the specific value as an array object, and each object parameter is as follows:

Field	Type	Whether Required	Description
`op`	string	Y	Compare relationship retrievable `eq(=),lt(<),leq(<=),gt(>),geq(>=)`
`target`	float	Y	Decision value

"success_when": [
  {
    "packet_loss_percent": [
      {
        "op": "leq",
        "target": 20
      }
    ]
  }
]

ICMP response time (response_time)

Fill in the specific time as an array object, and each object parameter is as follows:

Field	Type	Whether Required	Description
`func`	string	Y	Statistical type, take the value `avg,min,max,std`
`op`	string	Y	Comparison relation, take value `eq(=),lt(<),leq(<=),gt(>),geq(>=)`
`target`	string	Y	Decision value

"success_when": [
  {
     "response_time": [
        {
          "func": "avg",
          "op": "leq",
          "target": "50ms"
        }
      ],
  }
]

Network hop count (hops)

hops is an array object with the following parameters for each object:

Field	Type	Whether Required	Description
`op`	string	Y	Compare relationships, retrievable `eq(=),lt(<),leq(<=),gt(>),geq(>=)`
`target`	float	Y	Decision value

"success_when": [
  {
    "hops": [
      {
        "op": "eq",
        "target": 20
      }
    ]
  }
]

Number of packets grabbed (packets)

packets is an array object with the following parameters for each object:

Field	Type	Whether Required	Description
`op`	string	Y	Compare relationship retrievable `eq(=),lt(<),leq(<=),gt(>),geq(>=)`
`target`	float	Y	Decision value

"success_when": [
  {
    "packets": [
      {
        "op": "eq",
        "target": 20
      }
    ]
  }
]

WEBSOCKET Dial Test¶

Extra Field¶

Field	Type	Whether Required	Description
`url`	string	Y	Websocket connection address, such as ws://localhost:8080
`message`	string	Y	Websocket message sent after successful connection

The complete JSON structure is as follows:

{
  "WEBSOCKET": [
    {
      "name": "websocket-test",
      "url": "ws://localhost:8080",
      "message": "hello",
      "post_url": "https://<your-dataway-host>?token=<your-token>",
      "status": "OK",
      "frequency": "10s",
      "success_when_logic": "and",
      "success_when": [
        {
          "response_time": [
            {
              "is_contain_dns": true,
              "target": "10ms"
            }
          ],
          "response_message": [
            {
              "is": "hello1"
            }
          ],
          "header": {
            "status": [
              {
                "is": "ok"
              }
            ]
          }
        }
      ],
      "advance_options": {
        "request_options": {
          "timeout": "10s",
          "headers": {
            "x-token": "aaaaaaa",
            "x-header": "111111"
          }
        },
        "auth": {
          "username": "admin",
          "password": "123456"
        }
      }
    }
  ]
}

`success_when` Definition¶

Response time judgment (response_time)

response_time is an array object with the following parameters for each object:

Field	Type	Whether Required	Description
`target`	string	Y	Determining whether the response time is less than the value
`is_contain_dns`	bool	N	Indicates whether the response time includes DNS resolution time

"success_when": [
  {
    "response_time": [
      {
        "is_contain_dns": true,
        "target": "10ms"
      }
    ]
  }
]

Return a message decision (response_message）

response_message is an array object with the following parameters for each object:

Field	Type	Whether Required	Description
`is`	string	N	Whether the returned message is equal to the specified field
`is_not`	string	N	Whether the returned message is not equal to the specified field
`match_regex`	string	N	Whether the returned message contains a substring of the matching regular expression
`not_match_regex`	string	N	Whether the returned message does not contain a substring of the matching regular expression
`contains`	string	N	Whether the returned message contains the specified substring
`not_contains`	string	N	Whether the returned message does not contain the specified substring

for example:

"success_when": [
  {
    "response_message": [
      {
        "is": "reply",
      }
    ]
  }
]

Request to return header judgment（header）

header is a dictionary type object whose value for each object element is an array object with the following parameters:

Field	Type	Whether Required	Description
`is`	string	N	The header returned specifies whether the field is equal to the specified value
`is_not`	string	N	The header returned specifies whether the field is not equal to the specified value
`match_regex`	string	N	The header returned specifies whether the field contains a substring of the matching regular expression
`not_match_regex`	string	N	The header returned specifies whether the field does not contain a substring of the matching regular expression
`contains`	string	N	The header returned specifies whether the field contains the specified substring
`not_contains`	string	N	The header returned specifies whether the field does not contain the specified substring

for example:

"success_when": [
  {
    "header": {
       "Status": [
         {
           "is": "ok"
         }
       ]
    }
  }
]

`advance_options` Definition¶

Request option (request_options)

Field	Type	Whether Required	Description
`timeout`	string	N	Connection timeout
`headers`	map[string]string	N	Specify a set of headers on request

"advance_options": {
  "request_options": {
    "timeout": "30ms",
    "headers": {
      "X-Token": "xxxxxxxxxx"
    }
  },
}

Authentication information (auth)

Support for common user name and password authentication (Basic access authentication)。

Field	Type	Whether Required	Description
`username`	string	Y	user name
`password`	string	Y	user name and password

"advance_options": {
  "auth": {
    "username": "admin",
    "password": "123456"
  },
}

Template Function Usage Instructions¶

Version-1.80.0

When defining a monitoring task, you are allowed to embed template variables in strings.

Function List¶

The following template functions are currently supported:

timestamp function: Generate a timestamp

Returns the timestamp of the current UTC time, supporting multiple time units (s, ms, us, ns). It returns 0 by default (when the unit parameter is invalid).

Usage example:

{{ timestamp "ns" }} //1754379375400790109

date function: Format time string

Returns the string of the current local time according to the specified format, supporting two definition formats (rfc3339, iso8601).

Usage examples:

{{ date "rfc3339" }} // 2006-01-02T15:04:05Z07:00
{{ date "iso8601" }} // 2006-01-02T15:04:05Z

urlencode function: URL encoding

Performs URL encoding on the input string (in compliance with the application/x-www-form-urlencoded specification), which is used to handle special characters in URLs.

Usage example:

{{ urlencode "hello world" }} // hello+world

Instructions for Nested Use of Functions¶

Template functions support nested use, which means the return value of one function can be used as the parameter of another function.

Usage example:

{{ urlencode (date "iso8601") }} // 2006-01-02T15%3A04%3A05Z

Customize Dialtesting

Configuration¶

Configure Collector¶

Configure the Dial Test Task¶

Test Task Field Definition¶

Public Field¶

HTTP Dial Test¶

success_when Definition¶

advance_options Definition¶

post_script Definition¶

inject variables¶

Example¶

TCP Dial Test¶

Extra Field¶

success_when Definition¶

ICMP Dial Test¶

Extra Field¶

success_when Definition¶

WEBSOCKET Dial Test¶

Extra Field¶

success_when Definition¶

advance_options Definition¶

Template Function Usage Instructions¶

Function List¶

Instructions for Nested Use of Functions¶

Is this page helpful? ×

`success_when` Definition¶

`advance_options` Definition¶

`post_script` Definition¶

`success_when` Definition¶

`success_when` Definition¶

`success_when` Definition¶

`advance_options` Definition¶