OpenTelemetry Sampling Best Practices¶

Full trace data can certainly help relevant personnel promptly and accurately identify business issues. However, the probability of business problems occurring is generally low. Full trace sampling has its own advantages and disadvantages.

Advantages

Complete trace data

Disadvantages

Resource waste. Complete data significantly increases storage resource costs, and the cost of retrieving abnormal trace data also rises.

Based on full trace data collection, OpenTelemetry supports two types of samplers:

Probabilistic Sampler Processor (probabilisticsamplerprocessor)

Tail Sampler Processor (tailsamplingprocessor)

Probabilistic Sampler Processor ¶

As the name implies, the probabilistic sampler samples according to a certain probability. OpenTelemetry also supports two types of probabilistic sampling:

sampling.priority – semantic convention defined by OpenTracing

TraceId hash

The sampling.priority semantic convention takes precedence over TraceId hash. As the name suggests, TraceId hash sampling is based on the hash value determined by the TraceId. For TraceId hash to work, all collectors at a given tier (e.g., behind the same load balancer) must have the same hash_seed. Different hash_seed values can also be used for different collector tiers to support additional sampling requirements. For configuration specifications, see config.go.

The following configuration options can be modified:

hash_seed (no default): An integer used to compute the hash algorithm. Note that all collectors at a given tier (e.g., behind the same load balancer) should have the same hash_seed.

This is important when using multiple collector tiers to achieve the desired sampling rate, for example: 10% at the first tier, 10% at the second tier, resulting in an overall sampling rate of 1% (10% × 10%).

If all tiers use the same seed, all data passing through one tier will also pass through the next tier, regardless of the configured sampling rate. Using different seeds across tiers ensures that the sampling rate at each tier works as expected.

sampling_percentage (default = 0): The percentage of traces to sample; >= 100 means sampling all traces.

Configuring the Probabilistic Sampler Processor

processors:
  # Probabilistic sampler
  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 15.3

Enabling the Probabilistic Sampler Processor

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch,probabilistic_sampler]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Tail Sampler Processor ¶

The tail sampling processor samples traces based on a set of defined policies. Currently, this processor only works with a single collector instance. Technically, TraceId-aware load balancing can be used to support multiple collector instances, but this configuration has not been tested. For configuration specifications, see config.go.

The following configuration option is required:

policies (no default): The policies used to make sampling decisions

Multiple policies are currently supported. These include:

always_sample: Samples all traces

latency: Samples based on trace duration. Duration is determined by looking at the earliest start time and the latest end time, regardless of what happens in between.

numeric_attribute: Samples based on numeric attributes

probabilistic: Samples a certain percentage of traces

status_code: Samples based on status codes (OK, ERROR, or UNSET). Many people mistake this for the code in the response body JSON (which is common in many projects), but that is incorrect.

string_attribute: Samples based on string attribute value matching, supporting exact match and regular expression value match

rate_limiting: Samples based on rate limiting

and: Samples based on multiple policies, creating an AND policy

composite: Samples based on a combination of the above samplers, with each sampler having ordering and rate allocation. Rate allocation assigns a certain percentage of spans to each policy order. For example, if we set max_total_spans_per_second to 100, we can configure rate_allocation as follows:

test-composite-policy-1 = 50% of max_total_spans_per_second = 50 spans_per_second

test-composite-policy-2 = 25% of max_total_spans_per_second = 25 spans_per_second

To ensure the remaining capacity is filled, use always_sample as one of the policies

The following configuration options can also be modified:

decision_wait (default = 30 seconds): The wait time from the first span of a trace before making a sampling decision

num_traces (default = 50000): The number of traces kept in memory

expected_new_traces_per_sec (default = 0): The expected number of new traces (helps allocate data structures)

Example:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 10
    policies:      [
          {
            name: test-policy-1,
            type: always_sample
          },
          {
            name: test-policy-2,
            type: latency,
            latency: {threshold_ms: 5000}
          },
          {
            name: test-policy-3,
            type: numeric_attribute,
            numeric_attribute: {key: key1, min_value: 50, max_value: 100}
          },
          {
            name: test-policy-4,
            type: probabilistic,
            probabilistic: {sampling_percentage: 10}
          },
          {
            name: test-policy-5,
            type: status_code,
            status_code: {status_codes: [ERROR, UNSET]}
          },
          {
            name: test-policy-6,
            type: string_attribute,
            string_attribute: {key: key2, values: [value1, value2]}
          },
          {
            name: test-policy-7,
            type: string_attribute,
            string_attribute: {key: key2, values: [value1, val*], enabled_regex_matching: true, cache_max_size: 10}
          },
          {
            name: test-policy-8,
            type: rate_limiting,
            rate_limiting: {spans_per_second: 35}
         },
         {
            name: test-policy-9,
            type: string_attribute,
            string_attribute: {key: http.url, values: [\/health, \/metrics], enabled_regex_matching: true, invert_match: true}
         },
         {
            name: and-policy-1,
            type: and,
            and: {
              and_sub_policy:               [
                {
                  name: test-and-policy-1,
                  type: numeric_attribute,
                  numeric_attribute: { key: key1, min_value: 50, max_value: 100 }
                },
                {
                    name: test-and-policy-2,
                    type: string_attribute,
                    string_attribute: { key: key2, values: [ value1, value2 ] }
                },
              ]
            }
         },
         {
            name: composite-policy-1,
            type: composite,
            composite:              {
                max_total_spans_per_second: 1000,
                policy_order: [test-composite-policy-1, test-composite-policy-2, test-composite-policy-3],
                composite_sub_policy:                  [
                    {
                      name: test-composite-policy-1,
                      type: numeric_attribute,
                      numeric_attribute: {key: key1, min_value: 50, max_value: 100}
                    },
                    {
                      name: test-composite-policy-2,
                      type: string_attribute,
                      string_attribute: {key: key2, values: [value1, value2]}
                    },
                    {
                      name: test-composite-policy-3,
                      type: always_sample
                    }
                  ],
                rate_allocation:                  [
                    {
                      policy: test-composite-policy-1,
                      percent: 50
                    },
                    {
                      policy: test-composite-policy-2,
                      percent: 25
                    }
                  ]
              }
          },
        ]

For a detailed example of using the processor, see tail_sampling_config.yaml.

Comparison Between Probabilistic Sampler Processor and Probabilistic Policy in Tail Sampler Processor¶

The probabilistic sampler processor and the probabilistic tail sampling processor policy work very similarly: based on a configurable sampling percentage, they will sample a fixed ratio of received traces. However, depending on the overall processing pipeline, you should prefer one over the other.

As a rule of thumb, if you want to add probabilistic sampling and...

...you are not already using the tail sampling processor: Use the probabilistic sampler processor. Running the probabilistic sampler processor is more efficient than the tail sampling processor. The probabilistic sampling policy makes decisions based on the TraceId, so waiting for more spans to arrive does not affect its decision.

...you are already using the tail sampling processor: Add the probabilistic sampling policy. You are already incurring the cost of running the tail sampling processor, so adding the probabilistic policy is negligible. Additionally, using this policy within the tail sampling processor ensures that traces sampled by other policies are not discarded.

Demo¶

This demo mainly pushes OpenTelemetry data to Guance

Prerequisites¶

Download the source code: https://github.com/lrwh/observable-demo/tree/main/opentelemetry-collector-sampling
Ensure DataKit is installed

Service Name	Port	Description
otel-collector		otel/opentelemetry-collector-contrib:0.69.0
springboot_server	8080:8080	opentelemetry-agent version 1.21.0, source code: https://github.com/lrwh/observable-demo/tree/main/springboot-server

Enable OpenTelemetry collection in DataKit.

Start the Services¶

docker-compose up -d

The following examples mainly use the tail sampling policy as the test scenario.

Configuring the Tail Sampler Processor

processors:
  # Tail sampler
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 100
    policies:
      [
        {
          name: policy-1,
          type: status_code,
          status_code: {status_codes: [ERROR]}
        },
        {
          name: policy-2,
          type: probabilistic,
          probabilistic: {sampling_percentage: 20}
        }
      ]

The above rules are in an OR relationship, meaning sampling occurs if either policy-1 or policy-2 is satisfied.

Enabling the Tail Sampler Processor

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch,tail_sampling]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Using `probabilistic`¶

  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 100
    policies:
      [
        {
          name: policy-2,
          type: probabilistic,
          probabilistic: {sampling_percentage: 20}
        }
      ]

Access the service API gateway 5 times, each time with a normal response.

curl http://localhost:8080/gateway

Go to Guance to view trace information. According to the sampling rule, at most one trace of data will be sampled.

Using `status_code`¶

processors:
  # Tail sampler
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 100
    policies:
      [
        {
          name: policy-1,
          type: status_code,
          status_code: {status_codes: [ERROR]}
        }
      ]

Set the client to start.

curl http://localhost:8080/setClient?c=true

The current demo does not start the client service, so subsequent calls to the gateway endpoint will generate errors.

Access the gateway service 5 times, returning {"msg":"client 调用失败","code":500}
Go to Guance to view trace information. According to the sampling rule, if an error occurs, all error traces are sampled.

Using `span_count`¶

If the number of spans is less than or equal to 2, do not report. Configuration as follows:

  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 100
    policies:
      [
        {
         name: p2,
         type: span_count,
         span_count: {min_spans: 3 }
         }
      ]

Using `string_attribute`¶

Scenario: Only collect traces for GET requests

  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 100
    policies:
      [
        {
         name: policy-string,
         type: string_attribute,
         string_attribute: {key: http.method, values: [ GET ] }
         }
      ]

string_attribute does not work?

Yes, you read that correctly. string_attribute is not guaranteed to work 100% of the time. Take http.method as an example. Suppose we match a GET request, but that GET request also calls a POST request within the same trace. For the complete trace, there is both GET and POST, which creates a conflict. Because the GET matches the criteria, the trace is sampled, so it will not be filtered out just because the trace also contains a POST.

Using `and`¶

and: Sampling and reporting only when all AND conditions are met; otherwise, discard.

Scenario: Report GET requests, and sample according to a 20% sampling rate; discard otherwise.

  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 100
    policies:
      [
        {
         name: and-policy-1,
         type: and,
         and: {
           and_sub_policy:
             [
                 {
                     name: test-and-policy-2,
                     type: string_attribute,
                     string_attribute: { key: http.method, values: [ GET ] }
                 },
                 {
                     name: policy-2,
                     type: probabilistic,
                     probabilistic: {sampling_percentage: 20}
                 }
            ]
         }
      }
    ]

Notes on Sampling Formatting¶

Incorrect sampling formatting can also cause sampling to fail. It is recommended to add spaces between each word, symbol, and numeric value.

Incorrect format

{
 name: p2,
 type: span_count,
 span_count:{min_spans: 3 }
 }

or

{
 name: p2,
 type: span_count,
 span_count: {min_spans:3 }
}

Correct format

{
 name: p2,
 type: span_count,
 span_count: {min_spans: 3 }
}

OpenTelemetry Sampling Best Practices¶

Probabilistic Sampler Processor¶

Tail Sampler Processor¶

Comparison Between Probabilistic Sampler Processor and Probabilistic Policy in Tail Sampler Processor¶

Demo¶

Prerequisites¶

Start the Services¶

Using probabilistic¶

Using status_code¶

Using span_count¶

Using string_attribute¶

Using and¶