OpenTelemetry Go SDK¶

The OpenTelemetry Go SDK collects telemetry from Go applications through APIs, SDKs, and framework instrumentation libraries. Unlike runtime auto-instrumentation solutions such as the Java Agent, Go applications usually need to initialize the SDK in code and wrap web frameworks, HTTP clients, databases, or message queues with the corresponding instrumentation libraries.

This article uses DataKit's OpenTelemetry collector to receive OTLP data and forward it to Guance:

Go application + OpenTelemetry Go SDK -> OTLP -> DataKit -> Guance

The example in this document reports Trace and Metric data through OTLP/HTTP + Protobuf. Trace and Metric in OpenTelemetry Go are stable; the maturity and ecosystem support of the Log SDK may still change, so verify the current official status before using it in production.

Prerequisites¶

• Go 1.23 or higher;
• DataKit is installed and DataKit is connected to the target Guance workspace;
• The network reachability of Go applications to DataKit: OTLP/HTTP uses DataKit HTTP port 9529, OTLP/gRPC uses 4317 by default;
• Confirm that the framework or component used by the application has a matching OpenTelemetry Go instrumentation library, or plan to create spans and metrics manually through the OpenTelemetry API.

1. Enable the OpenTelemetry collector¶

Enter conf.d/opentelemetry in the DataKit installation directory. If the collector configuration has not been created yet, copy the sample file:

cd /usr/local/datakit/conf.d/opentelemetry
sudo cp opentelemetry.conf.sample opentelemetry.conf

Verify that opentelemetry.conf contains at least the following receive configuration:

[[inputs.opentelemetry]]
  # Add custom attributes to this whitelist if you want to keep them as tags in Guance.
  # Dots in attribute names will be converted to underscores, for example team.name -> team_name.
  customer_tags = ["team", "project"]

  [inputs.opentelemetry.http]
    http_status_ok = 200
    trace_api = "/otel/v1/traces"
    metric_api = "/otel/v1/metrics"
    logs_api = "/otel/v1/logs"

  [inputs.opentelemetry.grpc]
    addr = "127.0.0.1:4317"
    max_payload = 16777216

The above configuration enables the following receiving addresses:

If the application and DataKit are not on the same host, the DataKit HTTP listening address, firewall, or other network access controls need to be adjusted according to the actual deployment. When using OTLP/gRPC, you also need to change addr to a listening address accessible to the application, such as 0.0.0.0:4317. Do not expose the OTLP receiving port directly to the public network.

Restart DataKit for the configuration to take effect:

sudo datakit service restart

Check if the DataKit HTTP service is reachable:

curl http://127.0.0.1:9529/v1/ping

2. Application access to OpenTelemetry¶

Install the Go SDK and instrumentation libraries¶

Install the official SDK, OTLP/HTTP Exporter and net/http instrumentation library in the Go project directory:

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
go get go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
go mod tidy

go.mod and go.sum record dependency versions. In production, commit both files and run compilation, unit tests, and trace regression tests after upgrading OpenTelemetry dependencies.

Initialize the SDK¶

The following example completes both:

1. Read resource attributes from OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES;
2. Create OTLP/HTTP Trace Exporter and Metric Exporter;
3. Register global TracerProvider, MeterProvider and W3C context propagator;
4. Use otelhttp to wrap HTTP Handler, and create a business sub-Span and custom Counter;
5. Refresh and close the Provider when the process exits to avoid data loss in the buffer.

package main

import (
    "context"
    "errors"
    "fmt"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/propagation"
    sdkmetric "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func setupOTelSDK(ctx context.Context) (func(context.Context) error, error) {
    res, err := resource.New(
        ctx,
        resource.WithFromEnv(),
        resource.WithTelemetrySDK(),
        resource.WithHost(),
        resource.WithOS(),
        resource.WithProcess(),
    )
    if err != nil {
        return nil, fmt.Errorf("create resource: %w", err)
    }

    traceExporter, err := otlptracehttp.New(ctx)
    if err != nil {
        return nil, fmt.Errorf("create trace exporter: %w", err)
    }

    tracerProvider := sdktrace.NewTracerProvider(
        sdktrace.WithResource(res),
        sdktrace.WithSampler(
            sdktrace.ParentBased(sdktrace.TraceIDRatioBased(1.0)),
        ),
        sdktrace.WithBatcher(traceExporter),
    )

    metricExporter, err := otlpmetrichttp.New(ctx)
    if err != nil {
        _ = tracerProvider.Shutdown(ctx)
        return nil, fmt.Errorf("create metric exporter: %w", err)
    }

    meterProvider := sdkmetric.NewMeterProvider(
        sdkmetric.WithResource(res),
        sdkmetric.WithReader(
            sdkmetric.NewPeriodicReader(
                metricExporter,
                sdkmetric.WithInterval(30*time.Second),
            ),
        ),
    )

    otel.SetTracerProvider(tracerProvider)
    otel.SetMeterProvider(meterProvider)
    otel.SetTextMapPropagator(
        propagation.NewCompositeTextMapPropagator(
            propagation.TraceContext{},
            propagation.Baggage{},
        ),
    )

    shutdown := func(ctx context.Context) error {
        return errors.Join(
            meterProvider.Shutdown(ctx),
            tracerProvider.Shutdown(ctx),
        )
    }
    return shutdown, nil
}

func main() {
    ctx, stop := signal.NotifyContext(
        context.Background(),
        os.Interrupt,
        syscall.SIGTERM,
    )
    defer stop()

    shutdown, err := setupOTelSDK(ctx)
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        shutdownCtx, cancel := context.WithTimeout(
            context.Background(),
            5*time.Second,
        )
        defer cancel()
        if err := shutdown(shutdownCtx); err != nil {
            log.Printf("shutdown OpenTelemetry: %v", err)
        }
    }()

    meter := otel.Meter("example/order-service")
    requestCounter, err := meter.Int64Counter("app.request.count")
    if err != nil {
        log.Fatal(err)
    }

    mux := http.NewServeMux()
    mux.Handle("/hello", otelhttp.NewHandler(
        http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            requestCtx, span := otel.Tracer("example/order-service").Start(
                r.Context(),
                "prepare-response",
            )
            defer span.End()

            span.SetAttributes(attribute.String("app.route", "/hello"))
            requestCounter.Add(requestCtx, 1)
            _, _ = fmt.Fprintln(w, "hello from OpenTelemetry Go SDK")
        }),
        "GET /hello",
    ))

    server := &http.Server{
        Addr:              ":8080",
        Handler:           mux,
        ReadHeaderTimeout: 5 * time.Second,
    }

    go func() {
        <-ctx.Done()
        shutdownCtx, cancel := context.WithTimeout(
            context.Background(),
            5*time.Second,
        )
        defer cancel()
        if err := server.Shutdown(shutdownCtx); err != nil {
            log.Printf("shutdown HTTP server: %v", err)
        }
    }()

    log.Println("listening on http://127.0.0.1:8080")
    if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
        log.Fatal(err)
    }
}

In a real project, install the instrumentation library required by each component you use. For example, the standard library net/http uses otelhttp; other web frameworks, databases, or message queues should use the matching package from the OpenTelemetry Registry and wrap the corresponding Handler, Transport, Client, or Driver as instructed.

Configure the reported address and start¶

Next, use OTLP/HTTP + Protobuf to report to the local DataKit. OTEL_EXPORTER_OTLP_ENDPOINT is the base endpoint. The Trace and Metric exporters append /v1/traces and /v1/metrics respectively, which map to DataKit's /otel/v1/* routes.

export OTEL_SERVICE_NAME="order-service"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=prod,service.version=1.0.0,team=backend"

export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:9529/otel"
export OTEL_EXPORTER_OTLP_INSECURE="true"
export OTEL_EXPORTER_OTLP_COMPRESSION="gzip"

go run .

Make a request to generate traces and metrics:

curl http://127.0.0.1:8080/hello

Metrics are exported every 30 seconds by default in this example through sdkmetric.WithInterval(30*time.Second). After waiting for one export cycle, you can view traces for service=order-service in Guance and query the app.request.count metric.

Using OTLP/gRPC¶

The Go SDK's OTLP transport is determined by the Exporter package used in the code. This example uses otlptracehttp and otlpmetrichttp directly, just setting OTEL_EXPORTER_OTLP_PROTOCOL=grpc will not switch it to gRPC.

To use OTLP/gRPC, install gRPC Exporter:

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go get go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc
go mod tidy

Then replace the exporter package and initialization function in the code with:

traceExporter, err := otlptracegrpc.New(ctx)
metricExporter, err := otlpmetricgrpc.New(ctx)

And use gRPC address without /v1/traces, /v1/metrics paths:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4317"
export OTEL_EXPORTER_OTLP_INSECURE="true"

3. Data reporting parameters¶

Resource parameters¶

This example reads the following standard environment variables via resource.WithFromEnv():

Set at least service.name, deployment.environment.name, and service.version so Guance can perform service attribution, environment filtering, and version analysis. Custom resource attributes must be added to the DataKit customer_tags whitelist before they can be retained as tags. . in attribute names is converted to _.

OTLP/HTTP parameters¶

otlptracehttp.New() and otlpmetrichttp.New() will read the following environment variables directly. Signal-specific parameters take precedence over general parameters:

If your app and DataKit are not on the same host, you should replace 127.0.0.1 in the example with a DataKit address that your app can access.

SDK code parameters¶

The following parameters are controlled by the Go SDK initialization code and will not automatically take effect by setting a common environment variable with the same name:

This example creates the OTLP exporter directly, so pay special attention to the following:

• OTEL_EXPORTER_OTLP_PROTOCOL does not change the HTTP or gRPC exporter already selected in code;
• OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER will not close the Exporter created directly in this example;
• The OpenTelemetry Go core SDK currently does not automatically apply all common SDK environment variables, especially do not assume that OTEL_SDK_DISABLED, OTEL_TRACES_SAMPLER or OTEL_PROPAGATORS will automatically take effect in custom initialization code;
• If you need to dynamically select and initialize the Exporter through standard environment variables, you can evaluate the official Contrib's autoexport package and verify its behavior in the test environment.

Verification and troubleshooting¶

1. Execute curl http://127.0.0.1:9529/v1/ping to confirm that the application can reach DataKit;
2. Start the application and access /hello, confirm that there is no create trace exporter, create metric exporter or OTLP export error in the application log;
3. Wait for at least one metric export cycle;
4. Query traces for order-service in the APM service list of Guance;
5. When the data cannot be queried, check the DataKit opentelemetry collector configuration, the network applied to DataKit, the reporting URL and the DataKit log;
6. When duplicate Span occurs, check whether the same Handler, Transport, Database Client or Driver is packaged repeatedly.

References¶

• OpenTelemetry collector configuration
• OpenTelemetry Go official documentation
• OpenTelemetry Go Getting Started
• OpenTelemetry Go Exporters
• OpenTelemetry Go instrumentation library
• OpenTelemetry OTLP Exporter configuration