콘텐츠로 이동

OpenTelemetry Operator 모범 사례

작성자: 류루이(刘锐)

OpenTelemetry Operator는 Kubernetes Operator의 한 구현입니다.

주로 다음 작업을 관리합니다:

  • OpenTelemetry Collector
  • Auto-instrumentation: OpenTelemetry 계측 라이브러리를 사용하여 워크로드를 자동으로 계측

Guance 수집기 DataKit은 OpenTelemetry 설계 개념을 도입했으며 OTLP 프로토콜과 호환됩니다. 따라서 OpenTelemetry Collector를 거치지 않고 데이터를 직접 DataKit으로 푸시할 수 있으며, OpenTelemetry Collector의 exporter를 OTLP로 설정하고 주소를 DataKit으로 지정할 수도 있습니다.

두 가지 방식을 사용하여 APM 데이터를 Guance에 통합하겠습니다.

  • APM 데이터가 OpenTelemetry Collector를 통해 Guance로 푸시됨;
  • APM 데이터가 Guance로 직접 푸시됨.

전제 조건

  • k8s 환경
  • Guance 계정

OpenTelemetry 관련 구성 요소 설치

OpenTelemetry Operator 설치

  • opentelemetry-operator.yaml 다운로드
wget https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
  • opentelemetry-operator.yaml 설치
[root@k8s-master ~]# kubectl apply -f opentelemetry-operator.yaml 
namespace/opentelemetry-operator-system created
customresourcedefinition.apiextensions.k8s.io/instrumentations.opentelemetry.io created
customresourcedefinition.apiextensions.k8s.io/opentelemetrycollectors.opentelemetry.io created
serviceaccount/opentelemetry-operator-controller-manager created
role.rbac.authorization.k8s.io/opentelemetry-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/opentelemetry-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/opentelemetry-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/opentelemetry-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/opentelemetry-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/opentelemetry-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/opentelemetry-operator-proxy-rolebinding created
service/opentelemetry-operator-controller-manager-metrics-service created
service/opentelemetry-operator-webhook-service created
deployment.apps/opentelemetry-operator-controller-manager created
certificate.cert-manager.io/opentelemetry-operator-serving-cert created
issuer.cert-manager.io/opentelemetry-operator-selfsigned-issuer created
mutatingwebhookconfiguration.admissionregistration.k8s.io/opentelemetry-operator-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/opentelemetry-operator-validating-webhook-configuration created
  • pod 확인
[root@k8s-master df-demo]# kubectl get pod -n opentelemetry-operator-system
NAME                                                         READY   STATUS    RESTARTS   AGE
opentelemetry-operator-controller-manager-7b4687df88-9s967   2/2     Running   0          26h

OpenTelemetry Collector 설치

  • opentelemetry-collector.yaml 작성
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: demo
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      logging:
      otlp:
        endpoint: "http://datakit-service.datakit:4319" # 트레이스 데이터를 Guance 플랫폼으로 출력
        tls:
          insecure: true
        #compression: none # gzip 사용 안 함

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging,otlp]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging]
  • opentelemetry-collector.yaml 실행
kubectl apply -f opentelemetry-collector.yaml
  • pod 확인
[root@k8s-master ~]# kubectl get pod 
NAME                                 READY   STATUS    RESTARTS   AGE
demo-collector-59b9447bf9-dz47k      1/1     Running   0          61m

Instrumentation 설치

OpenTelemetry Operator는 OpenTelemetry 자동 계측 라이브러리를 주입하고 구성할 수 있습니다. 현재 Apache HTTPD, DotNet, Go, Java, NodeJS 및 Python을 지원합니다.

자동 계측을 사용하려면 SDK 및 계측 구성을 사용하여 계측 리소스를 구성하세요.

  • opentelemetry-instrumentation.yaml 작성
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4317 # Opentelemetry collector 주소
    # endpoint: http://datakit-service.datakit:4319 # Guance datakit opentelemetry collector 주소
  propagators:
    - tracecontext
    - baggage
    - b3
  #sampler:   
      #type: parentbased_traceidratio
      #argument: "0.25"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
  • exporter: 데이터 업로드 주소로, Opentelemetry collector 또는 otlp 프로토콜 데이터를 수신할 수 있는 다른 수집기일 수 있습니다.
  • propagators: 트레이스 데이터 전파자입니다. 더 많은 전파자 동작에 대해서는 문서를 참조하세요.
  • sampler: 샘플링

  • java\nodejs\python: 각 언어의 에이전트로, 실제 프로젝트 요구 사항에 따라 입력합니다.

  • opentelemetry-instrumentation.yaml 실행

kubectl apply -f opentelemetry-instrumentation.yaml
  • instrumentation 확인
[root@k8s-master ~]# kubectl get instrumentation
NAME                 AGE   ENDPOINT                     SAMPLER   SAMPLER ARG
my-instrumentation   65m   http://demo-collector:4317

또는 kubectl get otelinst 명령어를 사용하여 확인할 수 있습니다.

[root@k8s-master ~]#  kubectl get otelinst
NAME                 AGE   ENDPOINT                     SAMPLER   SAMPLER ARG
my-instrumentation   71m   http://demo-collector:4317

Guance

Kubernetes DataKit 설치

Kubernetes DataKit 설치

OpenTelemetry 수집기 구성

  • datakit.yaml의 DaemonSet 아래에 volumeMounts 추가:
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: daemonset-datakit
  name: datakit
  namespace: datakit
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: daemonset-datakit
  template:
    metadata:
      labels:
        app: daemonset-datakit
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      ...
      volumeMounts:
        - mountPath: /usr/local/datakit/conf.d/opentelemetry/opentelemetry.conf
          name: datakit-conf
          subPath: opentelemetry.conf
      ....
  • datakit.yaml의 ConfigMap data 아래에 opentelemetry.conf 추가
apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:
    opentelemetry.conf: |-
        [[inputs.opentelemetry]]
            [inputs.opentelemetry.grpc]
            trace_enable = true
            metric_enable = true
            addr = "0.0.0.0:4319" # 기본값 4317
            [inputs.opentelemetry.http]
            enable = false
            http_status_ok = 200
  • DataKit 재시작

애플리케이션

여기서는 JAVA 애플리케이션 springboot-server를 준비했습니다.

  • springboot-server.yaml 작성
apiVersion: v1
kind: Service
metadata:
  name: springboot-server
  labels:
    app: springboot-server
spec:
  selector:
    app: springboot-server
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 31010
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: springboot-server
spec:
  selector:
    matchLabels:
      app: springboot-server
  replicas: 1
  template:
    metadata:
      labels:
        app: springboot-server
      annotations:
        sidecar.opentelemetry.io/inject: "true"
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
      - name: app
        image: registry.cn-shenzhen.aliyuncs.com/lr_715377484/springboot-server
        ports:
          - containerPort:  8080
            protocol: TCP
  • springboot-server.yaml 실행
kubectl apply -f springboot-server.yaml
  • pod 확인
[root@k8s-master ~]# kubectl get pod -owide
NAME                                 READY   STATUS    RESTARTS   AGE    IP                NODE        NOMINATED NODE   READINESS GATES
demo-collector-59b9447bf9-dz47k      1/1     Running   0          24h    100.111.156.98    k8s-node1   <none>           <none>
springboot-server-64b78f4487-9hv9r   1/1     Running   0          24h    100.111.156.108   k8s-node1   <none>           <none>
  • pod 상세 정보 확인
root@k8s-master ~]# kubectl describe pod springboot-server-64b78f4487-9hv9r
Name:         springboot-server-64b78f4487-9hv9r
Namespace:    default
Priority:     0
Node:         k8s-node1/172.31.22.247
Start Time:   ...
Labels:       app=springboot-server
              pod-template-hash=64b78f4487
Annotations:  cni.projectcalico.org/containerID: 5700e2ab666a8bbc32b1ac84cc3d98137a7e186ca5cf4b0b6e7407ac8139d391
              cni.projectcalico.org/podIP: 100.111.156.108/32
              cni.projectcalico.org/podIPs: 100.111.156.108/32
              instrumentation.opentelemetry.io/inject-java: true
              sidecar.opentelemetry.io/inject: true
Status:       Running
IP:           100.111.156.108
IPs:
  IP:           100.111.156.108
Controlled By:  ReplicaSet/springboot-server-64b78f4487
Init Containers:
  opentelemetry-auto-instrumentation:
    Container ID:  containerd://c5747d8217b43fcb1a8eac00fbd33d70c7b25d1a3f0faaccdacea94c8b1e016b
    Image:         ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
    Image ID:      ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java@sha256:f903e6eb067f28cba1f37b6ac592b511c61ce0bf2a73f6e7619359ac5d500d85
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
      /javaagent.jar
      /otel-auto-instrumentation/javaagent.jar
    ...
    Mounts:
      /otel-auto-instrumentation from opentelemetry-auto-instrumentation (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lbmf6 (ro)
Containers:
  app:
    Container ID:   containerd://0db185a75e9eeb5eed97aaf8e707f4bd30f210e404f5fae98fc0d55a300a4470
    Image:          registry.cn-shenzhen.aliyuncs.com/lr_715377484/springboot-server
    Image ID:       registry.cn-shenzhen.aliyuncs.com/lr_715377484/springboot-server@sha256:bf394ec31566653bc6aa0e56dfc94a602bde3d95dfb08ac96d7f33c5dc00005e
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      ...
    Ready:          True
    Restart Count:  0
    Environment:
      JAVA_TOOL_OPTIONS:                    -javaagent:/otel-auto-instrumentation/javaagent.jar
      OTEL_SERVICE_NAME:                   springboot-server
      OTEL_EXPORTER_OTLP_ENDPOINT:         http://demo-collector:4317
      OTEL_RESOURCE_ATTRIBUTES_POD_NAME:   springboot-server-64b78f4487-9hv9r (v1:metadata.name)
      OTEL_RESOURCE_ATTRIBUTES_NODE_NAME:   (v1:spec.nodeName)
      OTEL_PROPAGATORS:                    tracecontext,baggage,b3
      OTEL_RESOURCE_ATTRIBUTES:            k8s.container.name=app,k8s.deployment.name=springboot-server,k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=springboot-server-64b78f4487
    Mounts:
      /otel-auto-instrumentation from opentelemetry-auto-instrumentation (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lbmf6 (ro)

Init Containers: 초기화 컨테이너로, opentelemetry-auto-instrumentation sidecar를 실행했습니다.

기본 JAVA 애플리케이션에 주입된 환경 변수

    Environment:
      JAVA_TOOL_OPTIONS:                    -javaagent:/otel-auto-instrumentation/javaagent.jar
      OTEL_SERVICE_NAME:                   springboot-server
      OTEL_EXPORTER_OTLP_ENDPOINT:         http://demo-collector:4317
      OTEL_RESOURCE_ATTRIBUTES_POD_NAME:   springboot-server-64b78f4487-9hv9r (v1:metadata.name)
      OTEL_RESOURCE_ATTRIBUTES_NODE_NAME:   (v1:spec.nodeName)
      OTEL_PROPAGATORS:                    tracecontext,baggage,b3
      OTEL_RESOURCE_ATTRIBUTES:            k8s.container.name=app,k8s.deployment.name=springboot-server,k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=springboot-server-64b78f4487

이로써 JAVA 애플리케이션에 opentelemetry-auto-instrumentation sidecar가 성공적으로 주입되었습니다.

Trace 데이터 생성

  • 호스트에서 다음 명령을 실행하여 trace 데이터를 생성합니다.

    [root@k8s-master ~]# curl http://100.111.156.108:8080/gateway
{"msg":"下单成功","code":200}
    100.111.156.108은 pod의 ip입니다.

  • svc의 포트에 접근하여 trace 데이터를 생성할 수도 있습니다. 

    [root@k8s-master ~]# curl http://localhost:31010/gateway
{"msg":"下单成功","code":200}

애플리케이션 로그 정보 확인

[root@k8s-master ~]# kubectl logs -f springboot-server-64b78f4487-9hv9r
....
2023-08-* 16:34:17.454 [http-nio-8080-exec-8] INFO  c.z.o.s.c.ServerController - [auth,74] traceId=a1b510158fc09c55c04de2d9472d10d7 spanId=61b6bd8264f7d8b1 - this is auth
2023-08-* 16:34:17.456 [http-nio-8080-exec-5] INFO  c.z.o.s.f.CorsFilter - [doFilter,32] traceId=a1b510158fc09c55c04de2d9472d10d7 spanId=62370160a0fc0738 - url:/billing,header:
accept              :application/json, application/*+json
traceparent             :00-a1b510158fc09c55c04de2d9472d10d7-057f9e068e9cc007-01
b3              :a1b510158fc09c55c04de2d9472d10d7-057f9e068e9cc007-1
user-agent              :Java/1.8.0_212
host                :localhost:8080
connection              :keep-alive

2023-08-* 16:34:17.456 [http-nio-8080-exec-5] INFO  c.z.o.s.c.ServerController - [billing,82] traceId=a1b510158fc09c55c04de2d9472d10d7 spanId=9514404368a2d4fd - this is method3,null

traceparent, b3와 같은 trace 관련 정보가 생성된 것을 확인할 수 있습니다.

로그에도 traceId와 spanId가 생성된 것을 확인할 수 있습니다. 로그를 trace와 연결하는 방법에 대한 자세한 내용은 로그 연동 문서를 참조하세요.

애플리케이션 데이터를 Guance로 직접 푸시

  • opentelemetry-instrumentation.yaml 파일의 OTEL_EXPORTER_OTLP_ENDPOINT 조정
[root@k8s-master ~]# kubectl describe pod springboot-server-64b78f4487-t7gph |grep OTEL_EXPORTER_OTLP_ENDPOINT
      OTEL_EXPORTER_OTLP_ENDPOINT:         http://datakit-service.datakit:4319
  • 다음 yaml을 다시 실행
kubectl delete -f opentelemetry-instrumentation.yaml 
kubectl apply -f opentelemetry-instrumentation.yaml 

kubectl delete -f springboot-server.yaml
kubectl apply -f springboot-server.yaml

  • 애플리케이션 url에 접근하여 trace 데이터를 생성합니다.

  • Guance 계정에 로그인하여 트레이스 뷰를 확인합니다.

트레이스 상세

소스 코드

opentelemetry-operator-demo

문서 평가

이 페이지가 도움이 되었나요?