Istio¶

Istio のパフォーマンス指標を表示します。Incoming Request Volume、Incoming Success Rate、Incoming Requests By Source And Response Code、Outgoing Requests By Destination And Response Code などを含みます。

収集設定¶

前提条件¶

Kubernetes がデプロイ済みであること
DataKit がデプロイ済みであること。Kubernetes クラスター向けの <Datakit のインストール> を参照してください
Istio がデプロイ済みであること

説明: サンプルの Istio バージョンは 1.11.2 です。

DataKit 設定¶

apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data: # 以下は追加部分
  prom_istiod.conf: |-
    [[inputs.prom]] 
      url = "http://istiod.istio-system.svc.cluster.local:15014/metrics"
      source = "prom-istiod"
      metric_types = ["counter", "gauge"]
      interval = "60s"
      tags_ignore = ["cache","cluster_type","component","destination_app","destination_canonical_revision","destination_canonical_service","destination_cluster","destination_principal","group","grpc_code","grpc_method","grpc_service","grpc_type","reason","request_protocol","request_type","resource","responce_code_class","response_flags","source_app","source_canonical_revision","source_canonical-service","source_cluster","source_principal","source_version","wasm_filter"]
      metric_name_filter = ["istio_requests_total","pilot_k8s_cfg_events","istio_build","process_virtual_memory_bytes","process_resident_memory_bytes","process_cpu_seconds_total","envoy_cluster_assignment_stale","go_goroutines","pilot_xds_pushes","pilot_proxy_convergence_time_bucket","citadel_server_root_cert_expiry_timestamp","pilot_conflict_inbound_listener","pilot_conflict_outbound_listener_http_over_current_tcp","pilot_conflict_outbound_listener_tcp_over_current_tcp","pilot_conflict_outbound_listener_tcp_over_current_http","pilot_virt_services","galley_validation_failed","pilot_services","envoy_cluster_upstream_cx_total","envoy_cluster_upstream_cx_connect_fail","envoy_cluster_upstream_cx_active","envoy_cluster_upstream_cx_rx_bytes_total","envoy_cluster_upstream_cx_tx_bytes_total","istio_request_duration_milliseconds_bucket","istio_request_duration_seconds_bucket","istio_request_bytes_bucket","istio_response_bytes_bucket"]
      #measurement_prefix = ""
      measurement_name = "istio_prom"
      #[[inputs.prom.measurements]]
      # prefix = "cpu_"
      # name ="cpu"
      [inputs.prom.tags]
        app_id="istiod"
  #### ingressgateway
  prom-ingressgateway.conf: |-
    [[inputs.prom]] 
      url = "http://istio-ingressgateway-ext.istio-system.svc.cluster.local:15020/stats/prometheus"
      source = "prom-ingressgateway"
      metric_types = ["counter", "gauge"]
      interval = "60s"
      tags_ignore = ["cache","cluster_type","component","destination_app","destination_canonical_revision","destination_canonical_service","destination_cluster","destination_principal","group","grpc_code","grpc_method","grpc_service","grpc_type","reason","request_protocol","request_type","resource","responce_code_class","response_flags","source_app","source_canonical_revision","source_canonical-service","source_cluster","source_principal","source_version","wasm_filter"]
      metric_name_filter = ["istio_requests_total","pilot_k8s_cfg_events","istio_build","process_virtual_memory_bytes","process_resident_memory_bytes","process_cpu_seconds_total","envoy_cluster_assignment_stale","go_goroutines","pilot_xds_pushes","pilot_proxy_convergence_time_bucket","citadel_server_root_cert_expiry_timestamp","pilot_conflict_inbound_listener","pilot_conflict_outbound_listener_http_over_current_tcp","pilot_conflict_outbound_listener_tcp_over_current_tcp","pilot_conflict_outbound_listener_tcp_over_current_http","pilot_virt_services","galley_validation_failed","pilot_services","envoy_cluster_upstream_cx_total","envoy_cluster_upstream_cx_connect_fail","envoy_cluster_upstream_cx_active","envoy_cluster_upstream_cx_rx_bytes_total","envoy_cluster_upstream_cx_tx_bytes_total","istio_request_duration_milliseconds_bucket","istio_request_duration_seconds_bucket","istio_request_bytes_bucket","istio_response_bytes_bucket"]
      #measurement_prefix = ""
      measurement_name = "istio_prom"
      #[[inputs.prom.measurements]]
      # prefix = "cpu_"
      # name ="cpu"
  #### egressgateway
  prom-egressgateway.conf: |-
    [[inputs.prom]] 
      url = "http://istio-egressgateway-ext.istio-system.svc.cluster.local:15020/stats/prometheus"
      source = "prom-egressgateway"
      metric_types = ["counter", "gauge"]
      interval = "60s"
      tags_ignore = ["cache","cluster_type","component","destination_app","destination_canonical_revision","destination_canonical_service","destination_cluster","destination_principal","group","grpc_code","grpc_method","grpc_service","grpc_type","reason","request_protocol","request_type","resource","responce_code_class","response_flags","source_app","source_canonical_revision","source_canonical-service","source_cluster","source_principal","source_version","wasm_filter"]
      metric_name_filter = ["istio_requests_total","pilot_k8s_cfg_events","istio_build","process_virtual_memory_bytes","process_resident_memory_bytes","process_cpu_seconds_total","envoy_cluster_assignment_stale","go_goroutines","pilot_xds_pushes","pilot_proxy_convergence_time_bucket","citadel_server_root_cert_expiry_timestamp","pilot_conflict_inbound_listener","pilot_conflict_outbound_listener_http_over_current_tcp","pilot_conflict_outbound_listener_tcp_over_current_tcp","pilot_conflict_outbound_listener_tcp_over_current_http","pilot_virt_services","galley_validation_failed","pilot_services","envoy_cluster_upstream_cx_total","envoy_cluster_upstream_cx_connect_fail","envoy_cluster_upstream_cx_active","envoy_cluster_upstream_cx_rx_bytes_total","envoy_cluster_upstream_cx_tx_bytes_total","istio_request_duration_milliseconds_bucket","istio_request_duration_seconds_bucket","istio_request_bytes_bucket","istio_response_bytes_bucket"]
      #measurement_prefix = ""
      measurement_name = "istio_prom"
      #[[inputs.prom.measurements]]
      # prefix = "cpu_"
      # name ="cpu"

apiVersion: apps/v1
kind: DaemonSet
...
spec:
  template
    spec:
      containers:
      - env:
        volumeMounts: # 以下は追加部分
        - mountPath: /usr/local/datakit/conf.d/prom/prom_istiod.conf
          name: datakit-conf
          subPath: prom_istiod.conf
        - mountPath: /usr/local/datakit/conf.d/prom/prom-ingressgateway.conf
          name: datakit-conf
          subPath: prom-ingressgateway.conf
        - mountPath: /usr/local/datakit/conf.d/prom/prom-egressgateway.conf
          name: datakit-conf
          subPath: prom-egressgateway.conf

DataKit を再デプロイします。

kubectl delete -f datakit.yaml
kubectl apply -f  datakit.yaml

アプリケーション接続設定¶

Envoy のメトリクス収集器を有効化します。

業務 Pod に以下の annotations を追加します（具体的には spec.template.metadata 配下）。これで Envoy のメトリクスデータを収集できます。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
spec:
  template:
    metadata:
    ...
      annotations:  # 以下は追加部分
        datakit/prom.instances: |
          [[inputs.prom]]
            url = "http://$IP:15020/stats/prometheus"
            source = "bookinfo-istio-product"
            metric_types = ["counter", "gauge"]
            interval = "60s"
            tags_ignore = ["cache","cluster_type","component","destination_app","destination_canonical_revision","destination_canonical_service","destination_cluster","destination_principal","group","grpc_code","grpc_method","grpc_service","grpc_type","reason","request_protocol","request_type","resource","responce_code_class","response_flags","source_app","source_canonical_revision","source_canonical-service","source_cluster","source_principal","source_version","wasm_filter"]
            metric_name_filter = ["istio_requests_total","pilot_k8s_cfg_events","istio_build","process_virtual_memory_bytes","process_resident_memory_bytes","process_cpu_seconds_total","envoy_cluster_assignment_stale","go_goroutines","pilot_xds_pushes","pilot_proxy_convergence_time_bucket","citadel_server_root_cert_expiry_timestamp","pilot_conflict_inbound_listener","pilot_conflict_outbound_listener_http_over_current_tcp","pilot_conflict_outbound_listener_tcp_over_current_tcp","pilot_conflict_outbound_listener_tcp_over_current_http","pilot_virt_services","galley_validation_failed","pilot_services","envoy_cluster_upstream_cx_total","envoy_cluster_upstream_cx_connect_fail","envoy_cluster_upstream_cx_active","envoy_cluster_upstream_cx_rx_bytes_total","envoy_cluster_upstream_cx_tx_bytes_total","istio_request_duration_milliseconds_bucket","istio_request_duration_seconds_bucket","istio_request_bytes_bucket","istio_response_bytes_bucket"]
            #measurement_prefix = ""
            measurement_name = "istio_prom"
            #[[inputs.prom.measurements]]
            # prefix = "cpu_"
            # name = "cpu"
            [inputs.prom.tags]
            namespace = "$NAMESPACE"

パラメータの説明

url：Exporter のアドレス
source：収集器名
metric_types：メトリクスタイプのフィルタ
measurement_name：収集後のメトリクスセット名
interval：メトリクス収集頻度、秒単位
$IP：Pod の内部 IP に対するワイルドカード
$NAMESPACE：Pod が所属する namespace
tags_ignore：無視する tag
metric_name_filter：保持するメトリクス名

メトリクス¶

以下のデータ収集では、デフォルトで host というグローバル tag が追加されます（tag 値は DataKit が動作しているホスト名です）。また、設定内で [inputs.{{.InputName}}.tags] を使って他のタグを指定できます。

 [inputs.{{.InputName}}.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"
  # ...

メトリクス詳細¶

指標	説明	データ型	単位
`istio_agent_process_virtual_memory_bytes`	バイト単位の仮想メモリサイズ	int	B
`istio_agent_go_memstats_alloc_bytes`	割り当て済みで、なお使用中のバイト数	int	B
`istio_agent_go_memstats_heap_inuse_bytes`	使用中のヒープバイト数	int	B
`istio_agent_go_memstats_stack_inuse_bytes`	スタックアロケータが使用中のバイト数	int	B
`istio_agent_go_memstats_last_gc_time_seconds`	最後のガベージコレクションが行われてからの 1970 年基準の経過秒数	int	s
`istio_agent_go_memstats_next_gc_bytes`	次回のガベージコレクションが発生する時点のヒープバイト数	int	B
`istio_agent_process_cpu_seconds_total`	ユーザー時間とシステム時間を合わせた CPU 使用時間の合計（秒）	int	count
`istio_agent_outgoing_latency`	外向きリクエスト（例: トークン交換サーバー、CA など）の遅延（ミリ秒）	int	count
`istio_requests_total`	リクエスト総数	int
`istio_agent_pilot_xds`	XDS を使ってこの pilot に接続しているエンドポイント数	int	count
`istio_agent_pilot_xds_pushes`	lds、rds、cds、eds の Pilot のビルドおよび送信エラー	int	count
`istio_agent_pilot_xds_expired_nonce`	期限切れ nonce を持つ XDS リクエストの総数	int	count
`istio_agent_pilot_push_triggers`	push が発生した回数の合計。push の理由でラベル付けされます	int	count
`istio_agent_pilot_endpoint_not_ready`	未準備状態として見つかったエンドポイント	int	count
`envoy_cluster_upstream_cx_total`	envoy cluster upstream cx total	int	count
`istio_agent_pilot_conflict_inbound_listener`	競合している inbound listener の数	int	count
`istio_agent_pilot_conflict_outbound_listener_http_over_current_tcp`	現在の wildcard tcp listener と競合する wildcard http listener の数	int	count
`istio_agent_pilot_conflict_outbound_listener_tcp_over_current_tcp`	現在の tcp listener と競合する tcp listener の数	int	count
`istio_agent_pilot_conflict_outbound_listener_tcp_over_current_http`	現在の wildcard http listener と競合する wildcard tcp listener の数	int	count

APM 収集¶

1 Zipkin 収集器を有効化する¶

datakit.yaml を修正し、ConfigMap を通じて zipkin.conf を DataKit の /usr/local/datakit/conf.d/zipkin/zipkin.conf ディレクトリにマウントします。以下のように datakit.yaml を修正します。

apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data: # 以下は追加部分
  zipkin.conf: |-
    [[inputs.zipkin]]
      pathV1 = "/api/v1/spans"
      pathV2 = "/api/v2/spans"

apiVersion: apps/v1
kind: DaemonSet
...
spec:
  template
    spec:
      containers:
      - env:
        volumeMounts: # 以下は追加部分
        - mountPath: /usr/local/datakit/conf.d/zipkin/zipkin.conf
          name: datakit-conf
          subPath: zipkin.conf

kubectl delete -f datakit.yaml
kubectl apply -f  datakit.yaml

トレースデータは zipkin.istio-system の Service に送られ、送信ポートは 9411 です。
DataKit のデプロイ時にトレース指標収集のための Zipkin 収集器は有効化済みです。DataKit サービスの namespace は datakit、ポートは 9529 なので、ここでは変換が必要です。

2 ClusterIP の Service を定義する¶

apiVersion: v1
kind: Service
metadata:
  name: datakit-service-ext
  namespace: datakit
spec:
  selector:
    app: daemonset-datakit
  ports:
    - protocol: TCP
      port: 9411
      targetPort: 9529

デプロイ完了後、クラスター内部のコンテナは datakit-service-ext.datakit.svc.cluster.local:9411 を使って DataKit の 9529 ポートへアクセスできます。

3 ExternalName の Service を定義する¶

apiVersion: v1
kind: Service
metadata:
  name: zipkin
  namespace: istio-system
spec:
  type: ExternalName
  externalName: datakit-service-ext.datakit.svc.cluster.local

デプロイ完了後、クラスター内部のコンテナから zipkin.istio-system.svc.cluster.local:9411 を使って DataKit にデータを送信できます。

ログ¶

DataKit のデフォルト設定では、コンテナが /dev/stdout に出力するログを収集します。

ログ収集の詳細については、以下を参照してください。

ベストプラクティス¶

ベストプラクティスには、Istio のインストール、Istio 付属プロジェクトのデプロイ、RUM/APM の関連付けなどの拡張手順が含まれます。詳細はIstio に基づくマイクロサービス可観測性のベストプラクティスを参照してください。