Kodo-X Split¶

Note

Please read this document carefully to ensure that you have proper backup measures and a rollback plan in place during implementation to ensure the safety and stability of the system.

Introduction¶

When handling a large number of write requests, resource consumption in kodo-x may concentrate on specific types of processing tasks, especially Metrics data processing, leading to other data not being consumed in time, thus causing data backlog issues.

Solution ideas:

Create an independent kodo-x instance dedicated to processing Metrics data to reduce resource usage on the existing instance.
Modify the existing kodo-x instance to stop processing Metrics data and focus on other tasks.

Architecture Diagram¶

Before splitting:

After splitting:

Prerequisites¶

Have operation permissions for Guance cluster
Plan scheduling resources for kodo-x and the split kodo-x-metric in advance

Implementation Plan¶

1. Service Backup¶

kubectl get deploy -n forethought-kodo kodo-x -o yaml > kodo-x-deploy.yaml

2. Create kodo-x-metric Service¶

2.1 Create kodo-x-metric ConfigMap¶

kubectl get cm -n forethought-kodo kodo-x -o yaml > kodo-x-metric-cm.yaml

Modify the following information:

Change name to kodo-x-metric
Remove resourceVersion and uid fields

kubectl apply -f kodo-x-metric-cm.yaml

2.2 Create kodo-x-metric Deployment¶

kubectl get deploy -n forethought-kodo kodo-x -o yaml > kodo-x-deploy.yaml

Modify the following information:

Change metadata-name to kodo-x-metric
Remove resourceVersion and uid fields under metadata
Change name under volumeMounts and volumes to kodo-x-metric

...
metadata:
  name: kodo-x-metric    ##### Change to kodo-x-metric
  namespace: forethought-kodo 
  resourceVersion: "129925433"   ##### Remove
  uid: e4f0c541-4d91-47d4-82c8-91eec2757cb2   ##### Remove
...
        volumeMounts:
        - mountPath: /kodo/config/config.yaml
          name: kodo-x-metric  ##### Change to kodo-x-metric
          subPath: config.yaml

      volumes:
      - configMap:
          defaultMode: 420
          name: kodo-x-metric ##### Change to kodo-x-metric
          optional: false
        name: kodo-x-metric ##### Change to kodo-x-metric

kubectl apply -f kodo-x-deploy.yaml

3. Create kodo-x-metric Service¶

3.1 Modify kodo-x-metric Parameters¶

Configuration file: kodo-x-metric.yaml (other configurations are the same as the existing kodo-x.yaml)

global:
    metric_workers: 32  # Increase Metrics data processing capability
    object_workers: 8
    keyevent_workers: 8
    workers: 0  # Disable default worker
    log_workers: 0  # Disable log processing
    backup_log_workers: 0  # Disable backup log processing

Refer to the Application Service Configuration Guide for more information.

Restart the service:

kubectl rollout restart -n forethought-kodo deploy kodo-x-metric

3.2 Modify kodo-x Parameters¶

The existing kodo-x instance will only handle non-Metrics type tasks, further optimizing resource allocation.

Configuration file: kodo-x.yaml

global:
    metric_workers: -1  # Disable Metrics data processing
    object_workers: -1  # Disable object processing
    keyevent_workers: -1  # Disable key event processing