Skip to content

Insufficient Log Engine Storage

Solution 1: Scale Up

You can scale up horizontally or vertically.

  • Horizontal scaling: Add data nodes to the cluster.
  • Vertical scaling: Expand the data disks of the data nodes.

Solution 2: Clean Data

  1. First, use the df -h command to check the disk space of the corresponding ES and determine whether it is full.
  2. Use the following command to view index storage:
$ kubectl exec -ti -n middleware <es_pod_name> -- curl -XGET -u <user>:<password> 127.0.0.1:9200/_cat/indices?v

health status index                                                                                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   wksp_4bcd96fc753e4a87a7f035717e3492f4_logging-000002                                 v8EAVH4FR32gSGIeMGKtSw   1   0          0            0       60.0gb           60.0gb
green  open   wksp_4bcd96fc753e4a87a7f035717e3492f4_logging-000001                                 blKeVx7mT1edx-EJFf0PBA   1   0          0            0       60.1gb           60.1gb
green  open   wksp_4bcd96fc753e4a87a7f035717e3492f4_logging-000003                                 XGDzVakRT4yLx3KdgJpl6Q   1   0          0            0       59.9gb           59.9gb
green  open   wksp_4bcd96fc753e4a87a7f035717e3492f4_logging-000004                                 dXaJwIuKQMKj1W-sGQMgYA   1   0          8            0     53.6kb         53.6kb
green  open   .monitoring-es-7-2023.05.31                                                           hWN-pIKWSpyxHGLAuyrR-w   1   0    3342741      1023346      1.3gb          1.3gb
green  open   .monitoring-es-7-2023.05.30                                                           06VnZgosSuCZR4mUXewBxg   1   0    3386407       931888      1.3gb          1.3gb
green  open   wksp_ae656f5d29764fdd904707a2564b7517_keyevent-000002                                 PecXZi1sTGOvFgEhpbXifA   1   0          0            0       208b           208b
green  open   .monitoring-es-7-2023.06.01                                                           obPVFxemQeGVQC2flCtUiA   1   0    3466792       519395      1.4gb          1.4gb
green  open   .monitoring-es-7-2023.06.02                                                           HcxlyaH1R7-R7Wh1jF12bw   1   0     390356       151592    292.7mb        292.7mb
green  open   .infini_activities-00001                                                              f8f6rb1wTCe7IZDEECrKmQ   1   0        460            0    252.1kb        252.1kb
  1. As shown above, each index has its size and number. Delete indexes with smaller numbers first to free up space.

Index Name Explanation: The index name is composed of Workspace ID + Data Type + Number.

Note: Delete indexes based on the same Workspace ID and data type but different numbers, in ascending order of the number. Do not delete the index with the largest number. The smaller the number, the older the data. Generally, the index with the largest number is the one currently being written to.

Using the query results above as an example, the wksp_4bcd96fc753e4a87a7f035717e3492f4_logging index has four numbered indexes: 000001, 000002, 000003, and 000004. You can delete up to 000001, 000002, and 000003 in order.

Delete command:

$ kubectl exec -ti -n middleware <es_pod_name> -- curl -XDELETE -u <user>:<password> 127.0.0.1:9200/<index>

Feedback

Is this page helpful?