Skip to content

Using tmpfs in WAL

In some cases, if the disk performance is insufficient, we can use a portion of memory to meet the disk read/write requirements of WAL.

  1. Create a tmpfs mount directory on the host:

    sudo mkdir -p /mnt/wal-ramdisk

  2. Create a tmpfs disk using 1 GiB of memory space:

    sudo mount -t tmpfs -o size=1G tmpfs /mnt/wal-ramdisk

  3. Check the mount status:

    $ df -h /mnt/wal-ramdisk/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.0G     0  1.0G   0% /mnt/wal-ramdisk

  4. Modify the WAL directory in datakit.conf:

    [dataway.wal]
  path = "/mnt/wal-ramdisk"

  1. On the Kubernetes Node, create a ramdisk directory:

    # Here we change the directory; by default, in datakit.yaml, the Node's /root/datakit_cache
# is already mounted to the /usr/local/datakit/cache directory in the Datakit container
mkdir -p /root/datakit_cache/ramdisk

  2. Create a 1 GiB tmpfs:

    mount -t tmpfs -o size=1G tmpfs /root/datakit_cache/ramdisk

  3. Add the following environment variable in datakit.yaml, then restart the Datakit container:

    - name: ENV_DATAWAY_WAL_PATH
  value: /usr/local/datakit/cache/ramdisk
Adjust tmpfs size as needed

By default, the disk space for each category in WAL is set to 2 GiB, which is generally sufficient. In a tmpfs scenario, it may not be practical to allocate such a large amount of memory for each category. Here, only 1 GiB (i.e., all data categories share 1 GiB of tmpfs space) of memory is used to meet the disk requirements of WAL. This may be enough under conditions where the data volume is not large and the network (between Datakit and Dataway) is ok.

If the host (or Kubernetes Node) restarts, the data in WAL will be lost, but a Datakit restart will not affect this.

After setting this up, you will see a ramdisk directory in the cache directory. Once Datakit starts, if WAL is generated, you will see various data category disk files in the ramdisk directory:

# Enter /usr/local/datakit/cache or /mnt/wal-ramdisk/
$ du -sh ramdisk/*
8.0K    ramdisk/custom_object
0       ramdisk/dialtesting
4.0K    ramdisk/dynamic_dw
4.0K    ramdisk/fc
4.0K    ramdisk/keyevent
4.3M    ramdisk/logging
1000K   ramdisk/metric
4.0K    ramdisk/network
4.0K    ramdisk/object
4.0K    ramdisk/profiling
4.0K    ramdisk/rum
4.0K    ramdisk/security
4.0K    ramdisk/tracing

Feedback

Is this page helpful? ×