Flameshot

Flameshot is a lightweight automated profiling tool running in Sidecar mode. It monitors the resource usage (CPU/Memory) of target processes and automatically triggers underlying Profilers (such as async-profiler) when preset thresholds are reached, enabling non-intrusive on-site snapshot collection.

Core Concepts¶

Operating Mode¶

Flameshot is deployed using the Sidecar Container pattern. It must run in the same Pod as the main business container (Main Container) and have PID namespace sharing enabled.

1. Monitor: Flameshot continuously polls the resource levels of target processes within the main container.
2. Trigger: When thresholds are met (e.g., CPU > 80%) or an HTTP API request is received, a collection task is triggered.
3. Execute: Based on the configured language type (currently supporting Java and Go), it invokes the corresponding profiler workflow for the target process.
4. Collect: The generated Profile files (e.g., .jfr or .pprof) are subsequently uploaded to the data observability center.
5. Timed: After configuring FLAMESHOT_AUTO_PROFILING, it periodically collects profiling data for all matched processes. The sample duration defaults to 30 seconds and can be adjusted through FLAMESHOT_AUTO_PROFILING_DURATION.
6. OOM Summary: When a container oom_kill increment is detected, Flameshot tries to automatically parse -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath=... from the target Java process arguments. If the dump file is generated inside the shared volume, Flameshot finds the corresponding .hprof and uploads a summary log.

Use Cases¶

• Production Safety Net: Automatically preserve on-site evidence before a service crashes due to CPU spikes or memory leaks.
• Performance Stress Test Analysis: Cooperate with stress testing platforms to automatically collect performance hotspots under high load.

Configuration¶

All Flameshot behaviors are controlled via environment variables. Configuration is divided into Global Settings and Profiling Policies.

Global Environment Variables¶

These variables control the basic behavior of the Sidecar container.

If DataKit is deployed as a DaemonSet and exposes port 9529 through hostNetwork/hostPort, it is recommended that Flameshot connects directly to the DataKit on the same node as the application Pod instead of using a normal Service domain that may route requests to another node:

- name: NODE_IP
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP
- name: FLAMESHOT_DATAKIT_ADDR
  value: "http://$(NODE_IP):9529/profiling/v1/input"

This keeps each application Pod uploading Profile data to the DataKit instance on its own node, which makes troubleshooting clearer and preserves the node-local collection model.

Profiling Policy Configuration¶

Target monitoring rules are defined via the FLAMESHOT_PROCESSES environment variable. The value must be a standard JSON Array string.

To maintain readability in Kubernetes YAML, it is strongly recommended to use YAML's block scalar syntax (|) for writing the JSON configuration, as shown below:

    env:
      # ... other environment variables ...
      - name: FLAMESHOT_PROCESSES
        value: |
          [
            {
              "service": "user-service",
              "language": "java",
              "command": "^java.*user-service\\.jar$",
              "duration": "60s",
              "events": "cpu,alloc",
              "cpu_usage_percent": 80,
              "mem_usage_percent": 80,
              "mem_usage_mb": 1024,
              "mem_usage_percent_emergency": 92,
              "mem_usage_mb_emergency": 1536,
              "heap_dump_on_memory_emergency": true,
              "emergency_duration": "10s",
              "tags": [
                "env:prod",
                "version:v1.2"
              ]
            }
          ]

Common Field Descriptions:

• service (String): Service name reported to the observability center.
• language (String): Target process language. Currently supports java, go, and golang.
• command (String): Regular expression to match the process command line.
• duration (String): Duration of a single collection (e.g., 30s, 1m). Note: To avoid execution timeouts, it is recommended not to exceed 5 minutes.
• emergency_duration (String): Shorter profiling duration used after an emergency memory hit. 10s or 15s is recommended.
• pprof_url (String): Go pprof HTTP base URL, for example http://127.0.0.1:6060. Required when language is go or golang.
• pprof_types (List): Go pprof types. Supported values are cpu, goroutine, heap, mutex, and block, matching the existing Go pull mode in the profile input.
• pprof_timeout (String): Go pprof request timeout. It should be greater than the CPU profile duration.
• tags (List): List of custom tags; recommended to include meta-information like env, version.
• cpu_usage_percent (Int): CPU trigger threshold (0-N). Values may exceed 100 in multi-core environments.
• mem_usage_percent (Int): Average memory-percentage threshold (0-100), evaluated by the latest 5 points.
• mem_usage_mb (Int): Average RSS threshold in MB, evaluated by the latest 5 points.
• mem_usage_percent_emergency (Int): Instant emergency memory-percentage threshold (0-100). A single hit triggers immediately.
• mem_usage_mb_emergency (Int): Instant emergency RSS threshold in MB. A single hit triggers immediately.
• heap_dump_on_memory_emergency (Bool): Whether this process rule may trigger proactive Heap Dump on emergency memory hits. When omitted, it defaults to enabled if FLAMESHOT_HEAP_DUMP_ENABLED=true.
• cpu_usage_percent, mem_usage_percent, and mem_usage_mb skip threshold checks when omitted or set to 0.
• When FLAMESHOT_POD_MEM_LIMIT is configured, mem_usage_percent and mem_usage_percent_emergency prefer Pod-limit memory percentage instead of host memory percentage.
• When FLAMESHOT_HEAP_DUMP_ENABLED=true, emergency memory hits enqueue a jmap Heap Dump task. This requires the Sidecar to execute the jmap referenced by FLAMESHOT_HEAP_DUMP_JMAP_PATH. If hprof object storage upload is configured, the generated .hprof is uploaded and the event includes hprof_object_key, hprof_download_url, and upload status.

Language Specifics¶

Flameshot invokes different underlying tools depending on the technology stack of the monitored application.

import (
    "net/http"
    _ "net/http/pprof"
)

func main() {
    go http.ListenAndServe("127.0.0.1:6060", nil)
}

- name: POD_IP
  valueFrom:
    fieldRef:
      fieldPath: status.podIP
- name: FLAMESHOT_PROCESSES
  value: |
    [
      {
        "service": "go-app",
        "language": "go",
        "command": "^/app/go-app$",
        "pprof_url": "http://$(POD_IP):6060",
        "pprof_types": ["cpu", "goroutine", "heap", "mutex", "block"],
        "duration": "30s",
        "pprof_timeout": "45s"
      }
    ]

{
  "service": "go-app",
  "language": "go",
  "command": "^/app/go-app",
  "pprof_url": "http://127.0.0.1:6060",
  "pprof_types": ["cpu", "goroutine", "heap", "mutex", "block"],
  "duration": "30s",
  "pprof_timeout": "45s",
  "tags": ["env:prod", "version:v1"]
}

Deployment¶

Kubernetes Sidecar Deployment¶

For Flameshot to work correctly, the Pod configuration must meet the following three conditions:

1. Shared Process Namespace (shareProcessNamespace: true).
2. Shared Storage Volume (EmptyDir).
3. System Capabilities (Capabilities).

YAML Example:

apiVersion: v1
kind: Pod
metadata:
  name: java-app-profiled
spec:
  # 1. [Core] Enable PID sharing so Sidecar can see the Java process
  shareProcessNamespace: true

  volumes:
  - name: shared-data
    emptyDir: {}

  containers:
  # Business Container
  - name: my-app
    image: my-app:latest
    volumeMounts:
    - name: shared-data
      mountPath: /data # Must match Sidecar configuration

  # Flameshot Sidecar
  - name: flameshot
    image: pubrepo.jiagouyun.com/datakit/flameshot:latest
    env:
      - name: FLAMESHOT_PROFILING_PATH
        value: "/data"
      # ... other environment variables ...

    # 2. [Core] Grant ptrace capability
    securityContext:
      capabilities:
        add: ["SYS_PTRACE"]

    # 3. [Core] Mount the same directory
    volumeMounts:
    - name: shared-data
      mountPath: /data

DataKit DaemonSet Notes¶

When DataKit is deployed as a DaemonSet, Flameshot should preferably upload Profile data through the current node IP:

- name: NODE_IP
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP
- name: FLAMESHOT_DATAKIT_ADDR
  value: "http://$(NODE_IP):9529/profiling/v1/input"

Also make sure DataKit satisfies all of the following:

1. The profile input is enabled and registers the Profiling upload endpoint:

   [[inputs.profile]]
     endpoints = ["/profiling/v1/input"]
   

2. The Profiling API is allowed for non-localhost access. If DataKit has HTTP API allow-listing enabled, add /profiling/v1/input to ENV_HTTP_PUBLIC_APIS:

   - name: ENV_HTTP_PUBLIC_APIS
     value: /otel/v1/trace,/otel/v1/metric,/otel/v1/logs,/profiling/v1/input
   

   If this variable already contains other APIs, append /profiling/v1/input to the existing list instead of replacing the whole value. Otherwise, DataKit may return:

   datakit.publicAccessDisabled: api /profiling/v1/input disabled from external IP, only loopback(localhost) allowed
   

3. The DataKit listen address is reachable through the node IP. A common DaemonSet configuration is hostNetwork: true, hostPort: 9529, and ENV_HTTP_LISTEN=0.0.0.0:9529.

OOM HProf Summary Requirements¶

If you want Flameshot to automatically recover .hprof summary information after a Java OOM, all of the following conditions must be met:

1. Enable -XX:+HeapDumpOnOutOfMemoryError in the application JVM arguments.
2. Configure -XX:HeapDumpPath=/data/... in the JVM arguments, and make sure the path is inside the shared volume.
3. Enable FLAMESHOT_OOM_HPROF_ENABLED=true for the Flameshot Sidecar.
4. It is recommended to set FLAMESHOT_OOM_HPROF_MATCH_WINDOW explicitly so the matching window is operationally unambiguous.

For example:

java \
  -XX:+HeapDumpOnOutOfMemoryError \
  -XX:HeapDumpPath=/data/dumps/app.hprof \
  -jar app.jar

Notes:

• Flameshot now discovers HeapDumpPath directly from the target Java process arguments. There is no separate configuration item for the .hprof path.
• FLAMESHOT_OOM_HPROF_ENABLED only enables the Flameshot-side recovery workflow; it does not inject HeapDump-related flags into the target JVM.
• If the target process does not enable HeapDumpOnOutOfMemoryError, or if HeapDumpPath is not inside a shared volume, Flameshot can only record the OOM event and cannot locate the .hprof file.
• If the container is terminated before the dump is fully written, .hprof may still be unavailable.

Docker Local Testing¶

If you need to test in a local Docker environment, use the following command to start Flameshot and monitor the target container.

Prerequisites:

• Use --pid="container:<target_id>" or shared volumes (depending on the specific Docker version).

Test Image: pubrepo.jiagouyun.com/datakit/flameshot:1.85.1-testing_testing-iss-2876

Startup Command Example:

docker run -d \
  --name flameshot-debug \
  --volumes-from <YOUR_JAVA_APP_CONTAINER> \
  -e FLAMESHOT_DATAKIT_ADDR="http://datakit:9529/profiling/v1/input" \
  -e FLAMESHOT_PROCESSES='[{"service":"local-test","command":"java","language":"java","cpu_usage_percent":10}]' \
  pubrepo.jiagouyun.com/datakit/flameshot:1.85.1-testing_testing-iss-2876

API Reference¶

Flameshot provides an HTTP interface allowing users or automated O&M scripts to manually trigger collection tasks.

Manual Triggering¶

Interface Address: GET /v1/profile

Semantic Explanation: This interface is used to generate a Profile dataset on demand, not to retrieve monitoring metrics.

Request Parameters:

Usage Examples:

1. Trigger collection by PID:

   # Collect CPU and memory allocation data for the process with PID 1234 for 30 seconds
   curl "http://localhost:8089/v1/profile?pid=1234&duration=30s&events=cpu,alloc"
   

2. Trigger collection by process name regex:

   # Collect data for the process matching tmall.jar for the default duration
   curl "http://localhost:8089/v1/profile?command=^java\\b.*tmall\\.jar$"
   

JFR format¶

async-profiler events notes:

Troubleshooting¶

1. Cannot collect data?

   • Check if shareProcessNamespace: true is enabled in the Pod.
   • Check if the Sidecar has SYS_PTRACE capability.
   • For Go applications, check if pprof_url is reachable from the Flameshot Sidecar.

   • For Go applications, first verify the pprof endpoint from inside the Flameshot container:

     curl -v "http://<pprof-host>:6060/debug/pprof/"
     curl -v "http://<pprof-host>:6060/debug/pprof/goroutine?debug=0"
     

   • If the log contains connect: connection refused, the target IP:Port is not listening. Check inside the Pod:

     netstat -lntp | grep ':6060'
     ss -ltnp | grep ':6060'
     

   If there is no LISTEN, the application has not enabled pprof, or it is listening on a different port. Seeing only 60602 ... ESTABLISHED does not mean port 6060 is listening. - If pprof listens on the Pod IP, configure pprof_url as http://$(POD_IP):6060; if it listens on loopback or all addresses, http://127.0.0.1:6060 can be used.

2. File not uploaded?

   • Check if FLAMESHOT_PROFILING_PATH is correctly mounted between the two containers.
   • The system automatically manages file life cycles and will attempt to delete temporary files after collection is complete.
   • Go profiling uploads .pprof files as in-memory attachments and usually does not depend on local files. If the Go profiling log already shows upload to DataKit err, check the HTTP status code and response body returned by DataKit first.

   • If DataKit returns 403 with datakit.publicAccessDisabled, /profiling/v1/input has not been allowed for non-localhost access. Add it to DataKit configuration:

     - name: ENV_HTTP_PUBLIC_APIS
       value: /otel/v1/trace,/otel/v1/metric,/otel/v1/logs,/profiling/v1/input
     

   • If DataKit returns input "profile" is not enabled for API "/profiling/v1/input", the profile input is not enabled. Enable it with:

     [[inputs.profile]]
       endpoints = ["/profiling/v1/input"]
     

Changelog¶

0.2.2 (2026-5-12)¶

Bug Fixes¶

• Fix
  • Fixed duplicated host, env, version, service, and other tags in uploaded Profiling tags_profiler metadata.
• Change
  • Removed high-watermark jcmd snapshot support and related configuration items.
  • Added cgroup memory-pressure diagnostic fields to make threshold, current cgroup memory, and cgroup limit easier to verify.

0.2.1 (2026-2-11)¶

Optimize¶

• optimize
  • In a container environment, use the configured resource size as the base value for threshold calculation.

0.2.0 (2026-2-4)¶

New Features¶

• Add config
  • Support configuring scheduled Profiling execution via the environment variable FLAMESHOT_AUTO_PROFILING
• optimize
  • Optimize the threshold processing logic for configuration.

0.1.0 (2025-12-17)¶

The first official release of Flameshot, focusing on providing automated profiling capabilities for Java applications in containerized environments.

New Features¶

• Core Architecture:
  • Support for Kubernetes Sidecar Mode deployment, utilizing shared PID namespaces for non-intrusive monitoring.
  • Support for Linux AMD64 and ARM64 multi-architecture execution.
• Language Support:
  • Java: Deep integration with async-profiler, supporting various event collections like CPU, Alloc, Lock, etc.
  • Automatic detection and adaptation to the target container's JDK environment.
• Trigger Mechanism:
  • Threshold Trigger: Support for automatic triggering based on CPU usage (cpu_usage_percent) and memory usage/amount (mem_usage_percent/mem_usage_mb).
  • API Trigger: Provided HTTP interface GET /v1/monitor (Note: should be /v1/profile as per API section), supporting manual trigger by PID or regex process name matching.
• Data Integration:
  • Support for automatically reporting generated .jfr or flame graph data to DataKit.
  • Support for flexible multi-process monitoring policies and tags (tags) via the FLAMESHOT_PROCESSES environment variable.