Cocos Creator Session Replay (Experimental)¶

This document describes Cocos Creator Session Replay initialization, Camera selection, performance parameters, touch privacy, and node privacy rules.

Prerequisites¶

Session Replay depends on:

1. A native Android or iOS build;
2. RUM already initialized;
3. A valid RUM View currently active;
4. Using the Cocos SDK and native dependencies in the current SDK version combinations.

If there is no valid RUM Context, the SDK skips the current frame and does not generate replay data independent of RUM. It is recommended to enable autoTrack.scenes or manually call guanceSdk.rum.startView() before frame capture.

Install Standalone Replay Package¶

Starting from Cocos SDK 0.1.0-alpha.6, Session Replay is provided separately by @cloudcare/cocos-session-replay. Install exactly the same version of both packages in the Cocos project root:

npm install @cloudcare/cocos-sdk@0.1.0-alpha.7 @cloudcare/cocos-session-replay@0.1.0-alpha.7
npx --no-install guance-cocos install --project . --replay

Reopen Creator, then generate and compile the native project. For iOS projects using CocoaPods, run pod install as well. For rules on preserving SPM configuration and switches, see App Access.

The base package does not include the Replay APIs, frame capture implementation, or the Replay native dependencies. The following example saves the combined instance as the exported guanceSdk in observability.ts; other modules reuse this instance.

Initialization¶

import { guanceSdk as baseSdk } from '@cloudcare/cocos-sdk/creator3';
import { withSessionReplay } from '@cloudcare/cocos-session-replay/creator3';

export const guanceSdk = withSessionReplay(baseSdk);

guanceSdk.start({
  ...,
  rum: {
    androidAppId: 'android-rum-app-id',
    iosAppId: 'ios-rum-app-id',
  },
  replay: {
    sampleRate: 1,
    sessionOnErrorSampleRate: 0,
    captureFps: 2,
    maxImageDimension: 720,
    imagePolicy: {
      quality: 'medium',
    },
    touchPrivacy: 'show',
  },
  autoTrack: {
    scenes: true,
  },
});

For Creator 2, change the import entry of both packages to /creator2. Composition must happen before SDK initialization or Hybrid attach(); subsequent .replay, start({ replay }), and attach({ replay }) calls are made on the combined instance.

Invalid sampling rates, FPS, image dimensions, or traffic policies throw a TypeError or RangeError at the TypeScript layer.

Image Size Policy¶

captureFps independently controls the target frame capture frequency, while imagePolicy controls image encoding quality and per-frame size. To use per-frame limits, explicitly pass imagePolicy:

replay: {
  captureFps: 3,
  maxImageDimension: 720,
  imagePolicy: {
    quality: 'medium',
    maxFrameBytes: 40 * 1024,
  },
}

imagePolicy supports the following fields:

An explicitly set maxImageDimension and imagePolicy.maxFrameBytes override the preset values of the quality grade. Android V2 uses WebP and iOS V2 uses JPEG; when encoding exceeds the limit, quality is reduced first, then dimensions are lowered. If the image still exceeds the limit, the image is not saved, while pending touch records can still be saved.

0.1.0-alpha.7 removes the 60-second scroll budget for images, the budget-triggered frame rate reduction, and the 100 KiB first-frame burst allowance. When upgrading, remove imagePolicy.maxBytesPerMinute and imagePolicy.adaptiveCapture; these two fields are no longer part of the configuration API. captureFps is a target frequency; deduplication, frame capture, and encoding time still affect the actual output. The SDK processes only one frame at a time and skips missed capture slots when the capture time slot is exceeded; it does not queue concurrent tasks.

When imagePolicy is not passed, the legacy image path is retained with no encoded byte limit, and the per-frame limits above do not apply. Passing imagePolicy: {} applies the medium preset. Once explicitly enabled, the native V2 image interfaces are required; if these interfaces are unavailable or return oversized images, an error is reported rather than falling back to legacy interfaces that cannot guarantee size limits.

Camera Selection¶

By default, the first Camera found in the current scene is used. Multi-Camera projects should explicitly specify the Camera to replay:

import { guanceSdk } from './observability';

export function selectReplayCamera(camera: unknown): void {
  guanceSdk.setReplayCamera(camera);
}

setReplayCamera() stores only one Camera at a time. When called multiple times, the Camera passed in the latest call overrides the previously set Camera. The SDK does not capture or composite multiple Cameras simultaneously. If the previous frame is being captured at the time of the call, the new Camera takes effect from the next capture.

After switching the main Camera, call setReplayCamera() again. If there is no available Camera in the scene, the current frame is skipped.

In 0.1.0-alpha.7, Creator 3 captures frames using a separate offscreen Camera, avoiding modification of the rendering target of the display Camera and avoiding impact on input coordinates that depend on the Camera. This adjustment does not change the single-Camera capture limit; after upgrading, verify multi-Camera behavior, screen orientation changes, and node mask rendering.

Touch Privacy¶

touchPrivacy controls whether Session Replay records touch positions:

Touch collection is part of Session Replay and is independent of autoTrack.actions: even if autoTrack.actions is disabled, touchPrivacy: 'show' still records Replay touches; conversely, enabling Action auto-collection while keeping touchPrivacy: 'hide' means touch positions still do not appear in the Replay. Even when the previous and next frames are visually unchanged, pending touch operations are saved separately to the Replay.

0.1.0-alpha.7 fixes the issue in Creator 3 where touches consumed by UI controls were missing from the Replay, and preserves normal input dispatch. After enabling touchPrivacy: 'show', verify both the business response of controls such as buttons and joysticks and the touch positions in the Replay to confirm that operations are not recorded twice. This fix does not extend the collection scope of autoTrack.actions.

The current Cocos API only supports global touch privacy settings and does not support per-node touch privacy overrides. guanceSdk.replay.setPrivacy(node, 'hide') only hides the node's visual content and does not hide touch records at that position. Sensitive pages should use touchPrivacy: 'hide' when starting Session Replay.

Node Privacy¶

All EditBox nodes use mask by default. You can also set rules for specific nodes:

guanceSdk.replay.setPrivacy(accountNode, 'mask');
guanceSdk.replay.setPrivacy(secretPanelNode, 'hide');
guanceSdk.replay.setPrivacy(publicNode, 'unmask');

unmask only removes custom rules. If the node is still an EditBox, the default mask continues to apply.

Privacy regions are calculated based on the node's world-coordinate rectangle. Custom rendering, particles, shaders, RenderTextures, or visual content that extends beyond the node's bounding box are not automatically derived as privacy regions and must be checked scene by scene in integration tests.

Managing Masks Per Page¶

Rules set by setPrivacy() are bound to the passed node instances and do not switch automatically by page name or RUM View. When different pages need to mask different content, the business logic for page entry and exit should manage the rules for each page's nodes.

The following example switches between a "Diagnostics Page" and an "Action Page" in the same scene: diagnosticsPage and motionPage are the root nodes of the two pages, and privateTokenNode is an ordinary node inside the diagnostics page that needs masking. After SDK initialization or Hybrid attach() completes, call the corresponding functions on page switches:

function showDiagnostics() {
  // Reapply the rule on every entry to ensure the masking rule exists before the sensitive node is shown.
  guanceSdk.replay.setPrivacy(privateTokenNode, 'mask');
  motionPage.active = false;
  diagnosticsPage.active = true;
}

function showMotion() {
  // Hide the sensitive page first, then remove the custom rules of that page's nodes.
  diagnosticsPage.active = false;
  guanceSdk.replay.setPrivacy(privateTokenNode, 'unmask');
  motionPage.active = true;
}

If the action page also has sensitive nodes, set mask or hide for each of those nodes before they are shown, and clear the corresponding rules on exit. When a page contains multiple custom-masked nodes, each node must be managed individually.

Additional considerations for the page lifecycle:

• Setting only the page or parent node to active = false does not remove the registered custom rules. Hiding a page is not a substitute for calling unmask; otherwise, the mask region of the old page may affect subsequent frames.
• Before a page exits or its nodes are destroyed, clear the custom rules registered for that page. This logic can be placed in the project's page manager or in the component's enable, disable, and destroy callbacks.
• Reapply the rules when re-entering a page. If a page is destroyed and rebuilt, the newly created node instances must be passed in.
• Switching RUM View names does not automatically set or clear node rules. Hybrid enterCocos() / leaveCocos() manages collection attribution; pages still need to manage node masks themselves.
• unmask only removes the custom rule of the specified node; it does not disable global privacy protection or cancel the default EditBox mask.

These rules affect the rendering of nodes in Cocos frame capture. Hybrid native pages continue to use the Session Replay privacy configuration of the host Native SDK, while touch positions are controlled separately by touchPrivacy.

During verification, cover "enter page → switch to another page → re-enter"; confirm in the Replay that sensitive content remains masked continuously and that no residual masks remain on other pages.

For more privacy recommendations, see Data and Privacy.

Runtime Behavior¶

• Uses Cocos RenderTexture to capture RGBA frames.
• When imagePolicy is not configured, the legacy image path is retained; when explicitly configured, Android uses WebP V2 and iOS uses JPEG V2, and the actual encoded size is checked.
• Privacy masks are applied before image compression.
• Frames whose content is identical or approximately static are skipped; changes to the View, frame dimensions, or privacy rules force a change frame.
• When an image is skipped due to deduplication or encoding size, pending touch records are still saved separately.
• When the previous frame is still being processed, no new frame is processed concurrently.
• A single frame capture or encoding failure only discards the current frame and does not stop subsequent Session Replay collection.
• Temporary images are deleted from the application temporary directory after being written to the Native SDK.

Traffic Calculation¶

The following only calculates the image payload of a single recorded session; 1 KiB = 1024 bytes, 1 MiB = 1048576 bytes. Replay Segments, touch metadata, upload protocol overhead, and retries must be counted separately.

Calculation Method¶

Measured image traffic (MiB/min) = total bytes of post-deduplication images ÷ corresponding recording duration (seconds) × 60 ÷ 1048576
Planned image traffic (MiB/min) ≈ average encoded bytes per image × expected images saved per minute ÷ 1048576
Full-frame image theoretical limit (MiB/min) ≈ maxFrameBytes × captureFps × 60 ÷ 1048576

Downloads generated by repeated playback are deduplicated by image resource ID; multiple records reusing the same image do not accumulate the resource size repeatedly. The statistics duration and image set must correspond; the total image size of the entire Replay cannot be divided by the duration of only one of its Views. What is downloaded from the cloud is the image payload size. The real total upload traffic on the device must be measured on the device side.

Full-Frame Theoretical Limits for Six Configurations¶

The table assumes every frame is saved and reaches the per-frame size limit. It is an approximate full-frame image upper limit, not a typical consumption value, total upload limit, or scroll quota. For example, the theoretical image limit for medium at 3 FPS is 40960 × 3 × 60 ÷ 1048576 ≈ 7.03 MiB/min; actual images are usually smaller than 40 KiB.

Sample Estimate Ranges for Six Configurations¶

The following table uses 21 existing Battle images re-encoded with libwebp 1.6.0 according to the Android policy, estimating by multiplying the minimum and maximum image sizes by 120 or 180 images per minute. The lower values represent the reference lower bound for this set of dynamic samples and are not the minimum traffic for all scenarios.

These are second-pass encoding estimates of already compressed images and have not yet been continuously recorded on real devices for each of the six configurations. low is scaled to 480×270; medium/high use 720×405 source images without upscaling, so the high data does not represent the traffic of 960 px frame capture. This table is not applicable to inferring iOS JPEG traffic and is not a guaranteed range for other scenarios.

Android On-Device Reference¶

On September 13, 2026, the Battle scene of the Creator 2.4.9 Android example from the Guance Cocos development package was captured using captureFps: 3, quality: 'medium', 720 px long edge, and a 40 KiB per-frame limit. All 83 WebP images from this scene were obtained:

This measurement is a device reference from the Guance development package and does not represent a separately completed TrueWatch on-device traffic test. For this Android Battle scene, approximately 2.0–2.1 MiB/min can be used as a reference for image capacity planning. This on-device result is slightly higher than the second-pass encoding estimate for medium in the previous table; on-device data should be preferred. low/high still only have sample re-encoding estimates and cannot be claimed as verified on real devices.

The raw recording of the full playback spans 68.953 seconds, while the console session header displays 48.15 seconds; calculation should use the corresponding raw recording duration. Two images outside the Battle scene could not be retrieved from the server, so the total download amount for the full playback is not a reference for complete upload traffic. Missing images, protocol overhead, and retries cannot be derived from the image estimates above.

Session Sampling and Overall Capacity¶

When frames are identical or approximately static, deduplication reduces the number of images; capture and encoding time can also lower the actual frame rate. Production capacity should be estimated based on the average image size and save frequency on target devices and real scenarios, measuring high-motion scenes, menus, and static pages separately.

Session sampling controls the number of sessions participating in recording and does not change the image limits of an individual recorded session. If sessions are sampled uniformly and recording durations and scene distributions are similar, the ordinary sampled portion can be roughly estimated:

Overall image traffic ≈ total session minutes × sampleRate × average image traffic per minute of recorded sessions

Also combine RUM sampling and the actual minutes that participated in recording. Recording, Segments, and network overhead generated by error-session additional sampling are counted separately. When accurate billing or network capacity data is required, use device-side upload statistics as the source of truth.

Upgrade to alpha.7¶

Upgrade both the base package and the Replay package in the Cocos project root and update the native integration:

npm install @cloudcare/cocos-sdk@0.1.0-alpha.7 @cloudcare/cocos-session-replay@0.1.0-alpha.7
npx --no-install guance-cocos install --project . --replay

• Projects already using the alpha.6 split-package API continue to reuse the instance returned by withSessionReplay(baseSdk).
• Remove imagePolicy.maxBytesPerMinute and imagePolicy.adaptiveCapture. Use captureFps to control frequency, and use maxImageDimension and imagePolicy.maxFrameBytes to control per-image size. Keeping these removed fields in old JavaScript configuration does not enable the per-minute budget.
• Every image is subject to the per-frame limit, including the first frames of new Views and screen orientation changes. The scroll budget-based frame rate reduction and the separate first-frame burst allowance are no longer available. Re-evaluate actual traffic; see Traffic Calculation.
• Reopen Creator, then generate and compile the native project; run pod install again when using CocoaPods. Replacing JavaScript only does not update the native image processing logic.
• Verify the Creator 3 display, button and joystick response, Replay touch positions, and check View switching, landscape/portrait changes, and privacy masks.

Migrating from the Integrated Package¶

When upgrading from 0.1.0-alpha.5 or earlier:

1. Upgrade the base package to the released 0.1.0-alpha.6 or later, install the Replay package of the same version, and re-run the installer with --replay.
2. Call withSessionReplay(baseSdk) before the first start() / attach(), so business code reuses the combined instance.
3. Change the previously independently exported setReplayCamera(camera) to guanceSdk.setReplayCamera(camera).
4. Move the imports of Replay types such as FTSessionReplayConfig, FTHybridSessionReplayConfig, and FTReplayImagePolicy to @cloudcare/cocos-session-replay/creator2 or /creator3. For complete configurations containing Replay, use FTCocosReplayConfig and FTCocosHybridReplayConfig respectively.
5. Regenerate and compile the native project; replacing JavaScript only does not install the new Replay Bridge.

If Replay is no longer used, remove the composition and Replay configuration, run the installer with --no-replay, and rebuild. This switch removes the native integration managed by the installer. For pausing frame capture at runtime, still use replay.stop() below; use leaveCocos() in Hybrid scenarios.

Start and Stop¶

With the Replay package installed and composed, if replay was not passed in guanceSdk.start(), Session Replay can also be started separately after RUM initialization and after a View has started:

guanceSdk.replay.start({
  sampleRate: 1,
  captureFps: 1,
  maxImageDimension: 720,
  touchPrivacy: 'show',
});

Stop frame capture:

guanceSdk.replay.stop();

guanceSdk.shutdown() also stops frame capture. Do not start Session Replay twice by simultaneously using guanceSdk.start({ replay: ... }) and guanceSdk.replay.start().

Hybrid mode with a native host does not use the start and stop interfaces above. Session Replay is initialized by the native host in native recorder mode, and Cocos automatically switches to the external capture source between enterCocos() and leaveCocos(). See Native Host Hybrid Access.

Performance Recommendations¶

• Start verification with captureFps: 1 and imagePolicy: { quality: 'medium' }; increase to 2 fps only when a more continuous picture is needed.
• Enable sampling only in business environments where replay is required.
• On low-end devices with GPU, memory, or disk pressure, first switch to low, then reduce the longest-edge dimension and the sampling rate.
• Verify image orientation and mask positions in landscape/portrait switching, multi-Camera, complex UI, and low-frame-rate scenarios.