Send Google Cloud Logs to <<>> with Pub/Sub

Use a Cloud Logging Sink to select application logs, then forward them to Guance through Pub/Sub, Cloud Run and DataKit. This guide uses a standalone DataKit on a private host. GKE is not required. The forwarder supports jsonPayload and textPayload; audit protoPayload is unsupported.

Cloud Logging Sink → Pub/Sub → Cloud Run forwarder → private DataKit → Guance.

The forwarder source and README are public and licensed under Apache License 2.0. Keep the DataWay URL and token in DataKit; the forwarder does not need them.

Before you start¶

• Use Cloud Shell, or install gcloud and Git and sign in to Google Cloud. Enable billing on the project.
• Your account needs permission to enable APIs, create service accounts, read and write logs, manage sinks, Pub/Sub and firewall rules, deploy Cloud Run and grant IAM roles. It also needs iam.serviceAccounts.actAs on the build, runtime and push service accounts created below.
• Prepare a VPC subnet in the Cloud Run region, with a /26 or larger address range, and a route to the DataKit host. See Direct VPC egress for network requirements.
• The firewall example uses a Compute Engine VM in the same project and VPC. For another private host, have your network administrator configure the equivalent route and firewall rules before continuing.

Replace the values below. Run all gcloud commands in the same shell; install and configure DataKit on its host. The examples create new resources in one project; use unused names or adjust the names throughout the guide.

PROJECT_ID="my-project"
REGION="us-central1"
NETWORK="my-vpc"
SUBNET="cloud-run-subnet"
DATAKIT_IPV4="10.0.1.10"
DATAKIT_VM="datakit-vm"
DATAKIT_ZONE="us-central1-a"

gcloud services enable run.googleapis.com pubsub.googleapis.com \
  logging.googleapis.com cloudbuild.googleapis.com artifactregistry.googleapis.com \
  compute.googleapis.com iam.googleapis.com --project "$PROJECT_ID"

PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')

1. Prepare DataKit¶

Install DataKit on the private host using the installation command from your Guance workspace. The host must be able to reach the workspace's DataWay endpoint.

In datakit.conf, change the existing [http_api] listener to the host's private IPv4 address:

[http_api]
  listen = "10.0.1.10:9529"

Use the same address as DATAKIT_IPV4, then restart DataKit. Keep TCP 9529 private; allow only the Cloud Run subnet and approved administration sources.

For the Compute Engine VM, add a target tag and a rule allowing the Cloud Run subnet:

SUBNET_CIDR=$(gcloud compute networks subnets describe "$SUBNET" \
  --project "$PROJECT_ID" --region "$REGION" --format='value(ipCidrRange)')
gcloud compute instances add-tags "$DATAKIT_VM" \
  --project "$PROJECT_ID" --zone "$DATAKIT_ZONE" --tags pubsub-datakit
gcloud compute firewall-rules create allow-pubsub-datakit \
  --project "$PROJECT_ID" --network "$NETWORK" --direction INGRESS \
  --action ALLOW --rules tcp:9529 --target-tags pubsub-datakit \
  --source-ranges "${SUBNET_CIDR:?Subnet CIDR is required}"

If the VM uses a host firewall, allow the same subnet there. Check that existing rules do not allow other sources. On the DataKit host, check the listener with the following command, replacing the example IP. Expect HTTP 200; step 5 checks the Cloud Run connection.

curl -i --fail --show-error --max-time 5 "http://10.0.1.10:9529/v1/ping"

The forwarder writes JSON records to /v1/write/logging. No additional logstreaming collector is needed. The destination must use an IPv4 address in 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16.

2. Deploy the forwarder¶

Back in your gcloud shell, create separate build, runtime and push accounts. Only the build account receives roles/run.builder; specify it explicitly during deployment. See build service accounts for the required deployer permissions.

gcloud iam service-accounts create forwarder-build --project "$PROJECT_ID"
gcloud iam service-accounts create forwarder-run --project "$PROJECT_ID"
gcloud iam service-accounts create forwarder-push --project "$PROJECT_ID"

gcloud projects add-iam-policy-binding "$PROJECT_ID" \
  --member "serviceAccount:forwarder-build@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role roles/run.builder

git clone https://github.com/TrueWatchTech/google-cloud-log-forwarder.git
cd google-cloud-log-forwarder

cat > env.yaml <<EOF
DATAKIT_URL: "http://${DATAKIT_IPV4}:9529/v1/write/logging"
EXPECTED_PROJECT_ID: "${PROJECT_ID}"
EXPECTED_SUBSCRIPTION: "projects/${PROJECT_ID}/subscriptions/application-logs-push"
MEASUREMENT: "gcp_pubsub"
EOF

gcloud run deploy pubsub-log-forwarder \
  --source . --project "$PROJECT_ID" --region "$REGION" \
  --no-allow-unauthenticated \
  --build-service-account "projects/${PROJECT_ID}/serviceAccounts/forwarder-build@${PROJECT_ID}.iam.gserviceaccount.com" \
  --service-account "forwarder-run@${PROJECT_ID}.iam.gserviceaccount.com" \
  --network "$NETWORK" --subnet "$SUBNET" --vpc-egress private-ranges-only \
  --cpu 1 --memory 512Mi --min-instances 0 --max-instances 1 --concurrency 1 \
  --env-vars-file env.yaml

Wait for deployment to finish successfully and display a service URL. If it fails with a permission error just after granting a role, allow a few minutes for IAM changes to take effect, then retry the deployment. Keep env.yaml free of credentials. The CPU, memory and instance settings above are starting values, not measured production minimums. Adjust them after testing your workload.

3. Enable authenticated push¶

Grant Invoker permission on this Cloud Run service only. Ensure the Pub/Sub service agent exists, allow it to create tokens for the push account, then create the subscription:

SERVICE_URL=$(gcloud run services describe pubsub-log-forwarder \
  --project "$PROJECT_ID" --region "$REGION" --format='value(status.url)')

gcloud run services add-iam-policy-binding pubsub-log-forwarder \
  --project "$PROJECT_ID" --region "$REGION" \
  --member "serviceAccount:forwarder-push@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role roles/run.invoker

gcloud pubsub topics create application-logs --project "$PROJECT_ID"
gcloud beta services identity create \
  --service pubsub.googleapis.com --project "$PROJECT_ID"
gcloud iam service-accounts add-iam-policy-binding \
  "forwarder-push@${PROJECT_ID}.iam.gserviceaccount.com" \
  --project "$PROJECT_ID" \
  --member "serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com" \
  --role roles/iam.serviceAccountTokenCreator

gcloud pubsub subscriptions create application-logs-push \
  --project "$PROJECT_ID" --topic application-logs --ack-deadline 60 \
  --push-endpoint "${SERVICE_URL}/pubsub" \
  --push-auth-service-account "forwarder-push@${PROJECT_ID}.iam.gserviceaccount.com" \
  --push-auth-token-audience "$SERVICE_URL"

Keep payload unwrapping disabled: the forwarder expects the Pub/Sub JSON envelope. Cloud Run validates the OIDC token; keep authentication and the Invoker IAM check enabled. See authenticated push requirements.

4. Create a test Sink¶

Start with the dedicated log ID pubsub-guide-demo so that you can test without forwarding unrelated logs:

gcloud logging sinks create application-logs \
  "pubsub.googleapis.com/projects/${PROJECT_ID}/topics/application-logs" \
  --project "$PROJECT_ID" \
  --log-filter="logName=\"projects/${PROJECT_ID}/logs/pubsub-guide-demo\" AND (jsonPayload:* OR textPayload:*)"

SINK_WRITER=$(gcloud logging sinks describe application-logs \
  --project "$PROJECT_ID" --format='value(writerIdentity)')

gcloud pubsub topics add-iam-policy-binding application-logs \
  --project "$PROJECT_ID" --member "$SINK_WRITER" --role roles/pubsub.publisher

The Sink sends matching new entries after it is configured; it does not replay historical logs. Finish the sample check below before selecting your application logs. See log routing.

5. Verify delivery¶

Write one JSON entry and one plain-text entry to the test log:

TEST_ID="pubsub-$(date -u +%Y%m%dT%H%M%SZ)"
gcloud logging write pubsub-guide-demo \
  "{\"message\":\"${TEST_ID}-json\"}" \
  --payload-type=json --severity=INFO --project "$PROJECT_ID"
gcloud logging write pubsub-guide-demo "${TEST_ID}-text" \
  --payload-type=text --severity=INFO --project "$PROJECT_ID"

Read the test entries and the forwarder's receipts:

gcloud logging read \
  "logName=\"projects/${PROJECT_ID}/logs/pubsub-guide-demo\" AND SEARCH(\"${TEST_ID}\")" \
  --project "$PROJECT_ID" --freshness=1h --limit=10 --format=json
gcloud logging read \
  "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"pubsub-log-forwarder\" AND resource.labels.location=\"${REGION}\" AND jsonPayload.receipt:*" \
  --project "$PROJECT_ID" --freshness=1h --limit=20 --format=json

1. The first command should return both entries. Record each entry's insertId, timestamp and payload. If they are not visible yet, wait briefly and run it again.
2. In the second command's jsonPayload results, find each matching google_insert_id with http_status: 204 and a 2xx datakit_status. For failures, check reason and the troubleshooting table.
3. In Guance Logs, select source gcp_pubsub and a time range covering the test. Find each record by its google_insert_id.
4. Check that both records have the expected message and that log_timestamp matches the original timestamp. The JSON entry's message contains a JSON string; the text entry keeps its original text.

Cloud Run returning 204 or DataKit returning 2xx does not prove indexing. Verify both records in Guance. Before production use, test throughput, reliability, sizing and application overhead with your own workload.

6. Select your application logs¶

Ensure your application logs already appear in Cloud Logging, then replace the test filter with a narrow application filter. This GKE example selects stdout and stderr from one application container. Replace the cluster, namespace and container names. Select the business container, excluding DataKit; k8s_container also excludes the Cloud Run forwarder.

APP_FILTER=$(cat <<'EOF'
resource.type="k8s_container"
resource.labels.cluster_name="my-cluster"
resource.labels.namespace_name="my-namespace"
resource.labels.container_name="my-app"
(log_id("stdout") OR log_id("stderr"))
(jsonPayload:* OR textPayload:*)
EOF
)
gcloud logging read "$APP_FILTER" \
  --project "$PROJECT_ID" --freshness=1h --limit=5 --format=json

Check that the results contain only the intended application logs. If there are no results, generate an application log and check the filter in Logs Explorer before continuing. For applications outside GKE, use their resource type and application labels. Then update the Sink:

gcloud logging sinks update application-logs \
  --project "$PROJECT_ID" --log-filter="$APP_FILTER"

After the update takes effect, generate a new application log. Find it in Cloud Logging with APP_FILTER, then compare its identifier, timestamp and message with the record in Guance as in step 5. The pubsub-guide-demo test logs no longer match this Sink. Always exclude the forwarder and DataKit's own logs to avoid a forwarding loop.

Field mapping¶

Each Cloud Logging entry becomes one DataKit log record. The default source is gcp_pubsub, set by MEASUREMENT.

For status, DEFAULT and INFO map to info, and EMERGENCY maps to emerg. Other severity names map to lowercase. JSON payload keys are not expanded into separate fields, and resource labels are not mapped.

Troubleshooting¶

The forwarder limits the push request body to 1 MiB and the mapped record to 512 KiB. Pub/Sub delivery is at least once, so retries can duplicate records. Preserve google_insert_id for correlation; it does not provide automatic deduplication.

Messages that remain unacknowledged can expire after the subscription's retention period. Monitor subscription backlog and consider a dead-letter topic for messages that repeatedly fail.