Serverless Workers on GCP Cloud Run - Python SDK

View Markdown

On a GCP Cloud Run worker pool, you run a standard long-lived Temporal Worker. Register Workflows and Activities the same way you would with any other Worker, and Temporal Cloud scales the pool up and down as work arrives and drains. Worker Versioning is required for Serverless Workers; set a versioning behavior on each Workflow.

The GCP-specific integration is the OpenTelemetry plugin, which exports traces and Temporal Core metrics to a collector running alongside your Worker.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see Deploy a Serverless Worker on GCP Cloud Run.

Add observability with OpenTelemetry

The temporalio.contrib.gcp.OpenTelemetryPlugin configures observability with defaults suited to a Cloud Run worker pool. By default, the plugin configures:

OTLP gRPC export to localhost:4317, the collector sidecar endpoint.
service.name from the Cloud Run-provided CLOUD_RUN_WORKER_POOL environment variable.
A replay-safe OpenTelemetry tracer provider.
Temporal Core metrics with a 60-second export interval.

The underlying metrics and traces are the same ones the Python SDK emits in any environment. For general observability concepts and the full list of available metrics, see Observability - Python SDK and the SDK metrics reference.

Pass the plugin to Client.connect. Opt into add_temporal_spans=True to trace named operations such as RunWorkflow:GreetingWorkflow:

gcp_open_telemetry/worker.py

# Endpoint, service name, Core metrics, and tracer provider all use the GCP
# plugin defaults. The opt-in adds named Temporal operation spans.
plugin = OpenTelemetryPlugin(add_temporal_spans=True)
client = await Client.connect(
    settings.address,
    namespace=settings.namespace,
    api_key=settings.api_key,
    tls=True,
    plugins=[plugin],
)
worker = Worker(
    client,
    task_queue=settings.task_queue,
    workflows=[GreetingWorkflow],
    activities=[compose_greeting],
    graceful_shutdown_timeout=WORKER_GRACEFUL_SHUTDOWN_TIMEOUT,
)

Named Temporal operation spans are opt-in. The endpoint, service name, tracer provider, and Core metrics use the plugin defaults, but you must set add_temporal_spans=True to emit spans for Workflow and Activity operations.

Run the collector as a sidecar

The plugin exports to a collector you run as a second container in the Worker Pool. Use the Google-Built OpenTelemetry Collector, which detects the Cloud Run resource, authenticates through the Worker Pool's service account, exports metrics to Google Managed Service for Prometheus, and exports traces through the Google Cloud Telemetry API.

Provide the following collector configuration. It receives OTLP on localhost:4317, detects the GCP resource, and routes metrics and traces to their respective pipelines:

gcp_open_telemetry/collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

processors:
  batch/traces:
    send_batch_max_size: 200
    send_batch_size: 200
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_percentage: 65
    spike_limit_percentage: 20
  resource_detection:
    detectors: [gcp]
    timeout: 10s
  transform/collision:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["exported_location"], attributes["location"])
          - delete_key(attributes, "location")
          - set(attributes["exported_cluster"], attributes["cluster"])
          - delete_key(attributes, "cluster")
          - set(attributes["exported_namespace"], attributes["namespace"])
          - delete_key(attributes, "namespace")
          - set(attributes["exported_job"], attributes["job"])
          - delete_key(attributes, "job")
          - set(attributes["exported_instance"], attributes["instance"])
          - delete_key(attributes, "instance")
          - set(attributes["exported_project_id"], attributes["project_id"])
          - delete_key(attributes, "project_id")
  transform/set_project_id:
    error_mode: ignore
    trace_statements:
      - set(resource.attributes["gcp.project_id"], resource.attributes["gcp.project.id"]) where resource.attributes["gcp.project.id"] != nil
      - set(resource.attributes["gcp.project_id"], resource.attributes["cloud.account.id"]) where resource.attributes["gcp.project_id"] == nil and resource.attributes["cloud.account.id"] != nil

exporters:
  googlemanagedprometheus:
  otlp_grpc:
    endpoint: telemetry.googleapis.com:443
    compression: none
    balancer_name: pick_first
    auth:
      authenticator: googleclientauth

extensions:
  googleclientauth:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [googleclientauth, health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource_detection, transform/collision]
      exporters: [googlemanagedprometheus]
    traces:
      receivers: [otlp]
      processors:
        [memory_limiter, resource_detection, transform/set_project_id, batch/traces]
      exporters: [otlp_grpc]

The metrics pipeline forwards each SDK export directly, without a batch processor, so a runtime shutdown-time export cannot collide with a periodic export on the same Managed Prometheus time series. Traces use a dedicated five-second batch processor.

To verify telemetry after deploying, confirm that Trace Explorer contains RunWorkflow:GreetingWorkflow with service.name equal to the Worker Pool name, and that Metrics Explorer contains prometheus.googleapis.com/temporal_workflow_completed_total/counter.

Add observability with OpenTelemetry​

Run the collector as a sidecar​

Add observability with OpenTelemetry

Run the collector as a sidecar