Google Cloud Pub/Sub

Stream Junction events to Google Cloud Pub/Sub as an ETL pipeline destination, with message attributes, compression, and ordering details.

ETL Pipelines integration is available for the Scale plan.

Event Structure

Each event is published with these Cloud Pub/Sub message attributes:

Type	Key	Value
Attribute	event-type	Junction event type, e.g., daily.data.activity.created
Attribute	blob-type	json: UTF-8 JSON document
Attribute	blob-codec	gzip: The blob is compressed by gzip. null or absent: The blob is uncompressed.
Attribute	idempotency-key	The event delivery ID; only unique amongst all events from the same user and of the same event type.
Ordering Key	-	Junction User ID (If you have enabled Event Ordering)

Payloads smaller than 1KiB would not be compressed. You must check blob-codec to identify if decompression of the message payload blob is necessary.

At this time, Junction publishes exclusively JSON blobs (blob-type == json).

Having said that, you are strongly encouraged to check for and drop any events with unrecognized blob-type and blob-codec.

Pub/Sub event ordering

If you wish to receive events for each Junction user in their publication order, Junction supports Pub/Sub Event Ordering, and uses the Junction User ID as the event ordering key.

Junction publishes all events through the us-central1 and europe-west1 PubSub regional endpoints.

Pub/Sub Exactly Once Delivery is discouraged from being used together with Event Ordering. Our own trial suggested that the combination would struggle to move a high volume of events.

IAM Permissions

Grant the relevant IAM Principals on your topic with these pre-defined roles:

• Pub/Sub Publisher (roles/pubsub.publisher)
• Pub/Sub Viewer (roles/pubsub.viewer)

Or more specifically these permissions if you wish to create a custom IAM role with a minimized grant:

• pubsub.topics.get
• pubsub.topics.publish

Environment	Region	Service Account
Sandbox	US	[e-mail]
Sandbox	Europe	[e-mail]
Production	US	[e-mail]
Production	Europe	[e-mail]

Configuration

Org Management API is available for the Scale plan.

You can manage your Google Cloud Pub/Sub destination through the Org Management API Team ETL Pipeline endpoints.

A basic configuration would look like this:

{
  "team_id": "TEAM_ID",
  "preferences": {
    "enabled": ["cloud_pubsub"],
    "preferred": "cloud_pubsub"
  },
  "cloud_pubsub": {
      "project": "my-gcp-project-123456",
      "topic": "event-ingestion",
      "message_ordering": true
  },
  "push_historical_data": true
}