Google Pub/Sub to BigQuery the Simple Way

A hands-on guide to implementing BigQuery Subscriptions in Pub/Sub for simple message and streaming ingestion

Jim Barlow

Published in

Towards Data Science

8 min readSep 21, 2023

Google’s latest planet-scale data warehouse subscription-based streaming ingestion water-borne military capability: BigSub. In this case, the Pub never made it to General Availability, so you will have to get your pints elsewhere. Photo by Thomas Haas on Unsplash

Motivation

I have encountered many situations in the past where I wanted to get Pub/Sub messages into a BigQuery table, but I never managed to find a particularly simple way of…

Google Pub/Sub to BigQuery the Simple Way

A hands-on guide to implementing BigQuery Subscriptions in Pub/Sub for simple message and streaming ingestion

Motivation

Written by Jim Barlow