Christophe Blefari

OCTOBER 9, 2023

Chat with BigQuery data — this is a recycling of all the chatbot use-cases, once again. It's an example where you can use natural language to access BigQuery data. Heap is a product analytics solution to understand better you funnel acquisition performance. This is a must-read and a good showcase of what you can do.

Python 130

Python Data Database Java 130

Christophe Blefari

FEBRUARY 16, 2024

Why you need LLMOps — A great post that encapsulates all the words needed to understand what needs to be done when it comes to put LLM in production. Introducing vector search in BigQuery — RAG is everywhere and BigQuery enters the game. Here the English paper on arxiv. Snowflake lowers the cost of task from x1.5

Food 130

Food Data SQL Government 130

Hevo

MARCH 22, 2024

When migrating data from one BigQuery dataset to another, you can organize and manage your data more effectively for different purposes, including development, testing, and production. Additionally, migration from one […]

Datasets 52

Datasets Data Management Data Integration 52

Hevo

MARCH 22, 2024

When migrating data from one BigQuery dataset to another, you can organize and manage your data more effectively for different purposes, including development, testing, and production. Additionally, migration from one […]

Datasets 52

Datasets Data Management 52

Christophe Blefari

JANUARY 20, 2023

The main thing to notice is that the average company is a Finance company using Airflow with BigQuery and they struggle—like you probably—to hire people. For that you can follow this overview about Vertex AI—the Google Cloud Platform manage machine learning product. Don't target 100% coverage — Yes.

Google Cloud 130

Google Cloud Data Hadoop Business Intelligence 130

Christophe Blefari

APRIL 5, 2024

A talker did a survey during his keynote about the query engines used by the audience and Spark is still leading to BigQuery/Snowflake/Athena. How we built a 19 PiB logging platform with ClickHouse — ClickHouse is a tech company and you can see it from the blogpost. When is the data model finished?

SQL 130

SQL Data Python Datasets 130

Christophe Blefari

JANUARY 27, 2023

Awesome raw ideas with sharks circling around waiting for a new productivity highness. Finally Noah shared how he improved data quality by removing 80% of the tests and Ronald proposed a framework to create data products in Airflow. The difference with similar product is the possibility to also sync to Postgres. It gives ideas.

Data 130

Data Algorithm Data Pipeline SQL 130

Monte Carlo

MARCH 29, 2023

Today, I’m excited to share that Monte Carlo’s data observability platform has achieved a Google Cloud Ready – BigQuery designation. By earning this designation, Monte Carlo has proven its data observability platform meets a core set of stringent functionality and interoperability requirements when integrating with BigQuery.

Google Cloud 52

Google Cloud Designing Cloud Programming 52

Monte Carlo

MARCH 29, 2023

Today, I’m excited to share that Monte Carlo’s data observability platform has achieved a Google Cloud Ready – BigQuery designation. By earning this designation, Monte Carlo has proven its data observability platform meets a core set of stringent functionality and interoperability requirements when integrating with BigQuery.

Google Cloud 52

Google Cloud Designing Cloud Programming 52

Christophe Blefari

JULY 22, 2023

As cloud databases like Snowflake and BigQuery brought it years ago, this was mandatory for the Postgres stack. Taking charge of tables: introducing OpenHouse for big data management — New data product at LinkedIn: OpenHouse. Also vertical product like this can give ideas to marketplace on what can be great reportings.

Data 130

Data Database Big Data Data Engineering 130

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. In this episode Vishnu Venkataraman shares the journey that he and his team have taken to build and evolve their systems and improve the product offerings that they are able to support.

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

OCTOBER 30, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. By the time errors have made their way into production, it’s often too late and damage is done.

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Weekly

FEBRUARY 18, 2024

Benn Stancil: The problem was the product - How the modern data stack got lost Generated via DallE-3 The week is full of retrospect for the Modern Data Stack :-) The author shares a unique point of view on how the lack of collaboration among the MDS companies leads to its loss of popularity. Please add your thoughts if you live in Europe.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Data Engineering Podcast

NOVEMBER 13, 2022

Summary Building data products is an undertaking that has historically required substantial investments of time and talent. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.

Building 130

Building Metadata MongoDB MySQL 130

Data Engineering Podcast

JULY 10, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. What are some of the strategies you find useful for identifying scope and targets for testing/monitoring of data products?

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

OCTOBER 2, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. As with every data asset/product, the lineage graph is only as useful as the data that it stores.

IT 100

IT Food PostgreSQL MongoDB 100

Christophe Blefari

APRIL 28, 2023

Last but not least a more "traditional" AI category: End-to-end ML modeling in BigQuery — BigQuery added over the last years a lot of ML capabilities to the engine. What if with SQLglot you could detect a breaking change before it happens in production? The video also greatly showcase ChatGPT plugins.

SQL 100

SQL Food PostgreSQL Data 100

Christophe Blefari

FEBRUARY 24, 2023

At the moment the best way to develop in your data warehouse is still in the query editor of BigQuery / Snowflake / etc., Measuring everything — This blog is a proposition and a signal for you that you should measure absolutely everything to understand what is happening in your product. Often they are right.

Kafka 130

Kafka Data Lake Data Storage Data 130

Data Engineering Podcast

JUNE 12, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Data Engineering Podcast

NOVEMBER 6, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. By the time errors have made their way into production, it’s often too late and damage is done.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Ascend.io

APRIL 3, 2023

Continuous Productivity Improvements Our team is dedicated to helping you get the most value possible from your intelligent data pipelines. If you look closely, you will see evidence of these in your infrastructure bill and overall productivity on the Ascend platform. Along those lines, this is a week of many small improvements.

Data Pipeline 52

Data Pipeline AWS Cloud Engineering 52

Booking.com Engineering

DECEMBER 2, 2022

We make extensive use of Google BigQuery in PPC due to the scale of our business (up to 2 million room nights booked per day). BigQuery saves us substantial time — instead of waiting for hours in Hive/Hadoop, our median query run time is 20 seconds for batch, and 2 seconds for interactive queries[3].

Systems 52

Systems Cloud MySQL Relational Database 52

Monte Carlo

MARCH 12, 2024

You can export these results to a BigQuery table, Looker dashboard, or other aggregated report to do a full analysis. You can then leverage a data observability tool, like Monte Carlo, to automatically parse the metadata and query logs in BigQuery to detect incidents and display end-to-end data lineage. Courtesy of Google Cloud.

Google Cloud 52

Google Cloud Metadata SQL Data Lake 52

ThoughtSpot

APRIL 2, 2024

Over the past couple of years, ThoughtSpot and Google have collaborated on a series of seamless user experiences—enabling deployments on Google Cloud Platform, creating the ability to live query entire Google BigQuery analytics catalogs, and integrating key Looker Modeling functionality just to name a few.

Google Cloud 59

Google Cloud Cloud Accessible Accessibility 59

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. How has the emergence of the "modern data stack" influenced the product direction?

Building 130

Building IT Metadata MongoDB 130

Monte Carlo

AUGUST 22, 2023

One ecommerce company was able to investigate and improve their poor performing dbt models, while a personal care product company used Performance to detect and disable a query taking up a disproportionate amount of compute within a BigQuery slot. Additional integrations, alerting, and root cause analysis functionality is planned.

Metadata 52

Metadata Data Pipeline Data Warehouse Utilities 52

Monte Carlo

MAY 31, 2023

They built their stack around the Google Cloud Platform , including BigQuery, Looker, Airflow, and Fivetran. That small group of data pros encompassed data science and data visualization specialists, a production support team to handle incidents, and a few data engineers. They use Looker to consume the data.”

Google Cloud 52

Google Cloud Data Engineering Data Engineer Data 52

Christophe Blefari

JULY 8, 2023

Read dbt metrics documentation As an extension I've seen 2 things this week that I feel makes sense here: VulcanSQL — A data API framework for DuckDB, Snowflake, BigQuery, PostgreSQL. Paperspace is an all-in-one SaaS product to develop, train and deploy AI applications. Redpanda is a great product for developers.

Kafka 130

Kafka PostgreSQL Transportation Data 130

Christophe Blefari

SEPTEMBER 1, 2023

After Excel, Notebooks get a second youth — Meta explained how they schedule Jupyter Notebooks in production , Google announced the BigQuery studio with embedded Notebooks in the UI and Jupyter released Jupyter AI (you call it with %ai ) to bring Gen AI to the notebook. New features in Airflow — with 2.7

Food 130

Food Data SQL Python 130

Data Engineering Weekly

FEBRUARY 19, 2023

Netflix: Scaling Media Machine Learning at Netflix Netflix writes about media machine learning infrastructure and media-focused ML infrastructure to reduce the time from ideation to productization for media ML practitioners. The blog mostly focuses on how it optimizes the batch pipeline using Airflow and BigQuery.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Ascend.io

MARCH 6, 2023

Expanded Cost Metrics of Data Planes This upgrade continues to simplify the observability of Databricks, Snowflake, and BigQuery infrastructure supporting your data pipelines. If you recall, you can configure these data clouds through the Ascend platform, as we outlined in Link Multiple Data Clouds to Ascend on February 6.

Management 52

Management Data Pipeline Data Ingestion Cloud 52

Ascend.io

MARCH 6, 2023

Expanded Cost Metrics of Data Planes This upgrade continues to simplify the observability of Databricks, Snowflake, and BigQuery infrastructure supporting your data pipelines. If you recall, you can configure these data clouds through the Ascend platform, as we outlined in Link Multiple Data Clouds to Ascend on February 6.

Management 52

Management Data Pipeline Data Ingestion Cloud 52

Data Engineering Weekly

APRIL 30, 2023

Automated Sanity Checks It is crucial to guarantee the reliability and accuracy of ML models before deploying them to production. Intuit provides insightful information about performing a set of tests called machine learning model sanity checks in a pre-production environment. It follows a simple four-step process.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Knowledge Hut

SEPTEMBER 21, 2023

It provides more than 200 cloud services and products that enable organizations to build, run, and manage applications and services through Microsoft-managed data centers. GCP provides over 150 products and services to help build, deploy, run, and analyze large workloads quickly. So, let’s dive in! What Is Azure? to its users.

Cloud 52

Cloud Google Cloud Cloud Computing Cloud Storage 52

Hevo

FEBRUARY 15, 2023

And, through which marketing channel do our customers rate our product the best?’ Replicating data from Freshsales to BigQuery […] ‘Hey, how many customers do we receive through email? Do you have an answer to such questions from your team’s data analysts? If yes, great! If not, don’t worry. I’m here to help you with that.

Data 52

Data 52

Towards Data Science

JANUARY 16, 2024

Image by the author Late 1990s to Early 2000s - Productize Demand for reporting kept growing. Result: Companies started to sell pre-configured data warehouses as products. Examples include: Amazon Redshift, Google BigQuery, Snowflake, Azure Synapse Analytics, Databricks etc. Architectures became complex. So what was missing?

Data Warehouse 67

Data Warehouse Data Architecture Relational Database NoSQL 67

Data Engineering Podcast

DECEMBER 11, 2022

Planetscale is a serverless option for your MySQL workloads that lets you focus on your applications without having to worry about managing the database or fight with differences between development and production. By the time errors have made their way into production, it’s often too late and damage is done.

Database 100

Database MySQL Data Lake MongoDB 100