Snowflake

SEPTEMBER 13, 2023

Organizations can now leverage Salesforce data directly in Snowflake via zero-ETL data sharing to accelerate decision-making and help streamline business processes. This provides a reliable, secure and highly scalable zero-ETL data-sharing pattern.

Finance 95

Finance AWS Accessible Accessibility 95

Data Engineering Podcast

APRIL 9, 2023

When it was difficult to wire together the event collection, data modeling, reporting, and activation it made sense to buy monolithic products that handled every stage of the customer data lifecycle. The Machine Learning Podcast helps you go from idea to production with machine learning. Don't forget to check out our other shows.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data. Don't forget to check out our other shows.

Data Pipeline 130

Data Pipeline BI SQL Business Intelligence 130

Data Engineering Podcast

JUNE 17, 2023

ML, reverse ETL, etc.) The Machine Learning Podcast helps you go from idea to production with machine learning. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. ML, reverse ETL, etc.) Don't forget to check out our other shows.

Data Lake 130

Data Lake Metadata Data Architecture Machine Learning 130

Data Engineering Podcast

MAY 14, 2023

The Machine Learning Podcast helps you go from idea to production with machine learning. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. The Machine Learning Podcast helps you go from idea to production with machine learning.

Data Lake 147

Data Lake Machine Learning Data Warehouse AWS 147

Data Engineering Podcast

MAY 28, 2023

The Machine Learning Podcast helps you go from idea to production with machine learning. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. The Machine Learning Podcast helps you go from idea to production with machine learning.

Data Lake 162

Data Lake Machine Learning Data Warehouse Education 162

Data Engineering Podcast

JUNE 4, 2023

What are the changes in the broader data ecosystem that have had the most influence on your product goals and customer adoption? Agile Data Engine is a DataOps Management platform for designing, deploying, operating and managing data products, and managing the whole lifecycle of a data warehouse.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Lake 130

Netflix Tech

DECEMBER 14, 2023

Psyberg, An Incremental ETL Framework Using Iceberg Abhinaya Shetty and Bharath Mummadisetty, Data Engineers from Netflix’s Membership Data Engineering team, introduce Psyberg, an incremental ETL framework.

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Data Engineering Podcast

MARCH 10, 2023

Summary With the rise of the web and digital business came the need to understand how customers are interacting with the products and services that are being sold. Product analytics has grown into its own category and brought with it several services with generational differences in how they approach the problem.

Data Warehouse 130

Data Warehouse Data Lake Machine Learning Data Science 130

Monte Carlo

MARCH 28, 2023

As part of this movement, Fivetran and dbt fundamentally altered the data pipeline from ETL to ELT. Zero-ETL What it is : A misnomer for one thing; the data pipeline still exists. This process actually doesn’t involve ETL and instead provides expanded access to the data where it’s stored. Let’s look closer at each.

Data Warehouse 126

Data Warehouse Raw Data Data Pipeline Software Engineer 126

Data Engineering Podcast

MAY 21, 2023

The Machine Learning Podcast helps you go from idea to production with machine learning. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. The Machine Learning Podcast helps you go from idea to production with machine learning.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Podcast

DECEMBER 28, 2022

Summary With all of the messaging about treating data as a product it is becoming difficult to know what that even means. Vishal Singh is the head of products at Starburst which means that he has to spend all of his time thinking and talking about the details of product thinking and its application to data.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

RudderStack

NOVEMBER 2, 2021

In this update, we announce a beta version of Reverse ETL Mirror Sync Mode. Read the post for more information, and sign up to try Mirror Mode today.

40

40

Data Engineering Podcast

APRIL 16, 2023

The Machine Learning Podcast helps you go from idea to production with machine learning. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. The Machine Learning Podcast helps you go from idea to production with machine learning.

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Netflix Tech

NOVEMBER 14, 2023

Write-Audit-Publish (WAP) Process This is the general pattern we use in our ETL pipelines. Write Apply the ETL business logic to the input data identified in Step 1 and write to an unpublished iceberg snapshot based on the Psyberg mode b. The ETL then appends this data to the corresponding partitions within the fact tables.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Snowflake

FEBRUARY 27, 2024

Many of these providers make their products available on Snowflake Marketplace for Snowflake customers to purchase — and they use our integrated Marketplace Monetization capabilities to simplify the process and speed up procurement and sales cycles. Pricing is, of course, controlled and defined by the provider.

Bytes 87

Bytes Electronics Banking Data 87

Data Engineering Podcast

JUNE 11, 2023

The Machine Learning Podcast helps you go from idea to production with machine learning. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. The Machine Learning Podcast helps you go from idea to production with machine learning.

Project 130

Project Building Data Lake Machine Learning 130

Towards Data Science

NOVEMBER 4, 2023

If you don’t have time to learn a new ETL framework you can create a simple data connector yourself. ETL frameworks explosion We can witness a “Cambrian explosion” of various ETL frameworks for data extraction and transformation. Luigi [8] is one of them and it helps to create ETL pipelines. SQL, XML, XLS, CSV and JSON.

Data Engineering 88

Data Engineering Data Engineer Engineering BI 88

Data Engineering Podcast

JANUARY 21, 2024

Recent years have seen the adoption of reverse ETL, cloud data warehouses, and sophisticated product analytics suites. The Machine Learning Podcast helps you go from idea to production with machine learning. Recent years have seen the adoption of reverse ETL, cloud data warehouses, and sophisticated product analytics suites.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

DECEMBER 25, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. By the time errors have made their way into production, it’s often too late and damage is done. Not Endless DAGs!

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

NOVEMBER 13, 2022

Summary Building data products is an undertaking that has historically required substantial investments of time and talent. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.

Building 130

Building Metadata MongoDB MySQL 130

Data Engineering Podcast

FEBRUARY 5, 2023

The Machine Learning Podcast helps you go from idea to production with machine learning. The Machine Learning Podcast helps you go from idea to production with machine learning. What do you have planned for the future of the podcast? Don't forget to check out our other shows. What do you have planned for the future of the podcast?

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Towards Data Science

APRIL 3, 2023

As part of this movement, Fivetran and dbt fundamentally altered the data pipeline from ETL to ELT. Zero-ETL What it is : A misnomer for one thing; the data pipeline still exists. This process actually doesn’t involve ETL and instead provides expanded access to the data where it’s stored. Let’s look closer at each.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Warehouse 98

Knowledge Hut

APRIL 23, 2024

Managing an EDW requires the creation of intricate data models, formulation of data integration strategies, and the implementation of ETL (Extract, Transform, and Load) procedures to bring data into the warehouse. Data accuracy, completeness, and consistency depend on efficient ETL operations.

Database 52

Database Management Relational Database SQL 52

Data Engineering Weekly

DECEMBER 18, 2023

[link] Instacart: Monte Carlo, Puppetry and Laughter: The Unexpected Joys of Prompt Engineering Instacart writes an article exploring prompt techniques used for internal productivity tooling. RudderStack just launched Trino as a Reverse ETL source. It's the only Trino Reverse ETL solution that supports warehouse-based CDC.

Data Engineering 67

Data Engineering Data Engineer Engineering Food 67

Monte Carlo

DECEMBER 4, 2023

At the heart of data engineering lies the ETL process—a necessary, if sometimes tedious, set of operations to move data across pipelines for production. So let’s take a little ChatGPT speaking course and look at a few example prompts for each stage of the ETL process.

PostgreSQL 52

PostgreSQL Data Lake ETL Tools MySQL 52

Data Engineering Podcast

JULY 9, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack Your host is Tobias Macey and today I'm interviewing Max Beauchemin about the concept of entity-centric data modeling for analytical use cases Interview Introduction How did you get involved in the area of data management?

Database-centric 130

Database-centric Machine Learning SQL Data Engineering 130

Engineering at Meta

AUGUST 29, 2023

Notebooks are also being used widely for creating reports and workflows (for example, performing data ETL ) that need to be repeated at certain intervals. In this post, we’ll explain how we married Bento with our batch ETL pipeline framework called Dataswarm (think Apache Airflow ) in a privacy and lineage-aware manner.

SQL 80

SQL Metadata Coding Data Analysis 80

Christophe Blefari

MARCH 2, 2024

I'll give a talk—in French—about how to put in production machine learning at a small scale. And let's be honest this is a productivity boost, especially when you want to write docstrings and comments. Final reminder, next week there is La Conférence MLOps which will take place in Paris on March 7th.

Data 162

Data Python Database BI 162

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Building 130

Building IT Metadata MongoDB 130

Christophe Blefari

DECEMBER 16, 2022

Usual readers of the Data News might notice that I don't go softly with dbt Labs when it comes to their Cloud product, but this is a reality, if I caricature a bit right now dbt Cloud is only a web IDE with the capabilities to run your models, it should be commodity, the real value of dbt exists only in Core for the moment and in the community.

Kafka 130

Kafka Data SQL Cloud 130

Data Engineering Podcast

DECEMBER 17, 2023

The team at 5X understand the pain involved and the barriers to productivity and set out to solve it by pre-integrating the best tools from each layer of the stack. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

APRIL 2, 2023

What are the product categories that were only viable when capital was plentiful and easy to obtain? What are the product categories that you think will be swallowed by adjacent concerns, and which are likely to consolidate to remain competitive? The Machine Learning Podcast helps you go from idea to production with machine learning.

Hadoop 130

Hadoop Machine Learning Python Architecture 130

Snowflake

DECEMBER 21, 2023

They had plenty of data spread across multiple sources, but the time from idea to production was months, quarters or even years. Patch enables software engineers to build highly scalable and performant production apps using Snowflake data, without ETL or changes to your existing infrastructure, and with no boilerplate code in the application.

Software Engineer 82

Software Engineer Software Engineering Database Data Pipeline 82

Data Engineering Weekly

DECEMBER 24, 2023

No 1 rule of the product experience is “Don’t make the user think”; for me, “prompting” makes me think a lot. Your rollout can make/break the core experience of your product without showing much visual change. RudderStack just launched Trino as a Reverse ETL source.

Data Engineering 62

Data Engineering Data Engineer Engineering Deep Learning 62

Knowledge Hut

JANUARY 30, 2024

Categorical Data: Data extraction can also involve categorizing and extracting information like product categories, customer segments, or geographical regions. This can involve SQL queries or ETL (Extract, Transform, Load) processes. This can include news articles, product prices, or social media posts.

ETL Tools 52

ETL Tools Database-centric Data Mining Data Cleanse 52

Data Engineering Podcast

OCTOBER 30, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130