Data Engineering Podcast

APRIL 3, 2022

Signup for the SaaS product at dataengineeringpodcast.com/acryl Hey Data Engineering Podcast listeners, want to learn how the Joybird data team reduced their time spent building new integrations and managing data pipelines by 93%? How do you approach the build vs. buy decision while designing and evolving your data platform?

Designing 100

Designing Data Warehouse Data Engineering Data Engineer 100

Striim

NOVEMBER 30, 2022

When you run Striim on AWS, it lets you create real-time data pipelines for Redshift, S3, Kinesis, Databricks, Snowflake and RDS for enterprise workloads. It comes with pre-built data connectors that can automate your data movement from any source to AWS Redshift or S3 within a few minutes.

AWS 52

AWS Cloud Management Google Cloud 52

Christophe Blefari

NOVEMBER 11, 2022

When it comes to storage it's mainly a row-based vs. a column-based discussion, which in the end will impact how the engine will process data. Machine learning at Riot Games If you play video games like me you'll like this video. Ian describes how Riot Games uses data and what machine learning means.

BI 130

BI Data Warehouse Data Database 130

ProjectPro

JANUARY 31, 2022

Launched in 2014, Snowflake is one of the most popular cloud data solutions on the market. With around 5774 companies using it, Snowflake has recently been added to the top 20 most valued worldwide unicorns and the top 10 most expensive US unicorns. Table of Contents Snowflake Overview and Architecture What is Snowflake Data Warehouse?

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

Data Engineering Podcast

MAY 1, 2022

What were the factors that you considered in the various build vs. buy decisions? What are the most interesting, unexpected, or challenging lessons that you have learned while working on designing and implementing your platform? __init__ to learn about the Python language, its community, and the innovative ways it is being used.

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Data Engineering Podcast

APRIL 12, 2021

It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. what is the lineage of this asset vs. what is the distribution of values in this column, etc.) __init__ Episode Snowflake Podcast Episode Spark Airflow Podcast.__init__

Data Warehouse 100

Data Warehouse Data Pipeline BI Data 100

Data Engineering Podcast

DECEMBER 25, 2022

Summary Encryption and security are critical elements in data analytics and machine learning applications. Output data can be streamed into a data lake for query engines like Presto, Trino or Spark SQL, a data warehouse like Snowflake or Redshift., or any other destination you choose. Pricing for SQLake is simple.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Knowledge Hut

APRIL 26, 2024

Where are we going to learn more about this? Predictive causal analytics, prescriptive analytics and machine learning are some tools used to make decisions and predictions in data science. statistical analysis, machine learning, artificial intelligence, etc.). But what is Data Science? What are its salient characteristics?

Data Science 52

Data Science Machine Learning Hadoop Programming Language 52

Monte Carlo

APRIL 4, 2023

Data mesh vs data warehouse is an interesting framing because it is not necessarily a binary choice depending on what exactly you mean by data warehouse (more on that later). This era has been shaped and defined by the rise of cloud based analytical databases such as Snowflake, Redshift, and BigQuery. What is a Data Mesh?

Data Warehouse 52

Data Warehouse Data Governance Data Architecture 52

Data Engineering Weekly

FEBRUARY 5, 2023

The ML for large-scale production systems highlights the improvement made from the existing heuristic in the YouTube cache replacement algorithm with a new hybrid algorithm that combines a simple heuristic with a learned model, improving the byte miss ratio at the peak by ~9%. Pipelines for data in motion can quickly turn into DAG hell.

Data Engineering 57

Data Engineering Data Engineer Engineering Food 57

Data Engineering Weekly

MARCH 19, 2023

Map table vs. using complex data structure? Debezium Kafka Snowflake Airflow & dbt If anyone is confused with the modern data landscape, the blog is an excellent reminder that you only need a handful of systems to process your data. BuzzFeed shares seven learning on integrating generative AI.

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Towards Data Science

NOVEMBER 4, 2023

Cloud platform leaders made DWH (Snowflake, BigQuery, Redshift, Firebolt) infrastructure management really simple and in many scenarios they will outperform and dedicated in-house infrastructure management team in terms of cost-effectiveness and speed. So here are a few things to consider that can help us answer these questions.

Data Engineering 87

Data Engineering Data Engineer Engineering BI 87

Data Engineering Podcast

MARCH 15, 2021

In this episode the head of data platform for DoorDash, Sudhir Tonse, discusses the technologies that they are using, the approach that they take to adding new systems, and how they think about priorities for what to support for the whole company vs what to leave as a specialized concern for a single team.

Management 100

Management Data Warehouse PostgreSQL Kafka 100

Monte Carlo

AUGUST 25, 2023

Some of the well-known players in the data warehouse sphere include Amazon Redshift, Google BigQuery, and Snowflake. Exploring data lakes A data lake is a reservoir designed to handle both structured and unstructured data, frequently employed for streaming, machine learning, or data science scenarios.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Workfall

JULY 4, 2023

In this blog, we will cover: DBT Overview Snowflake Overview DBT With Snowflake Hands-On Understanding DBT Project Structure Creating Your First Model Conclusion DBT Overview In the magical world of DBT, there are three key concepts that bring order and clarity to the realm of data: models, transformations, and tests.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Monte Carlo

JANUARY 5, 2024

A visualization of the flow of data in data lakehouse architecture vs. data warehouse and data lake. Data lakehouses, like Databricks’ Delta Lake , Amazon Redshift Spectrum , and now Snowflake for Data Lakehouse , help to unify data warehousing, machine learning, and analytics. Image courtesy of Databricks.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

A visualization of the flow of data in data lakehouse architecture vs. data warehouse and data lake. Data lakehouses, like Databricks’ Delta Lake , Amazon Redshift Spectrum , and now Snowflake for Data Lakehouse , help to unify data warehousing, machine learning, and analytics. Image courtesy of Databricks.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Podcast

JUNE 28, 2021

It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. What are you optimizing for on the consistency vs. availability spectrum of CAP? What are you optimizing for on the consistency vs. availability spectrum of CAP?

Data Analysis 100

Data Analysis Scala Data Warehouse Kafka 100

Data Engineering Weekly

JANUARY 29, 2023

The conversation around data observability points out the growing gap in data observability [aka finding the things] vs. fixing the data quality. The author writes an interesting benchmark result comparing Databricks autoscaling vs. fixed computing. Get the new platform guide, and take the next step in your journey to data trust.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Monte Carlo

AUGUST 19, 2023

I’ve also updated this post to include additional layers frequently found within a modern data platform such as data governance, access management, and machine learning. Snowflake, a cloud data warehouse, is a popular choice among data teams when it comes to quickly scaling up a data platform. It depends.

Building 52

Building BI Data Lake Data Governance 52

AltexSoft

AUGUST 29, 2023

If your organization fits into one of these categories and you’re considering implementing advanced data management and analytics solutions, keep reading to learn how data lakes work and how they can benefit your business. Data warehouse vs. data lake in a nutshell. Traditional data warehouses have been around for decades.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

MAY 27, 2018

For complete visibility into the health of your pipeline, including deployment tracking, and powerful alerting driven by machine-learning, DataDog has got you covered. I want to go into stream VS batch here What are the most challenging components to scale? What is Alooma and what is the origin story?

Data Pipeline 100

Data Pipeline MongoDB Scala Google Cloud 100

Data Engineering Weekly

DECEMBER 18, 2022

link] Confessions of a Data Guy: Dataframe Showdown – Polars vs. Spark vs. Pandas vs. DataFusion. Motherbrain: Disrupting private capital using machine learning and an event-driven architecture The blog is an exciting one giving a peak into the private capital ventures approach to finding startup investment strategy.

Data Engineering 52

Data Engineering Data Engineer Engineering Relational Database 52

Monte Carlo

APRIL 24, 2023

By accommodating various data types, reducing preprocessing overhead, and offering scalability, data lakes have become an essential component of modern data platforms , particularly those serving streaming or machine learning use cases. See our post: Data Lakes vs. Data Warehouses. It works in both directions.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

AltexSoft

JANUARY 26, 2022

The story behind a new data job: Analytics engineer vs data engineer vs data analyst vs data scientist vs ML engineer. They commonly prepare data and build machine learning (ML) models. Get acquainted with how data is prepared for machine learning projects in our dedicated video.

Engineering 59

Engineering Software Engineer Software Engineering Data Warehouse 59

Data Engineering Weekly

JANUARY 22, 2023

Data Quality: Shift Left, Bring Consumers Closer Event Source vs. Outbox Pattern vs. CDC - The challenges and opportunities Data Contract - An Executive Overview Data Contract - Why does Everyone Talks about Data Contract now? Learn the 10 most important things to consider when choosing a data observability platform.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Data Engineering Weekly

APRIL 16, 2023

Redshift is no longer a true competitor in the warehouse space. However, as the data stack consolidation accelerates, we will see a much better permission model from Snowflake and Databricks. #3 One of the things that stands out for me in the article is the debate around pre-aggregation vs. lazy aggregation. Airflow is obsolete.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

U-Next

SEPTEMBER 7, 2022

Amazon Redshift. . The Snowflake database. . Data Lake Vs. Data Warehouse: Latest Industry Stats . Data Lake vs. Data Warehouse: Similarities . Data Lake vs. Data Warehouse: Differences . As training data increases, deep learning requires scalability. Data warehouses include the following examples: .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Monte Carlo

JULY 5, 2023

All too often, teams tasked with becoming “data-driven” get excited about new technologies (Snowflake! We spoke with Mark Stange-Tregear, VP of Data at Babylist, and Michael Weiss, Senior Product Manager at Nasdaq, to learn how each has embarked on his respective journey to build a modern data stack premised on overarching data trust.

High Quality Data 52

High Quality Data Data Software Engineer Software Engineering 52

ProjectPro

AUGUST 24, 2021

Data professionals who work with raw data like data engineers, data analysts, machine learning scientists , and machine learning engineers also play a crucial role in any data science project. Big Data Tools: Without learning about popular big data tools, it is almost impossible to complete any task in data engineering.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

AltexSoft

MARCH 14, 2023

A modern data stack can help the company manage and analyze this data effectively by using cloud-based data warehouses like Snowflake , data integration tools like Stitch, and data visualization software like Power BI. Some popular solutions include Snowflake, Google’s BigQuery, and Amazon Redshift. Designed to be modular.

IT 59

IT Data Warehouse Data Governance Data Lake 59

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Is Snowflake a data lake or data warehouse? Data Lake vs Data Warehouse - The Introduction What is a Data warehouse?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

dbt Developer Hub

NOVEMBER 28, 2021

Learning when to use surrogate keys and bring them into your project is a critical skill for any analytics professional. You can also run into problems if the fields are different datatypes (string vs numeric), so sometimes you need to cast as well. Meet the surrogate key. What’s a surrogate key? ​

PostgreSQL 52

PostgreSQL Data Warehouse SQL Datasets 52

ProjectPro

JANUARY 19, 2022

Explore this page further and learn everything about data engineers to find the answer. Furthermore, we will also lay out a learning path on how to become a data engineer that will help one explore this exciting domain. Good knowledge of various machine learning and deep learning algorithms will be a bonus.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

dbt Developer Hub

AUGUST 17, 2022

Identifying the problem ​ This project runs on Snowflake, so all the examples below show the Snowflake UI. We recommend reading through this article, then brushing up on cloud data warehouses and query optimization to help supplement the learnings here. The next step was to check out what the Snowflake query plan looked like.

Data Warehouse 59

Data Warehouse Datasets Cloud Coding 59

Monte Carlo

AUGUST 9, 2023

Data engineering workflow vs DevOps workflow As you can see, there are a lot of similarities between DevOps and DataOps workflows — but each discipline involves its own unique processes. When you’re ready to learn more about how data observability can keep your data reliable and trustworthy throughout your data stack, contact our team.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

AltexSoft

OCTOBER 21, 2022

Read our article on event-driven architecture and Pub/Sub to learn more about this powerful communication paradigm. cloud data warehouses — for example, Snowflake , Google BigQuery, and Amazon Redshift. Learn about scheduled online and live meetups on the Kafka Events page. Kafka cluster and brokers.

Kafka 93

Kafka Hadoop ETL Tools Big Data 93