Data Engineering Podcast

AUGUST 3, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. Vinoth Chandar helped to create the Hudi project while at Uber to address this challenge.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

MAY 29, 2022

In this episode they explain how their platform optimizes your Snowflake warehouses to reduce cost, as well as identifying improvements that you can make in your queries to reduce their contribution to your bill. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own.

Cloud 100

Cloud Data Warehouse Data Lake Data 100

Data Engineering Podcast

SEPTEMBER 1, 2021

By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more.

Data Lake 100

Data Lake Cloud AWS SQL 100

Monte Carlo

MARCH 4, 2024

And as crazy as it sounds at first blush, innovations in table formats got prime air-time at both Snowflake and Databricks annual conferences in 2023. At a broad feature level this translates to machine learning monitors that understand how these aspects of your data platform normally behave and alert you when anomalies occur.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Data Engineering Podcast

DECEMBER 8, 2019

And for your machine learning workloads, they just announced dedicated CPU instances. You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.

Data Warehouse 100

Data Warehouse Cloud AWS Relational Database 100

Monte Carlo

AUGUST 25, 2023

Some of the well-known players in the data warehouse sphere include Amazon Redshift, Google BigQuery, and Snowflake. Exploring data lakes A data lake is a reservoir designed to handle both structured and unstructured data, frequently employed for streaming, machine learning, or data science scenarios. Let’s dive in.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Data Engineering Podcast

OCTOBER 16, 2022

In this episode Jason Hughes explains what it means for a lakehouse to be "open" and describes the different components that the Dremio team build and contribute to. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services.

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

MAY 22, 2022

By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more.

Scala 100

Scala SQL Data Data Workflow 100

Data Engineering Podcast

JULY 16, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. Go to dataengineeringpodcast.com/97things today to get your copy!

High Quality Data 100

High Quality Data Data Engineering Data Engineer Coding 100

Monte Carlo

NOVEMBER 14, 2023

Over the last few months, Apache Iceberg has come to the forefront as a promising new open-source table format that removes many of the largest barriers to lakehouse adoption – namely, the high-latency and lack of OLTP (Online Transaction Processing) support afforded by Apache Hive. I think it’s safe to say it’s getting pretty cold in here.

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64

Data Engineering Podcast

NOVEMBER 20, 2021

By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more.

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Monte Carlo

APRIL 24, 2023

By accommodating various data types, reducing preprocessing overhead, and offering scalability, data lakes have become an essential component of modern data platforms , particularly those serving streaming or machine learning use cases. Databricks Data Catalog and AWS Lake Formation are examples in this vein.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Data Engineering Weekly

JUNE 18, 2023

Register Today Leo Godin: Understanding DBT Runtime Environment One of my favorite way to learn new systems is to tail its logs and starring at it for some time. I found fewer blogs focus on a technical deep-dive into the internals of dbt, and this one follows my favorite way of learning!!! Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering MySQL 52

Data Engineering Podcast

MAY 1, 2022

Summary Building a data platform is an iterative and evolutionary process that requires collaboration with internal stakeholders to ensure that their needs are being met. Yotpo has been on a journey to evolve and scale their data platform to continue serving the needs of their organization as it increases the scale and sophistication of data usage.

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Data Engineering Podcast

DECEMBER 4, 2021

By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more.

BI 100

BI Data Warehouse Data Pipeline Designing 100

ProjectPro

FEBRUARY 18, 2022

This data warehouse is accessible to data analysts and scientists and helps them perform data science tasks like data visualization , statistical analysis, machine learning model creation, etc. The big data analytics market is expected to grow at a CAGR of 13.2 percent, reaching USD 549.73 billion in 2028.

Project 52

Project AWS Kafka Healthcare 52

Snowflake

MARCH 30, 2023

A 2016 data science report from data enrichment platform CrowdFlower found that data scientists spend around 80% of their time in data preparation (collecting, cleaning, and organizing of data) before they can even begin to build machine learning (ML) models to deliver business value. ML workflow, ubr.to/3EJHjvm

Engineering 83

Engineering Raw Data Data Science Scala 83

Data Engineering Podcast

MARCH 22, 2021

It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Summary The world of business is becoming increasingly dependent on information that is accurate up to the minute.

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Cloudera

JUNE 2, 2021

As critical elements in supplying trusted, curated, and usable data for end-to-end analytic and machine learning workflows, the role of data pipelines is becoming indispensable. Data pipelines are in high demand in today’s data-driven organizations. To keep up, data pipelines are being vigorously reshaped with modern tools and techniques.

Data Pipeline 91

Data Pipeline Data Warehouse Machine Learning Data Architect 91

Knowledge Hut

DECEMBER 29, 2023

Then the data engineers study this data to learn about different patterns that can help curate effective business strategies. Snowflake This tool is the best option for companies who wish to switch to a cloud-based solution. The highlight features of Apache Hive are workload management and better security.

Data Engineering 40

Data Engineering Data Engineer Engineering BI 40

Knowledge Hut

JUNE 20, 2023

A novice data scientist prepared to start a rewarding journey may need clarification on the differences between a data scientist and a machine learning engineer. Many people are learning data science for the first time and need help comprehending the two job positions. In addition, they want to know their daily duties and obligations.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Monte Carlo

JUNE 14, 2022

Snowflake Summit 2022 has been different – and I mean this in the best way possible. Without context, you wouldn’t be sure if Snowflake CEO Frank Slootman or Sting (he is in his Las Vegas residency after all) was about to take the stage. Whatever the reason, the conference started like someone shot a starter’s pistol.

Cloud 52

Cloud Data Ingestion Government Python 52

Data Engineering Podcast

OCTOBER 21, 2019

And for your machine learning workloads, they just announced dedicated CPU instances. You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.

Cloud 100

Cloud Data Lake Hadoop Programming Language 100

ProjectPro

AUGUST 24, 2021

Data professionals who work with raw data like data engineers, data analysts, machine learning scientists , and machine learning engineers also play a crucial role in any data science project. Most of us have observed that data scientist is usually labeled the hottest job of the 21st century, but is it the only most desirable job?

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

ProjectPro

NOVEMBER 30, 2021

When working on real-time business problems, data scientists build models using various Machine Learning or Deep Learning algorithms. This guide provides definitions, a step-by-step tutorial, and a few best practices to help you understand ETL pipelines and how they differ from data pipelines. Can the source be single or multiple?

Process 52

Process Data Pipeline Data Warehouse AWS 52

Data Engineering Podcast

MAY 19, 2019

And for your machine learning workloads, they just announced dedicated CPU instances. You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.

Data Analytics 100

Data Analytics Building Engineering Data Warehouse 100

AltexSoft

JULY 29, 2022

What does it take to store all New York Times articles published between 1855 and 1922? Depending on how you measure it, the answer will be 11 million newspaper pages or… just one Hadoop cluster and one tech specialist who can move 4 terabytes of textual data to a new location in 24 hours. The Hadoop toy. Source: The Wall Street Journal.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

Monte Carlo

JANUARY 23, 2024

The speed of ETL and analytics has increased Maxime recalls a time, not too long ago, when data engineering would require running Hive jobs for hours at a time, frequent context switching between jobs and managing different elements of your data pipeline. Enter, the data engineer. What is a data engineer today? What do data engineers do?

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Data Engineering Weekly

JULY 23, 2023

I'm happy to see the wide variety of talk proposals from people management, Gen AI, Machine Learning, modern data stack, and more!!! Write SQL queries without learning SQL? See how it works today. I tried my best to write last week, but the travel took its toll. While on travel Aswin & I were busy preparing for DewCon.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Data Engineering Weekly

NOVEMBER 6, 2022

link] Meta: Improving Instagram notification management with machine learning and causal inference Meta writes about a similar system of improving notification with ML. Sign up free to test out the tool today. Editor’s Note: #TwitterMigration Mastodon & the definition of Data Contract Last week I switched from Twitter to Mastodon.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

ProjectPro

MAY 31, 2021

A big data project is a data analysis project that uses machine learning algorithms and different data analytics techniques on a large dataset for several purposes, including predictive modeling and other advanced analytics applications. Ace your big data interview by adding some unique and exciting Big Data projects to your portfolio.

Big Data 52

Big Data Coding Project Hadoop 52

ProjectPro

JANUARY 19, 2022

Explore this page further and learn everything about data engineers to find the answer. Furthermore, we will also lay out a learning path on how to become a data engineer that will help one explore this exciting domain. What is the precise reason behind it? That needs to be done because raw data is painful to read and work with.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Databand.ai

AUGUST 30, 2023

Learn more in our detailed guide to DataOps architecture (coming soon) Key Features of DataOps Tools DataOps tools come with a variety of features designed to optimize and streamline data processes. These tools provide a common platform for these teams to work together, share insights, and make collective decisions.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

AltexSoft

MARCH 14, 2023

A modern data stack can help the company manage and analyze this data effectively by using cloud-based data warehouses like Snowflake , data integration tools like Stitch, and data visualization software like Power BI. This development has paved the way for a suite of cloud-native data tools that are user-friendly, scalable, and affordable.

IT 59

IT Data Warehouse Data Governance Data Lake 59

AltexSoft

JULY 18, 2023

This article will expose Apache Spark architecture, assess its advantages and disadvantages, compare it with other big data technologies, and provide you with the path to learning this impactful instrument. To some, the word Apache may bring images of Native American tribes celebrated for their tenacity and adaptability. What is Apache Spark?

Big Data 52

Big Data Data Process Process Hadoop 52

Cloudera

DECEMBER 12, 2018

Greg Rahn: Toward the end of that eight-year stint, I saw this thing coming up called Hadoop and an engine called Hive. Hi Greg, thank you for joining us today. I would like to start off by asking you to tell us about your background and what kicked off your 20-year career in relational database technology? Michael Moreno: Nice!

Data Warehouse 42

Data Warehouse Relational Database Hadoop BI 42

Big Data Tools

JUNE 8, 2022

News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. It’s the start of June. That means it’s time to start taking summer vacations and enjoying some fresh juice alongside your fresh news! Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52