Knowledge Hut

SEPTEMBER 26, 2023

The demand for knowledgeable data engineers that can plan, create, and maintain sophisticated data infrastructure is growing as the amount of data created by enterprises continues to increase dramatically. The success of our career as an Azure Data Engineer depends on our ability to master several different talents.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

JUNE 12, 2022

Summary Unstructured data takes many forms in an organization. From a data engineering perspective that often means things like JSON files, audio or video recordings, images, etc. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Knowledge Hut

OCTOBER 3, 2023

Within the Microsoft Azure ecosystem, the role of an Azure data engineer stands out as one of the most sought-after positions. What Does an Azure Data Engineer Do? Azure Data engineers collaborate with Azure AI services built on top of Azure Cognitive Services APIs to offer end customers a variety of pre-built models.

Data Engineering 52

Data Engineering Data Engineer Engineering Cloud Computing 52

Knowledge Hut

DECEMBER 28, 2023

As technology evolves, cloud platforms have emerged as the cornerstone of modern data management. Its comprehensive suite of services can handle data at scale. It’s no surprise that the demand for certified Azure data engineers has skyrocketed. Who is an Azure Data Engineer?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering 52

Data Engineering Data Engineer Engineering Programming Language 52

Data Engineering Podcast

OCTOBER 2, 2022

Alvin is a platform that aims to provide a low effort solution for data lineage capabilities focused on simplifying the work of data engineers. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

IT 100

IT Food PostgreSQL MongoDB 100

Knowledge Hut

FEBRUARY 27, 2023

A data engineer is a key member of an enterprise data analytics team and is responsible for handling, leading, optimizing, evaluating, and monitoring the acquisition, storage, and distribution of data across the enterprise. Data Engineers indulge in the whole data process, from data management to analysis.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

Knowledge Hut

DECEMBER 26, 2023

Some of the reasons why this book is ideal for beginner-level students are listed below: It covers topics that are fundamental in the field of data science The language is easy to comprehend You will learn the basics of statistics in data science Important topics like distribution, randomization, sampling, and the like are covered in depth.

Data Science 52

Data Science Scala Programming Language R (Programming) 52

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares his journey from building a consumer product to launching a data pipeline service and how his frustrations as a product owner have informed his work at Hevo Data. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

OCTOBER 30, 2022

Summary One of the most impactful technologies for data analytics in recent years has been dbt. It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts.

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database 100

Database MySQL Cloud MongoDB 100

ProjectPro

FEBRUARY 21, 2023

With so many data engineering certifications available , choosing the right one can be a daunting task. There are over 133K data engineer job openings in the US, but how will you stand out in such a crowded job market? The answer is- by earning professional data engineering certifications! AWS or Azure?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Knowledge Hut

SEPTEMBER 29, 2023

This growth is creating a strong demand for data experts, especially Azure data engineers. But who are Azure data engineers, and what do they do? Moreover, what benefits can you expect from a career in Azure Data Engineering? Why Should You Get an Azure Data Engineer Certification?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

OCTOBER 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Mark Van de Wiel explains how they integrated CDC functionality into their existing product, discusses the nuances of different approaches to change data capture from various sources. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

He discusses the challenges of making data collection purposeful and efficient without distracting employees from their primary duties and how business owners can use the provided analytics to support their staff in their duties. Data teams are increasingly under pressure to deliver. Closing Announcements Thank you for listening!

Hospitality 100

Hospitality Food MongoDB Scala 100

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Podcast

AUGUST 13, 2022

In this episode Shinji Kim discusses the challenges of data discovery and how to collect and preserve additional context about each piece of information so that you can find what you need when you don’t even know what you’re looking for yet. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Metadata 100

Metadata MongoDB Scala MySQL 100

Data Engineering Podcast

JULY 3, 2022

Summary The perennial challenge of data engineers is ensuring that information is integrated reliably. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. Data teams are increasingly under pressure to deliver.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

AUGUST 21, 2022

In this episode Shruti Bhat gives her view on the state of the ecosystem for real-time data and the work that she and her team at Rockset is doing to make it easier for engineers to build those experiences. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.

Lambda Architecture 100

Lambda Architecture MongoDB Scala MySQL 100

ProjectPro

FEBRUARY 16, 2023

ETL is a critical component of success for most data engineering teams, and with teams harnessing it with the power of AWS, the stakes are higher than ever. Data Engineers and Data Scientists require efficient methods for managing large databases, which is why centralized data warehouses are in high demand.

AWS 52

AWS Data Management ETL Tools Management 52

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 100

Building MongoDB Scala MySQL 100

Data Engineering Podcast

AUGUST 6, 2022

In this episode Frank Liu explains how the open source Milvus vector database is implemented to speed up machine learning development cycles, how to think about proper storage and scaling of these vectors, and how data engineering and machine learning teams can collaborate on the creation and maintenance of these data sets.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Cloud Academy

JANUARY 27, 2022

Data engineering is the process of designing and implementing solutions to collect, store, and analyze large amounts of data. The data then gets prepared in formats to be used by people such as business analysts, data analysts, and data scientists. What does a data engineer do?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

AUGUST 13, 2022

Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. Data teams are increasingly under pressure to deliver.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode Gopal Erinjippurath discusses the data engineering challenges of building and serving those data sets, and how they are distilling complex climate information into consumable facts so you don’t have to be an expert to understand it. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

Ascend.io

SEPTEMBER 5, 2023

In the fast-evolving landscape of cloud data solutions, Snowflake has consistently been at the forefront of innovation, offering enterprises sophisticated tools to optimize their data management. Snowpark is a library equipped with an API that developers can use for querying and processing data within the Snowflake Data Cloud.

IT 59

IT Scala Java Programming Language 59

Ascend.io

OCTOBER 4, 2023

Snowpark stands out as a game-changer for data engineers. It empowers them to tap into the familiar terrain of languages like Scala, Java, and Python, but with the unique advantage of not having to move data out of Snowflake. This entails loading data from particular tables into Snowpark DataFrames.

Scala 52

Scala Data Ingestion Python Java 52

Data Engineering Podcast

OCTOBER 28, 2018

In this episode he explains the rationale for the effort, the challenges that it has posed, the development that has been done to make it work, and the benefits that it provides to the Netflix data platform teams. Can you start by outlining the motivation for choosing Jupyter notebooks as the core interface for your data teams?

Scala 100

Scala Python Data Engineering Data Engineer 100

Knowledge Hut

DECEMBER 26, 2023

Data Science is a huge umbrella with a plethora of roles available in the field such as a Data Scientist, Data Engineer, BI Developer, Data and Analytics Manager, etc. You can execute this by learning data science with python and working on real projects.

Data Science 97

Data Science BI Business Intelligence Data Mining 97

Data Engineering Podcast

JULY 27, 2020

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management What are the pieces of advice that you wish you had received early in your career of data engineering? If you hand a book to a new data engineer, what wisdom would you add to it?

Systems 100

Systems Building Scala Java 100

Data Engineering Podcast

APRIL 29, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. The current goal for most companies is to be “data driven” How would you define that concept?

Business Intelligence 100

Business Intelligence Scala Hadoop Machine Learning 100

AltexSoft

FEBRUARY 11, 2023

Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. Bad data management be like, Source: Makeameme Data architects are sometimes confused with other roles inside the data science team. Feel free to enjoy it.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Data Engineering Podcast

NOVEMBER 18, 2019

Summary With the constant evolution of technology for data management it can seem impossible to make an informed decision about whether to build a data warehouse, or a data lake, or just leave your data wherever it currently rests. And don’t forget to thank them for their continued support of this show!

Data Lake 100

Data Lake Scala Data Warehouse Hadoop 100