Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

IT 100

IT Metadata MongoDB MySQL 100

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. Let us first get a clear understanding of why Data Science is important. What is the need for Data Science?

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Knowledge Hut

SEPTEMBER 27, 2023

In the current data-driven environment, starting a business intelligence (BI) journey has become crucial. Come explore with me the special features that set Power BI apart from other business intelligence tools in the current day. Power BI is a robust business intelligence (BI) platform developed by Microsoft.

BI 52

BI Business Intelligence Certification Data Security 52

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Knowledge Hut

DECEMBER 28, 2023

This certification, often referred to as the Azure Data Engineer Associate certification, validates the competency of individuals in implementing Azure data solutions. It’s a testament to their ability to create scalable, efficient and secure data pipelines. What is the Azure Data Engineer Certification?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Meltano

FEBRUARY 22, 2023

Testing a data application is similar to testing any software application in many ways, just with a strong focus on testing data-related issues. But testing problems like failing data workflows, mismatches in data reconciliation after ETL, and data quality issues means that you’re not only testing the code but also the data itself.

Data 52

Data Transportation Data Workflow Data Lake 52

Monte Carlo

MARCH 24, 2023

Data migration refers to the movement of data from one location (warehouse or database), format, or SaaS application, to another. Data migrations are also highly visible projects with severe consequences to both the organization and project manager for failed attempts. In reality, it’s never quite that simple.

Data Warehouse 52

Data Warehouse AWS Cloud Data Governance 52

AltexSoft

MARCH 14, 2023

These sources commonly include databases, SaaS products, and event streams. Databases store key information that powers a company’s product, such as user data and product data. The ones that keep only relational data in a tabular format are called SQL or relational database management systems (RDBMSs).

IT 59

IT Data Warehouse Data Governance Data Lake 59

Monte Carlo

APRIL 24, 2023

While we won’t get into the minutia of every consideration for every level of the data stack, it’s important to recall these five considerations as they’ll nonetheless steer the direction of our conversation. At Uber, for example, data ingestion scale quickly surpassed what out-of-the-box solutions could support.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Monte Carlo

JUNE 14, 2023

5 Data pipeline architecture designs and their evolution The Hadoop era , roughly 2011 to 2017, arguably ushered in big data processing capabilities to mainstream organizations. Data then, and even today for some organizations, was primarily hosted in on-premises databases with non-scalable storage.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

AltexSoft

SEPTEMBER 21, 2023

This endeavor would unwittingly plant the seeds for Elasticsearch, a technology that today drives data search and analytics for businesses around the globe. First publicly introduced in 2010, Elasticsearch is an advanced, open-source search and analytics engine that also functions as a NoSQL database.

Engineering 52

Engineering NoSQL Programming Language Java 52

phData: Data Engineering

MAY 11, 2023

Data literacy is increasing fast, and team members capable of administrating a business unit’s transactional system can also perform the cloud ETL of that system data to a Snowflake database via Matillion. In addition, breaking the human and data workflows into modular components enables this simplification.

Coding 52

Coding Software Engineer Software Engineering Government 52

phData: Data Engineering

JANUARY 3, 2022

This commonly introduces: Database or Data Warehouse API/EDI Integrations ETL software Business intelligence tooling By leveraging off-the-shelf tooling, your company separates disciplines by technology. Databases, schemas, tables, view, etc.

IT 52

IT AWS Software Engineer Software Engineering 52