Data Engineering Podcast

AUGUST 13, 2022

Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Christophe Blefari

APRIL 8, 2023

I still firmly believe that this is not the role of a data engineer. A data engineer should still be a software engineer working with data, empowering others with tooling and apps. Data modeling should not be a required data engineer skill. Enters the analytics engineer.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

I still firmly believe that this is not the role of a data engineer. A data engineer should still be a software engineer working with data, empowering others with tooling and apps. Data modeling should not be a required data engineer skill. Enters the analytics engineer.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Grouparoo

OCTOBER 6, 2021

It was a fun experience and I think we made a good choice by picking 97 Things Every Data Engineer Should Know. This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

DataKitchen

FEBRUARY 21, 2023

On 24 January 2023, Gartner released the article “ 5 Ways to Enhance Your Data Engineering Practices.” Data team morale is consistent with DataKitchen’s own research. We surveyed 600 data engineers , including 100 managers, to understand how they are faring and feeling about the work that they are doing.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services.

Architecture 130

Architecture Technology Google Cloud Metadata 130

Databand.ai

JULY 19, 2023

Learn more in our detailed guide to data lineage visualization (coming soon) Integration with Multiple Data Sources Data lineage tools are designed to integrate with a wide range of data sources, including databases, data warehouses, and cloud-based data platforms.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

DataKitchen

NOVEMBER 4, 2021

It often takes months to progress from a data lake to the final delivery of insights. One data engineer called it the “last mile problem.” . In our many conversations about data analytics, data engineers, analysts and scientists have verbalized the difficulty of creating analytics in the modern enterprise.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Knowledge Hut

JULY 3, 2023

This process consists of feeding data from various sources and building it to be available for analysis, storage, or next processing. To achieve this goal, pursuing Data Engineer certification can be highly beneficial. What is Real-Time Data Ingestion? Why is Real-Time Data Ingestion Important?

Data Ingestion 52

Data Ingestion Pipeline-centric Google Cloud Media 52

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is Data Science? What are the roles and responsibilities of a Data Engineer? And many more.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

ProjectPro

DECEMBER 16, 2021

They need to know everything about the data and apply various mathematical and statistical tools to identify the most significant features using feature selection, feature engineering , feature transformation, etc. On the other hand, Data Scientists work in coordination with data analysts and data engineers.

Machine Learning 52

Machine Learning Engineering Pipeline-centric Database-centric 52

ProjectPro

JUNE 1, 2017

Datos IO has extended its on-premise and public cloud data protection to RDBMS and Hadoop distributions. Its RecoverX distributed database backup product of latest version v2.0 Cloudera is more inclined on becoming a product centric business with 23% of its revenue coming from services past year in comparison to 31% for Hortonworks.

Hadoop 52

Hadoop Medical Pipeline-centric Database-centric 52

Data Engineering Podcast

APRIL 7, 2019

Summary How much time do you spend maintaining your data pipeline? This was a fascinating conversation with someone who has spent his entire career working on simplifying complex data problems. Managing and auditing access to your servers and databases is a problem that grows in difficulty alongside the growth of your teams.

Data Pipeline 100

Data Pipeline Pipeline-centric Database-centric AWS 100

Cloudera

SEPTEMBER 17, 2020

With growing disparate data across everything from edge devices to individual lines of business needing to be consolidated, curated, and delivered for downstream consumption, it’s no wonder that data engineering has become the most in-demand role across businesses — growing at an estimated rate of 50% year over year.

Data Pipeline 104

Data Pipeline Data Engineering Data Engineer Engineering 104

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Ascend.io

AUGUST 29, 2023

A star-studded baseball team is analogous to an optimized “end-to-end data pipeline” — both require strategy, precision, and skill to achieve success. Just as every play and position in baseball is key to a win, each component of a data pipeline is integral to effective data management.

Data Pipeline 52

Data Pipeline Pipeline-centric Database-centric Data Ingestion 52

DareData

JULY 29, 2020

One paper suggests that there is a need for a re-orientation of the healthcare industry to be more "patient-centric". Furthermore, clean and accessible data, along with data driven automations, can assist medical professionals in taking this patient-centric approach by freeing them from some time-consuming processes.

Healthcare 52

Healthcare Data Pipeline Medical Pipeline-centric 52

Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Dataquest

JANUARY 25, 2017

Data scientists have become extremely sought after , and for good reason — a skilled data scientist can add incredible value to a business. But what about data engineers? A data scientist is only as good as the data they have access to. A data engineer transforms data into a useful format for analysis.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Knowledge Hut

SEPTEMBER 26, 2023

It offers a wide range of services, including computing, storage, databases, machine learning, and analytics, making it a versatile choice for businesses looking to harness the power of the cloud. This cloud-centric approach ensures scalability, flexibility, and cost-efficiency for your data workloads.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Monte Carlo

JULY 28, 2022

But perhaps one of the most common reasons for data quality challenges are software feature updates and other changes made upstream by software engineers. These are particularly frustrating, because while they are breaking data pipelines constantly, it’s not their fault. He suggested : “Private vs. public methods.

Pipeline-centric 52

Pipeline-centric Software Engineer Software Engineering Database-centric 52

Monte Carlo

JANUARY 23, 2024

, Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Today’s LLMs can already process enormous amounts of unstructured data, automating much of the monotonous work of data science. That implies working with new patterns like vector databases.” RAG workflow.

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

AltexSoft

JULY 18, 2023

Its flexibility allows it to operate on single-node machines and large clusters, serving as a multi-language platform for executing data engineering , data science , and machine learning tasks. Before diving into the world of Spark, we suggest you get acquainted with data engineering in general.

Big Data 52

Big Data Data Process Process Hadoop 52

RandomTrees

NOVEMBER 29, 2023

The choice of tools and adopting the right methodologies becomes the drive for any business and in this the selection on the database technologies is very crucial. Having said this, the need to understand all the primary components of a data warehouse and then designing the data vault components is described.

Data Warehouse 52

Data Warehouse Data Lake Database-centric Data Cleanse 52

phData: Data Engineering

JULY 18, 2023

Slow Response to New Information: Legacy data systems often lack the computation power necessary to run efficiently and can be cost-inefficient to scale. This typically results in long-running ETL pipelines that cause decisions to be made on stale or old data.

Data Warehouse 52

Data Warehouse Pipeline-centric Government Data 52