Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. 🍒 We'll send exclusive The Data Stack Show swag just for participating!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Knowledge Hut

FEBRUARY 27, 2023

A data engineer is a key member of an enterprise data analytics team and is responsible for handling, leading, optimizing, evaluating, and monitoring the acquisition, storage, and distribution of data across the enterprise. Data Engineers indulge in the whole data process, from data management to analysis.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

Christophe Blefari

APRIL 8, 2023

Data News entering in town ( credits ) Hey you, if I wasn't late in my newsletter writing it wouldn't be me. But here is your usual Data News. Data modeling Dear readers, I have to confess something. I did not care about data modeling for years. Data modeling should not be a required data engineer skill.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

Data News entering in town ( credits ) Hey you, if I wasn't late in my newsletter writing it wouldn't be me. But here is your usual Data News. Data modeling Dear readers, I have to confess something. I did not care about data modeling for years. Data modeling should not be a required data engineer skill.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Grouparoo

OCTOBER 6, 2021

It was a fun experience and I think we made a good choice by picking 97 Things Every Data Engineer Should Know. This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Podcast

AUGUST 13, 2022

Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

DataKitchen

FEBRUARY 21, 2023

On 24 January 2023, Gartner released the article “ 5 Ways to Enhance Your Data Engineering Practices.” Data team morale is consistent with DataKitchen’s own research. We surveyed 600 data engineers , including 100 managers, to understand how they are faring and feeling about the work that they are doing.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Cloudera

DECEMBER 13, 2023

Implementing a cohesive data vision LGIM’s Global Chief Operating Office is responsible for the company’s technology, data, client servicing and the management of its strategic operating agenda. LGIM’s data ecosystem had become fragmented over time as it had grown.

Pipeline-centric 89

Pipeline-centric Professional Services Datasets BI 89

Ascend.io

OCTOBER 20, 2023

Snowflake has swiftly risen to prominence in the data world, and its consumption-based pricing model is well-known among data professionals. The Nuances of Snowflake Costing Snowflake’s pricing strategy is an exemplification of its user-centric approach: pay for what you use. So Why Do Snowflake Costs Become Prohibitive?

Pipeline-centric 52

Pipeline-centric IT Data Pipeline Bytes 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? How can we interoperate between the data domains ?

Architecture 130

Architecture Technology Google Cloud Metadata 130

Pipeline Data Engineering

MARCH 2, 2022

Data engineering salon. News and interesting reads about the world of data. The Unbundling of Airflow Gorkem Yurtseven, Co-Founder, Features and Labels A diverse set of tools is unbundling Airflow and this diversity is causing substantial fragmentation in modern data stack. What the hell happened?

Pipeline-centric 52

Pipeline-centric BI Consulting Data Warehouse 52

Ascend.io

JUNE 21, 2023

Creating business value from the onslaught of data can feel like captaining a high-tech vessel through uncharted waters. Data teams across business areas are cranking out data sets in response to impatient business requests, while simultaneously trying to build disparate DataOps processes from scratch.

Pipeline-centric 52

Pipeline-centric Data Pipeline Metadata Data 52

Snowflake

JUNE 28, 2023

At Snowflake, we’re helping data scientists, data engineers, and application developers build faster and more efficiently in the Data Cloud. To make it even easier to process data with Snowpark Python UDFs and Stored Procedures, we have added support for Python 3.9 R) and hardware (e.g.

Pipeline-centric 57

Pipeline-centric Programming Language Python Government 57

Databand.ai

JULY 19, 2023

Data Lineage Tools: Key Capabilities and 5 Notable Solutions Ryan Yackel July 19, 2023 What Are Data Lineage Tools? Data lineage tools provide a visual representation of your data’s journey across multiple systems and transformations. Data lineage tools are not a new concept.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

DataKitchen

NOVEMBER 4, 2021

Data organizations often have a mix of centralized and decentralized activity. DataOps concerns itself with the complex flow of data across teams, data centers and organizational boundaries. It expands beyond tools and data architecture and views the data organization from the perspective of its processes and workflows.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Monte Carlo

AUGUST 2, 2022

Initial thoughts on our data team’s data mesh implementation plan and moving toward the four data mesh principles of domain data ownership, data as a product, self-service, and federated governance.

Pipeline-centric 52

Pipeline-centric Government Finance Data 52

Knowledge Hut

JULY 3, 2023

In today's fast-paced and data-driven world, users increasingly depend on real-time intuition to get an aggressive side and define a plan of action. This is where real-time data ingestion comes into the picture. Data is collected from various sources such as social media feeds, website interactions, log files and processing.

Data Ingestion 52

Data Ingestion Pipeline-centric Google Cloud Media 52

DataKitchen

AUGUST 27, 2021

DataOps has become an essential methodology in pharmaceutical enterprise data organizations, especially for commercial operations. Companies that implement it well derive significant competitive advantage from their superior ability to manage and create value from data.

Pharmaceutical 98

Pharmaceutical Pipeline-centric Data Lake Data Analytics 98

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is Data Science? What are the roles and responsibilities of a Data Engineer? What is Data Science?

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

ProjectPro

DECEMBER 16, 2021

Are you a newbie in the data science domain ready to embark on a rewarding journey but are confused between the roles of a Machine Learning Engineer vs Data Scientist? Data Science is an emerging discipline and so are the roles and job titles pretty much evolving.

Machine Learning 52

Machine Learning Engineering Pipeline-centric Database-centric 52

Cloudera

SEPTEMBER 17, 2020

For enterprise organizations, managing and operationalizing increasingly complex data across the business has presented a significant challenge for staying competitive in analytic and data science driven markets. CDP data lifecycle integration and SDX security and governance. Enterprise Data Engineering From the Ground Up.

Data Pipeline 103

Data Pipeline Data Engineering Data Engineer Engineering 103

Ascend.io

MAY 7, 2024

to bring its cutting-edge automation platform that revolutionizes modern data engineering. .” to bring its cutting-edge automation platform that revolutionizes modern data engineering. will strengthen clients’ data security postures by quickly identifying and mitigating potential risks.

Pipeline-centric 52

Pipeline-centric Data Cleanse Data Security Data Pipeline 52

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Data Engineering Podcast

APRIL 7, 2019

Summary How much time do you spend maintaining your data pipeline? This was a fascinating conversation with someone who has spent his entire career working on simplifying complex data problems. They have built an easy to use platform that lets you leverage your company’s single sign on for your data platform.

Data Pipeline 100

Data Pipeline Pipeline-centric Database-centric AWS 100

ProjectPro

JUNE 1, 2017

Datos IO has extended its on-premise and public cloud data protection to RDBMS and Hadoop distributions. RecoverX is described as app-centric and can back up applications data whilst being capable of recovering it at various granularity levels to enhance storage efficiency. now provides hadoop support. Zdnet.com, May 8, 2017.

Hadoop 52

Hadoop Medical Pipeline-centric Database-centric 52

Ascend.io

AUGUST 29, 2023

A star-studded baseball team is analogous to an optimized “end-to-end data pipeline” — both require strategy, precision, and skill to achieve success. Just as every play and position in baseball is key to a win, each component of a data pipeline is integral to effective data management.

Data Pipeline 52

Data Pipeline Pipeline-centric Database-centric Data Ingestion 52

DareData

JULY 29, 2020

The Challenges of Medical Data In recent times, there have been several developments in applications of machine learning to the medical industry. Odds are that your local hospital, pharmacy or medical institution's definition of being data-driven is keeping files in labelled file cabinets, as opposed to one single drawer.

Healthcare 52

Healthcare Data Pipeline Medical Pipeline-centric 52

Monte Carlo

MARCH 28, 2024

In recent years, we’ve seen all sorts of new job titles emerge that would have been inscrutable just a decade or two ago – cloud architect, data reliability engineer , data product manager , director of hybrid working, and yes, DataOps engineer. So what exactly IS a DataOps engineer? It depends!

Pipeline-centric 75

Pipeline-centric Engineering BI Google Cloud 75

Data Engineering Weekly

JANUARY 8, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Pipelines for data in motion can quickly turn into DAG hell.

Data Engineering 40

Data Engineering Data Engineer Engineering Pipeline-centric 40

Towards Data Science

MAY 19, 2023

Data Entropy — More Data, More Problems? How to navigate and embrace complexity in a modern data organisation. Business users are unable to find and access data assets critical to their workflows. Data engineers spend countless hours troubleshooting broken pipelines. The list goes on.

Pipeline-centric 105

Pipeline-centric Data Software Engineer Software Engineering 105

Data Engineering Podcast

MAY 7, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How do you manage permissions/auditability of updating or amending profile data? Let us know if you have opinions there!

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Ascend.io

JUNE 8, 2023

Because of its transformative potential, data has graduated from being merely a by-product of business operations to a critical asset in its own right. Is it possible to treat data not just as a necessary operational output, but as a product that holds immense strategic value? Without them, data products can’t exist.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Dataquest

JANUARY 25, 2017

From helping cars drive themselves to helping Facebook tag you in photos , data science has attracted a lot of buzz recently. Data scientists have become extremely sought after , and for good reason — a skilled data scientist can add incredible value to a business. But what about data engineers?

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Snowflake

JUNE 28, 2023

Since its launch two years ago, Snowpark has been empowering data scientists, data engineers, and application developers to streamline their architectures, accelerate development, and boost the performance of data engineering and ML/AI workloads on Snowflake.

Python 52

Python Accessible Accessibility Pipeline-centric 52

Monte Carlo

NOVEMBER 14, 2023

Imagine, if you will, a world where data just… flows. This is the world that data orchestration tools aim to create. Data orchestration tools minimize manual intervention by automating the movement of data within data pipelines. So, what makes a good data orchestration tool? No hiccups.

Pipeline-centric 40

Pipeline-centric Google Cloud Data Workflow Python 40

Monte Carlo

JULY 28, 2022

There are virtually an unlimited number of ways data can break. But perhaps one of the most common reasons for data quality challenges are software feature updates and other changes made upstream by software engineers. In fact, most of the time they are unaware of these data quality challenges.

Pipeline-centric 52

Pipeline-centric Software Engineer Software Engineering Database-centric 52