Data Engineering Weekly

MARCH 10, 2024

Big thanks to our insightful speakers, Hareshkumar Selvakumar - Talks about his work on Data Products for PayPal. Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses. Data engineers build the systems that store and process sensitive information.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Science Blog: Data Engineering

DECEMBER 23, 2022

Warum Data Engineering der Data Science in Bedeutung und Berufschancen längst die Show stiehlt, dabei selbst ebenso einem stetigen Wandel unterliegt. Was ein Data Engineer wirklich können muss Der Data Scientist als sexiest Job des 21. Data Engineer, der gefragteste Job des 21sten Jahrhunderts?

Data Engineering 52

Data Engineering Data Engineer Engineering Business Intelligence 52

Cloudera

JANUARY 15, 2021

Cloud data warehouses allow users to run analytic workloads with greater agility, better isolation and scale, and lower administrative overhead than ever before. The results demonstrate superior price performance of Cloudera Data Warehouse on the full set of 99 queries from the TPC-DS benchmark. Introduction.

Data Warehouse 92

Data Warehouse Cloud Consulting SQL 92

Cloudera

DECEMBER 11, 2020

In a previous blog post on CDW performance, we compared Azure HDInsight to CDW. In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to EMR 6.0 (also powered by Apache Hive-LLAP) on Amazon using the TPC-DS 2.9 More on this later in the blog.

Data Warehouse 107

Data Warehouse Metadata Datasets Data 107

Data Engineering Weekly

NOVEMBER 26, 2023

It’s the end of the year, and there will be a lot of buzz about what the next five years in data engineering might bring. We saw a fleet of announcements from Data Catalogs tools on how LLM can help to auto-generate documentation [See: How Generative AI Is Making Data Catalogs Smarter ]. But I had my doubts.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Data Science Blog: Data Engineering

MAY 20, 2024

Continuous Integration and Continuous Delivery (CI/CD) for Data Pipelines: It is a Game-Changer with AnalyticsCreator! The need for efficient and reliable data pipelines is paramount in data science and data engineering. It offers full BI-Stack Automation, from source to data warehouse through to frontend.

Data Pipeline 52

Data Pipeline Data Lake BI Data Warehouse 52

Data Engineering Weekly

NOVEMBER 5, 2023

The blog is a classic case study for data engineers who like to build SQL-like abstractions. link] BlaBlaCar: Controlling our Data Platform Costs at BlaBlaCar Cloud data warehouses bring flexibility and user-friendliness with additional infrastructure costs.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Weekly

AUGUST 20, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. But 4 years later, in 2023 — where has the data mesh gotten us?

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability. To measure, but not track.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Data Engineering Weekly

FEBRUARY 19, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. Redshift is no longer a true competitor in the warehouse space.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. So, let's shape the future of Data Engineering together.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Weekly

DECEMBER 21, 2022

The Data world Before Hadoop Era We must walk through memory lane to understand why functional data engineering is critical. Let’s reference what the data world looked like before the Hadoop era. Any blog is incomplete if it does not include a Gartner prediction, so let’s start with one. Why is it so?

Data Engineering 57

Data Engineering Data Engineer Engineering Hadoop 57

Data Engineering Weekly

FEBRUARY 26, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Perhaps merge Data LakeHouse & Data Warehouse in the next edition?

Data Engineering 57

Data Engineering Data Engineer Engineering Raw Data 57

Knowledge Hut

JUNE 30, 2023

The demand for experienced data engineers continuously expands in today's data-driven environment. Books on data engineering serve as essential resources to guide you through the vast terrain of data engineering. What is Data Engineering? Who are Data Engineers?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Weekly

APRIL 23, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. 🍒 We'll send exclusive The Data Stack Show swag just for participating! .

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? Data science as a discipline was going through its adolescence of self-affirming and defining itself. We were pioneers.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Uber Engineering

AUGUST 14, 2019

Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog.

Data Warehouse 99

Data Warehouse Designing Engineering Utilities 99

ProjectPro

FEBRUARY 16, 2023

The demand for skilled data engineers who can build, maintain, and optimize large data infrastructures does not seem to slow down any sooner. At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data. use SQL, compared to 61.7%

Data Engineering 52

Data Engineering Data Engineer SQL Engineering 52

Data Engineering Weekly

MARCH 5, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Editor’s Note: Who Owns the Data Contract?

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Knowledge Hut

SEPTEMBER 26, 2023

The demand for knowledgeable data engineers that can plan, create, and maintain sophisticated data infrastructure is growing as the amount of data created by enterprises continues to increase dramatically. The success of our career as an Azure Data Engineer depends on our ability to master several different talents.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Weekly

OCTOBER 30, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Databand.ai

DECEMBER 13, 2022

The Top 25 Data Engineering Influencers and Content Creators on LinkedIn Ryan Yackel 2022-12-13 10:23:19 Interested in data engineering? LinkedIn is full of influencers sharing new ideas and sparking conversations on all kinds of topics, and data engineering is no exception. You’ve come to the right place.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Monte Carlo

DECEMBER 27, 2022

It definitely holds true for our Data Downtime blog with these five articles driving a majority of our traffic in 2022. Table of Contents What is a Data Mesh — and How Not to Mesh it Up What is Data Observability? The 5 Pillars You Need To Know Is the Modern Data Warehouse Broken?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

NOVEMBER 26, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Knowledge Hut

DECEMBER 28, 2023

Its comprehensive suite of services can handle data at scale. It’s no surprise that the demand for certified Azure data engineers has skyrocketed. Today, Azure Data Engineer certification is an invaluable asset for those looking to excel in the field of data engineering. Who is an Azure Data Engineer?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! What are Data Engineering Tools?

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

FreshBI

JANUARY 31, 2022

The objective of this blog Building reliable intelligence at the speed of business can be a challenging task. A well-designed data engineering strategy ensures that your analytics resources are spent on uncovering insights rather than laying foundations.

BI 52

BI Data Engineering Data Engineer Engineering 52

Big Data Tools

NOVEMBER 9, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

NOVEMBER 9, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Knowledge Hut

FEBRUARY 16, 2023

Demand for data engineers is at a peak today globally due to the massive amount of data that companies accumulate and work with this data to draw actionable insights and make better business decisions. To do that, the generated data must be interpreted to be understood by the end user. What Does a Data Engineer Do?

Data Engineering 52

Data Engineering Data Engineer Engineering Healthcare 52

Big Data Tools

OCTOBER 10, 2022

I am now delighted to have the privilege of returning to the task of collecting for you the most exciting news from the world of data engineering. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. Why should data engineers care about this?

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

OCTOBER 10, 2022

I am now delighted to have the privilege of returning to the task of collecting for you the most exciting news from the world of data engineering. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. Why should data engineers care about this?

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Weekly

NOVEMBER 6, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Martin kindly stepped in for me to give the update for my promised blog posts.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Cloudera

AUGUST 20, 2020

This blog illustrates how Cloudera Data Engineering (CDE), using Apache Spark , can be used to produce reports based on the PPP data while addressing each of the challenges outlined above. A mock scenario for the Texas Legislative Budget Board (LBB) is set up below to help a data engineer manage and analyze the PPP data.

Programming 62

Programming Data Engineering Data Engineer Engineering 62

Analytics Vidhya

FEBRUARY 6, 2023

Introduction The demand for data to feed machine learning models, data science research, and time-sensitive insights is higher than ever thus, processing the data becomes complex. To make these processes efficient, data pipelines are necessary. appeared first on Analytics Vidhya.

Amazon Web Services 218

Amazon Web Services Data Pipeline Machine Learning Data Science 218

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. This is something I struggle with, I really like writing, I really like this newsletter, I really like the blog, but it takes me one day per week to be done. Often they are right.

Kafka 130

Kafka Data Lake Data Storage Data 130