7 Steps to Mastering Data Engineering
KDnuggets
APRIL 12, 2024
The only data engineering roadmap you need for an introduction to concepts, tools, and techniques to collect, store, transform, analyze, and model data.
Data Engineer Related Topics 
What Data Engineers Really Do?
Analytics Vidhya
JUNE 25, 2023
In a data-driven world, behind-the-scenes heroes like data engineers play a crucial role in ensuring smooth data flow. A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines.
Data Engineering 
220
220 Join 16,000+
Insiders
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Top 20 Data Engineering Project Ideas [With Source Code]
Analytics Vidhya
SEPTEMBER 20, 2023
Data engineering plays a pivotal role in the vast data ecosystem by collecting, transforming, and delivering data essential for analytics, reporting, and machine learning. Aspiring data engineers often seek real-world projects to gain hands-on experience and showcase their expertise.
Data Engineering 268
268 
How To Hire Junior Data Engineers
Seattle Data Guy
JANUARY 17, 2023
With all the recent data events I have put together I inevitably run into new data engineers who are either finishing up college or looking to transition into a data engineer or data scientist position. In fact I have talked to several newly graduated engineers who are struggling to find work.
Data Engineering 223
223 
Mr. Pavan’s Data Engineering Journey Drives Business Success
Analytics Vidhya
JUNE 24, 2023
He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Thus, providing valuable insights into the field of data engineering. Introduction We had an amazing opportunity to learn from Mr. Pavan.
Data Engineering 204
204 
Data Engineering for Streaming Data on GCP
Analytics Vidhya
APRIL 3, 2023
Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers. Nevertheless, setting up a streaming data pipeline to power such dashboards may […] The post Data Engineering for Streaming Data on GCP appeared first on Analytics Vidhya.
Data Engineering 251
251 
Modern Data Engineering with MAGE: Empowering Efficient Data Processing
Analytics Vidhya
JUNE 20, 2023
Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.
Data Process 217
217 
Data Engineer vs Data Scientist: Which Career to Choose?
Analytics Vidhya
JULY 25, 2023
In the world of data, two crucial roles play a significant part in unlocking the power of information: Data Scientists and Data Engineers. But what sets these wizards of data apart? Welcome to the ultimate showdown of Data Scientist vs Data Engineer! appeared first on Analytics Vidhya.
Data Engineering 200
200 
Most Essential 2023 Interview Questions on Data Engineering
Analytics Vidhya
FEBRUARY 7, 2023
Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.
Data Engineering 240
240 
Free Data Engineering Course for Beginners
KDnuggets
FEBRUARY 12, 2024
Interested in data engineering but don't know where to start? Get up to speed in data engineering fundamentals with this free course.
Data Engineering 122
122 
The State of Data Engineering at Data Day Texas 2024
Jesse Anderson
JANUARY 29, 2024
The premier of my latest talk covering The State of Data Engineering. This starts with data warehousing and goes into data science. I finish off by showing how data engineering can avoid the same fate as data warehousing and data science.
Data Engineering 130
130 
GPT and LLMs from a Data Engineering Perspective
Jesse Anderson
SEPTEMBER 14, 2023
There has been quite a bit of writing covering GPT and LLMs from data science and business perspectives. I haven’t seen much from the data engineering side. Let me share my perspective, having been in data and AI for a while and using LLMs before they became popular. How can we use LLMs in data engineering?
Data Engineering 147
147 
Top 20 Data Engineering Project Ideas with Source Code
Analytics Vidhya
SEPTEMBER 20, 2023
Data engineering plays a pivotal role in the vast data ecosystem by collecting, transforming, and delivering data essential for analytics, reporting, and machine learning. Aspiring data engineers often seek real-world projects to gain hands-on experience and showcase their expertise.
Data Engineering 130
130 
Ownership and Borrowing in Rust – Data Engineering Gold Mine.
Confessions of a Data Guy
FEBRUARY 7, 2023
Some of the things I’m going to talk about, well … all of it, is probably fairly obvious to most Rust folk, but it’s enjoyable to learn what new […] The post Ownership and Borrowing in Rust – Data Engineering Gold Mine. appeared first on Confessions of a Data Guy.
Data Engineering 221
221 
DuckDB vs Polars for Data Engineering.
Confessions of a Data Guy
APRIL 16, 2023
You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. I haven’t seen this since Databricks and Snowflake first came out and started throwing mud at each other.
Data Engineering 147
147 
Data Engineering Weekly #166
Data Engineering Weekly
APRIL 7, 2024
I strongly believe the concept of Data Product will play a bigger role in data engineering. link] Hussein Jundi: Data Engineering - Architectures & Strategies for Handling Sensitive Data The rapid adoption of AI brings challenges for data engineering to design systems to handling sensitive data.
What Is Apache Airflow – Data Engineering Consulting
Seattle Data Guy
FEBRUARY 11, 2023
Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?
Consulting 147
147 
Data Engineering Weekly #167
Data Engineering Weekly
APRIL 14, 2024
link] Arpit Choudhury: Growth needs to speak the language of Engineering and Data Software engineers and data engineers don’t speak the same language It reminds me of this famous quote The Author highlights that the growth team should be aware of the skill differences.
Brief History of Data Engineering
Jesse Anderson
DECEMBER 12, 2022
Big data projects were given to data scientists and data warehouse teams, where the projects subsequently failed. As clearly evident as that sounds now, my writing about needing data engineering went heavily against the grain of everything that was written at the time.
Data Engineering 130
130 
Our First Netflix Data Engineering Summit
Netflix Tech
DECEMBER 14, 2023
Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr. In this video, Sr.
Data Engineering 103
103 
7 Data Engineering Projects To Put On Your Resume
Seattle Data Guy
MAY 20, 2023
Starting new data engineering projects can be challenging. Data engineers can get stuck on finding the right data for their data engineering project or picking the right tools.
Data Engineering 147
147 
What Is The State Of Data Engineering And Infrastructure In 2023
Seattle Data Guy
JANUARY 17, 2023
Are data engineers all learning Rust? Our team is putting together an all day event focused on helping answer some… Read more The post What Is The State Of Data Engineering And Infrastructure In 2023 appeared first on Seattle Data Guy. Are Snowflake and Databricks still fighting over total cost of ownership?
Data Engineering 211
211 
Rust for Data Engineering
Simon Späti
OCTOBER 19, 2022
Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.
Data Engineering 130
130 
Rust for Data Engineering
Simon Späti
OCTOBER 19, 2022
Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.
Data Engineering 130
130 
The Ultimate Data Engineering Chadstack. Running Rust inside Apache Airflow.
Confessions of a Data Guy
OCTOBER 6, 2023
I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling […] The post The Ultimate Data Engineering Chadstack. appeared first on Confessions of a Data Guy. At the moment Rust and Airflow are at least somewhere at the top of that list. Running Rust inside Apache Airflow.
Data Engineering 147
147 
The Role of DevOps and CI/CD in Data Engineering
Confessions of a Data Guy
SEPTEMBER 9, 2023
Nothing screams “why are flying by night,” than coming into a Data Team only to find no tests, no docs, no deployments, no Docker, no nothing. […] The post The Role of DevOps and CI/CD in Data Engineering appeared first on Confessions of a Data Guy.
Data Engineering 162
162 
4 Ways To Setup Your Data Engineering Game.
Confessions of a Data Guy
JUNE 8, 2023
It gives a fairly unique view into the wide range of Data Engineering companies, jobs, projects people are working on, tech stacks, and problems that are being faced. One thing I’ve come […] The post 4 Ways To Setup Your Data Engineering Game. appeared first on Confessions of a Data Guy.
Data Engineering 130
130 
AWS Lambdas. Useful for Data Engineering?
Confessions of a Data Guy
MARCH 20, 2023
For a lot of my Data Engineering career I didn’t really think about or use AWS lambdas, I just saw them as little annoying flies […] The post AWS Lambdas. Useful for Data Engineering? appeared first on Confessions of a Data Guy.
AWS 130
130 
How to become a valuable data engineer
Start Data Engineering
JUNE 13, 2023
Introduction So you are a new data engineer (or looking for a DE job) and want to better yourself as a data engineer. Money & Time 2.2. Technical skills 3. Build impactful projects 4. Conclusion 5. Further reading 1.
Data Engineering 130
130 
Data Engineering Weekly #163
Data Engineering Weekly
MARCH 17, 2024
It also introduces emerging standards like the Open Data Contract Standard and Data Product Descriptor Specification. As you know, I’m fascinated by data products and the potential to change the data engineering practice. Can we measure the cost of data incidents?
Introduction to Historical Loads – for Data Engineers.
Confessions of a Data Guy
NOVEMBER 5, 2022
There are probably few things in life that will strike more fear and tumult in the heart of the Data Engineer than historical loads. How could it possibly be, just take a bunch of data stored somewhere and shove it into a table. […] The post Introduction to Historical Loads – for Data Engineers.
Data Engineering 130
130 
5 Free Courses to Master Data Engineering
KDnuggets
NOVEMBER 30, 2023
Data engineers must prepare and manage the infrastructure and tools necessary for the whole data workflow in a data-driven company.
Data Engineering 119
119 
The Only Free Course You Need To Become a Professional Data Engineer
KDnuggets
JANUARY 26, 2024
Data Engineering ZoomCamp offers free access to reading materials, video tutorials, assignments, homeworks, projects, and workshops.
Data Engineering 149
149 
What's new on the cloud for data engineers - part 12 (10.2023-02.2024)
Waitingforcode
FEBRUARY 13, 2024
It's time for another part of "What's new on the cloud for data engineers" Let's see what happened in the last 5 months.
Data Engineering 130
130 
How I use Gen AI as a Data Engineer
Towards Data Science
MARCH 26, 2024
In this article we dive into some practical examples for Data Engineers Continue reading on Towards Data Science » Generative AI is all the rage.
Data Engineering Weekly #159
Data Engineering Weekly
FEBRUARY 18, 2024
Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. link] Sponsored: Data modeling and exploration in Playground 2.0 Elevate your data skills!
How Data Engineering Teams Power Machine Learning With Feature Platforms
Data Engineering Podcast
JULY 2, 2023
In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?
Machine Learning 130
130 
Data Engineering Weekly #156
Data Engineering Weekly
JANUARY 28, 2024
I had a chance to meet some of the amazing humans of data engineering. The article narrates various stages, including business intelligence, data platform, event-bus backbone, and machine learning, leading to decentralizing data engineering expertise and code.
9 Skills You Need to Become a Data Engineer
KDnuggets
NOVEMBER 2, 2022
A data engineer is a fast-growing profession with amazing challenges and rewards. Which skills do you need to become a data engineer? In this post, we’ll take a look at both hard and soft skills.
Data Engineering 159
159 
GPT-based data engineering accelerators
RandomTrees
FEBRUARY 2, 2024
GPT-based data engineering accelerators make the working of data more accessible. These accelerators use GPT models to do data tasks faster, fix any issues, and save a lot of time. GPT models change data in simple language and also provide summaries and explanations. One can rely on this information.
Free Data Engineering Courses
KDnuggets
MAY 30, 2022
Get into the highly in-demand world of data engineering for free and earn 6 figures salary.
Data Engineering 160
160 
Data Engineering Weekly #161
Data Engineering Weekly
MARCH 3, 2024
Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup. There will be food, networking, and real-world talks around data engineering.
The Chaos Data Engineering Manifesto
Towards Data Science
FEBRUARY 24, 2023
The Chaos Data-Engineering Manifesto Another lesson we can learn from software engineers: break stuff to make it more reliable. While this idea isn’t completely foreign to data engineering, it can certainly be described as an extremely uncommon practice. Data is different. It’s terrifying. I’m afraid so.
Data Engineering Weekly #157
Data Engineering Weekly
FEBRUARY 4, 2024
Joe Reis: Definition of Data Modeling & What Data Modeling Is not Joe raised a very fundamental question in data engineering. What is Data Modeling, and what is not? He rightly points out that many data engineers will name any of the modeling techniques as data modeling.
Let's personalize your content