7 Python Libraries Every Data Engineer Should Know
KDnuggets
APRIL 25, 2024
Interested in switching to data engineering? Here’s a list of Python libraries you’ll find super helpful.
Snowflake’s New Python API Empowers Data Engineers to Build Modern Data Pipelines with Ease
Snowflake
APRIL 17, 2024
Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. Recognizing this shift, Snowflake is taking a Python-first approach to bridge the gap and help users leverage the power of both worlds.
Data Pipeline 
102
102 Join 16,000+
Insiders
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Webinars
Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications
From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success
Understanding User Needs and Satisfying Them
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
How to learn data engineering
Christophe Blefari
JANUARY 20, 2024
Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?
Data Engineering 130
130 Webinars
Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications
From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success
Understanding User Needs and Satisfying Them
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
Rust for Data Engineering
Simon Späti
OCTOBER 19, 2022
Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.
Data Engineering 130
130 
Rust for Data Engineering
Simon Späti
OCTOBER 19, 2022
Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.
Data Engineering 130
130 
Modern Data Engineering with MAGE: Empowering Efficient Data Processing
Analytics Vidhya
JUNE 20, 2023
Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.
Data Process 217
217 
Python for Data Engineering
Ascend.io
SEPTEMBER 14, 2023
The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?
Brief History of Data Engineering
Jesse Anderson
DECEMBER 12, 2022
Big data projects were given to data scientists and data warehouse teams, where the projects subsequently failed. As clearly evident as that sounds now, my writing about needing data engineering went heavily against the grain of everything that was written at the time. Now people are excited about Rust.
Data Engineering 130
130 
AWS Lambdas – Python vs Rust. Performance and Cost Savings.
Confessions of a Data Guy
FEBRUARY 26, 2023
Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. At least it’s an excuse to […] The post AWS Lambdas – Python vs Rust.
AWS 356
356 
DuckDB vs Polars for Data Engineering.
Confessions of a Data Guy
APRIL 16, 2023
You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. I haven’t seen this since Databricks and Snowflake first came out and started throwing mud at each other.
Data Engineering 147
147 
How Data Engineering Teams Power Machine Learning With Feature Platforms
Data Engineering Podcast
JULY 2, 2023
In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?
Machine Learning 130
130 
What Is Apache Airflow – Data Engineering Consulting
Seattle Data Guy
FEBRUARY 11, 2023
Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?
Consulting 147
147 
Data Engineering Weekly #163
Data Engineering Weekly
MARCH 17, 2024
It also introduces emerging standards like the Open Data Contract Standard and Data Product Descriptor Specification. As you know, I’m fascinated by data products and the potential to change the data engineering practice. Can we measure the cost of data incidents?
Useful classes for data engineers - Scala & Java
Waitingforcode
FEBRUARY 3, 2023
In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. The part for Python will follow next week! We all have our habits and as programmers, libraries and frameworks are definitely a part of the group.
Scala 130
130 
The Role of DevOps and CI/CD in Data Engineering
Confessions of a Data Guy
SEPTEMBER 9, 2023
Nothing screams “why are flying by night,” than coming into a Data Team only to find no tests, no docs, no deployments, no Docker, no nothing. […] The post The Role of DevOps and CI/CD in Data Engineering appeared first on Confessions of a Data Guy.
Data Engineering 162
162 
The Ultimate Data Engineering Chadstack. Running Rust inside Apache Airflow.
Confessions of a Data Guy
OCTOBER 6, 2023
I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling […] The post The Ultimate Data Engineering Chadstack. appeared first on Confessions of a Data Guy. At the moment Rust and Airflow are at least somewhere at the top of that list. Running Rust inside Apache Airflow.
Data Engineering 147
147 
Data Engineering Weekly #159
Data Engineering Weekly
FEBRUARY 18, 2024
Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. link] Sponsored: Data modeling and exploration in Playground 2.0 Elevate your data skills!
Tired of your Data Engineering Role?
Towards Data Science
AUGUST 19, 2023
How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.
Delivering Modern Enterprise Data Engineering with Cloudera Data Engineering on Azure
Cloudera
JULY 13, 2021
After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . Prerequisites for deploying CDP Data Engineering on Azure can be found here.
Data Engineering 124
124 
Python for Data Engineers
Towards Data Science
OCTOBER 21, 2023
Advanced ETL techniques for beginners Continue reading on Towards Data Science »
Effective Pandas Patterns For Data Engineering
Data Engineering Podcast
JANUARY 30, 2022
Summary Pandas is a powerful tool for cleaning, transforming, manipulating, or enriching data, among many other potential uses. As a result it has become a standard tool for data engineers for a wide range of applications. What are the main tasks that you have seen Pandas used for in a data engineering context?
Data Engineering 100
100 
10 Skills to Ace Your Data Engineering Interviews
Start Data Engineering
OCTOBER 11, 2021
Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.
Data Engineering 130
130 
Top 12 Data Engineering Project Ideas [With Source Code]
Knowledge Hut
JUNE 26, 2023
Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?
Azure Data Engineer Roles and Responsibilities 2024
Knowledge Hut
MARCH 15, 2024
At the same time, it has opened up a wealth of opportunities for data engineers. With businesses harnessing the power of Azure’s services, the need for skilled data engineers has topped the charts. Speaking from experience, the data engineers in this role are right in the thick of it all.
Maintain Your Data Engineers' Sanity By Embracing Automation
Data Engineering Podcast
JULY 10, 2022
Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.
Data Engineering 100
100 
The Symbiotic Relationship Between AI and Data Engineering
Ascend.io
FEBRUARY 28, 2024
The rise of generative AI is changing more than just technology; it’s reshaping our professional landscapes — and yes, data engineering is directly experiencing the impact. How does AI recalibrate the workload and priorities of data teams? How can data engineers harness the power of AI?
Azure Data Engineer Roles and Responsibilities in 2024
Knowledge Hut
MARCH 20, 2024
At the same time, it has opened up a wealth of opportunities for data engineers. With businesses harnessing the power of Azure’s services, the need for skilled data engineers has topped the charts. Speaking from experience, the data engineers in this role are right in the thick of it all.
Who is a Big Data Engineer? Skills, Responsibilities, Salary
Knowledge Hut
MARCH 13, 2024
Wondering what is a big data engineer? As the name suggests, Big Data is associated with ‘big’ data, which hints at something big in the context of data. Big data forms one of the pillars of data science. Big data has been a hot topic in the IT sector for quite a long time.
Big Data 52
52 
Who is a Big Data Engineer? Skills, Responsibilities, Salary
Knowledge Hut
MARCH 13, 2024
Wondering what is a big data engineer? As the name suggests, Big Data is associated with ‘big’ data, which hints at something big in the context of data. Big data forms one of the pillars of data science. Big data has been a hot topic in the IT sector for quite a long time.
Big Data 52
52 
Skills you should have as a Data Engineer
Team Data Science
JANUARY 8, 2021
The purpose of this post is to expose you to the skills needed as a data engineer; now let’s look into them Understand the fundamental skill Recently, functions of computer engineering have become more important in organizations that are handling vast volumes of data, including data in diverse formats.
Data Engineering 130
130 
Top 10 Azure Data Engineer Job Opportunities in 2024 [Career Options]
Knowledge Hut
MARCH 28, 2024
Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 All these numbers point to one thing–increased job roles and careers, especially when we talk about data engineering jobs in Azure, which are on the rise every year. Let’s get started.
Data Engineering Weekly #164
Data Engineering Weekly
MARCH 24, 2024
[link] Freshworks: Modernizing analytics data ingestion pipeline from legacy engine to distributed processing engine The article discusses Freshworks' journey in modernizing its analytics data platform to handle increasing volumes of data efficiently.
Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte
Data Engineering Podcast
MAY 22, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.
Machine Learning 130
130 
Data Engineering Weekly #147
Data Engineering Weekly
SEPTEMBER 24, 2023
Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. See how it works today. The Polars DataFrame support for JavaScript is a game changer.
15+ Must Have Data Engineer Skills in 2023
Knowledge Hut
NOVEMBER 28, 2023
The contemporary world experiences a huge growth in cloud implementations, consequently leading to a rise in demand for data engineers and IT professionals who are well-equipped with a wide range of application and process expertise. Data Engineer certification will aid in scaling up you knowledge and learning of data engineering.
Will GenAI Replace Data Engineers? No — And Here’s Why.
Monte Carlo
FEBRUARY 7, 2024
Data engineering is no exception. Already in the wee months of 2024, GenAI is beginning to upend the way data teams think about ingesting, transforming, and surfacing data to consumers. As familiar workflows evolve, it naturally begs a question: will GenAI replace data engineers? At least, not anytime soon.
Data Engineering Weekly #137
Data Engineering Weekly
JULY 2, 2023
Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. So, let's shape the future of Data Engineering together.
dbt Core, Snowflake, and GitHub Actions: pet project for Data Engineers
Towards Data Science
DECEMBER 1, 2023
Architecture Referencing Joe Reis’s “Fundamentals of Data Engineering,” let’s review our project in alignment with the defined stages of the data lifecycle: Data Engineering Lifecycle [1] Data Generation — Google Calendar, Fivetran If you’re a Google Calendar user, chances are you’ve accumulated a wealth of data there.
Data Engineering Weekly #123
Data Engineering Weekly
MARCH 19, 2023
Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real time with your own JavaScript or Python code. link] Sanjeev Mohan: What Exactly is a Data Product? Is chatGPT a data product? Is Data a product? from the browser or mobile app).
Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast
Data Engineering Podcast
JULY 17, 2022
Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.
Data Engineering 100
100 
Data Engineering Weekly #142
Data Engineering Weekly
AUGUST 13, 2023
Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. See how it works today. Editor’s Note: DewCon.ai
Jobprofil des Data Engineers
Data Science Blog: Data Engineering
DECEMBER 23, 2022
Warum Data Engineering der Data Science in Bedeutung und Berufschancen längst die Show stiehlt, dabei selbst ebenso einem stetigen Wandel unterliegt. Was ein Data Engineer wirklich können muss Der Data Scientist als sexiest Job des 21. Data Engineer, der gefragteste Job des 21sten Jahrhunderts?
5 Skills Data Engineers Should Master to Keep Pace with GenAI
Monte Carlo
FEBRUARY 27, 2024
If you’re a data engineer experiencing GenAI-induced whiplash, you’re not alone. On one hand, everyone’s talking about whether GenAI’s not-insignificant data engineering skills are going to automate away their jobs. They need robust data pipelines, high-quality data, well-guarded privacy, and cost-effective scalability.
How to Become an Azure Data Engineer? 2023 Roadmap
Knowledge Hut
NOVEMBER 17, 2023
The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.
Let's personalize your content