7 Python Libraries Every Data Engineer Should Know
KDnuggets
APRIL 25, 2024
Interested in switching to data engineering? Here’s a list of Python libraries you’ll find super helpful.
Snowflake’s New Python API Empowers Data Engineers to Build Modern Data Pipelines with Ease
Snowflake
APRIL 17, 2024
Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. Recognizing this shift, Snowflake is taking a Python-first approach to bridge the gap and help users leverage the power of both worlds.
Join 16,000+
Insiders
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Webinars
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Communication
Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications
From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success
Understanding User Needs and Satisfying Them
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
AWS Lambdas – Python vs Rust. Performance and Cost Savings.
Confessions of a Data Guy
FEBRUARY 26, 2023
Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. At least it’s an excuse to […] The post AWS Lambdas – Python vs Rust.
AWS 
356
356 Webinars
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Communication
Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications
From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success
Understanding User Needs and Satisfying Them
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
Python for Data Engineering
Ascend.io
SEPTEMBER 14, 2023
The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?
Isolated Python Environments using Docker
Analytics Vidhya
FEBRUARY 6, 2023
Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project.
Python 218
218 
Rust for Data Engineering
Simon Späti
OCTOBER 19, 2022
Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.
Data Engineering 130
130 
Rust for Data Engineering
Simon Späti
OCTOBER 19, 2022
Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.
Data Engineering 130
130 
Brief History of Data Engineering
Jesse Anderson
DECEMBER 12, 2022
Big data projects were given to data scientists and data warehouse teams, where the projects subsequently failed. As clearly evident as that sounds now, my writing about needing data engineering went heavily against the grain of everything that was written at the time. Now people are excited about Rust.
Data Engineering 130
130 
Modern Data Engineering with MAGE: Empowering Efficient Data Processing
Analytics Vidhya
JUNE 20, 2023
Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.
Data Process 217
217 
Useful classes for data engineers - Scala & Java
Waitingforcode
FEBRUARY 3, 2023
In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. The part for Python will follow next week! We all have our habits and as programmers, libraries and frameworks are definitely a part of the group.
Scala 130
130 
How To Build and Open Source PYPI Python Package
Confessions of a Data Guy
MARCH 25, 2024
Ever wondered how to build and end-to-end project for an Open Source Python Package that gets published to PYPI? link] The post How To Build and Open Source PYPI Python Package appeared first on Confessions of a Data Guy.
Python 100
100 
Python for Data Engineers
Towards Data Science
OCTOBER 21, 2023
Advanced ETL techniques for beginners Continue reading on Towards Data Science »
Why I Love Rust, but Deploy Python
Confessions of a Data Guy
FEBRUARY 25, 2024
I love to write Rust … but I deploy Python. Even when I know I […] The post Why I Love Rust, but Deploy Python appeared first on Confessions of a Data Guy. I’m not sure if others have this same problem, maybe they are lucky, they get to build in their favorite language 24/7, it’s their tool of choice.
Python 113
113 
Data Engineering Weekly #163
Data Engineering Weekly
MARCH 17, 2024
It also introduces emerging standards like the Open Data Contract Standard and Data Product Descriptor Specification. As you know, I’m fascinated by data products and the potential to change the data engineering practice. Can we measure the cost of data incidents?
How Data Engineering Teams Power Machine Learning With Feature Platforms
Data Engineering Podcast
JULY 2, 2023
In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?
Machine Learning 130
130 
Snowflake To Acquire Ponder, Boosting Python Capabilities In the Data Cloud
Snowflake
OCTOBER 23, 2023
One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers. Snowflake customers are already harnessing the power of Python through Snowpark , a set of runtimes and libraries that securely deploy and process non-SQL code directly in Snowflake.
Python 138
138 
What are Data Access Object and Data Transfer Object in Python?
Analytics Vidhya
FEBRUARY 6, 2023
This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? Especially while working with databases, it is often considered a good practice to follow a design pattern. appeared first on Analytics Vidhya.
Accessible 234
234 
Data Engineering Weekly #159
Data Engineering Weekly
FEBRUARY 18, 2024
Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. link] Sponsored: Data modeling and exploration in Playground 2.0 Elevate your data skills!
DuckDB vs Polars for Data Engineering.
Confessions of a Data Guy
APRIL 16, 2023
You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. I haven’t seen this since Databricks and Snowflake first came out and started throwing mud at each other.
Data Engineering 147
147 
What Is Apache Airflow – Data Engineering Consulting
Seattle Data Guy
FEBRUARY 11, 2023
Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?
Consulting 147
147 
The Role of DevOps and CI/CD in Data Engineering
Confessions of a Data Guy
SEPTEMBER 9, 2023
Nothing screams “why are flying by night,” than coming into a Data Team only to find no tests, no docs, no deployments, no Docker, no nothing. […] The post The Role of DevOps and CI/CD in Data Engineering appeared first on Confessions of a Data Guy.
Data Engineering 162
162 
The Ultimate Data Engineering Chadstack. Running Rust inside Apache Airflow.
Confessions of a Data Guy
OCTOBER 6, 2023
I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling […] The post The Ultimate Data Engineering Chadstack. appeared first on Confessions of a Data Guy. At the moment Rust and Airflow are at least somewhere at the top of that list. Running Rust inside Apache Airflow.
Data Engineering 147
147 
Delivering Modern Enterprise Data Engineering with Cloudera Data Engineering on Azure
Cloudera
JULY 13, 2021
After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . Prerequisites for deploying CDP Data Engineering on Azure can be found here.
Data Engineering 126
126 
Tired of your Data Engineering Role?
Towards Data Science
AUGUST 19, 2023
How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.
Python Scopes and Their Built-in Functions
Knowledge Hut
FEBRUARY 1, 2024
Variables in Python are fundamental containers used for storing and manipulating data in a program. In Python programming, variables are the backbone of data manipulation and program logic. They hold and transform data, allowing for the execution of algorithms and the management of large datasets.
Python 72
72 
10 Skills to Ace Your Data Engineering Interviews
Start Data Engineering
OCTOBER 11, 2021
Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.
Data Engineering 130
130 
Effective Pandas Patterns For Data Engineering
Data Engineering Podcast
JANUARY 30, 2022
Summary Pandas is a powerful tool for cleaning, transforming, manipulating, or enriching data, among many other potential uses. As a result it has become a standard tool for data engineers for a wide range of applications. What are the main tasks that you have seen Pandas used for in a data engineering context?
Data Engineering 100
100 
Streamlit vs Gradio – A Guide to Building Dashboards in Python
Analytics Vidhya
FEBRUARY 20, 2023
This blog is a tutorial for building intuitive frontend interfaces for Machine Learning models using two popular open-source libraries […] The post Streamlit vs Gradio – A Guide to Building Dashboards in Python appeared first on Analytics Vidhya.
Python 152
152 
Managing Python dependencies for Spark workloads in Cloudera Data Engineering
Cloudera
APRIL 30, 2021
If the users are already familiar with Python then PySpark provides a python API for using Apache Spark. When users work with PySpark they often use existing python and/or custom Python packages in their program to extend and complement Apache Spark’s functionality. Install Python dependencies on all nodes in the Cluster.
Python 61
61 
Maintain Your Data Engineers' Sanity By Embracing Automation
Data Engineering Podcast
JULY 10, 2022
Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.
Data Engineering 100
100 
Top 10 Azure Data Engineer Job Opportunities in 2024 [Career Options]
Knowledge Hut
MARCH 28, 2024
Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 All these numbers point to one thing–increased job roles and careers, especially when we talk about data engineering jobs in Azure, which are on the rise every year. Let’s get started.
The Symbiotic Relationship Between AI and Data Engineering
Ascend.io
FEBRUARY 28, 2024
The rise of generative AI is changing more than just technology; it’s reshaping our professional landscapes — and yes, data engineering is directly experiencing the impact. How does AI recalibrate the workload and priorities of data teams? How can data engineers harness the power of AI?
Skills you should have as a Data Engineer
Team Data Science
JANUARY 8, 2021
The purpose of this post is to expose you to the skills needed as a data engineer; now let’s look into them Understand the fundamental skill Recently, functions of computer engineering have become more important in organizations that are handling vast volumes of data, including data in diverse formats.
Data Engineering 130
130 
Who is a Big Data Engineer? Skills, Responsibilities, Salary
Knowledge Hut
MARCH 13, 2024
Wondering what is a big data engineer? As the name suggests, Big Data is associated with ‘big’ data, which hints at something big in the context of data. Big data forms one of the pillars of data science. Big data has been a hot topic in the IT sector for quite a long time.
Big Data 52
52 
Who is a Big Data Engineer? Skills, Responsibilities, Salary
Knowledge Hut
MARCH 13, 2024
Wondering what is a big data engineer? As the name suggests, Big Data is associated with ‘big’ data, which hints at something big in the context of data. Big data forms one of the pillars of data science. Big data has been a hot topic in the IT sector for quite a long time.
Big Data 52
52 
Azure Data Engineer Roles and Responsibilities 2024
Knowledge Hut
MARCH 15, 2024
At the same time, it has opened up a wealth of opportunities for data engineers. With businesses harnessing the power of Azure’s services, the need for skilled data engineers has topped the charts. Speaking from experience, the data engineers in this role are right in the thick of it all.
Top 12 Data Engineering Project Ideas [With Source Code]
Knowledge Hut
JUNE 26, 2023
Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?
Azure Data Engineer Roles and Responsibilities in 2024
Knowledge Hut
MARCH 20, 2024
At the same time, it has opened up a wealth of opportunities for data engineers. With businesses harnessing the power of Azure’s services, the need for skilled data engineers has topped the charts. Speaking from experience, the data engineers in this role are right in the thick of it all.
15+ Must Have Data Engineer Skills in 2023
Knowledge Hut
NOVEMBER 28, 2023
The contemporary world experiences a huge growth in cloud implementations, consequently leading to a rise in demand for data engineers and IT professionals who are well-equipped with a wide range of application and process expertise. Data Engineer certification will aid in scaling up you knowledge and learning of data engineering.
Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte
Data Engineering Podcast
MAY 22, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.
Machine Learning 130
130 
Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast
Data Engineering Podcast
JULY 17, 2022
Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.
Data Engineering 100
100 
Data Engineering Weekly #164
Data Engineering Weekly
MARCH 24, 2024
[link] Freshworks: Modernizing analytics data ingestion pipeline from legacy engine to distributed processing engine The article discusses Freshworks' journey in modernizing its analytics data platform to handle increasing volumes of data efficiently.
Introducing Python User-Defined Table Functions (UDTFs)
databricks
NOVEMBER 7, 2023
have brought an exciting feature to the table: Python user-defined table functions (UDTFs). Apache Spark™ 3.5 and Databricks Runtime 14.0 In this blog p.
Python 105
105 
KDnuggets Top Posts for January 2023: SQL and Python Interview Questions for Data Analysts
KDnuggets
MARCH 22, 2023
SQL and Python Interview Questions for Data Analysts • 5 SQL Visualization Tools for Data Engineers • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2 • Top Free Resources To Learn ChatGPT • Free TensorFlow 2.0
SQL 112
112 
Let's personalize your content