Python - Data Engineering Digest

AWS Lambdas – Python vs Rust. Performance and Cost Savings.

Confessions of a Data Guy

FEBRUARY 26, 2023

Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. At least it’s an excuse to […] The post AWS Lambdas – Python vs Rust.

AWS

AWS Python Data Engineering Data Engineer

Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python

Seattle Data Guy

NOVEMBER 19, 2024

Most of us will turn to Python and our trusty list of Python libraries and start plugging away. Of course, there are many challenges… Read more The post Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python appeared first on Seattle Data Guy.

Python

Python Data Data Science Data Engineering

Isolated Python Environments using Docker

Analytics Vidhya

FEBRUARY 6, 2023

Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project.

Python

Python Project Management Data Engineering

Webinars

How to Achieve High-Accuracy Results When Using LLMs

MORE WEBINARS

Building cost effective data pipelines with Python & DuckDB

Start Data Engineering

MAY 28, 2024

KISS: DuckDB + Python = easy to debug and quick to develop 4. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Use DuckDB 4.4. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5.

Data Pipeline

Data Pipeline Python Building Data

Apache Airflow® Crash Course: From 0 to Running your Pipeline in the Cloud

Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines.

Cloud

What are Data Access Object and Data Transfer Object in Python?

Analytics Vidhya

FEBRUARY 6, 2023

This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? Especially while working with databases, it is often considered a good practice to follow a design pattern. appeared first on Analytics Vidhya.

Accessible

Accessible Accessibility Python Database

10 GitHub Repositories to Master Python

KDnuggets

APRIL 10, 2024

Learn Python through tutorials, blogs, books, project work, and exercises. Access all of it on GitHub for free and join a supportive open-source community.

Python

Python Accessible Accessibility Project

ChatGPT as a Python Programming Assistant

KDnuggets

JANUARY 20, 2023

Is ChatGPT useful for Python programmers, specifically those of us who use Python for data processing, data cleaning, and building machine learning models? Let's give it a try and find out.

Python

Python Programming Machine Learning Data Process

7 Python Libraries Every Data Engineer Should Know

KDnuggets

APRIL 25, 2024

Here’s a list of Python libraries you’ll find super helpful. Interested in switching to data engineering?

Python

Python Data Engineering Data Engineer Engineering

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

Data Engineering Podcast

JUNE 25, 2023

__init__ covers the Python language, its community, and the innovative ways it is being used. __init__ covers the Python language, its community, and the innovative ways it is being used. Closing Announcements Thank you for listening! Don't forget to check out our other shows. Closing Announcements Thank you for listening!

Data Engineering

Data Engineering Data Engineer Python Engineering

7 Free Online Python REPLs

KDnuggets

SEPTEMBER 24, 2024

Running Python code directly in your browser is incredibly convenient, eliminating the need for Python environment setup and allowing instant code execution without dependency or hardware concerns. I am a strong advocate of using a cloud-based IDE for working with data, machine learning, and learning Python as a beginner.

Python

Python Machine Learning Coding Programming

5 Free Python Courses for Data Science Beginners

KDnuggets

MAY 31, 2024

Are you a data science beginner looking to learn Python? Start learning today with these 5 free courses.

Data Science

Data Science Python Data

FastAPI Tutorial: Build APIs with Python in Minutes

KDnuggets

JUNE 13, 2024

Want to build APIs with Python? Learn how to do so using FastAPI with this step-by-step tutorial.

Python

Python Building

5 Simple Steps to Automate Data Cleaning with Python

KDnuggets

MAY 3, 2024

Automate your data cleaning process with a practical 5-step pipeline in Python, ideal for beginners.

Python

Python Data Process

10 Python Libraries Every Data Scientist Should Know

KDnuggets

AUGUST 12, 2024

Check out these Python libraries for data science that you can't do without. Want to take the next step in your journey to becoming a data scientist?

Python

Python Data Science Data

7 Best Platforms to Practice Python

KDnuggets

APRIL 23, 2024

Looking to level up your Python skills and ace coding interviews? Start practicing today on these platforms.

Python

Python Coding

11 Python Magic Methods Every Programmer Should Know

KDnuggets

NOVEMBER 28, 2023

Want to support the behavior of built-in functions and method calls in your Python classes? Magic methods in Python let you do just that! So let’s uncover the method behind the magic.

Python

Mastering Python for Data Science: Beyond the Basics

KDnuggets

MARCH 28, 2024

This article serves as a detailed guide on how to master advanced Python techniques for data science. It covers topics such as efficient data manipulation with Pandas, parallel processing with Python, and how to turn models into web services.

Data Science

Data Science Python Data Process

10 Built-In Python Modules Every Data Engineer Should Know

KDnuggets

SEPTEMBER 2, 2024

Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks. Interested in data engineering?

Python

Python Data Engineering Data Engineer Engineering

3 Ways of Building Python Projects using GPT-4o

KDnuggets

AUGUST 7, 2024

Learn about essential AI tools that can help you develop Python projects faster and with fewer bugs using natural language.

Python

Python Project Building

Is Python OOP the Devil? Or Savior?

Confessions of a Data Guy

JUNE 3, 2024

Nothing will raise the hackles on the backs of hairy and pale programmers who’ve been stuck in their mom’s basement for a decade like bringing up OOP (Object Oriented Programming), especially in the context of Python. appeared first on Confessions of a Data Guy.

Python

Python Programming IT Data

Build An AI Application with Python in 10 Easy Steps

KDnuggets

MARCH 14, 2024

Explore the fundamental steps for creating a successful AI Application with Python and other tools.

Python

Python Building

A New Python Package Manager

KDnuggets

NOVEMBER 14, 2024

Manage Python projects, run scripts and tools, handle dependencies, and install packages—all with the uv tool.

Python

Python Management Project

Tips for Handling Large Datasets in Python

KDnuggets

NOVEMBER 29, 2024

Here are some tips to make working with such large datasets in Python simpler. Working with large datasets is common but challenging.

Datasets

Datasets Python

Free Harvard Course: Introduction to AI with Python

KDnuggets

DECEMBER 18, 2023

Looking for a great course to learn Artificial Intelligence with Python? Check out this free course from Harvard University.

Python

5 Free University Courses to Learn Python

KDnuggets

DECEMBER 14, 2023

Looking for the best resources to learn Python programming? Check out these free university courses.

Python

Python Programming

Mastering f-strings in Python

KDnuggets

NOVEMBER 7, 2024

Discover how to leverage Python's f-strings (formatted string literals) to write cleaner, more efficient, and more readable code.

Python

Python Coding

5 Python Best Practices for Data Science

KDnuggets

MAY 29, 2024

Level up your Python skills for data science with these by following these best practices.

Data Science

Data Science Python Data

10 Python Libraries Every Developer Should Know

KDnuggets

DECEMBER 2, 2024

In this article, we’ll go over Python libraries for tasks like logging, unit testing, data handling, and more — each with features that can simplify your application development.

Python

Python Data

Mastering Python: 7 Strategies for Writing Clear, Organized, and Efficient Code

KDnuggets

MAY 17, 2024

Optimize Your Python Workflow: Proven Techniques for Crafting Production-Ready Code

Python

Python Coding

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically

DataKitchen

FEBRUARY 17, 2025

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically As a data engineer, ensuring data quality is both essential and overwhelming. Writing SQL, Python, or YAML-based rules should not be a prerequisite for their involvement.

SQL

SQL Python Government Data Engineering

Partial Functions in Python: A Guide for Developers

KDnuggets

SEPTEMBER 19, 2024

In Python, functions often require multiple arguments, and you may find yourself repeatedly passing the same values for certain parameters. Python’s built-in functools module allows you to create partial functions. This is where partial functions can help.

Python

Convert Bytes to String in Python: A Tutorial for Beginners

KDnuggets

JULY 15, 2024

Strings are common built-in data types in Python. Let’s learn how to convert bytes to string in Python. But sometimes, you may need to work with bytes instead.

Bytes

Bytes Python Data

5 Free Courses to Master Python for Data Science

KDnuggets

FEBRUARY 7, 2024

Want to learn Python to kickstart your career in data? Here are five free courses to help you master Python for data science.

Data Science

Data Science Python Data

How to Learn Python the Lazy Way

KDnuggets

OCTOBER 18, 2024

It is a guide for lazy people who want to learn Python and earn dollars. The title says everything.

Python

Python IT

Beginner’s Guide to Building LLM Apps with Python

KDnuggets

JUNE 6, 2024

In this article, you will be impacted by the knowledge you need to start building LLM apps with Python programming language.

Python

Python Building Programming Language Programming

10 Python Libraries Every Data Analyst Should Know

KDnuggets

NOVEMBER 19, 2024

Here's a list of Python libraries you cannot do without. Interested in data analytics?

Python

Python Data Analytics Data

Beginner’s Guide to Machine Learning with Python

KDnuggets

JUNE 4, 2024

Master the Fundamentals of Predictive Modeling with Python: An In-Depth Guide to Machine Learning Algorithms and Sci-kit Learn Implementation.

Machine Learning

Machine Learning Python Algorithm

10 Useful Python One-Liners for Data Cleaning

KDnuggets

OCTOBER 29, 2024

Here are some useful Python one-liners for common data cleaning tasks.

Python

Python Data

Pip Install YOU: A Beginner’s Guide to Creating Your Python Library

KDnuggets

AUGUST 22, 2024

Have you ever wanted to create your library in Python? Well, it’s achievable and surprisingly quite easy!

Python

Replacing Pandas with Polars. A Practical Guide.

Confessions of a Data Guy

JANUARY 19, 2023

For those using Python, it’s probably one of the […] The post Replacing Pandas with Polars. I haven’t used Pandas in many a year, decades, or whatever. We’ve all been there, done that. Pandas I mean. I would dare say it’s a rite of passage for most data folk. A Practical Guide.

Python

Python IT Data Big Data

A Gentle Introduction to Rust for Python Programmers

KDnuggets

JANUARY 15, 2025

Python programmers will find Rust's syntax familiar but with more control over memory and performance. Rust is a systems programming language that offers high performance and safety.

Python

Python Programming Language Programming Systems

Don’t Manage Your Python Environments, Just Use Docker Containers

KDnuggets

JANUARY 28, 2025

Python environment management can sometimes give you that awful feeling in the pit of your stomach. So don't do it: just use Docker containers.

Python

Python Management AWS IT

Building Interactive Data Science Applications with Python

KDnuggets

OCTOBER 25, 2024

Using Python to build engaging and interactive applications where users can pass in an input, get and feedback and make use of multimedia elements such as images, videos, and audio.

Python

Python Building Data Science Data

AWS Lambdas – Python vs Rust. Performance and Cost Savings.

Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python

Webinars

Trending Sources

Isolated Python Environments using Docker

Webinars

Building cost effective data pipelines with Python & DuckDB

Apache Airflow® Crash Course: From 0 to Running your Pipeline in the Cloud

What are Data Access Object and Data Transfer Object in Python?

10 GitHub Repositories to Master Python

ChatGPT as a Python Programming Assistant

7 Python Libraries Every Data Engineer Should Know

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

7 Free Online Python REPLs

5 Free Python Courses for Data Science Beginners

FastAPI Tutorial: Build APIs with Python in Minutes

5 Simple Steps to Automate Data Cleaning with Python

10 Python Libraries Every Data Scientist Should Know

7 Best Platforms to Practice Python

11 Python Magic Methods Every Programmer Should Know

Mastering Python for Data Science: Beyond the Basics

10 Built-In Python Modules Every Data Engineer Should Know

3 Ways of Building Python Projects using GPT-4o

Is Python OOP the Devil? Or Savior?

Build An AI Application with Python in 10 Easy Steps

A New Python Package Manager

Tips for Handling Large Datasets in Python

Free Harvard Course: Introduction to AI with Python

5 Free University Courses to Learn Python

Mastering f-strings in Python

5 Python Best Practices for Data Science

10 Python Libraries Every Developer Should Know

Mastering Python: 7 Strategies for Writing Clear, Organized, and Efficient Code

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically

Partial Functions in Python: A Guide for Developers

Convert Bytes to String in Python: A Tutorial for Beginners

5 Free Courses to Master Python for Data Science

How to Learn Python the Lazy Way

Beginner’s Guide to Building LLM Apps with Python

10 Python Libraries Every Data Analyst Should Know

Beginner’s Guide to Machine Learning with Python

10 Useful Python One-Liners for Data Cleaning

Pip Install YOU: A Beginner’s Guide to Creating Your Python Library

Replacing Pandas with Polars. A Practical Guide.

A Gentle Introduction to Rust for Python Programmers

Don’t Manage Your Python Environments, Just Use Docker Containers

Building Interactive Data Science Applications with Python

Stay Connected