article thumbnail

How to test PySpark code with pytest

Start Data Engineering

Ensure the code’s logic is working as expected with tests 2.1. pytest: A powerful Python library for testing 2.2.1. Set context, run code, check results & clean up 2.2.2. Introduction 2. Test types for data pipelines 2.2. Tests are identified by their name 2.2.3. Use fixture to create fake data for testing 2.2.4.

Coding 130
article thumbnail

Data Pipeline Design Patterns - #2. Coding patterns in Python

Start Data Engineering

Introduction Sample project Code design patterns 1. Singleton, & Object pool patterns Python helpers 1. Functional design 2. Factory pattern 3. Strategy pattern 4. Dataclass 3. Context Managers 4. Testing with pytest 5.

Designing 147
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

How To Write Efficient Python Code: A Tutorial for Beginners

KDnuggets

Are you a programmer looking to get better at Python? Learn some of Python’s features that’ll help you write more elegant and Pythonic code.

Python 121
article thumbnail

Enhance Your Python Coding Style with Ruff

KDnuggets

Ruff's 700+ built-in lint rules, reimplemented in Rust for speed, provide comprehensive linting and formatting to enforce clean and consistent Python code.

Coding 106
article thumbnail

Reading and Processing JSON with Rust vs Python.

Confessions of a Data Guy

Have you ever wondered about being explicit in your code vs being vague? I think about this a lot as I’m writing code on a daily basis. I’ve found I like being explicit and verbose when writing code, rather than being vague in what I’m doing most of the time. appeared first on Confessions of a Data Guy.

Python 100
article thumbnail

Announcing FawltyDeps - a dependency checker for your Python code

Tweag

It is a truth universally acknowledged that the Python packaging ecosystem is in need of a good dependency checker. If you work with Python, and care about keeping your projects lean and repeatable, then this is for you. The dependency is now installed in your Python virtual environment or on your system. 3rd-party imports).

Python 142
article thumbnail

Top 15 Python IDEs and Code Editors to Use in 2024

Knowledge Hut

Over the years, Python language has evolved enormously with the contribution of developers. Python is one of the most popular programming languages. For this feature, Python encloses certain code editors and python IDEs used for software development say, Python itself. What is a Code Editor?

Python 97