article thumbnail

Python Essentials for Data Engineers

Start Data Engineering

Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct (..)

Python 148
article thumbnail

AWS Lambdas – Python vs Rust. Performance and Cost Savings.

Confessions of a Data Guy

Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. At least it’s an excuse to […] The post AWS Lambdas – Python vs Rust.

AWS 356
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Building cost effective data pipelines with Python & DuckDB

Start Data Engineering

KISS: DuckDB + Python = easy to debug and quick to develop 4. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Use DuckDB 4.4. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5.

article thumbnail

Is Python OOP the Devil? Or Savior?

Confessions of a Data Guy

Nothing will raise the hackles on the backs of hairy and pale programmers who’ve been stuck in their mom’s basement for a decade like bringing up OOP (Object Oriented Programming), especially in the context of Python. appeared first on Confessions of a Data Guy.

Python 130
article thumbnail

Isolated Python Environments using Docker

Analytics Vidhya

Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project.

Python 218
article thumbnail

Building Open-Source Python Packages – SparklePop

Confessions of a Data Guy

One of the things I love about Python is its flexibility and huge community, a community that puts out a never-ending stream of useful packages for the average Software Engineer.

Python 100
article thumbnail

Google Fires Python. What Next?

Confessions of a Data Guy

I thought Python was going to live forever. Recently Google announced it was laying off its entire North American-based Python team that was supporting Google’s special needs with Python, in favor of cheaper offshore workers. Apparently, some of these […] The post Google Fires Python.

Python 100