Marc Lamberti

OCTOBER 1, 2023

Airflow Sensors are one of the most common tasks in data pipelines. Why? Because a Sensor waits for a condition to be true to complete. Do you need to wait for a file? Check if an SQL entry exists? Delay the execution of a DAG? That’s the few possibilities of the Airflow Sensors. If you want to make complex and robust data pipelines, you have to understand how Sensors work genuinely.

Data Pipeline 243

Data Pipeline SQL Algorithm Coding 243

Data Engineering Podcast

OCTOBER 1, 2023

Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI.

Building 162

Building BI SQL Machine Learning 162

Confessions of a Data Guy

OCTOBER 1, 2023

So perhaps you’re thinking it’s time to use Rust on your next project. You’ll find plenty of primers on how to get your feet wet in the language (and if you somehow made it this far without that much, The Book is that starting point), but maybe you’re feeling a bit lost amidst the seas […] The post Introduction to using Rust Libraries (cargo and crates) appeared first on Confessions of a Data Guy.

Project 130

Project IT Data Data Engineering 130

KDnuggets

OCTOBER 1, 2023

Explore the essentials of Google Cloud Platform for data science and ML, from account setup to model deployment, with hands-on project examples.

Google Cloud 136

Google Cloud Cloud Data Science Project 136

Advertisement

Generative AI is upending the way product developers & end-users alike are interacting with data. Despite the potential of AI, many are left with questions about the future of product development: How will AI impact my business and contribute to its success? What can product managers and developers expect in the future with the widespread adoption of AI?

Data

Data Engineering Weekly

OCTOBER 1, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. See how it works today. Dropbox: Is this a date? Using ML to identify date formats in file names Naming things is always one of the hardest problems in computer science.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Rock the JVM

OCTOBER 1, 2023

by Herbert Kateu 1. Introduction RPC stands for Remote Procedure Call, it’s a client-server communication protocol where one program can request a service on a different address that may be on the same or different system connected by a network. It enables users to work with remote procedures as if they were local. In this article we will cover gRPC which is a modern Open Source RPC framework designed by Google that uses Protocol Buffers for data serialization and HTTP/2 as a transport layer. gR

Scala 63

Scala Metadata Transportation Java 63