Christophe Blefari

JUNE 9, 2023

Rethinking the newsletter ( credits ) Here's a new edition of the Data News newsletter. Since my 2-year anniversary post, I've been struggling to find the right writing rhythm. I can tease you that I'll create content about "Create a data platform in 2023", with live examples.

Data 130

Data Government SQL Coding 130

Workfall

SEPTEMBER 18, 2023

Reading Time: 9 minutes Imagine your data as pieces of a complex puzzle scattered across different platforms and formats. This is where the power of data integration comes into play. Meet Airbyte, the data magician that turns integration complexities into child’s play. In this blog, we will cover: What is Airbyte?

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

Christophe Blefari

MARCH 24, 2023

I really don't know how to deal with news overflow about the Gen AI landscape. Out of the box Mage provide all-in-one web editor to write data pipelines with a great UX. But actually Mage and actual Airflow version are—almost—the same the only difference is the UX when developing pipelines.

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Christophe Blefari

APRIL 28, 2023

To add more spice to this Carlin wrote what happened to the Semantic Layer. To add more spice to this Carlin wrote what happened to the Semantic Layer. This week summer time arrived in Berlin and it was awesome. I managed to move forward with my client projects this week and it also feels relieving. " (cf.

SQL 100

SQL Food PostgreSQL Data 100

Christophe Blefari

SEPTEMBER 28, 2023

Make your data stack take-off ( credits ) Hello, another edition of Data News. This week, we're going to take a step back and look at the current state of data platforms. What are the current trends and why are people fighting around the concept of the modern data stack. But you should temper your excitation.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Snowflake

MARCH 2, 2023

Snowflake enables organizations to be data-driven by offering an expansive set of features for creating performant, scalable, and reliable data pipelines that feed dashboards, machine learning models, and applications. But before data can be transformed and served or shared, it must be ingested from source systems.

Kafka 119

Kafka Data Ingestion Data Pipeline Cloud Storage 119

Lyft Engineering

MARCH 6, 2024

DEED In this post, we’ll cover how Lyft upgrades Python at scale — 1500+ repos spanning 150+ teams — and the latest iteration of the tools and strategy we’ve built to optimize both the overall time to upgrade and the work required from our engineers. Here’s your first assignment. Python, How Do I L̶o̶v̶e̶ Use Thee?

Python 72

Python Metadata Datasets Coding 72

Cloudera

MARCH 14, 2023

We just announced the general availability of Cloudera DataFlow Designer , bringing self-service data flow development to all CDP Public Cloud customers. In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities.

Data Pipeline 81

Data Pipeline Designing Kafka Metadata 81

Cloudera

JUNE 26, 2023

Here are some tips and tricks of the trade to prevent well-intended yet inappropriate data engineering and data science activities from cluttering or crashing the cluster. For data engineering and data science teams, CDSW is highly effective as a comprehensive platform that trains, develops, and deploys machine learning models.

ETL Tools 76

ETL Tools Programming Language Datasets Data Pipeline 76

Marc Lamberti

OCTOBER 20, 2023

Templates and Macros in Apache Airflow allow passing data to your DAGs at runtime. Imagine that you want to execute an SQL request with the execution date of your DAG. How can you do that? How can you use the DAG ID when you send notifications to know which DAG to look at? Why Airflow templating? You get it.

SQL 130

SQL Python Coding Metadata 130

Monte Carlo

MARCH 18, 2024

Data quality audits are a key way to ensure your business decisions are being informed by fresh, accurate, and up-to-date data. But, whether your data resides in a warehouse, lake, lakehouse, or on-prem, it’s important to take the time to assess the accuracy of your organization’s data.

Datasets 52

Datasets Data Pipeline BI Data 52

Snowflake

JANUARY 9, 2024

In today’s dynamic business landscape, numerous organizations are transitioning to the Snowflake Data Cloud, seeking more agile, secure and efficient solutions to manage and activate customer data. Data ingestion: Hakkoda leads the entire data ingestion process.

Data Ingestion 97

Data Ingestion Cloud Architecture Data Pipeline 97

Data Engineering Podcast

SEPTEMBER 4, 2022

Summary Data observability is a product category that has seen massive growth and adoption in recent years. Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Marc Lamberti

SEPTEMBER 24, 2023

Airflow taskgroups are meant to replace SubDAGs, the historical way of grouping your tasks. Visually group tasks to organize complex data pipelines. Grouping the tasks per step ( process and store ) Grouping the tasks per path ( a and b ) How you group your tasks is a matter of preference rather than a technical question.

Coding 130

Coding Python Process Data Pipeline 130

Towards Data Science

JULY 4, 2023

How Hamilton can help you write more maintainable Airflow DAGs An abstract representation of how Airflow & Hamilton relate. This post is written in collaboration with Thierry Jean and originally appeared here. Just to recap, Airflow is the industry standard to orchestrate data pipelines. Image from Pixabay.

Python 65

Python Data Science Machine Learning Coding 65

Knowledge Hut

DECEMBER 25, 2023

We are also required to know about DevOps, which is a practice of harmonizing development and operations whereby the entire pipeline from development, testing, deployment, continuous integration, and feedback is automated. These two in themselves are full-time jobs and most people make careers out of one of them.

Java 96

Java Database PostgreSQL Project 96

Monte Carlo

FEBRUARY 8, 2024

Data observability has been one of the hottest emerging data engineering technologies the last several years. This momentum shows no signs of stopping with data quality and reliability becoming a central topic in the data product and AI conversations taking place across organizations of all types and sizes.

BI 59

BI Data Pipeline Data Warehouse Data 59

DareData

FEBRUARY 28, 2024

In this post of the PyTorch Introduction, we’ll learn how to use custom datasets with PyTorch, particularly tabular, vision and text data PyTorch is one of the hottest libraries in the Deep Learning field right now. Learn how to work with non-linear activation functions and how to solve non-linear problems.

Datasets 52

Datasets Deep Learning Data Machine Learning 52

Knowledge Hut

DECEMBER 26, 2023

While some of these frameworks, languages, and software engineering tools might significantly speed up and simplify your work, others might leave you with much to regret. Your project's success depends on your ability to make the right decisions. Do you find it difficult to follow them? What is a Software Engineer?

Software Engineer 90

Software Engineer Software Engineering Engineering Java 90

dbt Developer Hub

AUGUST 2, 2023

Materialized views are now an out of the box materialization in your dbt project once you upgrade to the latest version of dbt v1.6 Materialized views are now an out of the box materialization in your dbt project once you upgrade to the latest version of dbt v1.6 They are a tool in your analytics engineering toolbox, one of many.

Datasets 52

Datasets Kafka SQL Cloud 52

Data Engineering Podcast

JUNE 29, 2020

This was a great conversation about the complexities of working in a niche domain of data analysis and how to build a pipeline of high quality data from collection to analysis. If you hand a book to a new data engineer, what wisdom would you add to it?

Data Collection 100

Data Collection Management High Quality Data Metadata 100

Cloudera

SEPTEMBER 21, 2021

Many customers looking at modernizing their pipeline orchestration have turned to Apache Airflow, a flexible and scalable workflow manager for data engineers. But now it has become very simple and secure with our release of the Cloudera Airflow provider , which gives users the best of Airflow and CDP data services.

Python 94

Python Cloud Data Workflow Accessible 94

DataKitchen

SEPTEMBER 20, 2021

Data organizations don’t always have the budget or schedule required for DataOps when conceived as a top-to-bottom, enterprise-wide transformational change. DataOps can and should be implemented in small steps that complement and build upon existing workflows and data pipelines. Figure 1: The four phases of Lean DataOps.

Data Pipeline 98

Data Pipeline Data Cleanse Process Architecture 98

Monte Carlo

AUGUST 15, 2023

Imagine you’ve been building houses with a hammer and nails for most of your career, and I gave you a nail gun. And while this analogy isn’t a perfect encapsulation of how some data teams operate after moving from on-premises to a modern data stack, it’s close. IMAGE COURTESY OF SHANE MURRAY. Let’s dive in.

Data Storage 88

Data Storage Cloud Metadata Media 88

Workfall

AUGUST 14, 2023

Reading Time: 7 minutes In today’s data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights.

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

DataKitchen

AUGUST 9, 2021

Below is our third post (3 of 5) on combining data mesh with DataOps to foster greater innovation while addressing the challenges of a decentralized architecture. We’ve talked about data mesh in organizational terms (see our first post, “ What is a Data Mesh? ”) and how team structure supports agility.

Pharmaceutical 98

Pharmaceutical Raw Data Data Lake Data 98

Scott Logic

APRIL 10, 2024

This blog post serves as a dev diary of the process, covering our challenges, contributions made and attempts to validate them. This blog post serves as a dev diary of the process, covering our challenges, contributions made and attempts to validate them.

Cloud Storage 52

Cloud Storage Cloud AWS Metadata 52

Lyft Engineering

NOVEMBER 29, 2023

Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data. For our customers, this means faster analytics on near real-time data and decision making. An example of how we use Druid rollup at Lyft.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

dbt Developer Hub

JULY 17, 2023

Whether you are creating your pipelines into dbt for the first time or just adding a new model once in a while, good documentation and testing should always be a priority for you and your team. How can we make this process faster and less painful? Why do we avoid it like the plague then?

Coding 52

Coding Database Programming Process 52

Towards Data Science

AUGUST 15, 2023

The modern data stacks allow you to do things differently, not just at a larger scale. Photo by Massimo Botturi on Unsplash Imagine you’ve been building houses with a hammer and nails for most of your career, and I gave you a nail gun. Take advantage of it. Image courtesy of Shane Murray and the author. Let’s dive in.

Cloud 83

Cloud Metadata Media Machine Learning 83

Workfall

DECEMBER 20, 2022

Reading Time: 10 minutes Refer to Part 2 of this blog, to know How to Write Unit Tests for Forms in an Angular 15 Application Using Jasmine? In this blog, we will cover: What is Unit Testing? About Angular About Jasmine Hands-on Conclusion What Is Unit Testing? Why Is Unit Testing Important?

Coding 52

Coding Project Programming Building 52

Workfall

JANUARY 17, 2024

Let’s find out how FastAPI makes it easy with pagination, breaking down content for a smoother ride. Reading Time: 12 minutes Navigating lots of info can be tricky. We will do a step-by-step implementation with FastAPI, a top-notch Python tool, which helps organize info better. Let’s get started! Hands-on Conclusion What is Pagination?

Python 76

Python MySQL Coding Database 76

DataKitchen

NOVEMBER 18, 2022

Part 3: Considering the Elements of Data Journeys. This is the third post in DataKitchen’s four-part series on DataOps Observability. As he thinks through the various journeys that data take in his company, Jason sees that his dashboard idea would require extracting or testing for events along the way.

Datasets 52

Datasets Data Lake Data Pipeline Database 52

DoorDash Engineering

JANUARY 23, 2024

At delivery creation, KafkaConsumer reads the data from Redis, creates a delivery assignment constraint (DAC) object, and then stores it in a DAC table via delivery service. Time-consuming process to add new preferences : DeepRed considers a lot of signals and preferences during delivery assignments.

Architecture 74

Architecture Metadata Bytes Systems 74

Marc Lamberti

JULY 6, 2021

A Helm chart is a collection of multiple Kubernetes YAML manifests describing every components of your application. That my friend, makes your deployments much easier than ever before. In today’s tutorial, you are going to discover how to run Airflow on Kubernetes within just 10 mins thanks to the official Helm chart.

Accessible 130

Accessible Accessibility Data Pipeline Building 130

Monte Carlo

APRIL 4, 2024

Monte Carlo launched the data observability category back in 2019, and over the last five years, data observability has been on a steady rise—from data quality nice-to-have to an essential component of your data and AI reliability strategy. What makes data observability different? Let’s talk about it.

Data 40

Data Data Lake Data Warehouse AWS 40

Knowledge Hut

FEBRUARY 6, 2024

DevOps is the practice of methodically monitoring numerous development-related areas, beginning with formulating an operation plan, carrying out development work involving integrating applications and testing, and finishing with deployment and operations. DevOps can incorporate several engineering best practices for successful operation.

Coding 52

Coding Systems Certification Designing 52