Data Engineering Podcast

APRIL 19, 2021

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Lots, buildings, units. Lots, buildings, units.

Data Pipeline 100

Data Pipeline Machine Learning Data Warehouse Datasets 100

Monte Carlo

JULY 27, 2023

For every asset…either a table or a report, it is to be tagged and assigned to a team. However, filtering and grouping queries by warehouse, schema, destination table, dbt model, or Airflow job is typically not as easy. The approach we followed was first to fix the tracking,” said Bushar at BlaBlaCar.

Data 52

Data Architecture Data Warehouse Data Engineering 52

Knowledge Hut

JUNE 20, 2023

Understanding how to construct data pipelines utilizing tools like Apache NiFi, Apache Kafka, and Apache Airflow is crucial. Knowing to branch, merge, and tag ideas and experience using version control systems like Git is crucial. Being familiar with cloud-based computing and storage platforms like Azure, AWS, and GCP is critical.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Monte Carlo

MARCH 12, 2024

DAGs are the means of orchestrating all of these moving parts, and two of the most popular solutions for creating and executing these DAGs are Apache Airflow and Databricks Workflows. For Airflow lineage, Monte Carlo relies on query tagging to ingest DAGs and tasks related to tables. How Does It Work?

BI 52

BI Metadata Data Pipeline Data Engineering 52

AltexSoft

NOVEMBER 7, 2022

But apparently, things were much more difficult before Apache Airflow appeared. This article covers Airflow’s pros and gives a clue why, despite all its virtues, it’s not a silver bullet. What is Apache Airflow? Source: Apache Airflow. No wonder, they represent over 54 percent of Apache Airflow active users.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Zalando Engineering

JUNE 9, 2022

The ROI pipeline is a batch based data- and machine learning pipeline powered by Databricks Spark and orchestrated by Apache Airflow. As mentioned earlier, we are using Airflow to orchestrate the overall pipeline. The Airflow code is stored in a github repository. The following section explains the problem in more depth.

Database 52

Database Coding Python AWS 52

ProjectPro

NOVEMBER 14, 2015

Big Data technologies used: AWS EC2, AWS S3, Flume, Spark, Spark Sql, Tableau, Airflow Big Data Architecture: This implementation is deployed on AWS EC2 and uses flume for ingestion, S3 as a data store, Spark SQL tables for processing, Tableau for visualization, and Airflow for orchestration.

Hadoop 52

Hadoop Project Big Data Healthcare 52

ProjectPro

FEBRUARY 18, 2022

The data includes video title, channel title, publishing time, tags, views, likes and dislikes, description, etc. Source Code- Build an End-to-End ETL Pipeline on AWS EMR Cluster AWS Snowflake Data Pipeline using Kinesis and Airflow For this ETL project, create a data pipeline starting with EC2 logs.

Project 52

Project AWS Kafka Healthcare 52

Airbnb Tech

APRIL 13, 2021

Apache Airflow , Apache Hive, Apache Spark ) and extensive analytics infrastructure (i.e., Airbnb CUR Pipeline The loading and transformation of raw CUR files into the Airbnb CUR Foundation is performed in an Airflow pipeline, which runs daily. Project Name  — This is a user-defined tag which is surfaced in the CUR data.

AWS 52

AWS Raw Data Amazon Web Services Cloud 52

Databand.ai

JULY 28, 2022

In this very simplified example, we can see an ELT: Some pipeline tasks, probably running by Airflow , are scraping external data sources and collecting data from there. Lineage by data tagging: A transformation engine that tags every movement or change in data allows for lineage by data tagging.

Metadata 52

Metadata Data Lake Datasets Data Warehouse 52

Monte Carlo

MARCH 1, 2022

With a single line of code, data engineers can add coverage for Airflow jobs, external tables, S3 buckets, Salesforce applications, HubSpot reports, internally built reports, and other endpoints. Downstream reporting beyond the business intelligence layer. Aggregating metadata. Programmatically spin up custom data observability monitors.

Python 52

Python Metadata Business Intelligence Data Engineering 52

Monte Carlo

FEBRUARY 8, 2024

If you had an Airflow job that caused your data to fail, the real culprit wouldn’t be the data but a system issue. Airflow Operator Can monitoring and alerting be configured from within your Airflow jobs, without breaking a workflow? Sometimes the issue is hidden within this web of dependencies.

BI 59

BI Data Pipeline Data Warehouse Data 59

DareData

AUGUST 12, 2021

Every data artifact should be clearly described, and tagged with short and easily understandable terms. You can, for example, supply data engineers with pre-built forms that contain a set of agreed-upon tags and a few free-text fields for descriptions. There are a few ways to make the documentation process less cumbersome.

Transportation 52

Transportation Data Lake Data Warehouse Data 52

ProjectPro

NOVEMBER 30, 2021

These include self-contained toolkits like Pygrametl, Apache Airflow, Luigi for workflow management, and Pandas for data movement and processing. ETL Pipeline Airflow Best Practices Here are some best practices for maintaining an Airflow ETL pipeline. Choose the Best Provider of Airflow ETL Services.

Process 52

Process Data Pipeline Data Warehouse AWS 52

phData: Data Engineering

JANUARY 3, 2022

The biggest gain with using Git over Subversion is that your developer’s branching and tagging can be separate from the central repository. First off, you have to define a branching and tagging strategy which takes time to do well. You can tag default (latest) containers along with specific versions of your application.

IT 52

IT AWS Software Engineer Software Engineering 52

ProjectPro

AUGUST 24, 2021

Project Idea: We’ll explore the usage of Apache Airflow for managing workflows. Source Code: Visualize Daily Wikipedia Trends with Hive, Zeppelin, and Airflow (projectpro.io) 7) Data Aggregation Data Aggregation refers to collecting data from multiple sources and drawing insightful conclusions from it.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

ProjectPro

JANUARY 25, 2022

PySpark, Hive, HDFS, Kafka, NiFi, Airflow, Tableau, and AWS QuickSight are the technologies used in this project. This Twitter Support project aims to listen to live tweets with needed tags and publish them to a Kafka topic. It requires using NiFi to ingest real-time streaming data from an external API.

Big Data 52

Big Data Data Process Process Kafka 52

AltexSoft

SEPTEMBER 21, 2023

With an active discussion forum boasting 129,375 users, community members can easily find answers to their questions across various topic categories and tags. The rapid response time, with questions often answered within approximately an hour, testifies to the high level of the community’s engagement.

Engineering 52

Engineering NoSQL Programming Language Java 52

AltexSoft

JULY 18, 2023

Just make sure to tag your question with ‘apache-spark’ to get the right attention. They have separate lists for user questions and development discussions. Stack Overflow : It is a well-known resource for programmers of all skill levels. You can use it to ask questions and get high-quality answers.

Big Data 52

Big Data Data Process Process Hadoop 52

Monte Carlo

APRIL 21, 2021

Daniel and her team now manage over 30,000 tables in Big Query and leverage a data tech stack that includes Monte Carlo, Google Cloud Platform, Airflow, Rivery, and other enterprise tools, conducting experiments and enabling the business to base decisions on data in real-time. “We We have a lot of data,” Daniel said. “It

Google Cloud 40

Google Cloud Data Retail BI 40

Monte Carlo

JANUARY 23, 2023

Data teams must now determine if it is: A systems issue : Did an Airflow job not run? If a failed Airflow job created a freshness issue? Additionally, tables can be tagged (perhaps to denote its reliability level) with descriptions perhaps (to include information on SLAs and other relevant information).

IT 52

IT BI Government Data Governance 52

Monte Carlo

MAY 11, 2023

And these are also the same APIs that power Monte Carlo’s web-based application, SDK CLI, and Airflow provider which we’ll get into soon enough, don’t you worry. Airflow provider Want to bring Monte Carlo utilities into your pipeline? What does it mean to be “developer-friendly”? Extracting insights back into your data platform.

Data Engineering 52

Data Engineering Data Engineer Engineering IT 52

Robinhood

FEBRUARY 7, 2024

We chose Apache Airflow for workflow management, Amazon (AWS) S3 for cloud-based storage, and Hive Metastore as our data catalog. For production jobs, we built libraries to trigger spark-submit from Airflow workers packaged with application code. Job Submission Process The job submission process was key.

Process 75

Process Hadoop Architecture Accessible 75

Monte Carlo

MAY 17, 2023

For example, Fivetran (ingestion), dbt (transformation), and Airflow (orchestration) all have these capabilities. We knew we were missing a lot of data and wanted to keep better track of our website through Google Analytics and Google Tag Manager. We had complicated queries, many duplications, inconsistent logic—it was quite a mess.

Data 52

Data Data Pipeline Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

For example, Fivetran (ingestion), dbt (transformation), and Airflow (orchestration) all have these capabilities. We knew we were missing a lot of data and wanted to keep better track of our website through Google Analytics and Google Tag Manager. We had complicated queries, many duplications, inconsistent logic—it was quite a mess.

Data Pipeline 52

Data Pipeline Data Data Engineering Data Engineer 52