Ascend.io

DECEMBER 19, 2023

You’re extracting and loading data first, then transforming it in Snowflake’s cloud data warehouse. That’s what we call a data pipeline. These features make Snowflake a great solution for data engineering, but the question arises: why might you still need a separate ETL tool?

ETL Tools 52

ETL Tools IT Data Pipeline Data Warehouse 52

Data Engineering Podcast

OCTOBER 15, 2021

They also share their ambitions for the near future of adding data observability and data quality management features. Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

JULY 31, 2022

Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. The only thing worse than having bad data is not knowing that you have it.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 3, 2022

In this episode they explain how the utility is implemented to run quickly and how you can start using it in your own data workflows to ensure that your data warehouse isn’t missing any records from your source systems. Can you describe what the data diff tool is and the story behind it?

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode Gopal Erinjippurath discusses the data engineering challenges of building and serving those data sets, and how they are distilling complex climate information into consumable facts so you don’t have to be an expert to understand it. Data stacks are becoming more and more complex.

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

Data Engineering Weekly

OCTOBER 30, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. The highlights are that 59% of folks think data catalogs are sometimes helpful.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Knowledge Hut

SEPTEMBER 25, 2023

You will be in charge of creating and maintaining data pipelines, data storage solutions, data processing, and data integration to enable data-driven decision-making inside a company. They guarantee that the data is efficiently cleaned, converted, and loaded.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Knowledge Hut

DECEMBER 28, 2023

Today, organizations seek skilled professionals who can harness data’s power to drive informed decisions. As technology evolves, cloud platforms have emerged as the cornerstone of modern data management. Its comprehensive suite of services can handle data at scale. What is the Azure Data Engineer Certification?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. To start, can you share your definition of what constitutes a "Data Lakehouse"?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

JANUARY 1, 2022

In this episode Emily Riederer shares her work to create a controlled vocabulary for managing the semantic elements of the data managed by her team and encoding it in the schema definitions in her data warehouse. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Warehouse 100

Data Warehouse BI Data Workflow Coding 100

Data Engineering Podcast

JUNE 1, 2021

Summary Every part of the business relies on data, yet only a small team has the context and expertise to build and maintain workflows and data pipelines to transform, clean, and integrate it. RudderStack’s smart customer data pipeline is warehouse-first. task-based, data assets, etc.)

Data Pipeline 100

Data Pipeline Data Warehouse Data Workflow Data 100

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. These tools help organizations implement DataOps practices by providing a unified platform for data teams to collaborate, share, and manage their data assets.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Data Engineering Podcast

JANUARY 21, 2024

In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how he is applying the capabilities of cloud data warehouses to the challenge of building more comprehensive experiences for end-users through a modern customer data platform (CDP).

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Monte Carlo

MAY 14, 2024

Delta Lake : Released by Databricks in 2019, Delta Lake was created to bring reliability and robustness to data lakes, incorporating ACID (Atomicity, Consistency, Isolation, Durability) transactions into Apache Spark to maintain data integrity across complex transformations and updates.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Can you describe what your conception of a data contract is?

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

APRIL 3, 2022

Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. In this episode Satish Jayanthi explains how he is building a framework to allow enterprises to move quickly while maintaining guardrails for data workflows.

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Data Engineering Podcast

NOVEMBER 14, 2021

In this episode Michael Harper advocates for proactive data quality and starting with the source, rather than being reactive and having to work backwards from when a problem is found. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days.

Data Warehouse 100

Data Warehouse BI Data Workflow Data 100

Data Engineering Podcast

OCTOBER 29, 2021

In this episode Ahmed Elsamadisi shares his journey to find a more flexible and universal data model in the form of the "activity schema" that is powering the Narrator platform, and how it has allowed his customers to perform self-service exploration of their business domains without being blocked by schema evolution in the data warehouse.

Data Warehouse 100

Data Warehouse BI Data Workflow Data Pipeline 100

Data Engineering Podcast

NOVEMBER 27, 2021

If you are having trouble answering questions for your business with the data that you generate and collect internally, then it is definitely worthwhile to explore the information available from external sources. Start trusting your data with Monte Carlo today! No more scripts, just SQL. Who are your target customers?

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Meltano

OCTOBER 5, 2022

We’ll explain what a data engineer is, what the job entails, and how to become a data engineer. Plus, we’ll explain how data engineers use Meltano, our DataOps platform, for efficient data management. What Is Data Engineering?

Data Engineering 52

Data Engineering Data Engineer Engineering Raw Data 52

Data Engineering Weekly

JANUARY 15, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Take control of your customer data today.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is the role of a Data Engineer? Let us now understand the basic responsibilities of a Data engineer.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Meltano

OCTOBER 5, 2022

In the same way, a DataOps engineer designs the data assembly line that enables data scientists to derive insights from data analytics faster and with fewer errors. DataOps engineers improve the speed and quality of the data development process by applying DevOps principles to data workflow, known as DataOps.

Engineering 52

Engineering Raw Data Data Pipeline ETL Tools 52

DataKitchen

APRIL 13, 2021

Airflow — An open-source platform to programmatically author, schedule, and monitor data pipelines. Apache Oozie — An open-source workflow scheduler system to manage Apache Hadoop jobs. They make it easy to deploy and manage your own Apache Airflow webserver, so you can get straight to writing workflows.

Consulting 123

Consulting Machine Learning Data Science Data Pipeline 123

AltexSoft

MARCH 14, 2023

As the volume and complexity of data continue to grow, organizations seek faster, more efficient, and cost-effective ways to manage and analyze data. In recent years, cloud-based data warehouses have revolutionized data processing with their advanced massively parallel processing (MPP) capabilities and SQL support.

IT 59

IT Data Warehouse Data Governance Data Lake 59