dbt Developer Hub

APRIL 30, 2023

My next gig was in consulting where I bootstrapped my way into data engineering and had to learn the whole gamut below. Armed with them, I finally saw the magic of the modern data stack and what problems it could solve. Armed with them, I finally saw the magic of the modern data stack and what problems it could solve.

Data Engineering 64

Data Engineering Data Engineer Engineering Finance 64

Grouparoo

OCTOBER 6, 2021

It was a fun experience and I think we made a good choice by picking 97 Things Every Data Engineer Should Know. This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Advancing Analytics: Data Engineering

JUNE 21, 2019

Hear me out – back in the on-premises days we had data loading processes that connect directly to our source system databases and perform huge data extract queries as the start of one long, monolithic data pipeline, resulting in our data warehouse. Till next time.

Data Warehouse 52

Data Warehouse Business Intelligence Data Data Validation 52

Ascend.io

AUGUST 31, 2023

Secondly , the rise of data lakes that catalyzed the transition from ELT to ELT and paved the way for niche paradigms such as Reverse ETL and Zero-ETL. Still, these methods have been overshadowed by EtLT — the predominant approach reshaping today’s data landscape. Read More: What is ETL?

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Databand.ai

AUGUST 30, 2023

Here are some of the reasons why DataOps tools are important: Improved Collaboration DataOps tools enable better collaboration between data teams, including data engineers, data scientists, and data analysts. One of the key features of IBM Databand is its ability to automate data validation and anomaly detection.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Airbnb Tech

NOVEMBER 28, 2023

However, for all of our uncertified data, which remained the majority of our offline data, we lacked visibility into its quality and didn’t have clear mechanisms for up-leveling it. How could we scale the hard-fought wins and best practices of Midas across our entire data warehouse?

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

ProjectPro

FEBRUARY 8, 2023

ETL stands for Extract, Transform, and Load, which involves extracting data from various sources, transforming the data into a format suitable for analysis, and loading the data into a destination system such as a data warehouse. Focussed on designing, building, and maintaining large-scale data processing systems.

ETL Tools 52

ETL Tools Data Cleanse Data Warehouse Big Data 52

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 82

Data Warehouse Database-centric Metadata Cloud 82

Data Engineering Podcast

JUNE 16, 2019

This conversation was useful for getting a better idea of the challenges that exist in large scale data analytics, and the current state of the tradeoffs between data lakes and data warehouses in the cloud. Support the show and get your data projects in order! When is Delta Lake the wrong choice?

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Meltano

JANUARY 25, 2023

A curated list of the top 9 must read blogs on data. The data world is in turmoil and lots of exciting things happen every day, week and year. At Meltano we’re ourselves avid users of data, data engineers, data PMs, and data enthusiasts through and through. Happy reading!

Pipeline-centric 52

Pipeline-centric Database-centric Data Warehouse SQL 52

Knowledge Hut

APRIL 19, 2023

If you want to learn more about ETL processes check out this Data Engineer course. It is the process of extracting data from various sources, transforming it into a format suitable for analysis, and loading it into a target database or data warehouse. What is ETL? ETL stands for Extract, Transform, and Load.

Data Warehouse 52

Data Warehouse Business Intelligence ETL Tools Data Pipeline 52

Databand.ai

JULY 19, 2023

A Beginner’s Guide [SQ] Niv Sluzki July 19, 2023 ELT is a data processing method that involves extracting data from its source, loading it into a database or data warehouse, and then later transforming it into a format that suits business needs. The data is loaded as-is, without any transformation.

Data Cleanse 52

Data Cleanse Raw Data Data Storage Data Warehouse 52

Monte Carlo

FEBRUARY 15, 2023

So, creating a reliable data platform that delivers bankable business value involves two critical components: Discovering data quality issues, And resolving data quality issues. Data testing is the first step in that equation. And discovering the impact of a data anomaly or change in the source data was just as tedious.

Data Validation 52

Data Validation Data Data Engineering Data Engineer 52

Monte Carlo

FEBRUARY 15, 2023

So, creating a reliable data platform that delivers bankable business value involves two critical components: Discovering data quality issues, And resolving data quality issues. Data testing is the first step in that equation. And discovering the impact of a data anomaly or change in the source data was just as tedious.

Data Validation 52

Data Validation Data Data Engineering Data Engineer 52

Striim

JANUARY 4, 2023

Companies need to analyze large volumes of datasets, leading to an increase in data producers and consumers within their IT infrastructures. These companies collect data from production applications and B2B SaaS tools (e.g., This data makes its way into a data repository, like a data warehouse (e.g.,

PostgreSQL 52

PostgreSQL Data Warehouse Data Lake Data 52

Monte Carlo

MAY 26, 2023

By the end, you’ll be well-prepared to conduct your own data symphony, ensuring every ‘instrument’ plays in tune and in harmony. What Do We Mean When We Talk About Data Consistency? In other words, every time a piece of data is recorded, no matter where in the system that happens, it should always look the same.

Datasets 52

Datasets Data Warehouse Data Validation Data 52

Ascend.io

SEPTEMBER 19, 2023

In the world of data engineering, the ETL (Extract, Transform, Load) approach has been the cornerstone for managing and processing data. However , the traditional methods of executing ETL are increasingly struggling to meet the escalating demands of today’s data-intensive environments.

ETL Tools 52

ETL Tools ETL Method Utilities Data Engineering 52

DataKitchen

SEPTEMBER 21, 2023

Bridging the Gap: How ‘Data in Place’ and ‘Data in Use’ Define Complete Data Observability In a world where 97% of data engineers report burnout and crisis mode seems to be the default setting for data teams, a Zen-like calm feels like an unattainable dream.

Raw Data 67

Raw Data Data Business Intelligence High Quality Data 67

Airbnb Tech

NOVEMBER 24, 2020

During this transformation, Airbnb experienced the typical growth challenges that most companies do, including those that affect the data warehouse. In the first post of this series, we shared an overview of how we evolved our organization and technology standards to address the data quality challenges faced during hyper growth.

Data Warehouse 52

Data Warehouse Certification Data Pipeline Data 52

Advancing Analytics: Data Engineering

JUNE 10, 2019

After the horror that was the “data silo” days, with clumps of data living in Access databases, Excel spreadsheets and isolated data stores, we’ve had a pretty good run with the classic Kimball data warehouse. However, a data warehouse is a large, sanitary data store. Till next time.

Data Lake 52

Data Lake Data Warehouse Data Data Validation 52

dbt Developer Hub

SEPTEMBER 28, 2022

That’s how life feels without an analytics engineer on your team. So let’s say that you have a business question, you have the raw data in your data warehouse , and you’ve got dbt up and running. When a data engineer gets stuck writing a model, they’ll need to reach out to the analyst for some additional discovery.

Engineering 52

Engineering Raw Data Datasets BI 52

Databand.ai

JULY 19, 2023

Despite these challenges, proper data acquisition is essential to ensure the data’s integrity and usefulness. Data Validation In this phase, the data that has been acquired is checked for accuracy and consistency.

Data Ingestion 52

Data Ingestion Process Data Cleanse Data Governance 52

Monte Carlo

MARCH 24, 2023

On the face of it, data migration should be fairly straightforward – just pick up your data and move it where it needs to go. After all, as data engineers aren’t we migrating data all the time within our ETL pipelines? Here are some tips for mitigating some of the risks of data migration.

Data Warehouse 52

Data Warehouse AWS Cloud Data Governance 52

Knowledge Hut

JANUARY 29, 2024

By mastering the art of Data Wrangling, individuals and organizations alike can unlock the true power of data, transforming it from a tangled web of information into a valuable asset that drives innovation, fuels growth, and guides them toward a future were data reigns supreme. What are the six steps of data wrangling?

Raw Data 52

Raw Data Data Mining Data Preparation Structured Data 52

Ascend.io

JUNE 8, 2023

Whether it be the marketing team seeking customer insights, the finance team working on budgeting, or executives crafting business strategies, data needs to be shared in a manner that aligns with their specific objectives and competencies. It is the stage where data truly becomes a product, delivering tangible value to its end users.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

AltexSoft

NOVEMBER 23, 2021

Before we get into more detail, let’s determine how data virtualization is different from another, more common data integration technique — data consolidation. Data virtualization vs data consolidation. You can learn more about how such data pipelines are built in our video about data engineering.

Process 69

Process Data Lake Metadata Data Warehouse 69

Monte Carlo

OCTOBER 1, 2020

In this article we introduce “Key Assets”, a new approach taken by the best data teams to surface your most important data assets for quick and reliable insights. Have you been 3/4ths of the way done with a data warehouse migration only to discover that you don’t know which data assets are right and which ones are wrong?

Data Warehouse 40

Data Warehouse Datasets Data Validation Machine Learning 40

ProjectPro

NOVEMBER 30, 2021

In extract-transform-load (ETL), data is obtained from multiple sources, transformed, and stored in a single data warehouse, with access to data analysts , data scientists , and business analysts for data visualization and statistical analysis model building, forecasting, etc.

Process 52

Process Data Pipeline Data Warehouse AWS 52

RandomTrees

SEPTEMBER 6, 2020

Snowflake Overview A data warehouse is a critical part of any business organization. Lot of cloud-based data warehouses are available in the market today, out of which let us focus on Snowflake. Snowflake is an analytical data warehouse that is provided as Software-as-a-Service (SaaS).

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

DataKitchen

APRIL 13, 2021

Piperr.io — Pre-built data pipelines across enterprise stakeholders, from IT to analytics, tech, data science and LoBs. Prefect Technologies — Open-source data engineering platform that builds, tests, and runs data workflows. Genie — Distributed big data orchestration service by Netflix. Telm.ai — Telm.ai

Consulting 120

Consulting Machine Learning Data Science Data Pipeline 120

Knowledge Hut

OCTOBER 30, 2023

Data Analysis: Perform basic data analysis and calculations using DAX functions under the guidance of senior team members. Data Integration: Assist in integrating data from multiple sources into Power BI, ensuring data consistency and accuracy. Define data architecture standards and best practices.

BI 52

BI Business Intelligence Data Cleanse Business Analyst 52

phData: Data Engineering

JANUARY 3, 2022

Software engineering practices define how to reliably and effectively build software and data products, delivering value faster to your customers. In this post, we will explore the complexities involved with software engineering with a focus on data engineering and data operations (DataOps).

IT 52

IT AWS Software Engineer Software Engineering 52

ProjectPro

JANUARY 31, 2023

Big Data Interview Questions and Answers Based on Job Role With the help of ProjectPro experts, we have compiled a list of interview questions on big data based on several job roles, including big data tester, big data developer, big data architect, and big data engineer. What are its benefits?

Big Data 52

Big Data Hadoop AWS Relational Database 52

ProjectPro

AUGUST 21, 2015

When compaction takes place, the old data will take the new block size so that the existing data is read correctly. HBase and Hive both are completely different hadoop based technologies-Hive is a data warehouse infrastructure on top of Hadoop whereas HBase is a NoSQL key value store that runs on top of Hadoop.

Hadoop 40

Hadoop Big Data Metadata Java 40