Ascend.io

FEBRUARY 28, 2024

The rise of generative AI is changing more than just technology; it’s reshaping our professional landscapes — and yes, data engineering is directly experiencing the impact. How does AI recalibrate the workload and priorities of data teams? How can data engineers harness the power of AI?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? Data science as a discipline was going through its adolescence of self-affirming and defining itself. We were pioneers.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Weekly

NOVEMBER 13, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering 52

Data Engineering Data Engineer Engineering Programming Language 52

Airbnb Tech

NOVEMBER 28, 2023

However, for all of our uncertified data, which remained the majority of our offline data, we lacked visibility into its quality and didn’t have clear mechanisms for up-leveling it. How could we scale the hard-fought wins and best practices of Midas across our entire data warehouse?

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. Imagine a team of skilled data engineers on an exciting quest to transform raw data into a treasure trove of insights.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. But they should!

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. But they should!

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Lyft Engineering

OCTOBER 3, 2023

High-quality data is necessary for the success of every data-driven company. It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Cloudera

OCTOBER 7, 2022

We’re excited to announce the general availability of the open source adapters for dbt for all the engines in CDP — Apache Hive , Apache Impala , and Apache Spark, with added support for Apache Livy and Cloudera Data Engineering. The Open Data Lakehouse .

Data Warehouse 81

Data Warehouse Data Lake Government High Quality Data 81

Monte Carlo

JULY 27, 2023

These specialists are also commonly referred to as data reliability engineers. To be successful in their role, data quality engineers will need to gather data quality requirements (mentioned in 65% of job postings) from relevant stakeholders.

Engineering 52

Engineering Healthcare Scala Data Warehouse 52

Monte Carlo

AUGUST 31, 2023

Understanding the “rise of data downtime” With a greater focus on monetizing data coupled with the ever present desire to increase data accuracy, we need to better understand some of the factors that can lead to data downtime. We’ll take a closer look at variables that can impact your data next.

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Databand.ai

JULY 5, 2023

What Are Data Observability Tools? Data observability tools are software solutions that oversee, analyze, and improve the performance of data pipelines. These tools offer data engineers insight into the health of their data infrastructure, by giving visibility into crucial metrics like latency, throughput, and error rates.

Data Pipeline 52

Data Pipeline Data Lake Data Warehouse Datasets 52

Monte Carlo

FEBRUARY 20, 2024

Choosing one tool over another isn’t just about the features it offers today; it’s a bet on the future of how data will flow within organizations. Matillion is an all-in-one ETL solution that stands out for its ability to handle complex data transformation tasks in all the popular cloud data warehouses.

Data Ingestion 52

Data Ingestion Google Cloud Kafka AWS 52

DataKitchen

SEPTEMBER 21, 2023

Bridging the Gap: How ‘Data in Place’ and ‘Data in Use’ Define Complete Data Observability In a world where 97% of data engineers report burnout and crisis mode seems to be the default setting for data teams, a Zen-like calm feels like an unattainable dream. What is Data in Use?

Raw Data 67

Raw Data Data Business Intelligence High Quality Data 67

Towards Data Science

MARCH 29, 2023

The Data Warehouse Toolkit (Kimball & Ross) The Data Warehouse Toolkit, 3rd Edition - Kimball Group I’m not going to bury the lead. If you work in data, you at the very least need to be familiar with dimensional modeling concepts, and I personally don’t think there’s a better way than by going straight to the source.

Data Warehouse 66

Data Warehouse BI Healthcare Database 66

Monte Carlo

OCTOBER 11, 2022

We’ll then discuss how they can be avoided with an organizational commitment to high-quality data. Imagine this You’re a data scientist with a swagger working on a predictive model to optimize a fast-growing company’s digital marketing spend. The data warehouse is a mess and devoid of semantic meaning.

IT 52

IT Datasets Data Warehouse Data Analysis 52

Airbnb Tech

NOVEMBER 24, 2020

During this transformation, Airbnb experienced the typical growth challenges that most companies do, including those that affect the data warehouse. In the first post of this series, we shared an overview of how we evolved our organization and technology standards to address the data quality challenges faced during hyper growth.

Data Warehouse 52

Data Warehouse Certification Data Pipeline Data 52

Monte Carlo

FEBRUARY 8, 2024

Data observability has been one of the hottest emerging data engineering technologies the last several years. This momentum shows no signs of stopping with data quality and reliability becoming a central topic in the data product and AI conversations taking place across organizations of all types and sizes.

BI 59

BI Data Pipeline Data Warehouse Data 59

Monte Carlo

AUGUST 10, 2023

Taken together, these three pillars and application management (APM) solutions like DataDog or Splunk, gave DevOps teams and software engineers valuable awareness and insights to predict future behavior, and in turn, trust their IT systems to meet SLAs. Enter data observability. Why is data observability important?

Data Pipeline 98

Data Pipeline Software Engineer Software Engineering Machine Learning 98

Monte Carlo

MAY 25, 2023

However, once data platforms scale beyond a certain point, managing jobs will quickly become unwieldy by in-house standards. Data orchestration supports the data engineering lifecycle by bringing automation to data pipelines. Source : Fundamentals of Data Engineering by Joe Reis and Matt Housley.

Data Pipeline 52

Data Pipeline Data Workflow Data Data Governance 52

Ascend.io

JUNE 8, 2023

Whether it be the marketing team seeking customer insights, the finance team working on budgeting, or executives crafting business strategies, data needs to be shared in a manner that aligns with their specific objectives and competencies. It is the stage where data truly becomes a product, delivering tangible value to its end users.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

DataKitchen

NOVEMBER 4, 2021

It often takes months to progress from a data lake to the final delivery of insights. One data engineer called it the “last mile problem.” . In our many conversations about data analytics, data engineers, analysts and scientists have verbalized the difficulty of creating analytics in the modern enterprise.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Monte Carlo

JANUARY 31, 2023

“Given that we are in the financial sector, we see quite disparate use-cases for both analytical and operational reporting which require high-levels of accuracy” says Checkout.com Senior Data Engineer Martynas Matimaitis. ML-based anomaly detection beats manual threshold basically any day of the week,” says Martynas. “If

Data 52

Data High Quality Data Data Engineering Data Engineer 52

Monte Carlo

MARCH 28, 2024

For example, when you think about a data warehouse , it’s really just a codebase—primarily composed of SQL—that’s serving internal customers like other analysts, data scientists, and product managers who are using that data to go and make business decisions. If data is your product, then it needs to be high quality.

Data 52

Data High Quality Data Machine Learning Accessible 52

Monte Carlo

AUGUST 8, 2023

By applying rules and checks, data validation testing verifies the data meets predefined standards and business requirements to help prevent data quality issues and data downtime. From this perspective, the data validation process looks a lot like any other DataOps process.

Data Validation 52

Data Validation Data Pipeline SQL Data 52

Monte Carlo

APRIL 26, 2022

DataOps is a discipline that merges data engineering and data science teams to support an organization’s data needs, in a similar way to how DevOps helped scale software engineering. One of the primary focuses of DataOps is data engineering automation. What is DataOps? Invest in self-service tooling.

IT 64

IT Data Pipeline Data Engineering Data Engineer 64

Monte Carlo

MAY 17, 2023

Data observability, an organization’s ability to fully understand the health and quality of the data in their systems, has become one of the hottest technologies in modern data engineering. Data Warehouse (Or Lakehouse) Migration 34. Integrate Data Stacks Post Merger 35.

Data 52

Data Data Pipeline Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

Data observability, an organization’s ability to fully understand the health and quality of the data in their systems, has become one of the hottest technologies in modern data engineering. Data warehouse (or Lakehouse) migration 34. Integrate Data Stacks Post Merger 35.

Data Pipeline 52

Data Pipeline Data Data Engineering Data Engineer 52

Monte Carlo

APRIL 21, 2022

Adopting a cloud data warehouse like Snowflake is an important investment for any organization that wants to get the most value out of their data. Snowflake Data Quality Feature #3: Object Tagging Object Tagging in Snowflake. Data quality isn’t simply about accuracy—it’s about providing value to the business.

Metadata 52

Metadata Bytes Government Data 52

DataKitchen

JANUARY 3, 2022

If the IT or data engineering team can’t respond with an enabling data platform in the required time frame, the business analyst does the necessary data work themselves. This ad hoc data engineering work often means coping with numerous data tables and diverse data sets using Alteryx, SQL, Excel or similar tools. .

Business Analyst 98

Business Analyst Data Lake Consulting Data Analytics 98

Databand.ai

JANUARY 24, 2023

DataOps helps ensure organizations make decisions based on sound data. Previously, organizations have grabbed their full dataset across multiple environments, put it all into a data warehouse, and surfaced information from there. In doing so, it allows teams to identify problems faster and, therefore, deliver solutions faster.

Retail 52

Retail Banking Government Manufacturing 52

Monte Carlo

APRIL 24, 2023

During data ingestion, raw data is extracted from sources and ferried to either a staging server for transformation or directly into the storage level of your data stack—usually in the form of a data warehouse or data lake. There are two primary types of raw data. When should you invest in dbt Cloud?

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Monte Carlo

AUGUST 17, 2021

And your entire data engineering team is all in on this “data as code” movement. The flip side of this data-driven coin: your stakeholders (CEO and CTO included) ping you nearly every other hour to ask you: “is my data up-to-date?” said Pablo Recio, a Data Engineer, Hotjar.

Insurance 40

Insurance Retail Data Pipeline Portfolio 40

DareData

JULY 29, 2020

With these points in mind, I argue that the biggest hurdle to the widespread adoption of these advanced techniques in the healthcare industry is not intrinsic to the industry itself, or in any way related to its practitioners or patients, but simply the current lack of high-quality data pipelines.

Healthcare 52

Healthcare Data Pipeline Medical Pipeline-centric 52

Monte Carlo

AUGUST 30, 2023

Data testing is the first step in many data engineers’ journey toward reliable data. dbt (data build tool) is a SQL-based command-line tool that offers native testing features. dbt also happens to be an incredibly handy tool for running many data quality tests.

SQL 52

SQL Datasets Database High Quality Data 52

AltexSoft

AUGUST 22, 2022

A data fabric is an architecture design presented as an integration and orchestration layer built on top of multiple disjointed data sources like relational databases , data warehouses , data lakes, data marts , IoT , legacy systems, etc., to provide a unified view of all enterprise data.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52