Data Engineering Podcast

JUNE 5, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Data Security 100

Data Security Metadata MongoDB Scala 100

Databand.ai

JULY 19, 2023

Data lineage tools are not a new concept. However, their importance has grown significantly in recent years due to the increasing complexity of data architectures and the growing need for data governance and compliance. In this article: Why Are Data Lineage Tools Important?

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Monte Carlo

APRIL 24, 2023

Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption. Databricks Data Catalog and AWS Lake Formation are examples in this vein. See our post: Data Lakes vs. Data Warehouses.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Data Engineering Podcast

APRIL 5, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

AltexSoft

AUGUST 29, 2023

Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture. This structure is made efficient by data engineering practices that include object storage. Data warehouse vs. data lake in a nutshell.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

SEPTEMBER 23, 2018

This includes modeling the lifecycle of your information as a pipeline from the raw, messy, loosely structured records in your data lake, through a series of transformations and ultimately to your data warehouse. What is your opinion on the relative merits of a data warehouse vs a data lake and are they mutually exclusive?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Data Engineering Podcast

NOVEMBER 9, 2020

In this episode Grant Seward explains how he built Tree Schema to be an easy to use and cost effective option for organizations to build their data catalogs. He also shares the internal architecture, how he approached the design to make it accessible and easy to use, and how it autodiscovers the schemas and metadata for your source systems.

Building 100

Building PostgreSQL BI Metadata 100

AltexSoft

FEBRUARY 11, 2023

It serves as a foundation for the entire data management strategy and consists of multiple components including data pipelines; , on-premises and cloud storage facilities – data lakes , data warehouses , data hubs ;, data streaming and Big Data analytics solutions ( Hadoop , Spark , Kafka , etc.);

Data Architect 87

Data Architect Certification Generalist Big Data 87

Monte Carlo

SEPTEMBER 12, 2022

But with the rise of tools such as Segment, Fivetran, Meltano, and Airbyte, it’s become relatively easy for teams to bring all of their data from external sources into a centralized place like a data warehouse. Maxime points to the decentralization of data governance as a possible solution. “As

BI 52

BI Data Governance ETL Tools Data Warehouse 52

Cloudera

SEPTEMBER 15, 2022

A key area of focus for the symposium this year was the design and deployment of modern data platforms. Luke: Let’s talk about some of the fundamentals of modern data architecture. What is a data fabric? Mark: Gartner states that a data fabric “enables frictionless access and sharing of data in a distributed data environment.”

Data Lake 79

Data Lake Analytics Application Cloud Storage Architecture 79

ThoughtSpot

AUGUST 23, 2023

The ThoughtSpot layup For the last 11 years, we have been working towards our vision: allowing business users to get data insights through an intuitive interface, with minimal effort; and meanwhile, giving data teams robust control over the semantics of data, governance, and security controls.

BI 87

BI Metadata Data Warehouse Data Governance 87

Monte Carlo

SEPTEMBER 20, 2022

“The way that our data flows across our stack is pretty integral to the way the company runs,” said Dylan. “We We take the data from our warehouse, and we actually push it back into our operational tooling for use by the individual teams. But a growing company means growing data needs. Image courtesy of Prefect.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Cloudera

MAY 12, 2021

HYBRID & MULTI-CLOUD INNOVATION. If you are working in an organization that is driving business innovation by unlocking value from data in multiple environments — in the private cloud or across hybrid and multiple public clouds — we encourage you to consider entering this category. SECURITY AND GOVERNANCE LEADERSHIP.

Food 66

Food Healthcare Government Medical 66

Cloudera

FEBRUARY 16, 2022

Each workspace is associated with a collection of cloud resources. In the case of CDP Public Cloud, this includes virtual networking constructs and the data lake as provided by a combination of a Cloudera Shared Data Experience (SDX) and the underlying cloud storage. Figure 03: lineage.yaml.

Machine Learning 81

Machine Learning Algorithm Government Metadata 81

Monte Carlo

AUGUST 25, 2023

At the same time, 81% of IT leaders say their C-suite has mandated no additional spending or a reduction of cloud costs. Data teams need to balance the need for robust, powerful data platforms with increasing scrutiny on costs. But, the options for data storage are evolving quickly. Let’s dive in.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Precisely

JANUARY 25, 2024

But today’s cloud platforms offer similar capabilities, with the added benefits of flexibility, elasticity, and virtually unlimited scalability. The cloud helps companies lower their capital expenditures, enhance operational efficiency, and innovate faster. They’re also valued for their rock-solid reliability, boasting 99.999% uptime.

Metadata 69

Metadata Data Governance Relational Database Government 69

Monte Carlo

MARCH 14, 2023

Discovering the data mesh BlaBlaCar’s data stack was built to support their end goal of supporting self service analytics at scale. BlaBlaCar has been a data-driven company for a long time. In an effective data mesh, data governance tools need to be standardized,” said Kineret. Image courtesy of BlaBlaCar.

Transportation 40

Transportation Government Google Cloud Architecture 40

Monte Carlo

FEBRUARY 26, 2024

A data catalog is a constantly updated inventory of the universe of data assets within an organization. It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems.

Metadata 52

Metadata Government Data Data Governance 52

Databand.ai

AUGUST 30, 2023

Data Cataloging Data cataloging is the process of organizing and categorizing data assets to make them easily discoverable and understandable for data teams. This enables data teams to quickly and easily find the data they need for their analytics projects.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Towards Data Science

JULY 24, 2023

A data engineering manager at a Fortune 500 company expressed the pain of on-prem limitations to me by saying: “Our analysts were unable to run the queries they wanted to run when they wanted to run them. Why are these things related, and more importantly, why should data leaders care? Take data governance for example.

Metadata 97

Metadata Data Warehouse Data Lake Data Science 97

AltexSoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Data warehouse vs data lake vs data lakehouse: What’s the difference.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

phData: Data Engineering

AUGUST 4, 2023

Data storage is a vital aspect of any Snowflake Data Cloud database. Within Snowflake, data can either be stored locally or accessed from other cloud storage systems. In Snowflake, there are three different storage layers available, Database, Stage, and Cloud Storage.

Cloud Storage 52

Cloud Storage Google Cloud Amazon Web Services Data Storage 52

Monte Carlo

JUNE 14, 2023

Some organizations choose to still leverage an ETL pattern in the cloud, particularly for production pipelines where data contracts can help reduce data downtime. With cost and physical compute/storage limitations largely lifted, data engineers started to optimize data pipeline architecture for speed and agility.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

AltexSoft

NOVEMBER 23, 2021

Nowadays, all organizations need real-time data to make instant business decisions and bring value to their customers faster. But this data is all over the place: It lives in the cloud, on social media platforms, in operational systems, and on websites, to name a few. Data virtualization vs data consolidation.

Process 69

Process Data Lake Metadata Data Warehouse 69

AltexSoft

AUGUST 22, 2022

According to Flexera’s 2022 State of the Cloud Report , 89 percent of respondents have a multi-cloud strategy with 80 percent having a hybrid cloud approach in place. Organizations have data stored in public and private clouds, as well as in various on-premises data repositories. Source: Flexera.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

Cloudera

MARCH 6, 2018

People are gravitating to the analytics services of the large public cloud providers because the “house-brand” offerings seem to be the easiest choice. Instead, they have separate data stores and inconsistent (if any) frameworks for data governance, management, and security. But these offerings are not integrated.

Cloud 40

Cloud Metadata Big Data AWS 40

Monte Carlo

AUGUST 30, 2023

This helps ensure your data quality is consistent over time, given that new data isn’t always guaranteed to follow the same properties as old data. Once you have a set of reliable schema tests, use a tool like dbt Cloud, Apache Airflow, or a task scheduler to run the tests on a regular basis.

SQL 52

SQL Datasets Database High Quality Data 52

ProjectPro

AUGUST 24, 2021

Data Warehousing: Data warehousing utilizes and builds a warehouse for storing data. A data engineer interacts with this warehouse almost on an everyday basis. Data Analytics: A data engineer works with different teams who will leverage that data for business solutions.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

ProjectPro

JANUARY 31, 2022

As the demand for big data grows, an increasing number of businesses are turning to cloud data warehouses. The cloud is the only platform to handle today's colossal data volumes because of its flexibility and scalability. Table of Contents Snowflake Overview and Architecture What is Snowflake Data Warehouse?

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

AltexSoft

JUNE 26, 2023

Apache HBase and Apache Cassandra are well-known columnar technologies belonging to the Hadoop big data ecosystem; graph, intended for graph structures where data points are connected through defined relationships — like in Neo4J, Amazon Neptune, and OrientDB. The difference between data warehouses, lakes, and marts.

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

AltexSoft

OCTOBER 8, 2021

Data integration defines the process of collecting data from a number of disparate source systems and presenting it in a unified form within a centralized location like a data warehouse. So, why is data integration such a big deal? On-premise data integration tools. Cloud-based data integration tools.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

ProjectPro

FEBRUARY 8, 2017

The need for speed to use Hadoop for sentiment analysis and machine learning has fuelled the growth of hadoop based data stores like Kudu and adoption of faster databases like MemSQL and Exasol. 2) Big Data is no longer just Hadoop A common misconception is that Big Data and Hadoop are synonymous.

Big Data 40

Big Data Hadoop Data Lake Data Governance 40

Snowflake

MARCH 30, 2023

A 2016 data science report from data enrichment platform CrowdFlower found that data scientists spend around 80% of their time in data preparation (collecting, cleaning, and organizing of data) before they can even begin to build machine learning (ML) models to deliver business value.

Engineering 82

Engineering Raw Data Data Science Scala 82

AltexSoft

JULY 29, 2022

A master node called NameNode maintains metadata with critical information, controls user access to the data blocks, makes decisions on replications, and manages slaves. Instruments like Apache ZooKeeper and Apache Oozie help better coordinate operations, schedule jobs, and track metadata across a Hadoop cluster. Let’s see why.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

ProjectPro

NOVEMBER 30, 2021

In extract-transform-load (ETL), data is obtained from multiple sources, transformed, and stored in a single data warehouse, with access to data analysts , data scientists , and business analysts for data visualization and statistical analysis model building, forecasting, etc.

Process 52

Process Data Pipeline Data Warehouse AWS 52

Ascend.io

MARCH 14, 2023

In this article, you’re going to learn the following: What a data mesh is Why it gained momentum The five core features of data mesh Why a company might consider building one Let’s dive in! What Is a Data Mesh? It provides a more distributed, decentralized, and resilient approach to data management.

Government 52

Government Architecture Data Lake Data 52