Cloudera

SEPTEMBER 24, 2020

Some of the most powerful results come from combining complementary superpowers, and the “dynamic duo” of Apache Hive LLAP and Apache Impala, both included in Cloudera Data Warehouse , is further evidence of this. Both Impala and Hive can operate at an unprecedented and massive scale, with many petabytes of data.

Data Warehouse 68

Data Warehouse SQL Engineering Metadata 68

Data Engineering Podcast

NOVEMBER 27, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Missing data? Atlan is the metadata hub for your data ecosystem. Missing data? Stale dashboards?

Data Process 100

Data Process Process Metadata Business Intelligence 100

Data Engineering Podcast

DECEMBER 25, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Missing data? Again, be prepared to have metadata challenges especially. Struggling with broken pipelines? Stale dashboards?

Building 100

Building Metadata Business Intelligence Data Lake 100

Data Engineering Podcast

SEPTEMBER 11, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Interview Introduction How did you get involved in the area of data management?

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

JUNE 19, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. No more scripts, just SQL.

Systems 100

Systems Building Metadata Data Warehouse 100

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Podcast

JUNE 12, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Data Engineering Podcast

JUNE 26, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Data Engineering Podcast

NOVEMBER 20, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Towards Data Science

NOVEMBER 4, 2023

I’d like to discuss some popular Data engineering questions: Modern data engineering (DE). Does your DE work well enough to fuel advanced data pipelines and Business intelligence (BI)? Are your data pipelines efficient? Often it is a data warehouse solution (DWH) in the central part of our infrastructure.

Data Engineering 86

Data Engineering Data Engineer Engineering BI 86

Data Engineering Podcast

JULY 17, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

JULY 3, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 31, 2022

Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. Atlan is the metadata hub for your data ecosystem. Can you describe what Manta is and the story behind it?

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

phData: Data Engineering

SEPTEMBER 19, 2023

Want to learn more about data governance? Check out our Data Governance on Snowflake blog! Metadata Management Data modeling methodologies help in managing metadata within the data lake. Metadata describes the characteristics, attributes, and context of the data.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Data Engineering Podcast

JUNE 5, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Data Security 100

Data Security Metadata MongoDB Scala 100

AltexSoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Data warehouse vs data lake vs data lakehouse: What’s the difference.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 90

Data Lake Data Warehouse BI SQL 90

Snowflake

APRIL 5, 2023

Two different data modeling approaches—dimensional data modeling and Data Vault—each have their own pros and cons. Modernizing a data warehouse with Snowflake Data Cloud is a smart investment that can provide significant benefits to businesses of all sizes, today more than ever as data models become ever more complex.

Data Warehouse 52

Data Warehouse Healthcare Unstructured Data Metadata 52

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Data Engineering Podcast

NOVEMBER 20, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. Instead, Facebook came to realize that the work we were doing transcended classic business intelligence. The traditional best practices of data warehousing are loosing ground on a shifting stack.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

Data Engineering Podcast

JANUARY 15, 2022

Summary Applications of data have grown well beyond the venerable business intelligence dashboards that organizations have relied on for decades. You can observe your pipelines with built in metadata search and column level lineage. You can observe your pipelines with built in metadata search and column level lineage.

Engineering 100

Engineering Electronics ETL Tools Data Pipeline 100

Ascend.io

AUGUST 31, 2023

Secondly , the rise of data lakes that catalyzed the transition from ELT to ELT and paved the way for niche paradigms such as Reverse ETL and Zero-ETL. Still, these methods have been overshadowed by EtLT — the predominant approach reshaping today’s data landscape. Read More: What is ETL?

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Monte Carlo

NOVEMBER 15, 2022

In our effort to provide comprehensive, end-to-end data observability for cloud native data stacks across the world, Monte Carlo is committed to integrating with the tools and technologies most requested by our customers. Power BI, Microsoft’s leading business intelligence platform. Our latest integration?

BI 52

BI Data Warehouse Telecommunication Business Intelligence 52

Monte Carlo

JANUARY 23, 2024

The data warehouse is the mirror of the organization in many ways. If people don’t agree on what they call things in the data warehouse or what the definition of a metric is, then this lack of consensus will be reflected downstream. Data debt or data pipeline chaos. But maybe that’s OK.”

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

AltexSoft

SEPTEMBER 23, 2021

One of the innovative ways to address this problem is to build a data hub — a platform that unites all your information sources under a single umbrella. This article explains the main concepts of a data hub, its architecture, and how it differs from data warehouses and data lakes. What is Data Hub?

Architecture 52

Architecture Data Lake Unstructured Data Data Warehouse 52

Hepta Analytics

FEBRUARY 14, 2022

This week, we got to think about our data ingestion design. We looked at the following: How do we ingest – ETL vs ELT Where do we store the data – Data lake vs data warehouse Which tool to we use to ingest – cronjob vs workflow engine NOTE : This weeks task requires good internet speed and good compute.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Monte Carlo

JUNE 28, 2022

Traditionally, data lakes held raw data in its native format and were known for their flexibility, speed, and open source ecosystem. By design, data was less structured with limited metadata and no ACID properties. Since then, Databricks has aggressively moved toward allowing users to add more structure to their data.

Data Lake 52

Data Lake Metadata AWS Data Warehouse 52

Monte Carlo

AUGUST 19, 2023

We’ll cover: What is a data platform? With companies moving their data platforms to the cloud, the emergence of cloud-native solutions ( data warehouse vs data lake or even a data lakehouse ) have taken over the market, offering more accessible and affordable options for storing data relative to many on-premises solutions.

Building 52

Building BI Data Lake Data Governance 52

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Data Large volumes of structured or unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40