Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Cloudera

JANUARY 15, 2021

Cloud data warehouses allow users to run analytic workloads with greater agility, better isolation and scale, and lower administrative overhead than ever before. The results demonstrate superior price performance of Cloudera Data Warehouse on the full set of 99 queries from the TPC-DS benchmark. Introduction.

Data Warehouse 92

Data Warehouse Cloud Consulting SQL 92

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. In order to turn this into a tractable problem one approach is to define and enforce contracts between producers and consumers of data. Atlan is the metadata hub for your data ecosystem.

Systems 100

Systems Metadata Building MongoDB 100

Grouparoo

OCTOBER 6, 2021

It was a fun experience and I think we made a good choice by picking 97 Things Every Data Engineer Should Know. This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Knowledge Hut

NOVEMBER 28, 2023

The contemporary world experiences a huge growth in cloud implementations, consequently leading to a rise in demand for data engineers and IT professionals who are well-equipped with a wide range of application and process expertise. Data Engineer certification will aid in scaling up you knowledge and learning of data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Generalist 52

Data Engineering Weekly

OCTOBER 30, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. Missing data?

Systems 100

Systems Building Metadata Data Warehouse 100

Ascend.io

APRIL 22, 2024

At Ascend, we believe it’s time to rethink data engineering from the ground up. As the world of data continues to evolve at a breakneck pace, we are thrilled to announce the next revolutionary step in our journey – Project Inception. Read More: The State of Data Engineering in 2023: Does Your Data Program Stack Up?

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Podcast

DECEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Ascend.io

AUGUST 31, 2023

Secondly , the rise of data lakes that catalyzed the transition from ELT to ELT and paved the way for niche paradigms such as Reverse ETL and Zero-ETL. Still, these methods have been overshadowed by EtLT — the predominant approach reshaping today’s data landscape.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Monte Carlo

MARCH 4, 2024

As any data engineer currently rolling their eyes would tell you, the reality has long been more nuanced. These questions just scratch the surface of the painful delightfully complex world of data engineering. Most major data cloud providers support both use cases. Is a materialized view a view or a table?

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Data Engineering Podcast

MARCH 27, 2022

In this episode Balaji Ganesan shares how his experiences building and maintaining Ranger in previous roles helped him understand the needs of organizations and engineers as they define and evolve their data governance policies and practices. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Hepta Analytics

FEBRUARY 14, 2022

This week, we got to think about our data ingestion design. We looked at the following: How do we ingest – ETL vs ELT Where do we store the data – Data lake vs data warehouse Which tool to we use to ingest – cronjob vs workflow engine NOTE : This weeks task requires good internet speed and good compute.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Query Google’s cloud data warehouse.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 90

Data Lake Data Warehouse BI SQL 90

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

Towards Data Science

FEBRUARY 6, 2024

The disconnection between the operational teams immersed in the day-to-day functions and those extracting business value from data generated in the operational processes still remains a significant friction point. Searching for data Imagine being a data engineer/analyst tasked with identifying the top-selling products within your company.

Systems 83

Systems Raw Data Metadata Data Cleanse 83

Monte Carlo

JUNE 28, 2022

To help organizations realize the full potential of their data lake and lakehouse investments, Monte Carlo, the data observability leader, is proud to announce integrations with Delta Lake and Databricks’ Unity Catalog for full data observability coverage. billion in 2020 to 17.60 billion in 2020 to 17.60

Data Lake 52

Data Lake Metadata AWS Data Warehouse 52

Workfall

JUNE 12, 2023

In this dynamic realm of data engineering, a monumental challenge takes centre stage: efficiently managing the ever-changing tides of real-time data. Data, the lifeblood of organisations, holds the key to unlocking untapped potential and propelling businesses forward. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Podcast

SEPTEMBER 23, 2018

This includes modeling the lifecycle of your information as a pipeline from the raw, messy, loosely structured records in your data lake, through a series of transformations and ultimately to your data warehouse. What are your thoughts on the landscape for managing data flows and migration and when to use which approach?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Cloudera

MARCH 17, 2023

The table format provides the necessary structure for the unstructured data that is missing in a data lake, using a schema or metadata definition, to bring it closer to a data warehouse. Some of the popular table formats are Apache Iceberg, Delta Lake, Hudi, and Hive ACID.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

Data Engineering Podcast

MARCH 22, 2021

In this episode Raghu Murthy, founder and CEO of Datacoral, does a deep dive on how he and his team manage change data capture pipelines in production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. S3/Parquet or Trino/Drill/etc.)

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Data Engineering Podcast

NOVEMBER 20, 2021

In this episode he shares the journey that he and his team at Elementl have taken to understand the state of the ecosystem and how they can provide a foundational layer for a holistic data platform. Missing data? Start trusting your data with Monte Carlo today! Struggling with broken pipelines? Stale dashboards?

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

APRIL 24, 2022

Summary There are very few tools which are equally useful for data engineers, data scientists, and machine learning engineers. WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model.

Machine Learning 100

Machine Learning Systems Data Lake Metadata 100

Monte Carlo

MAY 18, 2023

Data lineage tools are like detectives that help data professionals quickly sort through the tangled webs of interdependencies that make up the modern data stack. The Spark data lake lineage example is helpful for context here. Why do you need data lineage? “I

Metadata 64

Metadata Data Lake Government Data Governance 64

Data Engineering Podcast

AUGUST 6, 2022

Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. RudderStack helps you build a customer data platform on your warehouse or data lake.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Cloudera

JANUARY 19, 2022

Cloudera Data Platform (CDP) scored among the top 10 vendors on all four Analytical Use Cases — Data Warehouse, Logical Data Warehouse, Data Lake and Operational Intelligence in the Critical Capabilities for Cloud Database Management Systems for Analytics Use Cases.

Database 87

Database Cloud Data Warehouse Data Lake 87

ProjectPro

FEBRUARY 16, 2023

ETL is a critical component of success for most data engineering teams, and with teams harnessing it with the power of AWS, the stakes are higher than ever. Data Engineers and Data Scientists require efficient methods for managing large databases, which is why centralized data warehouses are in high demand.

AWS 52

AWS Data Management ETL Tools Management 52

DataKitchen

AUGUST 3, 2021

First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt. Second-generation – gigantic, complex data lake maintained by a specialized team drowning in technical debt. The product includes data and operations.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Data Engineering Weekly

JANUARY 6, 2023

Author: Chris Ricommni Recap Github: [link] [Don't forget to star it] I’m excited to release Recap , a dead simple data catalog for engineers written in Python. Recap makes it easy for engineers to build infrastructure and tools that need metadata. Recap is a data catalog for machines–a metadata service.

Metadata 40

Metadata ETL Tools MySQL Data Lake 40

Rockset

APRIL 26, 2023

Offline feature store : Detecting anomalies requires historical data in order to have a baseline for comparisons. This data tends to be slow changing and is stored in an offline feature store. This could be a cloud data warehouse, a data lake, or a database. The database has two primary jobs.

Architecture 52

Architecture Transportation Data Lake Insurance 52

ProjectPro

JANUARY 24, 2023

It is a popular ETL tool well-suited for big data environments and extensively used by data engineers today to build and maintain data pipelines with minimal effort. You can leverage AWS Glue to discover, transform, and prepare your data for analytics.

AWS 52

AWS Data Lake Scala ETL Tools 52

Cloudera

FEBRUARY 14, 2024

Introduction Apache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data — structured and unstructured. Expiring snapshots is a relatively cheap operation and uses metadata to determine newly unreachable files.

Bytes 57

Bytes Metadata Data Lake SQL 57

AltexSoft

FEBRUARY 11, 2023

It serves as a foundation for the entire data management strategy and consists of multiple components including data pipelines; , on-premises and cloud storage facilities – data lakes , data warehouses , data hubs ;, data streaming and Big Data analytics solutions ( Hadoop , Spark , Kafka , etc.);

Data Architect 87

Data Architect Certification Generalist Big Data 87

Netflix Tech

MARCH 7, 2024

Data: Fast Data Our main data lake is hosted on S3, organized as Apache Iceberg tables. For ETL and other heavy lifting of data, we mainly rely on Apache Spark. In many cases, the integration might be via shared tables in our data warehouse.

Systems 90

Systems Media Machine Learning Data Warehouse 90