Data Engineering Podcast

DECEMBER 8, 2019

Summary Data warehouses have gone through many transformations, from standard relational databases on powerful hardware, to column oriented storage engines, to the current generation of cloud-native analytical engines. If you are evaluating your options for building or migrating a data platform, then this is definitely worth a listen.

Data Warehouse 100

Data Warehouse Cloud AWS Relational Database 100

Data Engineering Podcast

JULY 3, 2022

Summary The ecosystem for data tools has been going through rapid and constant evolution over the past several years. These technological shifts have brought about corresponding changes in data and platform architectures for managing data and analytical workflows. Tired of deploying bad data?

Architecture 100

Architecture Metadata MongoDB MySQL 100

Monte Carlo

APRIL 4, 2023

Data mesh vs data warehouse is an interesting framing because it is not necessarily a binary choice depending on what exactly you mean by data warehouse (more on that later). Despite their differences, however, both approaches require high-quality, reliable data in order to function. What is a Data Mesh?

Data Warehouse 52

Data Warehouse Data Governance Data Architecture 52

Snowflake

JANUARY 30, 2023

By running data warehouse and data engineering workloads on Snowflake’s Data Cloud Ramp improves performance and user experience, while delivering powerful insights to customers quickly. How do you scale seamlessly, without worrying about keeping the lights on?

Data Architecture 52

Data Architecture Architecture Management Datasets 52

Acceldata

DECEMBER 19, 2022

Avoid these three data pitfalls when attempting to modernize your data architecture with a data lake or data warehouse.

Data Lake 52

Data Lake Data Warehouse Data Data Architecture 52

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, Alluxio, and Data Council.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Towards Data Science

JANUARY 16, 2024

My personal take on justifying the existence of Data Mesh A senior stakeholder at one my projects mentioned that they wanted to decentralise their data platform architecture and democratise data across the organisation. When I heard the words ‘decentralised data architecture’, I was left utterly confused at first!

Data Warehouse 67

Data Warehouse Data Architecture Relational Database NoSQL 67

Data Engineering Podcast

JULY 8, 2019

Summary The market for data warehouse platforms is large and varied, with options for every use case. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Data Warehouse 100

Data Warehouse MySQL Data Lake Hadoop 100

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Cloudyard

MARCH 4, 2023

Read Time: 5 Minute, 16 Second As we know Snowflake has introduced latest badge “Data Cloud Deployment Framework” which helps to understand knowledge in designing, deploying, and managing the Snowflake landscape. Respective Cloud would consume/Store the data in bucket or containers. Snowpipe to automate the ingestion process.

Architecture 52

Architecture Cloud Metadata Data Ingestion 52

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. While data vault has many benefits, it is a sophisticated and complex methodology that can present challenges to data quality.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

AltexSoft

AUGUST 29, 2023

Data lakes emerged as expansive reservoirs where raw data in its most natural state could commingle freely, offering unprecedented flexibility and scalability. This article explains what a data lake is, its architecture, and diverse use cases. Data warehouse vs. data lake in a nutshell.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

ProjectPro

JANUARY 31, 2022

As the demand for big data grows, an increasing number of businesses are turning to cloud data warehouses. The cloud is the only platform to handle today's colossal data volumes because of its flexibility and scalability. Launched in 2014, Snowflake is one of the most popular cloud data solutions on the market.

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

Data Engineering Podcast

JUNE 17, 2023

Summary Architectural decisions are all based on certain constraints and a desire to optimize for different outcomes. In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not possible for downstream use cases.

Data Lake 130

Data Lake Metadata Data Architecture Machine Learning 130

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineering 87

Data Engineering Data Engineer Engineering Data Lake 87

Scott Logic

APRIL 18, 2024

In this episode, Oliver Cronk, Andrew Carr and David Hope talk about the ever-changing world of data, with conversations moving from data warehouse to data lake, and data mesh to data fabric.

Data Lake 59

Data Lake Data Warehouse Data Architecture Architecture 59

Knowledge Hut

MARCH 28, 2024

They work together with stakeholders to get business requirements and develop scalable and efficient data architectures. Role Level Advanced Responsibilities Design and architect data solutions on Azure, considering factors like scalability, reliability, security, and performance.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Rockset

APRIL 26, 2023

We’ve noticed many common patterns across streaming data architectures and we’ll be sharing a blueprint for three of the most popular: anomaly detection, IoT, and recommendations. The majority of anomaly detectors require streaming data, real-time data and historical data in order to generate inferences.

Architecture 52

Architecture Transportation Data Lake Insurance 52

Monte Carlo

MAY 14, 2024

As organizations seek greater value from their data, data architectures are evolving to meet the demand — and table formats are no exception. It was designed to support high-volume data exchange and compatibility across different system versions, which is essential for streaming architectures such as Apache Kafka.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Weekly

DECEMBER 2, 2023

Revisiting The Current State of Data Infrastructure Let’s revisit the current state of the data infrastructure before discussing the S3 Express. There are two critical properties of data warehouse access patterns. Data freshness matters a lot—the more recent the data, the more frequently it is accessed.

IT 85

IT BI AWS Kafka 85

Monte Carlo

MARCH 6, 2024

Table of Contents Supporting Multi-Database Architectures Keep An Eye On This Space Supporting Multi-Database Architectures We’ve been aggressively adding these integrations to better support complex data architectures. End-to-end data lineage then helps trace these issues to their point of origin.

Database 52

Database MySQL Data Warehouse AWS 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. As data is expanding exponentially, organizations struggle to harness digital information's power for different business use cases. What is a Big Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Podcast

APRIL 3, 2022

Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. Coalesce is a platform designed to reduce repetitive work for common workflows by adopting a visual pipeline builder to support your data warehouse transformations.

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Simon Späti

NOVEMBER 28, 2018

However, this is still not common in the Data Warehouse (DWH) field. In my recent blog, I researched OLAP technologies, for this post I chose some open-source technologies and used them together to build a full data architecture for a Data Warehouse system. These days, everyone talks about open-source.

Data Warehouse 130

Data Warehouse Data Storage Data Architecture Architecture 130

Data Engineering Podcast

SEPTEMBER 23, 2018

This includes modeling the lifecycle of your information as a pipeline from the raw, messy, loosely structured records in your data lake, through a series of transformations and ultimately to your data warehouse. Can you walk through the stages of an ideal lifecycle for data within the context of an organizations uses for it?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Cloudera

JUNE 18, 2022

These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. In recent years, the term “data lakehouse” was coined to describe this architectural pattern of tabular analytics over data in the data lake.

Data Lake 90

Data Lake Data Warehouse BI SQL 90

Monte Carlo

APRIL 27, 2023

Empowered by the rise of the modern data stack , leading companies like Warner Music Group , Chime and Petsmart are now turning to the Composable CDP to unlock a modular customer data architecture that centers on the cloud data warehouse. Why not build one single store of customer data to activate from centrally?

Data Warehouse 72

Data Warehouse Data Collection Architecture Data Storage 72

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. But first, let’s define the data mesh design pattern. The past decades of enterprise data platform architectures can be summarized in 69 words.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Data Engineering Podcast

OCTOBER 2, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. What are the driving factors for building a real-time data platform?

Building 100

Building BI Data Architecture Architecture 100

Towards Data Science

FEBRUARY 15, 2023

In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture. The Lakehouse architecture was one of them. What is Delta Lake?

Data Lake 80

Data Lake Data Warehouse Hadoop Data Architecture 80

Snowflake

FEBRUARY 21, 2023

In this Q&A, we hear from Nico Acosta, CEO and Co-Founder of Propel, about how his company is building an API platform to equip developers to build with data, and why data architecture is the most important technical decision a company will make. We fully align with Snowflake’s vision of the Data Cloud.

AWS 120

AWS Raw Data Building Architecture 120

Towards Data Science

MAY 11, 2024

When architecting a transactional database or a data warehouse, it’s important not to forget about various types of technical columns… Continue reading on Towards Data Science »

Database Design 85

Database Design Database Designing Data Warehouse 85

Knowledge Hut

DECEMBER 21, 2023

What is Data Engineering? Data engineering is the method to collect, process, validate and store data. It involves building and maintaining data pipelines, databases, and data warehouses. The purpose of data engineering is to analyze data and make decisions easier.

Data Engineering 52

Data Engineering Data Engineer Engineering MongoDB 52

Cloudera

SEPTEMBER 15, 2022

Modern data platforms deliver an elastic, flexible, and cost-effective environment for analytic applications by leveraging a hybrid, multi-cloud architecture to support data fabric, data mesh, data lakehouse and, most recently, data observability. Luke: What is a modern data platform?

Data Lake 79

Data Lake Analytics Application Cloud Storage Architecture 79

Data Engineering Podcast

AUGUST 20, 2021

Summary Data lakes have been gaining popularity alongside an increase in their sophistication and usability. Despite improvements in performance and data architecture they still require significant knowledge and experience to deploy and manage. The data you’re looking for is already in your data warehouse and BI tools.

Building 100

Building Data Lake Data Warehouse SQL 100

phData: Data Engineering

APRIL 4, 2023

Today we want to introduce Fivetran’s support for Amazon S3 with Apache Iceberg, investigate some of the implications of this feature, and learn how it fits into the modern data architecture as a whole. Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

ProjectPro

JULY 22, 2016

For the same cost, organizations can now store 50 times as much data as in a Hadoop data lake than in a data warehouse. Data lake is gaining momentum across various organizations and everyone wants to know how to implement a data lake and why.

Data Lake 40

Data Lake Data Warehouse Hadoop Unstructured Data 40