KDnuggets

FEBRUARY 5, 2024

How and why the data lake architecture often fails to meet its promises. And how better governance helps mitigate such challenges.

Data Lake 104

Data Lake IT Government Data 104

Hevo

APRIL 20, 2024

In the early 2000s, organizations started dealing with more semi-structured and unstructured data, which consisted of images, videos, log files, text, and sensor data. They needed a storage solution that was more flexible than a data warehouse.

Data Lake 52

Data Lake Unstructured Data Data Warehouse Data 52

Start Data Engineering

APRIL 12, 2022

Introduction Data lakes and data warehouses Data lake Data warehouse Criteria to choose lake and warehouse tools Conclusion Further reading References Introduction With the data ecosystem growing fast, new terms are coming up every week.

Data Lake 130

Data Lake Data Warehouse Data 130

Precisely

MARCH 9, 2023

As cloud computing platforms make it possible to perform advanced analytics on ever larger and more diverse data sets, new and innovative approaches have emerged for storing, preprocessing, and analyzing information. In this article, we’ll focus on a data lake vs. data warehouse.

Data Lake 59

Data Lake Data Warehouse Hadoop Raw Data 59

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

KDnuggets

JANUARY 16, 2023

In this article, we will discuss the benefits of using SQL with a data lake and how it can help organizations unlock the full potential of their data.

Data Lake 108

Data Lake SQL Data IT 108

phData: Data Engineering

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Cloudyard

DECEMBER 13, 2022

Read Time: 4 Minute, 23 Second During this post we will discuss how AWS S3 service and Snowflake integration can be used as Data Lake in current organizations. How customer has migrated On Premises EDW to Snowflake to leverage snowflake Data Lake capabilities. Create S3 bucket to hold the tables data.

Data Lake 52

Data Lake AWS Metadata Data 52

Monte Carlo

APRIL 24, 2023

Data lakes are useful, flexible data storage repositories that enable many types of data to be stored in its rawest state. Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Data Engineering Podcast

JULY 22, 2019

Summary The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and access.

Data Lake 100

Data Lake Hadoop Data Architecture 100

Data Engineering Podcast

MAY 15, 2022

Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. Missing data? Struggling with broken pipelines? Stale dashboards?

Data Lake 100

Data Lake Building BI Architecture 100

Seattle Data Guy

MAY 7, 2024

Getting data out of source systems and into a data warehouse or data lake is one of the first steps in making it usable by analysts and data scientists. The question is how will your team do that?

Data Lake 130

Data Lake Data Warehouse Data Systems 130

Data Engineering Podcast

NOVEMBER 2, 2020

Summary Data lakes are gaining popularity due to their flexibility and reduced cost of storage. Along with the benefits there are some additional complexities to consider, including how to safely integrate new data sources or test out changes to existing pipelines.

Data Lake 100

Data Lake PostgreSQL Datasets Machine Learning 100

Grouparoo

JANUARY 11, 2022

When it comes to storing large volumes of data, a simple database will be impractical due to the processing and throughput inefficiencies that emerge when managing and accessing big data. This article looks at the options available for storing and processing big data, which is too large for conventional databases to handle.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Seattle Data Guy

FEBRUARY 10, 2024

Photo by Tiger Lily Data warehouses and data lakes play a crucial role for many businesses. It gives businesses access to the data from all of their various systems. As well as often integrating data so that end-users can answer business critical questions.

Data Lake 162

Data Lake Data Warehouse Data Accessible 162

KDnuggets

JANUARY 16, 2020

Whereas a data warehouse will need rigid data modeling and definitions, a data lake can store different types and shapes of data. In a data lake, the schema of the data can be inferred when it’s read, providing the aforementioned flexibility.

Data Lake 67

Data Lake Data Warehouse Data 67

phData: Data Engineering

APRIL 4, 2023

Today we want to introduce Fivetran’s support for Amazon S3 with Apache Iceberg, investigate some of the implications of this feature, and learn how it fits into the modern data architecture as a whole. Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Acceldata

DECEMBER 19, 2022

Avoid these three data pitfalls when attempting to modernize your data architecture with a data lake or data warehouse.

Data Lake 52

Data Lake Data Warehouse Data Data Architecture 52

Elder Research

MARCH 28, 2023

The post Is My Data Lake Actually a Data Swamp? appeared first on Elder Research.

Data Lake 52

Data Lake Data 52

Christophe Blefari

FEBRUARY 23, 2024

My ideas these days ( credits ) Hey, fresh Data News edition. This week I've participated to a round table about data and did a cool presentation about Engines. — Apache Arrow is an awesome library that powers a lot of innovations in the data space in the recent years. Is it lake or warehouse?

Data Lake 130

Data Lake PostgreSQL MongoDB Scala 130

Teradata

JUNE 13, 2019

Martin Wilcox examines the failure of data lakes.

Data Lake 102

Data Lake Data 102

Data Engineering Podcast

NOVEMBER 11, 2018

Summary A data lake can be a highly valuable resource, as long as it is well built and well managed. In this episode Yoni Iny, CTO of Upsolver, discusses the various components that are necessary for a successful data lake project, how the Upsolver platform is architected, and how modern data lakes can benefit your organization.

Data Lake 100

Data Lake Building Kafka Cloud 100

Start Data Engineering

MAY 11, 2022

Data lake structure 5. Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Introduction 2. Objective 3. Prerequisite 4.2 AWS Infrastructure costs 4.3 Code walkthrough 5.1 Generating user behavior metric 5.4. Checking results 6.

Data Engineering 246

Data Engineering Data Engineer Project Data Lake 246

Monte Carlo

JUNE 2, 2021

As companies leverage more and more data to drive decision-making and maintain their competitive edge, it’s crucial that this data is accurate and reliable. Over the last few years, data lakes have emerged as a must-have for the modern data stack. What is data observability?

Data Lake 52

Data Lake Machine Learning Unstructured Data Datasets 52

databricks

JANUARY 26, 2023

We thank Vishnu Vettrivel, Founder, and Alex Thomas, Principal Data Scientist, for their contributions. This is a collaborative post from Databricks and wisecube.ai.

Data Lake 112

Data Lake Building Data 112

Propel Data

OCTOBER 11, 2022

The main difference between data lakes and data warehouses is data lakes allow unstructured data, but data warehouses need structured data.

Data Lake 40

Data Lake Data Warehouse Unstructured Data Data Analytics 40

Preset

MAY 24, 2021

Visualize your data lake using AWS Athena and Apache Superset™. | AWS Athena Tutorial

Data Lake 52

Data Lake AWS Data 52

Data Engineering Podcast

MAY 22, 2022

Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

databricks

JUNE 6, 2023

Apache Parquet is one of the most popular open source file formats in the big data world today. Being column-oriented, Apache Parquet allows.

Data Lake 76

Data Lake Big Data Data Data Engineering 76

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Uber Engineering

MARCH 16, 2023

Uber’s Global Data Warehouse team leveraged Apache Hudi to drastically improve performance of traditional batch ETL pipelines by going incremental, improving business-critical data’s freshness, quality, and completeness.

Data Lake 98

Data Lake Data Warehouse Data 98

Data Engineering Weekly

APRIL 21, 2024

Meta: Introducing Meta Llama 3 - The most capable openly available LLM to date Meta is taking an interesting approach in the growing LLM market with the open source approach and distribution across all the leading cloud providers and data platforms. Counting is the hardest problem in data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

RudderStack

JANUARY 27, 2022

Learn more about how to send event data from RudderStack to your Data Lake and Delta Lake destinations.

Data Lake 40

Data Lake Data 40

Towards Data Science

JANUARY 26, 2023

How we deal with Big Data at Emplifi Continue reading on Towards Data Science »

Data Lake 68

Data Lake Building Data Science Big Data 68

Towards Data Science

MAY 3, 2023

Data Engineering Learn about slow change dimensions (SCD) and how to implement SCD Type 2 in VDK Photo by Joshua Sortino on Unsplash Data is the backbone of any organization, and in today’s fast-paced world, it is crucial to keep track of its versions. They store and manage current and historical data in a data warehouse.

Data Lake 87

Data Lake Data SQL Data Warehouse 87