Hevo

JULY 23, 2024

The term “Data Lake” was first introduced by James Dixon in 2010 as a form of storage to cope with evolving data needs due to advancements in IT and IoT. The close of the 19th century and beginning of the 20th century saw an influx of new digital tools like the mobile phone and smart […]

Data Lake 52

Data Lake Technology Data IT 52

Hevo

JULY 15, 2024

From mobile devices and IoT sensors to our online content, unprecedented amounts of data are generated. Traditional databases and warehouse technologies cannot handle this data volume and variety burst. This led to the rise of data lakes […]

Data Lake 52

Data Lake Data Database Technology 52

Hevo

JULY 15, 2024

From mobile devices and IoT sensors to our online content, unprecedented amounts of data are generated. Traditional databases and warehouse technologies cannot handle this data volume and variety burst. This led to the rise of data lakes […]

Data Lake 52

Data Lake Data Database Technology 52

Hevo

JULY 12, 2024

Building a data lake for reporting, analytics, and machine learning needs has become general practice. Data lakes allow us to ingest data from multiple sources in their raw formats in real time. This will enable us to scale any data size and save time in defining its schema and transformations.

Data Lake 52

Data Lake MySQL Building Machine Learning 52

KDnuggets

FEBRUARY 5, 2024

How and why the data lake architecture often fails to meet its promises. And how better governance helps mitigate such challenges.

Data Lake 116

Data Lake IT Government Data 116

KDnuggets

OCTOBER 30, 2023

A comparative overview of data warehouses, data lakes, and data marts to help you make informed decisions on data storage solutions for your data architecture.

Data Lake 128

Data Lake Data Warehouse Data Storage Data 128

Seattle Data Guy

AUGUST 2, 2023

A few months ago, I uploaded a video where I discussed data warehouses, data lakes, and transactional databases. However, the world of data management is evolving rapidly, especially with the resurgence of AI and machine learning.

Data Lake 130

Data Lake Data Warehouse Data Machine Learning 130

Start Data Engineering

APRIL 12, 2022

Introduction Data lakes and data warehouses Data lake Data warehouse Criteria to choose lake and warehouse tools Conclusion Further reading References Introduction With the data ecosystem growing fast, new terms are coming up every week.

Data Lake 130

Data Lake Data Warehouse Data 130

Hevo

JULY 12, 2024

Are you looking for a data lake tool that is scalable, cost-efficient, and accessible, can store your business’s historical data, and can help you perform intelligent analytics? To lift the weight off your shoulders, I have compiled a list of data lake tools. No worries.

Data Lake 40

Data Lake Data Accessible Accessibility 40

Hevo

APRIL 20, 2024

In the early 2000s, organizations started dealing with more semi-structured and unstructured data, which consisted of images, videos, log files, text, and sensor data. They needed a storage solution that was more flexible than a data warehouse.

Data Lake 52

Data Lake Unstructured Data Data Warehouse Data 52

Seattle Data Guy

DECEMBER 19, 2022

Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data.

Data Lake 130

Data Lake Data Warehouse Database Data Storage 130

Precisely

MARCH 9, 2023

Data warehouse vs. data lake, each has their own unique advantages and disadvantages; it’s helpful to understand their similarities and differences. In this article, we’ll focus on a data lake vs. data warehouse. It is often used as a foundation for enterprise data lakes.

Data Lake 58

Data Lake Data Warehouse Hadoop Raw Data 58

Towards Data Science

SEPTEMBER 27, 2023

Data Access API over Data Lake Tables Without the Complexity Build a robust GraphQL API service on top of your S3 data lake files with DuckDB and Go Photo by Joshua Sortino on Unsplash 1. This data might be primarily used for internal reporting, but might also be valuable for other services in our organization.

Data Lake 94

Data Lake Accessible Accessibility SQL 94

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 92

Data Lake Data Warehouse Cloud Unstructured Data 92

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

KDnuggets

JANUARY 16, 2023

In this article, we will discuss the benefits of using SQL with a data lake and how it can help organizations unlock the full potential of their data.

Data Lake 108

Data Lake SQL Data IT 108

Cloudyard

DECEMBER 13, 2022

Read Time: 4 Minute, 23 Second During this post we will discuss how AWS S3 service and Snowflake integration can be used as Data Lake in current organizations. How customer has migrated On Premises EDW to Snowflake to leverage snowflake Data Lake capabilities.

Data Lake 52

Data Lake AWS Metadata Data 52

phData: Data Engineering

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake? Consistency of data throughout the data lake.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Monte Carlo

APRIL 24, 2023

Data lakes are useful, flexible data storage repositories that enable many types of data to be stored in its rawest state. Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Data Engineering Podcast

NOVEMBER 20, 2021

Summary One of the perennial challenges posed by data lakes is how to keep them up to date as new data is collected. In this episode Ori Rafael shares his experiences from Upsolver and building scalable stream processing for integrating and analyzing data, and what the tradeoffs are when coming from a batch oriented mindset.

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Data Engineering Podcast

JUNE 1, 2020

Summary Data lakes offer a great deal of flexibility and the potential for reduced cost for your analytics, but they also introduce a great deal of complexity. In order to bring the DBA into the new era of data management the team at Upsolver added a SQL interface to their data lake platform.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Data Engineering Podcast

MAY 15, 2022

Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. When is a data lake architecture the wrong choice?

Data Lake 100

Data Lake Building BI Architecture 100

Data Engineering Podcast

JULY 22, 2019

Summary The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and access.

Data Lake 100

Data Lake Hadoop Data Architecture 100

KDnuggets

JANUARY 16, 2020

Whereas a data warehouse will need rigid data modeling and definitions, a data lake can store different types and shapes of data. In a data lake, the schema of the data can be inferred when it’s read, providing the aforementioned flexibility.

Data Lake 74

Data Lake Data Warehouse Data 74

Data Engineering Podcast

NOVEMBER 2, 2020

Summary Data lakes are gaining popularity due to their flexibility and reduced cost of storage. Along with the benefits there are some additional complexities to consider, including how to safely integrate new data sources or test out changes to existing pipelines.

Data Lake 100

Data Lake PostgreSQL Datasets Machine Learning 100

Grouparoo

JANUARY 11, 2022

This article looks at the options available for storing and processing big data, which is too large for conventional databases to handle. There are two main options available, a data lake and a data warehouse. What is a Data Warehouse? What is a Data Lake?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Data Engineering Podcast

JUNE 16, 2019

Summary Building and maintaining a data lake is a choose your own adventure of tools, services, and evolving best practices. The flexibility and freedom that data lakes provide allows for generating significant value, but it can also lead to anti-patterns and inconsistent quality in your analytics.

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Monte Carlo

AUGUST 25, 2023

That’s why it’s essential for teams to choose the right architecture for the storage layer of their data stack. But, the options for data storage are evolving quickly. Different vendors offering data warehouses, data lakes, and now data lakehouses all offer their own distinct advantages and disadvantages for data teams to consider.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Data Engineering Podcast

SEPTEMBER 1, 2021

Summary The Presto project has become the de facto option for building scalable open source analytics in SQL for the data lake. Another area that has been seeing a lot of activity is data lakes and projects to make them more manageable and feature complete (e.g. Hudi, Delta Lake, Iceberg, Nessie, LakeFS, etc.).

Data Lake 100

Data Lake Cloud AWS SQL 100

Snowflake

AUGUST 8, 2023

In 2022, the company realized that in order to monitor, analyze, counter, and prevent security threats, it needed both a modern data lake as well as a security information and event management (SIEM) solution. Then, they conducted insight mining by querying the data on different topics.

Data Lake 52

Data Lake Programming IT Building 52

Monte Carlo

JANUARY 23, 2023

When it comes to the data community, there’s always a debate broiling about something— and right now “data mesh vs data lake” is right at the top of that list. In this post we compare and contrast the data mesh vs data lake to illustrate the benefits of each and help discover what’s right for your data platform.

Data Lake 52

Data Lake Unstructured Data Business Intelligence Architecture 52

Data Engineering Podcast

JUNE 23, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Data lakes in various forms have been gaining significant popularity as a unified interface to an organization's analytics. When is Fabric the wrong choice?

Data Lake 162

Data Lake High Quality Data Hadoop Machine Learning 162

phData: Data Engineering

APRIL 4, 2023

Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format. Amazon S3 is an object storage service from Amazon Web Services (AWS) that offers industry-leading scalability, data availability, security, and performance.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Snowflake

JANUARY 24, 2023

CMS Chief Information Security Officer Rob Wood and Chief information Architect Amine Raounak led an in-depth presentation on their security data lake initiative with Snowflake. The security data lake project enables data-driven decision-making for system and product security at scale within CMS.

Data Lake 53

Data Lake IT Insurance AWS 53

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Visit [dataengineeringpodcast.com/data-council]([link] and use code *depod20* to register today!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262