Propel Data

AUGUST 1, 2022

Snowflake data platform is referred to as a data warehouse or data lake because it separates storage (data) from compute (processing power).

Data Warehouse 40

Data Warehouse Data Analytics Data Lake Data 40

Data Engineering Weekly

MARCH 3, 2024

The NVIDIA blog on Sovereign AI emphasizes the importance of countries developing artificial intelligence capabilities using local infrastructure, data, and workforce. The article concludes with a look at data contracts as a concrete example of these principles in practice. [link] Nvidia: What Is Sovereign AI?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Christophe Blefari

JANUARY 20, 2024

You'll be seen as the most technical person of a data team and you'll need to help regarding "low-level" stuff you team. You'll be also asked to put in place a data infrastructure. It means a data warehouse, a data lake or other concepts starting with data.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

JUNE 2, 2022

CDF-PC is a cloud native universal data distribution service powered by Apache NiFi on Kubernetes, ??allowing allowing developers to connect to any data source anywhere with any structure, process it, and deliver to any destination. This blog aims to answer two questions: What is a universal data distribution service?

Systems 99

Systems Data Lake Google Cloud Data Collection 99

Christophe Blefari

FEBRUARY 24, 2023

This is something I struggle with, I really like writing, I really like this newsletter, I really like the blog, but it takes me one day per week to be done. Measuring everything — This blog is a proposition and a signal for you that you should measure absolutely everything to understand what is happening in your product.

Kafka 130

Kafka Data Lake Data Storage Data 130

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 90

Data Lake Data Warehouse BI SQL 90

Cloudera

JUNE 25, 2021

This blog post outlines detailed step by step instructions to perform Hive Replication from an on-prem CDH cluster to a CDP Public Cloud Data Lake. CDP Data Lake cluster versions – CM 7.4.0, CDP Data Lake cluster versions – CM 7.4.0, Pre-Check: Data Lake Cluster.

Cloud 69

Cloud Data Lake Cloud Storage Metadata 69

Cloudera

MARCH 17, 2023

I took the free version of ChatGPT on a test drive (in March 2023) and asked some simple questions on data lakehouse and its components. Hopefully this blog will give ChatGPT an opportunity to learn and correct itself while counting towards my 2023 contribution to social good. I thought this was a fairly comprehensive list.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

Data Science Blog: Data Engineering

SEPTEMBER 24, 2021

To do this, we tick off the following points step by step: Database Concepts: ACID vs BASE Database Concepts: CAP Theorem Normalization of Databases (coming soon) Data Warehouse Data Models (coming soon) Data Lake Concepts (coming soon) The post Data Warehousing Basiscs appeared first on Data Science Blog (English only).

Data Lake 52

Data Lake Data Warehouse Data Transparency Database 52

Snowflake

DECEMBER 4, 2023

Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Snowflake’s support for Iceberg Tables is now in public preview, helping customers build and integrate Snowflake into their lake architecture.

Building 95

Building Metadata Cloud Storage AWS 95

Data Engineering Podcast

FEBRUARY 9, 2020

What are some of the foundational skills and knowledge that are necessary for effective modeling of data warehouses? How has the era of data lakes, unstructured/semi-structured data, and non-relational storage engines impacted the state of the art in data modeling?

Data Lake 100

Data Lake Data Warehouse Hadoop NoSQL 100

Data Engineering Weekly

FEBRUARY 4, 2024

link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. The blog is an excellent reminder no matter how standardized your tools are, the migration from one DB to another is always a nightmare.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Cloudera

DECEMBER 11, 2020

As mentioned in my previous blog on the topic , the recent shift to remote working has seen an increase in conversations around how data is managed. Without meeting GxP compliance, the Merck KGaA team could not run the enterprise data lake needed to store, curate, or process the data required to inform business decisions.

Data Lake 98

Data Lake Government Data Security Unstructured Data 98

Cloudera

MARCH 5, 2024

With built-in features like time travel, schema evolution, and streamlined data discovery, Iceberg empowers data teams to enhance data lake management while upholding data integrity. Learn more about the next generation of Cloudera Data Platform for Private Cloud.

Data Lake 86

Data Lake Data Storage Government Kafka 86

databricks

MAY 9, 2023

As organizations are maturing their data infrastructure and accumulating more data than ever before in their data lakes, Open and Reliable table formats.

Data Lake 93

Data Lake Data Data Engineering Data Engineer 93

Knowledge Hut

MARCH 28, 2024

This demonstrates the increasing need for Microsoft Certified Data Engineers. In this blog, I will explore Azure data engineer jobs and the top 10 job roles in this field where you can begin your career. Role Level Intermediate Responsibilities Design and develop data pipelines to ingest, process, and transform data.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Cloudera

OCTOBER 6, 2022

CIO blog post : “Digital transformation is a foundational change in how an organization delivers value to its customers.”. For example, we have some customers using their data platform originally established for compliance initiatives to drive new use cases. appeared first on Cloudera Blog. Strategies to maximize impact.

Data Lake 85

Data Lake Machine Learning Data Storage Cloud Computing 85

Christophe Blefari

MARCH 4, 2023

I'll try to think about it in the following weeks to understand where I go for the third year of the newsletter and the blog. This is funny to see their offering because they offer a "managed data warehouse storage", which means without the compute. So thank you for that. Stay tuned and let's jump to the content.

Machine Learning 130

Machine Learning AWS Data Data Lake 130

Data Engineering Podcast

APRIL 16, 2022

In this episode Demetrios Brinkmann and David Aponte share their perspectives on this rapidly changing space and what they have learned from their work building the MLOps community through blog posts, podcasts, and discussion forums. Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies.

Data Engineering 100

Data Engineering Data Engineer Engineering IT 100

Advancing Analytics: Data Engineering

SEPTEMBER 7, 2021

1 SCD Type 1 in SQL and Python Introduction With the move to cloud based Data Lake platforms there has often been criticism from the more traditional Data Warehousing community. By its definition, a data lake accepts any data, without oversight or governance. From Warehouse to Lakehouse Pt.1

Data Lake 52

Data Lake Datasets Data Warehouse SQL 52

Data Engineering Weekly

JUNE 25, 2023

link] Ziheng Wang: Open Vector Data Lakes We’ve seen the rise of vector databases in the market. The cost, vendor locking, and the properties of the data storage engine are frequent points of discussion in the vector database space. The blog narrates the essence of the asset model below.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Cloudera

FEBRUARY 14, 2024

Introduction Apache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data — structured and unstructured. Iceberg doesn’t delete the old data files. The rest of the blog will go into this in more detail.

Bytes 57

Bytes Metadata Data Lake SQL 57

Databand.ai

JULY 5, 2023

They provide insights into the health of data integration processes, detect issues in real-time, and enable teams to optimize data flows. Data lake and data warehouse monitoring: These tools monitor the performance, storage, and access patterns of data lakes and data warehouses, ensuring optimal performance and data availability.

Data Pipeline 52

Data Pipeline Data Lake Data Warehouse Datasets 52

Knowledge Hut

NOVEMBER 17, 2023

They enhance data pipelines, transform data, and guarantee the accuracy, integrity, and compliance of the data. Their job entails Azure data engineer skills like using big data, databases, data lakes, and analytics to help firms make efficient data-driven decisions.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

DataKitchen

AUGUST 3, 2021

First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt. Second-generation – gigantic, complex data lake maintained by a specialized team drowning in technical debt. The post What is a Data Mesh?

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Cloudera

JANUARY 30, 2024

Add appropriate contextual data (IT/business data), which is critical in AI analysis of manufacturing data. Eliminate data silos. Data from multiple sources must be centralized and stored on a common data lake so that you will have one source of truth across the value chain.

Manufacturing 77

Manufacturing Data Lake Data Science Cloud 77

Databand.ai

AUGUST 30, 2023

Data storage platforms can include traditional relational databases, NoSQL databases, data lakes, or cloud-based storage services. A DataOps architecture must consider the performance, scalability, and cost implications of the chosen data storage platform.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Scott Logic

APRIL 22, 2024

It’s been around since 2017, and we don’t intend to go into a full review of its features here—only a month ago, Mike Morgan and Steve Conway from our Leeds office published a comparative review of three cloud BI solutions, including QuickSight here on the Scott Logic blog.

BI 97

BI Software Engineer Software Engineering Algorithm 97

DataKitchen

NOVEMBER 4, 2021

Cloud computing has made it much easier to integrate data sets, but that’s only the beginning. Creating a data lake has become much easier, but that’s only ten percent of the job of delivering analytics to users. It often takes months to progress from a data lake to the final delivery of insights.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Cloudera

APRIL 4, 2022

The usage of data lakes and automation are helping facilitate the data sharing and collaboration across the healthcare ecosystem. appeared first on Cloudera Blog. Now is the time to reflect and learn from the events of the past two years. The post Why Can’t we Advance Healthcare and Life Sciences this Fast all the time?

Healthcare 77

Healthcare Data Lake Manufacturing Process 77

Data Engineering Podcast

DECEMBER 16, 2018

In recent years there has been a shift to using data lakes as a staging ground before performing transformations. What are some references that you recommend for anyone who is designing a new data platform? In recent years there has been a shift to using data lakes as a staging ground before performing transformations.

Data Pipeline 100

Data Pipeline Data Lake Data Warehouse Python 100

Cloudera

AUGUST 18, 2021

Data processed at the edge or in the cloud, for instance, is not effective if it follows the traditional lifecycle of “ingest, process, land, and analyze.” If the data goes into a data lake before analysis, extracting it can get pretty complex and time-consuming.

Medical 87

Medical Hospitality Data Lake Healthcare 87

Data Engineering Podcast

JULY 10, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Cloudera

JUNE 5, 2023

On May 3, 2023, Cloudera kicked off a contest called “Best in Flow” for NiFi developers to compete to build the best data pipelines. This blog is to congratulate our winner and review the top submissions. RK built some simple flows to pull streaming data into Google Cloud Storage and Snowflake.

Google Cloud 74

Google Cloud Cloud Storage Data Lake Data Pipeline 74

Cloudera

DECEMBER 14, 2021

Then, the company used Cloudera’s Data Platform as a foundation to build its own Network Real-time Analytics Platform (NRAP) and created the proper infrastructure to collect and analyze large-scale big data in real-time. . The post AI and ML: No Longer the Stuff of Science Fiction appeared first on Cloudera Blog.

Transportation 107

Transportation Telecommunication Banking Data Lake 107

Data Engineering Podcast

MAY 1, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Email hosts@dataengineeringpodcast.com ) with your story.

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Cloudera

JUNE 7, 2023

With persistent context across analytics and cloud environments, SDX simplifies data delivery and access with a unified multi-tenant model. This reduces risks and operational costs while enabling faster deployment of secure and governed data lakes for broader data access.

Government 79

Government Metadata Data Lake Data 79