Cloudera

SEPTEMBER 24, 2020

Some of the most powerful results come from combining complementary superpowers, and the “dynamic duo” of Apache Hive LLAP and Apache Impala, both included in Cloudera Data Warehouse , is further evidence of this. Both Impala and Hive can operate at an unprecedented and massive scale, with many petabytes of data.

Data Warehouse 70

Data Warehouse SQL Engineering Metadata 70

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Materialize]([link] Looking for the simplest way to get the freshest data possible to your teams?

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Monte Carlo

JANUARY 25, 2023

Over the past year, data contracts have taken the data world by storm as a novel approach to ensuring data quality at scale in production services. In the last couple of posts , I’ve focused on implementing data contracts in production services. Where to start implementing data contracts?

Data Warehouse 52

Data Warehouse Data High Quality Data Metadata 52

Cloudera

JANUARY 15, 2021

Cloud data warehouses allow users to run analytic workloads with greater agility, better isolation and scale, and lower administrative overhead than ever before. The results demonstrate superior price performance of Cloudera Data Warehouse on the full set of 99 queries from the TPC-DS benchmark. Introduction. higher cost.

Data Warehouse 94

Data Warehouse Cloud Consulting SQL 94

Data Science Blog: Data Engineering

MAY 20, 2024

Continuous Integration and Continuous Delivery (CI/CD) for Data Pipelines: It is a Game-Changer with AnalyticsCreator! The need for efficient and reliable data pipelines is paramount in data science and data engineering. CI/CD for Data Pipelines Data pipelines provide consistency, reduce errors, and increase efficiency.

Data Pipeline 52

Data Pipeline BI Data Lake Data Warehouse 52

Data Engineering Podcast

AUGUST 13, 2022

Summary Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking for. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Metadata 100

Metadata MongoDB Scala MySQL 100

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This is a real game-changer for data analysts on all levels and will make SQL development faster, easier, and less error-prone.

SQL 72

SQL Data Warehouse AWS Architecture 72

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Monte Carlo

APRIL 16, 2022

The data warehouse is the foundation of the modern data stack, so it caught our attention when we saw Convoy head of data Chad Sanderson declare, “ the data warehouse is broken ” on LinkedIn. Treating data like an API. Immutable data warehouses have challenges too.

Data Warehouse 52

Data Warehouse Architecture Data Government 52

Netflix Tech

OCTOBER 27, 2020

By Tianlong Chen and Ioannis Papapanagiotou Netflix has more than 195 million subscribers that generate petabytes of data everyday. Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy.

Data Warehouse 97

Data Warehouse Datasets Data Big Data 97

Data Engineering Podcast

DECEMBER 18, 2022

Summary One of the reasons that data work is so challenging is because no single person or team owns the entire process. This introduces friction in the process of collecting, processing, and using data. In order to reduce the potential for broken pipelines some teams have started to adopt the idea of data contracts.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Christophe Blefari

JUNE 16, 2023

I'm now under the Berlin rain with 20° When I write in these conditions I feel like a tortured author writing a depressing novel while actually today I'll speak about the AI Act, Python, SQL and data platforms. Mainly he unit tests macros (the logic) with his framework and test data with soda and dbt contracts.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Towards Data Science

NOVEMBER 4, 2023

Platform Specific Tools and Advanced Techniques Photo by Christopher Burns on Unsplash The modern data ecosystem keeps evolving and new data tools emerge now and then. In this article, I want to talk about crucial things that affect data engineers. Are your data pipelines efficient? Data warehouse exmaple.

Data Engineering 83

Data Engineering Data Engineer Engineering BI 83

Cloudera

DECEMBER 21, 2023

How long would it take you to find the data you need to even begin to come up with a data-driven response? Well imagine it no longer, as Cloudera’s SQL AI Assistant is exactly that! However, with the new SQL AI Assistant, this is no longer a chore. Imagine you’ve just started a new job working as a business analyst.

SQL 67

SQL Data Warehouse Business Analyst Data Schemas 67

Engineering at Meta

NOVEMBER 30, 2022

UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. A limiting type system: Initially, we used only the fixed set of built-in Hive data types ( string , integer , boolean , etc. )

SQL 76

SQL Data Warehouse Metadata Coding 76

Lyft Engineering

OCTOBER 3, 2023

High-quality data is necessary for the success of every data-driven company. It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Cloudera

FEBRUARY 9, 2021

Today’s customers have a growing need for a faster end to end data ingestion to meet the expected speed of insights and overall business demand. This ‘need for speed’ drives a rethink on building a more modern data warehouse solution, one that balances speed with platform cost management, performance, and reliability.

Data Warehouse 70

Data Warehouse Cloud Kafka Cloud Storage 70

Data Engineering Podcast

JUNE 19, 2022

Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Christophe Blefari

FEBRUARY 3, 2023

Delivering the data news ( credits ) Hey you, it's already February. The article shows our Netflix art creators are using past data to create new artworks. ebay, Variable Hub a data access layer for risk decisioning — Looks like a feature store but for risk topics. But BI Engineer for the modern data stack times.

BI 130

BI Google Cloud SQL Machine Learning 130

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? The Platform Integration Data Store Transformation Orchestration Presentation Transportation Observability Closing What’s changed? Over the last three years, my life has changed as well.

Building 80

Building Transportation Data Lake Metadata 80

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Christophe Blefari

NOVEMBER 18, 2022

Scracthing the surface ( credits ) Hey you, a new Friday means data news. This week feels a bit like old data news with a variety of articles on different cool topics while I navigate through the actual data trends. Next Monday I'll present "How to build a data dream team" at Y42 meetup.

Data Warehouse 130

Data Warehouse Data Python SQL 130

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. Missing data?

Systems 100

Systems Building Metadata Data Warehouse 100

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Cloudera

SEPTEMBER 9, 2018

We are proud to announce the general availability of Cloudera Altus Data Warehouse , the only cloud data warehousing service that brings the warehouse to the data. Modern data warehousing for the cloud. Cloudera Altus Data Warehouse is designed with agile data teams in mind.

Data Warehouse 40

Data Warehouse Metadata Cloud Storage Cloud 40

Data Engineering Podcast

JUNE 17, 2021

Summary Working with unstructured data has typically been a motivation for a data lake. Kirk Marple has spent years working with data systems and the media industry, which inspired him to build a platform for automatically organizing your unstructured assets to make them more valuable. No more scripts, just SQL.

Unstructured Data 100

Unstructured Data Data Warehouse Metadata Media 100

Netflix Tech

DECEMBER 3, 2022

by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow.

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Monte Carlo

APRIL 20, 2024

That’s where you really want to focus your data quality monitoring.” And it’s true – you need to go “deep” on certain data sets; but if you’re not monitoring broadly as well, you’re missing the boat—and a whole lot of bad data. What is data quality monitoring?

IT 52

IT Metadata Data Warehouse Machine Learning 52

Data Engineering Podcast

DECEMBER 4, 2022

Summary Business intelligence is the foremost application of data in organizations of all sizes. Zing Data is building a mobile native platform for business intelligence. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

phData: Data Engineering

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Towards Data Science

DECEMBER 23, 2023

Photo by Ian Taylor on Unsplash This tutorial guides you through an analytics use case, analyzing semi-structured data with Spark SQL. We’ll start with the data engineering process, pulling data from an API and finally loading the transformed data into a data lake (represented by MinIO ).

SQL 73

SQL Data Analytics Hadoop Raw Data 73

Data Engineering Podcast

NOVEMBER 27, 2022

Summary The data ecosystem has been growing rapidly, with new communities joining and bringing their preferred programming languages to the mix. This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. Atlan is the metadata hub for your data ecosystem. Missing data?

Data Process 100

Data Process Process Metadata Business Intelligence 100

Data Engineering Podcast

DECEMBER 29, 2022

Summary Making effective use of data requires proper context around the information that is being used. Rehgan Avon co-founded AlignAI to help address this challenge through a more purposeful platform designed to collect and distribute the knowledge of how and why data is used in a business. Missing data? Stale dashboards?

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

JULY 17, 2022

Summary There are extensive and valuable data sets that are available outside the bounds of your organization. Whether that data is public, paid, or scraped it requires investment and upkeep to acquire and integrate it with your systems. Atlan is the metadata hub for your data ecosystem.

Data Management 130

Data Management Management Metadata MongoDB 130