Data Engineering Weekly

OCTOBER 23, 2022

of my dear data friends say that they use @data_weekly to keep up with the data engineering landscape ❤️❤️❤️ 8:04 PM ∙ Oct 18, 2022 The top of my mind for this week is Data Catalog. link] It is almost two years since we published the metadata edition, but I keep thinking back.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This blog post aims to help you understand what you can do to get started with generative AI assisted SQL using Hue image version ​​2023.0.16.0

SQL 71

SQL Data Warehouse AWS Architecture 71

Snowflake

MAY 8, 2023

By leveraging Snowflake’s comprehensive metadata on query history and operator statistics , alongside Acceldata’s advanced query fingerprinting techniques, users can attain a deeper comprehension of their workloads and pinpoint the most resource-intensive or unoptimized queries.

Technology 111

Technology SQL BI Utilities 111

Cloudera

JUNE 18, 2022

It added metadata that described the logical and physical layout of the data, enabling cost-based optimizers, dynamic partition pruning, and a number of key performance improvements targeted at SQL analytics. The cloud native table format was open sourced into Apache Iceberg by its creators.

Data Lake 90

Data Lake Data Warehouse BI SQL 90

Cloudera

SEPTEMBER 15, 2022

The gathering in 2022 marked the sixteenth year for top data and analytics professionals to come to the MIT campus to explore current and future trends. ” NetApp provides a more robust definition of data fabric as “an architecture and set of data services that provide consistent capabilities across hybrid, multi-cloud environments.”

Data Lake 79

Data Lake Analytics Application Cloud Storage Architecture 79

Cloudera

JANUARY 5, 2023

What is certain is that having an enterprise data strategy aligned to the organization’s cloud strategy and business priorities will help the organization drive greater business value by improving operational efficiencies and unlocking new revenue streams. 3- Continued move to the public cloud and hybrid cloud, optimizing deployments.

Banking 52

Banking Machine Learning Insurance Data Architecture 52

dbt Developer Hub

AUGUST 23, 2022

This configuration allows you to define any metadata dictionary that you want. Given that the cost of storage in cloud warehouses is extremely cheap, it’s unlikely to be worth the effort to optimize for storage costs. Big thanks to Mike Fuller and Bennie Regenold for help ideating on this blog! OK, so which one?

Datasets 52

Datasets Metadata SQL Data Warehouse 52

Data Engineering Weekly

OCTOBER 30, 2022

link] Tomasz Tunguz: 9 Predictions for data in 2023 It is almost the end of 2022, and it's time for 2023 predictions. I found the blog helpful in understanding the generative model’s historical development and the path forward. Til about [link] Thank you, Anton Bryzgalov, for contributing the blog to the Data Engineering Weekly.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

The Pragmatic Engineer

NOVEMBER 3, 2022

Printing it loses all this metadata. pic.twitter.com/Rk9m1v1hmu — Leah Culver (@leahculver) October 28, 2022 Then, a few hours later, new instructions came from the top: stop printing and shred all documents with the code! Source: Manu Cornet’s blog. I talked with several software engineers before Musk took over.

Software Engineer 130

Software Engineer Software Engineering Coding Media 130

Rockset

AUGUST 2, 2023

In 2022, Windward embarked on several changes to its application prompting a reconsideration of its underlying data stack. In this blog, we’ll describe the new data platform for Windward and how it is API first, enables rapid product iteration and is architected for real-time, streaming data.

Database-centric 52

Database-centric PostgreSQL Transportation Insurance 52

Databand.ai

NOVEMBER 4, 2022

Re-Imagining Data Observability Ryan Yackel 2022-11-04 10:36:35 Data observability has become one of the hottest topics of the year – and for good reason. Now, the power of Databand will be available via a new integration with DataStage CP4D , IBM data fabric’s powerful cloud-native ingestion service. And it does so in real time.

Data 52

Data Data Pipeline Retail Metadata 52

Cloudera

DECEMBER 16, 2022

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. Cloudera has been recognized in this cloud DBMS report since its inception in 2020. We do it today when data is even bigger, and hybrid — and clouds — are expensive. This is unique.

Database 88

Database Cloud Systems Management 88

Cloudera

DECEMBER 12, 2022

Over the past few months industry analysts have been making some pretty controversial recommendations for data management in the cloud. His takeaway: “The world is very much going to be hybrid and multi-cloud.” Pat isn’t the only analyst talking hybrid and multi-cloud for data management, although he may be the most entertaining.

Cloud 90

Cloud Entertainment Telecommunication Insurance 90

Cloudera

APRIL 3, 2023

In this blog, we will share with you in detail how Cloudera integrates core compute engines including Apache Hive and Apache Impala in Cloudera Data Warehouse with Iceberg. We will publish follow up blogs for other data services. Iceberg basics Iceberg is an open table format designed for large analytic workloads.

Data Warehouse 79

Data Warehouse Metadata Java Data 79

dbt Developer Hub

AUGUST 30, 2022

Semantic layer, Python model support, the new dbt Cloud UI and IDE… there’s a lot our product team is excited to share with you at Coalesce in a few weeks. As Tristan says in his latest blog, that’s how this industry moves forward. Check our new dbt Cloud integrations page for what’s available today! dbt_artifacts v1.2.0:

Programming 52

Programming Consulting Python BI 52

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. A comprehensive list of benefits offered by Cosmos can be found in the linked blog.

Process 91

Process Pipeline-centric Media Metadata 91

AltexSoft

NOVEMBER 7, 2022

2022 Airflow user overview. Metadata database. A metadata database stores information about user permissions, past and current DAG and task runs, DAG configurations, and more. By default, Airflow handles metadata with SQLite which is meant for development only. Source: Apache Airflow. What is Apache Airflow used for?

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Cloudera

NOVEMBER 17, 2022

The Solution: CDP Private Cloud brings a next-generation hybrid architecture with cloud-native benefits to HBL’s data platform. CDP Private Cloud’s new approach to data management and analytics would allow HBL to access powerful self-service analytics. See other customers’ success here .

Banking 82

Banking Management Data Lake Professional Services 82

Cloudera

SEPTEMBER 9, 2022

analyst Sumit Pal, in “Exploring Lakehouse Architecture and Use Cases,” published January 11, 2022: “Data lakehouses integrate and unify the capabilities of data warehouses and data lakes, aiming to support AI, BI, ML, and data engineering on a single platform.” New innovations in the cloud have driven data explosions.

Architecture 97

Architecture Metadata Unstructured Data Machine Learning 97

Cloudera

AUGUST 4, 2021

Today, modern data governance programs must address not only an exponential increase in data volumes, but also cloud, hybrid architectures, and more reliance on advanced analytics. Cloudera provides secure data governance for hybrid architectures that span both cloud and on-premises data workloads. Companies indeed are taking notice.

Data Governance 83

Data Governance Government Metadata Data Security 83

Cloudera

APRIL 21, 2022

In 2022, it’s hard to believe, that for the first decades of the Information Age, the U.S. Its evolution to the present-day cloud-based package is a real-world case study that will likely live in IT textbooks for as long as use cases will be referenced. . billion for 2022 represents a full 11% of the nation’s total.

Electronics 84

Electronics Medical Hospitality Insurance 84

Cloudera

JUNE 30, 2022

No more lock-in, unnecessary data transformations, or data movement across tools and clouds just to extract insights out of the data. With Shared Data Experience (SDX) which is built in to CDP right from the beginning, customers benefit from a common metadata, security, and governance model across all their data. .

Data Lake 84

Data Lake Business Intelligence Metadata Data Warehouse 84

Monte Carlo

NOVEMBER 21, 2022

Prediction #1: Cloud Manages 75% Of All Data Workloads by 2024 (Tomasz) Image courtesy of Tomasz Tunguz. This was Tomasz’s first prediction , and based on an analyst report earlier this year showing the growth of cloud versus on-premises RDBMS revenue. Snowflake went from a hundred million in revenue to about 1.2

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Weekly

DECEMBER 4, 2022

The author discusses the need for richer metadata to support complex data lineage and evolving privacy requirements. I recently shared the thought and am excited to see Meta’s blog on static analysis of SQL queries. link] Barr Moses: What’s Next for Data Engineering in 2023? link] All rights reserved ProtoGrowth Inc, India.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

ProjectPro

JANUARY 31, 2022

As the demand for big data grows, an increasing number of businesses are turning to cloud data warehouses. The cloud is the only platform to handle today's colossal data volumes because of its flexibility and scalability. Launched in 2014, Snowflake is one of the most popular cloud data solutions on the market.

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

ProjectPro

NOVEMBER 7, 2016

This blog will give you an indepth insight into the architecture of hadoop and its major components- HDFS, YARN, and MapReduce. This blog post explains the Hadoop architecture and the factors to consider when designing and building a Hadoop cluster for production success. on 17th May 2022. The Apache Hadoop 3.3

Hadoop 40

Hadoop Architecture IT Big Data 40

Databand.ai

DECEMBER 29, 2022

The Top Data Strategy Influencers and Content Creators on LinkedIn Eitan Chazbani 2022-12-29 14:08:41 What’s the latest in the data world? Chad writes on data management, contracts, and products on his Substack blog and serves as an advisor and investor to several startups. But having the right network can make all the difference.

BI 52

BI Consulting Data Science Data Governance 52

Snowflake

JUNE 27, 2023

We’re going to summarize these new capabilities in this blog post. Between when we began tracking the SPI on August 25, 2022 to April 30, 2023, query duration time improved by 15 percent for customers’ stable workloads in Snowflake. This is one of the many ways Snowflake is helping customers get more value from the platform.

Data Governance 57

Data Governance Unstructured Data Government SQL 57

Cloudera

JUNE 16, 2022

CSP was recently recognized as a leader in the 2022 GigaOm Radar for Streaming Data Platforms report. For governance and security teams, the questions revolve around chain of custody, audit, metadata, access control, and lineage. Better yet, it works in any cloud environment. Not in the manufacturing space? Not to worry.

Kafka 84

Kafka Manufacturing Data Lake SQL 84