Databand.ai

SEPTEMBER 12, 2022

How to analyze dataset performance and schema changes in Databand Eric Jones 2022-09-12 13:06:42 “Why did my dataset schema change?” Databand helps fix this problem by capturing the metadata from your datasets and then alerting you when dataset operations change unexpectedly. Yeah, we hear this question a lot too.

Datasets 52

Datasets Metadata Data Engineering Data Engineer 52

Cloudera

OCTOBER 11, 2022

A recent VentureBeat article , “4 AI trends: It’s all about scale in 2022 (so far),” highlighted the importance of scalability. They all should work on shared data of any type – with common metadata management – ideally open. The post AI at Scale isn’t Magic, it’s Data – Hybrid Data appeared first on Cloudera Blog.

Data Science 89

Data Science Aggregated Data Data Consulting 89

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This blog post aims to help you understand what you can do to get started with generative AI assisted SQL using Hue image version ​​2023.0.16.0

SQL 71

SQL Data Warehouse AWS Architecture 71

dbt Developer Hub

AUGUST 23, 2022

This configuration allows you to define any metadata dictionary that you want. Big thanks to Mike Fuller and Bennie Regenold for help ideating on this blog! There are clear, intuitive relationships between these entities! In order to do this at scale, we’ll make use of the meta config of dbt model.

Datasets 52

Datasets Metadata SQL Data Warehouse 52

Cloudera

JUNE 7, 2022

Common security, governance, metadata, replication, and automation enable CDP to operate as an integrated system. Integration, metadata and governance capabilities glue the individual components together.” . The post The Future Is Hybrid Data, Embrace It appeared first on Cloudera Blog.

IT 106

IT Unstructured Data Data Architecture Government 106

Data Engineering Weekly

OCTOBER 30, 2022

link] Tomasz Tunguz: 9 Predictions for data in 2023 It is almost the end of 2022, and it's time for 2023 predictions. I found the blog helpful in understanding the generative model’s historical development and the path forward. Til about [link] Thank you, Anton Bryzgalov, for contributing the blog to the Data Engineering Weekly.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. In this blog post, we will discuss what we built in support of that goal and some of the lessons we learned along the way.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Cloudera

JUNE 18, 2022

It added metadata that described the logical and physical layout of the data, enabling cost-based optimizers, dynamic partition pruning, and a number of key performance improvements targeted at SQL analytics. The post The Future of the Data Lakehouse – Open appeared first on Cloudera Blog.

Data Lake 89

Data Lake Data Warehouse BI SQL 89

Snowflake

MAY 8, 2023

By leveraging Snowflake’s comprehensive metadata on query history and operator statistics , alongside Acceldata’s advanced query fingerprinting techniques, users can attain a deeper comprehension of their workloads and pinpoint the most resource-intensive or unoptimized queries.

Technology 110

Technology SQL BI Utilities 110

Cloudera

SEPTEMBER 15, 2022

The gathering in 2022 marked the sixteenth year for top data and analytics professionals to come to the MIT campus to explore current and future trends. In June 2022, Barr Moses of Monte Carlo expanded on her initial article defining data observability. The post Demystifying Modern Data Platforms appeared first on Cloudera Blog.

Data Lake 78

Data Lake Analytics Application Cloud Storage Architecture 78

Rockset

AUGUST 2, 2023

In 2022, Windward embarked on several changes to its application prompting a reconsideration of its underlying data stack. In this blog, we’ll describe the new data platform for Windward and how it is API first, enables rapid product iteration and is architected for real-time, streaming data.

Database-centric 52

Database-centric PostgreSQL Transportation Insurance 52

Cloudera

JANUARY 5, 2023

According to Flexera’s 2022 State of the Cloud Report , respondents self-estimated that their organizations wasted 32% of cloud spend in 2021, up from 30% the previous year. These features provide businesses with a common metadata, security, and governance model across all their data.

Banking 52

Banking Machine Learning Insurance Data Architecture 52

Cloudera

JUNE 2, 2021

As an important part of achieving better scalability, Ozone separates the metadata management among different services: . Ozone Manager (OM) service manages the metadata of the namespace such as volume, bucket and keys. Datanode service manages the metadata of blocks, containers and pipelines running on the datanode. .

Metadata 94

Metadata Hadoop Certification Algorithm 94

The Pragmatic Engineer

NOVEMBER 3, 2022

Printing it loses all this metadata. pic.twitter.com/Rk9m1v1hmu — Leah Culver (@leahculver) October 28, 2022 Then, a few hours later, new instructions came from the top: stop printing and shred all documents with the code! ” The policies were not shared, and he was cut off from all systems, and terminated.

Software Engineer 130

Software Engineer Software Engineering Coding Media 130

Databand.ai

NOVEMBER 4, 2022

Re-Imagining Data Observability Ryan Yackel 2022-11-04 10:36:35 Data observability has become one of the hottest topics of the year – and for good reason. Data observability provides an end-to-end view into exactly what’s happening with data pipelines across an organization’s data fabric. And it does so in real time.

Data 52

Data Data Pipeline Retail Metadata 52

Rockset

APRIL 18, 2023

Learn more about how to use vector search on the Rockset blog. In high-latency scenarios, fraud goes undetected causing millions in losses, security vulnerabilities are left unchecked giving attackers an open door, recommendations fail to incorporate the latest user interactions becoming irrelevant.

Machine Learning 52

Machine Learning Datasets Database Data Pipeline 52

Cloudera

DECEMBER 16, 2022

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. Many of our customers use multiple solutions—but want to consolidate data security, governance, lineage, and metadata management, so that they don’t have to work with multiple vendors.

Database 88

Database Cloud Systems Management 88

Cloudera

APRIL 3, 2023

In this blog, we will share with you in detail how Cloudera integrates core compute engines including Apache Hive and Apache Impala in Cloudera Data Warehouse with Iceberg. We will publish follow up blogs for other data services. Iceberg basics Iceberg is an open table format designed for large analytic workloads.

Data Warehouse 78

Data Warehouse Metadata Java Data 78

Airbnb Tech

NOVEMBER 28, 2023

billion cumulative guest arrivals as of year-end 2022, Airbnb’s growth pushed us to an inflection point where diminishing data quality began to hinder our data practitioners. In this blog post, we share our innovative approach to scoring data quality, Airbnb’s Data Quality Score (“DQ Score”).

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

DoorDash Engineering

MARCH 14, 2023

These localized mechanisms ( as demonstrated in a systematic study on the subject published at SoCC 2022 ) are useful in preventing individual services from being overloaded, but they are not very effective in dealing with complex failures that involve interactions between services, which are characteristic of microservice failures.

Metadata 134

Metadata Java Database Systems 134

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. In this blog post, we’ll dive into the details of these features and the benefits for customers. In addition to Iceberg External Tables, we introduced Native Iceberg Tables.

Metadata 86

Metadata AWS Data Lake Datasets 86

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

dbt Developer Hub

AUGUST 30, 2022

As Tristan says in his latest blog, that’s how this industry moves forward. Capture all the metadata generated by dbt at the end of an invocation (project nodes, success rate, test results, etc), and store directly in Snowflake, Databricks, or BigQuery for immediate analysis. REGISTER NOW What's new ​ dbt Core v1.3 dbt_artifacts v1.2.0:

Programming 52

Programming Consulting Python BI 52

Data Engineering Weekly

JANUARY 15, 2023

2023 predictions from the panel are; Unified metadata becomes kingmaker. 3:23 PM ∙ Jan 6, 2022 18 Likes 4 Retweets But I like the term “Lots of Islands of Simplicity, but it’s a complex toolchain.” The blog narrates the causal machine-learning approach in designing promotional artwork for their TV shows.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. A comprehensive list of benefits offered by Cosmos can be found in the linked blog.

Process 91

Process Pipeline-centric Media Metadata 91

Engineering at Meta

MAY 3, 2023

File metadata and packaging Here’s an example of a NodeStealer file. We also observed metadata on the file that attempts to disguise the file as a product of “MicrosoftOffice:” An example of file metadata. Based on publicly available information, the malware C2 domain was registered with Namecheap on December 27th, 2022.

Media 110

Media Metadata Coding Database 110

AltexSoft

NOVEMBER 7, 2022

2022 Airflow user overview. Metadata database. A metadata database stores information about user permissions, past and current DAG and task runs, DAG configurations, and more. By default, Airflow handles metadata with SQLite which is meant for development only. Source: Apache Airflow. What is Apache Airflow used for?

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Cloudera

DECEMBER 12, 2022

For a thoughtful and entertaining analysis, I strongly recommend you spend a few minutes watching the keynote session by Pat Moorhead, CEO Moor Insights & Strategy, at the Evolve 2022 Data event in New York. In March 2022, Gartner published A Strategic Roadmap for Migrating Data Management to the Cloud. His delivery — priceless.

Cloud 88

Cloud Entertainment Telecommunication Insurance 88

Cloudera

NOVEMBER 17, 2022

While Cloudera CDH was already a success story at HBL, in 2022, HBL identified the need to move its customer data centre environment from Cloudera’s CDH to Cloudera Data Platform (CDP) Private Cloud to accommodate growing volumes of data. The post Habib Bank manages data at scale with Cloudera Data Platform appeared first on Cloudera Blog.

Banking 81

Banking Management Data Lake Professional Services 81

Cloudera

AUGUST 4, 2021

The data is profiled and enhanced with rich metadata—including operational, social, and business context—creating trusted and reusable data assets and making them discoverable. Putting metadata to work here is like using an automated roadmap to locate the right information more quickly than having to scan large data loads from top to bottom.

Data Governance 82

Data Governance Government Metadata Data Security 82

LinkedIn Engineering

JANUARY 19, 2024

Co- Authors: Aditya Hedge and Saumi Bandyopadhyay 2022 was a year driven by change for the Talent Acquisition industry, with nearly 50k company mergers and acquisitions completed worldwide. The requests and processing metadata are streamed over Kafka. The request endpoint, entity processors are exposed over Rest.li.

Recruitment 88

Recruitment Data Process Process Kafka 88

Cloudera

SEPTEMBER 9, 2022

analyst Sumit Pal, in “Exploring Lakehouse Architecture and Use Cases,” published January 11, 2022: “Data lakehouses integrate and unify the capabilities of data warehouses and data lakes, aiming to support AI, BI, ML, and data engineering on a single platform.” This is the promise of the modern data lakehouse architecture.

Architecture 96

Architecture Metadata Unstructured Data Machine Learning 96

Cloudera

APRIL 21, 2022

In 2022, it’s hard to believe, that for the first decades of the Information Age, the U.S. billion for 2022 represents a full 11% of the nation’s total. Shared catalog of data, metadata aids compliance requirements. Unsurprisingly errors were all too common, as people were involved in every transaction.

Electronics 83

Electronics Medical Hospitality Insurance 83

Cloudera

JUNE 30, 2022

With Shared Data Experience (SDX) which is built in to CDP right from the beginning, customers benefit from a common metadata, security, and governance model across all their data. . In February 2022, we introduced Apache Iceberg as a technical preview within CDP. Why integrate Apache Iceberg with Cloudera Data Platform?

Data Lake 83

Data Lake Business Intelligence Metadata Data Warehouse 83

Databand.ai

JULY 28, 2022

Niv Sluzki 2022-07-28 10:20:02 The term “data lineage” has been thrown around a lot over the last few years. These techniques include: Pattern-based lineage: Evaluates metadata for patterns in tables, columns, and reports rather than relying on any code to perform data lineage. What is Data Lineage?

Metadata 52

Metadata Data Lake Datasets Data Warehouse 52

Data Engineering Weekly

DECEMBER 4, 2022

The author discusses the need for richer metadata to support complex data lineage and evolving privacy requirements. I recently shared the thought and am excited to see Meta’s blog on static analysis of SQL queries. link] Barr Moses: What’s Next for Data Engineering in 2023? link] All rights reserved ProtoGrowth Inc, India.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Rock the JVM

AUGUST 9, 2022

For a more concrete example, we are going to write a program that will parse markdown files, extract words identified as tags, and then regenerate those files with tag-related metadata injected back into them. code, which was officially released on June 24th, 2022. Set up We’re going to base this discussion off of the latest ZIO 2.0

Scala 40

Scala Bytes Kafka Programming 40