Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Extending Atlas’ metadata model. The example 1_typedef-server.json describes the server typedef used in this blog. .

Data Governance 93

Data Governance Government Metadata Datasets 93

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 94

Management Kafka Metadata Media 94

phData: Data Engineering

DECEMBER 16, 2021

If you missed our first blog on what a data catalog is , be sure to check it out! In this blog, we’ll explore what the ideal team to support your data catalog looks like. The Business Steward will be most familiar with the business drivers mentioned in the first blog in this series. How will Human-Driven Metadata be Captured?

Metadata 52

Metadata Datasets Data Education 52

Lyft Engineering

OCTOBER 3, 2023

Finally, as the subject of this blog post, we can assess data quality via batch compute analytics on our data warehouse, providing a comprehensive albeit slower evaluation compared to the previously mentioned methods. Verity Our solution was to build Verity, our in-house data quality platform.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Data Engineering Weekly

OCTOBER 30, 2022

The author narrates the data contract, why it is the foundation for building data mesh, and how it glue data catalogs, glossary and data quality systems. I found the blog helpful in understanding the generative model’s historical development and the path forward. The current state of these systems is inherently passive systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

phData: Data Engineering

DECEMBER 16, 2021

In our previous blog in this series , we spent a lot of time exploring why a data catalog is valuable and who you might need to support it. Where is your metadata stored? This feature will help build lineage and will let you see any current usage patterns. What Things Should I Consider Before Choosing a Data Catalog Tool?

Metadata 52

Metadata Datasets ETL Tools Cloud 52

Cloudera

AUGUST 10, 2021

With Apache Ozone on the Cloudera Data Platform (CDP) , they can implement a scale-out model and build out their next generation storage architecture without sacrificing security, governance and lineage. For example, writing a Spark dataset to Ozone or launching a DDL query in Hive that points to a location in Ozone. With CDP 7.1.4

Hadoop 104

Hadoop Kafka Datasets Government 104

Knowledge Hut

OCTOBER 26, 2023

In this blog, we'll explore a curated selection of beginner-friendly machine learning projects that will not only help you grasp the fundamentals but also inspire your passion for this ever-evolving technology. It includes the UCI machine learning repository and dataset.

Machine Learning 52

Machine Learning Project Datasets Algorithm 52

Data Engineering Podcast

AUGUST 5, 2019

Lyft has build the Amundsen platform to address the problem of data discovery and in this episode Tao Feng and Mark Grover explain how it works, why they built it, and how it has impacted the workflow of data professionals in their organization. How does it compare to what WeWork is building with Marquez?

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Knowledge Hut

OCTOBER 26, 2023

In this blog, we will show some interesting AWS project ideas for all professionals, including beginners, intermediate, and advanced. You can learn how to build scalable and reliable applications, manage infrastructure using automation tools, and create efficient solutions that are cost-effective. Why is Learning AWS Important?

AWS 52

AWS Coding Project Cloud Computing 52

Knowledge Hut

MAY 9, 2024

In this blog post, we will look at some of the world's highest paying data science jobs, what they entail, and what skills and experience you need to land them. Responsibilities A data scientist is responsible for identifying data sources, preprocessing data, building predictive models, and analyzing data systems for optimization.

Data Science 52

Data Science Data Mining Data Architect Programming Language 52

Monte Carlo

MARCH 3, 2021

After all, any solution dedicated to keeping data pipelines reliable should take its own medicine when it comes to building the actual product. Instead, we only extract query logs, metadata, and aggregated statistics about data usage to ensure that your most critical data assets are as trustworthy and reliable as possible. What’s next?

Metadata 52

Metadata Data Lake Business Intelligence Certification 52

AltexSoft

AUGUST 3, 2021

This blog entry is particularly helpful to anyone who wants to. Data annotation follows a meticulous process of adding metadata to a dataset. This metadata is always in the form of tags, which then can be added to various data types like text, images, and video. Explaining Data Annotation for ML. Use Tight Bounding Boxes.

Datasets 59

Datasets High Quality Data Metadata Raw Data 59

AltexSoft

FEBRUARY 11, 2023

If you are not familiar with the above-mentioned concepts, we suggest you to follow the links above to learn more about each of them in our blog posts. The daily tasks of a data architect require more of a strategic thinking, while a data engineer’s workload is more about building the software infrastructure, which are technical tasks.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Rockset

AUGUST 28, 2023

Use case 2: Creating materialized views or development datasets Rockset supports automatic real-time rollups on ingestion for some data sources. In this example, we will again work on our e-commerce dataset and we’ll leverage AWS API Gateway to create a webhook that our query lambda can call to export the results of a query into an S3 bucket.

AWS 52

AWS SQL Datasets Metadata 52

dbt Developer Hub

AUGUST 23, 2022

Those who have been building data warehouses for a long time have undoubtedly encountered the challenge of building surrogate keys on their data models. This configuration allows you to define any metadata dictionary that you want. product_id 1 always means mozzarella), you need to build in ORDER BY statements to your models.

Datasets 52

Datasets Metadata SQL Data Warehouse 52

Rockset

AUGUST 2, 2023

To support this, Windward’s data platform needed to enable rapid iteration so they could quickly innovate and build more AI capabilities. In this blog, we’ll describe the new data platform for Windward and how it is API first, enables rapid product iteration and is architected for real-time, streaming data.

Database-centric 52

Database-centric PostgreSQL Transportation Insurance 52

Rockset

APRIL 18, 2023

Join VP of Engineering Louis Brandy and product manager John Solitario for the talk From Spam Fighting at Facebook to Vector Search at Rockset: How to Build Real-Time Machine Learning at Scale on May 17th. After collecting data, each event in the dataset was manually labeled “sus” for unusual behavior or “evil” for malicious behavior.

Machine Learning 52

Machine Learning Datasets Database Data Pipeline 52

DataKitchen

JANUARY 5, 2021

Data Governance/Catalog (Metadata management) Workflow – Alation, Collibra, Wikis. Is it building up or tearing down the communication and relationships that are critical to your mission? Prepackaged datasets – clean, accurate, privacy and security-aware. Data Science Workflow – Kubeflow, Python, R.

Data Governance 59

Data Governance Government Metadata Data Science 59

Rockset

SEPTEMBER 14, 2021

With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side. In this blog, we introduce how the Kafka Integration with native support for Confluent Cloud and Apache Kafka works and walk through how to run real-time analytics on event streams from Kafka.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Rockset

MARCH 18, 2024

In this blog, we’ll discuss how RAG fits into the paradigm of real-time data processing and show an example product recommendation application using both Kafka and Flink on Confluent Cloud together with Rockset. Building a real-time, contextual and trustworthy knowledge base for AI applications revolves around RAG pipelines.

Cloud 64

Cloud Building Metadata Kafka 64

ProjectPro

JULY 23, 2021

This blog presents some of the most unique and innovative AWS projects from beginner to advanced levels. Before we get into the technicalities on how one can leverage any AWS service and build some exciting AWS projects, here is a quick overview of AWS to understanding the cloud platform and its services. Table of Contents What is AWS?

AWS 52

AWS Project Amazon Web Services Cloud Computing 52

Cloudera

MAY 22, 2018

Two key new capabilities, Experiments and Models, let data scientists build, train, and deploy models in a unified workflow; security enhancements automate user administration. As data scientists iteratively develop models, they often experiment with datasets, features, libraries, and algorithms as well as tuning hyperparameters.

Data Science 41

Data Science Machine Learning Metadata Big Data 41

Databand.ai

NOVEMBER 4, 2022

Specifically, Databand collects metadata from all key solutions in the modern data stack, builds a historical baseline based on common data pipeline behavior, alerts on anomalies and rules based on deviations, and resolves through triage by creating smart communication workflows. But when the data comes through, we see six columns.

Data 52

Data Data Pipeline Retail Metadata 52

ProjectPro

JANUARY 15, 2021

And honestly, there are a lot of real-world machine learning datasets around you that you can opt to start practicing your fundamental data science and machine learning skills, even without having to complete a comprehensive data science or machine learning course. Table of Contents What is a dataset in machine learning?

Datasets 40

Datasets Machine Learning Retail Banking 40

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. Apache Iceberg is a high-performance open table format for petabyte-scale analytic datasets. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata.

Metadata 88

Metadata Data Warehouse BI AWS 88

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Knowledge Hut

JULY 26, 2023

In this blog, we will deep dive into database system applications in DBMS, and their components and look at a list of database applications. Columnar stores, such as Apache Cassandra and Apache HBase, organize data by columns rather than rows, allowing for faster read and write operations on huge datasets.

Database 52

Database NoSQL Telecommunication MongoDB 52

Rockset

JUNE 7, 2021

The rule of thumb is that if you are looking to build a real-time data processing system then the push approach should be used. If the source system wants to change its underlying database or move the particular dataset elsewhere, the target doesn’t need to change as it would with a pull system.

IT 40

IT Kafka Database MongoDB 40

Rockset

DECEMBER 22, 2020

There are ways to build relationships in Elasticsearch documents, most common are: nested objects, parent-child joins, and application side joins. Building with Elasticsearch In Elasticsearch I have three indexes, one for each of the data types: product, purchase, and review. To do this we will need to build three separate queries.

SQL 40

SQL Data MongoDB Aggregated Data 40

ProjectPro

SEPTEMBER 6, 2021

Project Idea: You can build a system that collects the prices of a product from different eCommerce websites and prepares a list of them. The website has a product listings page that you can use to extract metadata of the games it hosts. Project Idea: Using WebMD’s database, you can build a drug recommendation system.

Project 52

Project Programming Language Entertainment Python 52

Cloudera

FEBRUARY 22, 2022

The data architecture layer is one such area where growing datasets have pushed the limits of scalability and performance. The data explosion has to be met with new solutions, that’s why we are excited to introduce the next generation table format for large scale analytic datasets within Cloudera Data Platform (CDP) – Apache Iceberg.

Metadata 105

Metadata Datasets BI SQL 105

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. This blog series follows the manufacturing, operations and sales data for a connected vehicle manufacturer as the data goes through stages and transformations typically experienced in a large manufacturing company on the leading edge of current technology.

Data Pipeline 114

Data Pipeline Building Manufacturing Data Warehouse 114

DareData

FEBRUARY 28, 2024

In this post of the PyTorch Introduction, we’ll learn how to use custom datasets with PyTorch, particularly tabular, vision and text data PyTorch is one of the hottest libraries in the Deep Learning field right now. We’ve used few custom datasets in our examples and previous blog posts. Let’s start!

Datasets 52

Datasets Deep Learning Data Machine Learning 52

Cloudera

JULY 19, 2023

Introduction Ozone is an Apache Software Foundation project to build a distributed storage platform that caters to the demanding performance needs of analytical workloads, content distribution, and object storage use cases. The hardware specifications are included at the end of this blog.

Management 69

Management Metadata Datasets Architecture 69

Cloudera

MAY 19, 2021

In the previous blog post in this series, we walked through the steps for leveraging Deep Learning in your Cloudera Machine Learning (CML) projects. RAPIDS brings the power of GPU compute to standard Data Science operations, be it exploratory data analysis, feature engineering or model building. Get the Dataset. Introduction.

Machine Learning 136

Machine Learning Datasets Data Science Raw Data 136

Rockset

NOVEMBER 7, 2023

As a result, indexing of newly ingested vectors and metadata does not negatively impact search performance. In this blog, we’ll dig into how Rockset has fully integrated vector search into its search and analytics database. Users can continuously stream and index vectors fully isolated from search. Each cell is defined by a centroid.

Cloud 64

Cloud Metadata Database SQL 64