The Pragmatic Engineer

NOVEMBER 3, 2022

Printing it loses all this metadata. On 29 Oct, Saturday morning, a new project kicked off, with the first demo due on Monday morning. The first demo was scheduled for Monday, 31 October, in the morning: only two days later. Building the verification functionality requires work on the backend, web, iOS and Android apps.

Software Engineer 130

Software Engineer Software Engineering Coding Media 130

Data Engineering Podcast

JUNE 5, 2022

The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems. Atlan is the metadata hub for your data ecosystem. Data teams are increasingly under pressure to deliver.

Data Security 100

Data Security Metadata MongoDB Scala 100

Christophe Blefari

JUNE 16, 2023

Metis: building Airbnb’s next generation data management platform — The new manifesto for every data governance company /S. Malloy's Near Term Roadmap — I've shared recently Malloy demo , which was awesome. in seed round (second to continue develop a data analysis platform build for exploration.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Rockset

APRIL 18, 2023

To highlight these new capabilities, we built a search demo using OpenAI to create embeddings for Amazon product descriptions and Rockset to generate relevant search results. In the demo, you’ll see how Rockset delivers search results in 15 milliseconds over thousands of documents. Why use vector search?

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

ThoughtSpot

JANUARY 31, 2024

We’ll also be rolling out the BI industry’s most advanced human-in-the-loop feedback (HITL) capabilities that center human knowledge, business context, and security to build trust and accuracy across your organization. If you’re ready to spark your own Data Renaissance with ThoughtSpot , I invite you to request a demo.

BI 89

BI Business Intelligence Metadata Machine Learning 89

Data Engineering Podcast

DECEMBER 16, 2019

Summary Building clean datasets with reliable and reproducible ingestion pipelines is completely useless if it’s not possible to find them and understand their provenance. The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Precisely

JULY 21, 2023

Precisely – the global leader in data integrity and a proud Connect with Confluent program member – helps you build trust in your data to derive the insights your business users need. The Confluent and Precisely partnership helps make this possible and addresses the challenges of tomorrow – watch the demo now.

Data Integration 52

Data Integration Kafka Bytes Banking 52

Precisely

MAY 31, 2023

When attempting to build a data strategy, the primary obstacle organizations face is a lack of resources. Teams are building complex, hybrid, multi-cloud environments, moving critical data workloads to the cloud, and addressing data quality challenges. The end result is that users can’t trust their data because it lacks integrity.

Data Integration 52

Data Integration Metadata Data Governance Data 52

Towards Data Science

JUNE 19, 2023

Docker was a game-changer, revolutionizing the way we design, build, and run our cloud applications. I use Compose daily at work and for my personal projects to build and manage my data pipelines, and today I’d like to show how you can build your ETL projects with Compose.

PostgreSQL 80

PostgreSQL SQL Python Project 80

Rockset

SEPTEMBER 26, 2023

With easy access to data streams through Rockset’s integration with Confluent Cloud, businesses can: Create a real-time knowledge base for AI applications: Build a shared source of real-time truth for all your operational and analytical data, no matter where it lives for sophisticated model building and fine-tuning.

Metadata 64

Metadata Kafka Cloud Database 64

Snowflake

DECEMBER 7, 2023

Ultimately, this allows retailers to build an identity spine and customer 360 view. Known data can serve as metadata around customers’ purchasing history, affinities and buying intent. The abundant supply of first-party consumer data means they can build advertising networks that can withstand competition from walled gardens.

Media 85

Media Retail Digital Media Data 85

Cloudera

MARCH 17, 2023

ChatGPT is an excellent resource for gaining high-level insights and building awareness of any technology. The table format provides the necessary structure for the unstructured data that is missing in a data lake, using a schema or metadata definition, to bring it closer to a data warehouse.

Education 62

Education Unstructured Data Data Lake Data Warehouse 62

Confluent

APRIL 10, 2019

I recommend you familiarize yourself with that blog post and the accompanying white paper before proceeding, because this blog post builds on those core concepts. For example, when Confluent Monitoring Interceptors are configured on Kafka clients, they write metadata to a Kafka topic called _confluent-monitoring. Try it out yourself.

Kafka 86

Kafka Metadata Java Cloud 86

Workfall

AUGUST 23, 2023

Hands-On Required installations: To perform the demo, you require the following installations: Visual Studio Code (IDE): It is a feature-rich source code editor designed to enhance coding speed and efficiency by offering a wide range of useful tools. These components will serve as the building blocks of our extension’s functionality.

Metadata 83

Metadata Coding Project Utilities 83

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. Now every team can build a customer360 in Snowflake with RudderStack Profiles.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Rockset

AUGUST 28, 2023

We’ll create a query lambda with the following SQL query on our ShopEvents collection: SELECT COUNT(*) As ItemsSold FROM "Demo-Ecommerce".ShopEvents We can do this by creating a SQL query that will get all Checkout events for the previous day: INSERT INTO "Demo-Ecommerce".OrdersAnalytics ShopEvents e INNER JOIN "Demo-Ecommerce".Products

AWS 52

AWS SQL Datasets Metadata 52

Data Engineering Podcast

SEPTEMBER 23, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Ascend.io

MAY 23, 2023

However, Ascend always keeps a record of every change in its metadata, so the tables themselves are not needed to look back at schemas, when changes were made, who made them, etc. This creates a clear, easily distinguishable trail of full-fledged artifacts that some businesses require, tracked in Ascend metadata.

Datasets 52

Datasets Data Pipeline Metadata Database 52

Monte Carlo

APRIL 24, 2023

However, one of the biggest trends in data lake technologies, and a capability to evaluate carefully, is the addition of more structured metadata creating “lakehouse” architecture. Amazon S3 and/or Lake Formation Amazon S3 is a popular storage platform to build and store data lakes thanks to its high availability and low latency access.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Snowflake

APRIL 17, 2023

Another of my Forrester reports, from back in 2014 and (embarrassingly) entitled Find a Date to the Data Dance , discussed finding partners to build these new data products and services. Snowflake makes it easy to build and deliver data and applications to customers across clouds.

Retail 69

Retail Metadata Cloud Data Warehouse 69

phData: Data Engineering

DECEMBER 16, 2021

Where is your metadata stored? This feature will help build lineage and will let you see any current usage patterns. Once Alation is being used to curate metadata, machine learning pattern recognition is used to determine usage patterns inside the catalog itself. Is your data just being scanned, or is it being moved?

Metadata 52

Metadata Datasets ETL Tools Cloud 52

Cloudera

DECEMBER 10, 2020

Drawing on more than a decade of experience in building and deploying massive scale data platforms on economical budgets, Cloudera has designed and delivered a cost-cutting cloud-native solution – Cloudera Data Warehouse (CDW), part of the new Cloudera Data Platform (CDP). These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Netflix Tech

OCTOBER 27, 2021

The GUI can be extended with plugins, allowing the community to build integrations to other systems, custom visualizations, and embed upcoming features of Metaflow directly into its views. We also had no need to build a model-monitoring GUI yet, which is a wholly separate problem domain.

Python 89

Python Machine Learning Data Science Metadata 89

dbt Developer Hub

JANUARY 24, 2023

The solution: Autoscaling CI ​ As of this writing, autoscaling CI is functionality built only within a python package, dbtc ; dbtc is an unaffiliated python interface to the dbt Cloud Metadata and Administrative APIs. yml Key pieces: pr : A pull request trigger specifies which branches cause a pull request build to run.

Metadata 52

Metadata Python Cloud Data Warehouse 52

Rockset

APRIL 18, 2023

While Rockset can be used as a stand-alone feature store for real-time ML, in today’s demo we’re going to walk through how to use Rockset with the Feast Feature Store which is tailored to make machine learning feature management a breeze. We will use one of the files for the demo but the steps below can be applied to all files.

Machine Learning 52

Machine Learning Datasets Database Data Pipeline 52

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 102

Cloud Building Insurance Data Engineering 102

Snowflake

NOVEMBER 1, 2023

The Snowpark Model Registry now builds on a native Snowflake model entity with built-in versioning support, role-based access control and a SQL API for more streamlined management catering to both SQL and Python users. Check out the Snowpark ML demo from Snowday to see the latest launches in action. What’s Next?

Building 105

Building Python SQL Programming Language 105

Data Engineering Podcast

JUNE 26, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Cloudera

FEBRUARY 8, 2024

The snapshotId of the source tables involved in the materialized view are also maintained in the metadata. A Note on Iceberg materialized view specification Currently, the metadata needed for materialized views is maintained in Hive Metastore and it builds upon the materialized views metadata previously supported for Hive ACID tables.

Metadata 89

Metadata Data Warehouse BI AWS 89

Rockset

NOVEMBER 7, 2023

As a result, indexing of newly ingested vectors and metadata does not negatively impact search performance. There are several methods to indexing vectors including building a graph, tree data structure and inverted file structure. Rockset supports streaming updates to metadata and vectors in an efficient way.

Cloud 64

Cloud Metadata Database SQL 64

Data Engineering Podcast

NOVEMBER 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Zalando Engineering

NOVEMBER 29, 2017

The Kafka Streams API is available as a Java library included in Apache Kafka that allows you to build real-time applications and microservices that process data from Kafka. Monitoring Consumer Lag By looking at the metadata of a Kafka Consumer Group, we can determine a few key metrics. What is Apache Kafka’s Streams API?

Kafka 40

Kafka AWS Amazon Web Services Utilities 40

Snowflake

JANUARY 23, 2024

We have been making it easier and faster to build and manage ML models with Snowpark ML , the Python library and underlying infrastructure for end-to-end ML workflows in Snowflake. Many developers and enterprises looking to use machine learning (ML) to generate insights from data get bogged down by operational complexity.

Machine Learning 120

Machine Learning Metadata Python Telecommunication 120

Snowflake

FEBRUARY 29, 2024

And because these tiles often need to perfectly connect with each other to build out or populate the map, the tiles often come in one of several shapes: a square, triangle or hexagon. This grid, along with the metadata that represents the details of how the grid is constructed and used, is referred to as a spatial index.

Insurance 85

Insurance SQL BI Consulting 85

Data Engineering Podcast

MAY 18, 2021

In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification. What is the current state of the ecosystem for generating and sharing metadata between systems? What are your goals for the OpenLineage effort?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. For metadata management, you can configure Snowflake to manage your Iceberg data or use an external Iceberg catalog.

Metadata 85

Metadata AWS Data Lake Datasets 85

Knowledge Hut

DECEMBER 29, 2023

Front-end development, or client-side development, involves building the User Interface (UI) of a website or a web application, that determines how every part of a website will look and how it will work. The “head” tags (<head> and </head>) contain the metadata or information about the website.

Portfolio 98

Portfolio Amazon Web Services Programming Language Coding 98