Snowflake

APRIL 11, 2024

At Snowflake, we believe in making the power of data accessible to all. Not only do we have a unique vantage point into the challenges faced by data analysts, we also possess rich metadata that feeds into Snowflake’s dedicated text-to-SQL model that Copilot leverages in combination with Mistral’s technology.

SQL 135

SQL AWS Data Analysis High Quality Data 135

Snowflake

DECEMBER 12, 2023

At our recent Snowday event, we announced a wave of Snowflake product innovations for easier application development, new AI and LLM capabilities, better cost management and more. If you missed the event or need a refresh of what was presented, watch any Snowday session on demand. Learn more about Iceberg Tables here. Learn more.

Metadata 86

Metadata Python AWS Government 86

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

Snowflake

APRIL 29, 2024

This strategy must incorporate business impact analyses as well as backup and recovery plans in the event of a security incident or loss of access to data. Access Control: Snowflake allows customers to define granular permissions for user roles, minimizing the risk of unauthorized access to sensitive data.

Transportation 81

Transportation Data Governance Government Consulting 81

Rockset

MARCH 18, 2024

For instance, GPT-4’s cutoff date was April 2023, so it would not be aware of any events or developments happening beyond that point of time. Real-time metadata filtering Streaming data on products in a catalog is used to generate vector embeddings as well as provide additional contextual information. What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Podcast

MARCH 18, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! Data Council Logo]([link] Join us at the event for the global data community, Data Council Austin.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Netflix Tech

JUNE 1, 2023

New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing. It also included metadata about ads, such as ad placement and impression-tracking events. We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch.

Algorithm 136

Algorithm Metadata Kafka Systems 136

François Nguyen

JANUARY 18, 2021

In every step,we do not just read, transform and write data, we are also doing that with the metadata. Every data governance policy about this topic must be read by a code to act in your data platform (access management, masking, etc.) Who has an access to this Data ? Last part, it was added the data security and privacy part.

Data Governance 130

Data Governance Metadata Government Data Pipeline 130

Data Engineering Podcast

AUGUST 21, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. Yeah…I can do that already!

Process 80

Process Kafka Datasets SQL 80

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Data Engineering Podcast

MAY 18, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. High-growth startups use Molecula’s feature store because of its unprecedented speed, cost savings, and simplified access to all enterprise data.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

ProjectPro

FEBRUARY 8, 2023

Users can schedule ETL jobs, and they can also choose the events that will trigger them. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. AWS Glue then creates data profiles in the catalog, a repository for all data assets' metadata, including table definitions, locations, and other features.

AWS 98

AWS Scala Metadata Data Lake 98

François Nguyen

MARCH 22, 2021

Data & Metadata : the data of the data product in many possible storages if needed but also the metadata (data on data) Infrastructure : you will need compute & storage but with the Serverless philisophy, we want to make it totally transparent and stay focus on the first two dimensions. What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

ThoughtSpot

JUNE 28, 2023

Getting started with version control APIs To begin, ensure that the following prerequisites are met: You have admin access (can administer Thoughtspot privilege) to connect ThoughtSpot to a Git repository and deploy commits. You have a Git repository and a branch that can be used as a default branch in ThoughtSpot.

Metadata 72

Metadata Accessible Accessibility Utilities 72

DoorDash Engineering

DECEMBER 5, 2023

Kafka is the pub-sub layer of the Iguazu pipeline, which provides real-time event delivery at DoorDash. Orchestrator service triggers an API call against the GitHub repo for Kafka Topics, creating a pull request, or PR, for a new topic and corresponding access control list (ACL) entry.

Kafka 90

Kafka Programming Language Metadata Architecture 90

Striim

FEBRUARY 5, 2024

It is not a standalone product but comprises key elements, including data integration, ensuring the smooth merging of data; data quality, maintaining high data standards; metadata management, organizing and understanding data context; and security, safeguarding data integrity.

Pharmaceutical 52

Pharmaceutical Data Cleanse Metadata Medical 52

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

IT 100

IT Metadata MongoDB MySQL 100

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

Data Engineering Podcast

MARCH 27, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Acryl]([link] The modern data stack needs a reimagined metadata management platform. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Towards Data Science

FEBRUARY 9, 2024

Eventing In many cases, the data platform itself needs to be responsible for, or at the very least inform, the generation of first party data. I break down eventing into two categories: Change Data Capture  — CDC The basic gist of CDC is using your database’s CRUD commands as a stream of data itself.

Building 81

Building Transportation Data Lake Metadata 81

Snowflake

OCTOBER 2, 2023

Access to a complete view of the customer is incredibly powerful. The Snowflake Data Cloud offering enables access to all of your customer data alongside infinite compute power to model it. That allows Profiles to automatically generate an identity graph and over 30 event-based user features out of the box.

Cloud 98

Cloud Building Insurance Data Engineering 98

Data Engineering Podcast

FEBRUARY 27, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Metadata repository Types of metadata (catalog, lineage, access, queries, etc.) Metadata repository Types of metadata (catalog, lineage, access, queries, etc.)

Designing 100

Designing Metadata Data Lake Relational Database 100

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Figure 1: sales group SELECT access.

Cloud 97

Cloud Kafka Metadata SQL 97

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. For metadata management, you can configure Snowflake to manage your Iceberg data or use an external Iceberg catalog.

Metadata 81

Metadata AWS Data Lake Datasets 81

Snowflake

NOVEMBER 2, 2023

To give customers flexibility for how they fit Snowflake into their architecture, Iceberg Tables can be configured to use either Snowflake or an external service like AWS Glue as the tables’s catalog to track metadata, with an easy one-line SQL command to convert to Snowflake in a metadata-only operation.

Data Lake 87

Data Lake Data Warehouse Cloud Unstructured Data 87

Cloudera

MARCH 14, 2023

When a draft is ready to be deployed in production, it is published to the Catalog, and can be productionalized with serverless DataFlow Functions for event-driven, micro-bursty use cases or auto-scaling DataFlow Deployments for low latency, high throughput use cases.

Data Pipeline 82

Data Pipeline Designing Kafka Metadata 82

Data Engineering Podcast

FEBRUARY 20, 2022

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. You can observe your pipelines with built in metadata search and column level lineage. What are some of the ways that you make your data accessible to AI/ML engineers?

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Snowflake

FEBRUARY 13, 2024

Snowpark Updates Model management with the Snowpark Model Registry – public preview Snowpark Model Registry is an integrated solution to register, manage and use models and their metadata natively in Snowflake. As a result, you should not rely on any forward-looking statements as predictions of future events. Learn more here.

Data Ingestion 83

Data Ingestion AWS Python Metadata 83

LinkedIn Engineering

DECEMBER 19, 2023

The framework incorporates proactive measures to monitor critical events, provides deployment insights, and issues proper notifications to enable seamless deployments. This metadata includes the namespace, file permissions, and the mapping of data blocks to datanodes. Accessibility of all namenodes. 0 missing blocks.

Big Data 92

Big Data Hadoop Metadata Data 92

Monte Carlo

JANUARY 4, 2024

The founders expect that all of their employees — 450 and counting — can access and query organizational data for their own analytics needs. As you can imagine, it’s crucial that Whatnot’s data is accessible, useful, and reliable. And as the cherry on top, Whatnot has one of the most robust self-serve data cultures we’ve encountered.

Data 40

Data Metadata Software Engineer Software Engineering 40

Rock the JVM

OCTOBER 1, 2023

In the first line of the file, we define syntax as proto3 which is the protobuf language version, next we specify package as com.rockthejvm.protos , this will be the namespace through which we access the case classes generated by Fs2Grpc. The Order service will be translated to a Scala trait through which we can access our function.

Scala 63

Scala Metadata Transportation Java 63

Snowflake

MARCH 29, 2023

But even without the catalog, Iceberg Tables are still accessible if the user directly points at appropriate file locations. This new Snowflake catalog is ideal for accessing Iceberg Tables managed and written by Snowflake’s engine. As a result, you should not rely on any forward-looking statements as predictions of future events.

Metadata 93

Metadata Scala Hadoop Relational Database 93

Netflix Tech

SEPTEMBER 29, 2022

Timestone clients can create queues, enqueue messages with user-defined deadlines and metadata, then dequeue these messages in an earliest-deadline-first (EDF) fashion. messages that belong to queue X and have metadata Y”) is also supported. Allow for dequeueing and queue depth querying using filters (metadata key-value pairs) 4.

Systems 85

Systems Metadata Media Kafka 85

DoorDash Engineering

APRIL 12, 2023

Additionally, having a centralized repository of metrics facilitates easy access to metrics, empowering all members of an organization to analyze experiment results. Data modeling Our platform requires access to data at the raw fact or event level, not just the aggregates. These measures are later aggregated to create metrics.

SQL 82

SQL Metadata Raw Data Government 82

Engineering at Meta

AUGUST 29, 2023

How BentoOperator works In order to address the majority of the concerns highlighted above, we perform the notebook execution state in a container without access to the network. In the event that the notebook execution fails for whatever reason, the schedule owner is automatically notified. We’re calling this: BentoOperator.

SQL 80

SQL Metadata Coding Data Analysis 80