Data Engineering Podcast

APRIL 22, 2018

For this reason metadata management systems are built to track the journey of your business data to aid in analysis, presentation, and compliance. What are some of the types of information that you classify and collect as metadata? What are some of the challenges that are typically faced by metadata management systems?

Business Intelligence 113

Business Intelligence Metadata Management Data Governance 113

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This is a real game-changer for data analysts on all levels and will make SQL development faster, easier, and less error-prone.

SQL 72

SQL Data Warehouse AWS Architecture 72

Rock the JVM

FEBRUARY 5, 2023

Enter Giannis: Flink SQL is a powerful high level API for running queries on streaming (and batch) datasets. Streaming (and Batch) SQL 1.1 Batch SQL Queries operate on static data, i.e. on data stored on disk, already available and the results are considered complete. This is called a Dynamic Table.

SQL 61

SQL Kafka Metadata Database 61

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. In a simple words dbt sits on top of your raw data to organise all your SQL queries that are defining your data assets. You can also add metadata on models (in YAML). When I write dbt, I often mean dbt Core.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Cloudera

DECEMBER 21, 2023

Now imagine you had a personal assistant who knew everything about your data sets and was an expert in SQL, sitting alongside you every step of the way to help you quickly problem solve, write optimized code, explain queries, and much more. Well imagine it no longer, as Cloudera’s SQL AI Assistant is exactly that!

SQL 67

SQL Data Warehouse Business Analyst Data Schemas 67

Cloudyard

JANUARY 13, 2023

Though we can use GET_DDL command to extract the metadata. Snowflake Sql Scripting. Read Time: 1 Minute, 35 Second During this post we will discuss the multiple ways to extract the DDL from your snowflake database. But what if our database has huge list of tables and running GET_DDL on each table is not feasible approach.

SQL 52

SQL Python Metadata Java 52

Data Engineering Podcast

AUGUST 13, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java.

Metadata 100

Metadata MongoDB Scala MySQL 100

Engineering at Meta

NOVEMBER 30, 2022

UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. UPM takes in an SQL query as input and represents it as a hierarchical data structure called a semantic tree.

SQL 76

SQL Data Warehouse Metadata Coding 76

Data Engineering Podcast

SEPTEMBER 17, 2023

Linked data technologies provide a means of tightly coupling metadata with raw information. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. With Materialize, you can! Hex brings everything together.

Building 189

Building BI SQL Python 189

Data Engineering Podcast

NOVEMBER 26, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can! Rudderstack : ![Rudderstack]([link]

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Towards Data Science

DECEMBER 23, 2023

Photo by Ian Taylor on Unsplash This tutorial guides you through an analytics use case, analyzing semi-structured data with Spark SQL. Analytics with Spark SQL : Finally, we dive deeper, exploring the depths of the ocean with Spark SQL. Plus, we'll utilise Docker to introduce a best practice for setting up the environment.

SQL 73

SQL Data Analytics Hadoop Raw Data 73

Snowflake

JANUARY 23, 2024

Snowpark ML Operations: Model management The path to production from model development starts with model management, which is the ability to track versioned model artifacts and metadata in a scalable, governed manner. The Snowpark Model Registry API provides simple catalog and retrieval operations on models.

Machine Learning 120

Machine Learning Metadata Python Telecommunication 120

Data Engineering Podcast

FEBRUARY 5, 2023

Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Rockset

DECEMBER 17, 2021

If you were one of the 15,000 people who attended Coalesce 2021 , you will likely remember SQL Draw, the Slack-based game combining SQL with cartesian geometry, art, creativity and teamwork. If you missed it, you can read more about SQL Draw on the Omnata website. Query Lambdas make it easy to create data APIs.

SQL 52

SQL NoSQL Database Design Metadata 52

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. For example, a single table named ‘Customers’ is actually an aggregation of metadata that manages and references several data files, ensuring that the table behaves as a cohesive unit.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

JUNE 19, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Cloudera

SEPTEMBER 24, 2020

Written in C++, which is very CPU efficient, with a very fast query planner and metadata caching, Impala is optimized for low latency queries. Impala also has a very efficient run-time execution framework, using code generation, process-to-process communication, massive parallelism, and metadata caching. Join us for Racing for Results!

Data Warehouse 70

Data Warehouse SQL Engineering Metadata 70

Netflix Tech

DECEMBER 14, 2023

Streaming SQL on Data Mesh using Apache Flink Mark Cho, Guil Pires and Sujay Jain, Engineers from the Netflix Data Platform talk about how a managed Streaming SQL using Apache Flink can help unlock new Stream Processing use cases at Netflix.

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 97

Building Metadata Cloud Storage AWS 97

Rockset

MARCH 18, 2024

There are several challenges that emerge when building real-time RAG pipelines: Real-time delivery of embeddings & updates Real-time metadata filtering Scale and efficiency for real-time data In the following sections, we’ll discuss these challenges broadly and delve into how they apply more specifically to vector search and vector databases.

Cloud 64

Cloud Building Metadata Kafka 64

Netflix Tech

DECEMBER 3, 2022

This logic consists of the following parts: DDL code, table metadata information, data transformation and a few audit steps. DDL Often, the first step in a data pipeline is to define the target table structure and column metadata via a DDL statement. For the workflow orchestration we use Netflix homegrown Maestro scheduler.

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Data Science Blog: Data Engineering

MAY 20, 2024

This includes the creation of SQL Code, DACPAC files, SSIS packages, Data Factory ARM templates, and XMLA files. Support for Various Data Warehouses and Databases : AnalyticsCreator supports MS SQL Server 2012-2022, Azure SQL Database, Azure Synapse Analytics dedicated, and more. Data Lakes : It supports MS Azure Blob Storage.

Data Pipeline 52

Data Pipeline BI Data Lake Data Warehouse 52

Data Engineering Podcast

DECEMBER 18, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don't forget to thank them for their continued support of this show!

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

DECEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Data Engineering Podcast

NOVEMBER 6, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Building 130

Building IT Metadata MongoDB 130

Christophe Blefari

JUNE 16, 2023

I'm now under the Berlin rain with 20° When I write in these conditions I feel like a tortured author writing a depressing novel while actually today I'll speak about the AI Act, Python, SQL and data platforms. The ultimate SQL guide — After the last canva on data interviews, here's a canva to learn SQL.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Podcast

DECEMBER 29, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don't forget to thank them for their continued support of this show!

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

NOVEMBER 20, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. From analyzing your metadata, query logs, and dashboard activities, Select Star will automatically document your datasets.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

JUNE 26, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Snowflake

NOVEMBER 1, 2023

Snowflake’s latest ML announcements Develop interactively with SQL and Python in Snowflake Notebooks Snowflake Notebooks, in private preview, is a new development interface that offers an interactive, cell-based programming environment for Python and SQL users to explore, process and experiment with data in Snowpark.

Building 105

Building Python SQL Programming Language 105

Snowflake

DECEMBER 12, 2023

To give customers flexibility for how they fit Snowflake into their data architecture, Iceberg Tables can be configured to use either Snowflake or an external service such as AWS Glue as the table’s catalog to track metadata, with an easy, one-line SQL command to convert the table’s catalog to Snowflake in a metadata-only operation.

Metadata 90

Metadata Python AWS Government 90

Monte Carlo

MARCH 12, 2024

Dataplex works with your metadata. As you add new data sources to your data stores, Dataplex leverages the structured and unstructured metadata with built-in data quality checks to maintain integrity. Implement Custom SQL Rules 3. The SQL expression should evaluate to true (pass) or false (fail) per row.

Google Cloud 52

Google Cloud Metadata SQL Data Lake 52

Towards Data Science

NOVEMBER 4, 2023

In this case, SQL stops being an option in many cases making it difficult to query the data for those users who are not familiar with programming. What I like about it is that it makes it really easy to work with various data file formats, i.e. SQL, XML, XLS, CSV and JSON. Data warehouse exmaple.

Data Engineering 83

Data Engineering Data Engineer Engineering BI 83

Data Engineering Podcast

OCTOBER 23, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Project 130

Project Metadata MongoDB MySQL 130

Rockset

NOVEMBER 7, 2023

As a result, indexing of newly ingested vectors and metadata does not negatively impact search performance. Rockset makes hybrid search as easy as a SQL WHERE clause. Rockset has exploited the power of the search index with an integrated SQL engine so your queries are always executed efficiently.

Cloud 64

Cloud Metadata Database SQL 64

LinkedIn Engineering

JUNE 26, 2023

Enter Flink We've recently adopted Apache Flink at Linkedin, and Flink SQL has changed the way we think about data pipelines and stream processing. But since the introduction of the Table API and Flink SQL, Flink has evolved to support more general-purpose data pipelines. Flink SQL) for a single runtime (e.g.

Data Pipeline 112

Data Pipeline Kafka MySQL SQL 112

Data Engineering Podcast

JULY 17, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Data Management 130

Data Management Management Metadata MongoDB 130