Data Engineering Podcast

SEPTEMBER 11, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Systems 100

Systems Metadata Building MongoDB 100

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 100

Cloud Building Insurance Data Engineering 100

Monte Carlo

AUGUST 25, 2023

A warehouse can be a one-stop solution, where metadata, storage, and compute components come from the same place and are under the orchestration of a single vendor. For metadata organization, they often use Hive, Amazon Glue, or Databricks. One advantage of data warehouses is their integrated nature.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Christophe Blefari

MARCH 1, 2023

You can also add metadata on models (in YAML). In a nutshell the dbt journey starts with sources definition on which you will define models that will transform these sources to something else you'll need in your downstream usage of the data. You can read dbt's official definitions.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Data Engineering Podcast

FEBRUARY 19, 2023

It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

IT 147

IT Data Lake Metadata Data Warehouse 147

Monte Carlo

AUGUST 30, 2023

dbt (data build tool) is a SQL-based command-line tool that offers native testing features. In dbt, the built-in schema test `not_null` can be used to build completeness checks. There are some limits to keep in mind as you build a dbt testing discipline. And one of the most popular approaches is dbt testing.

SQL 52

SQL Datasets Database High Quality Data 52

Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale.

Building 100

Building PostgreSQL BI Metadata 100

Tweag

NOVEMBER 1, 2023

For example, we build Nickel with Nix, and we need to make sure the version of wasm-bindgen we pull from Nixpkgs is the same as the one set in Cargo.toml. The actual Nix code to build Nickel calls to a mysterious override function, which is a bespoke mechanism of Nixpkgs to simulate this kind of overriding.

Metadata 104

Metadata Programming Programming Language Coding 104

Christophe Blefari

JANUARY 20, 2024

Every company out there has his own definition for the data engineer role. The idea behind is to solve data problem by building software. What is data engineering As I said it before data engineering is still a young discipline with many different definitions. Who are the data engineers? Is it really modern?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

ThoughtSpot

OCTOBER 9, 2023

ThoughtSpot continually invests in this area, giving users the confidence to build the correct Answers needed for their analysis—and ensuring they can trust the data they are shown. How ThoughtSpot builds trust with data catalog connectors For many, the data catalog is still the primary home for metadata enrichment and governance.

Metadata 83

Metadata Government Data Data Governance 83

The Pragmatic Engineer

AUGUST 3, 2023

Now after 7 years, Google has announced it will retire Firebase Dynamic Links, but with no definite successor lined up. ” Because of how useful this product is, especially for app developers building on top of it, this announcement came as a surprise. We will announce more information in Q3 2023.”

Metadata 159

Metadata Engineering Building Technology 159

Data Engineering Podcast

OCTOBER 28, 2019

In an effort to create a better abstraction for building data applications Nick Schrock created Dagster. In this episode he explains his motivation for creating a product for data management, how the programming model simplifies the work of building testable and maintainable pipelines, and his vision for the future of data programming.

Building 100

Building Data Pipeline Programming Language Metadata 100

Towards Data Science

MAY 7, 2024

Now you have to decide what sorts of things to test in order to figure out definitively if the tool is the right commitment for you and the team. This will help create an overall picture of logging, metadata, latency, CDC, and all the other things that come with a pipeline. Gather the team together and determine what these things are.

ETL Tools 84

ETL Tools Metadata Data Integration Data Science 84

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. But as they add more data sources, implement real-time streaming pipelines, and build out a modern data lake, the complexity compounds quickly.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Podcast

JULY 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineering 100

Data Engineering Data Engineer Lambda Architecture Engineering 100

Lyft Engineering

OCTOBER 3, 2023

Data quality is an amorphous term, with various definitions depending on the context. In Verity, we defined data quality as follows: Verity’s Definition of Data Quality The measure of how well data can be used as intended. Five aspects of data quality with the definition in italics and an example in quotes.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Now, it’s going to be about leveling up to building the actual implementation of the unstructured data.” Those who don’t embrace it will be left behind. With the right prompt (this is key!),

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. What do you see as the closest alternatives?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. It is only used in our GraphQL linter that executes during the build time and prevents additions of new values to enums which are marked as final. We will discuss this in detail in the next section. are some entities.

Metadata 86

Metadata Coding Banking Designing 86

LinkedIn Engineering

APRIL 25, 2023

In this blog post, we will cover how the GraphQL layer is architected for use by our internal engineers to build member and customer facing applications. We had two choices in hand - invest more in Deco to address the pressing issues, or adopt GraphQL and build the necessary infrastructure around it for our needs. Given that our Rest.li

Architecture 109

Architecture Metadata Java Transportation 109

François Nguyen

MARCH 22, 2021

How do we build data products ? This is really for us the definition of a self serve platform. ” Code : all the code necessary to build a data product (data pipelines, API, policies). As you can see, this is in the code part where you are building your data pipelines, a misnomer because this is an over simplification.

Architecture 130

Architecture Technology Google Cloud Metadata 130

DoorDash Engineering

APRIL 12, 2023

The Metrics Layer, also known as a Semantic Layer, is a critical component of the modern data stack that has recently received significant industry attention offers a powerful solution to the challenge of standardizing metric definitions. Experimentation is one of the primary use cases that relies on metrics.

SQL 82

SQL Metadata Raw Data Government 82

Towards Data Science

NOVEMBER 4, 2023

Indeed, why would we build a data connector from scratch if it already exists and is being managed in the cloud? Typical Airflow architecture includes a schduler based on metadata, executors, workers and tasks. """DAG definition for recommendation_bespoke model training.""" Consider this example below.

Data Engineering 86

Data Engineering Data Engineer Engineering BI 86

Marc Lamberti

SEPTEMBER 22, 2023

Digging into the official definition, you will see that Airflow is noted as an open-source platform that programmatically authors, schedules, and monitors workflows. As long as you know Python, you can build data pipelines and fully utilize AIrflow. Airflow stores metadata in it (DAG runs, XComs, Task instances, etc.

Data Pipeline 130

Data Pipeline Python Metadata Database 130

Cloudera

MARCH 14, 2023

Key requirements for building data pipelines Every data pipeline starts with a business requirement. Developers who are tasked with building these data pipelines are looking for tooling that: Gives them a development environment on demand without having to maintain it.

Data Pipeline 82

Data Pipeline Designing Kafka Metadata 82

Christophe Blefari

JUNE 16, 2023

Metis: building Airbnb’s next generation data management platform — The new manifesto for every data governance company /S. Data graph create an entity diagram from the semantic definition with the API offers you an endpoint to launch pre-aggregations jobs from your scheduler. Telmai raises $5.5m seed round.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Atlas provides open metadata management and governance capabilities to build a catalog of all assets, and also classify and govern these assets.

Data Governance 94

Data Governance Government Metadata Datasets 94

dbt Developer Hub

DECEMBER 5, 2021

It's more important to think about how features build upon themselves (and each other) rather than how quickly they do so.* We will build on this later, but right now, the important thing to do is create a model in dbt, give dbt a command, and see that it properly produces the view or table in our warehouse that we expect.

Project 52

Project Building Metadata BI 52

LinkedIn Engineering

JULY 19, 2023

Co-Authors: Sumedh Sakdeo , Lei Sun , Sushant Raikar , Stanislav Pak , and Abhishek Nath Introduction At LinkedIn, we build and operate an open source data lakehouse deployment to power Analytics and Machine Learning workloads. While functional, our current setup for managing tables is fragmented.

Big Data 78

Big Data Data Management Management Metadata 78

Data Engineering Podcast

MAY 18, 2021

In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification. What is the current state of the ecosystem for generating and sharing metadata between systems? What are your goals for the OpenLineage effort?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

ThoughtSpot

JUNE 28, 2023

Fine-grained personal access token that allows Read access to metadata and Read and Write access to code and commit statuses. Applying analytics-as-code principles, we initially developed a scriptable approach with TML to automate the lifecycle management of analytic content across environments. vcs/git/config/create REST API v2.0

Metadata 72

Metadata Accessible Accessibility Utilities 72

Snowflake

DECEMBER 12, 2023

To give customers flexibility for how they fit Snowflake into their data architecture, Iceberg Tables can be configured to use either Snowflake or an external service such as AWS Glue as the table’s catalog to track metadata, with an easy, one-line SQL command to convert the table’s catalog to Snowflake in a metadata-only operation.

Metadata 89

Metadata Python AWS Government 89

DoorDash Engineering

DECEMBER 5, 2023

Key terminology Here are definitions and links to further documentation about the tools we used. Exposes a unique hardware definition language (HDL) for configuring compute resources. All Topics, even those without any messages, include some metadata that is backed up to disk. Terraform : Infrastructure-as-code (IaC) platform.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. You can observe your pipelines with built in metadata search and column level lineage. What is your working definition of that term?

BI 100

BI Data Warehouse Metadata SQL 100

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. With Alteryx , you can create workflows without needing to code by using the provided automation building blocks. Atlan Atlan data catalog.

Metadata 52

Metadata Government Data Data Governance 52

Cloudera

SEPTEMBER 29, 2022

The data product is not just the data itself, but a bunch of metadata that surrounds it — the simple stuff like schema is a given. And during this time they will by definition have a hybrid architecture. What are the different definitions of hybrid architectures? There are many definitions of the hybrid data architecture.

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). But as models go, the idea is easier than the execution.

Government 80

Government Metadata Data Lake Data 80