Tweag

MAY 16, 2023

The manifest of a web app, the configuration of an Apache virtual host, an Infrastructure-as-Code (IaC) cloud deployment (Terraform, Kubernetes, etc.). That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure.

MySQL 131

MySQL Metadata Data Validation Coding 131

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Atlan is the metadata hub for your data ecosystem. What are some of the unique characteristics of that information?

Building 130

Building IT Metadata MongoDB 130

Monte Carlo

JULY 11, 2023

Data observability solutions also take testing a step further by providing root cause analysis tools that help teams proactively identify the root cause of data inaccuracies, as they relate to code, your data infrastructure, and even the data itself. This can include typographical errors, transposition of numbers, or incorrect formatting.

Data Cleanse 40

Data Cleanse Datasets Data Governance Government 40

Monte Carlo

AUGUST 30, 2023

More specifically, teams use dbt to write transformations as SQL queries, version control the code, and deploy transformations incrementally. For example, let’s say your company has customer data from both a Customer Relationship Management (CRM) system and a website’s analytics platform.

SQL 52

SQL Datasets Database High Quality Data 52

Monte Carlo

AUGUST 7, 2023

Prevention: Finally, data observability also provides mechanisms to prevent data issues from happening in the first place, like placing circuit breaking in pipelines and creating visibility around the impact code changes would make on data, among other proactive measures of preventing bad data from entering your pipelines in the first place.

Coding 52

Coding Data Software Engineer Software Engineering 52

Monte Carlo

MARCH 4, 2024

Table of Contents How Data Observability Works Made For The Lakehouse: Iceberg and Delta Transform in place: Dynamic Tables and Delta Live Tables Why This Matters How Data Observability Works Data observability provides full visibility into your data, systems, and code so you can find and fix bad data quickly.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. As data pipelines sprawl across disparate systems, it becomes harder to trace data lineage, track data quality, and quickly pinpoint issues when they arise.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Podcast

JULY 24, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. How have the definitions shifted over the past few decades?

Data Engineering 100

Data Engineering Data Engineer Lambda Architecture Engineering 100

Towards Data Science

MAY 26, 2023

Usually an external system is needed (e.g. The good news is, if you use Hamilton, there isn’t the need for another system to get lineage; making use of lineage is straightforward too! Lineage as Code With Hamilton, you do not need to add anything else, and you get lineage. Write code. Get lineage like this.

Metadata 82

Metadata Data Science Coding Machine Learning 82

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. The type system directives are available for the locations listed below. Though we practice defensive coding practices to handle unknown values, we still need a way to control the evolution of enums in a safe manner.

Metadata 86

Metadata Coding Banking Designing 86

LinkedIn Engineering

APRIL 25, 2023

Here are some of the primary differences compared to the widely used architecture in the industry: The GraphQL type system used at LinkedIn is completely autogenerated. Type System Generator The GraphQL type system used at LinkedIn is autogenerated by federating individual entity schemas from the Rest.li

Architecture 109

Architecture Metadata Java Transportation 109

Monte Carlo

JULY 10, 2023

That’s where the definition of data completeness comes in. Data profiling Data profiling uses a tool or programming language to surface metadata about your data. This can lead to data that’s incomplete in one system, even if it’s present in another. The same is true with data.

Data Collection 40

Data Collection Data Governance Government Data 40

Netflix Tech

JANUARY 10, 2024

Since then, the video pipeline has undergone substantial improvements and broad expansions: Starting with Standard Dynamic Range (SDR) at Standard-Definitions , we expanded the encoding pipeline to 4K and High Dynamic Range (HDR) which enabled support for our premium offering. This introductory blog focuses on an overview of our journey.

Process 91

Process Pipeline-centric Media Metadata 91

Engineering at Meta

OCTOBER 31, 2023

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. SCARF scans production data systems to identify tables or assets that are unused and safely removes them. This process can be unexpectedly difficult, because large software systems are inevitably interconnected.

Data 93

Data Metadata Coding Relational Database 93

Data Engineering Podcast

MAY 22, 2022

In this episode Ketan Umare and Haytham Abuelfutuh share the story of the Flyte project and how their work at Union is focused on supporting and scaling the code and community that has made Flyte successful. What are the patterns that you and the community have established to encourage discovery and reuse of granular task definitions?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. With Alteryx , you can create workflows without needing to code by using the provided automation building blocks. Atlan Atlan data catalog.

Metadata 52

Metadata Government Data Data Governance 52

Christophe Blefari

JUNE 16, 2023

Yann's vision goes toward AI systems learning and reasoning like animals and humans. Last week I shared nice QR Code generated with ControlNet, this week someone released a model on HuggingFace to do it QR Code Conditioned ControlNet (not related to the Chinese original paper) and you can even use the generator web UI.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Just like at first everyone had to code in a language, then everyone had to know how to incorporate packages from those languages — now we’re moving into, ‘ How do you incorporate AI that will write the code for you?’”

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. This is a great exploration of what it means to treat your data platform as a living system and apply state of the art engineering to it.

BI 100

BI Data Warehouse Metadata SQL 100

ProjectPro

FEBRUARY 8, 2023

When Glue receives a trigger, it collects the data, transforms it using code that Glue generates automatically, and then loads it into Amazon S3 or Amazon Redshift. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. You can produce code, discover the data schema, and modify it. Why Use AWS Glue?

AWS 98

AWS Scala Metadata Data Lake 98

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Ascend.io

FEBRUARY 28, 2024

The foundation of any AI system is high-quality data. Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. And crucially, what does the future hold for data engineering in an AI-driven world?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

LinkedIn Engineering

JUNE 26, 2023

However, we've found that this vertical self-service model doesn't work particularly well for data pipelines, which involve wiring together many different systems into end-to-end data flows. In those cases, creating an end-to-end data pipeline involves writing custom code to bridge the gaps.

Data Pipeline 112

Data Pipeline Kafka MySQL SQL 112

Towards Data Science

APRIL 6, 2023

Unit tests, which assess small pieces of code for performance on discrete, deterministic tasks, get overwhelmed by the variety of acceptable shapes and values that real-world data can take. Integrations : Your system should integrate with the core systems you already have, including databases, alerting (e.g., SelectStar).

Unstructured Data 79

Unstructured Data Metadata Data Coding 79

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 89

Datasets Metadata Algorithm Architecture 89

ThoughtSpot

JUNE 28, 2023

Applying analytics-as-code principles, we initially developed a scriptable approach with TML to automate the lifecycle management of analytic content across environments. Fine-grained personal access token that allows Read access to metadata and Read and Write access to code and commit statuses.

Metadata 72

Metadata Accessible Accessibility Utilities 72

Cloudera

NOVEMBER 13, 2020

The commonly-accepted best practice in database system design for years is to use an exhaustive search strategy to consider all the possible variations of specific database operations in a query plan. Metadata Caching. See the performance results below for an example of how metadata caching helps reduce latency.

Metadata 143

Metadata Coding SQL Database 143

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

DataKitchen

FEBRUARY 2, 2023

As an analogy, the DevOps space has seen consolidation in code storage, CI/CD, team workflow, value stream management, testing, and other tools into one platform. It’s code -> governance, not governance->code. This is because data teams realize that untested production systems produce error-prone output.

Metadata 52

Metadata BI Government ETL Tools 52

Data Engineering Podcast

FEBRUARY 6, 2022

Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. – You visually design the pipelines, and Prophecy generates clean Spark code with tests on git; then you visually schedule these pipelines on Airflow. How does it work? How does it work?

Data Pipeline 100

Data Pipeline Data Metadata Software Engineer 100

Monte Carlo

FEBRUARY 9, 2023

architecture (with some minor deviations) to achieve their data integration objectives around scalability and use of metadata. “A The other advantage is because we follow a standard design, we are able to generate a lot of our code using code templates and metadata. Curation Layer – Organizes the raw data. methodology.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Knowledge Hut

JULY 24, 2023

In the world of databases, data independence plays a vital role in making sure the flexibility and adaptability of database systems. It allows changes to be made at various levels of a database system without causing disruptions or requiring extensive modifications to the applications that rely on the data.

Database Design 52

Database Design Relational Database Metadata Database 52

Netflix Tech

OCTOBER 18, 2022

Due to its popularity, the number of workflows managed by the system has grown exponentially. The scheduler on-call has to closely monitor the system during non-business hours. As the usage increased, we had to vertically scale the system to keep up and were approaching AWS instance type limits.

Java 81

Java Data Machine Learning Systems 81

Data Engineering Podcast

MAY 3, 2021

They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Knowledge Hut

DECEMBER 29, 2023

They also ensure that the website loads correctly on all browsers (cross-browser), on different operating systems (cross-platform), and on different devices like mobiles, tablets, and computer screens (cross-device). For example, let us look at the code snippet below. <!DOCTYPE code files and assets.

Portfolio 98

Portfolio Amazon Web Services Programming Language Coding 98

Monte Carlo

JANUARY 23, 2024

“We can see this shift in the rise of data reliability engineering , and data engineering being responsible for managing (not building) data infrastructure and overseeing the performance of cloud-based systems.” Editing data like code. “If The data warehouse is the mirror of the organization in many ways. One example?

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Data Engineering Podcast

NOVEMBER 9, 2020

He also shares the internal architecture, how he approached the design to make it accessible and easy to use, and how it autodiscovers the schemas and metadata for your source systems. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. How is the Tree Schema system implemented?

Building 100

Building PostgreSQL BI Metadata 100