Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Gone are the months of complex data wrangling and the constraints of no-code SaaS tools.

Cloud 101

Cloud Building Insurance Data Engineering 101

Christophe Blefari

MARCH 1, 2023

You can also add metadata on models (in YAML). a macro — a macro is a Jinja function that either do something or return SQL or partial SQL code. You can read dbt's official definitions. All your business logic will be in the model select statements. You have to define sources in YAML files.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Towards Data Science

AUGUST 8, 2023

Pioneering Data Observability: Data, Code, Infrastructure, & AI The four dimensions of data observability: data, code, infrastructure, and ai? Unreliable data doesn’t live in a silo… it’s impacted by all three ingredients of the data ecosystem: data + code + infrastructure. You look at the code.

Coding 70

Coding Data Software Engineer Software Engineering 70

Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale.

Building 100

Building PostgreSQL BI Metadata 100

Tweag

NOVEMBER 1, 2023

If you control the source code, you can change the value provided to the configuration-as-a-function at the call-site. It also happens that one doesn’t control the call-site or the function’s code, because it’s in a library. In JSON, the best you can do is to copy and paste, praying for the different copies to not drift away with time.

Metadata 103

Metadata Programming Programming Language Coding 103

Monte Carlo

AUGUST 7, 2023

Prevention: Finally, data observability also provides mechanisms to prevent data issues from happening in the first place, like placing circuit breaking in pipelines and creating visibility around the impact code changes would make on data, among other proactive measures of preventing bad data from entering your pipelines in the first place.

Coding 52

Coding Data Software Engineer Software Engineering 52

Monte Carlo

AUGUST 30, 2023

dbt (data build tool) is a SQL-based command-line tool that offers native testing features. More specifically, teams use dbt to write transformations as SQL queries, version control the code, and deploy transformations incrementally. In dbt, the built-in schema test `not_null` can be used to build completeness checks.

SQL 52

SQL Datasets Database High Quality Data 52

The Pragmatic Engineer

AUGUST 3, 2023

These links were especially helpful for: Promotions and marketing campaigns QR codes Content sharing links that “just work” Converting desktop users to mobile ones The shutdown Dynamic links powered Firebase Invites: an app invite service where users could send app invite links to their friends, to drive installation of the app.

Metadata 159

Metadata Engineering Building Technology 159

Data Engineering Podcast

JULY 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineering 100

Data Engineering Data Engineer Lambda Architecture Engineering 100

Data Engineering Podcast

MAY 22, 2022

In this episode Ketan Umare and Haytham Abuelfutuh share the story of the Flyte project and how their work at Union is focused on supporting and scaling the code and community that has made Flyte successful. What are the core primitives that Flyte exposes for building up complex workflows? What do you see as the closest alternatives?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

François Nguyen

MARCH 22, 2021

How do we build data products ? This is really for us the definition of a self serve platform. How to build Data Products or never call me Data Pipeline any more You have this interesting schema in her second article on Data Mesh by Zhamak Dehghani : “Data mesh introduces the concept of data product as its architectural quantum.

Technology 130

Technology Architecture Google Cloud Metadata 130

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. It is easy to update the client code for web applications, but for the mobile native apps shipped to the app store, it is impossible to update the client code. We will discuss this in detail in the next section.

Metadata 85

Metadata Coding Banking Designing 85

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. But as they add more data sources, implement real-time streaming pipelines, and build out a modern data lake, the complexity compounds quickly.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Just like at first everyone had to code in a language, then everyone had to know how to incorporate packages from those languages — now we’re moving into, ‘ How do you incorporate AI that will write the code for you?’”

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Christophe Blefari

JUNE 16, 2023

Last week I shared nice QR Code generated with ControlNet, this week someone released a model on HuggingFace to do it QR Code Conditioned ControlNet (not related to the Chinese original paper) and you can even use the generator web UI. The past often changes, whether because of code or data. This is metrics drift. seed round.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

LinkedIn Engineering

APRIL 25, 2023

In this blog post, we will cover how the GraphQL layer is architected for use by our internal engineers to build member and customer facing applications. We had two choices in hand - invest more in Deco to address the pressing issues, or adopt GraphQL and build the necessary infrastructure around it for our needs.

Architecture 109

Architecture Metadata Java Transportation 109

Towards Data Science

MAY 7, 2024

Now you have to decide what sorts of things to test in order to figure out definitively if the tool is the right commitment for you and the team. This will help create an overall picture of logging, metadata, latency, CDC, and all the other things that come with a pipeline. Gather the team together and determine what these things are.

ETL Tools 77

ETL Tools Metadata Data Integration Data Science 77

Towards Data Science

MAY 26, 2023

Lineage as Code With Hamilton, you do not need to add anything else, and you get lineage. Write code. By virtue of writing code in the Hamiltonian way, you are defining computation within functions, and then specifying via function input arguments, how things connect, encoding lineage. Get lineage like this. Image by author.

Metadata 79

Metadata Data Science Coding Machine Learning 79

Towards Data Science

NOVEMBER 4, 2023

This would be the right way to go for data analyst teams that are not familiar with coding. Indeed, why would we build a data connector from scratch if it already exists and is being managed in the cloud? Typical Airflow architecture includes a schduler based on metadata, executors, workers and tasks.

Data Engineering 79

Data Engineering Data Engineer Engineering BI 79

ProjectPro

FEBRUARY 8, 2023

When Glue receives a trigger, it collects the data, transforms it using code that Glue generates automatically, and then loads it into Amazon S3 or Amazon Redshift. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. You can produce code, discover the data schema, and modify it. Why Use AWS Glue?

AWS 98

AWS Scala Metadata Data Lake 98

dbt Developer Hub

DECEMBER 5, 2021

It's more important to think about how features build upon themselves (and each other) rather than how quickly they do so.* We will build on this later, but right now, the important thing to do is create a model in dbt, give dbt a command, and see that it properly produces the view or table in our warehouse that we expect.

Project 52

Project Building Metadata BI 52

DoorDash Engineering

DECEMBER 5, 2023

Key terminology Here are definitions and links to further documentation about the tools we used. Terraform : Infrastructure-as-code (IaC) platform. Exposes a unique hardware definition language (HDL) for configuring compute resources. All Topics, even those without any messages, include some metadata that is backed up to disk.

Kafka 91

Kafka Programming Language Metadata Architecture 91

ThoughtSpot

JUNE 28, 2023

Applying analytics-as-code principles, we initially developed a scriptable approach with TML to automate the lifecycle management of analytic content across environments. Fine-grained personal access token that allows Read access to metadata and Read and Write access to code and commit statuses.

Metadata 72

Metadata Accessible Accessibility Utilities 72

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. How does it work?

BI 100

BI Data Warehouse Metadata SQL 100

Engineering at Meta

OCTOBER 31, 2023

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. This is the third and final post in our series on Meta’s Systematic Code and Asset Removal Framework (SCARF). SCARF coordinates several kinds of tasks for each data system: metadata collection (e.g.,

Data 98

Data Metadata Coding Relational Database 98

Data Engineering Podcast

MAY 18, 2021

In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification. What is the current state of the ecosystem for generating and sharing metadata between systems? What is the current state of the project?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. With Alteryx , you can create workflows without needing to code by using the provided automation building blocks. Atlan Atlan data catalog.

Metadata 52

Metadata Government Data Data Governance 52

Snowflake

DECEMBER 12, 2023

To give customers flexibility for how they fit Snowflake into their data architecture, Iceberg Tables can be configured to use either Snowflake or an external service such as AWS Glue as the table’s catalog to track metadata, with an easy, one-line SQL command to convert the table’s catalog to Snowflake in a metadata-only operation.

Metadata 89

Metadata Python AWS Government 89

LinkedIn Engineering

JUNE 26, 2023

These self-service integrations are typically owned by the teams that build and support the underlying infrastructure. Just as important, a growing number of use cases are driven by developers building applications. This means that developers frequently encounter gaps in self-service, requiring them to build their own solutions.

Data Pipeline 112

Data Pipeline Kafka MySQL SQL 112

Knowledge Hut

DECEMBER 29, 2023

Front-end development, or client-side development, involves building the User Interface (UI) of a website or a web application, that determines how every part of a website will look and how it will work. For example, let us look at the code snippet below. <!DOCTYPE The below code snippets show how styles are added using CSS. <html>

Portfolio 98

Portfolio Amazon Web Services Programming Language Coding 98

Towards Data Science

APRIL 6, 2023

In today’s age of digital business, an increasing number of decisions companies make when it comes to delivering customer experience, building trust, and shaping their business strategy begins with accurate data. That’s possibly correct — but the size of the distributional shift should definitely lead to an anomaly alert!

Unstructured Data 76

Unstructured Data Metadata Data Coding 76

Netflix Tech

NOVEMBER 13, 2023

The detailed temporal metadata SMAD provides about speech and music regions in a polyphonic audio mixture are a first step for structural audio segmentation, indexing and pre-processing audio for the following downstream tasks. The heavy usage of ambient sounds and sound effects blurs the boundaries between active music regions and non-music.

Datasets 89

Datasets Metadata Algorithm Architecture 89

Monte Carlo

JANUARY 23, 2024

In short, Maxime argues that to effectively scale data science and analytics in the future, data teams needed a specialized engineer to manage ETL, build pipelines, and scale data infrastructure. Enter, the data engineer. As a result, being a data engineer was both a blessing and a curse.

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Data Engineering Podcast

MAY 3, 2021

They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Monte Carlo

FEBRUARY 9, 2023

architecture (with some minor deviations) to achieve their data integration objectives around scalability and use of metadata. “A The other advantage is because we follow a standard design, we are able to generate a lot of our code using code templates and metadata. This layer has minimal transformation rules.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Monte Carlo

MARCH 28, 2023

Monte Carlo joined the fray and said, “Maybe having engineers manually code unit tests isn’t the best way to ensure data quality.” Today, data engineers continue to stomp on hard coded pipelines and on-premises servers as they march up the modern data stack slope of enlightenment. Speed to insights. Cons : Is it too much freedom?

Data Warehouse 126

Data Warehouse Raw Data Data Pipeline Software Engineer 126

Data Engineering Podcast

FEBRUARY 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. How does it work?

Data Pipeline 100

Data Pipeline Data Metadata Software Engineer 100