Data Engineering Digest

Data News — Week 24.02

Christophe Blefari

JANUARY 14, 2024

I actually cover data engineering and how to put data stuff into production. dbt meta tag — A list of the companies habing product features depending on the meta tag. Lead Data Engineer career guide — Detailed skillsets needed to be a lead data engineer. It's quite a broad subject.

Google Cloud

Google Cloud Data SQL Python

Building DoorDash’s Product Knowledge Graph with Large Language Models

DoorDash Engineering

APRIL 23, 2024

Historically, this SKU enrichment of extracting and tagging attributes has been a purely manual process led by contract operators. For instance, given the item name “Dove Silk Glow Body Wash 500 ml,” a token classifier would tag each entity in the item name as shown in Table 1. This approach further boosted our coverage.

Building

Building Retail Manufacturing Unstructured Data

Web Performance Regression Detection (Part 2 of 3)

Pinterest Engineering

MAY 24, 2024

Michelle Vu | Web Performance Engineer; Fighting regressions has been a priority at Pinterest for many years. We instrumented timings for when critical chunks of HTML, such as important script tags, preload tags, and the LCP image tag, are yielded from the server.

Engineering

Engineering Programming Process Coding

Webinars

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Understanding User Needs and Satisfying Them

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

Leading the Development of Profitable and Sustainable Products

How To Get Promoted In Product Management

MORE WEBINARS

4 Reasons Why Data Engineers Hate Google Tag Manager

RudderStack

NOVEMBER 2, 2021

Here are 4 of the reasons GTM and data engineers don’t get along plus a better solution. Modern teams need a more robust data integration solution than GTM.

Data Engineering

Data Engineering Data Engineer Engineering Management

Why did Google close its coding competitions after 20 years?

The Pragmatic Engineer

MARCH 3, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. Problems were modeled off a real Google engineering challenge. This means that two photos can, but don't have to, have tags in common.” We cover one out of five topics in yesterday's subscriber-only The Scoop issue.

Coding

Coding IT Software Engineer Software Engineering

Create a New React Project From Scratch [Step-by-Step Guide]

Knowledge Hut

OCTOBER 29, 2023

The user computer is a client and google search engine act as server. HTML is composed of various tags which defines the document and it's semantic. Semantic add meaning to the content as expressed by enclosing tags and it is not purely for visual presentation purpose. Then we need to use < p > tag.

Project

Project Programming Language Java Coding

Introducing Vector Search on Rockset: How to run semantic search with OpenAI and Rockset

Rockset

APRIL 18, 2023

We’re excited to introduce vector search on Rockset to power fast and efficient search experiences, personalization engines, fraud detection systems and more. Before vector search, search experiences primarily relied on keyword search, which frequently involved manually tagging data to identify and deliver relevant results.

Unstructured Data

Unstructured Data Metadata Machine Learning SQL

How DoorDash Migrated from StatsD to Prometheus

DoorDash Engineering

AUGUST 1, 2023

Challenges Faced With StatsD StatsD was a great asset for our early observability needs, but we began encountering constraints such as losing metrics during surge events, difficulties with naming/standardized tags, and a lack of reporting tools. These common tags are useful to create common dashboards and alerts to monitor service health.

AWS

AWS Transportation Programming Language Government

Now Featuring: Orchestration Lineage

Monte Carlo

MARCH 12, 2024

For example, some of the first questions data engineers want to know following an incident are: What DAG or workflow is populating this table? Now the data engineer can restart the DAG or take whatever additional steps may be required to resolve the issue. What was the result and duration of the recent task runs for this DAG?

BI

BI Metadata Data Pipeline Data Engineering

Bun: lessons from disrupting a tech ecosystem

The Pragmatic Engineer

SEPTEMBER 22, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). world by storm.

Programming Language

Programming Language Coding Project Engineering

Type-safe data processing pipelines

Tweag

APRIL 26, 2023

Defunctionalization — replacing functional arguments in higher-order functions with tags denoting the context of those functions (Wikipedia has a good article on this). Tags or strings In the above, the different transformations are denoted by different types ( data Prenex , for instance).

Data Process

Data Process Process Programming Data

How to get started with dbt

Christophe Blefari

MARCH 1, 2023

dbt is a development framework that combines modular SQL with software engineering best practices to make data transformation reliable, fast, and fun. It was the previous tag line dbt Labs had on their website. Jinja templating — Jinja is a templating engine that seems to exist forever in Python.

Data Warehouse

Data Warehouse SQL Metadata Raw Data

Inside Pollen's Software Engineering Salaries

The Pragmatic Engineer

JANUARY 12, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. Here are a few details from the data points, focusing on software engineering compensation. Source: The Pragmatic Engineer blog. Source: The Pragmatic Engineer blog. To get this newsletter every week, subscribe here.

Software Engineer

Software Engineer Software Engineering Engineering Raw Data

The Complete Front-End Developer Roadmap 2024

Knowledge Hut

DECEMBER 29, 2023

A Front-End Developer is a type of Software Engineer who works on front-end development. This markup language is made up of tags that label or indicate what the content should appear as. The HTML opening tag (<html>) and the HTML closing tag (</html>) indicate the start and end of a HTML structure.

Portfolio

Portfolio Amazon Web Services Programming Language Coding

Cloudera Uses CDP to Reduce IT Cloud Spend by $12 Million

Cloudera

OCTOBER 18, 2022

This was envisioned as a one-stop solution to serve the different personas around cloud cost awareness: from senior leaders down to the frontline engineer. The solution was prototyped in Cloudera Data Science Workbench (CDSW) , and is built using Python and PySpark, which is scheduled using Cloudera Data Engineering.

Cloud

Cloud IT Data Warehouse AWS

Transforming Delimited String Columns into Rows with Snowflake

RandomTrees

MARCH 22, 2024

Tagging and Categorization : Delimited string columns are commonly used for tagging or categorizing data. By splitting these tags into rows, we can analyze the distribution of tags, identify trends, and perform targeted analysis based on specific tags.

Media

Media Healthcare Datasets Electronics

Improving the code quality of your dbt models with unit tests and TDD

Towards Data Science

JUNE 2, 2023

How to improve the code quality of your dbt models with unit tests and TDD All you need to know to start unit testing your dbt SQL models Photo by Christin Hume on Unsplash If you are a data or analytics engineer, you are probably comfortable writing SQL models and testing for data quality with dbt tests.

Coding

Coding SQL Software Engineer Software Engineering

Data Catalog vs. Data Dictionary: 3 Essential Differences

Monte Carlo

MAY 20, 2024

User Annotations and Tags : Collaborative input from users such as comments, usage tips, and tags to improve discoverability and utility. It’s an essential resource for database administrators, data engineers, and developers who need to ensure that the data within a specific system or database is used consistently and correctly.

Datasets

Datasets Government Data Business Analyst

How Financial Platform Tide Automated GDPR Compliance With Atlan and Snowflake

Snowflake

NOVEMBER 13, 2023

After consulting with Atlan, Tide decided to use Atlan Playbooks to automatically identify, tag and classify data. With items from Snowflake appropriately tagged, Tide could mask PII such as account numbers, email addresses and phone numbers in the Atlan UI.

Finance

Finance Consulting Data Warehouse Data Governance

Lineage + Hamilton in 10 minutes

Towards Data Science

MAY 26, 2023

In Hamilton, the way to add extra metadata is through the use of the ` @tag ` and ` @tag_outputs ` decorators. This provides flexibility and the ability for you and your organization to define tags and values that make sense for your context. Use @tag to annotate functions. """ #. contents of function not important.

Metadata

Metadata Data Science Coding Machine Learning

Costwiz: Saving cost for LinkedIn enterprise on Azure

LinkedIn Engineering

JULY 27, 2023

Costwiz creates accountability by notifying organization owners to assign these recommendations to engineers or SREs and tracking the workflow of a recommendation. These tools expose the right set of data without us having to worry about engineering efforts for dedicated reporting UI or a workflow for email alerts and reporting dashboards.

Metadata

Metadata Utilities Cloud Data Lake

Migrating Netflix to GraphQL Safely

Netflix Tech

JUNE 14, 2023

By the summer of 2020, many UI engineers were ready to move to GraphQL. The GraphQL shim enabled client engineers to move quickly onto GraphQL, figure out client-side concerns like cache normalization, experiment with different GraphQL clients, and investigate client performance without being blocked by server-side migrations.

Utilities

Utilities Java Systems AWS

DotSlash: Simplified executable deployment

Engineering at Meta

FEBRUARY 6, 2024

config: github/workflows/dotslash-config.json # Tag for the release to to target. config: github/workflows/dotslash-config.json # Tag for the release to to target. The other required input is the ID of the release, which in GitHub, is defined by a Git tag. We look forward to hearing from you!

Metadata

Metadata Coding Building Project

Docker Vs Virtual Machines(VMs)

Knowledge Hut

MAY 2, 2024

Top Paas providers: AWS beanstalk , Oracle Cloud Platform (OCP) , Google App Engine IaaS – Infrastructure as a Service – Provide infrastructure such as servers, physical storage, networking, memory devices etc. Refer Diagram (10) that shows a Docker engine with 3 containers residing on host OS (MAC OS).

Python

Python Bytes Cloud Computing Amazon Web Services

Upgrade your Modern Data Stack

Christophe Blefari

SEPTEMBER 28, 2023

I often say that data engineering is boring, insanely boring. When you are a data engineer you're getting paid to build systems that people can rely on. The central storage can be cloud storage, a warehouse, a real-time system, while the SQL engine can be a data warehouse or a dedicated processing engine.

Cloud Storage

Cloud Storage Big Data Hadoop SQL

Understanding GraphQL Directives: Practical Use-Cases at Zalando

Zalando Engineering

OCTOBER 18, 2023

Certain directives are built into GraphQL, like @skip , @include , @deprecated , and @specifiedBy , and are supported by all GraphQL engines. In the second step, after execution, we match the error paths and omit the error tag for those extracted paths. See our blog post GraphQL persisted queries and schema stability for more details.

Metadata

Metadata Coding Banking Designing

Cloud Analytics Powered by FinOps

Cloudera

OCTOBER 30, 2023

Now development teams and engineers can spin up IT resources by themselves, just with a click of a button. Resource tagging CDP Public Cloud allows administrators to easily add tags to the Data Service and resources the platform deploys on the company’s cloud tenant.

Cloud

Cloud Finance Cloud Computing Government

Improving Meta’s global maps

Engineering at Meta

FEBRUARY 7, 2023

Recently, we partnered with the Daylight team at Meta to create Earth Tables, an extension of Daylight that simplifies OSM tags. We parsed OSM’s complicated building and building:part tags to refashion our building features from the ground up. The post Improving Meta’s global maps appeared first on Engineering at Meta.

Entertainment

Entertainment Transportation Data Schemas AWS

HTML Best Practices

Knowledge Hut

JULY 14, 2023

By sticking to these best practices, developers can produce readily maintainable code that enhances the overall user experience and improves search engine visibility and website performance. Consistent indentation, correct use of tags, and meaningful naming conventions contribute to the readability and clarity of the code.

Media

Media Coding Accessible Accessibility

Top 5 Reasons to Learn React JS in 2024

Knowledge Hut

DECEMBER 22, 2023

Search Engines hate heavy JavaScript pages because of reading troubles when the page is loaded in the browser. Also, if the basic HTML page contains nothing but some meta tags and script tags, the search engine will assume that our page is blank and the result is you get a poor ranking.

Coding

Coding Building Project Engineering

Building Netflix’s Distributed Tracing Infrastructure

Netflix Tech

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which We needed to increase engineering productivity via distributed request tracing. That is the first question our engineering teams asked us when integrating the tracer library.

Building

Building Transportation Metadata Java

Yelp Content As Embeddings

Yelp Engineering

APRIL 19, 2023

We need to tag, organize and rank online content to attain this goal. For this purpose, Yelp engineers have started using general embeddings on different data. Yelp aims to offer easily accessible high-quality content. It improves usability and efficiency for all kinds of model development.

Deep Learning

Deep Learning Accessible Accessibility Engineering

Announcing New Innovations for Snowflake Horizon

Snowflake

NOVEMBER 2, 2023

Integrations with other Apache Iceberg-compatible catalogs and engines , and with partners, further extend Snowflake Horizon across customers’ ecosystems. In addition, governors can take bulk action to propagate tags and policies to protect all downstream columns that have personally identifiable information.

Metadata

Metadata Government AWS Medical

ServiceNow Accelerates Data-Driven Innovation by Rearchitecting Its Enterprise Data Platform on Snowflake

Snowflake

DECEMBER 20, 2023

Performing one system upgrade required extensive engineering and testing, leading to excessive costs. Operational ease: Snowflake’s fully managed infrastructure and near-zero maintenance freed up engineers to focus on higher-impact projects. Building models and overseeing security was challenging for the team.

IT

IT Data Government Cloud

EC2 & Session Manager (Toronto Project)

Team Data Science

JUNE 6, 2020

Welcome back to this Toronto Specific data engineering project. We left off last time concluding finance has the largest demand for data engineers who have skills with AWS, and sketched out what our data ingestion pipeline will look like. I began building out the data ingestion pipeline by launching an EC2 instance.

Project

Project Management Data Ingestion AWS

From Big Data to Better Data: Ensuring Data Quality with Verity

Lyft Engineering

OCTOBER 3, 2023

This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. The lack of centralization in data quality also made the data discovery process inefficient, making it hard for data scientists and data engineers to identify trustworthy data. Yet this has come at the cost of quality.

Big Data

Big Data Metadata Data Warehouse Data

Our Learnings from the Early Days of Generative AI

LinkedIn Engineering

APRIL 6, 2023

It’s been an exciting few months at LinkedIn, as our engineering and product teams have been working hard to build some new and advanced AI-powered experiences for our members and customers. These are two really crucial tools that accelerated our generative AI exploration, and that we look forward to sharing more about in the future.

Accessible

Accessible Accessibility Technology Engineering

Essential Docker Commands You Must Learn

Knowledge Hut

JANUARY 12, 2024

docker pull Pull an image or a repository from a registry Syntax docker pull [OPTIONS] NAME[: TAG|@DIGEST] To download an image or set of images (i.e. Syntax: docker push [OPTIONS] NAME[: TAG] Use docker image push to share your images to the Docker Hub registry or to a self-hosted one. Example: $ docker kill my_container 9.

Architecture

Architecture Java Coding Systems

How Pie Insurance Created a Self-Serve Incident Triaging & Resolution Workflow with Monte Carlo and Slack

Monte Carlo

NOVEMBER 30, 2023

To ensure reliability beyond the data engineering team and across the organization, Ed and his team developed a self-serve incident resolution workflow that empowered business users to directly address issues, without involving data engineering at every step of the process. That’s where data observability came in.

Insurance

Insurance Data Engineering Data Engineer SQL

Migrating Policy Delivery Engines with (almost) Nobody Knowing

Pinterest Engineering

JANUARY 26, 2024

Jeremy Krach | Staff Security Engineer, Platform Security Background Several years ago, Pinterest had a short incident due to oversights in the policy delivery engine. To learn more about engineering at Pinterest, check out the rest of our Engineering Blog and visit our Pinterest Labs site.

Engineering

Engineering Architecture Systems Metadata

Google Shutting down Firebase Dynamic Links

The Pragmatic Engineer

AUGUST 3, 2023

👋 Hi, this is Gergely with a free issue of the Pragmatic Engineer Newsletter. The full edition additionally covers: Details behind Wise’s open engineering salary bands. I got more details from Timothee Ledure, the former engineering lead at Wise, at their Singapore office.

Metadata

Metadata Engineering Building Technology

Improving Efficiency Of Goku Time Series Database at Pinterest (Part?—?1)

Pinterest Engineering

NOVEMBER 22, 2023

Initial Architecture For Goku Short Term Ingestion Figure 1: Old push based ingestion pipeline into GokuS At Pinterest, we have a sidecar metrics agent running on every host that logs the application system metrics time series data points (metric name, tag value pairs, timestamp and value) into dedicated kafka topics.

Database

Database Bytes Kafka Architecture

Building and maintaining the skills taxonomy that powers LinkedIn's Skills Graph

LinkedIn Engineering

MARCH 21, 2023

One of the most exciting parts of our work is that we get to play a part in helping progress a skills-first labor market through our team’s ongoing engineering work in building our Skills Graph. Engineering vs PyTorch Figure 6: Sample Seed Skills Graph KGBert helps build a more accurate and complex taxonomy in less time.

Building

Building Recruitment Machine Learning Deep Learning

12 Responsive Web Design Principles You Need to Know in 2024

Knowledge Hut

MARCH 27, 2024

SEO Benefits: Search engines prioritize mobile-friendly websites, boosting search rankings for responsive designs and improving visibility. Viewport Meta Tag Set viewport meta tags to control the viewport size on mobile devices. Addresses potential issues that may arise due to variations in browser rendering engines.

Designing

Designing Media Accessible Accessibility

Data News — Week 24.02

Building DoorDash’s Product Knowledge Graph with Large Language Models

Webinars

Trending Sources

Web Performance Regression Detection (Part 2 of 3)

Webinars

4 Reasons Why Data Engineers Hate Google Tag Manager

Why did Google close its coding competitions after 20 years?

Create a New React Project From Scratch [Step-by-Step Guide]

Introducing Vector Search on Rockset: How to run semantic search with OpenAI and Rockset

How DoorDash Migrated from StatsD to Prometheus

Now Featuring: Orchestration Lineage

Bun: lessons from disrupting a tech ecosystem

Type-safe data processing pipelines

How to get started with dbt

Inside Pollen's Software Engineering Salaries

The Complete Front-End Developer Roadmap 2024

Cloudera Uses CDP to Reduce IT Cloud Spend by $12 Million

Transforming Delimited String Columns into Rows with Snowflake

Improving the code quality of your dbt models with unit tests and TDD

Data Catalog vs. Data Dictionary: 3 Essential Differences

How Financial Platform Tide Automated GDPR Compliance With Atlan and Snowflake

Lineage + Hamilton in 10 minutes

Costwiz: Saving cost for LinkedIn enterprise on Azure

Migrating Netflix to GraphQL Safely

DotSlash: Simplified executable deployment

Docker Vs Virtual Machines(VMs)

Upgrade your Modern Data Stack

Understanding GraphQL Directives: Practical Use-Cases at Zalando

Cloud Analytics Powered by FinOps

Improving Meta’s global maps

HTML Best Practices

Top 5 Reasons to Learn React JS in 2024

Building Netflix’s Distributed Tracing Infrastructure

Yelp Content As Embeddings

Announcing New Innovations for Snowflake Horizon

ServiceNow Accelerates Data-Driven Innovation by Rearchitecting Its Enterprise Data Platform on Snowflake

EC2 & Session Manager (Toronto Project)

From Big Data to Better Data: Ensuring Data Quality with Verity

Our Learnings from the Early Days of Generative AI

Essential Docker Commands You Must Learn

How Pie Insurance Created a Self-Serve Incident Triaging & Resolution Workflow with Monte Carlo and Slack

Migrating Policy Delivery Engines with (almost) Nobody Knowing

Google Shutting down Firebase Dynamic Links

Improving Efficiency Of Goku Time Series Database at Pinterest (Part?—?1)

Building and maintaining the skills taxonomy that powers LinkedIn's Skills Graph

12 Responsive Web Design Principles You Need to Know in 2024

Stay Connected