Towards Data Science

JUNE 16, 2023

How to Identify Your Business-Critical Data Practical steps to identifying business-critical data models and dashboards and drive confidence in your data Source: synq.io This article has been co-written with Lindsay Murphy Not all data is created equal. There’s a good reason for this. Source: synq.io

BI 82

BI Data ETL Tools Machine Learning 82

Engineering at Meta

FEBRUARY 7, 2023

This global, multi-scale map serves as a foundational layer — showing a variety of geographic features onto which we can blend other data, such as local points of interest or locally important features. Recently, we partnered with the Daylight team at Meta to create Earth Tables, an extension of Daylight that simplifies OSM tags.

Entertainment 133

Entertainment Transportation Data Schemas AWS 133

The Pragmatic Engineer

JANUARY 12, 2023

We cover one and a half out of eight topics in today’s subscriber-only issue, Inside Pollen's Transparent Compensation Data. In this week’s The Scoop, I analyzed this information and dissected it, going well beyond the raw data. To get this newsletter every week, subscribe here.

Software Engineer 183

Software Engineer Software Engineering Engineering Raw Data 183

Cloudera

OCTOBER 30, 2023

The legacy IT infrastructure to run the business operations — mainly data centers — has a deadline to shift to cloud-based services. The public cloud is increasingly becoming the preferred platform to host data analytics – related projects, such as business intelligence, machine learning (ML), and AI applications.

Cloud 74

Cloud Finance Cloud Computing Government 74

Christophe Blefari

SEPTEMBER 28, 2023

Make your data stack take-off ( credits ) Hello, another edition of Data News. This week, we're going to take a step back and look at the current state of data platforms. What are the current trends and why are people fighting around the concept of the modern data stack. Is the modern data stack dying?

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision. Enter the ELT.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Snowflake

APRIL 29, 2024

Identification of potential threats through data analysis and risk assessment processes to shift toward more proactive risk management. This strategy must incorporate business impact analyses as well as backup and recovery plans in the event of a security incident or loss of access to data. or better) for data in transit.

Transportation 62

Transportation Data Governance Government Consulting 62

Netflix Tech

JANUARY 25, 2023

Scalable Annotation Service — Marken by Varun Sekhri , Meenakshi Jindal Introduction At Netflix, we have hundreds of micro services each with its own data models or entities. Our team, Asset Management Platform, decided to create a generic service called Marken which allows any microservice at Netflix to annotate their entity.

Algorithm 113

Algorithm Media Metadata Data Ingestion 113

Yelp Engineering

APRIL 19, 2023

We need to tag, organize and rank online content to attain this goal. For this purpose, Yelp engineers have started using general embeddings on different data. This blog post discusses how the Content and Contributor Intelligence team generates low-dimensional representations of review text, business information and.

Deep Learning 59

Deep Learning Accessible Accessibility Engineering 59

Lyft Engineering

OCTOBER 3, 2023

High-quality data is necessary for the success of every data-driven company. It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Netflix Tech

OCTOBER 19, 2020

by Maulik Pandey Our Team?—? If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. That is the first question our engineering teams asked us when integrating the tracer library.

Building 93

Building Transportation Metadata Java 93

Towards Data Science

OCTOBER 25, 2023

Not too long ago, almost all data architectures and data team structures followed a centralized approach. As a data or analytics engineer, you knew where to find all the transformation logic and models because they were all in the same codebase. There was only one data team, two at most.

Data Pipeline 85

Data Pipeline SQL Data Architecture Data 85

LinkedIn Engineering

APRIL 6, 2023

It’s been an exciting few months at LinkedIn, as our engineering and product teams have been working hard to build some new and advanced AI-powered experiences for our members and customers. Overall, our team's work is contributing to an engineering culture at LinkedIn that values exploration, collaboration, and innovation.

Accessible 94

Accessible Accessibility Technology Engineering 94

François Nguyen

JANUARY 23, 2021

or why you should have a look at Data Observability ! This article is the second part on how Airbnb is managing data quality : “Part 2 — A New Gold Standard” The first part can be found here and it was just good principles about roles and responsabilities. Welcome to this strange way of managing data quality !

Finance 130

Finance Certification Data Pipeline Data 130

Data Engineering Podcast

DECEMBER 21, 2020

Summary Data governance is a term that encompasses a wide range of responsibilities, both technical and process oriented. One of the more complex aspects is that of access control to the data assets that an organization is responsible for managing. If you hand a book to a new data engineer, what wisdom would you add to it?

Data Governance 130

Data Governance Government Data Lake Banking 130

Rockset

MARCH 17, 2022

Real-time updates give you the power to insert, delete and update data in place. 1) Late Arriving Data in Time-Based Window Rollups ⌛️ Let's say you have a rollup that's counting events for each hour. A mutable database allows you to recompute the results with the late-arriving data. 2) Data Enrichment ?

Database 52

Database IT PostgreSQL MongoDB 52

Cloudera

JUNE 26, 2023

Here are some tips and tricks of the trade to prevent well-intended yet inappropriate data engineering and data science activities from cluttering or crashing the cluster. For data engineering and data science teams, CDSW is highly effective as a comprehensive platform that trains, develops, and deploys machine learning models.

ETL Tools 75

ETL Tools Programming Language Datasets Data Pipeline 75

dbt Developer Hub

JANUARY 23, 2023

Testing the quality of data in your warehouse is an important aspect in any mature data pipeline. One of the biggest blockers for developing a successful data quality pipeline is aggregating test failures and successes in an informational and actionable way. However, ensuring actionability can be challenging. on BigQuery.

Metadata 52

Metadata High Quality Data SQL Data Integration 52

Ripple Engineering

MARCH 30, 2023

As we’ve increased our investments in machine learning solutions, the applied science team has created many parts of the ML platform from scratch while figuring out how to make it work well with our existing software stack. Each cluster has a service account that has access to the requisite data.

Machine Learning 98

Machine Learning AWS Metadata Data Collection 98

Towards Data Science

JUNE 19, 2023

Photo by CHUTTERSNAP on Unsplash This article is about Meerschaum Compose , a tool for defining ETL pipelines in YAML and a plugin for the data engineering framework Meerschaum. As more data engineers built their pipelines using the platform, the dynamic nature of pipes meant that a solution was needed to provide project-level isolation.

PostgreSQL 83

PostgreSQL SQL Python Project 83

Netflix Tech

APRIL 27, 2023

A few years ago, we were paged by our SRE team due to our Metrics Alerting System falling behind — critical application health alerts reached engineers 45 minutes late! This user represented a platform team at Netflix, and their goal was to build alerting automation for their users.

Database 115

Database Architecture Consulting Systems 115

Cloudera

DECEMBER 21, 2020

In this blog we will take you through a persona-based data adventure, with short demos attached, to show you the A-Z data worker workflow expedited and made easier through self-service, seamless integration, and cloud-native technologies. In our data adventure we assume the following: . Company data exists in the data lake.

Banking 97

Banking Data Lake Data Data Warehouse 97

Netflix Tech

NOVEMBER 11, 2022

These timecode tags enable efficient discovery, freeing our creators from hours of categorizing footage so they can focus on creative decisions instead. Context, not control, guides the work for data scientists and algorithm engineers at Netflix. a fundamental ML research that utilizes multiple sources of information or modality (e.g.

Media 95

Media Machine Learning Algorithm Deep Learning 95

Snowflake

AUGUST 21, 2023

With demonstrable success across a range of industries, organizations are increasingly pursuing cutting-edge data mesh architectures to enhance self-service data use. Data-as-a-product: By considering data resources through a product lens, teams can adopt practices centered around quality and ease of use.

Pipeline-centric 52

Pipeline-centric Architecture Government Data Architect 52

DareData

NOVEMBER 28, 2023

In today's data-driven world, machine learning has emerged as a transformative force, empowering organizations to extract valuable insights from vast amounts of data. As the scope of the models and the data continues to scale, the role of a Data Scientist has evolved accordingly in the last years. But that is not all!

Data Science 52

Data Science Medical Machine Learning Data 52

Snowflake

DECEMBER 19, 2023

The best marketing is truly data-driven, creating powerful product promotions and offers through an understanding of customer needs and preferences. than reading data insights from a beautiful dashboard. It’s a potentially cumbersome and time-consuming process that too often requires moving or sharing access to sensitive customer data.

Building 80

Building Pipeline-centric Database-centric Digital Media 80

phData: Data Engineering

OCTOBER 12, 2022

This documentation can give different users insight into where data came from, what the profile of the data is, what the SQL looked like, and the DAG to know where the data is being used. It allows you to tag which final models are being used for a particular data product or dashboard.

Cloud 52

Cloud Metadata SQL Data Warehouse 52

Grouparoo

MARCH 30, 2021

Teams are centralizing their data in their data warehouse by loading data in and transforming it as necessary. Increasingly, we are seeing teams turn to dbt to do this transforming. sql files that, when run in the right order, create useful rollup tables or materialized views of the data. That's it!

Data Warehouse 52

Data Warehouse SQL Project Database 52

Snowflake

JUNE 14, 2023

Komodo Health is a technology platform company creating the new standard for real-world data and analytics by pairing the industry’s most complete view of patient encounters with enterprise software and machine learning that connects the dots between individual patient journeys and large-scale health outcomes.

Healthcare 52

Healthcare SQL Algorithm Architecture 52

Data Engineering Podcast

SEPTEMBER 14, 2020

On top of this foundation, the Hazelcast team has also built a streaming platform for reliable high throughput data transmission. In this episode Dale Kim shares how Hazelcast is implemented, the use cases that it enables, and how it complements on-disk data management systems.

Process 100

Process Unstructured Data Metadata Data Engineering 100

The Pragmatic Engineer

AUGUST 3, 2023

” Hearing the reasoning behind sunsetting FDL, I cannot help but wonder if there might have been some platform-level changes, for example, on iOS, where the Firebase team deemed it too problematic to keep supporting the smooth transition to the post-app-install experience. Data from Carta confirms an uptick in startup shutdowns.

Metadata 160

Metadata Engineering Building Technology 160

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Snowflake

SEPTEMBER 13, 2023

Building your data product is only the beginning. Or maybe you just have some really unique data that you think might be of use to someone. Pricing has stumped product teams since the dawn of time (well, almost). As the data economy began to take off, “ What’s my data worth? What to charge for my data?

Food 96

Food Retail Hospitality Data 96

Data Engineering Podcast

APRIL 19, 2021

Summary Most of the time when you think about a data pipeline or ETL job what comes to mind is a purely mechanistic progression of functions that move data from point A to point B. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Data Pipeline 100

Data Pipeline Machine Learning Data Warehouse Datasets 100

AltexSoft

AUGUST 25, 2021

Specifics of data used in NLP. Both in daily life and in business, we deal with massive volumes of unstructured text data : emails, legal documents, product reviews, tweets, etc. Often used for directing customer requests to an appropriate team, language detection highlights the languages used in emails and chats.

Process 139

Process Deep Learning Datasets Machine Learning 139

Data Engineering Weekly

JUNE 4, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Do you need data/ metrics at all? Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Netflix Tech

NOVEMBER 13, 2023

Music information retrieval There are a few studio use cases where music activity metadata is important, including quality-control (QC) and at-scale multimedia content analysis and tagging. To evaluate and benchmark our dataset, we manually labeled 20 audio tracks from various TV shows which do not overlap with our training data.

Datasets 89

Datasets Metadata Algorithm Architecture 89