Netflix Tech

MARCH 7, 2024

Berg , Romain Cledat , Kayla Seeley , Shashank Srikanth , Chaoying Wang , Darin Yu Netflix uses data science and machine learning across all facets of the company, powering a wide range of business applications from our internal infrastructure and content demand modeling to media understanding.

Systems 90

Systems Media Machine Learning Data Warehouse 90

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage.

Database 100

Database MySQL Data Lake MongoDB 100

Snowflake

SEPTEMBER 7, 2023

Snowpark is the set of libraries and runtimes that enables data engineers, data scientists and developers to build data engineering pipelines, ML workflows, and data applications in Python, Java, and Scala. Airbyte enables customers to continuously load data from external APIs and databases into Snowflake.

Amazon Web Services 119

Amazon Web Services AWS Government Python 119

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Data integration from source systems to their downstream destinations is the foundational step for any data product. With the increasing expecation for information to be instantly accessible, it drives the need for reliable change data capture. Data teams are increasingly under pressure to deliver.

Food 100

Food MongoDB Scala MySQL 100

Engineering at Meta

MAY 22, 2024

In recent years, Meta’s data management systems have evolved into a composable architecture that creates interoperability, promotes reusability, and improves engineering efficiency. Data is at the core of every product and service at Meta. We needed to change our thinking to be able to move faster.

Data Management 72

Data Management Management Data SQL 72

Monte Carlo

MARCH 28, 2024

While the DevOps methodology has been taking over the world of software development, data teams are just beginning to realize the benefits that a similar approach can bring to their world. It is a group of concepts that can be put into practice where there is data. Enter the nascent discipline of DataOps.

Data Pipeline 52

Data Pipeline Coding Machine Learning Data Warehouse 52

Knowledge Hut

APRIL 23, 2024

We live in a data-driven culture where familiarity with databases is crucial. Database management is crucial for businesses of all sizes to guarantee that their data is complete, safe, and easily available when needed. In the current digital era, data has become a crucial asset for businesses in all industries.

Database 52

Database Management Relational Database SQL 52

Data Engineering Podcast

APRIL 24, 2022

Summary There are very few tools which are equally useful for data engineers, data scientists, and machine learning engineers. WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model.

Machine Learning 100

Machine Learning Systems Data Lake Metadata 100

Data Engineering Podcast

JUNE 19, 2022

Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Podcast

NOVEMBER 6, 2022

Summary A lot of the work that goes into data engineering is trying to make sense of the "data exhaust" from other applications and services. Atlan is the metadata hub for your data ecosystem. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Ensuring Data Quality in New Data Sources (#1) Introduction to Data Evaluation in Data Observability Ensuring their quality and integrity before incorporating new data sources into production is paramount.

Data Cleanse 49

Data Cleanse Data Ingestion Data Datasets 49

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. RudderStack’s smart customer data pipeline is warehouse-first. How does it work?

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. This complicates the work involved in making end-to-end workflows visible and integrated. Data teams are increasingly under pressure to deliver.

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Weekly

APRIL 14, 2024

Meta: OpenEQA - From word models to world models Will AI agents soon become a common fixture in our homes and an integral part of our daily lives? link] Github: 4 ways GitHub engineers use GitHub Copilot The impact of LLM on software development is undeniable. Github shares some insights on how Github engineers use Github Copilot.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Data Engineering Podcast

NOVEMBER 13, 2022

Summary Building data products is an undertaking that has historically required substantial investments of time and talent. With the rise in cloud platforms and self-serve data technologies the barrier of entry is dropping. Atlan is the metadata hub for your data ecosystem.

Building 130

Building Metadata MongoDB MySQL 130

Data Engineering Podcast

JANUARY 1, 2022

Summary This has been an active year for the data ecosystem, with a number of new product categories and substantial growth in existing areas. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Missing data?

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

Christophe Blefari

SEPTEMBER 1, 2023

I've taken an unplanned 3-week break since the last Data News, let's be honest, it was necessary! I spent a few hours working on the fancy data stack project and articles are in the works, but it was idealistic to produce quality code and content while enjoying the summer. dbt tests: How to write fewer and better data tests?

Food 130

Food Data SQL Python 130

Data Engineering Podcast

OCTOBER 30, 2022

In its current incarnation it has become a ubiquitous need for analytics and opportunities to answer questions with data. In this episode Amir Orad discusses the Sisense platform and how it facilitates the embedding of analytics and data insights in every aspect of organizational and end-user experiences.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Picnic Engineering

OCTOBER 17, 2023

Building the data foundation for world’s best online supermarket The Importance of MDM In the fast-paced world of technology, it’s easy to be captivated by the allure of artificial intelligence. The truth is, as companies want to unlock the power of data this starts with strong MDM. Picnic’s MDM Journey Picnic was founded in 2015.

Data Management 52

Data Management Management Food Data 52

Monte Carlo

APRIL 20, 2024

That’s where you really want to focus your data quality monitoring.” And it’s true – you need to go “deep” on certain data sets; but if you’re not monitoring broadly as well, you’re missing the boat—and a whole lot of bad data. What is data quality monitoring?

IT 52

IT Metadata Data Warehouse Machine Learning 52

Knowledge Hut

MARCH 28, 2024

Data science has become one of the most trending fields today. Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 This demonstrates the increasing need for Microsoft Certified Data Engineers. That’s where data engineers are on the go.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

NOVEMBER 20, 2021

Summary The technology for scaling storage and processing of data has gone through massive evolution over the past decade, leaving us with the ability to work with massive datasets at the cost of massive complexity. Missing data? Start trusting your data with Monte Carlo today! Struggling with broken pipelines? Stale dashboards?

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Snowflake

MAY 9, 2024

From its start with efficient batch processing with data warehouses for descriptive analytics, and the inclusion of streaming data in real time to build recommendations, we find ourselves at the forefront of a new stage of evolution: generative AI (gen AI). The numbers back up Stuermer’s belief.

Unstructured Data 94

Unstructured Data Manufacturing Retail Data Warehouse 94

Monte Carlo

DECEMBER 20, 2023

“The data and AI space moves fast. Wondering what’s next for the future of data engineering and GenAI? Each year, we chat with one of the data industry’s pioneering leaders about their predictions for the modern data stack – and share a few of our own. Data teams will look like software teams 3.

Amazon Web Services 52

Amazon Web Services Metadata AWS Data 52

Monte Carlo

DECEMBER 20, 2023

“The data and AI space moves fast. Wondering what’s next for the future of data engineering and GenAI? Each year, we chat with one of the data industry’s pioneering leaders about their predictions for the modern data stack – and share a few of our own. Data teams will look like software teams 3.

Amazon Web Services 52

Amazon Web Services Metadata AWS Data 52

Data Engineering Podcast

APRIL 10, 2022

Summary Any time that you are storing data about people there are a number of privacy and security considerations that come with it. Privacy engineering is a growing field in data management that focuses on how to protect attributes of personal data so that the containing datasets can be shared safely.

Engineering 100

Engineering Data Lake Data Engineering Data Engineer 100

Towards Data Science

DECEMBER 1, 2023

Pet Project for Data/Analytics Engineers: Explore Modern Data Stack Tools — dbt Core, Snowflake, Fivetran, GitHub Actions. This hands-on experience will allow you to develop an end-to-end data lifecycle, from extracting data from your Google Calendar to presenting it in a Snowflake analytics dashboard. See Github repo.

Data Engineering 73

Data Engineering Data Engineer Project Engineering 73

dbt Developer Hub

OCTOBER 30, 2023

Hi all, I’m Kshitij, a senior software engineer on the Core team at dbt Labs. that leverages native zero-copy clone functionality on supported warehouses to copy entire schemas for free, almost instantly. On the surface, cloning and deferral seem similar – they’re both ways to save costs in the data warehouse.

BI 52

BI Datasets Metadata Data Warehouse 52

Christophe Blefari

JANUARY 20, 2023

Summer in coming ( credits ) Hey, new Friday, new Data News edition. Thank you for every recommendation you do about the blog or the Data News. The current state of data This week Benjamin Rogojan livestreamed an online conference featuring awesome data voices: state of data infra.

Google Cloud 130

Google Cloud Data Hadoop Machine Learning 130

Data Engineering Podcast

OCTOBER 16, 2022

Pathway is a streaming database engine that embeds artificial intelligence into the storage, with functionality designed to support the spatiotemporal data that is crucial for shipping and logistics. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Database 100

Database Metadata MongoDB MySQL 100

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. Missing data? Stale dashboards?

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

DECEMBER 11, 2021

In order to make it easier for developers to build customer profiles in a way that respects their privacy Serge Huber helped to create the Apache Unomi framework as an open source customer data platform. Missing data? Start trusting your data with Monte Carlo today! Struggling with broken pipelines? Stale dashboards?

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

Data Engineering Podcast

OCTOBER 2, 2022

This requires a significant amount of data to track shipments and operational characteristics of materials and goods. Roambee is a platform that collects, integrates, and analyzes all of that information to provide companies with the critical insights that businesses need to stay running, especially in a time of such constant change.

Metadata 100

Metadata Electronics MongoDB MySQL 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Regardless of how data is being used, it is critical that the information is trusted. The practice of data reliability engineering has gained momentum recently to address that question. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Building 100

Building Metadata MongoDB MySQL 100

Towards Data Science

FEBRUARY 24, 2023

The Chaos Data-Engineering Manifesto Another lesson we can learn from software engineers: break stuff to make it more reliable. Our main systems would scale—our articles were always served—but integration points across backend services would eventually buckle and burst under the sustained pressure of insane traffic levels.

Data Engineering 97

Data Engineering Data Engineer Engineering Software Engineer 97

Monte Carlo

NOVEMBER 27, 2023

When we introduced the concept of data observability four years ago, it resonated with organizations that had unlocked new value…and new problems thanks to the modern data stack. The answer today is the same as it was then: improve data product reliability by getting full context and visibility into your data systems.

Unstructured Data 59

Unstructured Data Data Pipeline Data Banking 59