Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

DataKitchen

DECEMBER 9, 2022

DataOps involves collaboration between data engineers, data scientists, and IT operations teams to create a more efficient and effective data pipeline, from the collection of raw data to the delivery of insights and results. Query> An AI, Chat GPT wrote this blog post, why should I read it? .

Machine Learning 52

Machine Learning Data Preparation Government Data Analytics 52

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. This is something I struggle with, I really like writing, I really like this newsletter, I really like the blog, but it takes me one day per week to be done. Monte Carlo also wrote this week about chaos engineering , with a manifesto.

Kafka 130

Kafka Data Lake Data Storage Data 130

Data Engineering Weekly

MAY 12, 2024

link] Gradient Flow: Learning from the Past - Comparing the Hype Cycles of Big Data and GenAI The blogs compare the hype cycle of Big Data with Gen AI. The blog narrates the initial hype of Big Data, followed by talent shortages and lead time to build production applications. Ready to give DoubleCloud a try?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Pinterest Engineering

MAY 13, 2024

The subsequent blog post will delve into how we looked into our specific needs, evaluated multiple candidates and decided on the adoption of a new database technology. Additionally, it was increasingly difficult to find HBase domain experts and the barriers to entry are very high for new engineers. took almost two years).

NoSQL 69

NoSQL MySQL Database Systems 69

Snowflake

DECEMBER 20, 2023

However, getting the required compute infrastructure into place, particularly GPUs for large language models (LLMs), is a real challenge. And once GPUs are obtained, operating the infrastructure requires specialized expertise and increases overhead costs that can constrain delivery of innovation.

Scala 119

Scala Government Java AWS 119

databricks

OCTOBER 3, 2023

This blog was written in collaboration with Sukh Sekhon, Software Engineer, Cloud Infrastructure and Helen Li, Sr. Director of Engineering at Exai Bio.

Software Engineer 89

Software Engineer Software Engineering Engineering Cloud 89

Data Engineering Podcast

FEBRUARY 26, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Are you tired of dealing with the headache that is the 'Modern Data Stack'? It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. We feel your pain. We feel your pain.

Building 147

Building Metadata Machine Learning Data Integration 147

Data Engineering Weekly

MARCH 31, 2024

The blog further emphasizes its increased investment in Data Mesh and clean data. The blog is an excellent overview of all the improvements made to PySpark in 2023. link] Uber: Scaling AI/ML Infrastructure at Uber The advancement in AI/ML brings significant challenges for infrastructure to scale and support.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Christophe Blefari

DECEMBER 30, 2022

💡 You can also read the 2021's must-read that I've done one year and half ago or how to learn data engineering that contains key articles to understand the field. ANALYTICS ENGINEERING We have to be honest in 2022 Analytics Engineering shaped up the data field and concentrated a lot of data discussions.

Data Warehouse 130

Data Warehouse BI Data Machine Learning 130

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers. Additionally, they needed the ability to experiment with streaming pipelines in batch mode.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Knowledge Hut

APRIL 23, 2024

Software engineering is a rapidly growing field with vast career opportunities. With emerging technologies like AI, machine learning, and blockchain, the demand for software engineers has skyrocketed. With emerging technologies like AI, machine learning, and blockchain, the demand for software engineers has skyrocketed.

Software Engineer 98

Software Engineer Software Engineering Engineering Programming Language 98

Cloudera

MAY 9, 2024

Hybrid Horses for Courses: The Right Cloud for AI from Pilot to Production at Scale Later, on May 14 at 12:40 pm BST , hear from Mark Samson, one of Cloudera’s solutions engineering directors, on whether a data center or cloud deployment is best for your organization’s data platform and architecture.

Banking 83

Banking Data Storage Data Analytics Cloud 83

Pinterest Engineering

JANUARY 9, 2024

In this blog we will deep dive into some of our recent advancements in machine learning modeling to connect pinners with the most relevant ads. This blog post will focus on our recent improvements on improving the models that predict offsite conversions given a user request on Pinterest.

Machine Learning 76

Machine Learning Architecture Engineering Utilities 76

Pinterest Engineering

OCTOBER 31, 2023

PinCompute aligns with the Platform as a Service (PaaS) cloud computing model, in that it abstracts away the undifferentiated heavy lifting of managing infrastructure and Kubernetes and enables users to focus on the unique aspects of their applications. Components that are able to be shared by multiple tenants (e.g.

Architecture 96

Architecture Pipeline-centric Accessible Accessibility 96

Data Engineering Weekly

APRIL 21, 2024

[link] Stripe: Shepherd - How Stripe adapted Chronon to scale ML feature development Airbnb recently open-sourced Chronon , a declarative feature engineering framework. The blog narrates how Chronon fits into Stripe’s online and offline requirements. Counting is the hardest problem in data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Data Science Blog: Data Engineering

SEPTEMBER 19, 2023

In the contemporary age of Big Data, Data Warehouse Systems and Data Science Analytics Infrastructures have become an essential component for organizations to store, analyze, and make data-driven decisions. Infrastructure as Code (IaC) can be a game-changer in this scenario. So why using IaC for Cloud Data Infrastructures?

Data Warehouse 52

Data Warehouse Coding Systems Cloud 52

Workfall

FEBRUARY 14, 2023

Reading Time: 7 minutes Platform Engineering has received a lot of attention, but there is some misunderstanding about what it is and, perhaps more importantly, how it differs from more well-known disciplines like SRE and DevOps. Platform Engineering is the rebranded DevOps or it is the next stage of DevOps evolution?

Engineering 77

Engineering Software Engineer Software Engineering AWS 77

DoorDash Engineering

MARCH 27, 2024

In such a multi-tenant architecture, the isolation is implemented at the infrastructure layer. We solve this using a fully multi-tenant architecture where data and traffic is isolated at the infrastructural layer with minimal interference with the application logic. Figure 2 below shows an overview of Asgard.

Kafka 103

Kafka Architecture Algorithm Systems 103

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. The slowest step in that process was provisioning Kafka Topics, which involved on-call engineer approval and was prone to failure, further increasing the on-call load. Built atop Prometheus.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Engineering at Meta

OCTOBER 24, 2023

In our last blog post on automatic product deprecation , we talked about the complexities of product deprecations, and a solution Meta has built called the Systematic Code and Asset Removal Framework (SCARF). Dead code removal at scale SCARF uses CodemodService to create code change requests for engineers to review.

Coding 129

Coding Programming Language Python MySQL 129

Data Engineering Weekly

MARCH 3, 2024

There will be food, networking, and real-world talks around data engineering. The NVIDIA blog on Sovereign AI emphasizes the importance of countries developing artificial intelligence capabilities using local infrastructure, data, and workforce. Part 1: Why did we need to build our own SIEM?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Uber Engineering

DECEMBER 19, 2019

Every day around the world, millions of trips take place across the Uber network, giving users more reliable transportation through ridesharing, bikes, and scooters, drivers and truckers additional opportunities to earn, employees and employers more convenient business travel, and hungry … The post Uber Infrastructure in 2019: Improving Reliability, (..)

Transportation 136

Transportation Engineering Architecture 136

Pinterest Engineering

MAY 2, 2023

It’s worth noting that the choice of the KV store technology is extremely important, because it can have a big impact on the overall efficiency (and ultimately, cost) of the entire infrastructure, as well as the complexity of the real-time indexing job. We need each dataset to store the last N events for a user. Handles duplicate values.

Lambda Architecture 65

Lambda Architecture Datasets Software Engineer Software Engineering 65

Cloudera

FEBRUARY 1, 2024

In this blog, we’ll cover the highlights of our recently published Day 2 Operations Guide and why it matters to enterprises. The rest of the blog covers precisely how the platform handles monitoring and troubleshooting of the platform when breakages happen.

Cloud 81

Cloud Architecture Utilities Designing 81

Pinterest Engineering

SEPTEMBER 12, 2023

Raymond Lee | Software Engineer II; Qingxian Lai | Sr. Software Engineer; Karthik Anantha Padmanabhan | Manager II, Engineering; Se Won Jang | Manager II, Engineering Photo by Claudio Schwarz on Unsplash Our mission at Pinterest is to bring everyone the inspiration to create the life they love.

Data Process 88

Data Process Process Datasets Scala 88

Knowledge Hut

NOVEMBER 20, 2023

DevOps engineers are a group of influential individuals who encapsulates depth of knowledge and years of hands-on experience around a wide variety of open-source technologies and tools. DevOps engineer always shows interest and comfort working with frequent, incremental code testing and deployment.

Engineering 98

Engineering Google Cloud Portfolio Certification 98

Lyft Engineering

SEPTEMBER 6, 2023

Our team, the Developer Infrastructure team, aims to build the best tools to enable microservice owners (our “customers”) to reliably and quickly test changes in a local and/or end-to-end environment. Routing overrides metadata with ProxyApp Staging overrides was well-received by our product engineers.

Building 80

Building Metadata Electronics Engineering 80

Pinterest Engineering

JANUARY 4, 2024

Rajath Prasad, Senior Engineering Manager Pinterest serves more than 480M monthly users and has grown to be a global destination for visual inspiration. Motivation As explained in this blog post , in 2019, Pinterest had four different key-value services with different storage engines including RocksDB, HBase, and HDFS.

Database 77

Database Building Datasets Relational Database 77

Data Engineering Weekly

SEPTEMBER 24, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. The blog specifically focuses on the following areas of LLM implementations.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Uber Engineering

JULY 22, 2021

We built a team with varied … The post Containerizing Apache Hadoop Infrastructure at Uber appeared first on Uber Engineering Blog.

Hadoop 145

Hadoop Machine Learning Engineering Architecture 145

Monte Carlo

JANUARY 23, 2024

In the world of data engineering, Maxime Beauchemin is someone who needs no introduction. One of the first data engineers at Facebook and Airbnb, he wrote and open sourced the wildly popular orchestrator, Apache Airflow , followed shortly thereafter by Apache Superset , a data exploration tool that’s taking the data viz landscape by storm.

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Engineering at Meta

SEPTEMBER 5, 2023

On the latest episode of the Meta Tech Podcast , Meta engineer Pascal Hartig ( @passy ) sits down with Dustin Shahidehpour and Katherine Zak, two software engineers at Meta, about their careers and what it’s really like to ship code at Meta. And what does Meta’s CI infrastructure look like?

Coding 80

Coding IT Software Engineer Software Engineering 80

Pinterest Engineering

SEPTEMBER 5, 2023

Pong Eksombatchai | Principal Engineer; Karthik Anantha Padmanabhan | Manager II, Engineering Image from [link] Pinterest’s mission is to bring everyone the inspiration to create a life they love. Both ML and ML platform engineers felt the full brunt of this issue.

Engineering 92

Engineering Deep Learning Architecture Project 92

LinkedIn Engineering

APRIL 25, 2023

since its inception in 2013, LinkedIn has built thousands of microservices to enable the exchange of data with our engineers and our external partners. Though this microservice architecture has worked out really well for our API engineers, when our clients need to fetch data they find themselves talking to several of these microservices.

Architecture 109

Architecture Metadata Java Transportation 109

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. Finally, relevant abstractions allow media algorithm developers to focus on the manipulation of video and audio signals rather than on infrastructural concerns.

Process 91

Process Pipeline-centric Media Metadata 91

Lyft Engineering

APRIL 24, 2024

See our previous blog post for more on the motivations behind TLC. Some recent examples can be found in our tech blog: reinforcement learning and structural causal modeling. Snapshot Photo from a Technical Training Session CS4DS Computer science and software engineering are fundamental data science skills. production-level code.

Data Science 66

Data Science Building Computer Science Software Engineer 66