Uber Engineering

JULY 22, 2021

We built a team with varied … The post Containerizing Apache Hadoop Infrastructure at Uber appeared first on Uber Engineering Blog. As Uber’s business grew, we scaled our Apache Hadoop (referred to as ‘Hadoop’ in this article) deployment to 21000+ hosts in 5 years, to support the various analytical and machine learning use cases.

Hadoop 145

Hadoop Machine Learning Engineering Architecture 145

Engineering at Meta

APRIL 11, 2024

An engineer shares how his team is building the foundation for AI infrastructure at Meta, from physical data centers to training AI models.

Building 49

Building Engineering Data 49

Uber Engineering

JANUARY 16, 2023

Uber’s infrastructure engineers deep dive into how they leverage Infrastructure as code to manage hundreds of thousands of servers across multiple cloud and on-prem providers.

Cloud 80

Cloud Coding Engineering Management 80

ArcGIS

OCTOBER 31, 2023

Learn how to use GIS and the geographic approach to improve the way you manage infrastructure and assets.

Management 67

Management Utilities 67

Knowledge Hut

DECEMBER 28, 2023

If you are wondering what is DevOps infrastructure, give this article a read. Infrastructure Automation in DevOps DevOps infrastructure or Infrastructure Automation under DevOps refers to a concept that revolves around the idea of managing infrastructure with the help of code. Why Use DevOps in Infrastructure?

IT 52

IT Coding Technology Designing 52

Hevo

SEPTEMBER 24, 2023

billion on modern data infrastructure in 2023.[1] To keep up with it, the data infrastructure had to evolve at a similar pace. According to HG Data, 1.5 million companies are slated to spend $133.6 1] It’s no surprise that the volume of data in our hands has grown exponentially over the past few years. This meant […]

Data 52

Data IT 52

Knowledge Hut

SEPTEMBER 22, 2023

I have always been fascinated by infrastructure and construction projects. It led me to think about building a career as an Infrastructure Project Manager. Recent data reveals an impressive 11% surge in senior infrastructure project manager salaries over the last five years in the US.

Project 52

Project Management Portfolio Transportation 52

Engineering at Meta

APRIL 11, 2024

An engineer shares how his team is building the foundation for AI infrastructure at Meta, from physical data centers to training AI models.

Building 40

Building Engineering Data 40

Uber Engineering

MAY 3, 2023

In this blog post we explain how we bootstrapped arm64 infrastructure using a relatively new toolchain in town: zig cc.

82

82

Netflix Tech

OCTOBER 19, 2020

Now let’s look at how we designed the tracing infrastructure that powers Edgar. This insight led us to build Edgar: a distributed tracing infrastructure and user experience. Our distributed tracing infrastructure is grouped into three sections: tracer library instrumentation, stream processing, and storage.

Building 93

Building Transportation Metadata Java 93

Towards Data Science

JANUARY 20, 2024

Automate resource provisioning with modern tools Continue reading on Towards Data Science »

Data Engineering 82

Data Engineering Data Engineer Engineering Data Science 82

Uber Engineering

DECEMBER 19, 2019

Every day around the world, millions of trips take place across the Uber network, giving users more reliable transportation through ridesharing, bikes, and scooters, drivers and truckers additional opportunities to earn, employees and employers more convenient business travel, and hungry … The post Uber Infrastructure in 2019: Improving Reliability, (..)

Transportation 132

Transportation Engineering Architecture 132

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. Building a data lake infrastructure is a highly beneficial endeavor, not just for the immediate use of data, but also for its potential to foster the growth of analytics across the organization.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

KDnuggets

JUNE 21, 2022

Collect validated user behavior data that’s ready to model on without prepwork. Take models built on one dataset and deploy & run them on another.

Datasets 160

Datasets Data 160

Uber Engineering

MAY 2, 2024

Accelerating Tomorrow: How Uber Turbocharges AI/ML Frontiers.

78

78

Data Engineering Podcast

FEBRUARY 17, 2020

Summary One of the biggest challenges in building reliable platforms for processing event pipelines is managing the underlying infrastructure. If you are daunted by the needs of your data infrastructure then it’s worth listening to how Josh and his team are approaching the problem.

AWS 100

AWS Google Cloud Kafka Media 100

Tweag

FEBRUARY 22, 2023

What follows are my takeaways on the trends and the trajectory of infrastructure and configuration management. This post is merely a patchwork of my biased, fresh impressions — as a programming language nerd rabbit-holed into devops and infrastructure by Nix — on my way back home. isn’t sustainable. Those are not curiosities any more.

Management 52

Management Programming Language Data Schemas Programming 52

Confluent

APRIL 20, 2023

In part 1, learn how to calculate Kafka costs stemming from infrastructure and the impact networking has on your cloud bill. Quantifying the cost of running Kafka is challenging.

Kafka 57

Kafka Cloud 57

Knowledge Hut

JULY 3, 2023

Use Cases of Real-time Ingestion Real-time ingestion provides organizations with infrastructure for implementing various data capture, data processing and data analyzing tools. Data Storage: Real-Time data ingestion infrastructure requires storage capable of handling and storing high amounts of data with low latency.

Data Ingestion 52

Data Ingestion Pipeline-centric Google Cloud Media 52

Palantir

MAY 6, 2021

Containerized infrastructure has become an industry-wide trend as engineering teams lean on the likes of Docker or Kubernetes to manage, deploy, and scale their environments; here, Palantir is no exception. Address the areas of Cilium’s logging telemetry that we’ve found valuable for alerting & detection.

Bytes 52

Bytes Metadata Engineering Process 52

LinkedIn Engineering

JUNE 2, 2023

In this post, we'll describe how the EGRI Data Foundations team (Team Asimov) leverages LinkedIn's cutting-edge data infrastructure tools such as Unified Metrics Platform , Pinot , and Datahub to ensure we can deliver data and insights robustly, securely, and at scale to a myriad of partners.

Building 75

Building Banking Datasets Media 75

Data Engineering Podcast

FEBRUARY 25, 2018

In this episode he discusses the challenges inherent in capturing and analyzing event data, the tools that his team is using to make it possible, and how this type of knowledge can be used to improve your critical infrastructure. What is Honeycomb and how did you get started at the company?

Kafka 100

Kafka AWS Data Pipeline Data 100

Cloudera

DECEMBER 1, 2020

In this blog post, let us take a look at how you can plan your infrastructure planning that you may have to do when deploying an operational database cluster on a CDP Private Cloud Base deployment. In the sections below, we take a look at multiple scenarios that you may encounter when planning infrastructure.

Database 58

Database Cloud Finance Metadata 58

Data Engineering Podcast

NOVEMBER 14, 2017

To surface the insights that they need to grow their business they need a robust data infrastructure to reliably capture all of those interactions. To surface the insights that they need to grow their business they need a robust data infrastructure to reliably capture all of those interactions. that they are posting.

Google Cloud 100

Google Cloud Data Ingestion Media AWS 100

Confluent

SEPTEMBER 19, 2023

Learn how a grocery delivery company guarantees same day delivery with Confluent as the backbone of their data streaming infrastructure.

Cloud 61

Cloud Data 61

Data Engineering Podcast

JUNE 19, 2022

Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. In order to lower the barrier to entry Ryan Buick created the Canvas application with a spreadsheet oriented workflow that is understandable to a wide audience.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Uber Engineering

MARCH 13, 2018

To build and maintain infrastructure at scale, easy access to the current state of the system is paramount.

Management 75

Management Engineering Accessible Accessibility 75

RudderStack

MAY 11, 2021

RudderStack briefly tells CDI and its core & the infrastructure for seizing, processing, and routing events. A typical example of a Data-Intensive Application.

Building 40

Building Data Process IT 40

Acceldata

FEBRUARY 10, 2023

Learn how Netflix uses data engineering best practices to optimize spend intelligence for its data infrastructure.

Data Engineering 52

Data Engineering Data Engineer IT Engineering 52

Uber Engineering

JANUARY 26, 2023

We recently containerized Hadoop NameNodes and upgraded hardware, improving NameNode RPC queue time from ~200 to ~20ms – A 10x improvement! With this radical change, Uber’s Hadoop customers are happier and admins rest more at night.

Hadoop 87

Hadoop Data 87

Propel Data

FEBRUARY 21, 2023

Terraform and CDK, the pros and cons of each, and how you can use infrastructure as code with Propel.

Coding 40

Coding Data Analytics Data 40

Confluent

SEPTEMBER 14, 2022

How gaming enterprises like Sony and Big Fish Games use Apache Kafka®, Confluent, and ksqlDB’s data streaming technologies for the best in-game experience, ROI, and real-time capabilities.

Kafka 121

Kafka Technology Data 121

Confluent

MARCH 17, 2021

In order for a technology like Confluent Cloud to make it easy to set data in motion, many different software systems are required to interact with each other using API […].

Cloud 89

Cloud Technology Systems IT 89

Engineering at Meta

MARCH 12, 2024

This announcement is one step in our ambitious infrastructure roadmap. By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s.

Building 145

Building Portfolio Utilities Data Storage 145

Netflix Tech

MARCH 5, 2019

Central engineering teams enable this operational model by reducing the cognitive burden on innovation teams through solutions related to securing, scaling and strengthening (resilience) the infrastructure. All these micro-services are currently operated in AWS cloud infrastructure.

Cloud 73

Cloud Building Amazon Web Services Metadata 73

Netflix Tech

JULY 8, 2020

By Torio Risianto, Bhargavi Reddy, Tanvi Sahni, Andrew Park Continue reading on Netflix TechBlog ».

Bytes 94

Bytes Data Cloud Storage AWS 94

RudderStack

MAY 12, 2021

RudderStack is in a chat with founder Soumyadeb Mitra to discuss the Open-source Data Infrastructure especially focusing on data privacy, safety, & reliability.

Data 40

Data 40