The Pragmatic Engineer

OCTOBER 17, 2024

The name comes from the concept of “spare cores:” machines currently unused, which can be reclaimed at any time, that cloud providers tend to offer at a steep discount to keep server utilization high. Spare Cores attempts to make it easier to compare prices across cloud providers. Source: Spare Cores. Tech stack.

Cloud 333

Cloud Metadata AWS Cloud Computing 333

Engineering at Meta

JANUARY 22, 2025

This information is then utilized to identify relevant matches with other people who have specified matched values in their dating preferences. To address this limitation, we utilize Privacy Probes , a key component of our PAI lineage technologies. Privacy Probes automate data flow discovery by collecting runtime signals.

Data Warehouse 116

Data Warehouse SQL Programming Language Data 116

KDnuggets

JUNE 11, 2025

But newer datasets—such as Amazon’s, Criteo’s, and now Yambda—offer the kind of scale and nuance needed to push models from academic novelty to real-world utility. Resources like MovieLens and Netflix Prize remain foundational for benchmarking and testing ideas.

Datasets 126

Datasets Metadata Machine Learning Data Science 126

The Pragmatic Engineer

MARCH 12, 2024

Responsible for building and maintaining developer tools so the programmer and copilot can do their jobs better; such as improving editors, building better debugging functionality, creating utility tools and macros, etc. The tester. They come up with test cases and data. Also responsible for scaffolding of tests. 

Engineering 322

Engineering Programming Language Hospitality Insurance 322

Engineering at Meta

APRIL 28, 2025

We shifted left by combining schematization together with annotations in code , in addition improving and utilizing multiple classification signals. Classification service : utilizes heuristic rules and machine learning models to classify data types with high accuracy. Strict measurements provided precision/recall guarantees.

Metadata 83

Metadata Data Utilities Data Warehouse 83

Cloudera

OCTOBER 10, 2024

Users may want to perform table maintenance functions, like expiring snapshots, removing old metadata files, and deleting orphan files, to optimize storage utilization and improve performance.

IT 84

IT Data Lake Metadata Data Warehouse 84

ArcGIS

MAY 27, 2025

Learn how to analyze the relationships between your network features and structures using ArcGIS Utility Network.

Utilities 62

Utilities Data Management Management Data 62

The Pragmatic Engineer

APRIL 11, 2024

This is also why banks utilize a 2-day “blackout period” which creates space to revert failed migrations or updates. In contrast, most other industries like ecommerce, utilities, airlines, etc, have no such luxury and must invest in zero-downtime migrations.

Banking 266

Banking Utilities Database Engineering 266

ProjectPro

JUNE 6, 2025

MongoDB Inc offers an amazing database technology that is utilized mainly for storing data in key-value pairs. Such flexibility offered by MongoDB enables developers to utilize it as a user-friendly file-sharing system if and when they wish to share the stored data. Users can utilize this data for self-analysis of their lifestyle.

MongoDB 66

MongoDB Coding Project NoSQL 66

Cloudera

DECEMBER 4, 2024

But now AWS customers will gain more flexibility, data utility, and complexity, supporting the modern data architecture. Today enterprises can leverage the combination of Cloudera and Snowflake—two best-of-breed tools for ingestion, processing and consumption of data—for a single source of truth across all data, analytics, and AI workloads.

AWS 89

AWS Raw Data Relational Database Government 89

Netflix Tech

MARCH 28, 2025

This scenario underscored the need for a new recommender system architecture where member preference learning is centralized, enhancing accessibility and utility across different models. Utilizing embeddings The model generates valuable embeddings for members and entities like videos, games, and genres.

Metadata 90

Metadata Bytes Entertainment Data Mining 90

Netflix Tech

DECEMBER 17, 2024

Our ecosystem enables engineering teams to run applications and services at scale, utilizing a mix of open-source and proprietary solutions.

Engineering 90

Engineering Entertainment Amazon Web Services Utilities 90

Engineering at Meta

JANUARY 10, 2025

Building foundations Where geology permits, utilize low- or no-concrete foundation designs for easier deployment into both greenfield and brownfield sites. Utilize cool surface materials and coatings (e.g., Lightweight flat-pack buildings suitable for a wide range of local and climatic conditions.

Transportation 110

Transportation Manufacturing Consulting Designing 110

Netflix Tech

DECEMBER 17, 2024

Our ecosystem enables engineering teams to run applications and services at scale, utilizing a mix of open-source and proprietary solutions. In turn, our self-serve platforms allow teams to create and deploy, sometimes custom, workloads more efficiently.

Cloud 88

Cloud Amazon Web Services AWS Datasets 88

ProjectPro

JUNE 6, 2025

MLflow Models: An MLflow Model is a standard setup for wrapping machine learning models that can be utilized in various downstream tools—for example, real-time serving from a REST API or batch inference on Apache Spark. It utilizes Docker containers to support the maintenance of machine learning systems.

Amazon Web Services 68

Amazon Web Services Machine Learning Datasets Algorithm 68

Engineering at Meta

DECEMBER 2, 2024

Andromeda is designed to maximize ads performance by utilizing the exponential growth in volume of eligible ads available to the retrieval stage. This model is built with low-latency, high-throughput, and memory-IO aware GPU operators, utilizing deep kernel fusion and advanced software pipelining techniques.

Engineering 92

Engineering Algorithm Utilities Machine Learning 92

ProjectPro

JUNE 6, 2025

From data exploration and processing to later stages like model training, model debugging, and, ultimately, model deployment, SageMaker utilizes all underlying resources like endpoints, notebook instances, the S3 bucket, and various built-in organization templates needed to complete your ML project.

Project 45

Project AWS Algorithm Machine Learning 45

Cloudera

JANUARY 7, 2025

A leading meal kit provider migrated its data architecture to Cloudera on AWS, utilizing Cloudera’s Open Data Lakehouse capabilities. Several organizations utilize multiple cloud providerssuch as AWS, Azure, and Google Cloudto enhance risk mitigation.

Cloud 82

Cloud Government AWS Unstructured Data 82

Engineering at Meta

FEBRUARY 3, 2025

The service continues to utilize TAI timestamps but can return UTC timestamps to clients via the API. Each time the library is called during the smearing period we adjust the return values based on the smearing algorithm, which shifts the time values 1 nanosecond every 62.5 microseconds.

Algorithm 98

Algorithm Utilities Systems Engineering 98

ProjectPro

JUNE 6, 2025

Amazon Redshift Node Configuration Comparison Utility Get Started to Learn Data Warehousing with Redshift Projects FAQ’s on AWS Redshift Projects 1. In the Amazon Redshift Node Configuration Comparsion Utility project, you will learn how to choose your workload on Redshift on the preferred instance type and the number of nodes.

Data Pipeline 64

Data Pipeline AWS Project Building 64

ProjectPro

JUNE 6, 2025

To understand the utility of AWS Athena, take this example of an E-commerce company. So, here is an AWS Athena Tutorial and a few AWS Athena project ideas that utilize a few of the big data tools discussed above. AWS Athena utilizes the Presto database and ANSI SQL in the background to perform complicated analytics tasks.

AWS 67

AWS Big Data SQL Raw Data 67

Scott Logic

MARCH 6, 2025

I am still incredibly optimistic about LLMs and their utility, but we dont need models that perform even better on some abstract benchmarks, we need models and tools that respond no, Im afraid I cannot do that, or can we break this task down, or can we discuss this a bit further.

Coding 104

Coding Building Utilities Programming 104

ProjectPro

JUNE 6, 2025

It is utilized as a valuable data review tool to ensure that the data is accurate and appropriate for future usage. Cluster mode should be utilized for deployment if the client computers are not near the cluster. Client mode can be utilized for deployment if the client computer is located within the cluster.

Hadoop 68

Hadoop Metadata Java Datasets 68

Snowflake

APRIL 9, 2025

SoFlo Solar SoFlo Solars SolarSync platform uses real-time AI data analytics and ML to transform underperforming residential solar systems into high-uptime clean energy assets, providing homeowners with savings while creating a virtual power plant network that delivers measurable value to utilities and grid operators.

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

Engineering at Meta

FEBRUARY 25, 2025

Weve even open sourced various examples and utilities we used to in our migration to manipulate Kotlin code. Meta has been working to shift its Android codebase from Java to Kotlin , a newer language for Android development that offers some key advantages over Java.

Java 107

Java IT Software Engineer Software Engineering 107

Pinterest Engineering

JUNE 12, 2025

However, offline ANN is also a valuable option, particularly when large-scale data processing, efficient resource utilization, and cost-effective operations are critical. To address infrastructure costs while enhancing online efficiency, an alternative approach involves utilizing the Offline ANN.

Architecture 52

Architecture Algorithm Utilities Data Storage 52

Netflix Tech

MARCH 31, 2025

Many productions utilize a large variety of vendors, making this collaboration a large technical puzzle. Our AnswerContent Hubs Media Production Suite(MPS) [link] Building a global scalable solution that could be utilized in a diversity of markets has been an exciting challenge.

Media 86

Media Amazon Web Services Metadata Utilities 86

ProjectPro

JUNE 6, 2025

Utility-Based Agents These agents go beyond goals to optimize results by considering various performance measures. Strength: Utility-based agents can balance trade-offs and provide highly optimized solutions. Utility-Based Agents maximize utility through evaluation of outcomes. How to become an AI agent?

Deep Learning 80

Deep Learning Algorithm Machine Learning Banking 80

ProjectPro

JUNE 6, 2025

These libraries provide diverse pre-trained models, tools, and utilities for generating sentence embeddings, catering to different NLP needs and enabling developers to leverage embeddings for various tasks. apply(get_embeddings) Step 6: Utilize the Embeddings You can now utilize the generated sentence embeddings for various NLP tasks.

Python 60

Python Utilities Datasets Architecture 60

The Pragmatic Engineer

JUNE 1, 2023

Hardcoding secrets in production is poor practice. We covered how Stack Overflow learned this the hard way, a few months back.

Education 364

Education Project PostgreSQL Software Engineer 364

ProjectPro

JUNE 6, 2025

The 3 Different Types of Machine Learning 1) Supervised Learning Algorithms Supervised learning algorithms are supervision-based machine learning techniques, meaning the machine utilizes labeled data for the training process to predict the output. Can R be used for Machine learning?

Machine Learning 75

Machine Learning Algorithm Medical Utilities 75

ProjectPro

JUNE 6, 2025

Project Description: This project aims to provide a practical understanding of MLOps on AWS, utilizing NLP techniques for LDA Topic Modelling in Python using the RACE Dataset. AWS services such as EC2, S3, and Lambda are utilized to deploy and manage the machine learning models on the cloud.

Data Science 69

Data Science Python Project Google Cloud 69

ProjectPro

JUNE 6, 2025

For instance, Airbnb utilizes AWS Kafka to handle data from diverse sources such as property listings, user searches, and bookings, enabling them to adjust pricing and maximize revenue dynamically. It utilizes a partitioned log model, where data is organized into ordered sequences of records called logs. How does Kafka Work?

Kafka 40

Kafka AWS Amazon Web Services Data Pipeline 40

DataKitchen

NOVEMBER 27, 2024

While conceptually appealing, this generality can limit their practical utility in driving meaningful improvements in data quality. For instance, a healthcare company might use such a dashboard to monitor the accuracy and consistency of patient demographic data utilized in predictive models for patient readmittance models.

Banking 86

Banking Data Pharmaceutical Consulting 86

Snowflake

APRIL 13, 2025

Managing and utilizing data effectively is crucial for organizational success in today's fast-paced technological landscape. The vast amounts of data generated daily require advanced tools for efficient management and analysis. Enter agentic AI, a type of artificial intelligence set to transform enterprise data management.

Data Management 93

Data Management Management Consulting Unstructured Data 93

The Pragmatic Engineer

SEPTEMBER 26, 2023

With remote work, engineers spend more time on video calls, which utilizes laptop resources like CPU, memory, and more.   At companies with large codebases, CDEs are a pragmatic way to increase the productivity of software engineers. Remote work. Covid-19 lockdowns and the rise of remote working have played a role in boosting CDEs.

Cloud 317

Cloud Software Engineer Software Engineering Cloud Computing 317

Data Engineering Weekly

FEBRUARY 18, 2025

In contrast, Fluss is a storage to store real-time streaming data but doesn't provide OLAP abilities; it utilizes external query engines to process/analyze data, such as Flink and StarRocks/Spark/Trino (on the roadmap). We utilize the same approach with Kafka, the synchronous replication, and the ISR (in-sync-replicas) strategy.

Kafka 73

Kafka Lambda Architecture SQL Data Lake 73