Analytics Vidhya

MARCH 12, 2025

Apache Iceberg is a modern table format designed to overcome the limitations of traditional Hive tables, offering improved performance, consistency, and scalability. Well also dive into […] The post How to Use Apache Iceberg Tables?

Data Lake 134

Data Lake Designing IT Data 134

Seattle Data Guy

JANUARY 18, 2025

Because they can preserve the visual layout of documents and are compatible with a wide range of devices and operating systems, PDFs are used for everything from business forms and educational material to creative designs.

IT 130

IT Education Data Designing 130

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Datasets 104

Datasets Computer Science Systems Kafka 104

KDnuggets

JUNE 10, 2025

It is embedded in Pythons design philosophy to emphasize that errors need to be shown explicitly, following the principle that its Easier to Ask Forgiveness than Permission (EAFP), which allows you to execute the code first before knowing whether theres an error. Python is designed to use objects only in their intended ways.

Python 97

Python Machine Learning Data Science Programming Language 97

The Pragmatic Engineer

JUNE 12, 2025

If you hired a competent engineer in 2024 to design a system that takes a prompt and pretends to be an AI, and they could use any tool they liked, and there were 700 devs for the project, what they built would look something like this: How to pretend to be an AI in 2024… just use an LLM!

Engineering 130

Engineering Media Coding Systems 130

ProjectPro

JUNE 6, 2025

Like in most protocol methods, messages sent do not guarantee that they've been delivered and processed, so RabbitMQ adopts message delivery acknowledgment and implements smart broker design in its architecture. That is why this design is also called dumb broker, the intelligent consumer. The flow of events in Kafka is as follows.

Kafka 72

Kafka Java Big Data Architecture 72

The Pragmatic Engineer

JUNE 1, 2023

Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js Phase 3: a basic UX (February) Migrate to React Design a car, from a vector image Move a car with animations. Including adding unit tests.

Education 364

Education Project PostgreSQL Software Engineer 364

Snowflake

JUNE 10, 2025

It’s the result of thoughtful design, rigorous discussion and collaboration across dozens of organizations and hundreds of individuals. And once we got to designing v3, we wanted to see what could be done differently to avoid the overhead of the conversion.” On disk you had Parquet files, in-memory had bitmaps.

Metadata 95

Metadata Software Engineer Software Engineering Technology 95

KDnuggets

JUNE 10, 2025

By Josep Ferrer , KDnuggets AI Content Specialist on June 10, 2025 in Python Image by Author DuckDB is a fast, in-process analytical database designed for modern data analysis. This design optimizes CPU cache usage and significantly accelerates analytical query performance.

Python 115

Python SQL Data Science Machine Learning 115

Jesse Anderson

FEBRUARY 11, 2025

We delve into the architecture of Kuzu, an embedded graph database designed for high performance, exploring its unique storage format, query planning techniques like Sideway Information Passing (SIP), and the rationale behind its schema-based approach.

Computer Science 130

Computer Science Database Design Software Engineer Software Engineering 130

The Pragmatic Engineer

NOVEMBER 21, 2023

Joshua also writes an excellent Substack newsletter about how to design products which customers love, how to operate live services at scale, grow and optimize your technology orgs, and the history of the tech industry. ARPANET was designed to maintain communications during outages. Subscribe here.

Engineering 338

Engineering Bytes Cloud Computing AWS 338

databricks

JUNE 11, 2025

Published: June 11, 2025 Announcements 5 min read by Ali Ghodsi , Stas Kelvich , Heikki Linnakangas , Nikita Shamgunov , Arsalan Tavakoli-Shiraji , Patrick Wendell , Reynold Xin and Matei Zaharia Share this post Keep up with us Subscribe Summary Operational databases were not designed for today’s AI-driven applications.

Entertainment 81

Entertainment Data Lake Manufacturing Consulting 81

KDnuggets

MAY 23, 2025

Design, test, and deploy multi-agent systems in hours using the powerful agentic frameworks.

Python 138

Python Designing Systems 138

Engineering at Meta

DECEMBER 2, 2024

Andromeda is Meta’s proprietary machine learning (ML) system design for retrieval in ad recommendation focused on delivering a step-function improvement in value to our advertisers and people. Andromeda is designed to maximize ads performance by utilizing the exponential growth in volume of eligible ads available to the retrieval stage.

Engineering 92

Engineering Algorithm Utilities Machine Learning 92

Snowflake

JANUARY 6, 2025

It has inspired original equipment manufacturers (OEMs) to innovate their systems, designs and development processes, using data to achieve unprecedented levels of automation. And yet, the ultimate goal has remained elusive. Digital twins help OEMs manage this complexity by providing a comprehensive view of a vehicle's performance over time.

Manufacturing 95

Manufacturing Cloud Computing Data Storage Technology 95

KDnuggets

MAY 21, 2025

This guide has been designed to help you navigate towards the right one for you, depending on your data and the problem to address. Amid so many different machine learning algorithms to choose from.

Machine Learning 117

Machine Learning Algorithm Designing Data 117

KDnuggets

JUNE 11, 2025

Privacy has been carefully considered in the design of the dataset. Researchers will also find value in the multimodal nature of the dataset, which includes precomputed audio embeddings for over 7.7 million tracks, enabling content-aware recommendation strategies out of the box.

Datasets 126

Datasets Metadata Machine Learning Data Science 126

The Pragmatic Engineer

SEPTEMBER 19, 2024

We moved to an enterprise CRM system and created a fresh design.    Throughout the project, we mostly talked about things users could see, like the new design, the easier way to update content, the content workflow, etc. Our content management was implemented with similar home-spun scripts maintained by the same developers.

Recruitment 293

Recruitment Java Coding Project 293

Confessions of a Data Guy

JANUARY 17, 2025

Databricks Labs has unveiled DQX, a Python-based Data Quality framework explicitly designed for PySpark DataFrames. A Deep Dive into Databricks Labs’ DQX: The Data Quality Game Changer for PySpark DataFrames Recently, a LinkedIn announcement caught my eyeand honestly, it had me on the edge of my seat.

Data 100

Data Python Designing IT 100

ProjectPro

JUNE 6, 2025

MLOps Project - Designing an MLOps Solution 10 MLOps Projects Ideas for Beginners to Practice in 2025 MLOps Project Idea Kickstart Your Journey to Becoming an MLOps Engineer What is MLOps ? MLOps Project - Designing an MLOps Solution The MLOps landscape is frequently evolving. Table of Contents What is MLOps ?

Project 66

Project Amazon Web Services Machine Learning Data Science 66

Snowflake

APRIL 9, 2025

AI agents can assist with research, analytics, reconciliation and more just one part of KAWAs AI-native platform designed to enable automation with transparency and enterprise-grade security. It connects structured and unstructured databases across sources and uses a no-code UI or Python for advanced and predictive analytics.

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

Precisely

OCTOBER 31, 2024

Key Takeaways: Data mesh is a decentralized approach to data management, designed to shift creation and ownership of data products to domain-specific teams. This complexity requires a mature data engineering team to design, implement, and manage it effectively.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

Engineering at Meta

FEBRUARY 4, 2025

Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. Were sharing how Meta built support for data logs, which provide people with additional data about how they use our products. What are data logs?

Accessible 89

Accessible Accessibility Raw Data Data Warehouse 89

KDnuggets

MAY 28, 2025

Learn the key habits that make Python functions readable by design. If your functions need comments to be understood, its probably time for a rewrite.

Python 91

Python Designing IT 91

Snowflake

MARCH 20, 2025

They have to stitch together extraction, embedding, retrieval and language models that arent designed to work well together. Currently, people implement RAG pipelines that look like Frankensteins monster. What advice would you give to others considering building their applications on Snowflake?

Certification 95

Certification Programming Designing Building 95

ProjectPro

JUNE 6, 2025

It requires a skillful blend of data engineering expertise and the strategic use of tools designed to streamline this process. Data pipelines consist of interconnected tools and processes designed to handle the intricacies of data processing, transformation, and delivery. That’s where data pipeline tools come in.

Data Pipeline 49

Data Pipeline Google Cloud Kafka AWS 49

Snowflake

JUNE 1, 2025

This move will allow us to deliver Snowflake Postgres, a new kind of Postgres designed to power the most demanding, mission-critical AI and transactional systems at enterprise scale and with enterprise confidence. Why Crunchy Data?

PostgreSQL 72

PostgreSQL Database Cloud Government 72

Netflix Tech

JANUARY 6, 2025

To address this, we introduced the term Title Health, a concept designed to help us communicate effectively and capture the nuances of maintaining each titles visibility and performance. Up next In the next iteration we will talk about how to design an observability endpoint that works for all personalization systems.

Metadata 93

Metadata Algorithm Systems Engineering 93

Snowflake

APRIL 2, 2025

Its vendor-neutral by design, and the Polaris governance structure and community-driven development ensures it remains so. Additionally, Snowflakes elastic, zero-ops data engine continues to evolve with capabilities purposefully designed to improve query performance and efficiency for Iceberg tables.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

ProjectPro

JUNE 6, 2025

Data warehousing tools are software applications designed to collect, store, manage, and analyze large volumes of data from various sources within an organization. Data warehousing tools are designed to handle such scenarios efficiently, enabling faster query performance and analysis, even on massive datasets.

Data Storage 58

Data Storage PostgreSQL Data Warehouse AWS 58