Rockset

APRIL 26, 2023

In part 1 , we covered the technology landscape for real-time analytics on streaming data. In this post, we’ll get to the details: how does one design an RTA system? Ingest performance : Similar to other use cases in this article, ingest performance is incredibly important when streaming data is arriving at high velocities.

Architecture 52

Architecture Transportation Data Lake Insurance 52

Ascend.io

AUGUST 31, 2023

Modern Cloud Data Platforms The native capabilities of the cloud providers have been joined by third-party services to offload that data into separate less costly systems that are optimized for analysis of that data. Source : A stream of sensor data represented as a directed acyclic graph.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Cloudera

SEPTEMBER 17, 2020

They were not able to quickly and easily query and analyze huge amounts of data as required. They also needed to combine text or other unstructured data with structured data and visualize the results in the same dashboards. Events or time-series data served by our real-time events or time-series data store solutions.

Unstructured Data 80

Unstructured Data Pharmaceutical Data Warehouse MySQL 80

Rockset

AUGUST 2, 2023

The steps Windward takes to create proprietary data and AI insights As Windward operated in a batch-based data stack, they stored raw data in S3. They used MongoDB as their metadata store to capture vessel and company data.

Database-centric 52

Database-centric PostgreSQL Transportation Insurance 52

Knowledge Hut

DECEMBER 21, 2023

Cons: Hive cannot function with real-time data. NoSQL This database management system has been designed in a way that it can store and handle huge amounts of semi-structured or unstructured data. Avro creates binary data which can be both compressed as well as split. Hive has high latency.

Hadoop 52

Hadoop Big Data NoSQL Unstructured Data 52

Silectis

JANUARY 3, 2021

Data Architecture Data architecture is a composition of models, rules, and standards for all data systems and interactions between them. Data Catalog An organized inventory of data assets relying on metadata to help with data management. Database A collection of structured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Towards Data Science

JANUARY 17, 2024

These are key in nearly all data pipelines, allowing for efficient data storage and easier querying and information extraction. They are designed to handle the challenges of big data like size, speed, and structure. Data engineers often face a plethora of choices.

Big Data 76

Big Data Data Data Storage SQL 76

Monte Carlo

MARCH 14, 2023

“We had some new use cases around data science that were hard to fit into our existing org. Up to this point, the BlaBlaCar data organization was designed around skillsets. They maintained a team of data analysts, a team of data engineers, and a team of data scientists.

Transportation 40

Transportation Government Google Cloud Architecture 40

Engineering at Meta

MARCH 18, 2024

In this post, we present the design behind Logarithm, and show how it powers AI training debugging use cases. We designed the system to support service-level guarantees on log freshness, completeness, durability, query latency, and query result completeness. Each log line can have zero or more metadata key-value pairs attached to it.

Engineering 89

Engineering Metadata Architecture Designing 89

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Data Engineering Podcast

JUNE 17, 2021

Kirk Marple has spent years working with data systems and the media industry, which inspired him to build a platform for automatically organizing your unstructured assets to make them more valuable. Supercharge your business teams with customer data using Hightouch for Reverse ETL today.

Unstructured Data 100

Unstructured Data Data Warehouse Metadata Media 100

Monte Carlo

AUGUST 25, 2023

Typically, data warehouses work best with structured data defined by specific schemas that organize your data into neat, well-labeled boxes. This same structure aids in maintaining data quality and simplifies how users interact with and understand the data.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Rather than defining schema upfront, a user can decide which data and schema they need for their use case.

Data Lake 84

Data Lake Data Warehouse Cloud Unstructured Data 84

Rockset

DECEMBER 22, 2020

We will also need to store this data in Elasticsearch. This will allow the front end to pass in the search terms and have the API execute the 3 queries and perform the join before sending the data back to the front end.

SQL 40

SQL Data MongoDB Aggregated Data 40

Engineering at Meta

FEBRUARY 6, 2024

The following is a hypothetical DotSlash file named node that is designed to run v18.19.0 The script we use to generate DotSlash files injects metadata about the build job that makes it straightforward to trace the provenance of the underlying artifacts. To that end, a DotSlash file is required to start with #!/usr/bin/env of Node.js.

Metadata 115

Metadata Coding Building Project 115

Zalando Engineering

JANUARY 30, 2023

Managing structured data in markdown is not ideal, despite the ability to use front matter for metadata. These conversations drive engineers to make changes to their design to fundamentally improve availability, or at least, to ensure their software facilitates easier intervention.

Management 58

Management Metadata Software Engineer Software Engineering 58

Precisely

OCTOBER 5, 2023

Determine what data you’ll need Once you’ve determined the use case, brainstorm and dig deeper into what your end goals are and what you need to know to get there. For example, will you need structured data, unstructured, or a combination?

Raw Data 69

Raw Data Insurance Datasets Telecommunication 69

ProjectPro

AUGUST 11, 2021

Data is collected and stored in data warehouses from multiple sources to provide insights into business data. Data warehouses store highly transformed, structured data that is preprocessed and designed to serve a specific purpose. Data from data warehouses is queried using SQL.

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

AltexSoft

AUGUST 22, 2022

In this post, we’ll attempt to explain the idea behind a data fabric, its architectural building blocks, the benefits it brings, and ways to approach its implementation. What is data fabric? to provide a unified view of all enterprise data. Data fabric architecture example. Data fabric architecture building blocks.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

Towards Data Science

MARCH 5, 2024

Summary ∘ Embrace data modeling best practices ∘ Master data operations for cost-effectiveness ∘ Design for efficiency and avoid unnecessary data persistence Disclaimer : BigQuery is a product which is constantly being developed, pricing might change at any time and this article is based on my own experience.

Bytes 75

Bytes Google Cloud Cloud Storage Utilities 75

Cloudera

SEPTEMBER 15, 2022

Apache Ozone is a distributed, scalable, and high-performance object store , available with Cloudera Data Platform (CDP), that can scale to billions of objects of varying sizes. Structured data (such as name, date, ID, and so on) will be stored in regular SQL databases like Hive or Impala databases. OBJECT_STORE Bucket (“OBS”).

Systems 86

Systems Hadoop Metadata Telecommunication 86

AltexSoft

JULY 18, 2023

Whether you’re a data scientist, software engineer, or big data enthusiast, get ready to explore the universe of Apache Spark and learn ways to utilize its strengths to the fullest. Maintained by the Apache Software Foundation, Apache Spark is an open-source, unified engine designed for large-scale data analytics.

Big Data 52

Big Data Data Process Process Hadoop 52

AltexSoft

NOVEMBER 10, 2021

In a nutshell, the lakehouse system leverages low-cost storage to keep large volumes of data in its raw formats just like data lakes. At the same time, it brings structure to data and empowers data management features similar to those in data warehouses by implementing the metadata layer on top of the store.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Data Engineering Weekly

MARCH 24, 2023

Architectural patterns for Data Creation There are three types of architecture patterns in data creation. Event Sourcing Change Data Capture [CDC] Outbox pattern 1. Event Sourcing Event sourcing is a system design pattern that writes the current state of the business process into a journal of records.

Engineering 67

Engineering Data Transportation Database 67

AltexSoft

NOVEMBER 1, 2023

NER for structuring unstructured data NER plays a pivotal role in converting unstructured text into structured data. Deep learning is a subfield of machine learning where artificial neural networks — complex algorithms designed to operate similarly to human brains — learn from vast data sets.

Deep Learning 52

Deep Learning Machine Learning Datasets Algorithm 52

AltexSoft

JUNE 26, 2023

From the perspective of data science, all miscellaneous forms of data fall into three large groups: structured, semi-structured, and unstructured. Key differences between structured, semi-structured, and unstructured data. Note, though, that not any type of web scraping is legal.

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

AltexSoft

JULY 29, 2022

No matter the actual size, each cluster accommodates three functional layers — Hadoop distributed file systems for data storage, Hadoop MapReduce for processing, and Hadoop Yarn for resource management. Hadoop distributed file system or HDFS is a data storage technology designed to handle gigabytes to terabytes or even petabytes of data.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

Snowflake

MARCH 30, 2023

External table registration and consumption as: Parquet : A popular column-oriented data file format for analytics AVRO : A serialization format for data under streaming pipelines ORC : An optimized row-columnar file format designed for Hive Iceberg : An open table format specifically designed for analytical queries.

Engineering 79

Engineering Raw Data Data Science Scala 79

AltexSoft

MARCH 2, 2023

Data modeling involves creating a conceptual representation of data objects and their relationships to each other, as well as the rules governing those relationships. To design an effective data governance program, it’s crucial to choose an operational model that fits your business size and structure.

Data Governance 52

Data Governance Government Healthcare Programming 52

AltexSoft

JUNE 7, 2021

HDFS master-slave structure. A HDFS Master Node, called a NameNode , keeps metadata with critical information about system files (like their names, locations, number of data blocks in the file, etc.) and keeps track of storage capacity, a volume of data being transferred, etc. Data storage options.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

AltexSoft

JULY 19, 2022

What data mesh is and is not. What data mesh IS. Data mesh is a set of principles for designing a modern distributed data architecture that focuses on business domains, not the technology used, and treats data as a product. For example, your organization has an HR platform that produces employee data.

Architecture 52

Architecture Data Lake Medical Datasets 52

Ascend.io

JULY 19, 2023

Establish standardized processes and guidelines for curating data products. To ensure consistency in the data product definitions across domains, these guidelines should at least cover: Metadata standards: Define a standard set of metadata to accompany every data product. Implement a publishing approval process.

Data Governance 40

Data Governance Government Metadata Data 40

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Think of an automatically updating encyclopedia for your data platform.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Netflix Tech

DECEMBER 14, 2018

A fundamental requirement for any lasting data system is that it should scale along with the growth of the business applications it wishes to serve. NMDB is built to be a highly scalable, multi-tenant, media metadata system that can serve a high volume of write/read throughput as well as support near real-time queries.

Media 94

Media Database Metadata Data Schemas 94

Netflix Tech

OCTOBER 31, 2018

The curious reader might have noticed that a majority of these characteristics relate to properties of the data managed by NMDB. Specifically, structured data that is modeled around the notion of a media timeline, with additional spatial properties. Hence, we designed it primarily around the notion of timed events.

Media 54

Media Metadata Data MongoDB 54

ProjectPro

JANUARY 31, 2022

Traditional data preparation platforms, including Apache Spark, are unnecessarily complex and inefficient, resulting in fragile and costly data pipelines. Snowflake's machine learning partners transfer most of their automated feature engineering down into Snowflake's cloud data platform.

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

ProjectPro

NOVEMBER 15, 2021

With SQL, machine learning, real-time data streaming, graph processing, and other features, this leads to incredibly rapid big data processing. DataFrames are used by Spark SQL to accommodate structured and semi-structured data. Calcite has chosen to stay out of the data storage and processing business.

Big Data 52

Big Data Project Metadata Programming Language 52