Monte Carlo

FEBRUARY 21, 2023

Today, as data sources become increasingly varied, data management becomes more complex, and agility and scalability become essential traits for data leaders, data fabric is quickly becoming the future of data architecture. If data fabric is the future, how can you get your organization up-to-speed?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Today, as data sources become increasingly varied, data management becomes more complex, and agility and scalability become essential traits for data leaders, data fabric is quickly becoming the future of data architecture. If data fabric is the future, how can you get your organization up-to-speed?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Snowflake

JULY 27, 2023

Over the course of this journey, HomeToGo’s data needs have evolved considerably. Combining these services is helping the company to make cost-effective, data-driven business decisions, as well as giving us the confidence that our data architecture is robust and secure.

Data Architecture 52

Data Architecture Architecture Building Structured Data 52

Cloudera

AUGUST 9, 2022

With all of the buzz around cloud computing, many companies have overlooked the importance of hybrid data. The truth is, the future of data architecture is all about hybrid. We’ve seen this from all of our customers and are emphasizing building and iterating on modern data architectures. Do we need more than one?

Data Architecture 91

Data Architecture Architecture Cloud Computing Cloud 91

Cloudera

SEPTEMBER 29, 2022

Each of these trends claim to be complete models for their data architectures to solve the “everything everywhere all at once” problem. Data teams are confused as to whether they should get on the bandwagon of just one of these trends or pick a combination. First, we describe how data mesh and data fabric could be related.

Architecture 74

Architecture Data Architecture Metadata Data Warehouse 74

DataKitchen

JULY 22, 2021

Learn about four data architectures patterns for agility - DataOps, Data Fabric, Data Mesh & Functional Data Engineering - & an example combining all four. The post DataOps: The Foundation for Your Agile Data Architecture first appeared on DataKitchen.

Data Architecture 52

Data Architecture Architecture Data Data Engineering 52

KDnuggets

FEBRUARY 10, 2022

Going forward, data professionals have found a new way to address the scalability of sources through data mesh.

Data Architecture 108

Data Architecture Architecture IT Data 108

KDnuggets

FEBRUARY 16, 2022

How to Learn Math for Machine Learning; Data Mesh & Its Distributed Data Architecture; 5 Ways to Apply AI to Small Data Sets; Top 5 Free Machine Learning Courses; Junior Data Scientist: The Next Level.

Machine Learning 104

Machine Learning Data Architecture Architecture IT 104

KDnuggets

APRIL 5, 2023

Learn about Apache Kafka architecture and its implementation using a real-world use case of a taxi booking app.

Kafka 107

Kafka Architecture Data Architecture Building 107

databricks

MAY 1, 2024

Modern data stacks provide the necessary flexibility and efficiency for analytics and AI. Learn how the Databricks Data Intelligence Platform makes use of them.

Data Architecture 87

Data Architecture Architecture Data 87

DataKitchen

AUGUST 3, 2021

The post Building a Modern Data Architecture for the 2020s first appeared on DataKitchen.

Data Architecture 59

Data Architecture Architecture Building Data 59

databricks

APRIL 26, 2023

One of Lakehouse's outstanding achievements is the ability to combine workloads for modern use cases, such as traditional BI, machine learning & AI.

Data Architecture 93

Data Architecture Architecture BI Machine Learning 93

DataKitchen

JULY 16, 2021

The post How to build a successful cloud data architecture first appeared on DataKitchen.

Data Architecture 52

Data Architecture Architecture Cloud Building 52

Snowflake

JANUARY 30, 2023

As we continue unlocking the full value of Snowflake, we’re becoming even more efficient in how we use data to deliver the best service to our customers.” The post <strong>Ramp Simplifies Data Architecture Management, Cuts Costs, and Delivers Market Insights to Customers at Scale</strong> appeared first on Snowflake.

Data Architecture 52

Data Architecture Architecture Management Datasets 52

Data Engineering Podcast

SEPTEMBER 7, 2020

For analytical use cases you often want to combine data across multiple sources and storage locations. This frequently requires cumbersome and time-consuming data integration. Summary Databases are limited in scope to the information that they directly contain.

Architecture 100

Architecture Data Architecture SQL Engineering 100

Data Engineering Podcast

MAY 29, 2022

Summary A large fraction of data engineering work involves moving data from one storage location to another in order to support different access and query patterns. Singlestore aims to cut down on the number of database engines that you need to run so that you can reduce the amount of copying that is required.

Architecture 100

Architecture Database Data Architecture PostgreSQL 100

Confluent

OCTOBER 30, 2020

A fundamental challenge with today’s “data explosion” is finding the best answer to the question, “So where do I put my data?” while avoiding the longer-term problem of data warehouses, […].

Data Architecture 113

Data Architecture Architecture Kafka Building 113

Hevo

JULY 17, 2024

Every data-centric organization uses a data lake, warehouse, or both data architectures to meet its data needs. Data Lakes bring flexibility and accessibility, whereas warehouses bring structure and performance to the data architecture.

Data Ingestion 52

Data Ingestion Data Lake Data Architecture Architecture 52

Rockset

JULY 21, 2020

Lessons from scaling facebook's online data infrastructure There are 3 growth numbers that stand out when I look back at the hyper-growth years of facebook from 2007 until 2015, when I was managing facebook's online data infrastructure team: user growth, team growth and infrastructure growth.

Data Architecture 40

Data Architecture Architecture Building Programming Language 40

Towards Data Science

JANUARY 16, 2024

When I heard the words ‘decentralised data architecture’, I was left utterly confused at first! In my then limited experience as a Data Engineer, I had only come across centralised data architectures and they seemed to be working very well. So what was missing?

Data Warehouse 68

Data Warehouse Data Architecture Relational Database NoSQL 68

Data Engineering Podcast

JUNE 16, 2024

Your host is Tobias Macey and today I'm interviewing Kevin Liu about his use of Trino and Iceberg for Stripe's data lakehouse Interview Introduction How did you get involved in the area of data management? Can you describe what role Trino and Iceberg play in Stripe's data architecture?

Data Lake 147

Data Lake High Quality Data Metadata Python 147

KDnuggets

OCTOBER 30, 2023

A comparative overview of data warehouses, data lakes, and data marts to help you make informed decisions on data storage solutions for your data architecture.

Data Lake 128

Data Lake Data Warehouse Data Storage Data 128

Simon Späti

MAY 26, 2023

.” If you’ve journeyed with us from Part 1, where we dove into the importance and history of data modeling, or joined us in Part 2 to explore various approaches and techniques, I’m delighted you’ve stuck around. In this third part, we’ll delve into data architecture patterns and their influence on data modeling.

Architecture 130

Architecture Data Engineering Data Engineer Engineering 130

Simon Späti

MAY 26, 2023

.” If you’ve journeyed with us from Part 1, where we dove into the importance and history of data modeling, or joined us in Part 2 to explore various approaches and techniques, I’m delighted you’ve stuck around. In this third part, we’ll delve into data architecture patterns and their influence on data modeling.

Architecture 130

Architecture Data Engineering Data Engineer Engineering 130

Cloudyard

JULY 22, 2024

By the end of this post, you’ll have a comprehensive understanding of how to leverage this powerful integration to enhance your data architecture. Lambda Kinesis Step6: Attach function to S3 Bucket: Attach Lambda Step7: Upload file to the Bucket and this would trigger Lambda function which in turn will call kinesis data stream.

AWS 52

AWS Data Ingestion Data Architecture Architecture 52

Data Engineering Podcast

JUNE 17, 2023

Sign up free at dataengineeringpodcast.com/rudderstack - Your host is Tobias Macey and today I'm interviewing Satish Jayanthi about the practice and promise of building a column-aware data architecture through intentional modeling Interview Introduction How did you get involved in the area of data management?

Data Lake 130

Data Lake Metadata Machine Learning Data Architecture 130

Simon Späti

APRIL 3, 2023

This article is the first in a three-part series that will shine a spotlight on the fascinating world of data modeling, delving into its crucial importance within the broader context of data engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Architecture 130

Simon Späti

APRIL 3, 2023

This article is the first in a three-part series that will shine a spotlight on the fascinating world of data modeling, delving into its crucial importance within the broader context of data engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Architecture 130

Confluent

JANUARY 19, 2021

On a recent episode of Streaming Audio, Gwen Shapira, Michael Noll, and Ben Stopford joined me to hold forth about the near future of Apache Kafka® and software architecture in […].

Kafka 43

Kafka Architecture Data Architecture Data 43

Cloudera

MARCH 28, 2024

But, even with the backdrop of an AI-dominated future, many organizations still find themselves struggling with everything from managing data volumes and complexity to security concerns to rapidly proliferating data silos and governance challenges.

Data Architecture 93

Data Architecture Architecture Data Lake Data 93

Cloudera

APRIL 11, 2024

By bringing together various partners, we provide comprehensive solutions that address specific use cases and modernize data architecture. Without our partners, achieving true hybridity, enabling AI capabilities, or modernizing data architecture would be challenging.

Food 90

Food Data Architecture Architecture Programming 90

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineering 89

Data Engineering Data Engineer Engineering Data Lake 89

Data Engineering Weekly

DECEMBER 2, 2023

The Current State of the Data Architecture S3 intelligent tiered storage provides a fine balance between the cost and the duration of the data retention. However, the real-time insight on accessing the recent data remains a big challenge. The combination of stream processing + OLAP storage like Pinot.

IT 85

IT BI AWS Kafka 85

Snowflake

OCTOBER 23, 2023

This flexibility in language and programming constructs not only optimizes data architecture but also turbocharges development cycles, fortifies governance, enhances security, and more.

Python 141

Python Cloud Data Science Machine Learning 141

Start Data Engineering

AUGUST 29, 2021

You are given a quick overview of the business and data architecture and are assigned your very first data engineering task. Breakdown into sub-tasks 3.3. Delivering the finished task 4. Conclusion 5. Further reading 1. Introduction Congratulations!

Data Engineering 130

Data Engineering Data Engineer Engineering Data Architecture 130

Cloudera

MAY 10, 2024

Iceberg, a high-performance open-source format for huge analytic tables, delivers the reliability and simplicity of SQL tables to big data while allowing for multiple engines like Spark, Flink, Trino, Presto, Hive, and Impala to work with the same tables, all at the same time.

Government 102

Government Data Governance Data Pipeline Big Data 102

Cloudera

MAY 28, 2024

It’s not enough for businesses to implement and maintain a data architecture. The unpredictability of market shifts and the evolving use of new technologies means businesses need more data they can trust than ever to stay agile and make the right decisions.

Data Architecture 87

Data Architecture Architecture Data Lake Data Warehouse 87