Pinterest Engineering

OCTOBER 31, 2023

Harry Zhang, Jiajun Wang, Yi Li, Shunyao Li, Ming Zong, Haniel Martino, Cathy Lu, Quentin Miao, Hao Jiang, James Wen, David Westbrook | Cloud Runtime Team Image Source: [link] Overview Modern compute platforms are foundational to accelerating innovation and running applications more efficiently.

Architecture 95

Architecture Pipeline-centric Accessible Accessibility 95

Data Engineering Podcast

FEBRUARY 11, 2024

There are also numerous technical considerations to be made, particularly if the producer and consumer of the data aren't using the same platforms. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Data Engineering Podcast

JUNE 4, 2023

Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow. What are the components that you needed to develop in-house to enable your platform goals? Can you describe how Agile Data Engine is architected?

Data Engineering 130

Data Engineering Data Engineer Engineering Data Lake 130

Data Engineering Podcast

FEBRUARY 4, 2024

Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises. Dagster offers a new approach to building and running data platforms and data pipelines. What is the specific niche that RisingWave addresses?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

MARCH 31, 2024

Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises. Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Engineering at Meta

DECEMBER 19, 2023

But, because of space limitations, we will only give examples of two existing components that played an important role: ZippyDB , our distributed key/value datastore, and Async , our aptly named asynchronous serverless function platform. By onboarding their code in this platform, their code automatically inherits hyperscale attributes.

Programming Language 141

Programming Language Engineering Database Coding 141

Data Engineering Podcast

JANUARY 28, 2024

Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises. Can you describe the architecture of the Scanner platform? Can you describe what Scanner is and the story behind it?

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. What is the engineering work that is involved in building a cohesive platform from those components?

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

JANUARY 21, 2024

In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how he is applying the capabilities of cloud data warehouses to the challenge of building more comprehensive experiences for end-users through a modern customer data platform (CDP).

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

NOVEMBER 20, 2022

Sifflet is a platform that brings your entire data stack into focus to improve the reliability of your data assets and empower collaboration across your teams. RudderStack helps you build a customer data platform on your warehouse or data lake. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

JANUARY 7, 2024

Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.

Data Process 147

Data Process Process Data Lake High Quality Data 147

Snowflake

FEBRUARY 21, 2024

We previously announced Snowflake’s Unistore workload , which continues Snowflake’s legacy of breaking down data silos by uniting transactional and analytical data in a consistent and governed platform. The most powerful aspect of Hybrid Tables is that it is just another table type in Snowflake’s unified platform.

Government 92

Government AWS Data Architect Architecture 92

Cloudera

FEBRUARY 1, 2024

Implementing the right platform is half the battle won — so congratulations on your choosing Cloudera’s industry-leading hybrid data platform for building your data solutions on a foundation of trusted data. Status indicators at the component level that illustrate the state of the platform: healthy, warning, and critical.

Cloud 82

Cloud Architecture Utilities Designing 82

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Engineering at Meta

JANUARY 31, 2023

We’ve made architecture changes to Meta’s event driven asynchronous computing platform that have enabled easy integration with multiple event-sources. We’re sharing our learnings from handling various workloads and how to tackle trade offs made with certain design choices in building the platform.

Transportation 105

Transportation Architecture Data Warehouse MySQL 105

Data Engineering Podcast

MARCH 17, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Snowflake

FEBRUARY 7, 2024

It embodies our commitment to provide transparency and accountability in continuously investing in the improvements that make Snowflake run your workloads faster. To calculate SPI, we identify a group of customer workloads that are stable and comparable in both amount of queries and data processed over the period presented.

SQL 78

SQL Database Designing Accessible 78

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Snowflake

NOVEMBER 30, 2023

At TCS , we help companies shift their enterprise data warehouse (EDW) platforms to the cloud as well as offering IT services. A “lift and shift” migration approach handles all of these requirements by moving the business workload and data together. Are there any upcoming release cycles that could affect the data model and workloads?

Data Warehouse 93

Data Warehouse Banking Data Cloud 93

Robinhood

FEBRUARY 7, 2024

Hadoop-Based Batch Processing Platform (V1) Initial Architecture In our early days of batch processing, we set out to optimize data handling for speed and enhance developer efficiency. Since etcd is shared across the whole K8s cluster, there was concern that Spark’s workload would impact K8s’ cluster health.

Process 74

Process Hadoop Architecture Accessible 74

Cloudera

MARCH 22, 2023

Over the past handful of years, systems architecture has evolved from monolithic approaches to applications and platforms that leverage containers, schedulers, lambda functions, and more across heterogeneous infrastructures. And, crucial for a hybrid data platform, it does so across hybrid cloud.

Data Governance 85

Data Governance Government Business Analyst Metadata 85

Cloudera

OCTOBER 4, 2023

Different data streams will have different characteristics, and having a platform flexible enough to adapt, with things like flexible partitioning for example, will be essential in adapting to different source volume characteristics. We get optimized price/performance on complex workloads over massive scale data.

Telecommunication 85

Telecommunication Data Warehouse Java Manufacturing 85

Cloudera

OCTOBER 30, 2023

The public cloud is increasingly becoming the preferred platform to host data analytics – related projects, such as business intelligence, machine learning (ML), and AI applications. Cloudera FinOps Capabilities CDP is a cloud-native platform helping companies accelerate cloud adoption to run their data and analytics workloads.

Cloud 75

Cloud Finance Cloud Computing Government 75

Towards Data Science

JULY 24, 2023

While there is no right answer, there is likely a sweet spot for most organizations’ data platforms. A decade later, a considerable amount of innovation in the data management industry is revolving around how different data platforms are coupling or decoupling storage and compute (don’t worry I include examples in the next section).

Metadata 96

Metadata Data Warehouse Data Lake Data Science 96

RandomTrees

FEBRUARY 22, 2024

How to Migrate Your Business to the Cloud Moving to the google cloud workload is one of the most impressive things your business can utilize to build your adaptability, flexibility, and productivity. One of the best procedures for accomplishing these objectives is to migrate workloads to google cloud.

Google Cloud 52

Google Cloud Cloud Utilities Machine Learning 52

Cloudera

DECEMBER 13, 2023

To enable LGIM to better utilize its wealth of data, LGIM required a centralized platform that made internal data discovery easy for all teams and could securely integrate external partners and third-party outsourced data pipelines. To realize this cohesive data vision, LGIM adopted Cloudera Data Platform (CDP) Public Cloud.

Pipeline-centric 89

Pipeline-centric Professional Services Datasets BI 89

Knowledge Hut

APRIL 23, 2024

By leveraging distributed computing and parallel processing, big data platforms enable organizations to extract meaningful insights and patterns from massive datasets. Moreover, big data platforms are highly scalable and can handle vast amounts of data, making them suitable for real-time analytics and processing massive data streams.

Data Warehouse 52

Data Warehouse Big Data Unstructured Data Hadoop 52

Knowledge Hut

MARCH 29, 2024

Databricks is a web-based platform for working with Apache Spark. Azure Databricks is a managed Databricks platform on Azure. It provides a unified data analytics platform for data engineers, data analysts, data scientists, and machine learning engineers. It provides end-to-end automated data engineering and ML solutions.

Data Lake 52

Data Lake Scala Machine Learning SQL 52

Knowledge Hut

MARCH 14, 2024

Moreover, it permits efficient threat protection for cloud-based hybrid workloads. With the Azure Security Center, you might receive threat management for any hybrid workload across several clouds. Microsoft offers a solution called Azure Security Centre that offers uniform security management for hybrid cloud workloads.

Architecture 52

Architecture Cloud Certification Machine Learning 52

Rockset

MARCH 27, 2024

Rockset is a database made for scalable read operation workloads. Rockset’s unique model of separating compute and storage as well as separating compute resources that serve different workloads enables querying data at scale. Data is always fresh in Rockset, with data latencies in the 10s-100s of milliseconds.

Database 52

Database SQL Datasets Architecture 52

Snowflake

APRIL 17, 2024

Snowflake’s new Python API (in public preview) marks a significant leap forward, offering a more streamlined, powerful solution for using Python within your data pipelines — and furthering our vision to empower all developers, regardless of experience, with a user-friendly and approachable platform.

Data Pipeline 99

Data Pipeline Python Data Engineering Data Engineer 99

Snowflake

JANUARY 16, 2024

Cost Management Get account-level spend and usage metrics through an intuitive UI – public preview Customers can go to a single, centralized location in Snowsight called the Cost Management Interface to obtain visibility into cost and price-for-performance metrics for their workloads.

Python 98

Python SQL Machine Learning AWS 98

Cloudera

JULY 18, 2023

How do I define SLAs for workloads? How do I validate the setting for the platform and services? How do I use system/service/platform metrics to debug Hive queries and improve their performance? After all, your organization will run different workloads during different periods. How many queries failed?

BI 73

BI Datasets Big Data Data Engineering 73

Snowflake

NOVEMBER 1, 2023

These features collectively help developers build more quickly within a unified platform, distribute products globally, deliver them securely, and scale without operational burden. Hex: Hex, a platform for collaborative analytics and data science, can soon be deployed entirely within the end-consumer’s account. Let’s dive in!

AWS 92

AWS Programming Language Database Data Science 92

Edureka

FEBRUARY 8, 2024

Choosing an instance type that aligns with your scalability requirements is crucial for efficiently handling traffic spikes, accommodating growing workloads, and ensuring a seamless user experience during peak periods. Key Factors to Consider- When choosing an AWS instance type, several key factors should guide your decision-making process.

AWS 52

AWS NoSQL Deep Learning Datasets 52

Snowflake

JANUARY 18, 2024

Snowflake helps you answer these questions and more by providing a cloud-first data platform that is flexible, secure, easy to use, cost-effective, and accommodates all your data in one place. Convenient workload elasticity Snowflake’s flexible, scalable architecture allows you to attach and detach services as needed.

Manufacturing 83

Manufacturing Unstructured Data Cloud Architecture 83

Snowflake

SEPTEMBER 5, 2023

Snowflake’s fully managed platform helps minimize TCO by achieving faster time to insights and production, decreasing unplanned downtime and operational risks, and reducing business costs through customers paying only for actual usage. And so TFS looked to Snowflake to serve as a core part of this new data platform, which it called EDP 2.0.

Retail 82

Retail Finance Data Pipeline SQL 82