Data Engineering Podcast

NOVEMBER 12, 2023

In this episode Eran Yahav shares the journey that he has taken in building this product and the ways that it enhances the ability of humans to get their work done, and when the humans have to adapt to the tool. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

NOVEMBER 19, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 10, 2023

It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack Data projects are notoriously complex. What other tools/products might teams be evaluating while they consider Anomstack?

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

DECEMBER 3, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Systems 130

Systems Designing Data Lake SQL 130

Snowflake

DECEMBER 5, 2023

Engaging with partners and customers — and showcasing what’s new on the Snowflake product front — made for a dynamic time in Las Vegas. Here are highlights from the collaborations, integrations and product enhancements that we were proud to dig in to throughout the week.

AWS 93

AWS Amazon Web Services Government Cloud Computing 93

Snowflake

FEBRUARY 1, 2024

They let filmmakers create scenes without having to bring in actors and a production crew, because the models can generate content from trained images.” The default for streaming services’ original content has been to lock them up exclusively. However, this is not great news for cable providers. Ready for a deeper dive?

Entertainment 82

Entertainment Media Data Governance Government 82

Cloudera

SEPTEMBER 26, 2023

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle.

Kafka 90

Kafka Technology IT Government 90

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 4) Building Data Products and why should you?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Christophe Blefari

MARCH 15, 2024

Assisted AI wars are around the corner — I'm only following the French news, but the government is proudly doubling its budget for "AI defense" From what I know, AI is mainly used as an information companion to find signals in the huge amount of data we generate, creating more efficient agents.

Metadata 272

Metadata Datasets Data Data Warehouse 272

François Nguyen

MARCH 7, 2021

From Data Domains to stream aligned teams Everything starts with the data domains : you will have to divide all the data of your organisation into data domains. Some are with a strong Data Governance flavour very focus on glossaries, data quality and data stewardship. This is one team who is focus on value and data as a product.

Government 130

Government Data Governance Data Metadata 130

Data Engineering Podcast

MARCH 27, 2022

Summary Data governance is a practice that requires a high degree of flexibility and collaboration at the organizational and technical levels. Privacera is an enterprise grade solution for cloud and hybrid data governance built on top of the robust and battle tested Apache Ranger project.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

JULY 23, 2023

To bring streaming data in reach of application engineers Matteo Pelati helped to create Dozer. In this episode he explains how investing in high performance and operationally simplified streaming with a familiar API can yield significant benefits for software and data teams together. Don't forget to check out our other shows.

Building 130

Building SQL Machine Learning Data Ingestion 130

Data Engineering Weekly

FEBRUARY 25, 2024

link] Chris Riccomini: From Samza to Flink: A Decade of Stream Processing This article offers a fascinating historical perspective on stream processing through the evolution of Apache Samza. Samza’s stream-stream join relies on Kafka’s key partition to shift the streaming operation burden to Kafka.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Snowflake

MARCH 18, 2024

Breaking down data silos, removing duplication, creating trusted data products, reducing the cost of data rework, ensuring more timely insights and cross-functional use cases, and improving user adoption. Many modern enterprises have far-flung operations, products and value chains that generate data globally and in a federated way.

Unstructured Data 82

Unstructured Data Government Data Pipeline Cloud 82

Knowledge Hut

APRIL 26, 2024

Datasets may also be confidential as they may contain sensitive information pertaining to a product, organization or government. In the real world, data is not open source , as it is confidential and may contain very sensitive information related to an item , user or product. Data is not available in a specific format.

Datasets 81

Datasets Machine Learning Deep Learning Finance 81

Snowflake

DECEMBER 12, 2023

At our recent Snowday event, we announced a wave of Snowflake product innovations for easier application development, new AI and LLM capabilities, better cost management and more. Data Pipelines Improved processing for streaming data with Dynamic Tables – public preview Streaming and CDC data can be challenging to handle.

Metadata 83

Metadata Python Government AWS 83

Knowledge Hut

APRIL 16, 2024

empowers enterprises to - Check the plans more quickly (Lean Startup cycle + Lean User experience) Enable fastest delivery (Scalable DevOps + Continuous Delivery pipeline) Simplify governance and improve portfolio performance (Lean Portfolio management + Lean budgets) Why is SAFe® crucial for the organizations? for teams (SP) Today, SAFe® 6.0

Certification 88

Certification Portfolio Consulting Programming 88

Rockset

MARCH 18, 2024

Flink is one of the most popular stream processing technologies, ranked as a top five Apache project and backed by a diverse committer community including Alibaba and Apple. Rockset customers using Flink often share how challenging it is to self-manage Flink for streaming transformations.

Cloud 64

Cloud Building Metadata Kafka 64

Monte Carlo

MARCH 25, 2024

Beyond our recognition in the category, we were also named one of G2Crowd’s 100 Fastest Growing Products for 2024 , signifying our unrivaled customer adoption across the U.S., Releasing new products by building solutions for pipeline runtime optimization with Performance product and ad-hoc data analysis Data Explorer.

MySQL 64

MySQL Government Data Governance Kafka 64

Striim

NOVEMBER 8, 2023

Data Mesh is revolutionizing event streaming architecture by enabling organizations to quickly and easily integrate real-time data, streaming analytics, and more. By exploring the benefits of using Data Mesh for your event streaming architecture, this article will help you decide if it’s the right solution for your organization.

Architecture 52

Architecture Generalist Government Datasets 52

Snowflake

MARCH 2, 2023

That proves to be a difficult task for data engineering teams that have to manage separate infrastructure for batch data and streaming data. To address this challenge, we are happy to announce the public preview of Snowpipe Streaming as the latest addition to our Snowflake ingestion offerings. How does Snowpipe Streaming work?

Kafka 119

Kafka Data Ingestion Data Pipeline Cloud Storage 119

Data Engineering Weekly

MARCH 10, 2024

Big thanks to our insightful speakers, Hareshkumar Selvakumar - Talks about his work on Data Products for PayPal. It distinguishes between intrusive and non-intrusive data quality management, advocating for clear governance to address quality issues. Thanks to Ideas2IT Technologies for hosting us in their fantastic space.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Towards Data Science

FEBRUARY 9, 2024

Streaming Although I briefly mentioned streaming in my 2021 article, you’ll see a renewed focus in the 2024 version. I’m a strong believer that data has to move at the speed of business , and the only way to truly accomplish this in modern times is through data streaming.

Building 81

Building Transportation Data Lake Metadata 81

Data Engineering Weekly

NOVEMBER 26, 2023

Once captured, there are a lot of market opportunities to move up the stack with Data Catalog, Data Governance, and Data Quality. Databricks is already doing a lot of these move-up stack products. Databricks announced a fleet of features for catalogs, governance, observability, and orchestration.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Snowflake

FEBRUARY 5, 2024

While it will unlock enormous productivity and open up countless opportunities, organizations will also have to mitigate the risks of ungoverned usage to deliver successful outcomes. Unified data access, strong governance and robust security combined with technical expertise and a clear understanding of business objectives will be essential.”

Unstructured Data 92

Unstructured Data Banking Government Insurance 92

Data Engineering Weekly

MARCH 24, 2024

OpenAI has more production deployments, but Google is catching on to it. link] Kai Waehner: The Data Streaming Landscape 2024 This is a comprehensive overview of the state of the data streaming landscape in 2024. The Gen AI budget is now rolling into regular software budgeting rather than an experimental budget.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Snowflake

APRIL 16, 2024

At the Marketing Data Cloud Forum , I moderated a conversation with Jason Davis, Co-Founder and CEO at Simon Data; Jay Goebel, General Manager of Data and Identity Partnerships at The Trade Desk; and Kathryn Murphy, SVP of Product and Design at Twilio. There’s a lack of clarity around what tool does what and what data goes where.

Cloud 71

Cloud Government Data AWS 71

François Nguyen

MARCH 22, 2021

How do we build data products ? How do we govern all these data products and domains ? How to build Data Products or never call me Data Pipeline any more You have this interesting schema in her second article on Data Mesh by Zhamak Dehghani : “Data mesh introduces the concept of data product as its architectural quantum.

Architecture 130

Architecture Technology Google Cloud Metadata 130

Cloudera

JUNE 7, 2022

We are a Gartner Peer Insights Customer Choice for Cloud DBMS products. Also, Cloudera DataFlow is rated highly in the GigaOm Radar for Streaming Data Platforms. Common security, governance, metadata, replication, and automation enable CDP to operate as an integrated system.

IT 108

IT Unstructured Data Data Architecture Government 108

Data Engineering Podcast

NOVEMBER 27, 2022

He also discusses the improvements that have been incorporated into FeatureBase to simplify integration with the rest of your data stack, and the SQL interface that was added to make working with the product easier. By the time errors have made their way into production, it’s often too late and damage is done.

Data Lake 100

Data Lake Data Warehouse MongoDB Data Pipeline 100

Snowflake

FEBRUARY 19, 2024

In 2024, the retail and consumer goods industry is expected to experience massive upheaval due to the proliferation of generative AI (gen AI) tools as well as changes in customer engagement and the general manner in which products are now sold. For their full insights, read the new report, Retail Data + AI Predictions 2024.

Retail 70

Retail Media Data Government 70

The Modern Data Company

APRIL 15, 2024

Unlocking Business Value with Data Product Strategy Adopting a Data Product Strategy isn’t just about upgrading your technology; it’s about elevating your entire organization to operate in a more informed, agile, and strategic manner. Think of it as the central hub of your data universe.

Raw Data 52

Raw Data Government Data Datasets 52

Snowflake

JUNE 28, 2023

That’s why at our annual user conference, Snowflake Summit 2023 , we unveiled new features that further extend data programmability in Snowflake for their language of choice, without having to compromise on governance. To make it even easier to process data with Snowpark Python UDFs and Stored Procedures, we have added support for Python 3.9

Pipeline-centric 54

Pipeline-centric Programming Language Python Government 54

Snowflake

SEPTEMBER 14, 2023

Companies hampered by legacy data architectures are often plagued by a high total cost of ownership (TCO), an inability to govern data, and a lack of scalability as their data volumes grow. “As Snowflake’s platform provides industry-leading features that ensure the highest standards of governance for your account, users, and data.

Data Architecture 89

Data Architecture Architecture Hadoop Data Warehouse 89

Knowledge Hut

APRIL 25, 2024

Instead of being a single process, big data analytics is a collection of numerous business-related procedures that may also involve data scientists, business management , and production teams. Data Governance Data governance is the process of ensuring that data is trustworthy, accurate, available, and usable.

Big Data 52

Big Data Data Analytics Manufacturing Retail 52

Knowledge Hut

APRIL 23, 2024

Big Data Use Cases in Manufacturing Quality Control and Defect Detection: Real-time analysis of manufacturing data is used for quality control and fault detection, ensuring excellent product quality and reducing rework. With this technology, companies find new trends, consumer demands, and future product prospects.

Big Data 52

Big Data Retail Recruitment Transportation 52

Monte Carlo

JANUARY 12, 2024

Streaming data , as opposed to processing it in batches, is essential for gaining real-time insight, but has implications when it comes to maintaining data quality – fresher data can mean a higher risk of acting on inaccurate or incomplete data (which can be be addressed using the principles of data observability ).

Big Data 59

Big Data Data Analytics Data Science Data Lake 59