Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. Despite this, it is still operationally challenging to deploy and maintain your own stream processing infrastructure. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Process 182

Process Building SQL BI 182

Data Engineering Weekly

MAY 5, 2024

In an ideal world, data should be thought of as “borrowed” (possibly unpermitted) and thus can be “returned,” and unlearning should enable such revocation. A key thought-provoking moment for me while reading the article is this quote. A key thought-provoking moment for me while reading the article is this quote.

Data Engineering 67

Data Engineering Data Engineer Engineering Google Cloud 67

Data Engineering Podcast

NOVEMBER 12, 2023

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Cloudera

MAY 10, 2024

Iceberg, a high-performance open-source format for huge analytic tables, delivers the reliability and simplicity of SQL tables to big data while allowing for multiple engines like Spark, Flink, Trino, Presto, Hive, and Impala to work with the same tables, all at the same time. The Iceberg community is deeply important to us at Cloudera.

Government 68

Government Data Governance Data Pipeline Big Data 68

Knowledge Hut

MAY 2, 2024

Most cutting-edge technology organizations like Netflix, Apple, Facebook, and Uber have massive Spark clusters for data processing and analytics. Why We Need Big Data Frameworks Big data is primarily defined by the volume of a data set. It is surprising to know how much data is generated every minute. As estimated by DOMO : Over 2.5

Scala 94

Scala Hadoop Datasets Java 94

Data Engineering Podcast

OCTOBER 22, 2023

Summary The primary application of data has moved beyond analytics. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With the broader audience comes the need to present data in a more approachable format. With Materialize, you can!

BI 162

BI SQL Machine Learning Programming Language 162

Data Engineering Podcast

OCTOBER 8, 2023

Summary The insurance industry is notoriously opaque and hard to navigate. Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. In this episode he shares his journey of data collection and analysis and the challenges of automating an intentionally manual industry.

Insurance 162

Insurance BI SQL Machine Learning 162

Rockset

MARCH 1, 2023

Every database built for real-time analytics has a fundamental limitation. When you deconstruct the core database architecture, deep in the heart of it you will find a single component that is performing two distinct competing functions: real-time data ingestion and query serving. This changes today.

Data Ingestion 144

Data Ingestion Database Architecture Cloud Storage 144

Snowflake

MAY 1, 2024

The ability to handle large volumes of real-time geospatial data and meet customers’ complex analysis demands is a particular point of pride for the BigGeo team. The 2024 Snowflake Startup Challenge began with over 900 applications from startups Powered by Snowflake in more than 100 countries. Let’s get to know the 2024 finalists.

Media 102

Media Building Data Warehouse Technology 102

Striim

APRIL 1, 2024

Focused on delivering real-time intelligence for AI and leveraging change data capture (CDC) from databases, Striim’s approach addresses the urgent need for thorough data integration, emphasizing the critical role of connecting disparate applications to fully realize their potential. Secure Connectivity: Secure, OAuth connectivity and SAML 2.0

Hospitality 52

Hospitality Transportation Google Cloud Cloud 52

Data Engineering Podcast

OCTOBER 1, 2023

It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI.

Building 162

Building BI SQL Machine Learning 162

Knowledge Hut

MAY 2, 2024

As per Apache, “ Apache Spark is a unified analytics engine for large-scale data processing ” Spark is a cluster computing framework, somewhat similar to MapReduce but has a lot more capabilities, features, speed and provides APIs for developers in many languages like Scala, Python, Java and R. billion (2019 - 2022).

Scala 52

Scala Hospitality Healthcare Retail 52

Data Engineering Weekly

MARCH 24, 2024

link] Kai Waehner: The Data Streaming Landscape 2024 This is a comprehensive overview of the state of the data streaming landscape in 2024. As we predicted in the key trends of 2023 about Apache Flink as a clear winner in the stream processing frameworks, we see Confluent offering Flink as a service.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Striim

NOVEMBER 17, 2023

This strategic alliance opens up a world of possibilities for businesses looking to leverage the power and speed of Striim’s real-time data streaming and integration capabilities to seamlessly move data into the Yellowbrick Data Warehouse and drive lightning-fast analytics.

Data Analytics 59

Data Analytics Data Warehouse Banking Data 59

Data Engineering Podcast

JULY 23, 2023

Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. To bring streaming data in reach of application engineers Matteo Pelati helped to create Dozer. Despite that, it is still a complex set of capabilities. Make your data team unstoppable with Hex.

Building 130

Building SQL Machine Learning Data Ingestion 130

Striim

APRIL 3, 2024

Recognizing these challenges, Striim, a leader in real time intelligence for AI and change data capture (CDC) from databases, has introduced a comprehensive solution: Striim Cloud for Application Integration. Data Transformation : Enables the transformation of data in-flight, delivering business-ready HubSpot data to BigQuery in real time.

Google Cloud 52

Google Cloud Transportation Data Integration Aggregated Data 52

Snowflake

FEBRUARY 12, 2024

One way telecom companies stay ahead of the curve is by attending the world’s largest and most influential telecom and connectivity event: Mobile World Congress (MWC) 2024, held February 26 – 29 in Barcelona, Spain. This year’s theme is all about powering connectivity. We’re at the heart of it.

Python 78

Python Manufacturing Cloud Government 78

Data Engineering Podcast

JUNE 4, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Can you describe the workflow for a team that is using Agile Data Engine to power their business analytics? What does CI/CD look like for a data warehouse?

Data Engineering 130

Data Engineering Data Engineer Engineering Data Lake 130

Workfall

JUNE 27, 2023

Reading Time: 7 minutes The world of creating real-time applications can get complex when we have to consider latency, fault tolerance, scalability, and possible data losses. Traditionally, web sockets were the go-to option when it came to real-time applications, but think of a situation whereby there’s server downtime.

Kafka 75

Kafka Architecture AWS Transportation 75

Knowledge Hut

JANUARY 11, 2024

Data scientists have been using data science techniques to solve challenging real-world issues in healthcare, agriculture, manufacturing, automotive, and many more. Qantas uses predictive analytics to mitigate losses Healthcare: Novo Nordisk is Driving innovation with NLP. Almost every industry uses data science in some way.

Data Science 98

Data Science Transportation Hospitality Banking 98

Knowledge Hut

DECEMBER 22, 2023

Microsoft developed it and combines business analytics, data visualization, and best practices. It was unable to perform ad hoc and real-time analysis efficiently. With it, they could refresh 20 billion rows of data in near real-time pull in the data faster and perform real-time analysis to derive insights from data.

BI 98

BI Business Intelligence Machine Learning SQL 98

Knowledge Hut

APRIL 25, 2024

Most businesses today understand how to gather the terabytes of data that constantly pour into their operations and utilize analytics to transform them into insightful information. Given its advantages, big data and analytics are crucial for any business trying to maximize its commercial potential. What is Big Data?

Big Data 97

Big Data Entertainment Transportation Banking 97

Christophe Blefari

JULY 3, 2023

Dynamic Tables — Dynamic tables are streaming tables. With Snowflake you can send real time data coming from Kafka, for instance, with dynamic tables you can create a table on top of the real time data that refreshes in real time, using only what's needed to compute the new state.

SQL 130

SQL Data Kafka AWS 130

Knowledge Hut

APRIL 23, 2024

Big data has revolutionized the world of data science altogether. With the help of big data analytics, we can gain insights from large datasets and reveal previously concealed patterns, trends, and correlations. It represents the vast quantity of data involved in big data analytics and the sheer scale of data being produced.

Big Data 52

Big Data Media Datasets Unstructured Data 52

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Jesse Anderson

DECEMBER 12, 2022

Hadoop didn’t support doing things in real-time, and Apache Storm was open sourced in 2011. It didn’t get wide adoption as it was a bit early for real-time, and the API was difficult to wield. It didn’t get wide adoption as it was a bit early for real-time, and the API was difficult to wield. In the beginning, there was Google.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Striim

DECEMBER 7, 2023

In the fast-paced world of healthcare, particularly in emergency rooms, the need for real-time data analytics has never been more pressing. The Urgency of Real-Time Data in Emergency Rooms Emergency rooms are the epicenters of acute healthcare services, where rapid decision-making is crucial.

Data Analytics 52

Data Analytics Healthcare Hospitality Medical 52

Christophe Blefari

FEBRUARY 3, 2024

hey ( credits ) Hello here, this is Christophe from Amsterdam. I hope you're doing good. I'm in Amsterdam for the day for the DuckCon #4. The DuckDB annual conference, and god I like Europe. Being able to travel by train from Berlin to Paris to Amsterdam while going to the west of France for a lecture in a week is something truly awesome.

MongoDB 130

MongoDB SQL Data Data Warehouse 130

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. Get ready to delve into fascinating data engineering project concepts and explore a world of exciting data engineering projects in this article. Therefore, the greatest thing you can do as a novice is to work on some real-time data engineering initiatives.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Workfall

JUNE 12, 2023

In this dynamic realm of data engineering, a monumental challenge takes centre stage: efficiently managing the ever-changing tides of real-time data. Timely analytics, the secret sauce that empowers companies to gain a competitive edge, are in high demand. The question then arises: how can they keep pace with this deluge of data?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Towards Data Science

APRIL 30, 2024

Intro Google Dataflow is a fully managed data processing service that provides serverless unified stream and batch data processing. It is the first choice Google would recommend when dealing with a stream processing workload. It is the first choice Google would recommend when dealing with a stream processing workload.

Google Cloud 62

Google Cloud Process Cloud Lambda Architecture 62

Towards Data Science

APRIL 3, 2023

Building a Mental Model for Engineers and Anyone in Between Stream Processing can be handled gently and with care, or wildly, and almost out of control! How you transition from a batch mindset to a streaming mindset although can also be tricky, so let’s start small and build. You be the judge of what future you’d rather embrace.

Process 86

Process Data Lake Systems Bytes 86

Striim

FEBRUARY 8, 2024

This blog delves into the transformative role real-time data analytics plays in streamlining ground operations, reducing delays, and boosting overall efficiency in the aviation industry. Integrating Real-Time Data for Smarter Operations Enter real-time data analytics, a game-changer for optimizing plane turnaround times.

MongoDB 52

MongoDB Data Analytics Data Data Integration 52

Knowledge Hut

NOVEMBER 2, 2023

The Azure Data Engineer certification aspirants frequently seek out real-world projects in order to obtain hands-on experience and demonstrate their skills. The Azure Data Engineer certification aspirants frequently seek out real-world projects in order to obtain hands-on experience and demonstrate their skills.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Rockset

AUGUST 28, 2023

Our team is on a mission to bring the power of search and AI to every digital disruptor in the world. Today, we are thrilled to announce a major milestone in our journey towards redefining search and analytics for the AI era. Thousands of enterprises started tinkering with AI when ChatGPT showed the world the art of the possible.

Metadata 78

Metadata Unstructured Data SQL Database 78

Snowflake

MAY 15, 2023

Unfortunately, marketers struggle with gaining such a view because most solutions make it difficult, if not impossible, to centralize data and deliver data-driven insights in real-time. This approach enables companies to decrease customer acquisition costs (CAC) and boost their return on ad spend (ROAS).

Data Collection 72

Data Collection Utilities AWS Cloud 72

Knowledge Hut

APRIL 24, 2024

Undertaking real-life projects equips you with a deep understanding of programming languages, tools, and frameworks, preparing you to face intricate challenges and devise efficient solutions. Engaging in software engineering projects not only helps sharpen your programming abilities but also enhances your professional portfolio.

Software Engineer 52

Software Engineer Software Engineering Coding Project 52