Cloudera

MARCH 2, 2023

It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. The CSP engine is powered by Apache Flink, which is the best-in-class processing engine for stateful streaming pipelines. Currently, Iceberg support in CSP is in technical preview mode.

Process 113

Process SQL Kafka Database 113

Netflix Tech

DECEMBER 14, 2023

Streaming SQL on Data Mesh using Apache Flink Mark Cho, Guil Pires and Sujay Jain, Engineers from the Netflix Data Platform talk about how a managed Streaming SQL using Apache Flink can help unlock new Stream Processing use cases at Netflix.

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Data Engineering Podcast

OCTOBER 15, 2023

In this episode Eric Sammer discusses why more companies are including real-time capabilities in their products and the ways that Decodable makes it faster and easier. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Process 182

Process Building SQL BI 182

Lyft Engineering

NOVEMBER 29, 2023

Real-time Ingestion Events from our real-time analytics pipeline were configured to be sent into our internal Flink application, streamed to Kafka, and written into Druid. Kinesis → Flink → ClickHouse : this ingestion scheme populates our events data in ClickHouse. This was our main form of ingestion.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Data Engineering Podcast

NOVEMBER 26, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Weekly

NOVEMBER 26, 2023

Databricks is already doing a lot of these move-up stack products. However, I do believe there is tons of innovation left in the deep engineering of LakeHouse, such as In a way, you can think of DuckDB -like systems as a feature of LakeHouse rather than a separate product. The sequence of Redshift migration is one part of it.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Pinterest Engineering

JULY 11, 2023

Our Flink clusters are multitenant and run jobs that concurrently process more than 20M msgs/sec across 12 clusters. Over the course of 2022 and early 2023, we’ve spent a significant period of time optimizing our Flink runtime environment and cluster configurations, and we’d like to share our learnings with you.

AWS 70

AWS Utilities Transportation Engineering 70

Knowledge Hut

APRIL 25, 2024

Apache Flink: Flink is an open-source stream processing framework that can perform high-speed analytics on live data streams. However, some of the most popular Big Data tools include Hadoop, Spark, and Flink. It includes Oracle's Exadata database system and other powerful Oracle software products.

Big Data 64

Big Data Technology NoSQL Hadoop 64

Cloudera

JULY 18, 2022

In this blog we will explore how we can use Apache Flink to get insights from data at a lightning-fast speed, and we will use Cloudera SQL Stream Builder GUI to easily create streaming jobs using only SQL language (no Java/Scala coding required). Apache Flink. Flink is a “streaming first” modern distributed system for data processing.

Process 86

Process Kafka Scala SQL 86

Data Engineering Podcast

MAY 21, 2023

The Machine Learning Podcast helps you go from idea to production with machine learning. The Machine Learning Podcast helps you go from idea to production with machine learning. Don't forget to check out our other shows. __init__ covers the Python language, its community, and the innovative ways it is being used.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Cloudera

JUNE 7, 2021

Today, it offers the breadth of products for managing data-in-motion from the edge to the cloud (or the enterprise). In the last two years, we have come out with several innovations and enhancements to our product offering. This enabled acceleration of building real-time streaming applications on Flink with a simple, easy-to-use GUI.

Kafka 101

Kafka Data Ingestion Architecture Cloud 101

Data Engineering Podcast

JULY 23, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack Modern data teams are using Hex to 10x their data impact. The Machine Learning Podcast helps you go from idea to production with machine learning. The Machine Learning Podcast helps you go from idea to production with machine learning.

Building 130

Building SQL Machine Learning Data Ingestion 130

Data Engineering Podcast

FEBRUARY 19, 2023

The Machine Learning Podcast helps you go from idea to production with machine learning. The Machine Learning Podcast helps you go from idea to production with machine learning. Signup for the SaaS product today at [dataengineeringpodcast.com/acryl]([link] Support Data Engineering Podcast

IT 147

IT Data Lake Metadata Data Warehouse 147

Data Engineering Weekly

MARCH 24, 2024

OpenAI has more production deployments, but Google is catching on to it. As we predicted in the key trends of 2023 about Apache Flink as a clear winner in the stream processing frameworks, we see Confluent offering Flink as a service. Data Mesh is certainly a hotly debated topic in the data industry.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Data Engineering Weekly

DECEMBER 25, 2023

While MDS has driven hyper-specialization in product categories, challenges in integration and overlapping tool categories have raised questions about its long-term viability. Streaming Architectures and the Rise of Apache Flink Streaming architectures have gained significant traction, with Apache Flink leading the way.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Netflix Tech

JANUARY 13, 2022

And we can’t discount the productivity impact it causes on data platform users. It motivates us to be proactive in detecting and handling failed workloads in our production environment, avoiding interruptions that could slow down our teams. If it crosses a threshold, Atlas sends a notification to Streaming Pensive.

Kafka 95

Kafka Big Data Data Machine Learning 95

Cloudera

MAY 1, 2023

By breaking down monolithic apps into microservices architectures, for example, or making modularized data products, organizations do their best to enable more rapid iterative cycles of design, build, test, and deployment of innovative solutions. An example of a project might be a fraud detection system implemented in Flink/SSB.

SQL 76

SQL Project Kafka Accessible 76

Christophe Blefari

FEBRUARY 23, 2024

Enabling near real-time data analytics on the data lake — Grab showcasing what they did with Flink and Hudi to enable real-time use-cases. Turning ideas into AI use cases — the Product Manager point of view. Cool ideas Open-source, current state and future hopes. Data will not tell you what to do.

Data Lake 130

Data Lake PostgreSQL MongoDB Scala 130

Cloudera

JUNE 7, 2021

Since then, we have been working hard to expose the full power of Apache Flink SQL and the existing Data Warehousing tools in CDP to combine it into a state-of-the-art real-time analytics platform. Flink SQL DDL and Catalog support. Flink SQL DDL Support. Feature Highlights. Improved Kafka and Schema Registry integration.

Kafka 101

Kafka SQL Accessible Accessibility 101

Christophe Blefari

OCTOBER 9, 2023

Confluent announced Kafka roadmap and Flink as a Cloud service — this is the result of Confluent acquisition of Immerok. Ensuring data quality with Verity — Lyft definition of data quality and a tour of the in-house product to address data quality in the data platform: Verity. Something that reminds dbt.

Python 130

Python Data Database Java 130

Christophe Blefari

OCTOBER 20, 2023

Still dbt Labs announcements were mainly towards dbt Cloud with great features to drive adoption of the paid product. They announced dbt Mesh a product enabling cross-project dependencies for teams with multiple dbt projects. With a product that has been launched in late 2022. As an echo of last bullet point.

Generalist 130

Generalist Entertainment NoSQL Datasets 130

Rock the JVM

FEBRUARY 5, 2023

Enter Giannis: Flink SQL is a powerful high level API for running queries on streaming (and batch) datasets. This is what is enables Unified Batch and Streaming Architecture and allows the use of a single API - like Flink SQL - to handle both Batch and Streaming data; no underlying code changes needed. Streaming (and Batch) SQL 1.1

SQL 63

SQL Kafka Metadata Database 63

Data Engineering Weekly

APRIL 14, 2024

Every survey demonstrates an increase in productivity in software development with copilot-type tools. However, TIL about Apache Paimon, a lake format that enables building a real-time lakehouse architecture with Flink and Spark for both streaming and batch operations.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Cloudera

OCTOBER 13, 2021

Cloudera produced a series of ebooks — Production Machine Learning For Dummies , Apache NiFi For Dummies , and Apache Flink For Dummies (coming soon) — to help simplify even the most complex tech topics. Learn about production machine learning (ML) and its difference from traditional ML workflows. There’s no need to panic.

Machine Learning 75

Machine Learning Data Ingestion Algorithm Technology 75

Cloudera

OCTOBER 4, 2023

For example: In manufacturing, fast-moving data provides the only way to detect — or even predict and prevent — defects in real time before they propagate across an entire production cycle. This will reduce defect rates, increasing product yield.

Telecommunication 85

Telecommunication Data Warehouse Java Manufacturing 85

Data Engineering Weekly

DECEMBER 3, 2023

Move Flink SQL to ClickHouse—certain Flink transformations can be directly done in the destination in ClickHouse. One of the highlights for me in the blog is the next step of ClickHouse adoption; as a believer in bringing “query to the data” rather than “data to query,” I love it.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Cloudera

DECEMBER 13, 2021

Cloudera’s security and engineering teams have identified the impact of this CVE across our product suite, and Cloudera customers have been sent detailed updates through Cloudera’s Technical Support Bulletins (TSB) and My Cloudera support cases. What Cloudera products and versions are affected?

Java 126

Java Coding Project Engineering 126

Christophe Blefari

FEBRUARY 24, 2023

But if you want to continue using the underlying tools here an overlook of Flink architecture or a few techniques you should know as a Kafka streams developer. Measuring everything — This blog is a proposition and a signal for you that you should measure absolutely everything to understand what is happening in your product.

Kafka 130

Kafka Data Lake Data Storage Data 130

Cloudera

JANUARY 19, 2022

Since then, we have seen great traction and a number of production implementations spanning from medium to extremely large in size. Some of these improvements and features are: Flink JAR submission (for Java UDF’s). Flink 1.14. Run/test/iterate on the CE environment, until your job is ready for production.

Java 86

Java Kafka SQL Cloud 86

Christophe Blefari

JANUARY 14, 2023

Analysis of Confluent buying Immerok — Jesse Anderson analyses last week news of Confluent (Kafka) buying Immerok (Flink) and what it implies in the real-time low-level technologies competition between Kafka / Flink / Spark.

Python 130

Python Kafka Data Scala 130

Cloudera

MARCH 5, 2024

This marks a significant milestone for the platform: according to IDC, today about half of the world’s enterprise production data under management is on-prem. This approach brings new compute engines into the fold, adding Spark, Flink, Impala, and NiFi, enabling concurrent access and processing of datasets within Iceberg.

Data Lake 87

Data Lake Government Data Storage Kafka 87

Christophe Blefari

DECEMBER 16, 2022

Usual readers of the Data News might notice that I don't go softly with dbt Labs when it comes to their Cloud product, but this is a reality, if I caricature a bit right now dbt Cloud is only a web IDE with the capabilities to run your models, it should be commodity, the real value of dbt exists only in Core for the moment and in the community.

Kafka 130

Kafka Data SQL Cloud 130

Pinterest Engineering

MAY 2, 2023

At Pinterest, most of our streaming jobs are built on top of Apache Flink , because Flink is a mature streaming framework with a lot of adoption in the industry. Compared with read-modify-write, which has to load all previous user sequence into Flink job, this approach uses far less memory and CPU.

Lambda Architecture 65

Lambda Architecture Datasets Software Engineer Software Engineering 65

Towards Data Science

FEBRUARY 9, 2024

It’s also worth mentioning that many vendors, such as Fivetran, have merged the best of OSS and venture capital funding into something called Product Led Growth, offering free tiers in their product offering that lower the barrier to entry into enterprise grade platforms.

Building 81

Building Transportation Data Lake Metadata 81

Data Engineering Weekly

NOVEMBER 27, 2022

Maybe Slack is 1% of the company implementing data engineering effectively to drive the product feature, but that is the point of implementing data contract and shifting left for an efficient data creation process. The blog captures the history of Flink SQL, its current state, and the challenges ahead of it.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Weekly

JULY 23, 2023

link] Sponsored: [New Report] The State of Data Products, 2023 Edition Data trust is on every data team’s mind, but how do you create and maintain it? To help answer those questions, we surveyed over 200 data teams to benchmark data product adoption rates and how data leaders can improve them. Are we reinventing Hive Metastore?!!

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Cloudera

SEPTEMBER 26, 2023

Learn more about how you can benefit from a well-supported data management platform and ecosystem of products, services and support by visiting the IBM and Cloudera partnership page. The post IBM Technology Chooses Cloudera as its Preferred Partner for Addressing Real Time Data Movement Using Kafka appeared first on Cloudera Blog.

Kafka 90

Kafka Technology IT Government 90