article thumbnail

Infoshare 2024: Stream processing fallacies, part 1

Waitingforcode

Otherwise, you wouldn't read stream processing fallacies! Last week I was speaking in Gdansk on the DataMass track at Infoshare. As it often happens, the talk time slot impacted what I wanted to share but maybe it's for good.

Process 130
article thumbnail

Stream processing models

Waitingforcode

If you're interested in stream processing, I bet your thinking is technology-based. It's not wrong, after all, the ability to use a tool gives you and me a job. However, for a long-term consideration it's better to reason in terms of patterns or models. Being aware of a more general vision helps assimilate new tools.

Process 130
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Event time skew in stream processing

Waitingforcode

Turns out, stream processing also has its skew but more related to time. As a data engineer you're certainly familiar with data skew. Yes, this bad phenomena where one task takes considerably more input than the others and often causes unexpected latency or failures.

Process 130
article thumbnail

Infoshare 2024: Stream processing fallacies, part 2

Waitingforcode

The blog shares the last fallacies for my 7 years stream processing journey.

Process 130
article thumbnail

LLMs in Production: Tooling, Process, and Team Structure

Speaker: Dr. Greg Loughnane and Chris Alexiuk

Greg Loughnane and Chris Alexiuk in this exciting webinar to learn all about: How to design and implement production-ready systems with guardrails, active monitoring of key evaluation metrics beyond latency and token count, managing prompts, and understanding the process for continuous improvement Best practices for setting up the proper mix of open- (..)

article thumbnail

Modern Data Engineering with MAGE: Empowering Efficient Data Processing

Analytics Vidhya

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

article thumbnail

Arbitrary stateful processing in PySpark with applyInPandasWithState

Waitingforcode

you can even write arbitrary stateful processing jobs! It's always a huge pleasure to see the PySpark API covering more and more Scala API features. Starting from Apache Spark 3.4.0 But since the API is a little bit different than the one available on the Scala side, I wanted to take a deeper look.

Process 147
article thumbnail

LLMOps for Your Data: Best Practices to Ensure Safety, Quality, and Cost

Speaker: Shreya Rajpal, Co-Founder and CEO at Guardrails AI & Travis Addair, Co-Founder and CTO at Predibase

Putting the right LLMOps process in place today will pay dividends tomorrow, enabling you to leverage the part of AI that constitutes your IP – your data – to build a defensible AI strategy for the future.

article thumbnail

BI Buyers Guide: Embedding Analytics in Your Software

This exhaustive guide with a foreword from BI analyst Jen Underwood dives deep into the BI buying process and explores how to decide what features you need. And as the number of vendors grows, it gets harder to make sense of it all. Don't go into the fray unarmed.

article thumbnail

How to Find and Test Assumptions in Product Development

Assumptions mapping is the process of identifying and testing your riskiest ideas. Watch this webinar with Laura Klein, product manager and author of Build Better Products, to learn how to spot the unconscious assumptions which you’re basing decisions on and guidelines for validating (or invalidating) your ideas.

article thumbnail

7+ Graphics Libraries to Enhance Your Embedded Analytics

You’ll learn: Seven graphics libraries developers can use to enhance in-app analytics Easy-to-use wireframe tools to help the design and approval process The importance of modernizing your embedded analytics Download the e-book to learn about the seven-plus graphics libraries to enhance your embedded analytics.

article thumbnail

How to Package and Price Embedded Analytics

Just by embedding analytics, application owners can charge 24% more for their product. How much value could you add? This framework explains how application enhancements can extend your product offerings. Brought to you by Logi Analytics.

article thumbnail

Monetizing Analytics Features: Why Data Visualizations Will Never Be Enough

Think your customers will pay more for data visualizations in your application? Five years ago they may have. But today, dashboards and visualizations have become table stakes. Discover which features will differentiate your application and maximize the ROI of your embedded analytics. Brought to you by Logi Analytics.

article thumbnail

New Study: 2018 State of Embedded Analytics Report

Why do some embedded analytics projects succeed while others fail? We surveyed 500+ application teams embedding analytics to find out which analytics features actually move the needle. Read the 6th annual State of Embedded Analytics Report to discover new best practices. Brought to you by Logi Analytics.

article thumbnail

Drive Better Decision-Making with Data Storytelling

A good data story is formed by three components: Data analysis - This is the basis of a strong story and mastering the data is an essential part of the process. Visualizations and insights are supported by a narrative that helps to expedite the decision-making process.