December, 2022

article thumbnail

Data Pipeline Design Patterns - #1. Data flow patterns

Start Data Engineering

1. Introduction 2. Source & Sink 2.1. Source Replayability 2.2. Source Ordering 2.3. Sink Overwritability 3. Data pipeline patterns 3.1. Extraction patterns 3.1.1. Time ranged 3.1.2. Full Snapshot 3.1.3. Lookback 3.1.4. Streaming 3.2. Behavioral 3.2.1. Idempotent 3.2.2. Self-healing 3.3. Structural 3.3.1. Multi-hop pipelines 3.3.2. Conditional/ Dynamic pipelines 3.3.3.

article thumbnail

Dataframe Showdown – Polars vs Spark vs Pandas vs DataFusion. Guess who wins?

Confessions of a Data Guy

There once was a day when no one used DataFrames that much. Back before Spark had really gone mainstream, Data Scientists were still plinking around with Pandas a lot. My My, what would your mother say? How things have changed. Now everyone wants a piece of the DataFrame pie. I mean it tastes so good, […] The post Dataframe Showdown – Polars vs Spark vs Pandas vs DataFusion.

Data 148
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

A Return to the Office (RTO) Wave?

The Pragmatic Engineer

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of five topics in today’s subscriber-only The Scoop issue. To get this newsletter every week, subscribe here. On Thursday, 29 November, Snap CEO Evan Spiegel, sent an email announcing Snap will mandate 4 days/week in the office, starting from January.

article thumbnail

Should We Get Rid Of ETLs?

Seattle Data Guy

AWS has jumped on the bandwagon of removing the need for ETLs. Snowflake announced this both with their hybrid tables and their partnership with Salesforce. Now, I do take a little issue with the naming “Zero ETLs”. Because at the very surface the functionality described is often closer to a zero integration future, which probably… Read more The post Should We Get Rid Of ETLs?

AWS 130
article thumbnail

Driving Business Impact for PMs

Speaker: Jon Harmer, Product Manager for Google Cloud

Move from feature factory to customer outcomes and drive impact in your business! This session will provide you with a comprehensive set of tools to help you develop impactful products by shifting from output-based thinking to outcome-based thinking. You will deepen your understanding of your customers and their needs as well as identifying and de-risking the different kinds of hypotheses built into your roadmap.

article thumbnail

Data News — Week 22.50

Christophe Blefari

Prepping me to deliver Christmas' Data News ( credits ) Hey you, the end of the year is coming soon. I really liked this year with you. It was super fun to write every Friday of the year my opinion on data topics, I don't know yet if next year I'll be able to pull out stuff without repeating myself, I hate repeating myself, but for sure I'll try and I'll continue.

Kafka 130
article thumbnail

Brief History of Data Engineering

Jesse Anderson

In the beginning, there was Google. Google looked over the expanse of the growing internet and realized they’d need scalable systems. They created MapReduce and GFS in 2004. They published the papers for them in the same year. Doug Cutting took those papers and created Apache Hadoop in 2005. Cloudera was started in 2008, and HortonWorks started in 2011.

More Trending

article thumbnail

I asked ChatGPT to write a blog post about Data Engineering. Here it is.

Confessions of a Data Guy

Data engineering is a vital field within the realm of data science that focuses on the practical aspects of collecting, storing, and processing large amounts of data. It involves designing and building the infrastructure to store and process data, as well as developing the tools and systems to extract valuable insights and knowledge from that […] The post I asked ChatGPT to write a blog post about Data Engineering.

article thumbnail

Building a Telegram Bot Powered by Apache Kafka and ksqlDB

Confluent

ksqlDB use case: see how apps can use ksqlDB to ingest, filter, enrich, aggregate, and query data directly with Kafka—no complex architectures or data stores needed.

Kafka 144
article thumbnail

Data warehouses vs Data Lakes vs Databases – Which One Do You Need

Seattle Data Guy

By Reseun McClendon Today, your enterprise must effectively collect, store, and integrate data from disparate sources to both provide operational and analytical benefits. Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data. Data analysts, data scientists, engineers, and managers all require a robust data storage solution for… Read more The post Data warehouses vs Data Lakes vs Databases – Which One Do You Need appeared first on

Data Lake 130
article thumbnail

Data News — Week 22.49

Christophe Blefari

This is what we call a Chat in French ( credits ) Hello there, this is Christophe, live from the human world. Last week have been totally driven by ChatGPT frenzy, the social networks I use to follow are spammed with conversation screenshots and hype. On my side I don't know what the future holds for us but for sure MaaS—Models as a Service—looks not bright to me.

Data 130
article thumbnail

How to Build an Experimentation Culture for Data-Driven Product Development

Speaker: Margaret-Ann Seger, Head of Product, Statsig

Experimentation is often seen as an aspirational practice, especially at smaller, fast-moving companies who are strapped for time and resources. So, how can you get your team making decisions in a more data-driven way while continuing to remain lean and maintaining ship velocity? In this webinar, Margaret-Ann Seger, Head of Product at Statsig, will teach you how to build an experimentation culture from the ground-up, graduating from just getting started with data-driven development to operating

article thumbnail

Best of 2022: 5 Most Popular Cybersecurity Blogs Of The Year

U-Next

Introduction. Are you a Cybersecurity enthusiast looking to know the latest trends and goings in the cybersecurity industry? Or are you just a tech enthusiast who likes to be updated with the ongoings around them? Then you are at the perfect place. As another year comes to an end, we decided the best way to look back was to revisit the most popular and sought-after blogs of Cybersecurity and list the same for all our Cybersecurity enthusiasts.

Education 105
article thumbnail

Data Science Minimum: 10 Essential Skills You Need to Know to Start Doing Data Science

KDnuggets

Data science is ever-evolving, so mastering its foundational technical and soft skills will help you be successful in a career as a Data Scientist, as well as pursue advance concepts, such as deep learning and artificial intelligence.

article thumbnail

What is Apache Arrow? Asking for a friend.

Confessions of a Data Guy

We’ve all been in that spot, especially in tech. You wanted to fit in, be cool, and look smart, so you didn’t ask any questions. And now it’s too late. You’re stuck. Now you simply can’t ask … you’re too afraid. I get it. Apache Arrow is probably one of those things. It keeps popping […] The post What is Apache Arrow?

IT 130
article thumbnail

Broadcom Modernizes Machine Learning and Anomaly Detection with ksqlDB

Confluent

Broadcom's Mainframe Operational Intelligence Product (MOI) collects and analyzes data at mass scale, using ksqlDB to improve anomaly detection and custom alarm filtering.

article thumbnail

Reimagined: Building Products with Generative AI

“Reimagined: Building Products with Generative AI” is an extensive guide for integrating generative AI into product strategy and careers featuring over 150 real-world examples, 30 case studies, and 20+ frameworks, and endorsed by over 20 leading AI and product executives, inventors, entrepreneurs, and researchers.

article thumbnail

Reducing Data Analytics Costs In 2023 – Doing More With Less

Seattle Data Guy

If you haven’t started looking for ways to improve your data analytics budget for 2023, then you’re probably already behind. The truth is that between all of the various economic indicators and investor letters, everyone is looking to improve audit all parts of their business. Especially where there has likely been bloat. One of those… Read more The post Reducing Data Analytics Costs In 2023 – Doing More With Less appeared first on Seattle Data Guy.

article thumbnail

Safety First: Using vehicle data to make us all better drivers

Teradata

Vehicle data is invaluable in improving the safety & safe operation of vehicles for their occupants & other drivers. The next gen of vehicles will use real-time analysis to make driving even safer.

Data 105
article thumbnail

Introducing Cloudera DataFlow Designer: Self-service, No-Code Dataflow Design

Cloudera

Cloudera has been providing enterprise support for Apache NiFi since 2015, helping hundreds of organizations take control of their data movement pipelines on premises and in the public cloud. Working with these organizations has taught us a lot about the needs of developers and administrators when it comes to developing new dataflows and supporting them in mission-critical production environments. .

Designing 104
article thumbnail

More Data Science Cheatsheets

KDnuggets

It's time again to look at some data science cheatsheets. Here you can find a short selection of such resources which can cater to different existing levels of knowledge and breadth of topics of interest.

article thumbnail

Entity Resolution Checklist: What to Consider When Evaluating Options

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

article thumbnail

Why Data Migrations Suck.

Confessions of a Data Guy

I’ve often wondered what purgatory would be like, doing penance for millennia into eternity. It would probably be doing data migrations. I suppose they are not all that dissimilar from normal software migrations, but there are a few things that make data migrations a little more horrible and soul-sucking. Data migrations are able to slow […] The post Why Data Migrations Suck. appeared first on Confessions of a Data Guy.

Data 130
article thumbnail

From Eager to Smarter in Apache Kafka Consumer Rebalances

Confluent

Major improvements to the Kafka consumer, Streams, and ksqlDB for incremental cooperative rebalancing while maintaining at-least-once and exactly-once guarantees.

Kafka 138
article thumbnail

Data Catalog - A Broken Promise

Data Engineering Weekly

Data catalogs are the most expensive data integration systems you never intended to build. Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. I know that is an expensive statement to make😊 To be fair, I’m a big fan of data catalogs, or metadata management , to be precise.

article thumbnail

Teradata VantageCloud Lake + Vcinity Technology: Taking the Management Costs out of Network Latency

Teradata

Teradata VantageCloud Lake + Vcinity is perfect for on-premises, hybrid, & multi-cloud solutions where long network latency might keep an enterprise from leveraging access to their sensitive data.

article thumbnail

The Big Payoff of Application Analytics

Outdated or absent analytics won’t cut it in today’s data-driven applications – not for your end users, your development team, or your business. That’s what drove the five companies in this e-book to change their approach to analytics. Download this e-book to learn about the unique problems each company faced and how they achieved huge returns beyond expectation by embedding analytics into applications.

article thumbnail

You Can’t Hit What You Can’t See

Cloudera

Full-stack observability is a critical requirement for effective modern data platforms to deliver the agile, flexible, and cost-effective environment organizations are looking for. For analytic applications to properly leverage a hybrid, multi-cloud ecosystem to support modern data architectures, data observability has become even more important. I spoke to Mark Ramsey of Ramsey International (RI) to dive deeper into that last subject.

article thumbnail

How To Overcome The Fear of Math and Learn Math For Data Science

KDnuggets

Many aspiring Data Scientists, especially when self-learning, fail to learn the necessary math foundations. These recommendations for learning approaches along with references to valuable resources can help you overcome a personal sense of not being "the math type" or belief that you "always failed in math.".

article thumbnail

Why Picnic picked Java

Picnic Engineering

Picking a tech stack for your startup isn’t something to do lightly. It’s a choice that will shape the future in many ways: how will the tech enable your emerging product and business, what talent can you attract, and how future-proof is the tech stack? When Picnic launched as the first app-only supermarket back in 2015 in The Netherlands, the tech landscape looked markedly different from today.

Java 59
article thumbnail

ksqlDB Execution Plans: Move Fast But Don’t Break Things

Confluent

Build fast, break nothing. Learn about the unique challenges Confluent's engineering team has faced building ksqlDB and continuously shipping the latest, greatest features.

Building 120
article thumbnail

Monetizing Analytics Features

Think your customers will pay more for data visualizations in your application? Five years ago, they may have. But today, dashboards and visualizations have become table stakes. Turning analytics into a source of revenue means integrating advanced features in unique, hard-to-steal ways. Download this white paper to discover which features will differentiate your application and maximize the ROI of your analytics.

article thumbnail

Functional Data Engineering - A Blueprint

Data Engineering Weekly

The Rise of Data Modeling Data modeling has been one of the hot topics in Data LinkedIn. Hadoop put forward the schema-on-read strategy that leads to the disruption of data modeling techniques as we know until then. We went through a full cycle that “schema-on-read ” led to the infamous GIGO (Garbage In, Garbage Out) problem in data lakes, as noted in this What Happened To Hadoop retrospect.

article thumbnail

Career stories: Next-gen systems, servers, and SREs

LinkedIn Engineering

Saira joined our Bangalore site reliability engineering (SRE) team to tackle large-scale, site engineering challenges and grow. She highlights for us the impactful work she found here �����from ushering in LinkedIn���s next-generation, server query system that runs over a fleet of 350,000 servers, to mentoring the next generation of female engineers: In my engineering career, I���ve always followed the path less taken.

Systems 55
article thumbnail

Clouderans Celebrate the Holiday Season by Giving Back

Cloudera

Holiday season is a time to reflect on your year and support those less fortunate than yourself. . Clouderans made a global impact by running a number of donation activities and local giving events to celebrate the season of giving. . November 29: Giving Tuesday—Global . Giving Tuesday, a day dedicated to donations and giving back, is the Tuesday after Thanksgiving in the US.

Food 88
article thumbnail

We Don’t Need Data Scientists, We Need Data Engineers

KDnuggets

As more people are entering the field of Data Science and more companies are hiring for data-centric roles, what type of jobs are currently in highest demand? There is so much data in the world, and it just keeps flooding in, it now looks like companies are targeting those who can engineer that data more than those who can only model the data.

article thumbnail

How To Package & Price Embedded Analytics

Just by embedding analytics, app owners can charge 24% more for their product. How much value could you add? This framework from Software Pricing Partners explains how application enhancements can extend your product offerings. You’ll learn: How to take a disciplined approach to pricing The three elements of the Packaging Decision Framework Ways to structure your new embedded analytics offering Download the White Paper to learn about How To Package & Price Embedded Analytics.