Snowflake

FEBRUARY 19, 2024

In 2024, the retail and consumer goods industry is expected to experience massive upheaval due to the proliferation of generative AI (gen AI) tools as well as changes in customer engagement and the general manner in which products are now sold. For their full insights, read the new report, Retail Data + AI Predictions 2024.

Retail 73

Retail Media Data Government 73

Knowledge Hut

APRIL 25, 2024

Big data in information technology is used to improve operations, provide better customer service, develop customized marketing campaigns, and take other actions to increase revenue and profits. It is especially true in the world of big data. It is especially true in the world of big data. What Are Big Data T echnologies?

Big Data 98

Big Data Technology NoSQL Hadoop 98

Knowledge Hut

MAY 2, 2024

As per Apache, “ Apache Spark is a unified analytics engine for large-scale data processing ” Spark is a cluster computing framework, somewhat similar to MapReduce but has a lot more capabilities, features, speed and provides APIs for developers in many languages like Scala, Python, Java and R. billion (2019 - 2022).

Scala 52

Scala Hospitality Healthcare Retail 52

Snowflake

MAY 9, 2024

From its start with efficient batch processing with data warehouses for descriptive analytics, and the inclusion of streaming data in real time to build recommendations, we find ourselves at the forefront of a new stage of evolution: generative AI (gen AI). The numbers back up Stuermer’s belief.

Unstructured Data 94

Unstructured Data Manufacturing Retail Data Warehouse 94

Data Engineering Weekly

APRIL 14, 2024

Every survey demonstrates an increase in productivity in software development with copilot-type tools. Discover how a universal semantic layer is transforming modern business intelligence, making data more accessible and reliable for organizations striving for informed business decisions.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Data integration from source systems to their downstream destinations is the foundational step for any data product. With the increasing expecation for information to be instantly accessible, it drives the need for reliable change data capture. Data teams are increasingly under pressure to deliver.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

NOVEMBER 6, 2022

Summary Despite the best efforts of data engineers, data is as messy as the real world. Entity resolution and fuzzy matching are powerful utilities for cleaning up data from disconnected sources, but it has typically required custom development and training machine learning models.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Weekly

NOVEMBER 26, 2023

The past few months have been a bit of a health hiccup for me 🤒, but hey, I'm all smiles 😊 about how things are turning out. It’s the end of the year, and there will be a lot of buzz about what the next five years in data engineering might bring. Databricks is already doing a lot of these move-up stack products.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Data Engineering Podcast

MAY 10, 2021

Summary Building an API for real-time data is a challenging project. The team at Tinybird wants to make it easy to turn a continuous stream of data into a production ready API or data product. RudderStack’s smart customer data pipeline is warehouse-first. Sound exciting?

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100

Data Engineering Weekly

SEPTEMBER 9, 2023

Aswin floated the idea of making it a meetup—inviting some data engineers and having an in-depth conversation about our field. In a half-joking manner, I threw out there, "Why not turn it into a full-blown data engineering conference?" Data modeling is my favorite topic. Panel: Is Modern Data Stack Dead?

Data Lake 62

Data Lake Architecture Data Engineering Data Engineer 62

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

OCTOBER 30, 2022

Summary One of the most impactful technologies for data analytics in recent years has been dbt. It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts.

Engineering 100

Engineering MongoDB Scala MySQL 100

Lyft Engineering

NOVEMBER 29, 2023

Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data. For our customers, this means faster analytics on near real-time data and decision making. An example of how we use Druid rollup at Lyft.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Knowledge Hut

APRIL 25, 2024

Big data can be summed up as a sizable data collection comprising a variety of informational sets. It is a vast and intricate data set. Big data has been a concept for some time, but it has only just begun to change the corporate sector. What is Big Data? What are the Benefits of Big Data?

Big Data 97

Big Data Entertainment Transportation Banking 97

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. RudderStack helps you build a customer data platform on your warehouse or data lake.

MongoDB 100

MongoDB Scala Google Cloud MySQL 100

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. Therefore, the greatest thing you can do as a novice is to work on some real-time data engineering initiatives.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Striim

FEBRUARY 8, 2024

The ability to swiftly turn around planes not only determines an airline’s efficiency but also impacts passenger satisfaction and operational profitability. Integrating Real-Time Data for Smarter Operations Enter real-time data analytics, a game-changer for optimizing plane turnaround times.

MongoDB 52

MongoDB Data Analytics Data Data Integration 52

Towards Data Science

APRIL 3, 2023

Building a Mental Model for Engineers and Anyone in Between Stream Processing can be handled gently and with care, or wildly, and almost out of control! By processing a smaller set of data, more often , you effectively divide and conquer a data problem that may otherwise be cost and time prohibitive.

Process 87

Process Data Lake Systems Bytes 87

Data Engineering Podcast

JULY 17, 2022

Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. RudderStack helps you build a customer data platform on your warehouse or data lake. Data teams are increasingly under pressure to deliver. In fact, while only 3.5%

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Cloudera

JUNE 2, 2022

Since 2015, the Cloudera DataFlow team has been helping the largest enterprise organizations in the world adopt Apache NiFi as their enterprise standard data movement tool. This need has generated a market opportunity for a universal data distribution service. Why does every organization need it when using a modern data stack?

Systems 104

Systems Data Lake Google Cloud Data Collection 104

Precisely

APRIL 29, 2024

It takes a balanced combination of technology (to break down silos, streamline data collection, and replace manual processes) and human care (to provide an empathetic, personalized experience through it all). Communication : clear, proactive updates through their preferred channels, minimizing the need to chase down information.

Insurance 64

Insurance Process Data Collection Accessible 64

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Cloudera

JUNE 26, 2023

Here are some tips and tricks of the trade to prevent well-intended yet inappropriate data engineering and data science activities from cluttering or crashing the cluster. For data engineering and data science teams, CDSW is highly effective as a comprehensive platform that trains, develops, and deploys machine learning models.

ETL Tools 78

ETL Tools Programming Language Datasets Data Pipeline 78

Cloudera

JULY 18, 2022

In part 1 of this blog we discussed how Cloudera DataFlow for the Public Cloud (CDF-PC), the universal data distribution service powered by Apache NiFi, can make it easy to acquire data from wherever it originates and move it efficiently to make it available to other applications in a streaming fashion. Data decays!

Process 91

Process Kafka Scala SQL 91

Data Engineering Podcast

OCTOBER 2, 2022

Summary Data lineage is something that has grown from a convenient feature to a critical need as data systems have grown in scale, complexity, and centrality to business. Alvin is a platform that aims to provide a low effort solution for data lineage capabilities focused on simplifying the work of data engineers.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

JULY 3, 2022

Summary The perennial challenge of data engineers is ensuring that information is integrated reliably. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. does exactly that.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Snowflake

JUNE 28, 2023

At Snowflake, we’re helping data scientists, data engineers, and application developers build faster and more efficiently in the Data Cloud. To make it even easier to process data with Snowpark Python UDFs and Stored Procedures, we have added support for Python 3.9 and unstructured data support, now in public preview.

Pipeline-centric 57

Pipeline-centric Programming Language Python Government 57

Knowledge Hut

DECEMBER 22, 2023

Not only academic streams but educational institutions, right from the elementary to the university level, are being transformed by AI systems. Agriculture AI is also largely benefitting the agricultural industry in terms of increased productivity. Education AI is set to transform the education sector in a big way.

Healthcare 98

Healthcare Manufacturing Education Retail 98

Data Engineering Podcast

OCTOBER 9, 2022

American Express has invested substantial time and effort in their Customer 360 product to achieve that understanding. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

JUNE 26, 2022

Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Master Data Management (MDM) is the process of building consensus around what the information actually means in the context of the business and then shaping the data to match those semantics.

Data Management 100

Data Management Management MongoDB Scala 100

Data Engineering Podcast

JUNE 8, 2020

Summary The landscape of data management and processing is rapidly changing and evolving. She discusses the work being done to address challenges in the areas of data quality, observability, discovery, and streaming. If you hand a book to a new data engineer, what wisdom would you add to it?

Data Management 100

Data Management Management Machine Learning Portfolio 100

Rockset

JANUARY 3, 2023

This blog compiles real-time data predictions from industry leaders so you know what’s coming in 2023. Confluent’s State of Data in Motion Report found that 97% of companies around the world are using streaming data, making it central to the data landscape.

Data Warehouse 52

Data Warehouse Education Database Data 52

Precisely

JANUARY 17, 2023

Technology helped to bridge the gap, as AI, machine learning, and data analytics drove smarter decisions, and automation paved the way for greater efficiency. Data integrity trends for 2023 promise to be an important year for all aspects of data management. Read The Corinium report to learn more.

Data Integration 97

Data Integration Data Cloud Computing Machine Learning 97

Cloudera

OCTOBER 13, 2021

Cloudera produced a series of ebooks — Production Machine Learning For Dummies , Apache NiFi For Dummies , and Apache Flink For Dummies (coming soon) — to help simplify even the most complex tech topics. What’s data analytics and why is everyone talking about it?”. Don’t know what data ingestion is? There’s no need to panic.

Machine Learning 77

Machine Learning Data Ingestion Algorithm Technology 77

Workfall

JUNE 12, 2023

In this dynamic realm of data engineering, a monumental challenge takes centre stage: efficiently managing the ever-changing tides of real-time data. Data, the lifeblood of organisations, holds the key to unlocking untapped potential and propelling businesses forward. In this blog, we will cover: What Is CDC and Its Benefits?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Knowledge Hut

APRIL 26, 2024

Also called data storage areas , they help users to understand the essential insights about the information they represent. Machine Learning without data sets will not exist because ML depends on data sets to bring out relevant insights and solve real-world problems. In the real world, data sets are huge.

Datasets 98

Datasets Machine Learning Deep Learning Finance 98