Data Engineering Weekly

MARCH 10, 2024

Editor’s Note: Chennai Meetup Wrap-Up & Preparation work started for DEWCon I am so grateful for the enthusiastic participants who made our Chennai Data Heroes- Community for Data Folks meetup vibrant! Big thanks to our insightful speakers, Hareshkumar Selvakumar - Talks about his work on Data Products for PayPal.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Cloudera

APRIL 20, 2021

What is Streaming Analytics? Streaming Analytics is a type of data analysis that processes data streams for real-time analytics. It continuously processes data from multiple streams and performs simple calculations to complex event processing for delivering sophisticated use cases.

Hospitality 93

Hospitality Retail Kafka Data Ingestion 93

Cloudera

SEPTEMBER 15, 2021

That’s where you find the ability to empower IoT devices to respond to events in real time by capturing and analyzing the relevant data. Edge computing relies on squeezing the power and functionality of a data center into a micro site as close to data sources as possible to enable real-time tasks.

Medical 117

Medical Retail Datasets Algorithm 117

Cloudera

JUNE 7, 2021

Cloudera has been named as a Strong Performer in the Forrester Wave for Streaming Analytics, Q2 2021. We are proud to have been named as one of “ The 14 providers that matter most ” in streaming analytics. And we have been saying the exact same thing all along – Data decays! It’s too late.

Kafka 101

Kafka Data Ingestion Architecture Cloud 101

Cloudera

JUNE 7, 2022

We live in a hybrid data world. In the past decade, the amount of structured data created, captured, copied, and consumed globally has grown from less than 1 ZB in 2011 to nearly 14 ZB in 2020. Impressive, but dwarfed by the amount of unstructured data, cloud data, and machine data – another 50 ZB.

IT 108

IT Unstructured Data Data Architecture Government 108

Cloudera

JANUARY 20, 2021

Most of what is written though has to do with the enabling technology platforms (cloud or edge or point solutions like data warehouses) or use cases that are driving these benefits (predictive analytics applied to preventive maintenance, financial institution’s fraud detection, or predictive health monitoring as examples) not the underlying data.

Manufacturing 105

Manufacturing Data Warehouse Kafka Retail 105

Cloudera

AUGUST 4, 2021

Today, preventative maintenance, where actions are performed regardless of actual condition, is giving way to Predictive, or Condition-Based, maintenance, where actions are based on actual, real-time insights into operating conditions. The key is active and ongoing monitoring of prognostic health data.

Unstructured Data 98

Unstructured Data Data Ingestion Government Machine Learning 98

Cloudera

OCTOBER 11, 2022

I recommend you read the entire piece, but to me the key takeaway – AI at scale isn’t magic, it’s data – is reminiscent of the 1992 presidential election, when political consultant James Carville succinctly summarized the key to winning – “it’s the economy”. It’s because of data. . Data science needs analytics.

Data Science 90

Data Science Aggregated Data Data Consulting 90

Cloudera

APRIL 9, 2021

This is part 4 in this blog series. This blog series follows the manufacturing and operations data lifecycle stages of an electric car manufacturer – typically experienced in large, data-driven manufacturing companies. The second blog dealt with creating and managing Data Enrichment pipelines.

Machine Learning 98

Machine Learning Manufacturing Data Collection Data Science 98

Cloudera

JUNE 22, 2023

These dis-integrated resources are “data platforms” in name only: in addition to their high maintenance costs, their lack of interoperability with other critical systems makes it difficult to respond to business change. The top-line benefits of a hybrid data platform include: Cost efficiency. Simplified compliance. Flexibility.

Data Cleanse 66

Data Cleanse Data Governance Unstructured Data Cloud Storage 66

ProjectPro

FEBRUARY 8, 2023

Did you know thousands of businesses, including over 80% of the Fortune 100, use Apache Kafka to modernize their data strategies? Apache Kafka is the most widely used open-source stream-processing solution for gathering, processing, storing, and analyzing large amounts of data. What is Apache Kafka Used For?

Kafka 52

Kafka Banking Medical Healthcare 52

Rockset

AUGUST 2, 2023

With 90% of trade being transported via sea , this data is crucial to keeping the global supply chain on track but can be difficult to disentangle and take action on. Windward fills this niche by providing actionable intelligence with real-time ETA tracking, carrier performance insights, risk monitoring and mitigation and more.

Database-centric 52

Database-centric PostgreSQL Transportation Insurance 52

Cloudera

JULY 29, 2021

Securely protecting healthcare data is critical for your organization’s success, whether data is ingested, streamed and stored in a data platform that runs in the public, private or hybrid cloud. Security and governance in a hybrid environment. Public, private, hybrid or on-premise data management platform.

Healthcare 100

Healthcare Unstructured Data Government Machine Learning 100

Cloudera

OCTOBER 26, 2020

Over time, we learn to set priorities, make a decision for one thing over the other, and compromise. Just like when it comes to data access in business. Enabling data access for end-users so they can drive insight and business value is a typical area of compromise between IT and users. Quickly onboard data.

Data Security 117

Data Security IT Unstructured Data Government 117

Cloudera

AUGUST 11, 2022

But information broadly, and the management of data specifically, is still “the” critical factor for situational awareness, streamlined operations, and a host of other use cases across today’s tech-driven battlefields. . Universal Data Distribution Solves DoD Data Transport Challenges. hardware in the skies, sea, and land.

Transportation 89

Transportation Data Ingestion Architecture Data 89

Cloudera

FEBRUARY 4, 2021

Financial services institutions need the ability to analyze and act on massive volumes of data from diverse sources in order to monitor, model, and manage risk across the enterprise. They need a comprehensive data and analytics platform to model risk exposures on-demand. End-to-end Data Lifecycle. Shared Data Experience (SDX).

Technology 88

Technology Data Management Management Insurance 88

Cloudera

MARCH 29, 2022

State and local governments generate and store enormous amounts of data essential to their ability to deliver citizen services. But how can they capitalize on all of their data to become engines of growth and innovation, empowering and enhancing their ability to provide services and better serve their communities?

Government 94

Government Cloud Cloud Computing Data Science 94

Cloudera

FEBRUARY 16, 2021

In the previous blog post , we looked at some of the application development concepts for the Cloudera Operational Database (COD). In this blog post, we’ll see how you can use other CDP services with COD. Integrated across the Enterprise Data Lifecycle . Cloudera Data Engineering to ingest bulk data and data from mainframes.

Database 83

Database Machine Learning Data Lake Kafka 83

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Databand.ai

MAY 30, 2023

6 Pillars of Data Quality and How to Improve Your Data Eric Jones May 30, 2023 What Is Data Quality? Data quality refers to the degree of accuracy, consistency, completeness, reliability, and relevance of the data collected, stored, and used within an organization or a specific context.

Data Cleanse 85

Data Cleanse Datasets Data Governance Data Validation 85

phData: Data Engineering

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Snowflake

MARCH 21, 2023

In February, Snowflake launched new features around streaming data ingestion and data governance and improved SQL experience and performance, with enhancements to Search Optimization Service and more. Read on to learn about everything new announced in February. Learn more here. Learn more here. Learn more here.

Retail 64

Retail Healthcare Data Ingestion Consulting 64

Cloudera

SEPTEMBER 24, 2021

One of the most substantial big data workloads over the past fifteen years has been in the domain of telecom network analytics. The Dawn of Telco Big Data: 2007-2012. To do so would – at the time – have been prohibitively expensive, and no one was really that interested anyway. The Explosion in Telco Big Data: 2012-2017.

Data Architect 91

Data Architect Government NoSQL Big Data 91

dbt Developer Hub

AUGUST 2, 2023

Jetblue has outgrown lambda views due to performance constraints (a view can only be so performant) and we are at another milestone in dbt’s journey to support streaming. Due to those features, they are also more aligned with what other data platforms are calling Materialized Views. Introduction ​ The year was 2020.

Datasets 52

Datasets Kafka SQL Cloud 52

Cloudera

JANUARY 5, 2023

In the past year, businesses who doubled down on digital transformation during the pandemic saw their efforts coming to fruition in the form of cost savings and more streamlined data management. 1- Treating data as a strategic business asset . 2- Operationalizing adaptive AI systems for quicker business decision-making.

Banking 52

Banking Machine Learning Insurance Data Architecture 52

Cloudera

OCTOBER 13, 2020

On September 24, 2019, Cloudera launched CDP Public Cloud (CDP-PC) as the first step in delivering the industry’s first Enterprise Data Cloud. CDP Machine Learning: a kubernetes-based service that allows data scientists to deploy collaborative workspaces with secure, self-service access to enterprise data. That Was Then.

Cloud 92

Cloud Data Warehouse AWS NoSQL 92

Cloudera

AUGUST 21, 2020

Live data-streaming offers businesses exciting new opportunities to transform the way they operate, leveraging real-time insights to drive better decision making and enhance operational efficiency. To start off, what are the advantages of a forward-looking data-in-motion strategy?

Banking 59

Banking Kafka Cloud Storage Government 59

Knowledge Hut

JULY 26, 2023

Veracity meaning in big data is the degree of accuracy and trustworthiness of data, which plays a pivotal role in deriving meaningful insights and making informed decisions. This blog will delve into the importance of veracity in Big Data, exploring why accuracy matters and how it impacts decision-making processes.

Big Data 52

Big Data Data Cleanse Retail Healthcare 52

Cloudera

FEBRUARY 8, 2021

Within the vehicle, current electronics and wiring infrastructures were not designed for this complex data wrangling capability. In addition, moving outside the vehicle, existing fragmented approaches for data management associated with the machine learning lifecycle are limiting the ability to deploy new use cases at scale.

Manufacturing 107

Manufacturing Machine Learning Data Ingestion Electronics 107

Rockset

JULY 28, 2022

As capable as it is, there are still instances where MongoDB alone can't satisfy all of the requirements for an application, so getting a copy of the data into another platform via a change data capture (CDC) solution is required. Generally, this means your applications would submit new data to a queue.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Data Engineering Weekly

JUNE 12, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. The architecture resembles any typical modern data catalog with search and integration features.

Data Engineering 57

Data Engineering Data Engineer Engineering AWS 57

Cloudera

JULY 12, 2021

The 2021 Data Impact Award (DIA) submissions are starting to stream in, and we know many of you are contemplating your entries – which we are excited to see. Data and fanning the flames of business transformation . But the most impactful uses of data are those that seek to have a broader positive impact on society.”.

Healthcare 106

Healthcare Consulting Media Big Data 106

Cloudera

JANUARY 27, 2022

Insurers are increasingly adopting data from smart devices and related technologies to support and service their customers better. I have been researching more about how we can use the new data from those devices to design more innovative insurance products while being aware that these should all be contingent upon customer opt-in.

Insurance 82

Insurance Data Government Data Governance 82

Cloudera

FEBRUARY 17, 2021

Streaming data (or data-in-motion) is one such technology space that thrived during these times. Everyone was looking for real-time insights by analyzing what is going on currently within their businesses and taking corrective action pro-actively.

Kafka 60

Kafka Food Manufacturing Healthcare 60

Datakin

AUGUST 5, 2021

Blog A real-time approach to data lineage Written by Ross Turk on August 5, 2021 A data ecosystem that spans multiple pipelines, teams, and platforms can be overwhelming. Some of them look through your source code, parsing SQL queries to determine which data sources and tables they SELECT from and INSERT into.

Metadata 52

Metadata Relational Database Datasets SQL 52

Databand.ai

DECEMBER 13, 2022

The Top 25 Data Engineering Influencers and Content Creators on LinkedIn Ryan Yackel 2022-12-13 10:23:19 Interested in data engineering? LinkedIn is full of influencers sharing new ideas and sparking conversations on all kinds of topics, and data engineering is no exception. You’ve come to the right place. Happy following!

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! While they go about it - enter big data data engineer tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52