Data Engineering Weekly

MAY 12, 2024

link] Gradient Flow: Learning from the Past - Comparing the Hype Cycles of Big Data and GenAI The blogs compare the hype cycle of Big Data with Gen AI. The blog narrates the initial hype of Big Data, followed by talent shortages and lead time to build production applications. It is one of the tools I wish Uber could open source.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Weekly

FEBRUARY 4, 2024

link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. The blog is an excellent reminder no matter how standardized your tools are, the migration from one DB to another is always a nightmare. How are you analyzing the cost of your infrastructure?

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Pinterest Engineering

FEBRUARY 23, 2023

In 2021, a group of client networking enthusiasts at Pinterest started thinking about adopting HTTP/3 (fka QUIC) for Pinterest, from traffic/CDN to client apps. Adopting HTTP/3 at Pinterest Strategy Safety and metrics came first. Apple’s HTTP/3 adoption rate continues to improve year over year. image, video).

Bytes 116

Bytes Media Software Engineer Software Engineering 116

Data Engineering Weekly

DECEMBER 3, 2023

Github writes an excellent blog to capture the current state of the LLM integration architecture. The blog is an excellent read to understand late-arriving data, backfilling, and incremental processing complications. I like the 3G model with Guardrails, Guidelines & Gadget, which I’m sure I will use more often :-).

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Data Engineering Podcast

JULY 9, 2023

What are your predictions for the future direction/adoption of ECM or other modeling techniques? What are your predictions for the future direction/adoption of ECM or other modeling techniques? What are the most interesting, unexpected, or challenging lessons that you have learned while working on ECM? When is ECM the wrong choice?

Database-centric 130

Database-centric Machine Learning SQL Data Engineering 130

Cloudera

DECEMBER 1, 2023

These government-led efforts have had a profound impact on the development and adoption of AI solutions in the public sector, paving the way for a future where data-driven decision-making and automation are the norm. The post Building Trust in Public Sector AI Starts with Trusting Your Data appeared first on Cloudera Blog.

Building 111

Building Government Transportation Data Governance 111

Pinterest Engineering

MARCH 20, 2024

Adopted by Pinterest multiple user facing surfaces, Ads, and Board. In this blog, we touch on: Technical design Key innovations Offline results Online results Technical Design Data Most Pins are associated with a landing page. Impact on engagement and monetization: Improve both engagement (e.g.

Machine Learning 73

Machine Learning Engineering Software Engineer Software Engineering 73

DoorDash Engineering

NOVEMBER 28, 2023

It is amusing for a human being to write an article about artificial intelligence in a time when AI systems, powered by machine learning (ML), are generating their own blog posts. As we saw users readily adopt this simple automation, we realized that simple tools can help our customers increase model development velocity over the long term.

Machine Learning 90

Machine Learning Pipeline-centric Data Science Designing 90

Cloudera

JUNE 2, 2022

Since 2015, the Cloudera DataFlow team has been helping the largest enterprise organizations in the world adopt Apache NiFi as their enterprise standard data movement tool. This blog aims to answer two questions: What is a universal data distribution service? Why does every organization need it when using a modern data stack?

Systems 105

Systems Data Lake Google Cloud Data Collection 105

Cloudera

DECEMBER 13, 2023

To realize this cohesive data vision, LGIM adopted Cloudera Data Platform (CDP) Public Cloud. Moreover, widespread adoption of the platform allows our employees to unlock value from our data assets,” comments Derrick Hastie, CTO at LGIM. The post Cloudera Customer Story appeared first on Cloudera Blog.

Pipeline-centric 91

Pipeline-centric Professional Services Datasets BI 91

Data Engineering Weekly

MARCH 3, 2024

This approach led to a successful expansion of Copilot access across the engineering team, resulting in a significant increase in productivity and adoption, demonstrating a commitment to enhancing developer experience while maintaining safety and security standards. link] Nvidia: What Is Sovereign AI?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Weekly

MARCH 19, 2023

The author defines Data Product as the combination of Datasets Domain Access It is an exciting time for the data industry as we are increasingly talking about philosophies to adopt data in an organization than technology complexities such as Hadoop, Spark, etc., Much of it focuses on model training, evaluation, and scoring.

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Uber Engineering

JULY 23, 2020

While only a few years ago, many people readily adopted microservice architectures due to the numerous benefits they provide such as … The post Introducing Domain-Oriented Microservice Architecture appeared first on Uber Engineering Blog.

Architecture 145

Architecture Engineering 145

Data Engineering Weekly

JUNE 8, 2023

I’m sure it will evolve as the adoption increases. A couple of thing stands out for me in the blog. The Event store is moving from S3/ parquet storage to DeltaLake storage—a sign of LakeHouse format adoption across the board. Instacart adoption of Databricks ecosystem along with Snowflake.

Architecture 57

Architecture SQL Data Pipeline Data Engineering 57

Data Engineering Weekly

SEPTEMBER 24, 2023

The blog talks about the limitations of rule engines and how LLM can enrich additional context to make the rule engine more effective. The blog specifically focuses on the following areas of LLM implementations. The blog is an excellent read about the “Scale First, Learn Last” problem in the data processing world.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Cloudera

MARCH 29, 2023

Matt (now Panther) turned out to be the best, most amazing companion I could ask for, and so I adopted him a few months later. Recently my other two fosters Maxie (a 7 year old Chi/terrier mix), and Bailey (a 7 month old Chi/pom puppy) were adopted. They are super easy going, very loving and it’s been a joy fostering them.

Medical 79

Medical Food Media Process 79

Yelp Engineering

JANUARY 29, 2023

The same principles of these systems can be adopted to filter out malformed data from datastores. This blog post deep dives into how we rebuilt one of our Cassandra(C*) clusters by removing malformed data using Yelp’s Data Pipeline.

Data Pipeline 75

Data Pipeline NoSQL Manufacturing Data 75

Edureka

FEBRUARY 13, 2024

This blog is a beginner’s guide to everything you need to know about PRINCE2 project management methodology! This blog guides you through these subsections: What Is PRINCE2? PRINCE2 (PRojects IN Controlled Environments) is one such framework that project managers use across the globe.

Project 52

Project Management Telecommunication Certification 52

Data Engineering Weekly

AUGUST 20, 2023

The blog narrates the potential possibilities of LLM’s impact in each stage of the data warehouse. link] Sponsored: Great Data Debate–The State of Data Mesh Since 2019, the data mesh has woven itself into every blog post, event presentation, and webinar. But 4 years later, in 2023 — where has the data mesh gotten us?

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Cloudera

MAY 13, 2021

While the whitepaper lays out and compares different engines, the purpose of this blog post is to define what factors the whitepaper looks at and why each one is important. Consideration #4: Enterprise adoption. The ease of enterprise adoption is our first big operational factor. Consideration #5: Enterprise operations.

Process 103

Process Engineering Kafka Architecture 103

Pinterest Engineering

NOVEMBER 7, 2023

For these reasons, and others detailed in our original PubSub Client blog post , our team has decided to invest in building, productionalizing, and most recently open-sourcing PubSub Client (PSC). years since our previous blog post, PSC has been battle-tested at large scale in Pinterest with notably positive feedback and results.

Kafka 99

Kafka Java Software Engineer Software Engineering 99

LinkedIn Engineering

OCTOBER 19, 2023

The LinkedIn engineering team has open-sourced over 75 projects across multiple categories, with several gaining widespread adoption and becoming part of the Apache Software Foundation. By 2019, Apache Beam pipelines were powering several critical use cases, and the programming model and framework saw extensive adoption across LinkedIn teams.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Knowledge Hut

NOVEMBER 16, 2023

One of the biggest challenges faced by corporations today when it comes to cloud adoption is the lack of cloud expertise. Thousands of businesses all across the world have adopted it. Nowadays, multi-cloud strategies are adopted by enterprises. You can even read AWS blogs or avail online video courses.

AWS 98

AWS Certification IT Amazon Web Services 98

Databand.ai

JULY 17, 2023

By adopting a set of best practices inspired by Agile methodologies, DevOps principles, and statistical process control techniques, DataOps helps organizations deliver high-quality data insights more efficiently. In some cases, organizations may benefit from adopting elements from both methodologies.

Data Pipeline 52

Data Pipeline Machine Learning High Quality Data BI 52

Cloudera

OCTOBER 30, 2023

As cloud security is not a big concern anymore and it is more mature (pointed out by 79 percent of all enterprises as a challenge), the market has seen an increase in the adoption of cloud services in the last few years. It is not strange to hear and know about companies adopting a cloud-first strategy in the short term.

Cloud 80

Cloud Finance Cloud Computing Government 80

Cloudera

SEPTEMBER 26, 2023

The post IBM Technology Chooses Cloudera as its Preferred Partner for Addressing Real Time Data Movement Using Kafka appeared first on Cloudera Blog. Learn more about how you can benefit from a well-supported data management platform and ecosystem of products, services and support by visiting the IBM and Cloudera partnership page.

Kafka 95

Kafka Technology IT Government 95

Cloudera

MAY 8, 2023

Alternatively, you could adopt Cloudera Observability and get tons of visibility immediately! The post Visibility and Transparency appeared first on Cloudera Blog.

Professional Services 80

Professional Services Data Architecture Data Engineering Data Engineer 80

Uber Engineering

JANUARY 19, 2023

In this blog post, we dive into the technical details behind solutions that helped us scale the adoption of Kerberos for authentication for the entire Data analytics infrastructure. Uber heavily relies on open source technologies for the Data analytics stack.

Data Analytics 59

Data Analytics Technology Data 59

Data Engineering Weekly

MAY 16, 2023

I won’t bore you with the importance of data quality in the blog. Let’s dive into both architectural patterns and see how to adopt them in real-time and batch data processing. One-Phase WAP The One-Phase WAP adopts a Zero-Copy data contract validation approach. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Yelp Engineering

APRIL 23, 2023

Since then, adoption of our new MVI architecture library has risen and we’ve seen some great performance and scalability wins. In this blog post, we’ll cover why we switched to MVI in the first place, how we managed to get performant screens by default, and our take on unit testing MVI.

Architecture 59

Architecture Management 59

Data Engineering Weekly

MARCH 12, 2023

We take three articles and do an in-depth analysis, and we hope to bring the author of the blogs to discuss more. The blog narrates the architecture to implement an outbox pattern with Debezium, the usage of the outbox router in Debezium, and lessons learned. We call it Data Engineering Weekly Radio.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Cloudera

JANUARY 30, 2024

Yet many manufacturing executives say they are challenged in adopting new technologies, including AI for new use cases. The lack of universal industrial data has been one of the major obstacles slowing the adoption of AI among mainstream manufacturers. The post Achieving Trusted AI in Manufacturing appeared first on Cloudera Blog.

Manufacturing 82

Manufacturing Data Lake Data Science Cloud 82

Lyft Engineering

APRIL 24, 2024

See our previous blog post for more on the motivations behind TLC. Some recent examples can be found in our tech blog: reinforcement learning and structural causal modeling. We want to foster a culture of continuous learning by providing resources and forums that are easily accessible to everyone.

Data Science 66

Data Science Building Computer Science Software Engineer 66

Snowflake

DECEMBER 4, 2023

Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Snowflake’s support for Iceberg Tables is now in public preview, helping customers build and integrate Snowflake into their lake architecture.

Building 96

Building Metadata Cloud Storage AWS 96

Cloudera

NOVEMBER 25, 2021

Advances in the performance and capability of Artificial Intelligence (AI) algorithms has led to a significant increase in adoption in recent years. Furthermore, AI adoption is becoming increasingly widespread and not just concentrated within a small number of organisations. in 2021 to USD $327 billion. Find out more.

Government 99

Government Insurance Algorithm Deep Learning 99

Cloudera

JUNE 27, 2023

Adopting an MDA is more efficient and cost effective than legacy data architectures which are siloed, poorly governed, with vast redundant data sets and data processes that are expensive and expose the company to risk. The post Five Ways A Modern Data Architecture Can Reduce Costs in Telco appeared first on Cloudera Blog.

Data Architecture 64

Data Architecture Architecture Data Governance Government 64

Precisely

FEBRUARY 12, 2024

For instance, telcos are early adopters of location intelligence – spatial analytics has been helping telecommunications firms by adding rich location-based context to their existing data sets for years. Despite that fact, valuable data often remains locked up in various silos across the organization.

Data Governance 64

Data Governance Government Telecommunication Machine Learning 64