Knowledge Hut

JANUARY 18, 2024

CISSP stands for Certified Information Systems Security Professional, and it is a certification in cyber security. This professional certification is developed and offered by (ISC)2, also known as International Information Systems Security Certification Consortium. This blog will help you understand the question “what is CISSP?

Systems 52

Systems Certification Consulting Education 52

Cloudera

JUNE 2, 2022

This blog aims to answer two questions: What is a universal data distribution service? In a recent customer workshop with a large retail data science media company, one of the attendees, an engineering leader, made the following observation: “Everytime I go to your competitor website, they only care about their system.

Systems 101

Systems Data Lake Google Cloud Data Collection 101

DataKitchen

DECEMBER 9, 2022

This can include the use of tools for data preparation, model training, and deployment, as well as technologies for monitoring and managing data-related systems and processes. One of the key benefits of DataOps observability is the ability to improve collaboration and communication across teams and systems.

Machine Learning 52

Machine Learning Data Preparation Government Data Analytics 52

databricks

SEPTEMBER 12, 2023

As the lakehouse becomes increasingly mission-critical to data-forward organizations, so too grows the risk that unexpected events, outages, and security incidents may derail.

Systems 86

Systems Data 86

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Rockset

SEPTEMBER 2, 2022

I recently had the good fortune to host a small-group discussion on personalization and recommendation systems with two technical experts with years of experience at FAANG and other web-scale companies. Garg also blogs regularly on real-time data and recommendation systems – read and subscribe here. That’s not machine learning.

Systems 52

Systems Building Machine Learning NoSQL 52

Cloudera

MARCH 6, 2024

As we enter into a new month, the Cloudera team is getting ready to head off to the Gartner Data & Analytics Summit in Orlando, Florida for one of the most important events of the year for Chief Data Analytics Officers (CDAOs) and the field of data and analytics. Add a Human To The Loop: An Introduction to RLHF & DPO.

Data Analytics 106

Data Analytics Data Technology Data Engineering 106

Snowflake

APRIL 29, 2024

DORA: Building a More Secure Financial System DORA, enacted in January 2023, moves beyond reactive measures, requiring FEs and their service providers to proactively identify vulnerabilities, prevent disruptions and plan for swift recovery from incidents.

Transportation 79

Transportation Data Governance Government Consulting 79

Knowledge Hut

JUNE 26, 2023

If you really want to build some cool backend projects, then this blog post is for you to provide a comprehensive guide for your career goals and guidance. Top 4 Backend Project Ideas - Beginner and Final Year Students Task Management System In this project you develop a backend system to manage tasks and projects.

Portfolio 98

Portfolio Project Food Media 98

Picnic Engineering

FEBRUARY 8, 2023

Delivering the right events at low latency and with a high volume is critical to Picnic’s system architecture. In our previous blog, Dima Kalashnikov explained how we configure our Internal services pipeline in the Analytics Platform. How did we implement the Saga pattern?

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! Late-arriving data is essentially delayed data due to system retries, network delays, batch processing schedules, system outages, delayed upstream workflows, or reconciliation in source systems.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Rockset

JULY 6, 2022

This is the fifth post in a series by Rockset's CTO and Co-founder Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. We'll be publishing more posts in the series in the near future, so subscribe to our blog so you don't miss them! Most notable are real-time event streams.

NoSQL 52

NoSQL SQL Systems PostgreSQL 52

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Utilities 135

Utilities Systems Architecture Coding 135

Snowflake

APRIL 9, 2024

In this blog, we’ll take a look at how Snowflake is using data sharing to benefit our SDR teams and marketing data analysts. Leads enter the system and get a score, the score is pushed to Salesforce and Marketo and used to inform predictions, and the SDR team can pick up that information to use in their daily applications.

BI 72

BI Data Ingestion Data Aggregated Data 72

Workfall

JUNE 12, 2023

In this blog, we will cover: What Is CDC and Its Benefits? It bridges gaps in data ecosystems, ensuring consistency and synchronisation across systems. Moreover, it facilitates the implementation of microservices architectures and event-driven systems, automating reactions to data changes without manual intervention.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Knowledge Hut

JANUARY 16, 2024

” ― Denis Waitley What are SAFe Agile Events (or) Ceremonies? The key link between these levels is the SAFe-specific events which help with smooth value delivery facilitation. The events help with alignment across teams, ARTs, etc thus helping in managing risk by providing a level-based cadence and synchronization.

Portfolio 97

Portfolio Business Analyst Programming Certification 97

Cloudera

FEBRUARY 1, 2024

In this blog, we’ll cover the highlights of our recently published Day 2 Operations Guide and why it matters to enterprises. Day 2 operations are akin to the housekeeping of a software system — vital for maintaining its health and stability.

Cloud 82

Cloud Architecture Utilities Designing 82

DoorDash Engineering

NOVEMBER 7, 2023

To do this, we built a large stateful Flink job to track user sessions using mobile/web client events. Deciding whether to trigger in the frontend or backend Backend events typically are generated by internal services, while frontend events are created by the mobile app or web.

Designing 95

Designing Coding Process Management 95

Workfall

MARCH 7, 2023

Reading Time: 6 minutes In order to create a Yew web application, one must create mechanisms to allow end users to interact with the system and provide data via online forms. For paper forms, this might involve manually entering the information into a computer system, while online forms typically capture the data automatically.

PostgreSQL 79

PostgreSQL Database AWS Data Collection 79

Knowledge Hut

APRIL 25, 2024

In this blog post, we will discuss such technologies. If you pursue the MSc big data technologies course, you will be able to specialize in topics such as Big Data Analytics, Business Analytics, Machine Learning, Hadoop and Spark technologies, Cloud Systems etc. Spark is a fast and general-purpose cluster computing system.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. I won’t bore you with the importance of data quality in the blog. The system can fail or apply a circuit breaker in the pipeline if the data contract fails.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Weekly

JUNE 8, 2023

However, as with any initial system iteration, the dbt model contract implementation has pros and cons. A couple of thing stands out for me in the blog. The Event store is moving from S3/ parquet storage to DeltaLake storage—a sign of LakeHouse format adoption across the board. Mine was over six years.

Architecture 57

Architecture SQL Data Pipeline Data Engineering 57

Zalando Engineering

MARCH 1, 2021

In order to provide an excellent customer experience, we must ensure our systems can technically handle high traffic events such as Cyber Week or other sales campaigns. Checkout and payments related systems are particularly important during sales events. We considered multiple options such as Locust, Vegeta and JMeter.

Systems 40

Systems Building Java Algorithm 40

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 3) DataOPS at AstraZeneca The AstraZeneca team talks about data ops best practices internally established and what worked and what didn’t work!!!

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Netflix Tech

APRIL 27, 2023

Engineers want their alerting system to be realtime, reliable, and actionable. A few years ago, we were paged by our SRE team due to our Metrics Alerting System falling behind — critical application health alerts reached engineers 45 minutes late! The internals here are outside the scope of this blog post.

Database 115

Database Architecture Consulting Systems 115

Cloudera

DECEMBER 19, 2023

Some of the primary operational problems highlighted at the PCN Government Innovation event include: Civil Government : A major challenge facing the civil government is the inefficient and cumbersome procurement process. Learn more about how Cloudera can help you on your AI journey. Trust your data. Trust your enterprise AI.

Government 87

Government Education Unstructured Data Datasets 87

Knowledge Hut

MARCH 19, 2024

I will explore the top 10 AWS applications and their use cases in this blog. AWS Lambda AWS Lambda is a serverless computing service that enables developers to run code in response to events without needing to work with servers. That shows how much AWS has to offer, and you must know about it if you’re a cloud computing enthusiast.

AWS 52

AWS Cloud Computing Amazon Web Services Relational Database 52

Data Engineering Weekly

APRIL 2, 2023

Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real-time with your own JavaScript or Python code. Meta: Presto - A Decade of SQL Analytics at Meta Presto and Kafka are the two systems that greatly impacted data infrastructure in the last decade.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Cloudera

JUNE 5, 2023

This blog is to congratulate our winner and review the top submissions. Cloudera VP of Engineering and NiFi founder Joe Witt, joined by principal committers Mark Payne and Matt Gillman, addressed the global community via a virtual event dubbed “ Meet the Committers.” On the verge of the release of NiFi 2.0,

Google Cloud 75

Google Cloud Cloud Storage Data Lake Data Pipeline 75

Data Engineering Weekly

FEBRUARY 4, 2024

link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. The challenge with the OLAP engine is that it is an additional system to maintain, and indexing everything is too costly. How are you analyzing the cost of your infrastructure?

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Lyft Engineering

NOVEMBER 29, 2023

Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Netflix Tech

AUGUST 30, 2021

In this blog post, we will focus on the latter feature set. In the Device Management Platform, this is achieved by having device updates be event-sourced through the control plane to the cloud so that NTS will always have the most up-to-date information about the devices available for testing.

Management 85

Management Kafka Transportation Cloud 85

Cloudera

OCTOBER 5, 2023

In a previous blog post , we explored the power of Cloudera Observability in providing high-level actionable insights and summaries for Hive service users. In this blog, we will delve deeper into the insight Cloudera Observability brings to queries executed on Hive. How do I detect problems due to skew? What information should I use?

SQL 75

SQL Systems Database Engineering 75

Workfall

APRIL 18, 2023

In this blog, we will discuss: Polling WebSockets GraphQL Subscriptions Apollo Client Features Community Integrations Why Apollo Client? Websockets are full-duplex or bi-directional meaning that information/data flow is from server to client and vice-versa as soon as the data is received or an event is triggered.

Transportation 66

Transportation BI Java Coding 66

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Scott Logic

JANUARY 23, 2024

The format makes for a fascinating event covering all kinds of diverse topics and ideas sourced from the attendees themselves. Sustainability Those of you who are regular visitors to the Scott Logic blog know that sustainability is something we spend a lot of time working on. Maybe that’s a blog for the future.

Government 52

Government Finance Consulting Technology 52

Data Engineering Weekly

APRIL 7, 2024

A key highlight for me, I spoke to multiple data people stuck in legacy systems and still inching their way to the cloud. link] Hussein Jundi: Data Engineering - Architectures & Strategies for Handling Sensitive Data The rapid adoption of AI brings challenges for data engineering to design systems to handling sensitive data.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70