Data Engineering Weekly

JUNE 12, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today. Should you?

Data Engineering 57

Data Engineering Data Engineer Engineering AWS 57

Cloudera

JANUARY 15, 2021

The Impala team continues to innovate to ensure continuing performance leadership. Some examples of recent optimizations in Impala include: New multithreading model (see dedicated blog post ). Impala use of KRPC (see dedicated blog post ). But don’t just take our word for it. higher cost. Overview of Cloudera Data Warehouse.

Data Warehouse 97

Data Warehouse Cloud Consulting SQL 97

Knowledge Hut

DECEMBER 22, 2023

This blog will guide you with some best practices that you should adopt while preparing for your PMP exam. Are you still trying to figure out a way to start preparing for your PMP® Exam? This will surely help you to successfully clear your PMP certification exam. Similarly, every person has got a unique learning habit.

Certification 98

Certification Education Project Management 98

Knowledge Hut

JANUARY 31, 2024

This blog reflects on the SDP in ITIL. It can be difficult to design IT services that meet business needs and customer expectations. Managing competing priorities, costs, risks, and responsibilities necessitates a clear process and cross-functional collaboration. Exploring the principles, processes, and importance of ITIL service design.

Designing 52

Designing Process IT Portfolio 52

Knowledge Hut

FEBRUARY 28, 2024

The ability to track multiple key performance indicators (KPIs) and metrics is critical to business success today. It is not feasible to analyze and interpret large amounts of data manually. Data visualization has become vital to understanding the various trends present in the data – both visible, as well as hidden trends.

BI 52

BI Business Intelligence Data Data Science 52

Netflix Tech

DECEMBER 8, 2021

Snare also provides a space for our security engineers to track what’s going on, drill down into various findings, follow their investigation flow, and ensure that findings are reaching their proper resolution. Snare can be broken down into the following parts: Detection, Enrichment, Reporting & Management, and Remediation.

AWS 78

AWS Metadata Cloud Utilities 78

Zalando Engineering

SEPTEMBER 1, 2021

Long time readers of this blog will remember that back in 2019, we published a feature on the benefits of rotating engineers between teams. I am always more than happy to double down my investment in them if I know that it will be maximised. This empowerment and growth mindset is deeply woven into our fabric.

Engineering 40

Engineering Architecture Process Systems 40

phData: Data Engineering

JULY 12, 2022

Logical modeling drills down into each of these and starts to define the structure of the data elements along with the relationships between them. Within data engineering , one of the most frequent tasks is modeling data into data marts and data products.

SQL 52

SQL Data Engineering Data Engineer Data 52

Cloudera

SEPTEMBER 9, 2021

The CDP Operational Database ( COD ) builds on the foundation of existing operational database capabilities that were available with Apache HBase and/or Apache Phoenix in legacy CDH and HDP deployments. Cloudera Machine Learning or Cloudera Data Warehouse), to deliver fast data and analytics to downstream components. Cloud-Native Consumption Model.

Database 96

Database AWS Relational Database Cloud 96

Rockset

MAY 21, 2021

In this blog, we’ll walk through real-time analytics use cases and some of the continual challenges on the implementation front. Data-driven decision making: Users want to drill down into the data, identify an insight and take action. The Dominos’ Pizza Tracker is the quintessential example of real-time analytics.

Retail 40

Retail Algorithm Big Data Data Analytics 40

Knowledge Hut

OCTOBER 13, 2023

As per Microsoft, “A Power BI report is a multi-perspective view of a dataset, with visuals representing different findings and insights from that dataset. A report can have a single visual or pages full of visuals.” Creating reports and dashboards can be very daunting if you do not have the right approach. Let's get started.

BI 52

BI Business Analyst Datasets Raw Data 52

Rockset

FEBRUARY 16, 2024

In this blog, we’ll describe how Klarna implemented real-time anomaly detection at scale, halved the resolution time and saved millions of dollars using Rockset. With a number of payment options, including direct payments, pay after delivery and installment plans, Klarna provides shoppers flexibility in how they pay with zero interest.

Architecture 59

Architecture SQL Data Warehouse Database 59

Monte Carlo

NOVEMBER 8, 2023

Monte Carlo’s third annual IMPACT: the Data Observability Summit just wrapped and impact was certainly the name of the game. IMPACT 2023 brought together some of the best and brightest in data and AI for one jam-packed day of content, including insights from industry luminaries like Tristan Handy of dbt Labs and Nga Phan of Salesforce AI.

Kafka 64

Kafka Database High Quality Data Datasets 64

Cloudera

MAY 13, 2021

Since my last blog, What you need to know to begin your journey to CDP , we received many requests for a tool from Cloudera to analyze the workloads and help upgrade or migrate to Cloudera Data Platform (CDP). The good news is Cloudera has a tried and tested tool, Workload Manager (WM) that meets your needs. How WM helps the Move to CDP.

Management 72

Management BI Data Warehouse Cloud 72

LinkedIn Engineering

APRIL 4, 2023

This blog post will provide an overview of how we approached metrics selection and design, system architecture and key product features. Authors: Viktoras Truchanovicius and Selina Zhang At LinkedIn, developer productivity and happiness has always been a priority. Our research found over 50 such dashboards and spreadsheets across engineering.

MySQL 122

MySQL Datasets Software Engineer Software Engineering 122

ProjectPro

NOVEMBER 18, 2021

Data science has been a trending buzzword in recent times. With wide applications in various sectors like healthcare, education, retail, transportation, media, and banking -data science applications are at the core of pretty much every industry out there. million people around the globe. petabytes of data every hour!

Data Science 52

Data Science Project Food Pharmaceutical 52

DoorDash Engineering

NOVEMBER 28, 2023

The following sections drill down a bit deeper into our processes, particularly in regards to collaboration and the human aspects of our processes. Regularly releasing updates to the App Store and Play Store is more complex than might be expected, especially for teams at scale and even more so when there are multiple apps to ship.

Management 90

Management Coding Building Process 90

Pinterest Engineering

MAY 17, 2023

In this blog post, we will walk through: The Warden Anomaly Detection Platform. Because these types of events can be found almost anywhere, opportunities and applications for anomaly detection are vast. With the positive results we are seeing, we are planning to continue to expand our anomaly detection work and use cases. What is Warden?

Algorithm 69

Algorithm Engineering Machine Learning Project 69

Cloudera

AUGUST 10, 2022

In this blog post we’ll introduce CSP-CE, show how easy and quick it is to get started with it, and list a few interesting examples of what you can do with it. In this blog post we’ll introduce CSP-CE, show how easy and quick it is to get started with it, and list a few interesting examples of what you can do with it.

Process 91

Process Kafka PostgreSQL MySQL 91

Rockset

FEBRUARY 24, 2022

While this blog post won’t dive deeply into Kinesis’ capabilities, it’s worth quickly noting three: Kinesis Data Streams enable continuous capture of gigabytes of data per second from an enormous number of sources. With increased adoption has come a flood of options for aggregating and analyzing events.

AWS 52

AWS Amazon Web Services Kafka SQL 52

Cloudera

NOVEMBER 25, 2020

We sat in a private conference room and he told me the story of his pharma startup. A small group of researchers set out to solve the black-box enigma of certain kinds of vicious cancers. There are so many cancers, so their vision was to focus on especially heinous ones. Almost in the same second, a big pharma acquired them. . Challenges Ahead.

Data Warehouse 58

Data Warehouse Unstructured Data Medical Pharmaceutical 58

DoorDash Engineering

NOVEMBER 15, 2022

Traditionally, experimentation velocity tradeoffs are focused on minimizing the false positive rate (FPR), which heavily leans towards maintaining the status quo. For example, in Figure 1 we show the Type I (false positive) and Type II (false negative) error rates from typical experiments. and a power of 0.8.

Building 52

Building Process Coding Architecture 52

Cloudera

SEPTEMBER 17, 2020

For enterprise organizations, managing and operationalizing increasingly complex data across the business has presented a significant challenge for staying competitive in analytic and data science driven markets. Native Apache Airflow and robust APIs for orchestrating and automating job scheduling and delivering complex data pipelines anywhere.

Data Pipeline 104

Data Pipeline Data Engineering Data Engineer Engineering 104

Cloudera

OCTOBER 20, 2020

Today we are introducing a new series of blog posts that will take a look at recent enhancements to Apache Impala. Introduction. Many of these are performance improvements, such as the feature described below which will give anywhere from a 2x to 7x performance improvement by taking better advantage of all the CPU cores.

Utilities 104

Utilities Data Warehouse Cloud SQL 104

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. It turns out that Apache Impala scales down with data just as well as it scales up. The entire collection is available here. Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge?

Metadata 144

Metadata Coding SQL Database 144

Cloudera

SEPTEMBER 20, 2021

In this blog series, we will discuss each of these deployments and the deployment choices made along with how they impact reliability. In our Kafka Summit 2021 presentation, we took a brief overview of many different configurations that have been observed to date. Serial and Parallel Systems Reliability . R s = R 1 * R 2 … R n.

Kafka 118

Kafka Systems Utilities Bytes 118

DareData

AUGUST 12, 2021

One of the main reasons is that there are several pitfalls that are easy to fall into, which only become apparent down the road when your data increases in size or number of users. In fact, this not a trivial endeavor at all. At which point it becomes difficult to fix the problems without disrupting your company's workflow.

Transportation 52

Transportation Data Lake Data Warehouse Data 52

Cloudera

MAY 16, 2018

They are creative in their thinking and collaborative in their efforts (including with parties who may not know they are complicit) to obtain their objectives and avoid getting caught. This post describes how native visual analytics on a Cloudera Enterprise Data Hub is the keystone that supports a holistic trade surveillance program.

Data Lake 42

Data Lake Electronics Media Unstructured Data 42

Zalando Engineering

JUNE 4, 2015

I’d like to drill down a bit deeper on one topic of great importance to us: development speed. My interest in this subject was recently piqued after I read Spotify’s blog post about how their mobile team sped up their Xcode build time in the Edit-Build cycle. I started to wonder if Zalando might achieve similar results.

Building 52

Building Coding Project Engineering 52

Rockset

NOVEMBER 6, 2019

In this blog, we will examine the use of three different data backends for event data - Apache Druid , Elasticsearch and Rockset. In this blog, we will examine the use of three different data backends for event data - Apache Druid , Elasticsearch and Rockset. Supporting separation of data access (read/write) mechanisms like CQRS.

Kafka 40

Kafka Data Lake SQL Hadoop 40

Zalando Engineering

NOVEMBER 22, 2017

Using Apache and the Kafka Streams API with Scala on AWS for real-time fashion insights This piece was originally published on confluent.io The Fashion Web Zalando, Europe’s leading online fashion platform, cares deeply about fashion. Our mission statement is to, “Reimagine fashion for the good of all”.

Kafka 40

Kafka Scala Hadoop Algorithm 40

Rockset

OCTOBER 1, 2019

In this blog post, we will explore a tool we built internally that gives us visibility into Kubernetes events, an excellent source of information about the state of the system, which we find useful in troubleshooting the system and understanding its long-term health. and the cause. kubectl describe pods jobworker-c5dc75db8-7m5ln.

SQL 40

SQL Systems Metadata Accessible 40

Confluent

FEBRUARY 13, 2019

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. The paradigm shift to the event streaming architecture often leads to a lightbulb moment of inspiration and clarity.

Architecture 90

Architecture Programming Kafka Database-centric 90

ProjectPro

AUGUST 24, 2021

And, out of these professions, this blog will discuss the data engineering job role. For beginners or peeps who are utterly new to the data industry, Data Scientist is likely to be the first job title they come across, and the perks of being one usually make them go crazy. Nevertheless, that is not the only job in the data world.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

phData: Data Engineering

JANUARY 3, 2022

DataOps: What Is It, Core Principles, and Tools For Implementation Nick Goble January 3, 2022 When building a successful company, it’s critical to have a strategy around how you build and scale your business from a technology and data perspective. Want to Save This eBook for Later? No problem! Table of Contents How Impactful is Your Data?

IT 52

IT AWS Software Engineer Software Engineering 52

Confluent

SEPTEMBER 6, 2019

How many of us drill down to see how we got there? Today, their online order space is broken down into five different services written in a variety of programming languages. First, what is event sourcing? Here’s an example. Consider your bank account: viewing it online, the first thing you notice is often the current balance.

Kafka 22

Kafka MySQL Banking Java 22

Cloudera

MARCH 27, 2023

We doubled-down on data governance in 2021, and in 18 short months we’re flying high, in part because we are standardizing our enterprise datasets. We doubled-down on data governance in 2021, and in 18 short months we’re flying high, in part because we are standardizing our enterprise datasets.

Finance 77

Finance Datasets Data Governance Government 77