Knowledge Hut

JULY 6, 2023

This blog walks you through the top Azure Monitoring and Development that every SRE and DevOps engineer must know. Azure Monitor Tools are essential to understand the alert's type, nature, and severity. Using time-based graphs and dashboards in these alert-based tools, it becomes easy to understand the root cause of the issue.

Amazon Web Services 52

Amazon Web Services Data Lake Java SQL 52

Rockset

AUGUST 28, 2023

Typical examples of repeating tasks include scaling compute resources to optimize their usage from a cost and performance perspective, sending automated e-mails or Slack messages with results of a SQL query, materializing views or doing periodic copies of data for development purposes, exporting data to S3 buckets, and so on.

AWS 52

AWS SQL Datasets Metadata 52

Data Engineering Podcast

SEPTEMBER 4, 2022

Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data.

IT 100

IT Metadata MongoDB MySQL 100

Pinterest Engineering

MAY 17, 2023

Isabel Tallam | Sw Eng, Real Time Analytics; Charles Wu | Sw Eng, Real Time Analytics; Kapil Bajaj | Eng Manager, Real Time Analytics Detecting anomalous events has been becoming increasingly important in recent years at Pinterest. However, over time a model’s accuracy can decrease as outside factors change.

Algorithm 68

Algorithm Engineering Machine Learning Project 68

Databand.ai

FEBRUARY 26, 2023

dbt Alerting and Monitoring with Databand Eitan Chazbani 2023-02-26 18:57:42 As organizations rely more heavily on dbt Core and Cloud to drive transformations, dbt alerting and monitoring has become increasingly important. This alert has been delivered to some alert receiver we’ve set up, so that might be an email address.

SQL 52

SQL Data Pipeline Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

Data observability, an organization’s ability to fully understand the health and quality of the data in their systems, has become one of the hottest technologies in modern data engineering. In less than three years it has gone from an idea sketched out in a Barr Moses blog post to climbing the Gartner Hype Cycle for Emerging Technology.

Data 52

Data Data Pipeline Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

Data observability, an organization’s ability to fully understand the health and quality of the data in their systems, has become one of the hottest technologies in modern data engineering. In less than three years it has gone from an idea sketched out in a Barr Moses blog post to climbing the Gartner Hype Cycle for Emerging Technology.

Data Pipeline 52

Data Pipeline Data Data Engineering Data Engineer 52

DataKitchen

JANUARY 25, 2021

It enables all of the stakeholders to view the real-time status of analytics build operations. The report shows the status of each data source: OK – received on time. Late – new file not present after a specific target delivery time. Received Late – file received after scheduled build start-time.

Data Pipeline 97

Data Pipeline Building Process Utilities 97

Edureka

MARCH 17, 2023

One of the key components of DevOps is automation, which can help reduce human error, improve communication and collaboration, and save time and effort for DevOps teams. In this blog post on ‘ChatGPT for DevOps Tasks’, we’ll explore how ChatGPT can be used in DevOps and the benefits it can provide.

Manufacturing 40

Manufacturing Machine Learning Healthcare Finance 40

Monte Carlo

MARCH 2, 2023

Assuming an hour of a data engineer’s time costs roughly $75 that totals to nearly a half million dollars in engineering time a year spent on resolving data anomalies. And that doesn’t factor in the damage data anomalies can wreck in the time between when they occur and when they are detected and resolved.

Food 52

Food Data SQL Data Pipeline 52

Monte Carlo

SEPTEMBER 6, 2022

We had the opportunity to interview him for our blog a few weeks ago on these and other tips for building out the first data team at your startup. Rather than building systems that detect and alert on breakages, build systems that don’t break.” John Steinmetz, VP of Data & Analytics at shiftkey, echoes this sentiment.

Software Engineer 52

Software Engineer Software Engineering Retail Data 52

DataKitchen

OCTOBER 11, 2021

All your code is version controlled in a system like GitHub, and you’re able to maintain history and track changes. The DataOps pipeline you have built has enough automated tests to catch errors, and error events are tied to some form of real-time alerts. DataOps Project Design and Implementation. Using Version Control.

Engineering 62

Engineering BI Business Analyst Metadata 62

Monte Carlo

MARCH 30, 2022

Like any entropic system, data breaks. In this blog post, I’ll dive into 8 reasons for hidden data quality issues (or, the “silent majority” of data downtime) and best practices for improving detection and tracking. They are delivering reports or executing a data dependent operation at a certain time.

Data Warehouse 52

Data Warehouse Data Machine Learning Data Engineering 52

Snowflake

JANUARY 31, 2023

A number of blog posts, mainly from competitors, call out Snowflake as expensive. At the same time, an accurate comparison is incredibly important. Let’s spend some time looking at the factors in this analysis and why we think Snowflake might be more cost efficient than you previously believed. So what gives?

Cloud 62

Cloud Data Programming Language SQL 62

Monte Carlo

JUNE 7, 2022

How did they do over time? IoT : Near real-time analysis of large volumes of time-series data from IoT devices and sensors. We are a data intensive SaaS application that monitors, alerts and provides lineage within our own UI. We can have alerting on all of our 3,500 tables. Who is their top customer?

Building 52

Building Raw Data Architecture Machine Learning 52

phData: Data Engineering

JANUARY 3, 2022

The separate disciplines begin to slow down due to the complexity and dependencies that have resulted over time. While there’s still an initial concept and inception of the team, the framework works on deliverables in a smaller time frame. You need additional resources to monitor and validate that your system is working as designed.

IT 52

IT AWS Software Engineer Software Engineering 52

Netflix Tech

AUGUST 13, 2020

Seth Katz , Janak Ramachandran , Jeff Butsch , Peter Lau , Ram Vaithilingam , and Greg Burrell Our Telltale Vision An alert fires and you get paged in the middle of the night. You’re half awake and wondering, “Is there really a problem or is this just an alert that needs tuning? Mantis real-time streaming data.

Algorithm 84

Algorithm Entertainment AWS Machine Learning 84