François Nguyen

JANUARY 23, 2021

or why you should have a look at Data Observability ! The second part is really how they do it and all the steps to have a “certification” They name it Midas, the famous king who can turn everything into gold ( with a not so good ending ). Of course you must document everything and have many validations.

Finance 130

Finance Certification Data Pipeline Data 130

Christophe Blefari

SEPTEMBER 25, 2023

Gen AI 🤖 Announcing Microsoft Copilot — Having everything under a common brand is great and Copilot is a great name. I'd not say that all the downsides are relevant blockers but still it outlines on of the biggest Airflow issue: everything is implicit. Data Economy 💰 Cisco acquired Splunk for $28b in cash.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Christophe Blefari

SEPTEMBER 25, 2023

Gen AI 🤖 Announcing Microsoft Copilot — Having everything under a common brand is great and Copilot is a great name. I'd not say that all the downsides are relevant blockers but still it outlines on of the biggest Airflow issue: everything is implicit. Data Economy 💰 Cisco acquired Splunk for $28b in cash.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Data Engineering Podcast

JANUARY 7, 2022

Summary Data observability is a set of technical and organizational capabilities related to understanding how your data is being processed and used so that you can proactively identify and fix errors in your workflows. You can observe your pipelines with built in metadata search and column level lineage. How does it work?

BI 100

BI Data Warehouse Metadata SQL 100

Data Engineering Podcast

SEPTEMBER 4, 2022

Summary Data observability is a product category that has seen massive growth and adoption in recent years. Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. You were one of the early entrants in the market of data quality/data observability products.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

FEBRUARY 26, 2023

observability, data quality, etc.) You can't optimize for everything all at once. observability, data quality, etc.) You can't optimize for everything all at once. What are the biggest challenges (technical and procedural) that you have encountered during your implementation?

Building 147

Building Metadata Machine Learning Data Integration 147

Christophe Blefari

SEPTEMBER 28, 2023

But every time I went to a conference in the past, I came back with ideas to change everything because someone introduced me to a new fancy stuff. We need to store, process and visualise data, everything else is just marketing. Last week it was Big Data London , this week it was Big Data & AI Paris. I wasn't able to go.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

DataKitchen

MARCH 5, 2024

Problems are often hidden, either unintentionally due to a lack of observability or deliberately as a cover-up. Assuming everything is accurate without rigorous verification is a gamble that most businesses cannot afford. Even minor errors can have substantial consequences in industries where precision is non-negotiable.

Data 60

Data Data Management Management Systems 60

Data Engineering Podcast

JANUARY 23, 2022

With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. You can observe your pipelines with built in metadata search and column level lineage. You can observe your pipelines with built in metadata search and column level lineage.

Building 100

Building Management Data Pipeline Metadata 100

Data Engineering Podcast

OCTOBER 19, 2020

To ensure that your pipelines are healthy you need a way to make them observable. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Can you start by giving your definition of observability in the context of data workflows?

Machine Learning 100

Machine Learning Data Engineering Data Engineer Data 100

Monte Carlo

NOVEMBER 20, 2023

Now with everything set, it’s time for the main event Celebrate, but continue to monitor How to handle challenges and rollbacks Best practices for successful data migrations Ready to elevate your next data migration with unparalleled data observability? That’s where data observability platforms like Monte Carlo come in.

MongoDB 52

MongoDB ETL Tools Database Data 52

Monte Carlo

MARCH 21, 2024

I maintain a sandbox environment for Monte Carlo’s solution engineers to demo our data observability solution. This will be tailored to data observability solutions specifically, but these principles apply to all data solutions. People and Process Data observability isn’t just a technology, it’s an operational process.

Data Warehouse 52

Data Warehouse Technology Database Machine Learning 52

Monte Carlo

JANUARY 30, 2024

” Table of Contents Monitoring and resolving data quality issues at scale Operationalizing data observability Earning data trust and proving ROI Monitoring and resolving data quality issues at scale JetBlue required a data observability platform that could operate effectively at scale.

Data 64

Data SQL Metadata Data Engineering 64

Monte Carlo

OCTOBER 31, 2023

I’m not sure what’s harder to believe – that we’re just a week away from IMPACT OR that it’s our third iteration of the world’s only Data Observability Summit ! 2023 has been a crazy year in data, with the rise of GenAI and LLMs eclipsing nearly everything else in tech.

Data Engineering 72

Data Engineering Data Engineer Data Engineering 72

Monte Carlo

DECEMBER 13, 2023

Over the last five years, data observability has leveled up from industry buzzword to a must-have element of every data stack. With data observability, teams can identify and proactively resolve incidents of data downtime — moments when data is partial, erroneous, missing, or otherwise inaccurate. The outcome? Win, win, win, win, win.

BI 52

BI Data Ingestion SQL Data Warehouse 52

Data Engineering Podcast

FEBRUARY 13, 2022

With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. You can observe your pipelines with built in metadata search and column level lineage. You can observe your pipelines with built in metadata search and column level lineage.

Building 100

Building Hadoop Data Pipeline Metadata 100

Data Engineering Podcast

FEBRUARY 6, 2022

With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. You can observe your pipelines with built in metadata search and column level lineage. You can observe your pipelines with built in metadata search and column level lineage.

Data Pipeline 100

Data Pipeline Data Metadata Software Engineer 100

Data Engineering Podcast

JANUARY 30, 2022

With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. You can observe your pipelines with built in metadata search and column level lineage. You can observe your pipelines with built in metadata search and column level lineage.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Pipeline 100

Monte Carlo

SEPTEMBER 19, 2023

The data space is evolving at break-neck speed, and IMPACT: The Data Observability Summit is returning to bring clarity to the chaos. Attendees will also get an exclusive look into Monte Carlo’s product roadmap and what’s next for the future of data observability, along with plenty of other surprises from the Monte Carlo team.

Insurance 52

Insurance Software Engineer Software Engineering Data 52

François Nguyen

FEBRUARY 28, 2021

We do not really know what it happening (see again data observability by Barr Moses) Let’s do it like the Toyota Way #1 “Create continuous process flow to bring problems to the surface” Capturing the data continuously is creating a flow and it is much more harder because you will to anticipate all the problems.

Data Pipeline 130

Data Pipeline Building Manufacturing BI 130

Scott Logic

JANUARY 16, 2023

He observes that once you know what you want, it’s quick to rewrite rough code that helped you figure that out. Another tip he had was on observability - that if you can’t reason about your code or failing tests via logs alone (i.e. don’t help us much if we haven’t modelled our domain well, he observes.

Generalist 83

Generalist Coding Java Education 83

Christophe Blefari

MARCH 24, 2023

Kestra vision is also very open, everything is accessible through APIs. Because OpenAI is not everything, some news of the alternative world. Initially this is a data observability tool but it turned out they added features like lineage and cataloging. Kestra is a YAML-based data pipeline tool mixed with string templating.

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Monte Carlo

FEBRUARY 9, 2024

Data observability is the cost of admission. In a world where free money is a thing of the past, data teams are doing everything they can to drive more stakeholder value at scale—and demonstrate that commitment to the executive team. Data observability is the cost of admission. Self-service is still relevant—and even more so 3.

Data 64

Data High Quality Data Architecture Coding 64

Engineering at Meta

OCTOBER 23, 2023

In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Fact4: Buck2 has no knowledge of any programming language built in Everything language-specific in Buck2 is described in Starlark. Buck2 is the recently open-sourced successor. We take our type safety very seriously!

Programming Language 109

Programming Language Java Python Programming 109

Monte Carlo

DECEMBER 20, 2023

So what is the optimal data observability architecture or data quality coverage strategy? Data testing capabilities are typically nested within data quality monitoring tools and both are nested within data observability solutions. Deploy data observability across all of your production pipelines, excluding the ad-hoc and experimental.

Architecture 40

Architecture Data Datasets Machine Learning 40

Data Engineering Podcast

OCTOBER 15, 2021

They also share their ambitions for the near future of adding data observability and data quality management features. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. What are your plans for Data Observability?

Metadata 100

Metadata BI Data Warehouse Government 100

Knowledge Hut

MARCH 14, 2024

Playtime – Dynamic Analysis: Now, let's run the software and observe it in action. Record Everything – Documentation: Write down everything you discover. Testing ensures everything works as expected. What does it do when you click buttons or input data? Popular Tool: Ghidra.

Software Engineer 52

Software Engineer Software Engineering Engineering Electronics 52

Christophe Blefari

DECEMBER 31, 2023

To be honest, everything's been going well, but I've found it hard to find time to write among other things. They still use technologies around JVM (Spark and Flink), but with no surprise everything resolves around Iceberg—which has been created at Netflix. And that's the problem. And they recorded it.

Data 100

Data Cloud Storage Datasets Python 100

Data Engineering Podcast

AUGUST 28, 2022

With Bigeye’s data observability platform, if there is an issue with your data or data pipelines you’ll know right away and can get it fixed before the business is impacted. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Cloudera

MARCH 17, 2023

Align everything to corporate strategy I lead data and analytics at Cloudera. Our CDA charter must be aligned with corporate strategy, but shouldn’t everything we do be aligned? Our CDA charter must be aligned with corporate strategy, but shouldn’t everything we do be aligned? We’re called Cloudera Data Analytics (CDA).

Data Architecture 79

Data Architecture Data Architecture Data Analytics 79

Edureka

APRIL 19, 2024

Let’s begin by installing Flowise and getting everything set up for use. After successfully installing NodeJS, open your command prompt and type ‘npm’ You’ll observe something similar to the example below. To get started with Flowise, we first need to install some necessary tools onto our systems.

Building 52

Building Engineering Coding Systems 52

DataKitchen

OCTOBER 27, 2022

I think it’s observability-led DataOps. The first step in solving that pain is to observe what’s happening with your data and analytics ‘estate’ and stick little thermometers at various points in the process and measure. Or maybe everything is perfect, and your dashboards are not being used.

Raw Data 52

Raw Data Data Pipeline Data Data Engineering 52

Towards Data Science

JANUARY 16, 2024

Organizations began to use relational databases for ‘everything’. Image by the author 2004 to 2010 — The elephant enters the room New wave of applications emerged — Social Media, Software observability, etc. So here goes my overly-simplified take. Databases were overwhelmed with transactional and analytical workloads.

Data Warehouse 68

Data Warehouse Data Architecture Relational Database NoSQL 68

Monte Carlo

AUGUST 17, 2023

I would get emails from the CEO, president, and others asking where the data is coming from or to explain the logic because there were situations where everything was correct, but numbers wouldn’t match. One of the benefits of data observability has been to standardize processes so they don’t have to code against bad data.

High Quality Data 52

High Quality Data Data Data Engineering Data Engineer 52

Picnic Engineering

JUNE 26, 2023

A glimpse at how a picking station looks like in our automated fulfillment center We are always looking for new ways to make this system better, identifying what can help the shoppers in their day to day work, while ensuring everything runs smoothly and efficiently. Recently we worked on one of those improvements.

Building 64

Building Systems Coding Project 64

Snowflake

MAY 2, 2023

And that is what observability is about: faster, more intelligent understanding of the state of entire networks and applications. Determining what kinds of observability data to use, and how to move, store, enrich, and analyze that data will help separate those that prosper from those that flounder. Enter observability.

Telecommunication 52

Telecommunication Metadata Data Portfolio 52

Lyft Engineering

APRIL 24, 2024

It is rare for a data scientist to be skilled in everything, especially in the first couple of years of their career. Over the years, we have successfully launched lecture series on topics such as experimentation, observational causal inference, structural causal modeling, and reinforcement learning.

Data Science 66

Data Science Building Computer Science Software Engineer 66