Christophe Blefari

FEBRUARY 16, 2024

Italy Sora ( credits ) Hey you, time for the Data News. Next Wednesday I will participate to a Data Night Talk a open discussion about AI & data engineering with other content creators. Next Wednesday I will participate to a Data Night Talk a open discussion about AI & data engineering with other content creators.

Food 130

Food Data SQL Government 130

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Monte Carlo

MARCH 25, 2024

When we launched the data observability category in 2020, we set out to solve a very real problem: data trust. In the preceding months, I met with hundreds of data leaders about what kept them up at night. What if there was a way to ensure that your data was trustworthy and reliable? EMEA, and other markets.

MySQL 64

MySQL Government Data Governance Kafka 64

Monte Carlo

MARCH 4, 2024

Well, we can liken the Data Observability vs. DataOps relationship to Beyonce (there’s a sentence I never expected to write!). Just as every Beyonce song is a hit, but not every hit song is by Beyonce, data observability is one key component of DataOps, but DataOps is more than just data observability.

Data Pipeline 52

Data Pipeline Data Data Science Software Engineer 52

The Pragmatic Engineer

MARCH 3, 2023

A few weeks ago, I set out why I’d be surprised if Apple goes down that path. So we took October’s data points and compared them with January’s. However, this move is facing a pushback I’ve not observed at any other Big Tech giant. Observations from the Pragmatic Engineer Talent Collective.

Recruitment 194

Recruitment Software Engineer Software Engineering Finance 194

Christophe Blefari

FEBRUARY 19, 2024

Modern data stack has been coined by US companies and VCs—mainly Fivetran / dbt Labs—as a word to quickly emphasis a way to build data stack in the cloud related to ELT. For these organisations the ideal of a modern data stack still resonate. Employees are stuck in hell regarding data tooling.

Data 100

Data Government SQL Technology 100

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Cloudera

MARCH 22, 2023

Cloudera Data Platform (CDP) is no different: it’s a hybrid data platform that meets organizations’ needs to get to grips with complex data anywhere, turning it into actionable insight quickly and easily. Monitoring alone will tell you when something’s not as it should be, but that’s not answering the question of “why?”

Data Governance 84

Data Governance Government Business Analyst Metadata 84

Christophe Blefari

JUNE 16, 2023

I'm now under the Berlin rain with 20° When I write in these conditions I feel like a tortured author writing a depressing novel while actually today I'll speak about the AI Act, Python, SQL and data platforms. Mainly he unit tests macros (the logic) with his framework and test data with soda and dbt contracts.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Precisely

FEBRUARY 26, 2024

Data flows from edge devices to core applications and back again. A myriad of data analytics tools provide up-to-the-minute insights to decision-makers throughout the organization. Observability helps IT teams to achieve mastery over this complexity. Today’s systems, however, are highly interconnected and constantly in flux.

IT 75

IT Banking Programming Language Systems 75

Monte Carlo

JANUARY 3, 2024

Data teams are scrambling to answer the call. Why should users pick you over ChatGPT? That quick check of the box feels like a step forward, but if you aren’t already thinking about how to connect LLMs with your proprietary data and business context to actually drive differentiated value, you’re behind. I’ll explain why below.)

Technology 113

Technology Database Data Governance Data Engineering 113

The Pragmatic Engineer

MARCH 12, 2024

” Brooks agrees with this observation, and suggests a radical solution: have as few senior programmers as possible, and build a team around each one – a bit like how a hospital surgeon leads a whole team. They come up with test cases and data. The tester. Also responsible for scaffolding of tests. 

Engineering 257

Engineering Programming Language Hospitality Programming 257

DataKitchen

MARCH 5, 2024

Why Not Hearing About Data Errors Should Worry Your Data Team In the chaotic lives of data & analytics teams, a day without hearing of any data-related errors is a blessing. Here are seven compelling reasons why you should care and be proactive, even when all seems well.

Data 60

Data Data Management Management Systems 60

The Pragmatic Engineer

OCTOBER 31, 2023

We're also observing elevated errors obtaining temporary credentials from the AWS Security Token Service, and are working in parallel to resolve these errors. We are now observing sustained recovery of the Lambda invoke error rates, and recovery of other affected AWS services. ” “What went wrong and why?“

AWS 216

AWS Google Cloud Cloud Food 216

Monte Carlo

MARCH 4, 2024

As any data engineer currently rolling their eyes would tell you, the reality has long been more nuanced. These questions just scratch the surface of the painful delightfully complex world of data engineering. These solutions also provide you the data lineage and root cause analysis insights you need to understand how to fix them.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Towards Data Science

AUGUST 29, 2023

Data Quality Chronicles Missing data, missing mechanisms, and missing data profiling Missing Data prevents data scientists to see the entire story the data has to tell. One of them was, unsurprisingly, Missing Data. Photo by Ronan Furuta on Unsplash. Image by Author. Let’s consider an example.

Datasets 107

Datasets Machine Learning Data Data Science 107

Monte Carlo

APRIL 1, 2024

But, for data engineers, there’s something else that comes pretty close to the top of that list: clean data. Data cleaning is an essential step to ensure your data is safe from the adage “garbage in, garbage out.” Define Clear Data Quality Standards 2. Implement Routine Data Audits 3. Table of Contents 1.

High Quality Data 52

High Quality Data Datasets Data Data Pipeline 52

DataKitchen

FEBRUARY 23, 2024

DataKitchen Resource Guide To Data Journeys & Data Observability & DataOps Data (and Analytic) Observability & Data Journey – Ideas and Background Data Journey Manifesto and Why the Data Journey Manifesto?

Data Validation 48

Data Validation Data Data Pipeline Data Solutions 48

Monte Carlo

NOVEMBER 9, 2023

And in many ways, LLMs are going to make data engineers more valuable – and that’s exciting! Still, it’s one thing to show your boss a cool demo of a data discovery tool or text-to-SQL generator – it’s another thing to use it with your company’s proprietary data, or even more concerning, customer data.

Unstructured Data 86

Unstructured Data Database Data Pipeline Architecture 86

Monte Carlo

APRIL 9, 2024

When data engineers tell scary stories around a campfire, it’s usually a cautionary tale about bad data. Data downtime can occur suddenly at any time—and often not when or where you’re looking for it. But just how much can data downtime actually cost your business? in inefficient operations during data downtime periods.

Data 52

Data Data Engineering Data Engineer Media 52

Monte Carlo

MAY 26, 2023

What is data freshness and why is it important? Data freshness, sometimes referred to as data timeliness, is the frequency in which data is updated for consumption. I would always frame the conversation in simple business terms and focus on the who, what, when, where, and why.

Data Pipeline 98

Data Pipeline Data Data Warehouse Machine Learning 98

Monte Carlo

JANUARY 19, 2024

Well, the same goes for your data. Just because it looked good yesterday doesn’t mean it’ll hold up tomorrow – and that’s why we’re talking about data integrity testing today. Data integrity testing is the process of ensuring data is fit for the task at hand and available to only those who should have access.

Data Integration 52

Data Integration Retail SQL Data Pipeline 52

Monte Carlo

JANUARY 16, 2024

They run on data powered by Fox. Factor in the advertising strategies, media production, partner programming, audience analytics…and you’re looking at an ocean of data that would fill even the deepest trench (we’d like a television show about that too, please!). So how does Fox’s data strategy support these complex data workflows?

Government 52

Government High Quality Data Data Governance Entertainment 52

Monte Carlo

MARCH 7, 2024

As GenAI has zoomed onto center stage in the daily data discourse, prompt engineering has been quietly along for the ride. And as we move into our undoubtedly AI-laden future, prompt engineering will be a necessary skill for every data engineer to develop. Table of Contents Why is prompt engineering important? Or so some say.

Engineering 59

Engineering High Quality Data Database Architecture 59

Monte Carlo

FEBRUARY 9, 2024

But, when you find a data leader who’s on the real AI journey first-hand (no, not Midjourney), it’s natural to have a few questions. Data asset optimization is the need of the hour 4. Data asset optimization is the need of the hour 4. Reliable AI needs reliable data. Data observability is the cost of admission.

Data 64

Data High Quality Data Architecture Coding 64

Monte Carlo

MARCH 18, 2024

Data quality audits are a key way to ensure your business decisions are being informed by fresh, accurate, and up-to-date data. But, whether your data resides in a warehouse, lake, lakehouse, or on-prem, it’s important to take the time to assess the accuracy of your organization’s data.

Datasets 52

Datasets Data Pipeline BI Data 52

Monte Carlo

AUGUST 31, 2023

Every data science team has that moment where it realizes it has outgrown its initial architecture. Our codebase and data science team were growing rapidly in parallel as well. Here’s how and why we did it. That moment came quickly for our team and our SQL-based feature store. Example features of a generic ML model.

SQL 64

SQL Data Science Machine Learning Algorithm 64

Engineering at Meta

APRIL 17, 2023

What the research is: Millisampler is one of Meta’s latest characterization tools and allows us to observe, characterize, and debug network performance at high-granularity timescales efficiently. Since the data is only aggregated flow-level header information, it does not contain any personally identifiable information (PII).

Bytes 111

Bytes Transportation Data Collection Coding 111

Monte Carlo

JUNE 6, 2023

Generative AI is taking the world by storm – here’s what it means for data engineering and why data observability is critical for this groundbreaking technology to succeed. And how will it impact data? It’s inevitable that generative AI will disrupt data, too. Let’s assess. Let’s assess.

Data Engineering 84

Data Engineering Data Engineer Engineering Business Intelligence 84

Monte Carlo

MARCH 28, 2024

While the DevOps methodology has been taking over the world of software development, data teams are just beginning to realize the benefits that a similar approach can bring to their world. At the Heart of Both DevOps and DataOps is Observability What Is The Difference Between DataOps and DevOps? Enter the nascent discipline of DataOps.

Data Pipeline 52

Data Pipeline Coding Machine Learning Data Warehouse 52

Monte Carlo

MARCH 18, 2024

That’s why Monte Carlo was the first data observability company to support vector databases and GenAI reliability. However, in order to realize the potential value of AI, we’ll first have to start thinking critically about how we develop AI applications—and the role data teams play in it.

Building 64

Building Healthcare Data Database 64

Monte Carlo

SEPTEMBER 27, 2023

When it comes to building data and AI products you can trust, Monte Carlo, the leader of the data observability category, and Alation, the data intelligence leader, are better together. Why is my data late?” “How Can I trust the data feeding this model? What’s wrong with this metric?” “Why

Data Governance 69

Data Governance Government Data Pipeline Data 69

Monte Carlo

JANUARY 10, 2024

What happens when you strip away all the noise of queries and pipelines and focus on the data itself? You get down to the intrinsic data quality. What’s the difference between intrinsic and extrinsic data quality? Intrinsic data quality is the quality of data assessed independently of its use case. Data Auditing 5.

Data Cleanse 52

Data Cleanse Data Engineering Data Engineer Engineering 52

Precisely

DECEMBER 12, 2023

quintillion exabytes of data every da y. It includes streaming data from smart devices and IoT sensors, mobile trace data, and more. Data is the fuel that feeds digital transformation. But with all that data, there are new challenges that may prompt you to rethink your data observability strategy.

IT 52

IT Medical Hospitality Data Pipeline 52

Cloudera

DECEMBER 1, 2022

Full-stack observability is a critical requirement for effective modern data platforms to deliver the agile, flexible, and cost-effective environment organizations are looking for. RI is a global leader in the design and deployment of large-scale, production-level modern data platforms for the world’s largest enterprises.

Data Lake 80

Data Lake Data Pipeline Data Governance Analytics Application 80

DoorDash Engineering

AUGUST 1, 2023

Accurate and reliable observability is essential when supporting a large distributed service, but this is only possible if your tools are equally scalable. Just when we most needed observability data, the system would leave us in the lurch. Just when we most needed observability data, the system would leave us in the lurch.

AWS 82

AWS Transportation Programming Language Government 82

Databand.ai

JUNE 8, 2023

Observability in Your Data Pipeline: A Practical Guide Eitan Chazbani June 8, 2023 Achieving observability for data pipelines means that data engineers can monitor, analyze, and comprehend their data pipeline’s behavior. This is part of a series of articles about data observability.

Data Pipeline 52

Data Pipeline Bytes Raw Data Data Collection 52