DataKitchen

SEPTEMBER 20, 2023

This webinar discusses how to make embarrassing data errors a thing of the past. We will start with how data engineers do not understand their data and have difficulty identifying problematic data records. We will finish with a demonstration of DataKitchen’s New DataOps Testgen Product.

IT 52

IT Data Engineering Data Engineer Data 52

DataKitchen

SEPTEMBER 20, 2021

Data organizations don’t always have the budget or schedule required for DataOps when conceived as a top-to-bottom, enterprise-wide transformational change. An essential part of the DataOps methodology is Agile Development , which breaks development into incremental steps. Figure 1 shows the four phases of Lean DataOps.

Data Pipeline 98

Data Pipeline Data Cleanse Process Architecture 98

DataKitchen

JANUARY 2, 2021

This is similar to findings in a joint Eckerson-DataKitchen DataOps survey. In this report, Gartner outlines recommendations to effectively operationalize AI solutions that involve the core management competencies of ModelOps, DataOps, and DevOps. Figure 1: Operational AI Requires ModelOps, DataOps, and DevOps Practices.

Data Engineering 52

Data Engineering Data Engineer Engineering Government 52

DataKitchen

AUGUST 19, 2021

Below is our fourth post (4 of 5) on combining data mesh with DataOps to foster innovation while addressing the challenges of a decentralized architecture. We’ve covered the basic ideas behind data mesh and some of the difficulties that must be managed. Figure 1: Data requirements for phases of the drug product lifecycle.

Pharmaceutical 98

Pharmaceutical Data Lake Data Warehouse Raw Data 98

DataKitchen

JUNE 12, 2023

Why the Data Journey Manifesto? So why another manifesto in the world? Why should I care? About seven years ago, we wrote the DataOps Manifesto. I spent much time de-categorizing DataOps: we are not discussing ETL, Data Lake, or Data Science.

ETL Tools 52

ETL Tools Data Science Data Lake Hadoop 52

DataKitchen

MAY 31, 2023

This webinar discusses how to make embarrassing data errors a thing of the past. We will start with how data engineers do not understand their data and have difficulty identifying problematic data records. We will finish with a demonstration of DataKitchen’s New DataOps Testgen Product.

IT 52

IT Data Engineering Data Engineer Data 52

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. A DataOps superstructure provides the foundation to address the many challenges inherent in operating a group of interdependent domains. The problem is not “you.”

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

DataKitchen

MARCH 16, 2023

How to measure your data analytics team? So it’s Monday, and you lead a data analytics team of perhaps 30 people. Like most leaders of data analytic teams, you have been doing very little to quantify your team’s success. What should be in that report about your data team? Introduction. You’ve got a new boss.

Data Analytics 52

Data Analytics Manufacturing Data Process 52

DataKitchen

OCTOBER 27, 2022

. Question: What is something the data industry is missing? I think it’s observability-led DataOps. The first step in solving that pain is to observe what’s happening with your data and analytics ‘estate’ and stick little thermometers at various points in the process and measure. Is it late?

Raw Data 52

Raw Data Data Pipeline Data Data Engineering 52

Monte Carlo

AUGUST 31, 2023

Data is a priority for your CEO, as it often is for digital-first companies, and she is fluent in the latest and greatest business intelligence tools. But what happens when the data is wrong? What about a frantic email from your CTO about “duplicate data” in a business intelligence dashboard? The root of data downtime?

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Big Data Tools

OCTOBER 10, 2022

I am now delighted to have the privilege of returning to the task of collecting for you the most exciting news from the world of data engineering. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. Why should data engineers care about this?

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

OCTOBER 10, 2022

I am now delighted to have the privilege of returning to the task of collecting for you the most exciting news from the world of data engineering. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. Why should data engineers care about this?

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

DataKitchen

JANUARY 18, 2023

Any good plumber can give you some hard-won tips on managing data pipelines effectively, insights that might save your career from going down the drain. That’s why a key goal of DataOps is to expose your best-laid plans to real-life urgencies faster and initiate that crucial feedback loop sooner.

Data Pipeline 52

Data Pipeline Data AWS Data Engineering 52

DataKitchen

JUNE 19, 2023

Our recent survey showed that 97% of data engineers report experiencing burnout in their day-to-day jobs. Given the complicated distributed systems we use to get value from data and the diversity of data, we need a simplifying framework. That idea is the Data Journey.

Data 52

Data Data Validation Utilities High Quality Data 52

DataKitchen

OCTOBER 2, 2023

Data Teams and Their Types of Data Journeys In the rapidly evolving landscape of data management and analytics, data teams face various challenges ranging from data ingestion to end-to-end observability. It explores why DataKitchen’s ‘Data Journeys’ capability can solve these challenges.

Data Ingestion 40

Data Ingestion Data Government Datasets 40

Databand.ai

DECEMBER 13, 2022

The Top 25 Data Engineering Influencers and Content Creators on LinkedIn Ryan Yackel 2022-12-13 10:23:19 Interested in data engineering? LinkedIn is full of influencers sharing new ideas and sparking conversations on all kinds of topics, and data engineering is no exception. You’ve come to the right place.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Monte Carlo

JULY 7, 2022

At the time, he was one of a very few people using the term “data contract.” Data contracts have since became one of the most discussed topics in data engineering. Forward: A Data Engineering Movement Afoot? By Barr Moses GoCardless’s ETL approach focuses on treating data like an API.

Data Warehouse 52

Data Warehouse Software Engineer Software Engineering Data 52

Databand.ai

AUGUST 30, 2023

DataOps Tools: Key Capabilities & 5 Tools You Must Know About Ryan Yackel August 30, 2023 What Are DataOps Tools? DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

DataKitchen

APRIL 18, 2024

Introducing DataKitchen’s Open Source Data Observability Software Today, we announce that we have open-sourced two complete, feature-rich products that solve the data observability problem: DataOps Observervability and DataOps TestGen. What is DataOps Observability? What is DataOps TestGen?

Data Analytics 48

Data Analytics Data SQL Software Engineer 48

DataKitchen

NOVEMBER 18, 2021

For several years now, the elephant in the room has been that data and analytics projects are failing. Gartner estimated that 85% of big data projects fail. Add all these facts together, and it paints a picture that something is amiss in the data world. . The top-line result was that 97% of data engineers are feeling burnout. .

Data Engineering 98

Data Engineering Data Engineer Engineering Government 98

DataKitchen

MARCH 8, 2023

These three objectives are interconnected and essential to the success of any data team. Delivering insight to customers without error is critical to the success of any data team. The team must ensure that the data they are working with is clean and accurate and that the analysis created from it is rigorous and reliable.

Manufacturing 52

Manufacturing Education Data Analytics Raw Data 52

Jesse Anderson

MARCH 6, 2023

Between January 24, 2023, and February 28, 2023, I ran a survey to get more data for my latest book Data Teams , and to update my previous survey from late 2020. This survey was designed to get information about how management uses data teams, the value they’re creating, and how they’re creating it. are in pre-production.

Consulting 130

Consulting Data Data Science Big Data 130

DataKitchen

JULY 27, 2023

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure. While working in Azure with our customers, we have noticed several standard Azure tools people use to develop data pipelines and ETL or ELT processes. We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure.

Data Pipeline 52

Data Pipeline BI Machine Learning Data Preparation 52

Monte Carlo

JULY 28, 2022

There are virtually an unlimited number of ways data can break. But perhaps one of the most common reasons for data quality challenges are software feature updates and other changes made upstream by software engineers. In fact, most of the time they are unaware of these data quality challenges.

Pipeline-centric 52

Pipeline-centric Software Engineer Software Engineering Database-centric 52

DataKitchen

OCTOBER 5, 2022

This is the first post in DataKitchen’s four-part series on DataOps Observability. Observability is a methodology for providing visibility of every journey that data takes from source to customer value across every tool, environment, data store, team, and customer so that problems are detected and addressed immediately.

Data Pipeline 70

Data Pipeline Datasets Engineering Data Engineering 70

Monte Carlo

SEPTEMBER 6, 2022

Say it with me: bad data is inevitable. It doesn’t care about how proactive you are at writing dbt tests, how perfectly your data is modeled, or how robust your architecture is. The possibility of a major data incident (Null value? John Steinmetz, VP of Data & Analytics at shiftkey, echoes this sentiment. Here’s why.

Software Engineer 52

Software Engineer Software Engineering Retail Data 52

DataKitchen

OCTOBER 5, 2022

This is the first post in DataKitchen’s four-part series on DataOps Observability. Observability is a methodology for providing visibility of every journey that data takes from source to customer value across every tool, environment, data store, team, and customer so that problems are detected and addressed immediately.

Data Pipeline 52

Data Pipeline Datasets Engineering Data Engineering 52

Monte Carlo

NOVEMBER 21, 2022

What’s next for the future of data engineering? Each year, we chat with one of our industry’s pioneering leaders about their predictions for the modern data stack – and share a few of our own. Pro-tip: be sure to check out his talk from IMPACT: The Data Observability Summit.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

DataKitchen

OCTOBER 25, 2022

Part 2: Introducing Data Journeys. This is the second post in DataKitchen’s four-part series on DataOps Observability. DataOps Observability to the Rescue. Jason knows he needs to implement changes, but what and where? The more he reads about DataOps, the more he knows his company needs to make the transition.

Data Pipeline 52

Data Pipeline Data Workflow Coding Building 52

phData: Data Engineering

JANUARY 3, 2022

DataOps: What Is It, Core Principles, and Tools For Implementation Nick Goble January 3, 2022 When building a successful company, it’s critical to have a strategy around how you build and scale your business from a technology and data perspective. What’s Data Strategy? Why Is This Challenging? Why is that?

IT 52

IT AWS Software Engineer Software Engineering 52

Monte Carlo

MAY 6, 2021

As data systems become increasingly distributed and companies ingest more and more data, the opportunity for error (and incidents) only increases. For decades, software engineering teams have relied on a multi-step process to identify, triage, resolve, and prevent issues from taking down their applications.

Management 40

Management Data Pipeline Software Engineer Software Engineering 40

phData: Data Engineering

MAY 11, 2023

Practices centered on software engineering principles can create a barrier to entry for teams with skilled data wranglers looking to take their infrastructure to the next level with cloud-native tools like Matillion for the Snowflake Data Cloud. Bitbucket, Github) to allow advanced workflows.

Coding 52

Coding Software Engineer Software Engineering Government 52

DataKitchen

APRIL 26, 2021

In early April 2021, DataKItchen sat down with Jonathan Hodges, VP Data Management & Analytics, at Workiva ; Chuck Smith, VP of R&D Data Strategy at GlaxoSmithKline (GSK) ; and Chris Bergh, CEO and Head Chef at DataKitchen, to find out about their enterprise DataOps transformation journey, including key successes and lessons learned.

Software Engineer 98

Software Engineer Software Engineering Machine Learning Education 98

Databand.ai

JANUARY 24, 2023

What is DataOps? The Ultimate Guide for Data Teams Eric Jones 2023-01-24 12:21:38 If you find yourself hearing a lot about DataOps and then subsequently asking, “ What is DataOps? This guide details everything you need to know about DataOps, including: What is DataOps? Why is DataOps important?

Retail 52

Retail Banking Government Manufacturing 52

DataKitchen

JUNE 24, 2021

In May 2021 at the CDO & Data Leaders Global Summit, DataKitchen sat down with the following data leaders to learn how to use DataOps to drive agility and business value. Kurt Zimmer, Head of Data Engineering for Data Enablement at AstraZeneca. Jim Tyo, Chief Data Officer, Invesco.

Pipeline-centric 84

Pipeline-centric Manufacturing Education Data Cleanse 84

Monte Carlo

MAY 19, 2022

In his seminal 2017 blog post, The Downfall of the Data Engineer , Maxime Beauchemin wrote that the data engineer had the worst seat at the table. So, if the data engineer no longer has the worst seat at the table, who then on the data team has inherited this unfortunate title?

Generalist 52

Generalist Data Governance Government Metadata 52

Monte Carlo

SEPTEMBER 12, 2023

Now, imagine all the data powering that platform. We had to know how their data team made it happen, especially in such a short span of time, so we recently sat down with Engineering Director Emmanuel Fuentes to learn more. They were among Whatnot’s earliest employees and together built the data foundation for the company.

Machine Learning 52

Machine Learning Data Entertainment SQL 52