Databand.ai

JUNE 8, 2023

Observability in Your Data Pipeline: A Practical Guide Eitan Chazbani June 8, 2023 Achieving observability for data pipelines means that data engineers can monitor, analyze, and comprehend their data pipeline’s behavior. This is part of a series of articles about data observability.

Data Pipeline 52

Data Pipeline Bytes Raw Data Data Collection 52

Hevo

FEBRUARY 2, 2024

Data pipelines have become an integral part of organizational workflows to keep pace with the rapidly evolving landscape of data management and analytics. By systematically processing and transporting data from its source to its destination, data pipelines help with data-driven decision-making.

Data Pipeline 40

Data Pipeline Transportation Data Data Management 40

Data Engineering Podcast

SEPTEMBER 30, 2019

Summary Building an end-to-end data pipeline for your machine learning projects is a complex task, made more difficult by the variety of ways that you can structure it. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, and Data Council.

Data Pipeline 100

Data Pipeline Machine Learning Media Data 100

Data Engineering Podcast

JUNE 26, 2022

Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Master Data Management (MDM) is the process of building consensus around what the information actually means in the context of the business and then shaping the data to match those semantics.

Data Management 100

Data Management Management MongoDB Scala 100

Data Engineering Podcast

JANUARY 26, 2020

Summary Data pipelines are complicated and business critical pieces of technical infrastructure. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Data Pipeline 100

Data Pipeline PostgreSQL Media Data Validation 100

Ascend.io

AUGUST 15, 2023

This same principle holds true in data management. It’s no surprise, then, that the quest for Fivetran alternatives is on the rise as organizations set their sights on a more holistic data approach. Defense: Saving Money with Intelligent Data Refresh In football, a solid defense does more than just stop goals.

Data Management 52

Data Management Management Data Ingestion Data Pipeline 52

Ascend.io

APRIL 12, 2023

Right now, at this moment, are you prepared to act on your company’s data? So when we talk about making data usable, we’re having a conversation about data integrity. Data integrity is the overall readiness to make confident business decisions with trustworthy data, repeatedly and consistently. If not, why?

Data Pipeline 52

Data Pipeline Data Integration Datasets Data 52

Ascend.io

MARCH 2, 2023

The system suffered frequent outages and regularly lost months of data. We couldn’t get data to the business fast enough, and the data we did ship wasn’t trustworthy. Our velocity of delivering new data products continued to diminish. The initial wave of work migrating 75 data projects is proceeding like clockwork!

Data Pipeline 52

Data Pipeline Media Portfolio Data Ingestion 52

Data Engineering Podcast

DECEMBER 16, 2018

Summary Every business needs a pipeline for their critical data, even if it is just pasting into a spreadsheet. As the organization grows and gains more customers, the requirements for that pipeline will change. Can you start by sharing your definition of a data pipeline? What are your thoughts on that approach?

Data Pipeline 100

Data Pipeline Data Lake Data Warehouse Python 100

Data Engineering Podcast

MAY 27, 2018

In this episode CTO and co-founder of Alooma, Yair Weinberger, explains how the platform addresses the common needs of data collection, manipulation, and storage while allowing for flexible processing. How do you manage the underlying infrastructure to support your SLA of 5 nines? What is Alooma and what is the origin story?

Data Pipeline 100

Data Pipeline MongoDB Scala Google Cloud 100

Data Engineering Podcast

JUNE 17, 2017

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Can you start by giving an overview of your pipeline and the type of workload that you are optimizing for?

Data Pipeline 100

Data Pipeline Kafka Business Intelligence Architecture 100

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. As data is expanding exponentially, organizations struggle to harness digital information's power for different business use cases.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Hevo

MAY 30, 2023

The healthcare industry has seen an exponential growth in the use of data management and integration tools in recent years to leverage the data at their disposal. Unlocking the potential of “Big Data” is imperative in enhancing patient care quality, streamlining operations, and allocating resources optimally.

Healthcare 52

Healthcare Data Pipeline Building Big Data 52

Data Engineering Podcast

NOVEMBER 13, 2022

Summary Building data products is an undertaking that has historically required substantial investments of time and talent. With the rise in cloud platforms and self-serve data technologies the barrier of entry is dropping. Atlan is the metadata hub for your data ecosystem.

Building 130

Building Metadata MongoDB MySQL 130

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. How is this distinct from other forms of data pipeline development and delivery?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Jesse Anderson

NOVEMBER 14, 2023

A Guest Post by Ole Olesen-Bagneux In this blog post I would like to describe a new data team, that I call ‘the data discovery team’. Data discovery is thought of in different ways in data science and in information science respectfully. In an enterprise data reality, searching for data is a bit of a hassle.

Metadata 147

Metadata Data Science Big Data Data 147

Data Engineering Podcast

NOVEMBER 6, 2022

Summary A lot of the work that goes into data engineering is trying to make sense of the "data exhaust" from other applications and services. Atlan is the metadata hub for your data ecosystem. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Building 130

Building IT Metadata MongoDB 130

ProjectPro

FEBRUARY 16, 2023

ETL is a critical component of success for most data engineering teams, and with teams harnessing it with the power of AWS, the stakes are higher than ever. Data Engineers and Data Scientists require efficient methods for managing large databases, which is why centralized data warehouses are in high demand.

AWS 52

AWS Data Management ETL Tools Management 52

Data Engineering Podcast

NOVEMBER 6, 2022

Summary Despite the best efforts of data engineers, data is as messy as the real world. Entity resolution and fuzzy matching are powerful utilities for cleaning up data from disconnected sources, but it has typically required custom development and training machine learning models.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

DECEMBER 4, 2022

Summary The term "real-time data" brings with it a combination of excitement, uncertainty, and skepticism. In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Towards Data Science

FEBRUARY 6, 2024

This generalisation makes their data models complex and cryptic and require domain expertise. Even harder to manage, a common setup within large organisations is to have several instances of these systems with some underlaying processes in charge of transmitting data among them, which could lead to duplications, inconsistencies, and opacity.

Systems 83

Systems Raw Data Metadata Data Cleanse 83

Data Engineering Podcast

DECEMBER 4, 2022

Summary Business intelligence is the foremost application of data in organizations of all sizes. Zing Data is building a mobile native platform for business intelligence. Atlan is the metadata hub for your data ecosystem. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Data Engineering Podcast

JANUARY 23, 2022

Summary Data engineering is a relatively young and rapidly expanding field, with practitioners having a wide array of experiences as they navigate their careers. Ashish Mrig currently leads the data analytics platform for Wayfair, as well as running a local data engineering meetup.

Building 100

Building Management Data Pipeline Metadata 100

Data Engineering Podcast

FEBRUARY 6, 2022

Summary There are many dimensions to the work of protecting the privacy of users in our data. When you need to share a data set with other teams, departments, or businesses then it is of utmost importance that you eliminate or obfuscate personal information. The only thing worse than having bad data is not knowing that you have it.

Data Pipeline 100

Data Pipeline Data Metadata Software Engineer 100

Databand.ai

AUGUST 30, 2023

Data Testing Tools: Key Capabilities and 6 Tools You Should Know Helen Soloveichik August 30, 2023 What Are Data Testing Tools? Data testing tools are software applications designed to assist data engineers and other professionals in validating, analyzing, and maintaining data quality.

Data Cleanse 52

Data Cleanse Data Validation Data Pipeline Datasets 52

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. In this episode Inbar Yogev and Lior Winner share the journey that they and their teams at Riskified have been on for their data platform. They also discuss how they have established a guild system for training and supporting data professionals in the organization.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. RudderStack’s smart customer data pipeline is warehouse-first. How does it work?

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Data Engineering Podcast

JANUARY 30, 2022

Summary Pandas is a powerful tool for cleaning, transforming, manipulating, or enriching data, among many other potential uses. As a result it has become a standard tool for data engineers for a wide range of applications. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Pipeline 100

Data Engineering Podcast

MAY 22, 2022

Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to address their internal needs for machine learning and integrated closely with Kubernetes as the execution manager.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

Precisely

MAY 13, 2024

Key Takeaways: Data integration is vital for real-time data delivery across diverse cloud models and applications, and for leveraging technologies like generative AI. The right data integration solution helps you streamline operations, enhance data quality, reduce costs, and make better data-driven decisions.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Databand.ai

JULY 11, 2023

Data Quality Platform: Benefits, Key Features, and How to Choose Eric Jones July 11, 2023 What Is a Data Quality Platform? A data quality platform is a software solution designed to help organizations manage, maintain, and improve the quality of their data. In this article: Why Do You Need a Data Quality Platform?

Data Cleanse 52

Data Cleanse Telecommunication High Quality Data BI 52

Databand.ai

JULY 19, 2023

Data Lineage Tools: Key Capabilities and 5 Notable Solutions Ryan Yackel July 19, 2023 What Are Data Lineage Tools? Data lineage tools provide a visual representation of your data’s journey across multiple systems and transformations. Data lineage tools are not a new concept.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Databand.ai

AUGUST 30, 2023

The DataOps framework is a set of practices, processes, and technologies that enables organizations to improve the speed, accuracy, and reliability of their data management and analytics operations. The core philosophy of DataOps is to treat data as a valuable asset that must be managed and processed efficiently.

Data Governance 52

Data Governance Data Pipeline Government Data Cleanse 52

The Modern Data Company

FEBRUARY 15, 2023

You can’t blame companies for playing the cards they’re given, but now data debt is costing companies more than they think, even when solutions seem to be working…for now. What Causes Data Debt? Data debt is similar to technical debt. Data, unlike technology, seems simple on the surface.

Government 98

Government Data Governance Raw Data Data 98