Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

The Modern Data Company

FEBRUARY 15, 2023

You can’t blame companies for playing the cards they’re given, but now data debt is costing companies more than they think, even when solutions seem to be working…for now. What Causes Data Debt? Data debt is similar to technical debt. Data, unlike technology, seems simple on the surface.

Government 98

Government Data Governance Raw Data Data 98

Databand.ai

AUGUST 30, 2023

Data Testing Tools: Key Capabilities and 6 Tools You Should Know Helen Soloveichik August 30, 2023 What Are Data Testing Tools? Data testing tools are software applications designed to assist data engineers and other professionals in validating, analyzing, and maintaining data quality.

Data Cleanse 52

Data Cleanse Data Validation Data Pipeline Datasets 52

Monte Carlo

JUNE 6, 2023

Sure, data quality is everyones’ problem. But who is responsible for data quality? Some organizations will attempt to diffuse the responsibility widely across data stewards, data owners, data engineering and governance committees, each owning a fraction of the data value chain.

Data Governance 98

Data Governance Government Data Data Engineering 98

Data Engineering Podcast

JUNE 5, 2022

Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Data governance is the binding force between these two parts of the organization. At what point does a lack of an explicit governance policy become a liability?

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Precisely

MAY 13, 2024

Key Takeaways: Data integration is vital for real-time data delivery across diverse cloud models and applications, and for leveraging technologies like generative AI. The right data integration solution helps you streamline operations, enhance data quality, reduce costs, and make better data-driven decisions.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Towards Data Science

JUNE 8, 2023

We examine which team structures are the best suited for efficiently improving data quality. Sure, data quality is everyones’ problem. Some organizations will attempt to diffuse the responsibility widely across data stewards, data owners, data engineering and governance committees, each owning a fraction of the data value chain.

Data Governance 73

Data Governance Government Generalist Data Engineering 73

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. In this episode Inbar Yogev and Lior Winner share the journey that they and their teams at Riskified have been on for their data platform. They also discuss how they have established a guild system for training and supporting data professionals in the organization.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Knowledge Hut

MARCH 28, 2024

Data science has become one of the most trending fields today. Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 This demonstrates the increasing need for Microsoft Certified Data Engineers. That’s where data engineers are on the go.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Monte Carlo

JANUARY 10, 2024

What happens when you strip away all the noise of queries and pipelines and focus on the data itself? You get down to the intrinsic data quality. What’s the difference between intrinsic and extrinsic data quality? Intrinsic data quality is the quality of data assessed independently of its use case.

Data Cleanse 52

Data Cleanse Data Engineering Data Engineer Engineering 52

Precisely

JANUARY 9, 2023

With global data creation projected to grow to more than 180 zettabytes by 2025 , it’s not surprising that more organizations than ever are looking to harness their ever-growing datasets to drive more confident business decisions. As data initiatives become more sophisticated, organizations will uncover new data quality challenges.

Data Integration 81

Data Integration Data Governance Government Data 81

Databand.ai

AUGUST 30, 2023

Building a Winning Data Quality Strategy: Step by Step Eitan Chazbani August 30, 2023 What Is a Data Quality Strategy? A data quality strategy details the processes, tools, and techniques employed to ensure your company’s data is accurate, consistent, complete, and up-to-date.

Building 52

Building Data Cleanse Data Governance Datasets 52

Data Engineering Podcast

JULY 12, 2021

Summary We have been building platforms and workflows to store, process, and analyze data since the earliest days of computing. Over that time there have been countless architectures, patterns, and "best practices" to make that task manageable. RudderStack’s smart customer data pipeline is warehouse-first.

Designing 100

Designing Data Warehouse Data Governance Consulting 100

Data Engineering Podcast

APRIL 26, 2021

Summary The reason for collecting, cleaning, and organizing data is to make it usable by the organization. In this episode Maxime Beauchemin discusses how data engineers can use Superset to provide self service access to data and deliver analytics. Give it a listen and then take it for a test drive today.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Podcast

APRIL 5, 2021

Summary One of the biggest obstacles to success in delivering data products is cross-team collaboration. This introduces a barrier to communication that is difficult to overcome, particularly in teams that have not reached a significant level of maturity in their data journey.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

FEBRUARY 24, 2020

Summary Misaligned priorities across business units can lead to tensions that drive members of the organization to build data and analytics projects without the guidance or support of engineering or IT staff. The availability of cloud platforms and managed services makes this a viable option, but can lead to downstream challenges.

IT 100

IT Data Lake Data Pipeline Media 100

Data Engineering Podcast

NOVEMBER 23, 2020

Summary One of the oldest aphorisms about data is "garbage in, garbage out", which is why the current boom in data quality solutions is no surprise. He also shares how his own product is designed and the challenges that are involved in building a system to help data engineers manage the complexity of a data platform.

Hadoop 100

Hadoop Data Pipeline BI Data Engineering 100

Data Engineering Podcast

DECEMBER 14, 2020

Summary As a data engineer you’re familiar with the process of collecting data from databases, customer data platforms, APIs, etc. At YipitData they rely on a variety of alternative data sources to inform investment decisions by hedge funds and businesses.

Data Analytics 100

Data Analytics Building MySQL Data 100

Databand.ai

AUGUST 30, 2023

Data testing tools: Key capabilities you should know Helen Soloveichik August 30, 2023 Data testing tools are software applications designed to assist data engineers and other professionals in validating, analyzing and maintaining data quality. There are several types of data testing tools.

Data Cleanse 52

Data Cleanse Data Pipeline Datasets Data Validation 52

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. No more scripts, just SQL.

Metadata 100

Metadata BI Data Warehouse Government 100

Ascend.io

JANUARY 10, 2024

Visual representation of Conway’s Law ( source ) Read More: The Chief AI Officer: Avoid The Trap of Conway’s Law Process: Ensuring Data Readiness The backbone of successful AI implementation is robust data management processes. AI models are only as good as the data they consume, making continuous data readiness crucial.

Data Pipeline 52

Data Pipeline Government Data Governance Raw Data 52

Monte Carlo

JULY 13, 2023

Data-driven organizations generate, collect, and store vast amounts of data. To effectively manage and analyze this data, data engineering teams must navigate a wide range of challenges, including data access, security, compliance, and data observability. Automating self-service access.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Governance 52

Data Engineering Podcast

OCTOBER 19, 2020

Summary In order for analytics and machine learning projects to be useful, they require a high degree of data quality. To ensure that your pipelines are healthy you need a way to make them observable. If you hand a book to a new data engineer, what wisdom would you add to it?

Machine Learning 100

Machine Learning Data Engineering Data Engineer Data 100

Databand.ai

JULY 19, 2023

Complete Guide to Data Ingestion: Types, Process, and Best Practices Helen Soloveichik July 19, 2023 What Is Data Ingestion? Data Ingestion is the process of obtaining, importing, and processing data for later use or storage in a database. However, the ultimate goal is simple: to prepare data for immediate use.

Data Ingestion 52

Data Ingestion Process Data Cleanse Data Governance 52

Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. In this episode Grant Seward explains how he built Tree Schema to be an easy to use and cost effective option for organizations to build their data catalogs.

Building 100

Building PostgreSQL BI Metadata 100

Monte Carlo

JUNE 16, 2023

Data integrity and quality may seem similar at first glance, and they are sometimes used interchangeably in everyday life, but they play unique roles in successful data management. When data has integrity, it means it wasn’t altered during storage, retrieval, or processing without authorization.

Data Integration 52

Data Integration Hospitality Banking Healthcare 52

The Modern Data Company

JULY 31, 2023

Modern Announces Partnership with Data Mesh Pioneers, ThoughtWorks In July, we collaborated with ThoughtWorks at the annual CDOIQ Conference in Cambridge, MA to discuss real-world Data Products implementation and best practices for Data Mesh. The data stack is the powerhouse behind AI/ML stacks and data applications.

Government 98

Government Data Governance Architecture Data Pipeline 98

Monte Carlo

SEPTEMBER 27, 2023

According to our annual survey , incidents of data downtime — moments when data is inaccurate, missing, or otherwise unreliable — nearly doubled year over year. This is likely driven by the finding that time-to-resolution for data quality issues increased by 166%.

Data Governance 52

Data Governance Government Data Data Pipeline 52

AltexSoft

FEBRUARY 11, 2023

Data is now one of the most valuable assets for any kind of business. The 11th annual survey of Chief Data Officers (CDOs) and Chief Data and Analytics Officers reveals 82 percent of organizations are planning to increase their investments in data modernization in 2023. What is a data architect?

Data Architect 87

Data Architect Certification Generalist Big Data 87

Data Engineering Podcast

NOVEMBER 30, 2020

Summary The first stage of every good pipeline is to perform data integration. With the increasing pace of change and the need for up to date analytics the need to integrate that data in near real time is growing. If you hand a book to a new data engineer, what wisdom would you add to it?

Data Integration 100

Data Integration Coding BI Kafka 100

Data Engineering Weekly

JUNE 18, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Data management is critical for any organization to succeed in this AI world.

Data Engineering 52

Data Engineering Data Engineer Engineering MySQL 52

Ascend.io

SEPTEMBER 5, 2023

In the fast-evolving landscape of cloud data solutions, Snowflake has consistently been at the forefront of innovation, offering enterprises sophisticated tools to optimize their data management. Snowpark is a library equipped with an API that developers can use for querying and processing data within the Snowflake Data Cloud.

IT 59

IT Scala Java Programming Language 59

Databand.ai

JULY 5, 2023

What Are Data Observability Tools? Data observability tools are software solutions that oversee, analyze, and improve the performance of data pipelines. In this article: Why Are Data Observability Tools Important? Faster Troubleshooting Data pipeline failures are expensive and damaging to organizations.

Data Pipeline 52

Data Pipeline Data Lake Data Warehouse Datasets 52

Precisely

MARCH 14, 2023

Then, data clouds from providers like Snowflake and Databricks made deploying and managing enterprise-grade data solutions much simpler and more cost-effective. Now, almost any company can build a solid, cost-effective data analytics or BI practice grounded in these new cloud platforms.

Data Integration 52

Data Integration High Quality Data BI Data 52

Striim

NOVEMBER 8, 2023

Data Mesh is revolutionizing event streaming architecture by enabling organizations to quickly and easily integrate real-time data, streaming analytics, and more. In this article, we will explore the advantages and limitations of data mesh, while also providing best practices for building and optimizing a data mesh with Striim.

Architecture 52

Architecture Generalist Government Datasets 52