Data Engineering Weekly

FEBRUARY 4, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Joe Reis: Definition of Data Modeling & What Data Modeling Is not Joe raised a very fundamental question in data engineering.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Knowledge Hut

APRIL 29, 2024

Replacing inherent social structures of a physical office, team stand-ups and in-person collaboration with tools like Slack, Microsoft Teams and Proofhub, knowledge sharing and learning new things can become much harder. Dig deep with books and blogs Books teach you to a greater depth and help you cross reference.

Data Science 98

Data Science Media Technology Big Data 98

Databand.ai

AUGUST 30, 2023

Data testing tools: Key capabilities you should know Helen Soloveichik August 30, 2023 Data testing tools are software applications designed to assist data engineers and other professionals in validating, analyzing and maintaining data quality. There are several types of data testing tools.

Data Cleanse 52

Data Cleanse Data Pipeline Datasets Data Validation 52

Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Databand.ai

AUGUST 30, 2023

Building a Winning Data Quality Strategy: Step by Step Eitan Chazbani August 30, 2023 What Is a Data Quality Strategy? A data quality strategy details the processes, tools, and techniques employed to ensure your company’s data is accurate, consistent, complete, and up-to-date.

Building 52

Building Data Cleanse Data Governance Datasets 52

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Databand.ai

AUGUST 30, 2023

Data Testing Tools: Key Capabilities and 6 Tools You Should Know Helen Soloveichik August 30, 2023 What Are Data Testing Tools? Data testing tools are software applications designed to assist data engineers and other professionals in validating, analyzing, and maintaining data quality.

Data Cleanse 52

Data Cleanse Data Validation Data Pipeline Datasets 52

Databand.ai

JULY 11, 2023

Data Quality Platform: Benefits, Key Features, and How to Choose Eric Jones July 11, 2023 What Is a Data Quality Platform? A data quality platform is a software solution designed to help organizations manage, maintain, and improve the quality of their data.

Data Cleanse 52

Data Cleanse Telecommunication High Quality Data BI 52

Databand.ai

JULY 19, 2023

Data Lineage Tools: Key Capabilities and 5 Notable Solutions Ryan Yackel July 19, 2023 What Are Data Lineage Tools? Data lineage tools provide a visual representation of your data’s journey across multiple systems and transformations. Data lineage tools are not a new concept.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Engineering at Meta

MARCH 20, 2024

Here’s how we are improving the RTC video quality for our apps with tools like the AV1 codec, the challenges we face, and how we mitigate those challenges. The last few decades have seen tremendous improvements in mobile phone camera quality as well as video quality for streaming video services. db on average.

Algorithm 86

Algorithm Manufacturing Media Coding 86

Data Engineering Weekly

APRIL 14, 2024

Every survey demonstrates an increase in productivity in software development with copilot-type tools. Discover how a universal semantic layer is transforming modern business intelligence, making data more accessible and reliable for organizations striving for informed business decisions.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Knowledge Hut

APRIL 26, 2024

Also called data storage areas , they help users to understand the essential insights about the information they represent. Machine Learning without data sets will not exist because ML depends on data sets to bring out relevant insights and solve real-world problems. In the real world, data sets are huge.

Datasets 98

Datasets Machine Learning Deep Learning Finance 98

Databand.ai

JUNE 8, 2023

Observability in Your Data Pipeline: A Practical Guide Eitan Chazbani June 8, 2023 Achieving observability for data pipelines means that data engineers can monitor, analyze, and comprehend their data pipeline’s behavior. This ensures the reliability and accuracy of data-driven decision-making processes.

Data Pipeline 52

Data Pipeline Bytes Raw Data Data Collection 52

Databand.ai

AUGUST 30, 2023

The DataOps framework is a set of practices, processes, and technologies that enables organizations to improve the speed, accuracy, and reliability of their data management and analytics operations. The core philosophy of DataOps is to treat data as a valuable asset that must be managed and processed efficiently.

Data Governance 52

Data Governance Data Pipeline Government Data Cleanse 52

Databand.ai

JULY 17, 2023

DataOps , short for Data Operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data management processes. It aims to streamline the entire data lifecycle—from ingestion and preparation to analytics and reporting.

Data Pipeline 52

Data Pipeline Machine Learning High Quality Data BI 52

Databand.ai

JULY 19, 2023

Complete Guide to Data Ingestion: Types, Process, and Best Practices Helen Soloveichik July 19, 2023 What Is Data Ingestion? Data Ingestion is the process of obtaining, importing, and processing data for later use or storage in a database. However, the ultimate goal is simple: to prepare data for immediate use.

Data Ingestion 52

Data Ingestion Process Data Cleanse Data Governance 52

Snowflake

NOVEMBER 30, 2023

At TCS , we help companies shift their enterprise data warehouse (EDW) platforms to the cloud as well as offering IT services. We’re extremely familiar with just how tricky a cloud migration can be, especially when it involves moving historical business data. How many tables and views will be migrated, and how much raw data?

Data Warehouse 91

Data Warehouse Banking Data Cloud 91

Cloudera

APRIL 11, 2024

Businesses around the world rely on an extensive network of partnerships to deliver quality customer experiences—and it’s no different here at Cloudera. By bringing together various partners, we provide comprehensive solutions that address specific use cases and modernize data architecture.

Food 85

Food Data Architecture Programming Architecture 85

Snowflake

FEBRUARY 22, 2024

A solid data foundation is critical to being able to successfully leverage AI solutions, and Snowflake’s platform allows both marketing solution providers and marketing professionals to capitalize on this movement of AI in marketing with data at the core. All these features make it faster and easier for marketers to experiment.

Technology 81

Technology Machine Learning Utilities Designing 81

Databand.ai

AUGUST 30, 2023

Data Consistency vs Data Integrity: Similarities and Differences Joseph Arnold August 30, 2023 What Is Data Consistency? Data consistency refers to the state of data in which all copies or instances are the same across all systems and databases. Data consistency is essential for various reasons.

Data Integration 52

Data Integration Data Cleanse Data Validation High Quality Data 52

Knowledge Hut

MARCH 18, 2024

In this blog post, I will guide you as to what a software project manager is. Key aspects of software project management include defining project goals, creating project plans, allocating resources, managing risks, monitoring progress, and ensuring quality. Quality Assurance: Delivering high-quality software is paramount.

Project 52

Project Management Certification Software Engineer 52

Workfall

SEPTEMBER 18, 2023

Reading Time: 9 minutes Imagine your data as pieces of a complex puzzle scattered across different platforms and formats. This is where the power of data integration comes into play. Meet Airbyte, the data magician that turns integration complexities into child’s play. In this blog, we will cover: What is Airbyte?

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

Knowledge Hut

MARCH 18, 2024

How you track and analyze project data is a critical component that can make all the difference in whether a project you manage succeeds or fails. Work performance data is even more important than this process, which is necessary. In this blog, I will discuss how WPD can be a great tool in project management and how you can master it.

Raw Data 52

Raw Data Data Designing Project 52

Knowledge Hut

MARCH 7, 2024

BAs are thus expected to have knowledge about key concepts of business analysis and be skilled in using different tools and techniques for eliciting, analyzing, and managing requirements. The BA may need to make decisions on which tools or techniques to use, which stakeholders to consult, which solution options to take, and so on.

Business Analyst 98

Business Analyst Consulting Data Analysis Finance 98

Cloudera

DECEMBER 1, 2023

These government-led efforts have had a profound impact on the development and adoption of AI solutions in the public sector, paving the way for a future where data-driven decision-making and automation are the norm. This requires a holistic approach that addresses the key areas of security, governance, and trustworthy data.

Building 104

Building Government Transportation Data Governance 104

Pinterest Engineering

FEBRUARY 22, 2024

In this blog post, we share our journey of unlocking AI-assisted development, from the initial idea to the General Availability (GA) stage. While Pinterest possesses extensive in-house AI expertise and builds many of our developer tools, we recognized developing everything from scratch was not essential to our core business.

Scala 78

Scala Engineering Software Engineer Software Engineering 78

Data Engineering Weekly

JANUARY 21, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Interesting article on the impact of search engine optimization (SEO) on the quality of search engine results. Visit rudderstack.com to learn more.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Knowledge Hut

MARCH 26, 2024

By working closely with management, Quality Managers play a vital role in ensuring procedures are effectively implemented throughout the organization. They collaborate closely with leadership to translate quality objectives into actionable plans. Who is the Quality Manager? Who is the Quality Manager?

Management 52

Management Manufacturing Data Analysis Certification 52

Netflix Tech

NOVEMBER 17, 2022

Bampis , Li-Heng Chen and Zhi Li When you are binge-watching the latest season of Stranger Things or Ozark, we strive to deliver the best possible video quality to your eyes. To do so, we continuously push the boundaries of streaming video quality and leverage the best video technologies.

Media 117

Media Architecture Algorithm Designing 117

Databand.ai

AUGUST 30, 2023

7 Data Testing Methods, Why You Need Them & When to Use Them Helen Soloveichik August 30, 2023 What Is Data Testing? Data testing involves the verification and validation of datasets to confirm they adhere to specific requirements. This is part of a series of articles about data quality.

Data Validation 52

Data Validation Data Integration Data Database 52

Lyft Engineering

APRIL 24, 2024

At Lyft, data scientists tackle challenging technical problems every day. To support and empower our data scientists, Lyft’s Technical Learning Council (TLC) provides diverse and high-quality continuous learning opportunities to hone their technical skills. See our previous blog post for more on the motivations behind TLC.

Data Science 66

Data Science Building Computer Science Software Engineer 66

Edureka

APRIL 19, 2024

In the fast-changing world of artificial intelligence, a new tool called Midjourney has captured the attention of artists, designers, and anyone interested in creativity. But before that, let’s outline the topics that we will be discussing in this blog: What Is Midjourney? This data can be in the form of text, audio, or image.

Algorithm 52

Algorithm Deep Learning Machine Learning Media 52

DataKitchen

FEBRUARY 23, 2024

DataKitchen Resource Guide To Data Journeys & Data Observability & DataOps Data (and Analytic) Observability & Data Journey – Ideas and Background Data Journey Manifesto and Why the Data Journey Manifesto?

Data Validation 48

Data Validation Data Data Pipeline Data Solutions 48

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. Imagine a team of skilled data engineers on an exciting quest to transform raw data into a treasure trove of insights. These guards are tests in DBT.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Databand.ai

JULY 5, 2023

What Are Data Observability Tools? Data observability tools are software solutions that oversee, analyze, and improve the performance of data pipelines. By employing these tools, teams can proactively detect issues before they become larger problems that affect business operations.

Data Pipeline 52

Data Pipeline Data Lake Data Warehouse Datasets 52

Cloudera

OCTOBER 31, 2023

This kind of question lends itself perfectly to data science approaches that enable quick and intuitive analysis of data across multiple sources. So far we’ve talked about quantity, but just as important is quality. This variable is largely out of your control, and can be dependent on the region you live in.

Data Mining 77

Data Mining Portfolio Machine Learning Data Science 77