Precisely

APRIL 11, 2023

It’s a common scenario across industries: a new strategic initiative is launching, and you need trustworthy, relevant data to analyze and support the best outcomes for the business. What existing data does the organization have to support my initiative? What existing data does the organization have to support my initiative?

Accessible 52

Accessible Accessibility Metadata Data Integration 52

Knowledge Hut

APRIL 25, 2024

Intrusion detection systems (IDS) are designed to identify suspicious and malicious activity through network traffic. So, let's get to know the meaning of an intrusion detection system and how it works. What Is a n Intrusion Detection System? Why Are Intrusion Detection Systems Important? and how it works.

Systems 52

Systems Recruitment Machine Learning Certification 52

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 85

Systems Metadata Media Kafka 85

Precisely

MAY 13, 2024

Key Takeaways: Data integration is vital for real-time data delivery across diverse cloud models and applications, and for leveraging technologies like generative AI. As enterprise technology landscapes grow more complex, the role of data integration is more critical than ever before.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Data Engineering Podcast

DECEMBER 4, 2022

Summary Business intelligence is the foremost application of data in organizations of all sizes. The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. Zing Data is building a mobile native platform for business intelligence.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Data Engineering Weekly

MARCH 24, 2024

link] Kai Waehner: The Data Streaming Landscape 2024 This is a comprehensive overview of the state of the data streaming landscape in 2024. The author goes beyond comparing the tools to various offerings from streaming vendors in stream processing and Kafka protocol-supported systems.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Knowledge Hut

MARCH 7, 2024

The year 2024 saw some enthralling changes in volume and variety of data across businesses worldwide. The surge in data generation is only going to continue. Foresighted enterprises are the ones who will be able to leverage this data for maximum profitability through data processing and handling techniques.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Christophe Blefari

FEBRUARY 3, 2023

Delivering the data news ( credits ) Hey you, it's already February. The article shows our Netflix art creators are using past data to create new artworks. ebay, Variable Hub a data access layer for risk decisioning — Looks like a feature store but for risk topics. Every week same analysis for me.

BI 130

BI Google Cloud SQL Machine Learning 130

Data Engineering Podcast

JULY 31, 2022

Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

AUGUST 6, 2022

Summary Data mesh is a frequent topic of conversation in the data community, with many debates about how and when to employ this architectural pattern. The team at AgileLab have first-hand experience helping large enterprise organizations evaluate and implement their own data mesh strategies.

Metadata 100

Metadata MongoDB MySQL Business Intelligence 100

Data Engineering Podcast

MARCH 18, 2023

Summary As with all aspects of technology, security is a critical element of data applications, and the different controls can be at cross purposes with productivity. In this episode Yoav Cohen from Satori shares his experiences as a practitioner in the space of data security and how to align with the needs of engineers and business users.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Data Engineering Podcast

AUGUST 28, 2022

For data engineers, this is a monumental undertaking. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. The only thing worse than having bad data is not knowing that you have it.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Data Engineering Podcast

MARCH 27, 2022

Summary Data governance is a practice that requires a high degree of flexibility and collaboration at the organizational and technical levels. The growing prominence of cloud and hybrid environments in data management adds additional stress to an already complex endeavor.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Weekly

DECEMBER 29, 2022

Data catalogs are the most expensive data integration systems you never intended to build. Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. How happy are you with your data catalogs?

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Metadata layer 4. API layer 5.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Metadata layer 4. API layer 5.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Arjun Mohnot , Jenchang Ho , Anthony Quigley , Xing Lin , Anil Alluri , Michael Kuchenbecker LinkedIn operates one of the world’s largest Apache Hadoop big data clusters. Historically, deploying code changes to Hadoop big data clusters has been complex.

Big Data 92

Big Data Hadoop Metadata Data 92

Data Engineering Podcast

SEPTEMBER 4, 2022

Summary Data observability is a product category that has seen massive growth and adoption in recent years. Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Ascend.io

FEBRUARY 28, 2024

The rise of generative AI is changing more than just technology; it’s reshaping our professional landscapes — and yes, data engineering is directly experiencing the impact. How does AI recalibrate the workload and priorities of data teams? How can data engineers harness the power of AI?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Weekly

MARCH 10, 2024

Editor’s Note: Chennai Meetup Wrap-Up & Preparation work started for DEWCon I am so grateful for the enthusiastic participants who made our Chennai Data Heroes- Community for Data Folks meetup vibrant! Big thanks to our insightful speakers, Hareshkumar Selvakumar - Talks about his work on Data Products for PayPal.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

phData: Data Engineering

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Data Engineering Weekly

JUNE 29, 2023

But hey, I met my friends after a long time and got my copy of “ Fundamentals of Data Engineering ” signed by Joe Reis & Matt Housely. If you’re starting data engineering, I highly recommend reading it. There are many exciting announcements from both the data conferences, and I want to highlight a few themes here.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

Monte Carlo

JANUARY 29, 2024

It’s one thing to get your data into a modern data cloud. Monte Carlo is thrilled to be part of the Snowflake Horizon partner ecosystem as we leverage many of the pre-built features Snowflake provides in order to help organizations reduce their data downtime and improve data quality at scale.

Metadata 64

Metadata High Quality Data Data Pipeline Machine Learning 64

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Monte Carlo

MAY 18, 2023

Data lineage tools are like detectives that help data professionals quickly sort through the tangled webs of interdependencies that make up the modern data stack. The first instinct of many early stage companies or budget strapped data teams is to turn to open source data lineage tools.

Metadata 64

Metadata Data Lake Government Data Governance 64

Precisely

MAY 31, 2023

To innovate, compete, and grow in the current macroeconomic environment, enterprises must approach data strategically. A sound data strategy doesn’t happen by accident; it’s built on a foundation of data integrity , including accuracy, consistency, and rich context. Many organizations still struggle with data integrity.

Data Integration 52

Data Integration Metadata Data Governance Data 52

Netflix Tech

NOVEMBER 14, 2023

In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing. Pipelines After Psyberg Let’s explore how different modes of Psyberg could help with a multistep data pipeline. Audit Run various quality checks on the staged data.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Cloudera

MARCH 3, 2022

Whether the enterprise uses dozens or hundreds of data sources for multi-function analytics, all organizations can run into data governance issues. Bad data governance practices lead to data breaches, lawsuits, and regulatory fines — and no enterprise is immune. . Everyone Fails Data Governance. In 2019, the U.K.’s

Data Governance 100

Data Governance Government Metadata Medical 100

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Snowflake

APRIL 29, 2024

DORA: Building a More Secure Financial System DORA, enacted in January 2023, moves beyond reactive measures, requiring FEs and their service providers to proactively identify vulnerabilities, prevent disruptions and plan for swift recovery from incidents. or better) for data in transit. How can Snowflake Help?

Transportation 84

Transportation Data Governance Government Consulting 84

Towards Data Science

DECEMBER 1, 2023

Pet Project for Data/Analytics Engineers: Explore Modern Data Stack Tools — dbt Core, Snowflake, Fivetran, GitHub Actions. This hands-on experience will allow you to develop an end-to-end data lifecycle, from extracting data from your Google Calendar to presenting it in a Snowflake analytics dashboard. See Github repo.

Data Engineering 67

Data Engineering Data Engineer Project Engineering 67

Cloudyard

DECEMBER 7, 2023

Read Time: 3 Minute, 0 Second In this blog post, we will delve into Snowflake’s Listing feature, which operates on the data sharing provider and consumer model. Data » Provider Studio >> select Review Provider Terms & Conditions. Data Replication Once done we will get the mail.

Metadata 52

Metadata SQL Data Cloud 52

Cloudera

JUNE 7, 2022

We live in a hybrid data world. In the past decade, the amount of structured data created, captured, copied, and consumed globally has grown from less than 1 ZB in 2011 to nearly 14 ZB in 2020. Impressive, but dwarfed by the amount of unstructured data, cloud data, and machine data – another 50 ZB.

IT 112

IT Unstructured Data Data Architecture Government 112

Knowledge Hut

JULY 24, 2023

In the world of databases, data independence plays a vital role in making sure the flexibility and adaptability of database systems. Data independence tells us about the ability to modify the database schema or organization without affecting the applications that use the data. What is Data Independence of DBMS?

Database Design 52

Database Design Relational Database Metadata Database 52

Data Engineering Weekly

JANUARY 6, 2023

Author: Chris Ricommni Recap Github: [link] [Don't forget to star it] I’m excited to release Recap , a dead simple data catalog for engineers written in Python. Recap makes it easy for engineers to build infrastructure and tools that need metadata. Unlike traditional data catalogs, Recap is designed to power software.

Metadata 40

Metadata ETL Tools MySQL Data Lake 40