Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. platform design, pipeline design, data usage, etc.) Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

FEBRUARY 20, 2022

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. that you need to account for in the design?

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Data Engineering Podcast

MARCH 27, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. How have the design and goals of the system changed or evolved since you started working on it? Can you describe what Privacera is and the story behind it?

Data Governance 130

Data Governance Government Cloud Building 130

François Nguyen

MARCH 22, 2021

Data & Metadata : the data of the data product in many possible storages if needed but also the metadata (data on data) Infrastructure : you will need compute & storage but with the Serverless philisophy, we want to make it totally transparent and stay focus on the first two dimensions. What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. Yeah…I can do that already!

Process 80

Process Kafka Datasets SQL 80

Data Engineering Podcast

JANUARY 23, 2022

In this episode he shares his career journey, the challenges related to management of data professionals, and the platform design that he and his team have built to power analytics at a large company. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow.

Building 100

Building Management Data Pipeline Metadata 100

Engineering at Meta

AUGUST 29, 2023

These three considerations shaped and influenced our design decisions. How BentoOperator works In order to address the majority of the concerns highlighted above, we perform the notebook execution state in a container without access to the network. The overall design for BentoOperator. An example of this is the SQL cell.

SQL 81

SQL Metadata Coding Data Analysis 81

LinkedIn Engineering

DECEMBER 19, 2023

The framework incorporates proactive measures to monitor critical events, provides deployment insights, and issues proper notifications to enable seamless deployments. The new Rolling Upgrade framework The new RU orchestration design significantly enhanced our big data components deployment process. Accessibility of all namenodes.

Big Data 92

Big Data Hadoop Metadata Data 92

Netflix Tech

MARCH 7, 2024

Since its inception , Metaflow has been designed to provide a human-friendly API for building data and ML (and today AI) applications and deploying them in our production infrastructure frictionlessly. Thanks to Arrow, data can be accessed through these libraries in a zero-copy fashion.

Systems 90

Systems Media Machine Learning Data Warehouse 90

Rockset

SEPTEMBER 26, 2023

AI-powered applications almost always need access to real-time data to deliver accurate results in a responsive user experience that the market has come to expect. As smart as ChatGPT appears to be, it can’t summarize current events accurately if it was last trained a year ago and not told what’s happening now.

Metadata 64

Metadata Kafka Cloud Database 64

Data Engineering Podcast

MARCH 22, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. Fivetran/Airbyte/Meltano/custom scripts) What are the moving pieces in a CDC workflow that need to be considered as you are designing the system?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Monte Carlo

MAY 18, 2023

You may even already have access to one of them. No-code Editor : It’s possible to augment the lineage captured from machine metadata with a drag and drop no-code editor. One reviewer said, “I’m in the process of setting up OpenMetadata in our org, ingesting metadata from Athena, Superset and Dagster.

Metadata 64

Metadata Data Lake Government Data Governance 64

Netflix Tech

SEPTEMBER 29, 2022

Timestone clients can create queues, enqueue messages with user-defined deadlines and metadata, then dequeue these messages in an earliest-deadline-first (EDF) fashion. messages that belong to queue X and have metadata Y”) is also supported. Allow for dequeueing and queue depth querying using filters (metadata key-value pairs) 4.

Systems 85

Systems Metadata Media Kafka 85

Netflix Tech

SEPTEMBER 28, 2021

We identified common design patterns and architectures scattered across various tools which were all duplicating efforts in some way or another. Metadata Driven : Plugin modules are defined from a configuration which could be injected at any point in the application lifecycle.

Building 96

Building Metadata Designing Coding 96

Data Engineering Podcast

MAY 3, 2021

They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. How does security factor into the design of robust DataOps systems?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

AUGUST 5, 2019

Upcoming events include the O’Reilly AI Conference, the Strata Data Conference, and the combined events of the Data Architecture Summit and Graphorum. Can you start by explaining what Amundsen is and the problems that it was designed to address? What has been the impact of Amundsen on the workflows of data teams at Lyft?

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

AltexSoft

AUGUST 29, 2023

A data lake is a centralized repository designed to hold vast volumes of data in its native, raw format — be it structured, semi-structured, or unstructured. Such an object storage model allows metadata tagging and incorporating unique identifiers, streamlining data retrieval and enhancing performance. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. For example, grouping the ones about metadata, discoverability, and column naming might have made a lot of sense.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Netflix Tech

DECEMBER 8, 2021

We currently process tens of millions of log records every minute and analyze these events to perform in-house custom detections. Once ingested, findings are then enriched and processed with additional metadata collected from Netflix’s internal data sources. Hopefully, we’ve made the mistakes and learned from those experiences.

AWS 78

AWS Metadata Cloud Utilities 78

Ascend.io

NOVEMBER 28, 2023

This includes job process sequencing, metadata synchronization, cataloging of data processing results, triggering data loads and quality checks, detecting when one task is done and triggering another, and setting the timing of scripts and system commands. In contrast, general data orchestration lacks this level of contextual insight.

Data Pipeline 52

Data Pipeline IT Data Metadata 52

Knowledge Hut

APRIL 25, 2024

Intrusion detection systems (IDS) are designed to identify suspicious and malicious activity through network traffic. Attackers continuously develop new exploits and affect techniques that are designed to affect the defense. Some attacks are for obtaining user credentials that grant access to the network and data.

Systems 52

Systems Recruitment Machine Learning Certification 52

Cloudera

SEPTEMBER 17, 2020

By enabling their event analysts to monitor and analyze events in real time, as well as directly in their data visualization tool, and also rate and give feedback to the system interactively, they increased their data to insight productivity by a factor of 10. . This led them to fall behind. Our solution: Cloudera Data Visualization.

Unstructured Data 80

Unstructured Data Data Warehouse Pharmaceutical MySQL 80

Databand.ai

JUNE 28, 2023

Most were designed for the best-case scenario. More people accessing the data and running their own pipelines and their own transformations causes errors and impacts data stability. Beware the black box scenario You know what goes in. You know what comes out. But what happens in between? And why the discrepancy?

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Maxime Beauchemin

JANUARY 20, 2017

In retrospect, complex SCD modeling techniques are not intuitive and reduce accessibility. Data engineers are also the “librarians” of the data warehouse, cataloging and organizing metadata, defining the processes by which one files or extract data from the warehouse.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Weekly

NOVEMBER 6, 2022

link] Pinterest: How Pinterest Leverages Realtime User Actions in Recommendation to Boost Homefeed Engagement Volume Looping in real-time user interaction events with the recommendation engine can significantly improve the user experience. Every data processing engine has one metadata store to integrate.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Hepta Analytics

FEBRUARY 14, 2022

This week, we got to think about our data ingestion design. The data is easily accessible and is easy to update. IT could also have sensors, subclass of operators that which wait for external event to happen. Designing the workflow is dependent on the use case. This was used to test our setup.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Cloudera

FEBRUARY 9, 2021

The design cost of this however is that most often the source data for a given partition is not growing fast enough to be accumulated at the expected file size within minutes. In the real-time layer, Kafka automatically cleans the historical events with configurable retention. Design Detail. Low Maintenance. Partition Strategy.

Data Warehouse 69

Data Warehouse Cloud Kafka Cloud Storage 69

Monte Carlo

APRIL 16, 2022

For businesses to truly gain value from the massive amounts of data they possess, teams need to take a step back and model their business semantically before defining entities and events through code for the express purpose of analytics. An engineering owner is established for each entity and event.

Data Warehouse 52

Data Warehouse Architecture Data Government 52

Knowledge Hut

DECEMBER 21, 2023

Cloud-based storage and analytical platform helps business to easily access their data whenever required irrespective of where it is located physically. Security : Cloud providers offer various security features which includes data encryption and access control, that can help businesses to protect their data from unauthorized access.

Cloud Computing 52

Cloud Computing Cloud Algorithm Healthcare 52

Cloudera

SEPTEMBER 1, 2020

It is designed to simplify deployment, configuration, and serviceability of Solr-based analytics applications. Includes a drag-n-drop style, GUI-based Search Dashboard Designer. Hue also has a Index Creation Designer – but we do not recommend using this until after GA. data best served through Apache Solr).

Cloud Storage 67

Cloud Storage Unstructured Data AWS Analytics Application 67

Cloudera

AUGUST 21, 2020

For example, if a credit card was used in the United States and shortly afterward the same card was used in Spain to withdraw the same amount, these two events in isolation could appear legitimate. However, in the context of time and geography, these two events point to a pattern of fraud.

Banking 60

Banking Kafka Cloud Storage Government 60

Netflix Tech

JUNE 4, 2019

Because microprocessors are so fast, computer architecture design has evolved towards adding various levels of caching between compute units and the main memory, in order to hide the latency of bringing the bits to the brains. We also want to leverage kernel PMC events to more directly optimize for minimal cache noise.

Machine Learning 100

Machine Learning Metadata Systems Data Collection 100

Cloudera

SEPTEMBER 15, 2022

July brings summer vacations, holiday gatherings, and for the first time in two years, the return of the Massachusetts Institute of Technology (MIT) Chief Data Officer symposium as an in-person event. A key area of focus for the symposium this year was the design and deployment of modern data platforms. What is a data fabric?

Data Lake 79

Data Lake Analytics Application Cloud Storage Architecture 79

Zalando Engineering

APRIL 4, 2018

Other than our in-house equivalent, the esteemed Zalando HackWeek , we were both completely inexperienced in participating in such events. Our event of choice was #picturepunk. It was an event hosted by the German Press Agency (DPA) and in cooperation with Adobe, Microsoft, and Google News Lab.

Media 52

Media Software Engineer Software Engineering Metadata 52

Monte Carlo

SEPTEMBER 20, 2022

While solutions like dbt tests failed to achieve scalability, alternatives like Google Cloud Platform’s native solutions didn’t appear to be designed for data at all. This left Dylan’s team with a gap to fill. These data monitors combine to track data accuracy, data completeness and general adherence to internal business rules.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Rockset

APRIL 18, 2023

Turn events into real-time features: Rockset turns events into features in real time with SQL ingest transformations. Ensure service-levels at scale: Rockset meets the strict latency requirements of real-time analytics and is designed for high availability and durability with no scheduled downtime.

Machine Learning 52

Machine Learning Datasets Database Data Pipeline 52

Rockset

NOVEMBER 7, 2018

Amazon S3) and Event Streams (e.g. The "Five-Minute Rule" captures succinctly the inherent economic trade-off between memory and storage in data store design (Appuswamy/ADMS@VLDB 2017). Such a database is designed to take full advantage of the abundance of networking, processing, and storage resources available in the cloud.

Database 40

Database Cloud Cloud Storage MySQL 40