Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. platform design, pipeline design, data usage, etc.) Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Cloudera

NOVEMBER 29, 2023

The Model Registry is designed to streamline these processes, offering a variety of tools and features. It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. You can track each iteration, compare changes, and ensure that you always have access to the version that suits your needs.

Machine Learning 84

Machine Learning Metadata Cloud Project 84

Data Engineering Podcast

JANUARY 23, 2022

In this episode he shares his career journey, the challenges related to management of data professionals, and the platform design that he and his team have built to power analytics at a large company. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow.

Building 100

Building Management Data Pipeline Metadata 100

phData: Data Engineering

SEPTEMBER 19, 2023

This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. Data modeling methodologies are systematic approaches used to design and define the structure and relationships of data within a system.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Databand.ai

JULY 19, 2023

Learn more in our detailed guide to data lineage visualization (coming soon) Integration with Multiple Data Sources Data lineage tools are designed to integrate with a wide range of data sources, including databases, data warehouses, and cloud-based data platforms. One of the unique features of Atlan is its human-centric design.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Cloudera

AUGUST 26, 2021

Ozone natively provides Amazon S3 and Hadoop Filesystem compatible endpoints in addition to its own native object store API endpoint and is designed to work seamlessly with enterprise scale data warehousing, machine learning and streaming workloads. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 109

Data Science Cloud Hadoop Metadata 109

Databand.ai

AUGUST 30, 2023

This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management. These metrics should be aligned with your organization’s goals and objectives, and should be designed to track the progress of your data quality strategy over time.

Building 52

Building Data Cleanse Data Governance Datasets 52

Workfall

AUGUST 23, 2023

These extensions are designed to enhance and customize the browsing experience by adding new features, modifying existing ones, or integrating with other web services. Once we access the browser console, we’ll see the weather data containing all the necessary information provided by the OpenWeather API.

Metadata 83

Metadata Coding Project Utilities 83

Data Engineering Podcast

MARCH 22, 2021

Your host is Tobias Macey and today I’m interviewing Raghu Murthy about his recent work of making change data capture more accessible and maintainable Interview Introduction How did you get involved in the area of data management? How has the design evolved as you have grown the scale and sophistication of your system?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

AltexSoft

FEBRUARY 11, 2023

This suggests that today, there are many companies that face the need to make their data easily accessible, cleaned up, and regularly updated. A data architect is an IT professional responsible for the design, implementation, and maintenance of the data infrastructure inside an organization. What is a data architect?

Data Architect 87

Data Architect Certification Generalist Big Data 87

Cloudera

JUNE 7, 2022

Cloudera Data Platform (CDP) is designed to address the critical requirements for modern data architectures today and tomorrow. Common security, governance, metadata, replication, and automation enable CDP to operate as an integrated system. Companies know that secure, well-governed and accessible data, on-demand, fuels growth.

IT 108

IT Unstructured Data Data Architecture Government 108

Rockset

NOVEMBER 14, 2022

In this blog, we will describe the SSB configuration, results and performance enhancements. Row-Store Cache We designed a Multiversion Concurrency Control (MVCC) cache for the row store to reduce the overhead of meta operations and thereby query latency when the working set fits into memory.

Metadata 52

Metadata Datasets Cloud Accessible 52

LinkedIn Engineering

DECEMBER 19, 2023

The new Rolling Upgrade framework The new RU orchestration design significantly enhanced our big data components deployment process. The new orchestrator agent design offers versatility and significantly improves the big data deployment process, making it smoother and less prone to issues. Accessibility of all namenodes.

Big Data 92

Big Data Hadoop Metadata Data 92

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. But first, let’s define the data mesh design pattern. A schema designer who moves from project to project may not understand the nuances behind the requirements of each data consumer.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Cloudera

DECEMBER 10, 2020

Drawing on more than a decade of experience in building and deploying massive scale data platforms on economical budgets, Cloudera has designed and delivered a cost-cutting cloud-native solution – Cloudera Data Warehouse (CDW), part of the new Cloudera Data Platform (CDP). CDW was designed to aggressively keep cloud costs under control.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Cloudera

JULY 15, 2021

This blog post provides an overview of best practice for the design and deployment of clusters incorporating hardware and operating system configuration, along with guidance for networking and security as well as integration with existing enterprise infrastructure. A minimum ensemble of 3 is required to achieve a majority consensus.

Architecture 97

Architecture Cloud Kafka Hadoop 97

Monte Carlo

MARCH 3, 2021

Our SOC 2 designation means that Monte Carlo has designed a set of internal controls, systems, policies, and procedures that meet industry best practices for protecting our customer data. We designed our product without having to store or access individual records, PII, or any other sensitive information. What’s next?

Metadata 52

Metadata Data Lake Business Intelligence Certification 52

Workfall

AUGUST 14, 2023

In this blog, we will explore the benefits of using DBT and Airflow together and how they make data pipeline management easier than ever before. In this blog, we will cover: Understanding DBT and Airflow What is DBT? Initialize the Airflow metadata database by running airflow initdb in your terminal. What is Airflow?

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Cloudera

APRIL 26, 2021

This can be done via the Ranger webUI, accessible from Cloudera Manager. But first, let’s take a quick overview of HBase method for access control. HBase Access Levels. HBase access levels are granted independently of each other and allow for different types of operations at a given scope. HBase Authorization.

Finance 94

Finance Metadata Accessible Accessibility 94

Cloudera

JANUARY 19, 2022

We continue to strengthen CDP’s security, governance and metadata management enabling customers to safely run complex data workloads spanning multiple clouds (public and/or private). CDP’s core design principles focus on providing a pre-integrated solution for a combination of use cases.

Database 87

Database Cloud Data Warehouse Data Lake 87

Netflix Tech

DECEMBER 8, 2021

Once ingested, findings are then enriched and processed with additional metadata collected from Netflix’s internal data sources. Step Function DAG for S3 Public Access Block Remediation Diagram from a remediation to enable S3’s public access block on a non-compliant bucket.

AWS 78

AWS Metadata Cloud Utilities 78

Data Engineering Podcast

AUGUST 5, 2019

Can you start by explaining what Amundsen is and the problems that it was designed to address? Once a dataset has been located, how does Amundsen simplify the process of accessing that data for analysis or further processing? Links Amundsen Data Council Presentation Strata Presentation Blog Post Lyft Airflow Podcast.__init__

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Data Engineering Weekly

NOVEMBER 6, 2022

I plan to write a series of blogs on Schemata and Data Contract in the coming weeks. Martin kindly stepped in for me to give the update for my promised blog posts. The blog narrates how Azure InterpretML service can help to understand the ML models' predictions better. I know I told you this before, so George R.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Ripple Engineering

MARCH 30, 2023

This blog outlines Ripple’s general design and approach for machine learning model lifecycle management using Databricks. Challenges and Assumptions We need to track model files and all related metadata for models like training metrics, date trained, etc. and store this information for a long time.

Machine Learning 98

Machine Learning AWS Metadata Data Collection 98

dbt Developer Hub

DECEMBER 11, 2023

The Semantic Layer lets teams define business metrics centrally, in dbt, and access them in multiple analytics tools through our semantic layer APIs. This blog post walks through the end-to-end process we used to set up product analytics for the dbt Semantic Layer using the dbt Semantic Layer.

Finance 64

Finance BI SQL Cloud 64

Maxime Beauchemin

JANUARY 20, 2017

The fact that ETL tools evolved to expose graphical interfaces seems like a detour in the history of data processing, and would certainly make for an interesting blog post of its own. In retrospect, complex SCD modeling techniques are not intuitive and reduce accessibility.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. This week, we got to think about our data ingestion design. The data is easily accessible and is easy to update. Designing the workflow is dependent on the use case. This was used to test our setup.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Cloudera

SEPTEMBER 17, 2020

A large oil and gas company was suffering over not being able to offer users an easy and fast way to access the data needed to fuel their experimentation. They implemented a culture where IT, LoB application developers, and business users were kept in tight communication, as they all accessed the same tool and could iterate faster.

Unstructured Data 80

Unstructured Data Data Warehouse Pharmaceutical MySQL 80

Cloudera

SEPTEMBER 1, 2020

It is designed to simplify deployment, configuration, and serviceability of Solr-based analytics applications. Includes a drag-n-drop style, GUI-based Search Dashboard Designer. Hue also has a Index Creation Designer – but we do not recommend using this until after GA. data best served through Apache Solr).

Cloud Storage 67

Cloud Storage Unstructured Data AWS Analytics Application 67

Knowledge Hut

SEPTEMBER 29, 2023

This blog aims to answer these questions, providing a straightforward and professional insight into the world of Azure Data Engineering. Design and develop data processing (25–30%): This component is concerned with ingesting and developing reliable data processing solutions. Then, you can create analytical layer serving designs.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Netflix Tech

MARCH 22, 2021

Focusing on impact Like many teams at Netflix, we are full-cycle developers , responsible for everything from design to ongoing development. Configuration management through Haze One of the patterns we identified across multiple teams was the need for a stable process around configuration and metadata management.

Metadata 65

Metadata Portfolio Java Engineering 65

Cloudera

JANUARY 5, 2023

In addition to managing data on premises and in public or private clouds, these modern data architectures are also intrinsically designed to handle complexities such as security and governance-related issues. They also address the concerns of IT teams in allowing access to organizational data. .

Banking 52

Banking Machine Learning Insurance Data Architecture 52

DataKitchen

MAY 4, 2023

Bergh went on to talk about how the software industry has tackled complexity by applying lean and agile principles such as DevOps and domain-driven design software products. Chris discussed the idea of decentralization and how well-designed domain interfaces can provide the freedom to innovate in data engineering.

Raw Data 52

Raw Data Data Datasets Metadata 52

RandomTrees

SEPTEMBER 6, 2020

Built on new SQL database engine, it provides a unique architecture designed for the cloud. This stage handles all the aspects of data storage like organization, file size, structure, compression, metadata, statistics. The data objects are accessible only through SQL query operations run using Snowflake.

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

Cloudera

FEBRUARY 9, 2021

The design cost of this however is that most often the source data for a given partition is not growing fast enough to be accumulated at the expected file size within minutes. Design Detail. In many large-scale solutions, data is divided into partitions that can be managed and accessed separately. Partition Strategy.

Data Warehouse 69

Data Warehouse Cloud Kafka Cloud Storage 69

Knowledge Hut

JULY 26, 2023

In this blog, we will deep dive into database system applications in DBMS, and their components and look at a list of database applications. Database applications are software programs or systems that are designed to organize and efficiently store, handle, and retrieve vast amounts of data. What are Database Applications?

Database 52

Database NoSQL Telecommunication MongoDB 52