phData: Data Engineering

OCTOBER 12, 2022

So you’ve heard all the talk around dbt , but now you’re working to determine if you should go with dbt Core or dbt Cloud and you’re wanting to know what advantages dbt Cloud has over the free dbt Core offering. However, dbt Cloud offers you much more than that.

Cloud 52

Cloud Metadata SQL Data Warehouse 52

Cloudera

JUNE 25, 2021

Many Cloudera customers are making the transition from being completely on-prem to cloud by either backing up their data in the cloud, or running multi-functional analytics on CDP Public cloud in AWS or Azure. Configure the required ports to enable connectivity from CDH to CDP Public Cloud (see docs for details).

Cloud 69

Cloud Data Lake Cloud Storage Metadata 69

Cloudera

SEPTEMBER 28, 2020

Cloudera delivers an enterprise data cloud that enables companies to build end-to-end data pipelines for hybrid cloud, spanning edge devices to public or private cloud, with integrated security and governance underpinning it to protect customers data. Attribute-based access control and SparkSQL fine-grained access control.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Cloudera

FEBRUARY 17, 2022

CDP Private Cloud Base is an on-premises version of Cloudera Data Platform (CDP). Provides self-service access to integrated, multi-function analytics on centrally managed and secured business data. Provides a consistent experience on Public Cloud, Multi-Cloud, and Private Cloud deployments. Upgrade HDP 3.1.5

Cloud 96

Cloud Data Professional Services Metadata 96

Cloudera

JANUARY 19, 2022

Gartner® recognized Cloudera in three recent reports – Magic Quadrant for Cloud Database Management Systems (DBMS), Critical Capabilities for Cloud Database Management Systems for Analytical Use Cases and Critical Capabilities for Cloud Database Management Systems for Operational Use Cases.

Database 87

Database Cloud Data Warehouse Data Lake 87

Cloudera

NOVEMBER 12, 2020

Cloud has given us hope, with public clouds at our disposal we now have virtually infinite resources, but they come at a different cost – using the cloud means we may be creating yet another series of silos, which also creates unmeasurable new risks in security and traceability of our data. A solution.

Cloud 102

Cloud Data Warehouse Banking Data 102

Cloudera

OCTOBER 15, 2021

Apache Ozone is a distributed, scalable, and high performance object store, available with Cloudera Data Platform Private Cloud. CDP Private Cloud uses Ozone to separate storage from compute, which enables it to handle billions of objects on-premises, akin to Public Cloud deployments which benefit from the likes of S3.

Cloud 86

Cloud Hadoop Data Analytics Metadata 86

Cloudera

SEPTEMBER 30, 2021

Our recent blog discussed the four paths to get from legacy platforms to CDP Private Cloud Base. In this blog and accompanying video, we will deep dive into the mechanics of running an in-place upgrade from CDH5 or CDH6 to CDP Private Cloud Base. As of CDP Private Cloud Base 7.1.6 CDP Upgrade.mp4 . Zookeeper data.

Cloud 70

Cloud PostgreSQL Metadata MySQL 70

Cloudera

NOVEMBER 29, 2023

It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. You can track each iteration, compare changes, and ensure that you always have access to the version that suits your needs. Its development is geared towards easing the transition between different cloud systems.

Machine Learning 84

Machine Learning Metadata Cloud Project 84

Snowflake

APRIL 29, 2024

This strategy must incorporate business impact analyses as well as backup and recovery plans in the event of a security incident or loss of access to data. Access Control: Snowflake allows customers to define granular permissions for user roles, minimizing the risk of unauthorized access to sensitive data.

Transportation 83

Transportation Data Governance Government Consulting 83

Cloudera

MARCH 3, 2022

Data analytics and machine learning can become a business and a compliance risk if data security, governance, lineage, metadata management, and automation are not holistically applied across the entire data lifecycle and all environments. One possible solution is to adopt a hybrid cloud strategy. .

Data Governance 96

Data Governance Government Metadata Medical 96

Cloudera

MARCH 2, 2023

It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. The post Streaming Ingestion for Apache Iceberg With Cloudera Stream Processing appeared first on Cloudera Blog. The Catalog Type should be set to Hive.

Process 112

Process SQL Kafka Database 112

Edureka

MAY 6, 2024

Whether you are a newbie or an experienced individual, if you want to explore more about the concepts of MLOPS, then you just click on the right blog. But before we begin, Let’s have a look at what we will be covering in this blog: What is MLOPS? Proper metadata management helps in understanding, grouping, and sorting data.

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

IT 100

IT Metadata MongoDB MySQL 100

Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). The data is protected but it is also accessible by the people who need it.

Government 79

Government Metadata Data Lake Data 79

Cloudera

FEBRUARY 9, 2021

Today, more and more customers are moving workloads to the public cloud for business agility where cost-saving and management are key considerations. Cloud object storage is used as the main persistent storage layer, which is significantly cheaper than block volumes. The Cost-Effective Data Warehouse Architecture. Cost-Effective.

Data Warehouse 69

Data Warehouse Cloud Kafka Cloud Storage 69

Cloudyard

DECEMBER 7, 2023

Read Time: 3 Minute, 0 Second In this blog post, we will delve into Snowflake’s Listing feature, which operates on the data sharing provider and consumer model. Consumer will create a Database from listing and Query the objects. Listing in Consumer Click on Get and it will start replicating the data from provider Account.

Metadata 52

Metadata SQL Data Cloud 52

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This blog post aims to help you understand what you can do to get started with generative AI assisted SQL using Hue image version ​​2023.0.16.0

SQL 71

SQL Data Warehouse AWS Architecture 71

Cloudera

MARCH 14, 2023

We just announced the general availability of Cloudera DataFlow Designer , bringing self-service data flow development to all CDP Public Cloud customers. In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities.

Data Pipeline 81

Data Pipeline Designing Kafka Metadata 81

Cloudera

MARCH 17, 2023

Hopefully this blog will give ChatGPT an opportunity to learn and correct itself while counting towards my 2023 contribution to social good. The one key component that is missing is a common, shared table format, that can be used by all analytic services accessing the lakehouse data. I thought this was a fairly comprehensive list.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

Cloudera

JUNE 7, 2022

Impressive, but dwarfed by the amount of unstructured data, cloud data, and machine data – another 50 ZB. The cause is hybrid data – the massive amounts of data created everywhere businesses operate – in clouds, on-prem, and at the edge. We can also do it with your preferred cloud – AWS, Azure or GCP.

IT 108

IT Unstructured Data Data Architecture Government 108

phData: Data Engineering

SEPTEMBER 19, 2023

This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. Data Governance and Security By defining data models, organizations can establish policies, access controls, and security measures to protect sensitive data.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Cloudera

AUGUST 3, 2021

In recent years the Cloud has been seen as a solution and panacea for many companies digital transformation strategies. However a recent Andereessen Horowitz study has shown that while the Cloud is a viable solution for start-up, expanding and emerging use cases, its true cost on market capitalization is vastly underestimated.

Manufacturing 108

Manufacturing Retail Entertainment Government 108

Netflix Tech

MARCH 5, 2019

All these micro-services are currently operated in AWS cloud infrastructure. Finally, provisioning our infrastructure itself is also becoming an increasingly complex task, so our data teams contribute to tools for diagnosis and automation of our cloud capacity management.

Cloud 73

Cloud Building Amazon Web Services Metadata 73

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If so, you know about the failed economics of some cloud-native solutions on the market today. If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Data Engineering Podcast

JANUARY 23, 2022

Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. You can observe your pipelines with built in metadata search and column level lineage. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud.

Building 100

Building Management Data Pipeline Metadata 100

Databand.ai

JULY 19, 2023

Learn more in our detailed guide to data lineage visualization (coming soon) Integration with Multiple Data Sources Data lineage tools are designed to integrate with a wide range of data sources, including databases, data warehouses, and cloud-based data platforms. It provides context for data, making it easier to understand and manage.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Data Engineering Weekly

MARCH 10, 2024

Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses. Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Cloudera

DECEMBER 9, 2022

Cloudera has been providing enterprise support for Apache NiFi since 2015, helping hundreds of organizations take control of their data movement pipelines on premises and in the public cloud. Now, we shift focus on the needs of developers and addressing the challenges they face when building dataflows in the cloud.

Designing 96

Designing Coding Google Cloud AWS 96

Cloudera

SEPTEMBER 29, 2022

Data teams have the impossible task of delivering everything (data and workloads) everywhere (on premise and in all clouds) all at once (with little to no latency). Then, we add hybrid architectures to the mix since they are here to stay and will not just be a “temporary state until we all move to the cloud.”. Data mesh defined.

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Netflix Tech

DECEMBER 8, 2021

Project by Netflix’s Cloud Infrastructure Security team ( Alex Bainbridge , Mike Grima , Nick Siow) Cloud security is a hard problem, but an even harder one is cloud security at scale. In 2020, we decided to reinvent how we handle cloud security findings by redefining how we write and respond to cloud detections.

AWS 78

AWS Metadata Cloud Utilities 78

Snowflake

MAY 8, 2023

By leveraging Snowflake’s comprehensive metadata on query history and operator statistics , alongside Acceldata’s advanced query fingerprinting techniques, users can attain a deeper comprehension of their workloads and pinpoint the most resource-intensive or unoptimized queries.

Technology 111

Technology SQL BI Utilities 111

Cloudera

SEPTEMBER 23, 2020

In this blogpost, we are going to take a look at some of the OpDB related security features of a CDP Private Cloud Base deployment. Comprehensive auditing is provided to enable enterprises to effectively and efficiently meet their compliance requirements by auditing access and other types of operations across OpDB (through HBase).

Database 81

Database Data Lake Metadata Java 81

Data Engineering Podcast

MARCH 22, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. e.g. APIs and third party data sources How can we integrage CDC into metadata/lineage tooling? Sign up free at dataengineeringpodcast.com/rudder today.

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Rockset

NOVEMBER 14, 2022

In this blog, we will describe the SSB configuration, results and performance enhancements. RocksDB provides an in-memory cache, called the block cache, that keeps frequently accessed data blocks in memory. The metadata lookup operation takes a significant proportion of the working set memory, impacting query latency.

Metadata 52

Metadata Datasets Cloud Accessible 52

Workfall

AUGUST 23, 2023

Inside this directory, we’ll create a file named ‘manifest.json’ This file will play a crucial role in defining the metadata, permissions, and settings for our extension. Once we access the browser console, we’ll see the weather data containing all the necessary information provided by the OpenWeather API.

Metadata 83

Metadata Coding Project Utilities 83

dbt Developer Hub

DECEMBER 11, 2023

The Semantic Layer lets teams define business metrics centrally, in dbt, and access them in multiple analytics tools through our semantic layer APIs. This blog post walks through the end-to-end process we used to set up product analytics for the dbt Semantic Layer using the dbt Semantic Layer. Measures: Next we added Measures.

Finance 64

Finance BI SQL Cloud 64