DataKitchen

AUGUST 3, 2021

First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt. Second-generation – gigantic, complex data lake maintained by a specialized team drowning in technical debt. The post What is a Data Mesh?

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Cloudera

JANUARY 19, 2022

Cloudera Data Platform (CDP) scored among the top 10 vendors on all four Analytical Use Cases — Data Warehouse, Logical Data Warehouse, Data Lake and Operational Intelligence in the Critical Capabilities for Cloud Database Management Systems for Analytics Use Cases.

Database 87

Database Cloud Data Warehouse Data Lake 87

DataKitchen

AUGUST 9, 2021

The domain also includes code that acts upon the data, including tools, pipelines, and other artifacts that drive analytics execution. The domain requires a team that creates/updates/runs the domain, and we can’t forget metadata: catalogs, lineage, test results, processing history, etc., …. How does one get access to a domain?

Pharmaceutical 98

Pharmaceutical Raw Data Data Lake Data 98

Monte Carlo

MARCH 3, 2021

We designed our product without having to store or access individual records, PII, or any other sensitive information. Instead, we only extract query logs, metadata, and aggregated statistics about data usage to ensure that your most critical data assets are as trustworthy and reliable as possible. What’s next?

Metadata 52

Metadata Data Lake Business Intelligence Certification 52

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

ProjectPro

JANUARY 24, 2023

If you are preparing for your ETL developer or data engineer interview , you must possess a solid fundamental knowledge of AWS Glue, as you’re likely to get asked questions that test your ability to handle complex big data ETL tasks. You can leverage AWS Glue to discover, transform, and prepare your data for analytics.

AWS 52

AWS Data Lake Scala ETL Tools 52

Cloudera

SEPTEMBER 15, 2022

Mark: The first element in the process is the link between the source data and the entry point into the data platform. At Ramsey International (RI), we refer to that layer in the architecture as the foundation, but others call it a staging area, raw zone, or even a source data lake. What is a data fabric?

Data Lake 79

Data Lake Analytics Application Cloud Storage Architecture 79

AltexSoft

FEBRUARY 11, 2023

What’s more, investing in data products, as well as in AI and machine learning was clearly indicated as a priority. This suggests that today, there are many companies that face the need to make their data easily accessible, cleaned up, and regularly updated. Let’s discuss and compare them to avoid misconceptions.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Data Engineering Weekly

OCTOBER 30, 2022

I found the blog helpful in understanding the generative model’s historical development and the path forward. link] Sponsored- [New eBook] The Ultimate Data Observability Platform Evaluation Guide Considering investing in a data quality solution? to solve some of the challenges with self-serving.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Cloudera

FEBRUARY 16, 2022

In the case of CDP Public Cloud, this includes virtual networking constructs and the data lake as provided by a combination of a Cloudera Shared Data Experience (SDX) and the underlying cloud storage. Each project consists of a declarative series of steps or operations that define the data science workflow.

Machine Learning 81

Machine Learning Algorithm Government Metadata 81

Cloudera

NOVEMBER 10, 2021

One such major change for CDH users is the replacement of Sentry with Ranger for authorization and access control. . For big data platforms like Cloudera’s stack that are used by multiple business units with many users, upgrading even minor versions must be a well-planned activity to reduce the impact to users and business.

Hadoop 74

Hadoop SQL Database Kafka 74

Cloudera

MAY 12, 2021

Use cases could include but are not limited to: workload analysis and replication, migrating or bursting to cloud, data warehouse optimization, and more. Data Security & Governance: Merck KGaA, Darmstadt, Germany — Established a data governance framework with their data lake to discover, analyze, store, mine, and govern relevant data.

Food 66

Food Healthcare Government Medical 66

Cloudera

NOVEMBER 12, 2020

Typically, when we talk about data warehousing at an enterprise level on the cloud, one of the biggest concerns is that moving workloads from on-premises to the cloud is not seamless and opens up new risks for data safety and security. Key areas of concern are: .

Cloud 102

Cloud Data Warehouse Banking Data 102

Knowledge Hut

SEPTEMBER 29, 2023

Moreover, what benefits can you expect from a career in Azure Data Engineering? This blog aims to answer these questions, providing a straightforward and professional insight into the world of Azure Data Engineering. Join us on this journey through the exciting realm of Azure Data Engineering.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 89

Data Lake Data Warehouse Cloud Unstructured Data 89

Cloudera

AUGUST 21, 2020

Data-in-motion is predominantly about streaming data so enterprises typically have two different ways or binary ways of looking at data. Once the data gets into a place where organizations are processing it in real-time, users need access to it in real-time as well.

Banking 60

Banking Kafka Cloud Storage Government 60

Monte Carlo

APRIL 1, 2021

Over the past few years, data lakes have emerged as a must-have for the modern data stack. But while the technologies powering our access and analysis of data have matured, the mechanics behind understanding this data in a distributed environment have lagged behind. Who has access to it?

Data Lake 52

Data Lake Unstructured Data Data Warehouse Metadata 52

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Did you know the global big data market will likely reach $268.4 Businesses are leveraging big data now more than ever.

AWS 98

AWS Scala Metadata Data Lake 98

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Cloudera

DECEMBER 15, 2022

With CDP, customers can deploy storage, compute, and access, all with the freedom offered by the cloud, avoiding vendor lock-in and taking advantage of best-of-breed solutions. With in-place table migration, you can rapidly convert to Iceberg tables since there is no need to regenerate data files. Only metadata will be regenerated.

Cloud 76

Cloud Metadata Google Cloud Data Warehouse 76

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. In this blog post, we’ll dive into the details of these features and the benefits for customers. In addition to Iceberg External Tables, we introduced Native Iceberg Tables.

Metadata 83

Metadata AWS Data Lake Datasets 83

Cloudera

AUGUST 30, 2022

Data is the fuel that drives government, enables transparency, and powers citizen services. For state and local agencies, data silos create compounding problems: Inaccessible or hard-to-access data creates barriers to data-driven decision making. A simple example from a recent article in StateTech makes this case.

Data Architecture 76

Data Architecture Architecture Data Lake NoSQL 76

Cloudera

NOVEMBER 17, 2022

The Solution: CDP Private Cloud brings a next-generation hybrid architecture with cloud-native benefits to HBL’s data platform. HBL started their data journey in 2019 when data lake initiative was started to consolidate complex data sources and enable the bank to use single version of truth for decision making.

Banking 82

Banking Management Data Lake Professional Services 82

Cloudera

MARCH 23, 2022

Apache Iceberg is a high-performance, open table format, born-in-the cloud that scales to petabytes independent of the underlying storage layer and the access engine layer. By being a truly open table format, Apache Iceberg fits well within the vision of the Cloudera Data Platform (CDP). What is Apache Iceberg? 2: Open formats.

Metadata 88

Metadata Data Architecture BI Machine Learning 88

Cloudera

JANUARY 21, 2021

With CDW, as an integrated service of CDP, your line of business gets immediate resources needed for faster application launches and expedited data access, all while protecting the company’s multi-year investment in centralized data management, security, and governance. Proprietary file formats mean no one else is invited in!

IT 93

IT Data Lake Data Warehouse Cloud Storage 93

ProjectPro

JULY 23, 2021

AWS (Amazon Web Services) is the world’s leading and widely used cloud platform, with over 200 fully featured services available from data centers worldwide. This blog presents some of the most unique and innovative AWS projects from beginner to advanced levels. Create an AWS EC2 instance and install Amazon Kinesis on it.

AWS 52

AWS Project Amazon Web Services Cloud Computing 52

Cloudera

AUGUST 4, 2021

Today, modern data governance programs must address not only an exponential increase in data volumes, but also cloud, hybrid architectures, and more reliance on advanced analytics. Trends toward more self-service data access and real-time applications have made data governance more visible and critical than ever before.

Data Governance 83

Data Governance Government Metadata Data Security 83

Cloudera

JUNE 30, 2022

With Cloudera’s vision of hybrid data , enterprises adopting an open data lakehouse can easily get application interoperability and portability to and from on premises environments and any public cloud without worrying about data scaling. Why integrate Apache Iceberg with Cloudera Data Platform?

Data Lake 84

Data Lake Business Intelligence Metadata Data Warehouse 84

LinkedIn Engineering

JULY 27, 2023

It’s now much easier to manage large infra requirements that have traditionally demanded an amalgamation of teams like DBA, Infra-SRE, Onprem-SMEs, network managers, and access control managers working together. In this blog post, we will share our progress, challenges, and lessons learned from our Costwiz journey.

Metadata 75

Metadata Utilities Cloud Data Lake 75

Cloudera

NOVEMBER 5, 2020

Registering an Environment provides CDP with access to your cloud provider account and identifies the resources in your cloud provider account that CDP services can access or provision. When you register an Environment in CDP, a Data Lake is automatically deployed for that environment. Cloudera Data Warehouse Security.

Cloud 70

Cloud Data Lake Data Warehouse Metadata 70

Cloudera

SEPTEMBER 9, 2022

The promise of a modern data lakehouse architecture. Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested.

Architecture 97

Architecture Metadata Unstructured Data Machine Learning 97

Data Engineering Weekly

DECEMBER 4, 2022

The article highlights the challenges of maintaining data models in a world where SQL data warehouses are no longer the primary data platform. The author discusses the need for richer metadata to support complex data lineage and evolving privacy requirements. link] All rights reserved ProtoGrowth Inc, India.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Cloudera

MARCH 31, 2021

Customers who have chosen Google Cloud as their cloud platform can now use CDP Public Cloud to create secure governed data lakes in their own cloud accounts and deliver security, compliance and metadata management across multiple compute clusters. Analyze static (Apache Impala) and streaming (Apache Flink) data.

Google Cloud 90

Google Cloud Cloud Amazon Web Services Cloud Storage 90

Cloudera

JUNE 6, 2019

The typical Cloudera Enterprise Data Hub Cluster starts with a few dozen nodes in the customer’s datacenter hosting a variety of distributed services. Once configured and secured, the cluster administrator (admin) gives access to a few individuals to onboard their workloads. A Better Approach: Virtual Private Clusters. With CM 6.2,

Metadata 85

Metadata Data Lake Cloud Big Data 85

Cloudera

DECEMBER 7, 2020

Apache Hadoop Distributed File System (HDFS) is the most popular file system in the big data world. The Apache Hadoop File System interface has provided integration to many other popular storage systems like Apache Ozone, S3, Azure Data Lake Storage etc. Migrating file systems thus requires a metadata update. .

Systems 58

Systems Hadoop Metadata Datasets 58

Cloudera

DECEMBER 16, 2022

Cloudera has long had the capabilities of a data lakehouse, if not the label. Cloudera enables an open data lakehouse architecture that combines all the flexibility of the data lake with the performance of the data warehouse, so enterprises can use all data — both structured and unstructured.

Database 88

Database Cloud Systems Management 88

AltexSoft

NOVEMBER 7, 2022

Metadata database. A metadata database stores information about user permissions, past and current DAG and task runs, DAG configurations, and more. By default, Airflow handles metadata with SQLite which is meant for development only. Full REST API: easy access for third parties. Since the 2.0 Web App Development.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52