Ascend.io

FEBRUARY 6, 2023

Data Flow – is an individual data pipeline. Data Flows include the ingestion of raw data, transformation via SQL and python, and sharing of finished data products. Data Plane – is the data cloud where the data pipeline workload runs, like Databricks, BigQuery, and Snowflake.

Cloud 52

Cloud Data Ingestion Raw Data Data Pipeline 52

Ascend.io

FEBRUARY 6, 2023

Data Flow – is an individual data pipeline. Data Flows include the ingestion of raw data, transformation via SQL and python, and sharing of finished data products. Data Plane – is the data cloud where the data pipeline workload runs, like Databricks, BigQuery, and Snowflake.

Cloud 52

Cloud Data Ingestion Raw Data Data Pipeline 52

Cloudera

MARCH 31, 2021

CDP Public Cloud is now available on Google Cloud. The addition of support for Google Cloud enables Cloudera to deliver on its promise to offer its enterprise data platform at a global scale. CDP Public Cloud is already available on Amazon Web Services and Microsoft Azure.

Google Cloud 92

Google Cloud Cloud Amazon Web Services Cloud Storage 92

Precisely

MAY 13, 2024

Key Takeaways: Data integration is vital for real-time data delivery across diverse cloud models and applications, and for leveraging technologies like generative AI. As enterprise technology landscapes grow more complex, the role of data integration is more critical than ever before.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Cloudera

OCTOBER 11, 2021

Customers can now seamlessly automate migration to Cloudera’s Hybrid Data Platform — Cloudera Data Platform (CDP) to dynamically auto-scale cloud services with Cloudera Data Engineering (CDE) integration with Modak Nabu. Cloud Speed and Scale. Customers using Modak Nabu with CDP today have deployed Data Lakes and.

Data Engineering 92

Data Engineering Data Engineer Cloud Engineering 92

Cloudera

SEPTEMBER 28, 2020

Cloudera delivers an enterprise data cloud that enables companies to build end-to-end data pipelines for hybrid cloud, spanning edge devices to public or private cloud, with integrated security and governance underpinning it to protect customers data. The customer is a heavy user of Kafka for data ingestion.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Data Engineering Podcast

NOVEMBER 20, 2022

report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. In fact, while only 3.5% That’s where our friends at Ascend.io

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Booking.com Engineering

DECEMBER 2, 2022

In this article, we want to illustrate our extensive use of the public cloud, specifically Google Cloud Platform (GCP). From data ingestion, data science, to our ad bidding[2], GCP is an accelerant in our development cycle, sometimes reducing time-to-market from months to weeks. What are PPC’s Challenges?

Systems 52

Systems Cloud MySQL Relational Database 52

Snowflake

NOVEMBER 29, 2023

Data cloud technology can accelerate FAIRification of the world’s biomedical patient data. Also, the associated business metadata for omics, which make it findable for later use, are dynamic and complex and need to be captured separately.

Metadata 84

Metadata Healthcare Medical Data Storage 84

Cloudera

FEBRUARY 9, 2021

Today’s customers have a growing need for a faster end to end data ingestion to meet the expected speed of insights and overall business demand. This ‘need for speed’ drives a rethink on building a more modern data warehouse solution, one that balances speed with platform cost management, performance, and reliability.

Data Warehouse 70

Data Warehouse Cloud Kafka Cloud Storage 70

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. Understand Change Data Capture — CDC.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Snowflake

FEBRUARY 13, 2024

Snowpark Updates Model management with the Snowpark Model Registry – public preview Snowpark Model Registry is an integrated solution to register, manage and use models and their metadata natively in Snowflake. Learn more here. This enables secure, simple, and isolated connectivity to internal stages. Learn more here.

Data Ingestion 87

Data Ingestion AWS Python Metadata 87

Data Engineering Podcast

JUNE 5, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Data Security 100

Data Security Metadata MongoDB Scala 100

Cloudera

SEPTEMBER 9, 2022

Cloudera has been supporting data lakehouse use cases for many years now, using open source engines on open data and table formats, allowing for easy use of data engineering, data science, data warehousing, and machine learning on the same data, on premises, or in any cloud.

Architecture 100

Architecture Metadata Unstructured Data Machine Learning 100

Monte Carlo

JANUARY 5, 2024

This architecture format consists of several key layers that are essential to helping an organization run fast analytics on structured and unstructured data. Table of Contents What is data lakehouse architecture? The 5 key layers of data lakehouse architecture 1. Ingestion layer 2. Metadata layer 4. API layer 5.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

This architecture format consists of several key layers that are essential to helping an organization run fast analytics on structured and unstructured data. Table of Contents What is data lakehouse architecture? The 5 key layers of data lakehouse architecture 1. Ingestion layer 2. Metadata layer 4. API layer 5.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

phData: Data Engineering

SEPTEMBER 19, 2023

Want to learn more about data governance? Check out our Data Governance on Snowflake blog! Metadata Management Data modeling methodologies help in managing metadata within the data lake. Metadata describes the characteristics, attributes, and context of the data.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Cloudera

AUGUST 17, 2022

What if you could access all your data and execute all your analytics in one workflow, quickly with only a small IT team? CDP One is a new service from Cloudera that is the first data lakehouse SaaS offering with cloud compute, cloud storage, machine learning (ML), streaming analytics, and enterprise grade security built-in.

Cloud Computing 115

Cloud Computing Cloud Storage Data Science Government 115

Rockset

AUGUST 28, 2023

We founded Rockset to empower everyone from Fortune 500 to a five-person startup to build powerful search and AI applications and scale them efficiently in the cloud. The ability to scale apps efficiently in the cloud is what is going to allow enterprises to continue to leverage AI. We obsess about efficiency in the cloud.

Metadata 78

Metadata Unstructured Data SQL Database 78

Towards Data Science

NOVEMBER 4, 2023

It simply makes sense to perform required data transformations, such as cleansing, enrichment and extraction in the place where data is being stored. Often it is a data warehouse solution (DWH) in the central part of our infrastructure. The downside of this approach is it’s pricing model though. ML model training using Airflow.

Data Engineering 83

Data Engineering Data Engineer Engineering BI 83

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Cloudera

DECEMBER 20, 2021

Understanding that the future of banking is data-driven and cloud-based, Bank of the West embraced cloud computing and its benefits, like remote capabilities, integrated processes, and flexible systems. Winner of the Data Impact Awards 2021: Security & Governance Leadership. You can become a data hero too.

Government 89

Government Data Security Banking Metadata 89

Towards Data Science

DECEMBER 1, 2023

This tool automates ELT (Extract, Load, Transform) process, integrating your data from the source system of Google Calendar to our Snowflake data warehouse. Storage —  Snowflake Snowflake, a cloud-based data warehouse tailored for analytical needs, will serve as our data storage solution.

Data Engineering 71

Data Engineering Data Engineer Project Engineering 71

Rockset

APRIL 18, 2023

Under the hood, Rockset utilizes its Converged Index technology, which is optimized for metadata filtering, vector search and keyword search, supporting sub-second search, aggregations and joins at scale. Fast Search: Combine vector search and selective metadata filtering to deliver fast, efficient results.

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

Cloudera

JUNE 30, 2022

No more lock-in, unnecessary data transformations, or data movement across tools and clouds just to extract insights out of the data. With Shared Data Experience (SDX) which is built in to CDP right from the beginning, customers benefit from a common metadata, security, and governance model across all their data. .

Data Lake 86

Data Lake Business Intelligence Metadata Data Warehouse 86

Data Engineering Podcast

AUGUST 6, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

AltexSoft

AUGUST 29, 2023

Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture. This structure is made efficient by data engineering practices that include object storage. Watch our video explaining how data engineering works.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Towards Data Science

JULY 24, 2023

Closely related to this is how those same platforms are bundling or unbundling related data services from data ingestion and transformation to data governance and monitoring. Why are these things related, and more importantly, why should data leaders care? Again, that answer will depend on who you are.

Metadata 96

Metadata Data Warehouse Data Lake Data Science 96

ProjectPro

FEBRUARY 8, 2023

For e.g., Finaccel, a leading tech company in Indonesia, leverages AWS Glue to easily load, process, and transform their enterprise data for further processing. Another leading European company, Claranet, has adopted Glue to migrate their data load from their existing on-premise solution to the cloud. How Does AWS Glue Work?

AWS 98

AWS Scala Metadata Data Lake 98

LinkedIn Engineering

JULY 27, 2023

Authors: Deven Walia, Vivek Subramaniam , Simon Desowza , and Karthik Subramanian Cloud services have completely changed the way we approach infrastructure management. However, the ease of these processes can lead to over-provisioning and under-utilization of cloud resources, resulting in increased operating expenses.

Metadata 75

Metadata Utilities Cloud Data Lake 75

Towards Data Science

MARCH 5, 2024

In that case, queries are still processed using the BigQuery compute infrastructure but read data from GCS instead. Such external tables come with some disadvantages but in some cases it can be more cost efficient to have the data stored in GCS. Load data For data ingestion Google Cloud Storage is a pragmatic way to solve the task.

Bytes 70

Bytes Google Cloud Cloud Storage Utilities 70

Cloudera

AUGUST 31, 2021

The key questions that need to be answered are: Do you have workloads you wish would run faster, but you just can’t make it happen without an expensive solution from your existing data warehouse? Are you looking for your data warehouse to support the hybrid multi-cloud? Copyright © 2021 Accenture. All rights reserved.

Data Warehouse 82

Data Warehouse Database-centric Metadata Cloud 82

Cloudera

APRIL 22, 2021

This CVD is built using Cloudera Data Platform Private Cloud Base 7.1.5 Apache Ozone is one of the major innovations introduced in CDP, which provides the next generation storage architecture for Big Data applications, where data blocks are organized in storage containers for larger scale and to handle small objects.

Pipeline-centric 100

Pipeline-centric Data Lake Hadoop Metadata 100

Monte Carlo

APRIL 24, 2023

Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption. Databricks Data Catalog and AWS Lake Formation are examples in this vein. It works in both directions.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

RandomTrees

SEPTEMBER 6, 2020

Snowflake Overview A data warehouse is a critical part of any business organization. Lot of cloud-based data warehouses are available in the market today, out of which let us focus on Snowflake. Snowflake is an analytical data warehouse that is provided as Software-as-a-Service (SaaS).

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

Cloudera

JUNE 16, 2022

Faster data ingestion: streaming ingestion pipelines. The DevOps/app dev team wants to know how data flows between such entities and understand the key performance metrics (KPMs) of these entities. She is a smart data analyst and former DBA working at a planet-scale manufacturing company. Getting started today.

Kafka 86

Kafka Manufacturing Data Lake SQL 86

Data Engineering Weekly

OCTOBER 30, 2022

Part of me wonders, shouldn’t it be a solved problem where the cloud providers can give these solutions out-of-the-box? link] ABN AMRO: Building a scalable metadata-driven data ingestion framework Data ingestion is a heterogenous system with multiple sources with its data format, scheduling & data validation requirements.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52