Cloudera

SEPTEMBER 28, 2021

Cloudera Data platform ( CDP ) provides a Shared Data Experience ( SDX ) for centralized data access control and audit in the Enterprise Data Cloud. The Ranger Authorization Service (RAZ) is a new service added to help provide fine-grained access control (FGAC) for cloud storage. Changes with file access control .

Accessible 62

Accessible Accessibility Cloud Cloud Storage 62

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Snowflake

MARCH 30, 2023

Collecting, cleaning, and organizing data into a coherent form for business users to consume are all standard data modeling and data engineering tasks for loading a data warehouse. Feature engineering: Data is transformed to support ML model training. “The features you use influence more than everything else the result.

Engineering 83

Engineering Raw Data Data Science Scala 83

Rockset

MARCH 18, 2024

That’s why we’re thrilled that Confluent Cloud is making it easier to use Flink, providing efficient and performant stream processing while saving engineers from complex infrastructure management. These additional inputs are referred to as metadata filtering.

Cloud 64

Cloud Building Metadata Kafka 64

Cloudera

APRIL 1, 2024

Our customers run some of the world’s most innovative, largest, and most demanding data science, data engineering, analytics, and AI use cases, including PB-size generative AI workloads. It gives the flexibility to teams to use their choice of tools/engines without worrying about ETL.

Cloud 104

Cloud Unstructured Data Metadata Datasets 104

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. But unfortunately, very little that happens in data engineering today is observable. Most data pipelines are built to move but not monitor.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Data Engineering Weekly

OCTOBER 30, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Til about [link] Thank you, Anton Bryzgalov, for contributing the blog to the Data Engineering Weekly.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Maxime Beauchemin

AUGUST 28, 2017

This post follows up on The Rise of the Data Engineer , a recent post that was an attempt at defining data engineering and described how this new role relates to historical and modern roles in the data space. Understanding and exposing the adversity that the role is facing is a first step towards finding solutions.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174

Knowledge Hut

APRIL 25, 2024

apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1 apiVersion: v1 kind: Pod metadata: name: Tomcat spec: containers: - name: Tomcat image: tomcat: 8.0 apiVersion : v1 kind: Pod metadata: name : nginx spec: containers : - name : nginx image : nginx:1.14.2

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? At the same time, data engineering was the slightly younger sibling, but it was going through something similar. I wasn’t promoted or assigned to this new role.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Weekly

DECEMBER 4, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. link] Barr Moses: What’s Next for Data Engineering in 2023?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Cloudera

JULY 13, 2023

Several compute engines such as Impala, Hive, Spark, and Trino have supported querying data in Iceberg table format by adopting this Java Library provided by the Apache Iceberg project. Different query engines such as Impala, Hive, and Spark can immediately benefit from using Apache Iceberg Java Library.

Metadata 70

Metadata Java PostgreSQL Data Warehouse 70

Data Engineering Weekly

NOVEMBER 6, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. I plan to write a series of blogs on Schemata and Data Contract in the coming weeks.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

IT 100

IT Metadata MongoDB MySQL 100

Cloudera

OCTOBER 11, 2021

Modak, a leading provider of modern data engineering solutions, is now a certified solution partner with Cloudera. Customers can now seamlessly automate migration to Cloudera’s Hybrid Data Platform — Cloudera Data Platform (CDP) to dynamically auto-scale cloud services with Cloudera Data Engineering (CDE) integration with Modak Nabu.

Data Engineering 92

Data Engineering Data Engineer Cloud Engineering 92

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. The session metadata table can then be read to determine the pipeline input. Metadata Recording : Metadata is persisted for traceability.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. The slowest step in that process was provisioning Kafka Topics, which involved on-call engineer approval and was prone to failure, further increasing the on-call load. Built atop Prometheus.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Cloudera

SEPTEMBER 9, 2022

Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. These challenges require architecture changes and adoption of new table formats that can support massive scale, offer greater flexibility of compute engine and data types, and simplify schema evolution. .

Architecture 100

Architecture Metadata Unstructured Data Machine Learning 100

Cloudera

DECEMBER 15, 2022

With CDP, customers can deploy storage, compute, and access, all with the freedom offered by the cloud, avoiding vendor lock-in and taking advantage of best-of-breed solutions. Only metadata will be regenerated. Newly generated metadata will then point to source data files as illustrated in the diagram below. .

Cloud 78

Cloud Metadata Google Cloud Data Warehouse 78

Snowflake

NOVEMBER 2, 2023

While many customers value the simplicity of fully managed storage and a single, multi-language and multi-cluster compute engine to power a variety of workloads, some customers would prefer to manage their own storage using open formats, which is why we’ve added support for Apache Iceberg.

Data Lake 91

Data Lake Data Warehouse Cloud Unstructured Data 91

Edureka

MAY 6, 2024

It involves collaboration between data scientists, ML engineers, and IT professionals to automate and optimize the end-to-end process of building, deploying, and maintaining machine learning applications. But before we begin, Let’s have a look at what we will be covering in this blog: What is MLOPS? Why do we need MLOPS?

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Tweag

MAY 16, 2023

Configuration is more often than not considered a second-class engineering discipline, like a side activity of software engineering. A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code. The | symbol attaches metadata to fields.

MySQL 131

MySQL Metadata Data Validation Coding 131

LinkedIn Engineering

APRIL 25, 2023

since its inception in 2013, LinkedIn has built thousands of microservices to enable the exchange of data with our engineers and our external partners. Though this microservice architecture has worked out really well for our API engineers, when our clients need to fetch data they find themselves talking to several of these microservices.

Architecture 109

Architecture Metadata Java Transportation 109

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. In this blog post, we’ll dive into the details of these features and the benefits for customers. In addition to Iceberg External Tables, we introduced Native Iceberg Tables.

Metadata 85

Metadata AWS Data Lake Datasets 85

Knowledge Hut

SEPTEMBER 29, 2023

This growth is creating a strong demand for data experts, especially Azure data engineers. But who are Azure data engineers, and what do they do? Moreover, what benefits can you expect from a career in Azure Data Engineering? Join us on this journey through the exciting realm of Azure Data Engineering.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. The transformation step takes place in a separate engine and the transformed data is stored in a data store. The data is easily accessible and is easy to update. This was used to test our setup.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Cloudera

MARCH 2, 2023

It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. The CSP engine is powered by Apache Flink, which is the best-in-class processing engine for stateful streaming pipelines.

Process 113

Process SQL Kafka Database 113

Cloudera

FEBRUARY 22, 2022

Today, we are announcing a private technical preview (TP) release of Iceberg for CDP Data Services in the public cloud, including Cloudera Data Warehousing ( CDW ) and Cloudera Data Engineering ( CDE ). . After the Iceberg tables become available in SDX, the next step is to enable the execution engines to leverage the new tables.

Metadata 107

Metadata Datasets BI SQL 107

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Snowflake

AUGUST 22, 2023

In an earlier blog post, I discussed how the notion of a product emerges from a business need. In either case, the process of discovery continues to capture the product requirements, including data sources; the specific data requirements such as quality, formats or frequency of refresh; and also analytics and engineering requirements.

Metadata 73

Metadata Data AWS Business Analyst 73

Data Engineering Weekly

MARCH 24, 2023

Any business in the modern digital era can be considered a workflow engine. A business process or workflow engine is a software system that enables businesses to execute well-defined steps to complete a user’s intention. It could be either booking a taxi or ordering your lunch; everything is a workflow engine.

Engineering 67

Engineering Data Transportation Database 67

Zalando Engineering

OCTOBER 18, 2023

Certain directives are built into GraphQL, like @skip , @include , @deprecated , and @specifiedBy , and are supported by all GraphQL engines. The query directives are generally useful for clients to express certain types of metadata for the query. We can access the request parameters (like pathname) by running GraphQL over HTTP.

Metadata 86

Metadata Coding Banking Designing 86

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. We expand on this feature later in this blog. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI.

Cloud 97

Cloud Kafka Metadata SQL 97

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. For a more in-depth description of these phases please refer to Impala: A Modern, Open-Source SQL Engine for Hadoop. Impala is architected to be the Speed-of-Thought query engine for your data. Metadata Caching. Query Planner Design.

Metadata 143

Metadata Coding SQL Database 143

Engineering at Meta

OCTOBER 31, 2023

In our first post on automating product deprecation , we discussed the complexities of product deprecations and introduced SCARF’s workflow management tools that guide engineers through a coordinated process to safely deprecate products. SCARF coordinates several kinds of tasks for each data system: metadata collection (e.g.,

Data 93

Data Metadata Coding Relational Database 93

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. Data and Metadata: Data inputs and data outputs produced based on the application logic. Introduction.

Architecture 124

Architecture Metadata Government Kafka 124