Scott Logic

NOVEMBER 9, 2023

There is a growing ecosystem of organisations that are promoting sustainability and environmental issues within software development, cloud computing, infrastructure, and digital services. It’s also worth checking out their Green Web Library which is a helpful catalogue of links to papers and data sources.

Amazon Web Services 52

Amazon Web Services Government Cloud Computing Consulting 52

Cloudera

DECEMBER 11, 2020

The Data Security and Governance category, at the annual Data Impact Awards, has never been so important. The sudden rise in remote working, a huge influx in data as the world turned digital, not to mention the never-ending list of regulations businesses need to remain compliant with (how many acronyms can you name in full?

Data Lake 98

Data Lake Government Data Security Unstructured Data 98

Snowflake

AUGUST 21, 2023

With demonstrable success across a range of industries, organizations are increasingly pursuing cutting-edge data mesh architectures to enhance self-service data use. Data-as-a-product: By considering data resources through a product lens, teams can adopt practices centered around quality and ease of use.

Pipeline-centric 52

Pipeline-centric Architecture Government Data Architect 52

Knowledge Hut

DECEMBER 28, 2023

Imagine having a framework capable of handling large amounts of data with reliability, scalability, and cost-effectiveness. In this blog, we'll talk about intriguing and real-time sample Hadoop projects with source codes that can help you take your data analysis to the next level. That's where Hadoop comes into the picture.

Hadoop 52

Hadoop Project Datasets Big Data 52

Knowledge Hut

SEPTEMBER 28, 2023

Welcome to this blog discussing the Power BI syllabus, where we'll delve into comprehensive exam preparation and offer valuable tips. It stands as a formidable business intelligence tool, providing an end-to-end solution for diverse data needs. It efficiently conserves space and enables exploration of multiple data variations.

BI 52

BI Business Intelligence Certification Raw Data 52

Cloudera

SEPTEMBER 11, 2020

In this blog post, we are going to take a look at some of the OpDB related security features of a CDP Private Cloud Base deployment. Data-at-rest encryption. Transparent data-at-rest encryption is available through the Transparent Data Encryption (TDE) feature in HDFS. . Mature key lifecycle management features.

Database 65

Database Big Data Ecosystem Transportation SQL 65

Cloudera

FEBRUARY 11, 2021

The Ranger plugin base is available only in Java, as most Hadoop ecosystem projects, including Ranger, are written in Java. Resource-based access control (RBAC) policies can be set up for Kudu in Ranger, but Kudu currently doesn’t support tag-based policies, row-level filtering or column masking. Conclusion.

Hadoop 52

Hadoop Metadata Java Database 52

Pinterest Engineering

OCTOBER 31, 2023

In this article, we discuss the PinCompute primitives, architecture, control plane and data plane capabilities, and showcase the value that PinCompute has delivered for innovation and efficiency at Pinterest. For example, removing managed fields will reduce up to 50% data size for PinCompute API calls.

Architecture 95

Architecture Pipeline-centric Accessible Accessibility 95

Tweag

NOVEMBER 2, 2022

The exact mechanism is out of scope for this article, but well described in this blog series. Well, the story in this blog post would be quite uninteresting if things were so simple. Well, the story in this blog post would be quite uninteresting if things were so simple. T aux x = A. const x ) ( A.

Coding 40

Coding Building Project Programming 40

Monte Carlo

MARCH 30, 2022

Say it with me: your data will never be perfect. Any team striving for completely accurate data will be sorely disappointed. Data testing , anomaly detection, and cataloging are important steps, but technology alone will not solve your data quality problem. Like any entropic system, data breaks.

Data Warehouse 52

Data Warehouse Data Machine Learning Data Engineering 52

LinkedIn Engineering

DECEMBER 13, 2023

In previous blog posts, we shared how we built the skills taxonomy behind our Skills Graph from the more than 41,000 skills across our platform. In this blog, we’ll examine how we use AI to extract skills from various content sources across LinkedIn and map these skills to our Skills Graph. AI is integral to this process.

Recruitment 83

Recruitment Utilities Designing Java 83

DareData

AUGUST 12, 2021

Introduction "Data is the 21st century oil". If you work anywhere in the vicinity of data, odds are you've heard some variation of this statement at least once. The goal of this blog post is to hopefully shed some light on what a healthy data ecosystem looks like, so that you might explore your company's data to its full potential.

Transportation 52

Transportation Data Lake Data Warehouse Data 52

Rock the JVM

AUGUST 9, 2022

He also has been diving into the ZIO ecosystem, and loves sharing his learnings. As a toy example, we’re going to take a brief foray into asynchronous operations, by connecting two streams to the same concurrent data structure. Broadly: A stream is a series of data elements that are made available over time.

Scala 40

Scala Bytes Kafka Programming 40

Cloudera

AUGUST 24, 2022

It unifies self-service data science and data engineering in a single, portable service as part of an enterprise data cloud for multi-function analytics on data anywhere. It’s built for the agility and power of cloud computing, but isn’t limited to any one cloud provider or data source. Prerequisites. docker.io).

Machine Learning 79

Machine Learning Building Metadata Programming Language 79

Snowflake

JUNE 28, 2023

Since its launch two years ago, Snowpark has been empowering data scientists, data engineers, and application developers to streamline their architectures, accelerate development, and boost the performance of data engineering and ML/AI workloads on Snowflake.

Python 52

Python Accessible Accessibility Pipeline-centric 52

Netflix Tech

SEPTEMBER 2, 2020

In an earlier blog post, we discussed Telltale , our health monitoring system. With request tracing and additional data from logs, events, metadata, and analysis, Edgar is able to show the flow of a request through our distributed system?—?what But, there are a few substantial differences in how Edgar approaches its data and its users.

Metadata 118

Metadata Transportation Engineering Systems 118

Cloudera

JANUARY 21, 2022

This blog will summarise the security architecture of a CDP Private Cloud Base cluster. The architecture reflects the four pillars of security engineering best practice, Perimeter, Data, Access and Visibility. Sensitive data is encrypted. Auditing has been setup for data in the metastore. Characteristics. Non-secure.

Architecture 107

Architecture Transportation Certification Government 107

Knowledge Hut

JANUARY 12, 2024

A private data centre, the public cloud , or even a developer's laptop, to name a few, can be the target. To check the Docker version installed use the following command; $ Docker --version The Docker version installed at the time of this blog is Docker version 20.10.12, build 20.10.12-0ubuntu2~20.04.1 0ubuntu2~20.04.1

Systems 97

Systems MySQL Architecture Certification 97

AltexSoft

NOVEMBER 7, 2022

You can hardly compare data engineering toil with something as easy as breathing or as fast as the wind. The platform went live in 2015 at Airbnb, the biggest home-sharing and vacation rental site, as an orchestrator for increasingly complex data pipelines. How data engineering works. What is Apache Airflow?

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Netflix Tech

DECEMBER 3, 2022

by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow.

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Datakin

OCTOBER 14, 2021

dbt is an amazing way to transform data within a data warehouse. Data lineage is super powerful like that. It is based on a pre-built sample project – a study of the Stack Overflow public data set – but you can apply this approach to a dbt project of your own. This view combines data from several tables.

Google Cloud 52

Google Cloud Datasets Bytes Metadata 52

AltexSoft

OCTOBER 21, 2022

Similar to Google in web browsing and Photoshop in image processing, it became a gold standard in data streaming, preferred by 70 percent of Fortune 500 companies. Apache Kafka is an open-source, distributed streaming platform for messaging, storing, processing, and integrating large data volumes in real time. What is Kafka?

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

AltexSoft

JUNE 9, 2023

Statistical language models leverage data patterns to make predictions about word sequences. Excelling in understanding context and interpreting meaning, transformers analyze the intricate relationships within sequential data — such as words in a text string. The transformer-model architecture.

Project 52

Project Java Hospitality Transportation 52

ProjectPro

OCTOBER 20, 2021

With so many pseudo-data scientists cropping up due to numerous data science bootcamps and courses that offer theoretical learning, the interview questions for AI and machine learning jobs are getting streamlined to filter those who understand how real-world implementation works. The following diagram explains Auto ML.

Machine Learning 52

Machine Learning Algorithm Government Data Science 52

Snowflake

JUNE 27, 2023

The world is undergoing a remarkable transformation fueled by data. Organizations have accumulated silos across their data infrastructure to support various workloads, languages, tools, and formats because of technology limitations. We’re going to summarize these new capabilities in this blog post.

Data Governance 57

Data Governance Unstructured Data Government SQL 57

LinkedIn Engineering

DECEMBER 13, 2022

In this blog post, we share more details on how LinkedIn performs observational causal inference at scale using our Ocelot platform. The tables show sample data for the control and treatment groups respectively. It aims to deliver estimates of causal relationships from observational data, along with robustness checks, to end users.

Data Preparation 55

Data Preparation Data Science Designing Data Pipeline 55

DataKitchen

JUNE 24, 2021

In May 2021 at the CDO & Data Leaders Global Summit, DataKitchen sat down with the following data leaders to learn how to use DataOps to drive agility and business value. Kurt Zimmer, Head of Data Engineering for Data Enablement at AstraZeneca. Jim Tyo, Chief Data Officer, Invesco. Data takes a long journey.

Pipeline-centric 84

Pipeline-centric Manufacturing Education Data Cleanse 84

nodeSWAT

JUNE 21, 2016

The following blog post is a long one, but hang in there, it will be worth it. Grab a cup of coffee and enjoy the deep dive into the NPM ecosystem. Various ways to configure NPM  — cache location and timeout, registry url, ignore-scripts , save-exact , sign-git-tag etc. It turned out there was a lot to discover. npmrc file.

Metadata 52

Metadata Google Cloud Coding Bytes 52

ProjectPro

MAY 31, 2021

Ace your big data interview by adding some unique and exciting Big Data projects to your portfolio. This blog lists over 20 big data projects you can work on to showcase your big data skills and gain hands-on experience in big data tools and technologies. Table of Contents What is a Big Data Project?

Big Data 52

Big Data Coding Project Hadoop 52

Robinhood

FEBRUARY 7, 2024

from Robinhood Data Infrastructure Robinhood adheres to a data-first philosophy. Every decision we make here (or every decision at the company), from feature rollouts to operational changes, is backed by data. When dealing with large-scale data, we turn to batch processing with distributed systems to complete high-volume jobs.

Process 75

Process Hadoop Architecture Accessible 75

Monte Carlo

MAY 17, 2023

Data observability, an organization’s ability to fully understand the health and quality of the data in their systems, has become one of the hottest technologies in modern data engineering. To help clarify, we reviewed hundreds of data observability deployments to identify 61 real data observability use cases and benefits.

Data 52

Data Data Pipeline Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

Data observability, an organization’s ability to fully understand the health and quality of the data in their systems, has become one of the hottest technologies in modern data engineering. To help clarify, we reviewed hundreds of data observability deployments to identify 61 real data observability use cases and benefits.

Data Pipeline 52

Data Pipeline Data Data Engineering Data Engineer 52