Knowledge Hut

APRIL 23, 2024

In the modern data-driven landscape, organizations continuously explore avenues to derive meaningful insights from the immense volume of information available. Two popular approaches that have emerged in recent years are data warehouse and big data. Data warehousing offers several advantages.

Data Warehouse 52

Data Warehouse Big Data Unstructured Data Hadoop 52

The Pragmatic Engineer

AUGUST 10, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Notion : for every company I talked to, I took notes on Notion to record what we talked about. 👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter.

Engineering 187

Engineering Recruitment Retail Software Engineer 187

DoorDash Engineering

FEBRUARY 22, 2023

Note: A few months back, our peers at the DashMart engineering team released an article elaborating on how they used CockroachDB changefeed to support real-time inventory changes. The gap between receiving data from the merchant and displaying data to the customer should be as small as possible.

Designing 124

Designing Database Architecture SQL 124

Christophe Blefari

NOVEMBER 18, 2022

Scracthing the surface ( credits ) Hey you, a new Friday means data news. This week feels a bit like old data news with a variety of articles on different cool topics while I navigate through the actual data trends. Next Monday I'll present "How to build a data dream team" at Y42 meetup.

Python 130

Python Data Warehouse Data SQL 130

Christophe Blefari

NOVEMBER 11, 2022

On a social note, today I've joined data-folks Mastodon server, you can follow me there. I'll speak about "How to build the data dream team" Let's jump onto the news. Ingredients of a Data Warehouse Going back to basics. Ian describes how Riot Games uses data and what machine learning means.

BI 130

BI Data Warehouse Data Database 130

Towards Data Science

MAY 3, 2023

Data Engineering Learn about slow change dimensions (SCD) and how to implement SCD Type 2 in VDK Photo by Joshua Sortino on Unsplash Data is the backbone of any organization, and in today’s fast-paced world, it is crucial to keep track of its versions. Note for images : all the images are by the author, except where explicitly stated.

Data Lake 86

Data Lake Data SQL Data Warehouse 86

Cloudera

DECEMBER 21, 2023

How long would it take you to find the data you need to even begin to come up with a data-driven response? Creating a query when you’ve new to a data model Whether you’re new to a role, or just new to a given data source, finding data is 90 percent of the query creation problem. That would be amazing wouldn’t it?

SQL 67

SQL Data Warehouse Business Analyst Data Schemas 67

Workfall

MARCH 28, 2023

In this blog, we will cover: GraphQL Hasura PostgreSQL Hands-on Conclusion GraphQL GraphQL is an API query language and runtime for answering queries with existing data. It allows you to run custom business logic over GraphQL by supporting data modeling, real-time querying, event programming, role-based authorization, and actions.

PostgreSQL 75

PostgreSQL Coding MySQL Database 75

Knowledge Hut

FEBRUARY 1, 2024

While building predictive models, if your results aren’t satisfactory, then the two things that can go wrong are data or models. Choosing the right data is the first step in any data science application. Then comes the data format. Data cleaning in data science plays a pivotal role in your analysis.

Data Science 52

Data Science Process Data Cleanse Datasets 52

Towards Data Science

APRIL 11, 2023

A hands-on example of how to apply geospatial index Introduction Geospatial Indexing is an indexing technique that provides an elegant way to manage location-based data. It makes geospatial data can be searched and retrieved efficiently so that the system can provide the best experience to its users. Let’s get started.

Bytes 61

Bytes Google Cloud Datasets Programming Language 61

Towards Data Science

DECEMBER 23, 2023

Photo by Ian Taylor on Unsplash This tutorial guides you through an analytics use case, analyzing semi-structured data with Spark SQL. We’ll start with the data engineering process, pulling data from an API and finally loading the transformed data into a data lake (represented by MinIO ).

SQL 74

SQL Data Analytics Hadoop Raw Data 74

Cloudera

FEBRUARY 14, 2024

Introduction Apache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data — structured and unstructured. Iceberg doesn’t delete the old data files. This can be an equality delete file or a positional delete file.

Bytes 56

Bytes Metadata Data Lake SQL 56

Knowledge Hut

DECEMBER 21, 2023

In the present-day world, almost all industries are generating humongous amounts of data, which are highly crucial for the future decisions that an organization has to make. This massive amount of data is referred to as “big data,” which comprises large amounts of data, including structured and unstructured data that has to be processed.

Hadoop 52

Hadoop Big Data NoSQL Unstructured Data 52

Monte Carlo

NOVEMBER 20, 2023

Data quality is important—no doubt about it. Bad data destroys trust in both the data products and the teams that produce them. But as data quality finally takes its rightful seat at the platform table, where should new data teams start? Data Observability? Table of Contents What is data quality?

Government 69

Government Data Data Governance Datasets 69

Towards Data Science

FEBRUARY 19, 2024

It worked surprisingly well, in my opinion, but I couldn’t help wondering if it would be better with more data. books, but there are millions in the Openlibrary data set , and I thought it was worthwhile to try adding more options to the search space. We 15x’ed the data, which gives the search much more to work with.

AWS 60

AWS Building Bytes Python 60

Data Engineering Podcast

APRIL 12, 2021

Summary "Business as usual" is changing, with more companies investing in data as a first class concern. As a result, the data team is growing and introducing more specialized roles. If you are wondering how to apply your talents and interests to working with data then this episode is a must listen.

Data Warehouse 100

Data Warehouse Data Pipeline BI Data 100

Workfall

JULY 25, 2023

Reading Time: 9 minutes Tired of manual data entry in Google Spreadsheets? Discover a simple and efficient way to automate your data handling using Python and Sheety API. Say goodbye to time-consuming tasks and hello to streamlined data integration – let’s get started! In this blog, we will cover: What Is Sheety API?

Python 76

Python Programming Language Coding Project 76

Snowflake

MARCH 30, 2023

A 2016 data science report from data enrichment platform CrowdFlower found that data scientists spend around 80% of their time in data preparation (collecting, cleaning, and organizing of data) before they can even begin to build machine learning (ML) models to deliver business value.

Engineering 81

Engineering Raw Data Data Science Scala 81

Towards Data Science

JULY 10, 2023

Many other fields have faced similar challenges and have created their own frameworks and processes that I can leverage when thinking about how to approach data modeling. These sources helped shape the following process for designing data models from scratch. It’s important to note, however, this process is tool-agnostic.

Designing 98

Designing Electronics Coding Engineering 98

RandomTrees

NOVEMBER 29, 2023

Here the practice of data warehousing and warehouse system is very important and the use of right modelling techniques has become a very important factor in todays’ competitive world. In this choice, Big Data will play an important role and its choice is also inevitably crucial in the Business Intelligence and related systems.

Data Warehouse 52

Data Warehouse Data Lake Database-centric Data Cleanse 52

Data Engineering Podcast

JULY 8, 2019

Summary The market for data warehouse platforms is large and varied, with options for every use case. ClickHouse is an open source, column-oriented database engine built for interactive analytics with linear scalability. It was interesting to learn about some of the custom data types and performance optimizations that are included.

Data Warehouse 100

Data Warehouse MySQL Data Lake Hadoop 100

Big Data Tools

DECEMBER 7, 2021

Hi, I’m Pasha Finkelshteyn , and I’ll be your guide today through this month’s installment of the Data Engineering Annotated Monthly. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

DECEMBER 7, 2021

Hi, I’m Pasha Finkelshteyn , and I’ll be your guide today through this month’s installment of the Data Engineering Annotated Monthly. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Knowledge Hut

JANUARY 18, 2024

Along with the ETL, data transformation and data modeling options. I've noticed a growing trend of businesses adopting Power BI and Fabric tools to elevate their data capabilities and refine decision-making processes. Before we proceed, let's have a look at what big data is and the types of big data.

BI 52

BI Data Cleanse Business Intelligence Datasets 52

Cloudera

NOVEMBER 10, 2021

Cloudera Data Platform (CDP) brings many improvements to customers by merging technologies from the two legacy platforms, Cloudera Enterprise Data Hub (CDH) and Hortonworks Data Platform (HDP). It is useful in defining and enforcing different levels of privileges on data for users on a Hadoop cluster.

Hadoop 75

Hadoop SQL Database Kafka 75

Grouparoo

OCTOBER 6, 2021

It was a fun experience and I think we made a good choice by picking 97 Things Every Data Engineer Should Know. This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Towards Data Science

JUNE 27, 2023

The Top 5 Features for Efficient Data Manipulation This April, pandas 2.0.0 was officially launched , making huge waves across the data science community. Due to its extensive functionality and versatility, pandas has secured a place in every data scientist’s heart. Photo by Yancy Min on Unsplash. Yep, pandas 2.0

Pipeline-centric 98

Pipeline-centric Data Science Machine Learning Datasets 98

dbt Developer Hub

NOVEMBER 22, 2022

Once your data warehouse is built out, the vast majority of your data will have come from other SaaS tools, internal databases, or customer data platforms (CDPs). Spreadsheets are the Swiss army knife of data processing. How big is the dataset? Does it have a consistent format? How frequently will it change?

Data Warehouse 52

Data Warehouse ETL Tools Google Cloud Cloud Storage 52

Towards Data Science

SEPTEMBER 27, 2023

Data Access API over Data Lake Tables Without the Complexity Build a robust GraphQL API service on top of your S3 data lake files with DuckDB and Go Photo by Joshua Sortino on Unsplash 1. Suppose that we have a data lake table that stores real-time statistics about our customers generated by some Spark application.

Data Lake 92

Data Lake Accessible Accessibility SQL 92

FreshBI

NOVEMBER 9, 2022

Multiple keys for single dimensions: This first use for SELECTCOLUMNS comes from a common case where two columns can appear in a fact table with IDs referring to one dimension table. Below, I have a simple data model, we can see we have an Employees table joined to fact table “Jobs” by the employee ID. Same file size impact as 1.

BI 52

BI Computer Science Project Education 52

Cloudera

NOVEMBER 15, 2022

It is one of the main data services that run on Cloudera Data Platform (CDP) Public Cloud. In this context, non-atomic renames could cause not only client read inconsistencies, but even data loss. User data in HBase . You can access COD from your CDP console.

Database 76

Database Cloud Cloud Storage Architecture 76

Towards Data Science

JUNE 6, 2023

Fitbit activity analysis with DuckDB Photo by Jake Hills on Unsplash Wearable fitness trackers have become an integral part of our lives, collecting and tracking data about our daily activities, sleep patterns, location, heart rate, and much more. What insights are buried within my archive of personal fitness activity data?

Data Warehouse 91

Data Warehouse Data SQL Data Analysis 91

Cloudera

AUGUST 26, 2021

Ozone natively provides Amazon S3 and Hadoop Filesystem compatible endpoints in addition to its own native object store API endpoint and is designed to work seamlessly with enterprise scale data warehousing, machine learning and streaming workloads. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 108

Data Science Cloud Hadoop Metadata 108

Snowflake

NOVEMBER 29, 2023

Data cloud technology can accelerate FAIRification of the world’s biomedical patient data. In other instances, the concern is primarily the risk of potential patient re-identification that comes with longitudinal data enrichment.

Healthcare 76

Healthcare Metadata Medical Data Storage 76

Data Science Blog: Data Engineering

NOVEMBER 15, 2023

In addition to Business Intelligence (BI), Process Mining is no longer a new phenomenon, but almost all larger companies are conducting this data-driven process analysis in their organization. The Event Log Data Model for Process Mining Process Mining as an analytical system can very well be imagined as an iceberg.

Architecture 52

Architecture Database-centric Process BI 52

Scott Logic

NOVEMBER 6, 2023

In this post we’ll take a whistle-stop tour around the topic of data, going from sources and types through to use cases and the technologies involved in processing data. Introduction Over the last decade we’ve heard more and more about data.

Data Lake 52

Data Lake Kafka BI Unstructured Data 52

Data Engineering Podcast

MARCH 15, 2021

Summary The team at DoorDash has a complex set of optimization challenges to deal with using data that they collect from a multi-sided marketplace. This is a valuable look at how to manage a large and growing data platform with that supports a variety of teams with varied and evolving needs.

Management 100

Management Data Warehouse PostgreSQL Kafka 100