AltexSoft

AUGUST 29, 2023

The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. This article explains what a data lake is, its architecture, and diverse use cases. Data warehouse vs. data lake in a nutshell.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Monte Carlo

FEBRUARY 15, 2023

Cloud data warehouses solve these problems. Belonging to the category of OLAP (online analytical processing) databases, popular data warehouses like Snowflake, Redshift and Big Query can query one billion rows in less than a minute. What is a data warehouse?

Data Warehouse 52

Data Warehouse Unstructured Data AWS Business Intelligence 52

Ascend.io

FEBRUARY 23, 2024

You are a data professional. This means that you are fully aware of the recent developments in AI and Large Language Models ( LLMs ). Exciting, isn’t it Yet, embarking on the AI adoption journey introduces a series of challenges, with one of the most significant being the readiness of your data platform.

Cloud Storage 52

Cloud Storage Data Ingestion Machine Learning Algorithm 52

Knowledge Hut

APRIL 25, 2024

They simplify data processing for our brains and give readers a quick overview of past, present, and future performance by helping the user to visualize otherwise complex and weighty raw data. Faster analysis, intuitive dashboards BI solutions are designed to do intense data processing on the cloud or your company's servers.

Business Intelligence 52

Business Intelligence BI Raw Data Data Lake 52

AltexSoft

MARCH 14, 2023

As the volume and complexity of data continue to grow, organizations seek faster, more efficient, and cost-effective ways to manage and analyze data. In recent years, cloud-based data warehouses have revolutionized data processing with their advanced massively parallel processing (MPP) capabilities and SQL support.

IT 59

IT Data Warehouse Data Governance Data Lake 59

Towards Data Science

MAY 3, 2023

VDK helps you easily perform complex operations, such as data ingestion and processing from different sources, using SQL or Python. You can use VDK to build data lakes and ingest raw data extracted from different sources, including structured, semi-structured, and unstructured data.

Data Lake 83

Data Lake Data SQL Data Warehouse 83

Data Engineering Weekly

JANUARY 15, 2023

Customers with the data catalogs, whether embedded or not, express three times high satisfaction with their analytical infrastructure - David Menninger. I write about how Data Catalogs are failing with the recent article Data Catalog - A Broken Promise. Data Catalogs are moving towards a feature, not a product.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Snowflake

MARCH 30, 2023

Collecting, cleaning, and organizing data into a coherent form for business users to consume are all standard data modeling and data engineering tasks for loading a data warehouse. Based on Tecton blog So is this similar to data engineering pipelines into a data lake/warehouse?

Engineering 83

Engineering Raw Data Data Science Scala 83

Maxime Beauchemin

AUGUST 28, 2017

This post follows up on The Rise of the Data Engineer , a recent post that was an attempt at defining data engineering and described how this new role relates to historical and modern roles in the data space. Conflicting nomenclature and inconsistent data across different namespaces, or “ data marts ” are problematic.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174

Snowflake

MARCH 21, 2023

In Snowflake’s 6 Data Science Trends in 2023 , one of the more prominent trends we identified was the emerging use of unified tools and infrastructure for SQL and Python. This knowledge-sharing ultimately creates more agile data engineering and MLprojects with better long-term results.

SQL 52

SQL Python Programming Language Scala 52

AltexSoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Data warehouse vs data lake vs data lakehouse: What’s the difference.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

AltexSoft

JANUARY 26, 2022

In recent years, it’s getting more common to see organizations looking for a mysterious analytics engineer. As you may guess from the name, this role sits somewhere in the middle of a data analyst and data engineer, but it’s really neither one nor the other. Here’s the video explaining how data engineers work.

Engineering 59

Engineering Software Engineer Software Engineering Data Warehouse 59

DoorDash Engineering

APRIL 12, 2023

Metrics are vital for measuring success in any data-driven company, but ensuring that these metrics are consistently and accurately measured across the organization can be challenging. These steps ensure that our data is always up-to-date and complete. delivery_id is a primary key for this source and the timestamp is in UTC.

SQL 82

SQL Metadata Raw Data Government 82

Monte Carlo

AUGUST 19, 2023

With companies moving their data platforms to the cloud, the emergence of cloud-native solutions ( data warehouse vs data lake or even a data lakehouse ) have taken over the market, offering more accessible and affordable options for storing data relative to many on-premises solutions.

Building 52

Building BI Data Lake Data Governance 52

Knowledge Hut

DECEMBER 26, 2023

What is Data Science? What are the roles and responsibilities of a Data Engineer? How does one become a Data engineer, and what skills are required? Our primary focus in this article will be to answer all these questions and take you a step forward towards your dream of becoming a Data engineer. And many more.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Cloudera

SEPTEMBER 15, 2022

Modern data platforms deliver an elastic, flexible, and cost-effective environment for analytic applications by leveraging a hybrid, multi-cloud architecture to support data fabric, data mesh, data lakehouse and, most recently, data observability. What is a data fabric?

Data Lake 81

Data Lake Analytics Application Cloud Storage Architecture 81

Knowledge Hut

NOVEMBER 16, 2023

This article illustrates the blend between a data scientist and Azure as a cloud platform. You can learn more about Data Science course that helps you understand how to tackle complex data science problems through practical exercises. Azure Synapse Analytics is a one-stop solution for all data analysis requirements.

Data Science 52

Data Science Technology Programming Language Cloud Computing 52

dbt Developer Hub

APRIL 30, 2019

This article does not seek to instruct you on how to design a final model for your stakeholders — it won’t cover whether you should denormalize everything into one wide master table , or have many tables that need to be joined together in the BI layer. Over time, we’ve developed internal conventions on how we structure them.

Project 40

Project Database-centric Raw Data Data Warehouse 40

Rockset

JANUARY 5, 2022

A good number of organizations have moved on from SQL databases, thinking there is no possibility that they could meet the demanding requirements of modern data applications. We’ll examine some commonly held misconceptions regarding SQL databases in this article. But nothing could be further from the truth.

Database 52

Database SQL NoSQL Raw Data 52

dbt Developer Hub

MARCH 9, 2022

In this article, we’ll explore a real-life baton pass between ML engineering and analytics engineering and highlighting where things went wrong. Luckily, the Modern Data Stack is making this baton pass smoother. Here’s what happened ​ After some initial planning, I knew we had this raw data living somewhere in our data warehouse.

Machine Learning 52

Machine Learning Raw Data SQL Python 52

Monte Carlo

JUNE 14, 2023

The modern data stack era , roughly 2017 to present data, saw the widespread adoption of cloud computing and modern data repositories that decoupled storage from compute such as data warehouses, data lakes, and data lakehouses. They also recently acquired Apache Flink , another streaming solution.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Monte Carlo

AUGUST 31, 2023

Data is a priority for your CEO, as it often is for digital-first companies, and she is fluent in the latest and greatest business intelligence tools. Your CTO is excited about migrating to the cloud, and constantly sends your team articles highlighting performance measurements against some of the latest technologies.

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Monte Carlo

JULY 7, 2022

They codify data needs in a data contract and schemas from consumers up front, and then deliver the data pre-modeled into the data warehouse. It’s becoming clear a data engineering movement is afoot as we have recently covered other organizations taking a similar approach: Convoy and Vimeo.

Data Warehouse 52

Data Warehouse Software Engineer Software Engineering Data 52

Monte Carlo

APRIL 1, 2021

is whether to choose a data warehouse or lake to power storage and compute for their analytics. While data warehouses provide structure that makes it easy for data teams to efficiently operationalize data (i.e., And it’s an increasingly relevant one for modern data teams.

Data Lake 52

Data Lake Unstructured Data Data Warehouse Metadata 52

Ascend.io

APRIL 12, 2023

What about data quality versus data integrity? You might be surprised that we keep referencing data integrity and not data quality, so let’s talk about why we don’t use these terms interchangeably. The broad objective is to improve data integrity and make data trustworthy and actionable for data-driven decisions.

Data Pipeline 52

Data Pipeline Data Integration Datasets Data 52

Monte Carlo

JULY 30, 2020

Most of all, however, your entire data team wishes there were an easier way to manage the growing needs of your organization, from fielding the never-ending stream of ad hoc queries to wrangling disparate data sources through a central ETL pipeline. Image courtesy of Monte Carlo Data.) (If Why use a data mesh?

IT 45

IT Data Lake Business Intelligence Data Architecture 45

Cloudera

MARCH 2, 2018

If you do an internet search for ‘data-driven disruption’ you can find articles about almost every industry being disrupted by digitalisation and new applications of data. While there are instances of data-driven efforts in the nonprofit sector, they are not as widespread as they can be.

Hospitality 43

Hospitality Healthcare Machine Learning Transportation 43

ProjectPro

NOVEMBER 14, 2015

The aim of this article is to mention some very common projects involving Apache Hadoop and Apache Spark. Get FREE Access to Data Analytics Example Codes for Data Cleaning, Data Munging, and Data Visualization 1. To this group, we add a storage account and move the raw data.

Hadoop 52

Hadoop Project Big Data Healthcare 52