Data Engineering Podcast

JUNE 19, 2022

In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance. RudderStack helps you build a customer data platform on your warehouse or data lake.

Metadata 130

Metadata MongoDB Scala MySQL 130

Knowledge Hut

SEPTEMBER 26, 2023

Born out of the minds behind Apache Spark, an open-source distributed computing framework, Databricks is designed to simplify and accelerate data processing, data engineering, machine learning, and collaborative analytics tasks. This flexibility allows organizations to ingest data from virtually anywhere.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Cloudera

JUNE 30, 2022

Over the past decade, Cloudera has enabled multi-function analytics on data lakes through the introduction of the Hive table format and Hive ACID. Companies, on the other hand, have continued to demand highly scalable and flexible analytic engines and services on the data lake, without vendor lock-in.

Data Lake 86

Data Lake Business Intelligence Metadata Data Warehouse 86

AltexSoft

MARCH 30, 2023

It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data. The relatively new storage architecture powering Databricks is called a data lakehouse. Databricks lakehouse platform architecture.

Scala 64

Scala Data Lake BI Google Cloud 64

Data Engineering Podcast

SEPTEMBER 18, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

Hospitality 100

Hospitality Food MongoDB Scala 100

Monte Carlo

AUGUST 19, 2023

We’ll cover: What is a data platform? Data-first companies have embraced data platforms as an effective way to aggregate, operationalize, and democratize data at scale across the organization.

Building 52

Building BI Data Lake Data Governance 52

Monte Carlo

APRIL 24, 2023

In this article, we’ll dive deep into the data presentation layers of the data stack to consider how scale impacts our build versus buy decisions, and how we can thoughtfully apply our five considerations at various points in our platform’s maturity to find the right mix of components for our organizations unique business needs.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Data Engineering Podcast

OCTOBER 2, 2022

Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. RudderStack helps you build a customer data platform on your warehouse or data lake. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

IT 100

IT Food PostgreSQL MongoDB 100

RandomTrees

NOVEMBER 29, 2023

The enterprise data warehouse systems must be able to provide valuable and relevant information to business processes and downstream applications including user queries and reporting requirements. This type of architecture is more preferred in any enterprise where agile is more predominant and also suits any data lake paradigms.

Data Warehouse 52

Data Warehouse Data Lake Database-centric Data Cleanse 52

Cloudera

JUNE 16, 2022

This blog aims to answer two questions as illustrated in the diagram below: How have stream processing requirements and use cases evolved as more organizations shift to “streaming first” architectures and attempt to build streaming analytics pipelines? Faster data ingestion: streaming ingestion pipelines.

Kafka 86

Kafka Manufacturing Data Lake SQL 86

WeCloudData

OCTOBER 19, 2021

Conclusion WeCloudData helped a client build a flexible data pipeline to address the needs from multiple business units requiring different sets, views and timelines of job market data. The client intends to build on and improve this data pipeline by moving towards a more serverless architecture and adding DevOps tools & workflows.

Consulting 52

Consulting Raw Data Data Lake Data Pipeline 52

WeCloudData

OCTOBER 19, 2021

Conclusion WeCloudData helped a client build a flexible data pipeline to address the needs from multiple business units requiring different sets, views and timelines of job market data. The client intends to build on and improve this data pipeline by moving towards a more serverless architecture and adding DevOps tools & workflows.

Consulting 52

Consulting Raw Data Data Lake Data Pipeline 52

Towards Data Science

JULY 24, 2023

A decade later, a considerable amount of innovation in the data management industry is revolving around how different data platforms are coupling or decoupling storage and compute (don’t worry I include examples in the next section). Why are these things related, and more importantly, why should data leaders care?

Metadata 95

Metadata Data Warehouse Data Lake Data Science 95

AltexSoft

JUNE 26, 2023

Read our article on Hotel Data Management to have a full picture of what information can be collected to boost revenue and customer satisfaction in hospitality. While all three are about data acquisition, they have distinct differences. Read our articles on structured vs unstructured data and unstructured data to learn more.

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

AltexSoft

MARCH 14, 2023

Known as the Modern Data Stack (MDS) , this suite of tools and technologies has transformed how businesses approach data management and analysis. What is a modern data stack? A data stack, in turn, focuses on data : It helps businesses manage data and make the most out of it. Modern data stack architecture.

IT 59

IT Data Warehouse Data Governance Data Lake 59

Snowflake

JULY 18, 2023

Data infrastructure that makes light work of complex tasks Built as a connected application from day one, the anecdotes Compliance OS uses the Snowflake Data Cloud for data ingestion and modeling, including a single cybersecurity data lake where all data can be analyzed within Snowflake.

Data Process 52

Data Process Process Data Lake Data Ingestion 52

Knowledge Hut

APRIL 25, 2023

Top Data Engineering Tools We've compiled a list of the top data engineering tools in 2023 that offer a range of functionalities, including data integration, processing, transformation, and visualization, to help data engineers extract actionable insights from data. Let’s take a look: 1.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Snowflake

AUGUST 16, 2023

Streaming and Continuous Pipelines & Data Onboarding Snowpipe Streaming – general availability Streaming pipelines are often challenging to build. Data comes in a continuous manner, and often a separate architecture is required to handle streaming data. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Silectis

JANUARY 3, 2021

This data engineering glossary is by no means exhaustive, but should provide some foundational context and information. Advanced Analytics The process of discovering deeper insights in data than typically enabled by most business intelligence (BI) tools. Big Data Large volumes of structured or unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Monte Carlo

JULY 18, 2023

A decade later, a considerable amount of innovation in the data management industry is revolving around how different data platforms are coupling or decoupling storage and compute (don’t worry I include examples in the next section). Why are these things related, and more importantly, why should data leaders care?

Metadata 52

Metadata Data Warehouse Data Lake Government 52

ProjectPro

DECEMBER 6, 2016

News on Hadoop-November 2016 Microsoft's Hadoop-friendly Azure Data Lake will be generally available in weeks. Microsoft's cloud-based Azure Data Lake will soon be available for big data analytic workloads. Azure Data Lake will have 3 important components -Azure Data Lake Analytics, Azure Data Lake Store and U-SQL.

Hadoop 52

Hadoop Data Lake BI Big Data 52

AltexSoft

JULY 29, 2022

a runtime environment (sandbox) for classic business intelligence (BI), advanced analysis of large volumes of data, predictive maintenance , and data discovery and exploration; a store for raw data; a tool for large-scale data integration ; and. a suitable technology to implement data lake architecture.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

LinkedIn Engineering

JULY 27, 2023

Our approach to building Costwiz For dashboards and alerts through regular emails, we decided to use native reporting tools available to us like Power BI to help us scale quickly. Instead of point-to-point integrations, the platform is built on Extract, Transform, Load (ETL) principles to handle data from various source systems.

Metadata 75

Metadata Utilities Cloud Data Lake 75

Monte Carlo

JULY 18, 2023

A decade later, a considerable amount of innovation in the data management industry is revolving around how different data platforms are coupling or decoupling storage and compute (don’t worry I include examples in the next section). Why are these things related, and more importantly, why should data leaders care?

Metadata 40

Metadata Data Warehouse Data Lake Government 40

Ascend.io

JANUARY 3, 2023

Common examples include: Internal teams, possibly in a mesh or fabric architecture, can combine different data products. For example, a customer preference engine can merge internal domain data into segmentation data to enable rapid insights and improve customer satisfaction.

Data 52

Data Data Lake Business Intelligence Big Data 52

ProjectPro

AUGUST 24, 2021

If you are a newbie in data engineering and are interested in exploring real-world data engineering projects, check out the list of best data engineering project examples below. With the trending advance of IoT in every facet of life, technology has enabled us to handle a large amount of data ingested with high velocity.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Knowledge Hut

JUNE 26, 2023

A complete end-to-end stream processing pipeline is shown here using an architectural diagram. The pipeline in this reference design collects data from two different sources, then conducts a join operation on related records from each stream, then enriches the output, and finally produces an average.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Snowflake

MARCH 30, 2023

3EJHjvm Once a business need is defined and a minimal viable product ( MVP ) is scoped, the data management phase begins with: Data ingestion: Data is acquired, cleansed, and curated before it is transformed. Feature engineering: Data is transformed to support ML model training. ML workflow, ubr.to/3EJHjvm

Engineering 83

Engineering Raw Data Data Science Scala 83

Rockset

NOVEMBER 6, 2019

With event-driven architectures powered by systems like Apache Kafka becoming more prominent, there are now many applications in the modern software stack that make use of events and messages to operate effectively. Tools like Apache Superset are commonly used to analyze and visualize the data in Druid.

Kafka 40

Kafka Data Lake SQL Hadoop 40

Confluent

SEPTEMBER 26, 2019

In the early days, many companies simply used Apache Kafka ® for data ingestion into Hadoop or another data lake. explains in more detail why many new integration architectures leverage Apache Kafka instead of legacy tools like RabbitMQ, ETL, and ESB. However, Apache Kafka is more than just messaging.

Kafka 20

Kafka BI SQL Datasets 20

ProjectPro

OCTOBER 20, 2021

AutoKeras focuses on making machine learning and deep learning more accessible with the help of Neural Architecture Search. Auto-Weka : Weka is a top-rated java-based machine learning software for data exploration. It is a function to find the best model with minimal knowledge or effort from the Data Scientist.

Machine Learning 52

Machine Learning Algorithm Government Data Science 52