Data Engineering Podcast

NOVEMBER 16, 2020

Summary The core mission of data engineers is to provide the business with a way to ask and answer questions of their data. This often takes the form of business intelligence dashboards, machine learning models, or APIs on top of a cleaned and curated data set.

Data Management 100

Data Management Management BI Business Intelligence 100

RandomTrees

NOVEMBER 29, 2023

Here the practice of data warehousing and warehouse system is very important and the use of right modelling techniques has become a very important factor in todays’ competitive world. In this choice, Big Data will play an important role and its choice is also inevitably crucial in the Business Intelligence and related systems.

Data Warehouse 52

Data Warehouse Data Lake Database-centric Data Cleanse 52

Monte Carlo

AUGUST 19, 2023

We’ll cover: What is a data platform? With companies moving their data platforms to the cloud, the emergence of cloud-native solutions ( data warehouse vs data lake or even a data lakehouse ) have taken over the market, offering more accessible and affordable options for storing data relative to many on-premises solutions.

Building 52

Building BI Data Lake Data Governance 52

Cloudera

JUNE 30, 2022

Supercharge your data lakehouse, make it open. The data lakehouse is not new to Cloudera or our customers. Now with Iceberg, CDP supports an open data lakehouse architecture that future-proofs our data platform for all our analytical workloads. We selected change data capture as our first use case on Iceberg.

Data Lake 86

Data Lake Business Intelligence Metadata Data Warehouse 86

Monte Carlo

APRIL 24, 2023

Data ingestion When we think about the flow of data in a pipeline, data ingestion is where the data first enters our platform. This data ingestion process can be accomplished by either querying the source directly, using upstream systems to publish events, or some combination of the two.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Monte Carlo

FEBRUARY 15, 2023

Cloud data warehouses solve these problems. Belonging to the category of OLAP (online analytical processing) databases, popular data warehouses like Snowflake, Redshift and Big Query can query one billion rows in less than a minute. What is a data warehouse?

Data Warehouse 52

Data Warehouse Unstructured Data AWS Business Intelligence 52

Towards Data Science

NOVEMBER 4, 2023

I’d like to discuss some popular Data engineering questions: Modern data engineering (DE). Does your DE work well enough to fuel advanced data pipelines and Business intelligence (BI)? Are your data pipelines efficient? Often it is a data warehouse solution (DWH) in the central part of our infrastructure.

Data Engineering 82

Data Engineering Data Engineer Engineering BI 82

Data Engineering Podcast

JUNE 5, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Data teams are increasingly under pressure to deliver.

Data Security 100

Data Security Metadata MongoDB Scala 100

phData: Data Engineering

SEPTEMBER 19, 2023

A rigid data model such as Kimball or Data Vault would ruin this flexibility and essentially transform your data lake into a data warehouse. However, some flexible data modeling techniques can be used to allow for some organization while maintaining the ease of new data additions.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 82

Data Warehouse Database-centric Metadata Cloud 82

Monte Carlo

JUNE 14, 2023

The modern data stack era , roughly 2017 to present data, saw the widespread adoption of cloud computing and modern data repositories that decoupled storage from compute such as data warehouses, data lakes, and data lakehouses.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

ProjectPro

JULY 22, 2016

For the same cost, organizations can now store 50 times as much data as in a Hadoop data lake than in a data warehouse. Data lake is gaining momentum across various organizations and everyone wants to know how to implement a data lake and why.

Data Lake 40

Data Lake Data Warehouse Hadoop Unstructured Data 40

AltexSoft

MARCH 30, 2023

What is Databricks Databricks is an analytics platform with a unified set of tools for data engineering, data management , data science, and machine learning. It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data.

Scala 64

Scala Data Lake BI Google Cloud 64

Cloudera

FEBRUARY 8, 2021

ECC will enrich the data collected and will make it available to be used in analysis and model creation later in the data lifecycle. Below is the entire set of steps in the data lifecycle, and each step in the lifecycle will be supported by a dedicated blog post(see Fig. 2 ECC data enrichment pipeline.

Data Pipeline 116

Data Pipeline Building Manufacturing Data Warehouse 116

AltexSoft

JUNE 26, 2023

Data collection is a methodical practice aimed at acquiring meaningful information to build a consistent and complete dataset for a specific business purpose — such as decision-making, answering research questions, or strategic planning. While all three are about data acquisition, they have distinct differences.

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. Instead, Facebook came to realize that the work we were doing transcended classic business intelligence. The traditional best practices of data warehousing are loosing ground on a shifting stack.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Rockset

JULY 29, 2022

Enterprise data warehouses (EDWs) became necessary in the 1980s when organizations shifted from using data for operational decisions to using data to fuel critical business decisions. Data warehouses are popular because they help break down data silos and ensure data consistency.

Data Analytics 52

Data Analytics Data Warehouse Datasets Google Cloud 52

DataKitchen

SEPTEMBER 21, 2023

Data in Place refers to the organized structuring and storage of data within a specific storage medium, be it a database, bucket store, files, or other storage platforms. In the contemporary data landscape, data teams commonly utilize data warehouses or lakes to arrange their data into L1, L2, and L3 layers.

Raw Data 67

Raw Data Data Business Intelligence High Quality Data 67

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Data Large volumes of structured or unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

AltexSoft

MARCH 14, 2023

As the volume and complexity of data continue to grow, organizations seek faster, more efficient, and cost-effective ways to manage and analyze data. In recent years, cloud-based data warehouses have revolutionized data processing with their advanced massively parallel processing (MPP) capabilities and SQL support.

IT 59

IT Data Warehouse Data Governance Data Lake 59

Ascend.io

JUNE 8, 2023

Is it possible to treat data not just as a necessary operational output, but as a product that holds immense strategic value? Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

ProjectPro

AUGUST 24, 2021

Data Warehousing: Data warehousing utilizes and builds a warehouse for storing data. A data engineer interacts with this warehouse almost on an everyday basis. Data Analytics: A data engineer works with different teams who will leverage that data for business solutions.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Knowledge Hut

APRIL 25, 2023

Data engineers add meaning to the data for companies, be it by designing infrastructure or developing algorithms. The practice requires them to use a mix of various programming languages, data warehouses, and tools. While they go about it - enter big data data engineer tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

phData: Data Engineering

JULY 18, 2023

With the birth of cloud data warehouses, data applications, and generative AI , processing large volumes of data faster and cheaper is more approachable and desired than ever. First up, let’s dive into the foundation of every Modern Data Stack, a cloud-based data warehouse.

Data Warehouse 52

Data Warehouse Pipeline-centric Government Data 52

Knowledge Hut

SEPTEMBER 26, 2023

Microsoft Azure's Azure Synapse, formerly known as Azure SQL Data Warehouse, is a complete analytics offering. Designed to tackle the challenges of modern data management and analytics, Azure Synapse brings together the worlds of big data and data warehousing into a unified and seamlessly integrated platform.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Cloudera

JUNE 16, 2022

CSP was recently recognized as a leader in the 2022 GigaOm Radar for Streaming Data Platforms report. Faster data ingestion: streaming ingestion pipelines. Figure 1: The evolution of Cloudera Stream Processing offering based on customers’ evolving streaming use cases and requirements.

Kafka 86

Kafka Manufacturing Data Lake SQL 86

ProjectPro

DECEMBER 7, 2021

Generally, data pipelines are created to store data in a data warehouse or data lake or provide information directly to the machine learning model development. Keeping data in data warehouses or data lakes helps companies centralize the data for several data-driven initiatives.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

ProjectPro

DECEMBER 6, 2016

Pentaho published a whitepaper titled “Hadoop and the Analytic Data Pipeline” that highlights the key categories which need to be focused on - Big Data Ingestion, Transformation, Analytics, Solutions. Source: [link] ) How Trifacta is helping data wranglers in Hadoop, the cloud, and beyond.Zdnet.com, November 4,2016.

Hadoop 52

Hadoop Data Lake BI Big Data 52

Knowledge Hut

JUNE 26, 2023

Finnhub API with Kafka for Real-Time Financial Market Data Pipeline Project Overview: The goal of this project is to construct a streaming data pipeline by making use of the real-time financial market data API provided by Finnhub.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

AltexSoft

MAY 14, 2021

Big Data analytics encompasses the processes of collecting, processing, filtering/cleansing, and analyzing extensive datasets so that organizations can use them to develop, grow, and produce better products. Big Data analytics processes and tools. Data ingestion. Data storage and processing.

Big Data 78

Big Data Data Analytics IT NoSQL 78

phData: Data Engineering

JANUARY 3, 2022

With that in place, data engineers can build data pipelines to allow data to flow out of the source systems. The result of this data pipeline is then stored in a separate location — generally in a highly available format for various business intelligence tools to query.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Governance 52

AltexSoft

JULY 29, 2022

Developed in 2006 by Doug Cutting and Mike Cafarella to run the web crawler Apache Nutch, it has become a standard for Big Data analytics. According to the study by the Business Application Research Center (BARC), Hadoop found intensive use as. a suitable technology to implement data lake architecture.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

ProjectPro

JANUARY 31, 2023

There are three steps involved in the deployment of a big data model: Data Ingestion: This is the first step in deploying a big data model - Data ingestion, i.e., extracting data from multiple data sources. The prepared data is then stored in a data warehouse or a similar repository.

Big Data 52

Big Data Hadoop AWS Relational Database 52

phData: Data Engineering

JANUARY 3, 2022

This commonly introduces: Database or Data Warehouse API/EDI Integrations ETL software Business intelligence tooling By leveraging off-the-shelf tooling, your company separates disciplines by technology. This required applying transformations and filters to the data for various business units.

IT 52

IT AWS Software Engineer Software Engineering 52

ProjectPro

OCTOBER 20, 2021

Having multiple data integration routes helps optimize the operational as well as analytical use of data. Experimentation in production Big Data Data Warehouse for core ETL tasks Direct data pipelines Tiered Data Lake 4. Data: Data Engineering Pipelines Data is everything.

Machine Learning 52

Machine Learning Algorithm Government Data Science 52