Data Engineering Podcast

JULY 22, 2019

Summary The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and access.

Data Lake 100

Data Lake Hadoop Data Architecture 100

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Knowledge Hut

OCTOBER 5, 2023

When it comes to cloud computing and big data, Amazon Web Services (AWS) has emerged as a leading name. As businesses’ reliance on cloud and big data increases, so does the demand for professionals who have the necessary skills and knowledge in AWS. Who is AWS Big Data Specialist?

Big Data 52

Big Data AWS Certification Amazon Web Services 52

Data Engineering Podcast

AUGUST 19, 2019

Summary Managing big data projects at scale is a perennial problem, with a wide variety of solutions that have evolved over the past 20 years. Designed as a fully integrated platform to meet the needs of enterprise grade analytics it provides a solution for the full lifecycle of data at massive scale.

Big Data 100

Big Data Hadoop Data Lake Media 100

Seattle Data Guy

MAY 7, 2024

Getting data out of source systems and into a data warehouse or data lake is one of the first steps in making it usable by analysts and data scientists. The question is how will your team do that?

Data Lake 130

Data Lake Data Warehouse Data Systems 130

databricks

JUNE 6, 2023

Apache Parquet is one of the most popular open source file formats in the big data world today. Being column-oriented, Apache Parquet allows.

Data Lake 74

Data Lake Big Data Data Data Engineering 74

Data Engineering Weekly

MAY 12, 2024

[link] Gradient Flow: Learning from the Past - Comparing the Hype Cycles of Big Data and GenAI The blogs compare the hype cycle of Big Data with Gen AI. The blog narrates the initial hype of Big Data, followed by talent shortages and lead time to build production applications.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Towards Data Science

JANUARY 26, 2023

How we deal with Big Data at Emplifi Continue reading on Towards Data Science »

Data Lake 69

Data Lake Building Data Science Big Data 69

Towards Data Science

FEBRUARY 28, 2023

Continue reading on Towards Data Science » Parquet vs ORC vs AVRO vs JSON. Which one to choose and how to use them?

Big Data 83

Big Data Data Science Data Data Lake 83

Seattle Data Guy

FEBRUARY 10, 2024

Photo by Tiger Lily Data warehouses and data lakes play a crucial role for many businesses. It gives businesses access to the data from all of their various systems. As well as often integrating data so that end-users can answer business critical questions.

Data Lake 162

Data Lake Data Warehouse Data Accessible 162

ProjectPro

SEPTEMBER 26, 2021

Big Data Engineer is one of the most popular job profiles in the data industry. This blog on Big Data Engineer salary gives you a clear picture of the salary range according to skills, countries, industries, job titles, etc. Big Data gets over 1.2 What does a big data engineer do?

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

Analytics Vidhya

FEBRUARY 27, 2023

Introduction Enterprises here and now catalyze vast quantities of data, which can be a high-end source of business intelligence and insight when used appropriately. Delta Lake allows businesses to access and break new data down in real time.

Data Lake 215

Data Lake Business Intelligence Designing Accessible 215

Data Engineering Podcast

MAY 22, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Stop struggling to speed up your data lake.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Data Engineering Podcast

OCTOBER 14, 2018

The Hive format is also built with the assumptions of a local filesystem which results in painful edge cases when leveraging cloud object storage for a data lake. One of the complicated problems in data modeling is managing table partitions. What are the unique challenges posed by using S3 as the basis for a data lake?

Data Lake 100

Data Lake Big Data Cloud Hadoop 100

ProjectPro

JULY 22, 2016

For the same cost, organizations can now store 50 times as much data as in a Hadoop data lake than in a data warehouse. Data lake is gaining momentum across various organizations and everyone wants to know how to implement a data lake and why.

Data Lake 40

Data Lake Data Warehouse Hadoop Unstructured Data 40

Knowledge Hut

MARCH 28, 2024

Role Level Intermediate Responsibilities Design and develop data pipelines to ingest, process, and transform data. Implemented and managed data storage solutions using Azure services like Azure SQL Database , Azure Data Lake Storage, and Azure Cosmos DB.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Christophe Blefari

JANUARY 20, 2024

📬 Subscribe to the excellent weekly newsletter 📬 A bit of context It's important to take a step back and to understand from where the data engineering is coming from. Data engineering inherits from years of data practices in US big companies. My advice on this point is to learn from others.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Towards Data Science

MAY 3, 2023

One such tool is the Versatile Data Kit (VDK), which offers a comprehensive solution for controlling your data versioning needs. VDK helps you easily perform complex operations, such as data ingestion and processing from different sources, using SQL or Python. Use VDK to build a data lake and merge multiple sources.

Data Lake 88

Data Lake Data SQL Data Warehouse 88

Confessions of a Data Guy

JANUARY 12, 2023

Anyone who’s been roaming around the forest of Data Engineering has probably run into many of the newish tools that have been growing rapidly around the concepts of Data Warehouses, Data Lakes, and Lake Houses … the merging of the old relational database functionality with TB and PB level cloud-based file storage systems.

Data Lake 162

Data Lake Relational Database Data Warehouse Data Engineering 162

Cloud Academy

MAY 2, 2018

Cloud Academy’s latest Learning Paths go broad and deep—covering CI/CD, machine learning, AI, big data, and even preparation for the first AWS certification designed for non-technical staff. This learning path focuses on getting your team up to speed using two key Azure services: Data Lake Analytics and Stream Analytics.

Google Cloud 52

Google Cloud Machine Learning Big Data AWS 52

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 92

Data Lake Data Warehouse BI SQL 92

Data Engineering Weekly

MARCH 3, 2024

The migration enhanced data quality, lineage visibility, performance improvements, cost reductions, and better reliability and scalability, setting a robust foundation for future expansions and onboarding.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Hevo

FEBRUARY 2, 2024

Data-driven organizations are searching for different storage solutions to manage the latency, volume, and resilience of big data and analytics. Initially, businesses used existing data lakes and warehouses in their tech stack to make the most out of data assets.

Architecture 40

Architecture Data Lake Big Data Data 40

Knowledge Hut

NOVEMBER 2, 2023

Azure Data Ingestion Pipeline Create an Azure Data Factory data ingestion pipeline to extract data from a source (e.g., Azure SQL Database, Azure Data Lake Storage). Data Aggregation Working with a sample of big data allows you to investigate real-time data processing, big data project design, and data flow.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Knowledge Hut

JANUARY 18, 2024

Data Science is an amalgamation of several disciplines, including computer science, statistics, and machine learning. As the world on the internet is becoming our second home, Big Data has exploded. Data Science is the study of this big data to derive a meaningful pattern.

Data Science 98

Data Science Amazon Web Services Finance Big Data 98

Knowledge Hut

NOVEMBER 17, 2023

To provide end users with a variety of ready-made models, Azure Data engineers collaborate with Azure AI services built on top of Azure Cognitive Services APIs. You must be able to create ETL pipelines using tools like Azure Data Factory and write custom code to extract and transform data if you want to succeed as an Azure Data Engineer.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Knowledge Hut

SEPTEMBER 25, 2023

They are moving their servers and on-premises data to Azure Cloud. What does all of this mean for Data Engineering professionals? In order to manage big data and other operational services, businesses are continuously in need of data engineers. Azure Data Engineers work with these and other solutions.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Big Data Tools

MAY 19, 2022

Apache Hudi 1.11.0 – This release of the well-known data lake has added many interesting changes. YuniKorn 1.0.0 – If you’ve been anxiously waiting for Kubernetes to come to data engineering, your wishes have been granted. is a scheduler targeting big data and ML workflows, and of course, it is cloud-native.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

MAY 19, 2022

Apache Hudi 1.11.0 – This release of the well-known data lake has added many interesting changes. YuniKorn 1.0.0 – If you’ve been anxiously waiting for Kubernetes to come to data engineering, your wishes have been granted. is a scheduler targeting big data and ML workflows, and of course, it is cloud-native.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Data Engineering Podcast

SEPTEMBER 7, 2020

In this episode he explains how it is designed to allow for querying and combining data where it resides, the use cases that such an architecture unlocks, and the innovative ways that it is being employed at companies across the world. What are the tradeoffs of using Presto on top of a data lake vs a vertically integrated warehouse solution?

Architecture 100

Architecture Data Architecture SQL Engineering 100

Towards Data Science

APRIL 10, 2023

A few Compelling Reasons to Choose Apache Iceberg for Data Lakes Continue reading on Towards Data Science »

Data Lake 96

Data Lake Data Science Data Big Data 96

Data Science Blog: Data Engineering

SEPTEMBER 24, 2021

Data Warehousing is applied Big Data Management and a key success factor in almost every company. Without a data warehouse, no company today can control its processes and make the right decisions on a strategic level as there would be a lack of data transparency for all decision makers.

Data Lake 52

Data Lake Data Warehouse Data Transparency Database 52

ProjectPro

JANUARY 24, 2023

With over 20 pre-built connectors and 40 pre-built transformers, AWS Glue is an extract, transform, and load (ETL) service that is fully managed and allows users to easily process and import their data for analytics. You can leverage AWS Glue to discover, transform, and prepare your data for analytics. If yes, which ones and how?

AWS 52

AWS Data Lake ETL Tools Scala 52

Data Engineering Podcast

APRIL 16, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Engineering 100

Data Engineering Data Engineer Engineering IT 100

Towards Data Science

SEPTEMBER 27, 2023

Data Access API over Data Lake Tables Without the Complexity Build a robust GraphQL API service on top of your S3 data lake files with DuckDB and Go Photo by Joshua Sortino on Unsplash 1. This data might be primarily used for internal reporting, but might also be valuable for other services in our organization.

Data Lake 94

Data Lake Accessible Accessibility SQL 94

Knowledge Hut

SEPTEMBER 27, 2023

Caching and data modeling methods help improve the dashboard speed. Big Data support The SPICE engine in QuickSight was built to handle huge datasets, making it suited for big data scenarios. Microsoft Power BI : The combination of BI and Azure Data Lake Storage is advantageous in big-data configurations.

BI 52

BI AWS Database-centric Data Lake 52

Cloudera

DECEMBER 14, 2021

Then, the company used Cloudera’s Data Platform as a foundation to build its own Network Real-time Analytics Platform (NRAP) and created the proper infrastructure to collect and analyze large-scale big data in real-time. . For this, the RTA transformed its data ingestion and management processes. .

Transportation 110

Transportation Telecommunication Banking Data Lake 110