AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

JUNE 17, 2023

In this episode Satish Jayanthi explores the benefits of incorporating column-aware tooling in the data modeling process. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 130

Data Lake Metadata Data Architecture Machine Learning 130

phData: Data Engineering

APRIL 4, 2023

Today we want to introduce Fivetran’s support for Amazon S3 with Apache Iceberg, investigate some of the implications of this feature, and learn how it fits into the modern data architecture as a whole. Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Monte Carlo

JANUARY 23, 2023

When it comes to the data community, there’s always a debate broiling about something— and right now “data mesh vs data lake” is right at the top of that list. In this post we compare and contrast the data mesh vs data lake to illustrate the benefits of each and help discover what’s right for your data platform.

Data Lake 52

Data Lake Unstructured Data Business Intelligence Architecture 52

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Towards Data Science

APRIL 3, 2023

The post-modern data stack is coming. If you don’t like change, data engineering is not for you. The most prominent, recent examples are Snowflake and Databricks disrupting the concept of the database and ushering in the modern data stack era. It’s important to note the data is still in a relatively raw state.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Warehouse 98

Monte Carlo

APRIL 4, 2023

Data mesh vs data warehouse is an interesting framing because it is not necessarily a binary choice depending on what exactly you mean by data warehouse (more on that later). Despite their differences, however, both approaches require high-quality, reliable data in order to function. What is a Data Mesh?

Data Warehouse 52

Data Warehouse Data Governance Data Architecture 52

Data Engineering Podcast

JULY 8, 2019

Summary The market for data warehouse platforms is large and varied, with options for every use case. ClickHouse is an open source, column-oriented database engine built for interactive analytics with linear scalability. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Data Warehouse 100

Data Warehouse MySQL Data Lake Hadoop 100

Towards Data Science

JANUARY 16, 2024

When I heard the words ‘decentralised data architecture’, I was left utterly confused at first! In my then limited experience as a Data Engineer, I had only come across centralised data architectures and they seemed to be working very well. Result: Data warehouse was born. So what was missing?

Data Warehouse 67

Data Warehouse Data Architecture Relational Database NoSQL 67

Edureka

FEBRUARY 9, 2023

Azure Data Engineering is a rapidly growing field that involves designing, building, and maintaining data processing systems using Microsoft Azure technologies. Contents: What is the role of an Azure Data Engineer?

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Monte Carlo

APRIL 1, 2021

Over the past few years, data lakes have emerged as a must-have for the modern data stack. But while the technologies powering our access and analysis of data have matured, the mechanics behind understanding this data in a distributed environment have lagged behind.

Data Lake 52

Data Lake Unstructured Data Data Warehouse Metadata 52

Knowledge Hut

JUNE 30, 2023

The demand for experienced data engineers continuously expands in today's data-driven environment. Books on data engineering serve as essential resources to guide you through the vast terrain of data engineering. What is Data Engineering? Who are Data Engineers?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Scott Logic

APRIL 18, 2024

In this episode, Oliver Cronk, Andrew Carr and David Hope talk about the ever-changing world of data, with conversations moving from data warehouse to data lake, and data mesh to data fabric.

Data Lake 59

Data Lake Data Warehouse Data Architecture Architecture 59

Knowledge Hut

NOVEMBER 28, 2023

The contemporary world experiences a huge growth in cloud implementations, consequently leading to a rise in demand for data engineers and IT professionals who are well-equipped with a wide range of application and process expertise. Data Engineer certification will aid in scaling up you knowledge and learning of data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Generalist 52

ProjectPro

FEBRUARY 16, 2023

The demand for skilled data engineers who can build, maintain, and optimize large data infrastructures does not seem to slow down any sooner. At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data. use SQL, compared to 61.7%

Data Engineering 52

Data Engineering Data Engineer SQL Engineering 52

Data Engineering Weekly

MAY 7, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. Sign up free to test out the tool today. What is Headless Lakehouse?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Query Google’s cloud data warehouse.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 90

Data Lake Data Warehouse BI SQL 90

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Modern platforms like Redshift , Snowflake , and BigQuery have elevated the data warehouse model.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Data Engineering Weekly

JANUARY 15, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Take control of your customer data today.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Towards Data Science

FEBRUARY 15, 2023

In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture. The data became useless. The Lakehouse architecture was one of them.

Data Lake 80

Data Lake Data Warehouse Hadoop Data Architecture 80

Data Engineering Podcast

AUGUST 20, 2021

Summary Data lakes have been gaining popularity alongside an increase in their sophistication and usability. Despite improvements in performance and data architecture they still require significant knowledge and experience to deploy and manage. No more scripts, just SQL. How are you using Cuelake in your work at Cuebook?

Building 100

Building Data Lake Data Warehouse SQL 100

Databand.ai

DECEMBER 13, 2022

The Top 25 Data Engineering Influencers and Content Creators on LinkedIn Ryan Yackel 2022-12-13 10:23:19 Interested in data engineering? LinkedIn is full of influencers sharing new ideas and sparking conversations on all kinds of topics, and data engineering is no exception. You’ve come to the right place.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Data Engineering Podcast

SEPTEMBER 23, 2018

This includes modeling the lifecycle of your information as a pipeline from the raw, messy, loosely structured records in your data lake, through a series of transformations and ultimately to your data warehouse. What are your views on that evolution and what is your litmus test for determining which data to keep?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

ProjectPro

JANUARY 19, 2022

Planning to land a successful job as an Azure Data Engineer? Read this blog till the end to learn more about the roles and responsibilities, necessary skillsets, average salaries, and various important certifications that will help you build a successful career as an Azure Data Engineer.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Cloudera

MARCH 17, 2023

As the use of ChatGPT becomes more prevalent, I frequently encounter customers and data users citing ChatGPT’s responses in their discussions. I love the enthusiasm surrounding ChatGPT and the eagerness to learn about modern data architectures such as data lakehouses, data meshes, and data fabrics.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Monte Carlo

JUNE 14, 2023

For example, data may be extracted from multiple sources, reorganized, and joined at different times before ultimately being served to its final destination. Why is data pipeline architecture important? Data then, and even today for some organizations, was primarily hosted in on-premises databases with non-scalable storage.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Towards Data Science

DECEMBER 15, 2023

Taking a hard look at data privacy puts our habits and choices in a different context, however. Data scientists’ instincts and desires often work in tension with the needs of data privacy and security. Anyone who’s fought to get access to a database or data warehouse in order to build a model can relate.

Machine Learning 93

Machine Learning Data Science Data Security Data Storage 93

Advancing Analytics: Data Engineering

JULY 2, 2019

The data engineer is an emerging role that’s rapidly growing in popularity… but what is it? The Data Science Engineer Let’s start with the original idea of the Data Engineer, the support of Data Science functions by providing clean data in a reliable, consistent manner, likely using big data technologies.

Business Intelligence 52

Business Intelligence Data Engineering Data Engineer Engineering 52

Monte Carlo

JANUARY 24, 2024

More data, more access to data, and more regulations mean data governance has become a higher-stakes game. As you prepare to roll out a successful data governance strategy at your organization, keep an eye on the following four trends that will shape the future of data governance.

Data Governance 52

Data Governance Government Data Lake Data Architecture 52

Monte Carlo

AUGUST 31, 2023

Data pipelines can handle both batch and streaming data, and at a high-level, the methods for measuring data quality for either type of asset are much the same. We’ll take a closer look at variables that can impact your data next. As data becomes more and more foundational to business, data teams will only grow.

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Knowledge Hut

SEPTEMBER 26, 2023

Microsoft Azure's Azure Synapse, formerly known as Azure SQL Data Warehouse, is a complete analytics offering. Designed to tackle the challenges of modern data management and analytics, Azure Synapse brings together the worlds of big data and data warehousing into a unified and seamlessly integrated platform.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Data Engineering Podcast

NOVEMBER 18, 2019

Summary With the constant evolution of technology for data management it can seem impossible to make an informed decision about whether to build a data warehouse, or a data lake, or just leave your data wherever it currently rests. How does it influence the relevancy of data warehouses or data lakes?

Data Lake 100

Data Lake Scala Data Warehouse Hadoop 100

ProjectPro

JULY 27, 2021

This blog is your one-stop solution for the top 100+ Data Engineer Interview Questions and Answers. In this blog, we have collated the frequently asked data engineer interview questions based on tools and technologies that are highly useful for a data engineer in the Big Data industry.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

ProjectPro

SEPTEMBER 26, 2021

Big Data Engineer is one of the most popular job profiles in the data industry. This blog on Big Data Engineer salary gives you a clear picture of the salary range according to skills, countries, industries, job titles, etc. Big Data gets over 1.2 What does a big data engineer do?

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

Ascend.io

NOVEMBER 21, 2023

The emergence of cloud data warehouses, offering scalable and cost-effective data storage and processing capabilities, initiated a pivotal shift in data management methodologies. Extract The initial stage of the ELT process is the extraction of data from various source systems.

Raw Data 52

Raw Data Data Warehouse Data Cleanse Data Integration 52