Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! What are Data Engineering Tools?

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Knowledge Hut

DECEMBER 28, 2023

Its comprehensive suite of services can handle data at scale. It’s no surprise that the demand for certified Azure data engineers has skyrocketed. Today, Azure Data Engineer certification is an invaluable asset for those looking to excel in the field of data engineering. Who is an Azure Data Engineer?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Knowledge Hut

FEBRUARY 27, 2023

A data engineer is a key member of an enterprise data analytics team and is responsible for handling, leading, optimizing, evaluating, and monitoring the acquisition, storage, and distribution of data across the enterprise. Data Engineers indulge in the whole data process, from data management to analysis.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Data Large volumes of structured or unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Knowledge Hut

JUNE 20, 2023

Let's find out the differences between a data scientist and a machine learning engineer below to make an informative decision. Data Engineer vs Machine Learning Engineer While there are similarities between a data engineer and a machine learning engineer, both play a key role in the technological world.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

APRIL 26, 2021

One of the most common and widely used methods of access is through a business intelligence dashboard. In this episode Maxime Beauchemin discusses how data engineers can use Superset to provide self service access to data and deliver analytics. RudderStack’s smart customer data pipeline is warehouse-first.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

dbt Developer Hub

APRIL 30, 2023

My next gig was in consulting where I bootstrapped my way into data engineering and had to learn the whole gamut below. Armed with them, I finally saw the magic of the modern data stack and what problems it could solve. Armed with them, I finally saw the magic of the modern data stack and what problems it could solve.

Data Engineering 64

Data Engineering Data Engineer Engineering Finance 64

Data Engineering Podcast

OCTOBER 22, 2021

Summary The market for business intelligence has been going through an evolutionary shift in recent years. One of the driving forces for that change has been the rise of analytics engineering powered by dbt. Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads?

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Data Engineering Podcast

NOVEMBER 4, 2021

Summary The precursor to widespread adoption of cloud data warehouses was the creation of customer data platforms. Acting as a centralized repository of information about how your customers interact with your organization they drove a wave of analytics about how to improve products based on actual usage data.

Data Warehouse 100

Data Warehouse Business Intelligence ETL System Data Lake 100

Databand.ai

DECEMBER 13, 2022

The Top 25 Data Engineering Influencers and Content Creators on LinkedIn Ryan Yackel 2022-12-13 10:23:19 Interested in data engineering? LinkedIn is full of influencers sharing new ideas and sparking conversations on all kinds of topics, and data engineering is no exception. You’ve come to the right place.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Monte Carlo

SEPTEMBER 1, 2022

Data engineering is a relatively young profession, even for the tech space. To put it in perspective, front-end engineering has twice the number of years in industry maturity. That’s because not only is data engineering talent in high demand, but the very nature of handling big data necessitates prioritization and focus.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Ascend.io

AUGUST 31, 2023

Still, these methods have been overshadowed by EtLT — the predominant approach reshaping today’s data landscape. In this article, we assess: The role of the data warehouse on one hand, and the data lake on the other; The features of ETL and ELT in these two architectures; The evolution to EtLT; The emerging role of data pipelines.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Podcast

DECEMBER 19, 2021

In this episode Pardhu Gunnam and Mars Lan explain how they have designed the architecture and user experience to allow everyone to collaborate on the data lifecycle and provide opportunities for automation and extensible workflows. Start trusting your data with Monte Carlo today! If this resonates with you, you’re not alone.

Systems 100

Systems Building Metadata Data Warehouse 100

Knowledge Hut

DECEMBER 26, 2023

Data Science is a combination of several disciplines including Mathematics and Statistics, Data Analysis, Machine Learning, and Computer Science. Data Science is a huge umbrella with a plethora of roles available in the field such as a Data Scientist, Data Engineer, BI Developer, Data and Analytics Manager, etc.

Data Science 97

Data Science BI Business Intelligence Data Mining 97

Data Engineering Podcast

DECEMBER 11, 2021

In this episode he explains how it can be used to build rich and useful profiles of your users, the system architecture that powers it, and some of the ways that it is being integrated into an organization’s broader data ecosystem. Start trusting your data with Monte Carlo today! If this resonates with you, you’re not alone.

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

Data Engineering Podcast

JANUARY 1, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

Data Engineering Podcast

DECEMBER 4, 2021

The whole process of hiring is an important organizational skill to cultivate and this is an interesting exploration of the specific challenges involved in finding data professionals. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform!

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Tired of deploying bad data?

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. Tired of deploying bad data?

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

Data Engineering Podcast

JULY 10, 2022

In this episode Inbar Yogev and Lior Winner share the journey that they and their teams at Riskified have been on for their data platform. They also discuss how they have established a guild system for training and supporting data professionals in the organization. Tired of deploying bad data?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Knowledge Hut

DECEMBER 29, 2023

There are many data science fields in which experts may contribute to the success of a business, and you can hone the abilities you need by specializing in data science subfields. Data Engineering and Warehousing The data is the lifeblood of every successful Data Science endeavor.

Data Science 98

Data Science Data Mining Deep Learning Programming Language 98

Advancing Analytics: Data Engineering

JUNE 21, 2019

Ah the ETL (Extract-Transform-Load) Window, the schedule by which the Business Intelligence developer sets their clock, the nail-biting nightly period during which the on-call support hopes their phone won’t ring. It’s a cornerstone of the data warehousing approach… and we shouldn’t have one. There, I said it. Till next time.

Data Warehouse 52

Data Warehouse Business Intelligence Data Data Validation 52

Data Engineering Podcast

NOVEMBER 20, 2021

In this episode he shares the journey that he and his team at Elementl have taken to understand the state of the ecosystem and how they can provide a foundational layer for a holistic data platform. Start trusting your data with Monte Carlo today! The data you’re looking for is already in your data warehouse and BI tools.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

MAY 15, 2022

Srivatsan Sridharan has had the opportunity to design, build, and run data lake platforms for both Yelp and Robinhood, with many valuable lessons learned from each experience. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Data Lake 100

Data Lake Building BI Architecture 100

Data Engineering Podcast

NOVEMBER 10, 2021

They also explain how the OpenMetadat project is aiming to be a common standard for defining and storing metadata for every use case in data platforms and the ways that they are architecting the reference implementation to simplify its adoption. Start trusting your data with Monte Carlo today! No more scripts, just SQL.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Data Engineering Podcast

MAY 8, 2022

In this episode Jon Herke shares how TigerGraph customers are taking advantage of those capabilities to achieve meaningful discoveries in their fields, the utilities that it provides for modeling and managing your connected data, and some of his own experiences working with the platform before joining the company. Missing data?

Database 100

Database Data Lake BI Business Intelligence 100

Data Engineering Podcast

JUNE 12, 2022

Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth. In this episode she shares her thoughts and insights on how to be intentional about establishing your own data team. Unstruk is the DataOps platform for your unstructured data.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

JULY 31, 2022

Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. The only thing worse than having bad data is not knowing that you have it.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

DECEMBER 21, 2020

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management What are the pieces of advice that you wish you had received early in your career of data engineering? If you hand a book to a new data engineer, what wisdom would you add to it?

Data Governance 130

Data Governance Government Data Lake Banking 130

Monte Carlo

AUGUST 31, 2023

Data is a priority for your CEO, as it often is for digital-first companies, and she is fluent in the latest and greatest business intelligence tools. What about a frantic email from your CTO about “duplicate data” in a business intelligence dashboard?

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

ProjectPro

JUNE 9, 2021

This blog on Data Science vs. Data Engineering presents a detailed comparison between the two domains. vs. What does a Data Engineer do? Are you a Data Scientist or a Data Engineer? Is data engineering more important than data science?

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Data Engineering Podcast

JANUARY 11, 2021

This is a fascinating conversation on the technical challenges involved, the opportunities that such as system provides, and the complexities inherent to building a successful business on open source. We also have tools for collaborating on software and analysis, but collaborating on data is still an underserved capability.

PostgreSQL 100

PostgreSQL Python Data Lake Computer Science 100