Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Towards Data Science

NOVEMBER 4, 2023

Platform Specific Tools and Advanced Techniques Photo by Christopher Burns on Unsplash The modern data ecosystem keeps evolving and new data tools emerge now and then. In this article, I want to talk about crucial things that affect data engineers. Data warehouse exmaple. What is it?

Data Engineering 86

Data Engineering Data Engineer Engineering BI 86

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? __init__ covers the Python language, its community, and the innovative ways it is being used.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

KDnuggets

FEBRUARY 13, 2023

SQL and Python Interview Questions for Data Analysts • Learn Machine Learning From These GitHub Repositories • Learn Data Engineering From These GitHub Repositories • The ChatGPT Cheat Sheet • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2

SQL 102

SQL Python Machine Learning Data Engineering 102

Confessions of a Data Guy

APRIL 16, 2023

I was wondering the other day … since Polars now has a SQL context and is getting more popular by the day, do I need DuckDB anymore? You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. These two tools are hot.

Data Engineering 147

Data Engineering Data Engineer Engineering SQL 147

Snowflake

MARCH 21, 2023

In Snowflake’s 6 Data Science Trends in 2023 , one of the more prominent trends we identified was the emerging use of unified tools and infrastructure for SQL and Python. Challenges with siloed data architecture SQL and Python are among the most popular languages in the modern data stack for transformations, analysis, and ML.

SQL 52

SQL Python Programming Language Scala 52

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is Data Science? What are the roles and responsibilities of a Data Engineer? And many more. And many more.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

ProjectPro

FEBRUARY 16, 2023

The demand for skilled data engineers who can build, maintain, and optimize large data infrastructures does not seem to slow down any sooner. At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data.

Data Engineering 52

Data Engineering Data Engineer SQL Engineering 52

Jesse Anderson

DECEMBER 12, 2022

They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Snowflake

OCTOBER 23, 2023

One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers. Snowflake customers are already harnessing the power of Python through Snowpark , a set of runtimes and libraries that securely deploy and process non-SQL code directly in Snowflake.

Python 141

Python Cloud Data Science Data Architecture 141

Analytics Vidhya

FEBRUARY 2, 2023

Introduction SQL injection is an attack in which a malicious user can insert arbitrary SQL code into a web application’s query, allowing them to gain unauthorized access to a database. We can use this to steal sensitive information or make unauthorized changes to the data stored in the database.

Database 204

Database SQL Coding Accessible 204

Start Data Engineering

OCTOBER 11, 2021

Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.

Data Engineering 130

Data Engineering Data Engineer Engineering Cloud Computing 130

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineering 83

Data Engineering Data Engineer Engineering SQL 83

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Visit: dataengineeringpodcast.com/data-council today. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. With Materialize, you can! Hex brings everything together. Hex brings everything together.

Building 189

Building BI SQL Python 189

Data Engineering Weekly

MARCH 31, 2024

Intuit: How Intuit data analysts write SQL 2x faster with the internal GenAI tool The productivity increase with GenAI is undeniable, and several startups are trying to solve the Text2SQL generation problem. My key highlight is that Excellent data documentation and “clean data” improve results.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Knowledge Hut

MAY 3, 2023

With companies increasingly relying on data-driven insights to make informed decisions, there has never been a greater need for skilled specialists who can manage and evaluate vast amounts of data. The roles of data analyst and data engineer have emerged as two of the most in-demand professions in today's job market.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Cleanse 98

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Knowledge Hut

MARCH 28, 2024

Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 All these numbers point to one thing–increased job roles and careers, especially when we talk about data engineering jobs in Azure, which are on the rise every year. Let’s get started.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. Data lakes are notoriously complex. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Building 147

Building Data Lake High Quality Data Machine Learning 147

KDnuggets

FEBRUARY 8, 2023

SQL and Python Interview Questions for Data Analysts • 20 Questions (with Answers) to Detect Fake Data Scientists: ChatGPT Edition, Part 2 • ChatGPT for Beginners • Python String Matching Without Complex RegEx Syntax • Learn Data Engineering From These GitHub Repositories

Python 81

Python SQL Data Engineering Data Engineer 81

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Data Engineering Podcast

APRIL 14, 2024

In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database. Data lakes are notoriously complex. How have the requirements and applications of NoSQL engines changed since they first became popular ~15 years ago?

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

JULY 10, 2022

Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Team Data Science

JANUARY 8, 2021

The purpose of this post is to expose you to the skills needed as a data engineer; now let’s look into them Understand the fundamental skill Recently, functions of computer engineering have become more important in organizations that are handling vast volumes of data, including data in diverse formats.

Data Engineering 130

Data Engineering Data Engineer Engineering Unstructured Data 130

Data Engineering Podcast

SEPTEMBER 24, 2023

In this episode Louis Brandy discusses the applications for vector search capabilities both in and outside of AI, as well as the challenges of maintaining real-time indexes of vector data. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

BI 147

BI SQL Machine Learning Python 147

Towards Data Science

DECEMBER 1, 2023

Architecture Referencing Joe Reis’s “Fundamentals of Data Engineering,” let’s review our project in alignment with the defined stages of the data lifecycle: Data Engineering Lifecycle [1] Data Generation —  Google Calendar, Fivetran If you’re a Google Calendar user, chances are you’ve accumulated a wealth of data there.

Data Engineering 73

Data Engineering Data Engineer Project Engineering 73

Data Engineering Podcast

SEPTEMBER 3, 2023

In this episode Adrian Brudaru explains how it works, the benefits that it provides over other data integration solutions, and how you can start building pipelines today. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Data Integration 162

Data Integration BI SQL Python 162

Data Engineering Podcast

APRIL 7, 2024

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

OCTOBER 15, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Process 182

Process Building SQL BI 182

Data Engineering Podcast

MARCH 24, 2024

In this episode Pete Hunt, CEO of Dagster labs, outlines these new capabilities, how they reduce the burden on data teams, and the increased collaboration that they enable across teams and business units. Data lakes are notoriously complex. __init__ covers the Python language, its community, and the innovative ways it is being used.

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

Data Engineering Podcast

FEBRUARY 6, 2022

In order to make geospatial analytics more maintainable and scalable there has been an increase in the number of database engines that provide extensions to their SQL syntax that supports manipulation of spatial data. Can you describe what spatial SQL is and some of the use cases that it is relevant for?

SQL 100

SQL Building IT Google Cloud 100

Data Engineering Podcast

SEPTEMBER 10, 2023

In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

BI 208

BI SQL Data Machine Learning 208

Knowledge Hut

DECEMBER 21, 2023

Data Science is the world's most rapidly growing sector and data engineers are at the forefront. In this article, we will understand the promising data engineer career outlook and what it takes to succeed in this role. What is Data Engineering? What are the Data Engineer Career Opportunities?

Data Engineering 52

Data Engineering Data Engineer Engineering MongoDB 52