10 Great Videos To Help You Learn Data Engineering
Seattle Data Guy
APRIL 19, 2024
It’s unavoidable that as businesses demand that their data teams implement AI, they will also realize that data engineers are a crucial piece of the data pipeline.
This site uses cookies to improve your experience. By viewing our content, you are accepting the use of cookies. To help us insure we adhere to various privacy regulations, please select your country/region of residence. If you do not select a country we will assume you are from the United States. View our privacy policy and terms of use.
Seattle Data Guy
APRIL 19, 2024
It’s unavoidable that as businesses demand that their data teams implement AI, they will also realize that data engineers are a crucial piece of the data pipeline.
Analytics Vidhya
JUNE 25, 2023
In a data-driven world, behind-the-scenes heroes like data engineers play a crucial role in ensuring smooth data flow. A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Communication
Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications
From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
Analytics Vidhya
SEPTEMBER 20, 2023
Data engineering plays a pivotal role in the vast data ecosystem by collecting, transforming, and delivering data essential for analytics, reporting, and machine learning. Aspiring data engineers often seek real-world projects to gain hands-on experience and showcase their expertise.
The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Communication
Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications
From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know
KDnuggets
APRIL 12, 2024
The only data engineering roadmap you need for an introduction to concepts, tools, and techniques to collect, store, transform, analyze, and model data.
Start Data Engineering
APRIL 22, 2024
Most data projects use Docker to set up the data infra locally (and often in production). Introduction 2. Docker concepts 2.1. Define the OS and its configurations with an image 2.2. Use the image to run containers 2.2.1. Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1.
Seattle Data Guy
JANUARY 17, 2023
With all the recent data events I have put together I inevitably run into new data engineers who are either finishing up college or looking to transition into a data engineer or data scientist position. In fact I have talked to several newly graduated engineers who are struggling to find work.
Analytics Vidhya
APRIL 3, 2023
Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers. Nevertheless, setting up a streaming data pipeline to power such dashboards may […] The post Data Engineering for Streaming Data on GCP appeared first on Analytics Vidhya.
Analytics Vidhya
JUNE 24, 2023
He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Thus, providing valuable insights into the field of data engineering. Introduction We had an amazing opportunity to learn from Mr. Pavan.
Analytics Vidhya
JULY 25, 2023
In the world of data, two crucial roles play a significant part in unlocking the power of information: Data Scientists and Data Engineers. But what sets these wizards of data apart? Welcome to the ultimate showdown of Data Scientist vs Data Engineer! appeared first on Analytics Vidhya.
Analytics Vidhya
FEBRUARY 7, 2023
Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.
Analytics Vidhya
JUNE 20, 2023
Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.
KDnuggets
FEBRUARY 12, 2024
Interested in data engineering but don't know where to start? Get up to speed in data engineering fundamentals with this free course.
Jesse Anderson
JANUARY 29, 2024
The premier of my latest talk covering The State of Data Engineering. This starts with data warehousing and goes into data science. I finish off by showing how data engineering can avoid the same fate as data warehousing and data science.
Jesse Anderson
SEPTEMBER 14, 2023
There has been quite a bit of writing covering GPT and LLMs from data science and business perspectives. I haven’t seen much from the data engineering side. Let me share my perspective, having been in data and AI for a while and using LLMs before they became popular. How can we use LLMs in data engineering?
Data Engineering Weekly
APRIL 7, 2024
I strongly believe the concept of Data Product will play a bigger role in data engineering. link] Hussein Jundi: Data Engineering - Architectures & Strategies for Handling Sensitive Data The rapid adoption of AI brings challenges for data engineering to design systems to handling sensitive data.
Confessions of a Data Guy
FEBRUARY 7, 2023
Some of the things I’m going to talk about, well … all of it, is probably fairly obvious to most Rust folk, but it’s enjoyable to learn what new […] The post Ownership and Borrowing in Rust – Data Engineering Gold Mine. appeared first on Confessions of a Data Guy.
Analytics Vidhya
SEPTEMBER 20, 2023
Data engineering plays a pivotal role in the vast data ecosystem by collecting, transforming, and delivering data essential for analytics, reporting, and machine learning. Aspiring data engineers often seek real-world projects to gain hands-on experience and showcase their expertise.
Data Engineering Weekly
APRIL 21, 2024
link] Canva: Scaling to Count Billions If you know how to count, you’re an excellent data engineer. Counting is the hardest problem in data engineering. RevenueCat writes about solving such challenges with the ingestion table & consolidation table pattern.
Jesse Anderson
DECEMBER 12, 2022
Big data projects were given to data scientists and data warehouse teams, where the projects subsequently failed. As clearly evident as that sounds now, my writing about needing data engineering went heavily against the grain of everything that was written at the time.
Confessions of a Data Guy
APRIL 16, 2023
You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. I haven’t seen this since Databricks and Snowflake first came out and started throwing mud at each other.
Seattle Data Guy
FEBRUARY 11, 2023
Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?
Start Data Engineering
JUNE 13, 2023
Introduction So you are a new data engineer (or looking for a DE job) and want to better yourself as a data engineer. Money & Time 2.2. Technical skills 3. Build impactful projects 4. Conclusion 5. Further reading 1.
Seattle Data Guy
MAY 20, 2023
Starting new data engineering projects can be challenging. Data engineers can get stuck on finding the right data for their data engineering project or picking the right tools.
Simon Späti
OCTOBER 19, 2022
Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.
Simon Späti
OCTOBER 19, 2022
Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.
Seattle Data Guy
JANUARY 17, 2023
Are data engineers all learning Rust? Our team is putting together an all day event focused on helping answer some… Read more The post What Is The State Of Data Engineering And Infrastructure In 2023 appeared first on Seattle Data Guy. Are Snowflake and Databricks still fighting over total cost of ownership?
Netflix Tech
DECEMBER 14, 2023
Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr. In this video, Sr.
Confessions of a Data Guy
SEPTEMBER 9, 2023
Nothing screams “why are flying by night,” than coming into a Data Team only to find no tests, no docs, no deployments, no Docker, no nothing. […] The post The Role of DevOps and CI/CD in Data Engineering appeared first on Confessions of a Data Guy.
Data Engineering Weekly
APRIL 14, 2024
link] Arpit Choudhury: Growth needs to speak the language of Engineering and Data Software engineers and data engineers don’t speak the same language It reminds me of this famous quote The Author highlights that the growth team should be aware of the skill differences.
Confessions of a Data Guy
OCTOBER 6, 2023
I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling […] The post The Ultimate Data Engineering Chadstack. appeared first on Confessions of a Data Guy. At the moment Rust and Airflow are at least somewhere at the top of that list. Running Rust inside Apache Airflow.
Snowflake
APRIL 17, 2024
This traditional SQL-centric approach often challenged data engineers working in a Python environment, requiring context-switching and limiting the full potential of Python’s rich libraries and frameworks. To get started, explore the comprehensive API documentation , which will guide you through every step.
Confessions of a Data Guy
JUNE 8, 2023
It gives a fairly unique view into the wide range of Data Engineering companies, jobs, projects people are working on, tech stacks, and problems that are being faced. One thing I’ve come […] The post 4 Ways To Setup Your Data Engineering Game. appeared first on Confessions of a Data Guy.
Data Engineering Weekly
MARCH 17, 2024
It also introduces emerging standards like the Open Data Contract Standard and Data Product Descriptor Specification. As you know, I’m fascinated by data products and the potential to change the data engineering practice. Can we measure the cost of data incidents?
Confessions of a Data Guy
MARCH 20, 2023
For a lot of my Data Engineering career I didn’t really think about or use AWS lambdas, I just saw them as little annoying flies […] The post AWS Lambdas. Useful for Data Engineering? appeared first on Confessions of a Data Guy.
Data Engineering Podcast
JULY 2, 2023
In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?
KDnuggets
NOVEMBER 30, 2023
Data engineers must prepare and manage the infrastructure and tools necessary for the whole data workflow in a data-driven company.
Data Engineering Weekly
FEBRUARY 18, 2024
Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. link] Sponsored: Data modeling and exploration in Playground 2.0 Elevate your data skills!
Confessions of a Data Guy
NOVEMBER 5, 2022
There are probably few things in life that will strike more fear and tumult in the heart of the Data Engineer than historical loads. How could it possibly be, just take a bunch of data stored somewhere and shove it into a table. […] The post Introduction to Historical Loads – for Data Engineers.
KDnuggets
JANUARY 26, 2024
Data Engineering ZoomCamp offers free access to reading materials, video tutorials, assignments, homeworks, projects, and workshops.
Waitingforcode
FEBRUARY 13, 2024
It's time for another part of "What's new on the cloud for data engineers" Let's see what happened in the last 5 months.
Start Data Engineering
FEBRUARY 22, 2024
Data Pipeline Logging Best Practices 3.1. Metadata: Information about pipeline runs, & data flowing through your pipeline 3.2. Introduction 2. Setup & Logging architecture 3. Obtain visibility into the code’s execution sequence using text logs 3.3. Understand resource usage by tracking Metrics 3.4.
Data Engineering Weekly
JANUARY 28, 2024
I had a chance to meet some of the amazing humans of data engineering. The article narrates various stages, including business intelligence, data platform, event-bus backbone, and machine learning, leading to decentralizing data engineering expertise and code.
Towards Data Science
MARCH 26, 2024
In this article we dive into some practical examples for Data Engineers Continue reading on Towards Data Science » Generative AI is all the rage.
Data Engineering Weekly
MARCH 3, 2024
Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup. There will be food, networking, and real-world talks around data engineering.
Expert insights. Personalized for you.
We have resent the email to
Are you sure you want to cancel your subscriptions?
Let's personalize your content