Data Engineering Weekly

NOVEMBER 27, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Weekly

FEBRUARY 26, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Perhaps merge Data LakeHouse & Data Warehouse in the next edition?

Data Engineering 57

Data Engineering Data Engineer Engineering Raw Data 57

Data Science Blog: Data Engineering

DECEMBER 23, 2022

Warum Data Engineering der Data Science in Bedeutung und Berufschancen längst die Show stiehlt, dabei selbst ebenso einem stetigen Wandel unterliegt. Was ein Data Engineer wirklich können muss Der Data Scientist als sexiest Job des 21. Data Engineer, der gefragteste Job des 21sten Jahrhunderts?

Data Engineering 52

Data Engineering Data Engineer Engineering Business Intelligence 52

ProjectPro

FEBRUARY 16, 2023

The demand for skilled data engineers who can build, maintain, and optimize large data infrastructures does not seem to slow down any sooner. At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data. use SQL, compared to 61.7%

Data Engineering 52

Data Engineering Data Engineer SQL Engineering 52

Edureka

FEBRUARY 9, 2023

The role of Azure Data Engineer is in high demand in the field of data management and analytics. As an Azure Data Engineer, you will be in charge of designing, building, deploying, and maintaining data-driven solutions that meet your organization’s business needs. Who is an Azure Data Engineer?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Mining 52

Monte Carlo

DECEMBER 27, 2022

It definitely holds true for our Data Downtime blog with these five articles driving a majority of our traffic in 2022. The 5 Pillars You Need To Know Is the Modern Data Warehouse Broken? What is Data Observability? Is the Modern Data Warehouse Broken? The titular pun might have helped too… Read more.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

U-Next

AUGUST 17, 2022

Introduction to 2022 Data Engineer Roles and Responsibilities. Companies and enterprises, large and small, are built on data. Data Engineer roles and responsibilities include aiding in the collection of issues and the delivery of remedies addressing customer demand and product accessibility.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Knowledge Hut

FEBRUARY 16, 2023

Demand for data engineers is at a peak today globally due to the massive amount of data that companies accumulate and work with this data to draw actionable insights and make better business decisions. To do that, the generated data must be interpreted to be understood by the end user. What Does a Data Engineer Do?

Data Engineering 52

Data Engineering Data Engineer Engineering Healthcare 52

ProjectPro

FEBRUARY 21, 2023

With so many data engineering certifications available , choosing the right one can be a daunting task. There are over 133K data engineer job openings in the US, but how will you stand out in such a crowded job market? The answer is- by earning professional data engineering certifications! AWS or Azure?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Silectis

FEBRUARY 17, 2021

In this blog post, we’ll walk you through how data science and data engineering are complementary disciplines. We’ll also delineate a third category: data analysis. We’ll explore how both data engineering and data science should be marshaled to make better decisions.

Data Science 52

Data Science Data Engineering Data Engineer Engineering 52

Cloudera

FEBRUARY 15, 2023

This month’s #ClouderaLife Spotlight features software engineer Amogh Desai. Snatching victory from the jaws of defeat Amogh and his fellow hackathon team members felt the rush of victory after winning Cloudera’s 2022 global hackathon in the product development category. One way he does this is through blog writing.

Software Engineer 81

Software Engineer Software Engineering Engineering Recruitment 81

Monte Carlo

MAY 24, 2022

Monte Carlo intends to use the capital infusion to continue improving experiences for its hundreds of customers, scale the data observability category to new verticals, and grow its U.S. and EMEA go-to-market and engineering teams. With their Series D, Monte Carlo has achieved a $1.6B

High Quality Data 52

High Quality Data Data Education Machine Learning 52

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. This is something I struggle with, I really like writing, I really like this newsletter, I really like the blog, but it takes me one day per week to be done. Hubert tries to explain stuff in the real-time category.

Kafka 130

Kafka Data Lake Data Storage Data 130

Christophe Blefari

AUGUST 4, 2023

As a disclaimer, this may not quite make sense in a corporate context, but since this is my blog, I'll do what I want. Race data will be partitioned per day, but as the Tour is already done, it will be a bit different than in real life environment. This is linear, I'll move data from A to B to C in to order to use it with D.

Google Cloud 225

Google Cloud MongoDB NoSQL Data 225

ProjectPro

JANUARY 19, 2022

Planning to land a successful job as an Azure Data Engineer? Read this blog till the end to learn more about the roles and responsibilities, necessary skillsets, average salaries, and various important certifications that will help you build a successful career as an Azure Data Engineer.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Data Engineering Podcast

MAY 7, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What are some of the categories of attributes that need to be managed in a prototypical customer profile?

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Data Engineering Weekly

MAY 16, 2023

In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive? I won’t bore you with the importance of data quality in the blog. Let’s talk about the data processing types. How to Fix It?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

ProjectPro

JUNE 9, 2021

This blog on Data Science vs. Data Engineering presents a detailed comparison between the two domains. vs. What does a Data Engineer do? Are you a Data Scientist or a Data Engineer? Is data engineering more important than data science?

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Cloudera

JANUARY 12, 2018

I mentioned in an earlier blog titled, “Staffing your big data team, ” that data engineers are critical to a successful data journey. That said, most companies that are early in their journey lack a dedicated engineering group. Image 1: Data Engineering Skillsets.

Big Data 54

Big Data Data Engineering Data Engineer Engineering 54

Snowflake

MARCH 30, 2023

Collecting, cleaning, and organizing data into a coherent form for business users to consume are all standard data modeling and data engineering tasks for loading a data warehouse. Feature engineering: Data is transformed to support ML model training. ML workflow, ubr.to/3EJHjvm

Engineering 81

Engineering Raw Data Data Science Scala 81

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Just connect it to your database/data warehouse/data lakehouse/whatever you’re using and let them do the rest.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Scott Logic

AUGUST 11, 2023

This relies on Google domination of the search engine market with Google Search enjoying a 92% market share. This is because Google search prioritises web pages that uses Google Ads, and the self proclaimed second largest search engine on the internet is Youtube which exclusively uses Google Ads. What are search engines?

Engineering 52

Engineering AWS Datasets Metadata 52

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. How does it work?

Metadata 130

Metadata MongoDB Scala MySQL 130

Data Engineering Podcast

SEPTEMBER 4, 2022

Summary Data observability is a product category that has seen massive growth and adoption in recent years. Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. The only thing worse than having bad data is not knowing that you have it.

IT 100

IT Metadata MongoDB MySQL 100

Christophe Blefari

MARCH 17, 2023

It also means that if you have any questions that you want me to ask you can send them to me beforehand 🫠 Gen AI 🤖 I will create a specific category for generative AI. Data with Rust — This is a handbook that will showcase how to work in data engineering with Rust.

Data 130

Data SQL Deep Learning Kafka 130

Christophe Blefari

DECEMBER 3, 2022

Joe Reis launched his Substack — Joe is the co-author of the great The Fundamentals of Data Engineering and his blog already have 2 articles I deeply recommend: No extra credit for complexity & Groundhog Days. This post tries to show how data team should work with finance team to avoid this spat.

Kafka 130

Kafka BI Finance Data 130

ProjectPro

JULY 27, 2021

This blog is your one-stop solution for the top 100+ Data Engineer Interview Questions and Answers. In this blog, we have collated the frequently asked data engineer interview questions based on tools and technologies that are highly useful for a data engineer in the Big Data industry.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Christophe Blefari

MARCH 31, 2023

We are slowly approaching the 2-years anniversary of the blog and the newsletter. To be honest time flies and I’d have preferred to do more for the blog in the start of the year but my freelancing activities and my laziness took me so much. This newsletter is about money ( credits ) Dear readers, already 3 months done in 2023.

Bytes 130

Bytes Data Google Cloud Education 130

Christophe Blefari

JULY 3, 2023

💡 If you want another view on both the conferences Ananth from Data Engineering Weekly wrote about the conferences extravaganza and a few trends he wanted to chat about. This is their way to democratise data to every employee. In the end what we want is a place to store data and an engine to query data.

SQL 130

SQL Data Kafka AWS 130

Data Engineering Weekly

MARCH 30, 2023

In this episode of Data Engineering Weekly Radio, we delve into modern data stacks under pressure and the potential consolidation of the data industry. We also touched on the role of data lineage and pondered whether the control of data lineage would translate to control over the entire data stack.

Machine Learning 52

Machine Learning Government AWS Data Engineering 52

Cloudera

APRIL 24, 2023

To this six-part series, where we’ll look at how to get control of the health of your Cloudera Data platform (CDP) environment. With each blog we’ll outline the symptoms and root causes of common environmental health challenges and prescribe solutions. That’s observability. If you’ve got the symptoms, the doctors are in.

Professional Services 81

Professional Services Datasets Data Pipeline Data Architecture 81

Knowledge Hut

FEBRUARY 15, 2023

As a result, the demand for software engineers is always high in this island nation. As a software engineer, this is the perfect time to jump on the bandwagon and pave your way toward a lucrative career. Who is a Software Engineer? Software engineers develop programs, applications, and operations systems for computers.

Software Engineer 52

Software Engineer Software Engineering Engineering Recruitment 52

Databand.ai

AUGUST 30, 2023

Data testing tools: Key capabilities you should know Helen Soloveichik August 30, 2023 Data testing tools are software applications designed to assist data engineers and other professionals in validating, analyzing and maintaining data quality. There are several types of data testing tools.

Data Cleanse 52

Data Cleanse Data Pipeline Datasets Data Validation 52

Picnic Engineering

JANUARY 9, 2024

Especially if the products are in a category the customer has never bought before. The original papers for all these models can be found linked on the RecBole website and we won’t provide any detail in this blog as it is not too relevant. Explore categories are easier! Well they all boil down to sparsity.

Machine Learning 91

Machine Learning Datasets Algorithm Systems 91

Knowledge Hut

APRIL 25, 2024

It is especially true in the world of big data. If you want to stay ahead of the curve, you need to be aware of the top big data technologies that will be popular in 2024. In this blog post, we will discuss such technologies. The two most popular data warehouse systems are Teradata and Oracle Exadata.

Big Data 64

Big Data Technology NoSQL Hadoop 64

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore).

Big Data 78

Big Data Data Management Management Metadata 78

Databand.ai

AUGUST 30, 2023

Data Testing Tools: Key Capabilities and 6 Tools You Should Know Helen Soloveichik August 30, 2023 What Are Data Testing Tools? Data testing tools are software applications designed to assist data engineers and other professionals in validating, analyzing, and maintaining data quality.

Data Cleanse 52

Data Cleanse Data Validation Data Pipeline Datasets 52