Data Engineering Weekly

NOVEMBER 27, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. I have a long list of thoughts on this conversation, which might need a blog post on its own.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Weekly

MAY 28, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. If you want to write a career guidance series for Data Engineering Weekly , Please DM me on LinkedIn.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions. Our audits would detect this and alert the on-call data engineer (DE).

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Scott Logic

AUGUST 11, 2023

This relies on Google domination of the search engine market with Google Search enjoying a 92% market share. This is because Google search prioritises web pages that uses Google Ads, and the self proclaimed second largest search engine on the internet is Youtube which exclusively uses Google Ads. What are search engines?

Engineering 52

Engineering AWS Datasets Metadata 52

Knowledge Hut

APRIL 26, 2024

The categories are : SUPERVISED UNSUPERVISED SEMI – SUPERVISED In supervised ML algorithms, the user knows both the Input and Output data before applying any algorithm on the data. T he application of “Re-engineering” has been evolving at present with Machine Learning.

Machine Learning 96

Machine Learning Algorithm Deep Learning Programming Language 96

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. This is something I struggle with, I really like writing, I really like this newsletter, I really like the blog, but it takes me one day per week to be done. Monte Carlo also wrote this week about chaos engineering , with a manifesto.

Kafka 130

Kafka Data Lake Data Storage Data 130

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. According to reports by DICE Insights, the job of a Data Engineer is considered the top job in the technology industry in the third quarter of 2020.

Data Engineering 83

Data Engineering Data Engineer Engineering Pipeline-centric 83

Pinterest Engineering

MAY 9, 2023

Erika Sun ML Engineer | Advertiser Growth Modeling Team; Ogheneovo Dibie Engineering Manager | Advertiser Growth Modeling Team Photo by Jason Blackeye on Unsplash Summary In this blog post, we describe a Machine Learning (ML) powered proactive churn prevention solution that was prototyped with our small & medium business (SMB) advertisers.

Consulting 80

Consulting Data Science Architecture Engineering 80

Data Engineering Weekly

FEBRUARY 26, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. the addition of the “Fully Managed” category is an exciting space to watch out for.

Data Engineering 57

Data Engineering Data Engineer Engineering Raw Data 57

Monte Carlo

DECEMBER 27, 2022

It definitely holds true for our Data Downtime blog with these five articles driving a majority of our traffic in 2022. Data contracts: 7 Lessons from GoCardless’ Implementation How to Build Your Own Data Platform The common thread: The data engineering world moves pretty fast, blink and you just might miss it.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Snowflake

MARCH 30, 2023

Collecting, cleaning, and organizing data into a coherent form for business users to consume are all standard data modeling and data engineering tasks for loading a data warehouse. Feature engineering: Data is transformed to support ML model training. “The features you use influence more than everything else the result.

Engineering 81

Engineering Raw Data Data Science Scala 81

Christophe Blefari

AUGUST 4, 2023

As a disclaimer, this may not quite make sense in a corporate context, but since this is my blog, I'll do what I want. Because I'm convinced that to learn data engineering you need to experiment real life pipelines running every day to experiment the morning firefighting. Actually this is just modern data engineering.

Google Cloud 225

Google Cloud MongoDB NoSQL Data 225

Knowledge Hut

FEBRUARY 16, 2023

Demand for data engineers is at a peak today globally due to the massive amount of data that companies accumulate and work with this data to draw actionable insights and make better business decisions. That's where the data engineer comes into the picture, making it a demanding profession today. What Does a Data Engineer Do?

Data Engineering 52

Data Engineering Data Engineer Engineering Healthcare 52

Data Science Blog: Data Engineering

DECEMBER 23, 2022

Warum Data Engineering der Data Science in Bedeutung und Berufschancen längst die Show stiehlt, dabei selbst ebenso einem stetigen Wandel unterliegt. Was ein Data Engineer wirklich können muss Der Data Scientist als sexiest Job des 21. Data Engineer, der gefragteste Job des 21sten Jahrhunderts? Jahrhunderts?

Data Engineering 52

Data Engineering Data Engineer Engineering Business Intelligence 52

Maxime Beauchemin

APRIL 26, 2023

Mastering AI-Powered Product Development: Introducing Promptimize for Test-Driven Prompt Engineering originally posted here-> [link] AI, AGI, LLM, and GPT are the buzzwords of the moment. Although we’re still figuring out new patterns, we know that prompt engineering is a crucial piece of the puzzle. What a stellar assistant!

SQL 148

SQL Database Engineering Software Engineer 148

Palantir

MARCH 16, 2022

Engineering for Impact: Problem solving with purpose at Palantir Editor’s note: In this blog post, we sit down with UK Health Lead Joanna Peller, recipient of Data IQ’s 2022 100 award, to discuss what she’s learned leading Palantir’s UK Health work during the pandemic. Describe your path to Palantir. Find out more.

Engineering 52

Engineering Healthcare Manufacturing Consulting 52

Snowflake

SEPTEMBER 5, 2023

Snowpark ML unifies data pre-processing, feature engineering, model training, and model management and integrated deployment into a single, easy-to-use Python toolkit. save_as_table(NEWS_DATA_TABLE_NAME) Here’s a breakdown of the task list for our use case: Assign a category to each news item in the table. json", lines=True).convert_dtypes()

Medical 117

Medical Python Government Datasets 117

Edureka

FEBRUARY 9, 2023

The role of Azure Data Engineer is in high demand in the field of data management and analytics. As an Azure Data Engineer, you will be in charge of designing, building, deploying, and maintaining data-driven solutions that meet your organization’s business needs. Contents: Who is an Azure Data Engineer?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Mining 52

ProjectPro

FEBRUARY 16, 2023

The demand for skilled data engineers who can build, maintain, and optimize large data infrastructures does not seem to slow down any sooner. At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data. of data engineer job postings on Indeed?

Data Engineering 52

Data Engineering Data Engineer SQL Engineering 52

Data Engineering Podcast

MAY 7, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What are some of the categories of attributes that need to be managed in a prototypical customer profile? When is Segment Unify the wrong choice?

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

DoorDash Engineering

DECEMBER 12, 2023

Unlike our restaurant business, where a typical merchant sells only a few dozen or at most hundreds of dishes or beverages, our new vertical business stores often carry hundreds of thousands of SKUs in thousands of categories. Collection generation: Our collections fall under three main categories, depending on how they are generated.

Retail 93

Retail Architecture Building Process 93

Knowledge Hut

FEBRUARY 15, 2023

As a result, the demand for software engineers is always high in this island nation. As a software engineer, this is the perfect time to jump on the bandwagon and pave your way toward a lucrative career. Who is a Software Engineer? Software engineers develop programs, applications, and operations systems for computers.

Software Engineer 52

Software Engineer Software Engineering Engineering Recruitment 52

Christophe Blefari

MARCH 17, 2023

It also means that if you have any questions that you want me to ask you can send them to me beforehand 🫠 Gen AI 🤖 I will create a specific category for generative AI. Data with Rust — This is a handbook that will showcase how to work in data engineering with Rust.

Data 130

Data SQL Deep Learning Kafka 130

DoorDash Engineering

APRIL 26, 2023

In this blog post, we’ll examine how DoorDash hopes to leverage Generative AI and revolutionize the delivery experience. By analyzing vast amounts of data, the models can suggest a list of items that are often ordered together for each meal category, such as breakfast, lunch, and dinner. ChatGPT, the version of GPT-3.5

Food 99

Food Unstructured Data Deep Learning SQL 99

Christophe Blefari

JANUARY 27, 2023

Fun story because one year ago I had an intern working with me on the blog to whom I had given the task to write code that was able to learn from my writings to generate a Data News edition. Back to the roots, a few engineering articles I did not know how to put together these articles, so here a few loose articles.

Data 130

Data Algorithm Data Pipeline SQL 130

Silectis

FEBRUARY 17, 2021

In this blog post, we’ll walk you through how data science and data engineering are complementary disciplines. We’ll also delineate a third category: data analysis. We’ll explore how both data engineering and data science should be marshaled to make better decisions. So then what is Data Engineering?

Data Science 52

Data Science Data Engineering Data Engineer Engineering 52

Christophe Blefari

JULY 3, 2023

💡 If you want another view on both the conferences Ananth from Data Engineering Weekly wrote about the conferences extravaganza and a few trends he wanted to chat about. Databricks acquires MosaicML for $1.3b— It should land in data economy category but you know. Also you might know Mode through Benn Stancil blog.

SQL 130

SQL Data Kafka AWS 130

Christophe Blefari

DECEMBER 3, 2022

Joe Reis launched his Substack — Joe is the co-author of the great The Fundamentals of Data Engineering and his blog already have 2 articles I deeply recommend: No extra credit for complexity & Groundhog Days. This time Apple engineers shared at Trino—good old Presto for lost people—Summit how they use it.

Kafka 130

Kafka BI Finance Data 130

Cloudera

APRIL 24, 2023

Out of the box, CDP performs superbly, but over time, if data architecture, data engineering, and DevOps best practices are not maintained, the Data City you’ve erected atop a solid CDP bedrock can become the wild, wild, west. Categories of CDP Environmental Health We’ve categorized aspects of environmental health for this series.

Professional Services 81

Professional Services Datasets Data Pipeline Data Architecture 81

Knowledge Hut

APRIL 25, 2024

In this blog post, we will discuss such technologies. Big data technologies can be categorized into four broad categories: batch processing, streaming, NoSQL databases, and data warehouses. Each category has its own strengths and weaknesses, so it's important to select the right tool for the job at hand.

Big Data 69

Big Data Technology NoSQL Hadoop 69

Picnic Engineering

JANUARY 9, 2024

Especially if the products are in a category the customer has never bought before. The original papers for all these models can be found linked on the RecBole website and we won’t provide any detail in this blog as it is not too relevant. Explore categories are easier! Well they all boil down to sparsity.

Machine Learning 91

Machine Learning Datasets Algorithm Systems 91

Data Engineering Podcast

SEPTEMBER 4, 2022

Summary Data observability is a product category that has seen massive growth and adoption in recent years. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder. A few years ago, the category barely existed – what’s changed?

IT 100

IT Metadata MongoDB MySQL 100

Christophe Blefari

MARCH 31, 2023

We are slowly approaching the 2-years anniversary of the blog and the newsletter. To be honest time flies and I’d have preferred to do more for the blog in the start of the year but my freelancing activities and my laziness took me so much. This newsletter is about money ( credits ) Dear readers, already 3 months done in 2023.

Bytes 130

Bytes Data Google Cloud Education 130

DoorDash Engineering

APRIL 23, 2024

Products from different categories are characterized by different sets of uniquely defining attributes. And attributes such as product category and size enable recommending more relevant substitutions when the original item is out of stock, giving customers a smooth fulfillment experience.

Building 52

Building Retail Manufacturing Unstructured Data 52

ProjectPro

FEBRUARY 21, 2023

With so many data engineering certifications available , choosing the right one can be a daunting task. There are over 133K data engineer job openings in the US, but how will you stand out in such a crowded job market? The answer is- by earning professional data engineering certifications! AWS or Azure? Cloudera or Databricks?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. How does it work?

Metadata 130

Metadata MongoDB Scala MySQL 130

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Systems 130

Systems Metadata Data Pipeline MongoDB 130