Data Engineering Weekly

MAY 12, 2024

This year, we added some additional features to bring the data community together. Gloss Genius: How We Migrated From dbt Cloud and Scaled Our Data Development Gloss Genius describes its migration journey from dbt cloud to Airflow + custom Github actions. If you want to speak at the conference , propose the talk here : [link].

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Podcast

JULY 9, 2023

Summary For business analytics the way that you model the data in your warehouse has a lasting impact on what types of questions can be answered quickly and easily. In this episode Maxime Beauchemin of Airflow and Superset fame shares his vision for the entity-centric data model and how you can incorporate it into your own warehouse design.

Database-centric 130

Database-centric Machine Learning SQL Data Engineering 130

Data Engineering Podcast

FEBRUARY 26, 2023

Summary There has been a lot of discussion about the practical application of data mesh and how to implement it in an organization. Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. Is a data mesh ever "done"? We feel your pain.

Building 147

Building Metadata Machine Learning Data Integration 147

Data Engineering Weekly

APRIL 7, 2024

dbt: 2024 State of Analytics Engineering The 2024 dbt’s state of analytical engineering report is out. Poor data quality and unlcear data ownership remains the top challenges for the data teams. Data Mesh continuously gaining popularity among the enterprises.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Knowledge Hut

APRIL 25, 2024

Big data in information technology is used to improve operations, provide better customer service, develop customized marketing campaigns, and take other actions to increase revenue and profits. It is especially true in the world of big data. It is especially true in the world of big data. What Are Big Data T echnologies?

Big Data 98

Big Data Technology NoSQL Hadoop 98

Scott Logic

APRIL 22, 2024

Amazon’s goal in this space is to develop AI tools that are able to be used by non-technical users, so that organisations are not hamstrung by any bottlenecks caused by the number of data scientists they have available. They see the potential of AI, but struggle to find practical applications for it.

BI 97

BI Software Engineer Software Engineering Algorithm 97

Knowledge Hut

APRIL 23, 2024

Software engineering is a rapidly growing field with vast career opportunities. With emerging technologies like AI, machine learning, and blockchain, the demand for software engineers has skyrocketed. With emerging technologies like AI, machine learning, and blockchain, the demand for software engineers has skyrocketed.

Software Engineer 98

Software Engineer Software Engineering Engineering Programming Language 98

Data Engineering Weekly

APRIL 21, 2024

Meta: Introducing Meta Llama 3 - The most capable openly available LLM to date Meta is taking an interesting approach in the growing LLM market with the open source approach and distribution across all the leading cloud providers and data platforms. The blog narrates how Chronon fits into Stripe’s online and offline requirements.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Data Engineering Weekly

APRIL 14, 2024

link] Github: 4 ways GitHub engineers use GitHub Copilot The impact of LLM on software development is undeniable. Github shares some insights on how Github engineers use Github Copilot. Meta: OpenEQA - From word models to world models Will AI agents soon become a common fixture in our homes and an integral part of our daily lives?

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Lyft Engineering

APRIL 24, 2024

At Lyft, data scientists tackle challenging technical problems every day. To support and empower our data scientists, Lyft’s Technical Learning Council (TLC) provides diverse and high-quality continuous learning opportunities to hone their technical skills. See our previous blog post for more on the motivations behind TLC.

Data Science 66

Data Science Building Computer Science Software Engineer 66

Snowflake

FEBRUARY 21, 2024

We previously announced Snowflake’s Unistore workload , which continues Snowflake’s legacy of breaking down data silos by uniting transactional and analytical data in a consistent and governed platform. They are powered by a completely new storage engine and integrated deeply into the existing Snowflake architecture.

Government 97

Government AWS Data Architect Architecture 97

Pinterest Engineering

FEBRUARY 22, 2024

Staff Software Engineer At Pinterest we are continuously looking for ways to improve our developer experience, and we have recently shipped AI-assisted development for everyone while balancing safety, security, and cost. Technical Program Manager; Joe Gordon | Sr. The first decision we had to make was build versus buy.

Scala 78

Scala Engineering Software Engineer Software Engineering 78

Data Engineering Weekly

FEBRUARY 4, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Joe Reis: Definition of Data Modeling & What Data Modeling Is not Joe raised a very fundamental question in data engineering.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Pinterest Engineering

MARCH 20, 2024

Graph based model: Leverage the Pinner’s curation data to build a heterogeneous graph that supports different types of entities. Most of the Pins are linked to off-site content to provide Pinners with inspiration and actionability. community guidelines ), functional, relevant, and actionable (e.g. Ads and Shopping) the off-site content is.

Machine Learning 73

Machine Learning Engineering Software Engineer Software Engineering 73

Pinterest Engineering

NOVEMBER 7, 2023

Jeff Xiang | Software Engineer, Logging Platform Vahid Hashemian | Software Engineer, Logging Platform Jesus Zuniga | Software Engineer, Logging Platform At Pinterest, data is ingested and transported at petabyte scale every day, bringing inspiration for our users to create a life they love.

Kafka 99

Kafka Java Software Engineer Software Engineering 99

Data Engineering Weekly

NOVEMBER 26, 2023

It’s the end of the year, and there will be a lot of buzz about what the next five years in data engineering might bring. It's like piecing together a puzzle from history to get a clearer picture of the future. I believe the impact of LLM will go further down in the stack with data storage formats in the coming years.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Pinterest Engineering

MAY 9, 2023

Erika Sun ML Engineer | Advertiser Growth Modeling Team; Ogheneovo Dibie Engineering Manager | Advertiser Growth Modeling Team Photo by Jason Blackeye on Unsplash Summary In this blog post, we describe a Machine Learning (ML) powered proactive churn prevention solution that was prototyped with our small & medium business (SMB) advertisers.

Consulting 81

Consulting Data Science Architecture Engineering 81

Snowflake

FEBRUARY 22, 2024

A lot of the buzz around AI focuses on its future potential. A solid data foundation is critical to being able to successfully leverage AI solutions, and Snowflake’s platform allows both marketing solution providers and marketing professionals to capitalize on this movement of AI in marketing with data at the core.

Technology 85

Technology Machine Learning Utilities Designing 85

Data Engineering Weekly

DECEMBER 3, 2023

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Github writes an excellent blog to capture the current state of the LLM integration architecture. Lackluster AI/ML results often stem from poor data quality.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Cloudera

FEBRUARY 1, 2024

Delivering transformational innovation and accurate business decisions requires harnessing the full potential of your organization’s entire data ecosystem. Ultimately, this boils down to how reliable and trustworthy the underlying data that feeds your insights and applications is.

Cloud 85

Cloud Architecture Utilities Designing 85

Netflix Tech

NOVEMBER 29, 2023

As noted in an earlier blog post , we began developing an HDR variant of VMAF; let’s call it HDR-VMAF. A vital aspect of such development is subjective testing with HDR encodes in order to generate training data. A vital aspect of such development is subjective testing with HDR encodes in order to generate training data.

Metadata 100

Metadata Electronics Algorithm Software Engineer 100

Cloudera

SEPTEMBER 13, 2023

It was with this team that I was able to grow into a principal engineer. Today, many of those same team members are still here at Cloudera driving the new ultramodern data platform. In the near future, I’ll be presenting in Halifax’s Community Over Code , discussing Cloudera’s open-source projects.

Technology 87

Technology Hadoop Kafka Project 87

Knowledge Hut

MARCH 20, 2024

The rise in demand for automation software engineers has become indispensable across industries. Automation Software Engineers are at the forefront of this revolution, playing a crucial role in developing and implementing automated solutions to enhance efficiency and productivity. Who is an Automation Software Engineer ?

Software Engineer 52

Software Engineer Software Engineering Engineering Programming Language 52

Knowledge Hut

MARCH 14, 2024

The rise in demand for automation software engineers has become indispensable across industries. Automation Software Engineers are at the forefront of this revolution, playing a crucial role in developing and implementing automated solutions to enhance efficiency and productivity. Who is an Automation Software Engineer ?

Software Engineer 52

Software Engineer Software Engineering Engineering Programming Language 52

RandomTrees

APRIL 29, 2024

In this domain, engineers and scientists tirelessly innovate to make it possible, that sometimes seemed impossible. Advanced AI algorithms combined with big data analytics have revolutionized the way researchers model complex scenarios and optimize vehicle performance. The center of this revolution is the rise of Generation AI (GenAI).

Transportation 52

Transportation Manufacturing Algorithm Machine Learning 52

Pinterest Engineering

AUGUST 3, 2023

An interview with Behnam Rezaei | Pinterest VP, Engineering At Pinterest, we’re on a mission to bring everyone the inspiration to create a life they love. Joining Pinterest in March 2023, Behnam Rezaei is Pinterest’s VP for Monetization — Machine Learning Engineering and Data Science based in San Francisco.

Engineering 77

Engineering Machine Learning Data Science Technology 77

Data Engineering Weekly

OCTOBER 1, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. What is the data behavior? See how it works today. Dropbox: Is this a date?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Data Engineering Weekly

SEPTEMBER 24, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Airflow isn’t meant to process the data. See how it works today.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Knowledge Hut

MARCH 19, 2024

Machine Learning Software Engineers are at the forefront of this revolution, applying their expertise to develop intelligent systems and algorithms. In this blog, I will describe the role of a Machine Learning Software Engineer, their responsibilities, required skills, and the path to becoming one.

Software Engineer 52

Software Engineer Software Engineering Machine Learning Engineering 52

Edureka

DECEMBER 5, 2023

These days, it’s common to hear the expression “prompt engineering,” which sends us into a mental rabbit hole. Prompt engineering: what is it? In data analysis, tailored prompts can extract specific insights. In data analysis, tailored prompts can extract specific insights. How does it work?

Engineering 52

Engineering Machine Learning Data Analysis Coding 52

Knowledge Hut

MARCH 13, 2024

People often associate software engineering with complex algorithms, commands, and data structures. Having worked in the software engineering field for many years, I can say that the code can sometimes seem like an alien language. What is Software Engineering? What is Software Engineering?

Software Engineer 52

Software Engineer Software Engineering Engineering Coding 52

Data Engineering Weekly

MARCH 10, 2024

Editor’s Note: Chennai Meetup Wrap-Up & Preparation work started for DEWCon I am so grateful for the enthusiastic participants who made our Chennai Data Heroes- Community for Data Folks meetup vibrant! Big thanks to our insightful speakers, Hareshkumar Selvakumar - Talks about his work on Data Products for PayPal.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Weekly

JANUARY 21, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Interesting article on the impact of search engine optimization (SEO) on the quality of search engine results. Visit rudderstack.com to learn more.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Workfall

JUNE 12, 2023

In this dynamic realm of data engineering, a monumental challenge takes centre stage: efficiently managing the ever-changing tides of real-time data. Data, the lifeblood of organisations, holds the key to unlocking untapped potential and propelling businesses forward. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Pinterest Engineering

JULY 18, 2023

Brian Overstreet | Software Engineer, Observability; Humsheen Geo | Software Engineer, Observability Time series is a critical part of Observability at Pinterest, powering 60,000 alerts and 5,000 dashboards. This blog post describes TScript and how we use it at Pinterest. Performing operations on the returned time series.

Database 74

Database Software Engineer Software Engineering Raw Data 74