Data Engineering Podcast

OCTOBER 15, 2023

Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can!

Process 182

Process Building SQL BI 182

Christophe Blefari

SEPTEMBER 15, 2023

Facing the News ( credits ) Hello Data News readers. If you're late to the party and you need fresh views on LLMs Daniel wrote an introduction demystifying the Large Language Models and Jesse wrote about LLMs impact from a Data Engineering perspective. — Hugo propose 7 hacks to optimise data warehouse cost.

Data Warehouse 130

Data Warehouse Data SQL Python 130

Christophe Blefari

APRIL 19, 2024

easy ( credits ) Hey, new Friday, new Data News. How we build Slack AI to be secure and private — How Slack uses VPC and Amazon SageMaker with your data secured and private. Data pipeline, incremental vs. full load — A comprehension comparison between 2 mode of ingestion with a decision tree about which one to pick.

MySQL 130

MySQL Data Datasets SQL 130

Christophe Blefari

DECEMBER 31, 2023

Before moving on to the Data News, a bit of personal news, in December, I took part in the MotherDuck meetup in Berlin. End of January, on the 31st I'll speak at a Modern Data Stack conf in Paris, still about DuckDB, but this time in French. Enjoy this last 2023 Data News. We're going to get to know each other.

Data 100

Data Cloud Storage Datasets Python 100

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community!

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Data Engineering Podcast

JULY 2, 2023

Summary Feature engineering is a crucial aspect of the machine learning workflow. In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Christophe Blefari

JANUARY 14, 2024

Back to the usual Data News—with a little delay, I'm sorry. It's a subject close to my heart and I was very happy to share it with you, because I never thought that Data News would become such a big part of my life. I actually cover data engineering and how to put data stuff into production.

Google Cloud 130

Google Cloud Data SQL Python 130

Knowledge Hut

DECEMBER 29, 2023

The market for analytics is flourishing, as is the usage of the phrase Data Science. Professionals from a variety of disciplines use data in their day-to-day operations and feel the need to understand cutting-edge technology to get maximum insights from the data, therefore contributing to the growth of the organization.

Data Science 98

Data Science Data Mining Deep Learning Programming Language 98

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

OCTOBER 1, 2023

Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can!

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

OCTOBER 22, 2023

Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the delivery mechanism for information. With Materialize, you can!

BI 162

BI SQL Machine Learning Programming Language 162

Knowledge Hut

JULY 4, 2023

Pattern recognition is a field of computer science that deals with the automatic identification of patterns in data. This can be done by finding regularities in the data, such as correlations or trends, or by identifying specific features in the data. What Is Pattern Recognition?

Machine Learning 98

Machine Learning Medical Algorithm Deep Learning 98

Data Engineering Podcast

OCTOBER 8, 2023

In this episode he shares his journey of data collection and analysis and the challenges of automating an intentionally manual industry. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can!

Insurance 162

Insurance BI SQL Machine Learning 162

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. The current data state is heavily dependent on infrastructure, wether it's cloud, on-premise or semi-related we need to understand where the data lands and where the code runs. Often they are right.

Kafka 130

Kafka Data Lake Data Storage Data 130

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Knowledge Hut

APRIL 26, 2024

In the age of big data processing, how to store these terabytes of data surfed over the internet was the key concern of companies until 2010. Now that the issue of storage of big data has been solved successfully by Hadoop and various other frameworks, the concern has shifted to processing these data.

Data Science 52

Data Science Machine Learning Hadoop Programming Language 52

Knowledge Hut

APRIL 25, 2024

Big data in information technology is used to improve operations, provide better customer service, develop customized marketing campaigns, and take other actions to increase revenue and profits. It is especially true in the world of big data. It is especially true in the world of big data. What Are Big Data T echnologies?

Big Data 98

Big Data Technology NoSQL Hadoop 98

Edureka

APRIL 24, 2024

Getting Trained on Data: To perform any task, first, the generative AI models need to be trained on massive datasets of existing content. This data can be retrieved from anything – books, blogs, pictures or images. Rather, you can opt for a good Prompt Engineering course online to kickstart your journey.

Pharmaceutical 52

Pharmaceutical Manufacturing Datasets Algorithm 52

Knowledge Hut

FEBRUARY 1, 2024

Every time someone uses the internet, more data is added. We need assistance from various Data Science methodologies to make sense of this enormous amount of data and use it for the company's objectives, etc. Data science for eCommerce has been one of the most impactful technologies in recent years.

Data Science 96

Data Science Retail Algorithm Manufacturing 96

Knowledge Hut

DECEMBER 26, 2023

According to the World Economic Forum, the amount of data generated per day will reach 463 exabytes (1 exabyte = 10 9 gigabytes) globally by the year 2025. Thus, almost every organization has access to large volumes of rich data and needs “experts” who can generate insights from this rich data.

Data Science 97

Data Science BI Business Intelligence Data Mining 97

Knowledge Hut

JANUARY 18, 2024

Data Science has risen to become one of the world's topmost emerging multidisciplinary approaches in technology. Recruiters are hunting for people with data science knowledge and skills these days. Data Scientists collect, analyze, and interpret large amounts of data. Choose data sets.

Data Science 98

Data Science Business Analyst ETL Method Data Architect 98

Pinterest Engineering

SEPTEMBER 29, 2023

Sanchay Javeria | Software Engineer, Ads Data Infrastructure To support metrics reporting for ads from external advertisers and real-time ad budget calculations at Pinterest, we run streaming pipelines using Apache Flink. Framework off-heap memory is reserved for Flink’s internal operations and data structures.

Utilities 93

Utilities Coding Kafka Engineering 93

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Data Process 90

Data Process Process Metadata Finance 90

Picnic Engineering

FEBRUARY 8, 2023

In this post, we will explain how our team automates the creation of new data pipeline deployments. We’ve decided to implement the Saga pattern to solve this issue! What is the Saga pattern? The Saga pattern is based on the idea of a “saga”: a long story of a hero’s journey. How did we implement the Saga pattern?

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Knowledge Hut

APRIL 25, 2024

In the world of data science, Spatial data science is at the heart of contributing to solving planet-threatening problems such as deciding the location of solar park installation, building urban resilience, predicting crop yield, population density analysis for immunization or disease mapping, and more. What is Spatial Data Science?

Data Science 52

Data Science Telecommunication Transportation Big Data 52

Pinterest Engineering

SEPTEMBER 20, 2023

ML Engineer & Inclusive AI Tech Lead; Bhawna Juneja | Sr. Machine Learning Engineer; Rohan Mahadev | Machine Learning Engineer II; Sujay Khandagale | Machine Learning Engineer II; Abhay Varmaraja | Machine Learning Engineer II Pinterest’s mission as a company is to bring everyone the inspiration to create a life they love.

Building 84

Building Pipeline-centric Machine Learning Datasets 84

Ascend.io

OCTOBER 25, 2023

Data teams are consistently challenged by a rapidly evolving technological landscape and escalating demands. Over the past four years, we have conducted an industry-wide DataAware Pulse Survey to capture the current state of data teams. In this article, we detail the six main patterns from the 2023 results.

Data Programming 52

Data Programming Data Engineering Data Engineer Programming 52

Knowledge Hut

MARCH 29, 2024

It can be difficult to distinguish between a software engineer and a software architect in the digital era. This post will clarify the argument between software engineers and architects and assist you in determining which would be most appropriate for you. What is a Software Engineer?

Software Engineer 52

Software Engineer Software Engineering Engineering Programming Language 52

Knowledge Hut

MAY 31, 2023

Currently, most students and working professionals prefer a Data Science Course to make a smooth transition in the data science field. These projects typically involve a collaborative team of software developers, data scientists, machine learning engineers, and subject matter experts. Let us get started!

Project 96

Project Healthcare Deep Learning Transportation 96

Knowledge Hut

MARCH 14, 2024

The process of disassembling a product, system, or program into its constituent parts, comprehending how they function, and then utilizing that knowledge to either reproduce or enhance the original is known as reverse engineering. I will be doing a detailed discussion of reverse engineering in software engineering in this article.

Software Engineer 52

Software Engineer Software Engineering Engineering Electronics 52

Data Engineering Weekly

JANUARY 28, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. I had a chance to meet some of the amazing humans of data engineering. Visit rudderstack.com to learn more.

Data Engineering 57

Data Engineering Data Engineer Engineering Recruitment 57

Data Engineering Weekly

APRIL 21, 2024

Meta: Introducing Meta Llama 3 - The most capable openly available LLM to date Meta is taking an interesting approach in the growing LLM market with the open source approach and distribution across all the leading cloud providers and data platforms. Counting is the hardest problem in data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Knowledge Hut

MARCH 26, 2024

On the other hand, AI, compared to human intelligence, is a product of human-designed algorithms, computational power, and data processing capabilities. AI systems are designed to analyze large amounts of data, identify patterns, and make predictions or decisions based on algorithms and computational models.

Algorithm 52

Algorithm Deep Learning Education Datasets 52

Ascend.io

APRIL 22, 2024

At Ascend, we believe it’s time to rethink data engineering from the ground up. As the world of data continues to evolve at a breakneck pace, we are thrilled to announce the next revolutionary step in our journey – Project Inception. Read More: The State of Data Engineering in 2023: Does Your Data Program Stack Up?

Project 52

Project Metadata Data Pipeline Data Engineering 52

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Modern platforms like Redshift , Snowflake , and BigQuery have elevated the data warehouse model.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Engineering at Meta

JANUARY 18, 2024

The milliseconds it takes for an ML model to transition from conceptualization to processing the initial training data can dramatically impact productivity and experimentation. It handles large volumes of data in groups (or batches) and allows us to train models, optimize parameters, and perform inference more effectively and swiftly.

Machine Learning 80

Machine Learning Python Coding Education 80

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes. But with vastly different architectural worldviews.

Data Lake 89

Data Lake Data Warehouse BI SQL 89