Data Engineering Podcast

OCTOBER 22, 2023

Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the delivery mechanism for information. With Materialize, you can!

BI 162

BI SQL Machine Learning Programming Language 162

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. Each time I look back and I see the amount of Fridays I've spent reading and writing I'm still surprised. Monte Carlo also wrote this week about chaos engineering , with a manifesto. Is it Kafka?

Kafka 130

Kafka Data Lake Data Storage Data 130

Knowledge Hut

APRIL 25, 2024

Big data in information technology is used to improve operations, provide better customer service, develop customized marketing campaigns, and take other actions to increase revenue and profits. It is especially true in the world of big data. It is especially true in the world of big data. What Are Big Data T echnologies?

Big Data 98

Big Data Technology NoSQL Hadoop 98

Knowledge Hut

MAY 2, 2024

As per Apache, “ Apache Spark is a unified analytics engine for large-scale data processing ” Spark is a cluster computing framework, somewhat similar to MapReduce but has a lot more capabilities, features, speed and provides APIs for developers in many languages like Scala, Python, Java and R. billion (2019 - 2022).

Scala 52

Scala Hospitality Healthcare Retail 52

Data Engineering Weekly

MARCH 24, 2024

link] Kai Waehner: The Data Streaming Landscape 2024 This is a comprehensive overview of the state of the data streaming landscape in 2024. As we predicted in the key trends of 2023 about Apache Flink as a clear winner in the stream processing frameworks, we see Confluent offering Flink as a service.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Christophe Blefari

FEBRUARY 18, 2023

When the Data News lands on Saturday ( credits ) In last week newsletter I've also share what is a metrics store, which led to a longer edition than usual and I saw that a few people did not like it this way. Last year DataOps has been used in many different ways to describe so many data-related different tasks.

Software Engineer 100

Software Engineer Software Engineering Data Validation Data 100

Pinterest Engineering

SEPTEMBER 29, 2023

Sanchay Javeria | Software Engineer, Ads Data Infrastructure To support metrics reporting for ads from external advertisers and real-time ad budget calculations at Pinterest, we run streaming pipelines using Apache Flink. To do this, it was helpful to revisit Flink’s memory model.

Utilities 93

Utilities Coding Kafka Engineering 93

Knowledge Hut

JANUARY 11, 2024

Data science has become popular in the last few years due to its successful application in making business decisions. Data scientists have been using data science techniques to solve challenging real-world issues in healthcare, agriculture, manufacturing, automotive, and many more.

Data Science 98

Data Science Transportation Hospitality Banking 98

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Towards Data Science

APRIL 3, 2023

Building a Mental Model for Engineers and Anyone in Between Stream Processing can be handled gently and with care, or wildly, and almost out of control! By processing a smaller set of data, more often , you effectively divide and conquer a data problem that may otherwise be cost and time prohibitive.

Process 86

Process Data Lake Systems Bytes 86

Data Engineering Podcast

OCTOBER 16, 2022

Pathway is a streaming database engine that embeds artificial intelligence into the storage, with functionality designed to support the spatiotemporal data that is crucial for shipping and logistics. Atlan is the metadata hub for your data ecosystem.

Database 100

Database Metadata MongoDB MySQL 100

Data Engineering Weekly

APRIL 28, 2023

Welcome to another episode of Data Engineering Weekly. Aswin and I select 3 to 4 articles from each edition of Data Engineering Weekly and discuss them from the author’s and our perspectives. 🤔] engineering. 42% say their top concern is “Data isn’t where business users need it.”

Consulting 52

Consulting Kafka Lambda Architecture Engineering 52

Lyft Engineering

OCTOBER 3, 2023

High-quality data is necessary for the success of every data-driven company. It enables everything from reliable business logic to insightful decision-making and robust machine learning modeling. It is now the norm for tech companies to have a well-developed data platform. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Data Engineering Podcast

JULY 12, 2021

Summary We have been building platforms and workflows to store, process, and analyze data since the earliest days of computing. Over that time there have been countless architectures, patterns, and "best practices" to make that task manageable. RudderStack’s smart customer data pipeline is warehouse-first.

Designing 100

Designing Data Warehouse Data Governance Consulting 100

Rockset

APRIL 18, 2023

We’re excited to introduce vector search on Rockset to power fast and efficient search experiences, personalization engines, fraud detection systems and more. Organizations have continued to accumulate large quantities of unstructured data, ranging from text documents to multimedia content to machine and sensor data.

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

Ascend.io

APRIL 22, 2024

At Ascend, we believe it’s time to rethink data engineering from the ground up. As the world of data continues to evolve at a breakneck pace, we are thrilled to announce the next revolutionary step in our journey – Project Inception. Read More: The State of Data Engineering in 2023: Does Your Data Program Stack Up?

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Podcast

JULY 19, 2021

Summary Data quality is a concern that has been gaining attention alongside the rising importance of analytics for business success. While those are useful tools, it is far better to prevent data errors before they become an outsized issue. RudderStack’s smart customer data pipeline is warehouse-first.

Management 100

Management Data Warehouse Data Pipeline Data 100

Cloudera

AUGUST 4, 2021

In fact, McKinsey points to a 50% reduction in downtime and a 40% reduction in maintenance costs when using IoT and data analytics to predict and prevent breakdowns. The key is active and ongoing monitoring of prognostic health data. Cloudera is foundational in how we track and govern our data.” .

Unstructured Data 98

Unstructured Data Data Ingestion Government Machine Learning 98

Data Engineering Weekly

NOVEMBER 5, 2023

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Netflix: Streaming SQL in Data Mesh Learnings from our journey In hindsight, we wish we had invested in enabling Flink SQL on the DataMesh platform much earlier.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Cloudera

OCTOBER 13, 2021

What’s data analytics and why is everyone talking about it?”. It turns out it’s much more than just modeling (when you train a computer to distinguish between a cat and a person , for example). . Production ML systems automate business operations and are capable of handling enormous amounts of real-time data.

Machine Learning 75

Machine Learning Data Ingestion Algorithm Technology 75

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Modern platforms like Redshift , Snowflake , and BigQuery have elevated the data warehouse model.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Data Engineering Weekly

NOVEMBER 27, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. The real-time agent assistant model is an exciting read.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Weekly

AUGUST 20, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. But 4 years later, in 2023 — where has the data mesh gotten us?

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Data Engineering Podcast

MARCH 8, 2021

Summary A majority of the time spent in data engineering is copying data between systems to make the information available for different purposes. After tasking some of his top engineers to consider the problem in a new light they created the Pilosa engine. In this episode H.O.

IT 100

IT Data Warehouse MongoDB Kafka 100

Data Engineering Podcast

MAY 1, 2022

Summary Building a data platform is an iterative and evolutionary process that requires collaboration with internal stakeholders to ensure that their needs are being met. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

The Pragmatic Engineer

DECEMBER 8, 2022

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. Starting February 27th 2023, we are adopting a “default together” model where team members are expected to spend (.) Twitter does have a process to ask for exceptions from RTO. Here are some of them.

Software Engineer 132

Software Engineer Software Engineering Medical Insurance 132

Jesse Anderson

JUNE 7, 2022

In 2021 I had the pleasure to first get to know and speak with Zhamak Dheghani, Director of Emerging Technologies at ThoughtWorks, in season one of the Data Dream Team series. I interviewed Zhamak last season as more of an introduction to Data Mesh. This episode is different to others in the Data Dream Team series.

Generalist 130

Generalist Software Engineer Software Engineering Data 130

Data Engineering Weekly

MARCH 26, 2023

Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real time with your own JavaScript or Python code. dbt: State of Analytics Engineering dbt publishes the state of analytical [data???🤔] 🤔] engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Workfall

JULY 18, 2023

Reading Time: 8 minutes Picture yourself in the bustling world of a leading streaming platform, where countless users rely on personalized recommendations for their next binge-watching adventure. Behind the scenes, a team of data wizards tirelessly crunches mountains of data to make those recommendations sparkle.

Datasets 77

Datasets Entertainment Data Workflow Data 77

Data Engineering Weekly

APRIL 23, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. I open-sourced Schemata last year, the industry's first Data Contract as a Code (DCC).

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Data Engineering Weekly

APRIL 2, 2023

Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real-time with your own JavaScript or Python code. Meta: Presto - A Decade of SQL Analytics at Meta Presto and Kafka are the two systems that greatly impacted data infrastructure in the last decade.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Podcast

JANUARY 13, 2020

This requires a new class of data storage which can accomodate that demand without having to rearchitect your system at each level of growth. YugabyteDB is an open source database designed to support planet scale workloads with high data density and full ACID compliance. A growing trend in database engines (e.g.

SQL 100

SQL PostgreSQL MongoDB Database Design 100

Monte Carlo

MAY 26, 2023

What is data freshness and why is it important? Data freshness, sometimes referred to as data timeliness, is the frequency in which data is updated for consumption. It is an important dimension of data quality and a pillar of data observability because recently refreshed data is more accurate, and thus more valuable.

Data Pipeline 98

Data Pipeline Data Data Warehouse Machine Learning 98

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Confluent

MARCH 19, 2019

Some background first—I wrote a post last year about how event streaming platforms are “not your dad’s message bus anymore.” Message brokers have evolved into intelligent event streaming platforms. The message brokers nowadays can be analytical engines by their own right. Streaming analytics has become mainstream.

Architecture 110

Architecture Telecommunication Transportation Database 110

Knowledge Hut

DECEMBER 26, 2023

In recent years, with the advent of technology, data has been considered to be a valuable asset in both large-scale and small-scale organizations. Data as a resource requires skilled professionals to be collected, interpreted, and stored safely. The demand for data scientists and career prospects has skyrocketed in the job market.

Data Science 52

Data Science Scala Programming Language R (Programming) 52

Knowledge Hut

JANUARY 9, 2024

Increases Your Earnings Potential As per reports by Glassdoor India, cloud engineers have an average salary of ₹6.82 Additionally, students learn about service and deployment models, SLAs, economic models, cloud security, enabling technologies, popular cloud stacks, and their use cases. lakh per annum.

Cloud Computing 52

Cloud Computing Cloud Amazon Web Services Cloud Storage 52