Knowledge Hut

SEPTEMBER 21, 2023

Questions such as which is better and easier to learn—AWS, Azure, or GCP— are often asked by organization leaders before starting out on their cloud journey. Azure's robust and secure infrastructure helps customers migrate to the cloud with confidence. Azure or Google Cloud—Which is better?

Cloud 52

Cloud Google Cloud Cloud Computing Cloud Storage 52

Striim

FEBRUARY 10, 2023

Additionally, operators can begin experimenting with machine learning by using real-time analytics to identify network events that could impact performance. According to a recent study by KX, US businesses could see a total revenue uplift of $2.6 trillion through investment in real-time data analytics.

Data Analytics 52

Data Analytics Telecommunication Insurance Google Cloud 52

Data Engineering Weekly

FEBRUARY 19, 2023

Netflix: Scaling Media Machine Learning at Netflix Netflix writes about media machine learning infrastructure and media-focused ML infrastructure to reduce the time from ideation to productization for media ML practitioners. The blog mostly focuses on how it optimizes the batch pipeline using Airflow and BigQuery.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Christophe Blefari

DECEMBER 31, 2023

A lot of things have been happening at the same time in my professional and personal life. To be honest, everything's been going well, but I've found it hard to find time to write among other things. And that's the problem. I want to do so many things at once. Yeah, you know. We're going to get to know each other.

Data 100

Data Cloud Storage Datasets Python 100

Christophe Blefari

SEPTEMBER 1, 2023

Back to school ( credits ) Hey, I'm back. I've taken an unplanned 3-week break since the last Data News, let's be honest, it was necessary! I spent a few hours working on the fancy data stack project and articles are in the works, but it was idealistic to produce quality code and content while enjoying the summer.

Food 130

Food Data SQL Python 130

Data Engineering Weekly

FEBRUARY 4, 2024

Visit rudderstack.com to learn more. link] Michiel De Muynck: 7 Lessons Learned migrating dbt code from Snowflake to Trino SQL standards are like toothbrushes: everyone agrees they're essential, but no one wants to use someone else's. What is Data Modeling, and what is not?

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Podcast

OCTOBER 9, 2022

Summary For any business that wants to stay in operation, the most important thing they can do is understand their customers. American Express has invested substantial time and effort in their Customer 360 product to achieve that understanding. And don’t forget to thank them for their continued support of this show!

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

AUGUST 3, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. Vinoth Chandar helped to create the Hudi project while at Uber to address this challenge.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

RandomTrees

FEBRUARY 22, 2024

How to Migrate Your Business to the Cloud Moving to the google cloud workload is one of the most impressive things your business can utilize to build your adaptability, flexibility, and productivity. Migrating your business to the cloud implies a ton of planning and attention.

Google Cloud 52

Google Cloud Cloud Utilities Machine Learning 52

Christophe Blefari

MAY 6, 2023

Even Amazon can't make sense of serverless or microservices — PrimeVideo tech team wrote an article that could be summarised by: we migrated from functions based approach to a monolith in a VM. Meta ones). If you already have access to OpenAI in Azure you can now use GPT-4 —only in preview yet. Internet found this ironical.

Programming Language 100

Programming Language Data Python Data Warehouse 100

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure. Let’s jump in!

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

Monte Carlo

NOVEMBER 20, 2023

Far from a mere “copy-paste” job, data migrations can be deceptively tricky and one of the most challenging aspects of a data engineer’s job. But as businesses pivot and technologies advance, data migrations are—regrettably—unavoidable. Table of Contents What makes data migrations complex?

MongoDB 52

MongoDB ETL Tools Database Data 52

Knowledge Hut

NOVEMBER 17, 2023

The days of struggling with complicated networking and on-premise server rooms are long gone. Thanks to cloud computing, services are now secure, reliable, and cost-effective. When we talk of top cloud computing providers, there are 2 names that are ruling the markets right now- AWS and Google Cloud.

Google Cloud 98

Google Cloud AWS Cloud Cloud Computing 98

Data Engineering Podcast

JULY 12, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. Go to dataengineeringpodcast.com/97things today to get your copy!

Designing 100

Designing Data Warehouse Data Governance Consulting 100

Data Engineering Podcast

JULY 27, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. Go to dataengineeringpodcast.com/97things today to get your copy!

Building 100

Building Management Kafka Data Warehouse 100

Data Engineering Podcast

DECEMBER 8, 2019

If you are evaluating your options for building or migrating a data platform, then this is definitely worth a listen. And for your machine learning workloads, they just announced dedicated CPU instances. SnowflakeDB has been leading the charge to take advantage of cloud services that simplify the separation of compute and storage.

Data Warehouse 100

Data Warehouse Cloud AWS Relational Database 100

Monte Carlo

JANUARY 5, 2024

Data lakehouses, like Databricks’ Delta Lake , Amazon Redshift Spectrum , and now Snowflake for Data Lakehouse , help to unify data warehousing, machine learning, and analytics. Snowflake announced Snowpipe for streaming and refactored their Kafka connector, and Google announced Pub/Sub could now be streamed directly into the BigQuery.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

Data lakehouses, like Databricks’ Delta Lake , Amazon Redshift Spectrum , and now Snowflake for Data Lakehouse , help to unify data warehousing, machine learning, and analytics. Snowflake announced Snowpipe for streaming and refactored their Kafka connector, and Google announced Pub/Sub could now be streamed directly into the BigQuery.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JULY 7, 2022

Data contracts could become a key piece of the data quality puzzle, and much can be learned from Andrew’s experience implementing them at GoCardless, detailed below. 7 Key Learnings From Our Experience Implementing Data Contracts A perspective from Andrew Jones, GoCardless One of the core values at GoCardless is to ask why.

Data Warehouse 52

Data Warehouse Software Engineer Software Engineering Data 52

Data Engineering Podcast

FEBRUARY 8, 2021

This is a great conversation about the lessons learned from real world use of a specific technology and how well it lives up to its promises. This is a great conversation about the lessons learned from real world use of a specific technology and how well it lives up to its promises.

Data Warehouse 100

Data Warehouse Building BI SQL 100

Monte Carlo

APRIL 27, 2023

Sigma Workbook Pages will automatically surface as lineage nodes and catalog entries once users have connected their Monte Carlo environment to Snowflake, BigQuery, Redshift, or Athena. Each interconnected layer can introduce and be impacted by poor data quality. What’s worse, we weren’t catching the issues on our own.”

BI 52

BI Business Intelligence Metadata Data Warehouse 52

dbt Developer Hub

NOVEMBER 28, 2021

Let’s export log events into BigQuery. “I They were often written in naked python scripts that only ran a SQL query + wrote data to BigQuery. More fundamentally, I think it’s important to call out that Airflow + dbt are spiritually aligned in purpose. In a Google Cloud Storage bucket. Let’s use GCP’s logging alerts to send emails.

Google Cloud 52

Google Cloud SQL Cloud Consulting 52

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Modern platforms like Redshift , Snowflake , and BigQuery have elevated the data warehouse model.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Monte Carlo

AUGUST 31, 2023

Your CTO is excited about migrating to the cloud, and constantly sends your team articles highlighting performance measurements against some of the latest technologies. But what happens when the data is wrong? If any of these situations hit home for you, you’re not alone. The root of data downtime? Unreliable data, and lots of it.

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Data Engineering Podcast

JULY 30, 2023

This makes it impossible to select a truly future-proof technology stack for your data platform, making an eventual migration inevitable. In this episode Gleb Mezhanskiy and Rob Goretsky share their experiences leading various data platform migrations, and the hard-won lessons that they learned so that you don't have to.

SQL 130

SQL Machine Learning Python Data 130

Databand.ai

DECEMBER 13, 2022

Whether you’re a data engineering pro looking to stay up to date on the latest trends or new to the space and want to learn more, following the right leaders and joining the right conversations can make all the difference when it comes to plugging into the data engineering community. You’ve come to the right place.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Christophe Blefari

JUNE 24, 2023

Iceberg in the clouds — Last week BigQuery announced Iceberg support in GA. ( credits ) Hey, this is the Data News. It's super hard to change habits, but it's how it is, the newsletter is going out on Saturday. I hope this edition finds you well. We'll be writing models moderating generative models.

PostgreSQL 130

PostgreSQL Data Data Engineering Data Engineer 130

Striim

APRIL 30, 2024

The transition involved moving from a traditional, batch-load dependent data warehousing approach to a more dynamic, cloud-based infrastructure, which encompassed their e-business platform and analytics powered by BigQuery. It strengthened business operations, empowering sophisticated machine learning projects and immediate data analysis.

Google Cloud 52

Google Cloud High Quality Data Business Intelligence Data Warehouse 52

Christophe Blefari

MAY 12, 2023

In Minds of Data I'll met people from the data ecosystem in order to learn more about them. I hope you will enjoy it as much as me. Let's start with with my newly created podcast Minds of Data. I really liked Benoit and Samy presentation about Cloud Composer —Managed Airflow on GCP. for IT security or medicine).

Data 130

Data Data Storage SQL Coding 130

Monte Carlo

FEBRUARY 15, 2023

A data warehouse is an online analytical processing system that stores vast amounts of data collected within a company’s ecosystem and acts as a single source of truth to enable downstream data consumers to perform business intelligence tasks, machine learning modeling, and more. Cloud data warehouses solve these problems.

Data Warehouse 52

Data Warehouse Unstructured Data AWS Business Intelligence 52

Data Engineering Weekly

MAY 21, 2023

I can also see a wave of migration from data warehouses to LakeHouse formats. Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Monte Carlo

JUNE 14, 2023

Be sure to read to the end where we share 6 data pipeline architecture diagrams used by real data teams at companies like JetBlue, Fox Networks, Drata, and more. What is data pipeline architecture? Data pipeline architecture is the process of designing how data is surfaced from its source system to the consumption layer.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Data Engineering Podcast

MARCH 15, 2021

Summary The team at DoorDash has a complex set of optimization challenges to deal with using data that they collect from a multi-sided marketplace. This is a valuable look at how to manage a large and growing data platform with that supports a variety of teams with varied and evolving needs.

Management 100

Management Data Warehouse PostgreSQL Kafka 100

Monte Carlo

NOVEMBER 15, 2022

Data teams that run on top of AWS and GCP can reap the benefits of Power BI to generate fast insights leveraging data from Snowflake, AWS Redshift, BigQuery, Databricks, and other leading data warehouse and lake providers. Read on to learn more about our new integration.

BI 52

BI Data Warehouse Telecommunication Business Intelligence 52

Data Engineering Weekly

JANUARY 29, 2023

Learn the 10 most important things to consider when choosing a data observability platform. Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Ripple Engineering

MAY 27, 2021

Our data analysts used to schedule queries on BigQuery for transformation workflows and test the transformed data manually. Our data analysts used to schedule queries on BigQuery for transformation workflows and test the transformed data manually. At Ripple , we are moving towards building complex business models out of raw data.

Building 52

Building Raw Data SQL Data 52

AltexSoft

JANUARY 26, 2022

They commonly prepare data and build machine learning (ML) models. Get acquainted with how data is prepared for machine learning projects in our dedicated video. Machine learning (ML) engineers combine software engineering expertise and machine learning. Learn more about what a machine learning engineer is in our video.

Engineering 59

Engineering Software Engineer Software Engineering Data Warehouse 59