Marc Lamberti

SEPTEMBER 22, 2023

The question is, How can we get the accuracy of each model in the task choosing_model to pick the best one? If you want to learn more about the differences between JSON/Pickle, click here. To access XComs, go to the user interface, then Admin and XComs. That value must be serializable in JSON or pickable. Time to practice.

MySQL 246

MySQL Data Pipeline Database Python 246

The Pragmatic Engineer

JUNE 22, 2023

In this article, we cover one out of five topics from today’s subscriber-only The Scoop issue. To get full issues twice a week, subscribe here. We anticipate that people on this team will remain on this team till the transition is complete - we anticipate this to take 9 months.

Google Cloud 234

Google Cloud Media Cloud Engineering 234

Christophe Blefari

MARCH 22, 2024

I don't go into too much detail about the magic of Data News, but every Friday is the same. At first, I'm: oh s**t, here we go again and 10 minutes later I'm lost in reading the content and picking too many articles to fit into a thousand word edition. I write in one-go starting from top to bottom.

Electronics 130

Electronics Media Data Python 130

The Pragmatic Engineer

AUGUST 3, 2023

We cover one out of five topics in today’s subscriber-only The Pulse issue. If you’re not yet a full subscriber, you missed this week’s deepdive: The 2023 tech market, as seen by hiring managers. To get full newsletters twice a week, subscribe here. We will announce more information in Q3 2023.”

Metadata 159

Metadata Engineering Building Technology 159

The Pragmatic Engineer

MARCH 3, 2023

We cover one out of five topics from The Scoop #39 , published two weeks ago, 23 February. To get full newsletters twice a week, subscribe here. Here are some of the results, visualized. Here are some of the results, visualized. So we took October’s data points and compared them with January’s.

Recruitment 193

Recruitment Software Engineer Software Engineering Finance 193

Christophe Blefari

DECEMBER 31, 2023

Hi, it's been a while since I last posted something here. To be honest, everything's been going well, but I've found it hard to find time to write among other things. However, some excellent articles have been written and I want to end 2023 with one last big wrap on these December articles. Yeah, you know.

Data 100

Data Cloud Storage Datasets Python 100

Marc Lamberti

SEPTEMBER 24, 2023

If you are new to Airflow, check out my course here; you will get it at a special discount. Use Case To better illustrate a concept, let’s begin with a use case: In this tutorial, we will use taskgroups to organize the giant DAG above. However, I wouldn’t go beyond three levels. Amazing, isn’t it? I’m kidding.

Coding 130

Coding Python Process Data Pipeline 130

The Pragmatic Engineer

OCTOBER 27, 2022

The observations on how Big Tech hiring will slow down have since been validated, with Meta not only laying off in November, but also rescinding offers in January 2023, and Amazon doing the same. In the latest issue , I deliberately did not go deep into recent layoffs, even though there have been some.

IT 130

IT Software Engineer Software Engineering Recruitment 130

Snowflake

NOVEMBER 15, 2023

BUILD 2023 is where AI gets real. We have more demos and hands-on virtual labs than ever before—and you won’t find a bunch of slideware here. Here are five reasons why this year’s global virtual summit is a must-attend: 1. Streamlit in the spotlight You asked, we listened. Let’s go build!

Building 112

Building Accessible Accessibility IT 112

Marc Lamberti

SEPTEMBER 29, 2023

Fasten your seatbelt, take a deep breath, and let’s go For a complete hands-on introduction to Apache Airflow, here is a 6-hour course at a discount. A data pipeline is nothing more than inputs that go through different steps executed in a specific order to produce outputs. For this tutorial, we will use the Astro CLI.

Data Pipeline 130

Data Pipeline Python AWS Project 130

The Pragmatic Engineer

AUGUST 10, 2023

In this article, we cover one out of five topics from today’s subscriber-only The Pulse issue. To get full issues twice a week, subscribe here. Before we start: plenty of people who subscribe to the newsletter utilize a learning and development budget their company provides. Read the full The Pulse here.

Retail 171

Retail Utilities Software Engineer Software Engineering 171

The Pragmatic Engineer

AUGUST 10, 2023

In this article, we cover one out of five topics from today’s subscriber-only The Pulse issue. To get full issues twice a week, subscribe here. Before we start: plenty of people who subscribe to the newsletter utilize a learning and development budget their company provides. Read the full The Pulse here.

Engineering 189

Engineering Recruitment Retail Software Engineer 189

Towards Data Science

JULY 10, 2023

This underscores the importance of thinking carefully about how we design models, not just on our own in dbt, but collectively with all of our stakeholders, to make sure the model is accurate and effective , and we don’t waste our time building each model 4–5 times before its useful. See the full canvas here. Image by author.

Designing 98

Designing Electronics Coding Engineering 98

Marc Lamberti

OCTOBER 1, 2023

By the way, if you are new to Airflow, check my course here , at a special discount. Hoping without delay, but we will come back to this later. Airflow brings many sensors, here is a non-exhaustive list of the most commonly used: The FileSensor : Waits for a file or folder to land in a filesystem. It is as simple as that.

Data Pipeline 243

Data Pipeline SQL Algorithm Coding 243

Data Engineering Weekly

NOVEMBER 26, 2023

Wow, can you believe it's almost time to wave goodbye 👋 to 2023? This year still feels like a fresh newbie, especially since we just bounced back from the pandemic rollercoaster 🎢 of last year. But here's a thought - why not look back to uncover the patterns? Who knows what insights we might uncover?

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Precisely

JANUARY 27, 2023

Data plays a central role here. The problem is that we don’t correlate the different levels of the data. For example, we get a retention report that says our retention is going up or down, and we get a CSAT report that says our CSATs are going up or down, but we don’t connect the dots.”

High Quality Data 105

High Quality Data Data Governance Government Banking 105

Christophe Blefari

FEBRUARY 3, 2024

hey ( credits ) Hello here, this is Christophe from Amsterdam. Being able to travel by train from Berlin to Paris to Amsterdam while going to the west of France for a lecture in a week is something truly awesome. But what problem are we trying to solve? In fact, I think we're trying to solve two different problems.

MongoDB 130

MongoDB SQL Data Data Warehouse 130

The Pragmatic Engineer

MARCH 30, 2023

Originally published on 30 March 2023. We cover one out of five topics in today’s subscriber-only The Scoop issue. To get full issues twice a week, subscribe here. in funding before going public in 2019, at a value of $29B. 👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter.

Software Engineer 286

Software Engineer Software Engineering Retail Media 286

Christophe Blefari

MARCH 4, 2023

I'll try to think about it in the following weeks to understand where I go for the third year of the newsletter and the blog. Here’s why your efforts to extract value from data are going nowhere — If data science is “making data useful,” then data engineering is “making data usable.”

Machine Learning 130

Machine Learning AWS Data Data Lake 130

The Pragmatic Engineer

MARCH 13, 2023

In case you somehow missed it: we went through the fastest bank run in history, in an event that impacted about half of all VC-funded startups in the US and UK. Deposits in Silicon Valley Bank, 1991-2023. Thursday: Bank run On Thursday, 9 March, we witnessed the fastest bank run on record. From here, things accelerated.

Banking 182

Banking Insurance Portfolio Media 182

Precisely

MAY 23, 2023

What are your goals in 2023, and which marketing trends can help you achieve them? I recently sat down with online publication, The CMO, to discuss five marketing trends to know about in 2023. Here’s a recap of what we covered: Top 5 Marketing Trends to Watch 1. Data integrity, again, has a critical role to play here.

Data Integration 98

Data Integration Technology Process IT 98

The Pragmatic Engineer

APRIL 6, 2023

We cover one out of five topics in today’s subscriber-only The Scoop issue. To get full newsletters twice a week, subscribe here. The site was launched in Januart 2023, shortly after California's pay transparency law went live. We use React on the frontend, Node.js We put the jobs data into Amazon S3.

Software Engineer 186

Software Engineer Software Engineering Datasets Database 186

ThoughtSpot

DECEMBER 19, 2023

2023 has been a year of breakthrough innovation for many, and a deer-in-headlights moment for others. Taking a page from last year’s book, we decided to look back on the top episodes of the year. Below, we break down the six episodes with the most unique listeners, sharing highlights and listener favorites from season four.

Entertainment 97

Entertainment Data Big Data Banking 97

Christophe Blefari

MAY 19, 2023

TWO YEARS — HAPPY BIRTHDAY 👋 Here is a special edition for me. I already told the story in Robin's podcast , here is a written version. We were a team of 6 people and our goal was to build the data platform for the company. We had set a goal: no broken Airflow pipelines in a 30-day sliding window.

Data 130

Data Data Engineering Data Engineer Hadoop 130

Snowflake

SEPTEMBER 12, 2023

The first annual Data Cloud Industry Day is here! Data Cloud Industry Day 2023 is a free virtual event on September 28, 2023, dedicated to what’s possible for you and your industry in the world of data. We know budgets are tight and we wouldn’t want anybody to miss out on learning how to use data to transform their organization.

Cloud 91

Cloud Manufacturing Retail Healthcare 91

The Pragmatic Engineer

JULY 13, 2023

We cover one out of six topics in today’s subscriber-only The Pulse issue. To get the full issues, twice a week, subscribe here. Before we start, a small change. Image source: Bicycling magazine This change made a major difference : “In 2015 we began shipping our bikes to the USA. And we were right.

Google Cloud 152

Google Cloud Retail Manufacturing Cloud 152

The Pragmatic Engineer

OCTOBER 10, 2022

’ A few days ago, Business Insider reported that Meta is doing “quiet layoffs,” which could mean as much as 15% of employees being let go. Here is what I have confirmed, though: 1. During my conversations, few bring this up early, but when we get to it, the speculation starts. Will Meta do layoffs?

Engineering 130

Engineering Software Engineer Software Engineering Recruitment 130

Christophe Blefari

MAY 18, 2023

As I often do I've overlooked the 70 presentations and here a medley of what I've liked. Data Council 2023 YouTube playlist My personal selection If you had only 3 videos to watch it should be the 3 following: Malloy an experimental language — This is my favourite talk. Formulas have metrics, relationships and weights.

Data 130

Data BI Consulting Data Science 130

Snowflake

JANUARY 31, 2024

Here are a few examples: “Snowflake Native Apps empower software service providers like MDO to seamlessly deploy applications directly within Snowflake,”” says Patrick Howerter, Co-Founder and CEO of My Data Outlet (MDO). Looking ahead, we project significant growth in 2024.” Check out MDO on Snowflake Marketplace. “We

AWS 100

AWS Government Programming Data Science 100

Knowledge Hut

NOVEMBER 16, 2023

So if you are wondering what is the average AWS certification salary in 2023, then this article can help you out. Since its popularity as a cloud computing service is constantly on the rise even in 2023, Amazon Web Services salary is something you can always rely on. But if you have an AWS certification, it can go higher.

AWS 98

AWS Amazon Web Services Cloud Computing Certification 98

Edureka

APRIL 19, 2024

In this blog post, we’ll delve into the fascinating world of Midjourney, exploring its capabilities, examining its impact on the artistic community, and providing insights into how this groundbreaking tool is reshaping the way we perceive and create visual art. May 3, 2023 V5.2 May 3, 2023 V5.2

Algorithm 52

Algorithm Deep Learning Machine Learning Media 52

The Pragmatic Engineer

FEBRUARY 2, 2023

Originally published 2 February 2023. We cover one out of seven topics in today’s subscriber-only The Scoop issue. To get this newsletter every week, subscribe here. Those going into the offices found themselves cut off from other teams, but also struggling for space.

IT 143

IT Software Engineer Software Engineering Media 143

Christophe Blefari

MARCH 24, 2023

📺 Watch the full replay Here are my takeaways about the event: Mage and Kestra have been both developed with Airflow flaws in mind, especially about deployment complexity, reusability and data sharing between tasks. Facing the reality, even if worker management seems easier in Mage, the deployment is not yet ready to go.

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

The Pragmatic Engineer

JANUARY 5, 2023

Originally published on 5 January 2023. We cover one out of seven topics in today’s subscriber-only The Scoop issue. To get this newsletter every week, subscribe here. Here’s the email CircleCI customers received late on Wednesday 4 January. We apologize for any disruption to your work.

Pipeline-centric 130

Pipeline-centric Database-centric Coding Accessible 130

Monte Carlo

DECEMBER 22, 2023

The team here at Monte Carlo has done our fair share of thinking — and writing — about generative AI over the last 12 months. Confidential | The Unvarnished Truth From 5 Anonymous Data Leaders At the end of 2023, what’s the AI verdict? As we hear all the time: garbage in, garbage out. Confidential, we did just that.

High Quality Data 52

High Quality Data Data Science Data Engineering Data Engineer 52

Snowflake

DECEMBER 11, 2023

According to Salesforce’s 8th annual State of Marketing Report , marketers cite measuring marketing ROI and attribution as their second biggest challenge—something that we here at Snowflake can certainly relate to. This means we can parse the URL to get insights ourselves.

Building 91

Building Cloud Media Algorithm 91

Scott Logic

JUNE 2, 2023

Every year at this time, I like to share my thoughts on the continuing relevance of Pride Month; you can read my posts here from 2021 and 2022. This Pride Month, we’re going to share insights from the Scott Logic team on what Pride and allyship mean to them, and why they value working in an inclusive environment. Happy Pride 2023.

Education 98

Education IT 98