Menu

Subscribe

An educational side project

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of four topics in today’s subscriber-only The Scoop issue. If you’re not yet a full subscriber, you missed this week’s deep-dive on Agoda’s private cloud setup. To get the full issues, twice a week, subscribe here.

I’d like to share a story about an educational side project which could prove fruitful for a software engineer who’s seeking a new job.

Juraj Majerik is an Amsterdam-based software engineer who decided to improve his applied knowledge of containerization, multiprocessing and observability. So he built a “clone” of a ridesharing app like Uber or Bolt: the app simulates riders requesting trips and drivers picking them up, then repeats this all over again. See it in action, here:

A screenshot of the interactive “Rides” app. See the full app here
A screenshot of the interactive “Rides” app. See the full app here

What grabbed my attention was how this app is much more than just a simulation. Juraj included system monitoring parts which monitor the server’s capacity he runs the app on:

The monitoring page on the Rides app
The monitoring page on the Rides app

And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used:

The systems design diagram for the Rides application
The systems design diagram for the Rides application

The app uses:

  • Node.js for the simulation engine
  • Go on the backend
  • PostgreSQL for the data layer
  • React and TypeScript on the frontend
  • Prometheus and Grafana for monitoring and observability

And if you were wondering how all of this was built, Juraj documented his process in an incredible, 34-part blog series. You can read this here.

This “Uber clone” offers a nice blueprint on how to tackle a complicated project. Thanks to the detailed documentation of Juraj’s progress, we can reconstruct how he built the project. This is educational, as it shows one possible way to tackle a complicated project, and one for which the creator could only devote time to on the side, and at weekends. Here’s how Juraj approached the undertaking:

Phase 0: make a plan (Oct). Instead of starting with coding, Juraj kicked off by sketching. He sketched out what he wanted the final product to look like:

The sketch Juraj made, before starting any coding
The sketch Juraj made, before starting any coding

And he sketched how he envisioned the observability part to work:

The sketch of the monitoring system
The sketch of the monitoring system

Phase 1: Infrastructure (October-November).

Before diving into coding, Juraj set up the infrastructure. I assume he did this to familiarize himself with infrastructure which he hasn’t necessarily used in production before. This side project offered a good opportunity to try it out.

  1. Set up a server on DigitalOcean (a virtual machine with 1GB memory and 25GB of disk space)
  2. Set up the domain, and configure the DNS
  3. Set up users on the server
  4. Set up SSH keys for more secure authentication
  5. Install Go on the server, which will power much of the backend
  6. Set up a HTTP server using Go
  7. Deploy for the first time, doing so manually, for now
  8. Enable HTTPS by registering a certificate
  9. Utilize environment variables to be used for configuration, instead of hardcoding configuration or secrets into the source code. Hardcoding secrets in production is poor practice. We covered how Stack Overflow learned this the hard way, a few months back.
  10. Serving a web page. This is the point where the app reached the “Hello world!” stage
  11. Improve deploys. Change deploys to be a one-command process, instead of multiple steps

See blog posts #1-11 for details on all the steps.

Phase 2: some business logic, and more infra (December-January)

  1. Draw a map using JavaScript to map onto an SVG format
  2. Build a graph and traverse it. Here’s a video of Juraj demonstrating this traversal.
  3. Set up Docker to package the application into containers
  4. Use Docker in production, and modify the deploy script to deploy using Docker
  5. Set up PostgreSQL to persist data on drivers, riders and trips
  6. Connect the Go backend to the PostgreSQL database
  7. Connect the backend and the database containers
  8. Docker Compose in production

See blog posts #12-20 for details on all the steps.

Phase 3: a basic UX (February)

  1. Migrate to React
  2. Design a car, from a vector image
  3. Move a car with animations. Including adding unit tests.
  4. Turn a car using rotations
  5. Tidying up the project: refactoring the files now, that the project’s structure is more clear
  6. Server-generated data
  7. Animation fixes

See blog posts #21-27 for details on all the steps.

Phase 4: “hardcore” coding (March)

This was the first phase where Juraj did no more infrastructure work, and focused only on “pure” business logic.

  • Simulation engine: this component simulates the behavior of drivers and customers
  • Multiprocessing for Node.js, to avoid blocking the event loop
  • Generating destinations, and making sure the start point is not too close to the destination
  • Matching drivers with customers: doing this similarly to how a service like Uber does
  • Route planner: tell the driver which route to choose to collect a  customer

See blog posts #28-32 for details on all the steps.

Phase 5: finalizing and monitoring (April)

  • Finalizing the simulation
  • Setting up monitoring & logging

See blog posts #33 & 34 for details of all the steps.

Standoud things about this side project

There are several things that I’ll highlight about this side project.

1. Persistence. Juraj built this project on the side, between October 2022 and April 2023, which is 7 months.

2. Documenting the steps. Every time Juraj made progress, he documented what he did, and how. This likely helped him to learn, and it also helps others wanting to understand, too.

3. Incremental progress. The project looks like a tough one to build from scratch on the side. But looking at the incremental steps, it is not nearly so daunting. Here are a few of the steps, taken directly from the blog:

4. You won’t get the layout of a project right, the first time. So refactor! Juraj was about 70% done with the project, when he went back to refactor the structure of the project. This wasn’t for the lack of planning: but because as you set up new infrastructure, things turn out a bit different than you expected.

This is just the way of software engineering, and there’s nothing to be embarrassed about it. As you learn more about the project, don’t be afraid to go back and change the project structure – or do other refactorings – to help your work, going forward. See the project structure Juraj set up at the end of Phase 3.

5. Infrastructure is important, and setting it up right can be a consuming task! In the first 3 phases of the project, Juraj spent a lot of time on infrastructure setup. It was only in Phase 4 that he was able to focus “purely” on the coding part.

I really like how this project showcases just how much time can go into infrastructure setup. At companies with dedicated platform teams, those teams take exactly this kind of load off other teams building greenfield projects.

Both as an engineer, and especially as an engineering manager, don’t forget there’s a real cost to setting up and then maintaining infrastructure. Much infrastructure work is invisible as it does not involve commits, and most engineers won’t document the time they spend on these tasks, like Juraj has. But this is work that still needs to be done!

Here are the learnings Juraj had with this project. I reached out to Juraj to ask how this project helped him. His answer:

“I've touched on many topics I haven't been exposed to in my day-to-day job, such as server configuration, setting up a deployment pipeline, animations in JavaScript, or using Docker.

I decided to get better at algorithms and data structures a few months ago, and this project complemented it nicely. For example, I implemented the map and its associated methods (e.g. path-finding) from scratch.

I also wanted to get better at setting up a full-stack project from scratch and understand why certain technologies are used. Docker is a good example - I didn't use it because I ‘wanted to’, but because the necessity for it arose. I was surprised how much time all of the infra work took me before I was able to start with the actual simulation logic!”

And how did Juraj find the time to work this much on the project?

“While I was at my previous job, I dedicated 1-2 hours a day, usually after work. After leaving in March due to a layoff, I decided not to start my job search immediately, but spent some more time finishing and polishing this project. That's when I really ramped up my effort on it and started seeing a lot of progress.”

If you are thinking of doing a side project with the goal of learning new technologies – and, to also be able to share those learnings, and show off the side project – I can recommend taking inspiration from this methodological approach Juraj took. Of course: don’t copy the exact approach, as-is. But planning first, documenting steps, and building a “production simulation” are all ideas that you could use in your own side projects as well. If you do: your side project will surely stand out from many of the other ones.

Your own projects often seem less impressive to you than they are to others. I was impressed by the implementation and attention to detail for this project – for example, going the extra mile for adding in monitoring for the server components: typical for a production service, but rarely seen in a side project. When I shared that I’m impressed with the execution with Juraj, his response was surprising, as he told me:

“Working on this side project every day, it no longer feels that impressive to me (as I think is common with software projects). But having read your fresh perspective, it certainly gives a lot of encouragement!”

When it’s day-to-day work, most engineers don’t think they’re doing anything complex, or special. And, in all fairness: no single step in Juraj’s project was complex. The complexity comes from the combination of simple steps. This is the beauty of software engineering: that everything that seems complex can be broken down to simple to understand – and simple to implement – steps.

Thanks a lot to Juraj for sharing this project with me. View the complete source code here. If you are hiring for full-stack engineers, Juraj is on the market – at least for now! Connect with him via his website, on LinkedIn or on Twitter.

This was one out of the four topics covered in this week’s The Scoop. A lot of what I share in The Scoop is exclusive to this publication, meaning it’s not been covered in any other media outlet before and you’re the first to read about it.

The full The Scoop edition additionally covers:

  • Why are many Snap employees selling their stock as soon as it vests, and not a day later? Snap is a very strange publicly traded company, where shareholders have precisely zero votes. I’ve talked with engineers and discovered a surprising level of distrust within the company. Is this tied to the governance model, or something else? Exclusive.
  • Analyzing the split of cuts at Lyft. How much were software engineers impacted by ride-hailing service Lyft’s large-scale cuts? I went through data based on Lyft’s talent directory; it looks like tech functions were hit harder than other areas. Exclusive.
  • Robinhood is no longer a remote-first company. Few companies were as vocal about becoming a remote-first company than Robinhood. But less than 18 months later, the company has made a u-turn. What can founders and leaders take from this avoidable policy reversal? Analysis.

Read the full The Scoop here.

Featured Pragmatic Engineer Jobs

The above jobs score at least 9/12 on The Pragmatic Engineer Test. Browse more senior engineer and engineering leadership roles with great engineering cultures, or add your own on The Pragmatic Engineer Job board and apply to join The Pragmatic Engineer Talent Collective.