Building a Custom PaaS (Part 1)

There is a certain comfort in pushing code to a managed platform like Heroku or Vercel. You run a command, a deployment process kicks off, and a few minutes later, your application is live on a secure, auto-generated URL.

Recently, while studying a system design diagram for a large-scale deployment architecture, it made me pause. Instead of just treating platforms like a black box, I wanted to understand exactly how they work under the hood. How does incoming traffic dynamically map to a container that was spun up just seconds ago?

I decided to build the core routing layer of my own Mini-PaaS from scratch. Here is the story of how I engineered the first version of this platform, why I eventually had to abandon my local Docker setup for Kubernetes, and the networking traps I fell into along the way.

The Architecture: How the Pieces Fit

To build a basic PaaS, you have to solve three distinct problems: intercepting the incoming web traffic, maintaining a map of where that traffic should go, and providing an API to update that map dynamically.

I split my architecture into three distinct components:

1. The Gateway (The Front Door): A Reverse Proxy built in C# using Microsoft's YARP (Yet Another Reverse Proxy). I chose C# because I am highly familiar with the .NET ecosystem, and YARP is excellent because it allows you to configure routing dynamically via code, rather than just static config files.

2. The Control Plane (The API): A Node.js/TypeScript Express server. This is the management layer where a new app deployment registers its target URL.

3. The Database (The Routing Map): A Redis instance. Because Redis lives entirely in memory, it is perfect for lightning-fast route lookups on every single web request.

How it works in practice: When I deploy a new application container, I make a POST request to the Node.js Control Plane. The API simply writes a key-value pair into Redis. In code, it looks something like this:

// Node.js API saving the route
await redisClient.set("app1.mydomain.com", "http://app-one-container:80");

When a user visits app1.mydomain.com, the request hits the C# Gateway. The Gateway reads the requested hostname, queries Redis for that specific key, retrieves the internal target URL, and proxies the traffic to the correct container.

The Limitation of Plain Docker

I initially built this entire setup using plain Docker containers on my laptop. It was a great proof of concept, but I quickly realized that standard Docker isn't enough to run a robust platform.

If my C# Gateway container crashed, it stayed dead. If I wanted to update an application without dropping traffic, I couldn't easily perform a rolling deployment. To make this a true platform capable of self-healing and real orchestration, I needed to step up to Kubernetes.

Migrating to Kubernetes was the right architectural choice, but it humbled me almost immediately.

Fixing the Database: Why Redis Needed a Persistent Volume

In my local Docker setup, Redis worked perfectly. But when I containerized it and pushed it into a Kubernetes Pod, I discovered a fundamental rule of orchestration: Pods are born to die.

Kubernetes Pods are completely ephemeral by default. When my Redis Pod was eventually recreated during testing, Kubernetes spun up a brand new, completely empty database. All my routing rules vanished instantly, and every app on the platform returned a 404 error.

To fix this, I had to learn how to attach permanent storage to temporary containers. I wrote a PersistentVolumeClaim (PVC) to request a dedicated storage drive outside the container's lifecycle:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

I updated my Redis Deployment YAML to mount this PVC and changed the Redis startup command to --appendonly yes, forcing it to write every route update to disk. Now, if the Redis Pod gets destroyed, the replacement Pod spins up, instantly mounts the 1Gi volume, reads the historical routes, and traffic keeps flowing without a single dropped packet.

The Networking Trap: Escaping Localhost

Once the database was stable, I moved on to containerizing the Node.js Control Plane and the C# Gateway.

Locally, all my apps communicated using localhost. My C# code looked something like this:

// Connecting to Redis locally
var multiplexer = ConnectionMultiplexer.Connect("localhost:6379");

When I deployed this into Kubernetes, my Gateway immediately went into a CrashLoopBackOff death spiral.

This was my biggest "Aha!" moment with Kubernetes networking. Inside a cluster, every Pod has its own isolated localhost. The Gateway was looking around its own empty network namespace for a Redis server that wasn't there.

I had to learn how Kubernetes Internal DNS works. Instead of relying on local IPs, you create a Kubernetes Service (essentially an internal load balancer) for your Pods. I updated all my connection strings to use the cluster's internal phonebook:

// Connecting to Redis via Kubernetes Internal DNS
var multiplexer = ConnectionMultiplexer.Connect("redis-service:6379");

Instantly, the cluster stabilized. The Gateway stopped crashing, located the Redis service, and began routing traffic properly.

I ran into a similar issue trying to test my Node.js API from my Mac. I used kubectl port-forward to drill a tunnel into the cluster to send a POST request, but the Node server rejected my connection. I realized that my Express app was defaulting to listening on 127.0.0.1 (internal only). Because the port-forwarded traffic was coming from outside the pod, I had to explicitly bind my Node server to 0.0.0.0 to accept the connections.

Looking Back at v1.0

Building this routing layer from scratch taught me significantly more about infrastructure than deploying to managed cloud providers ever did. I went from spinning up basic Docker containers to understanding core Kubernetes concepts:

• Self-Healing: Why CrashLoopBackOff happens and how K8s manages pod lifecycles.

• Internal DNS: How K8s Services abstract away dynamic IP addresses so pods can find each other reliably.

• State Management: Using PersistentVolumeClaims to decouple data from ephemeral compute instances.

Right now, my Mini-PaaS is a robust, resilient system. However, it’s not completely automated yet. To deploy a new app, I still have to manually hit my Node API to register the container's route in Redis.

A true PaaS should manage itself. So, for Part 2 of this series, I'll be upgrading my Node.js Control Plane into a native Kubernetes Controller. It will silently watch the cluster's API, and the second I deploy a new app, it will automatically read the YAML annotations, generate the route, and update Redis completely on its own without any manual intervention.

I've made the repository public here: https://github.com/dea1j/Mini-Paas.git