Self-hosted GitHub Actions with FireActions and Falcon

After once again running out of free GitHub action build minutes, I decided to revisit the self-hosted runners. I’ve been previously running both the daemon and the actions runner controller (ARC) on Kubernetes. Neither felt the right. On Kubernetes (without extra tweaks) you get the issues for nested Docker and isolation as by default the pods are just containers and not VMs. With the daemon option you also don’t really have isolation and would likely need to do extra work to cleanup the environment between builds.

FireActions runner

I knew FireCracker VMs could be a thing for this purpose. Light weight and quick to start. There are ways to run these under Kubernetes, but I settled on another option. Hostingner has an open source project called FireActions. With less than 200 GitHub stars it has not gotten much traction, but it does the work.

With FireActions you can run a pool of runners on your own hardware (or hosted VM). The builder images are Docker images, so it’s easy to build custom ones based on their supplied base image.

falconDev GHA Cache Server

By default also the self-hosted runners use the GitHub servers for caching. This obiously has it’s limits with the space and it does not make much sense to ship stuff like node_modules over the Internet. There are different options for solving the problem, but you don’t want to modify your actions scripts, some tricks need to be made. The caching APIs GitHub is using are not described, but developers have reverse engineered how they work. GHA Cache Server from falconDev provides the APIs.

GHA Cache Server comes with pluggable backend storage (filesystem, S3 compatible and Google Cloud Storage). It provides a HTTP API that matches the one GitHub provides. With small tweaks on the build image, you can direct the actions to use your local GHA Cache Server.

Distribution for Docker containers

One remaining piece in the puzzle is local cache for Docker images. Docker Hub has certain limits on pulling the images and of course it does not make much sense to keep on pulling the same stuff over and over again for each build. Since the FireActions virtual machines start from scratch every time, nothing gets saved on them. There are some options for pushing the Docker images to host machine. Instead, I decided to go with local installation of quite generically named CNCF Distribution.

In this case I’m using the Distribution just as proxy/cache for the Docker images. My final images are still getting published to GitHub container registry. Distribution is running under my local Kubernetes.

Setup

For this setup I have three old computers that I happened to have laying around. Even years old hardware is pretty ok when you compare it to average cloud VMs (old laptop - quad core CPU, 16GB memory, 512GB NVMe SSD…). Network wise it’s all easy. Everything works by opening connections from local to remote, so no need to worry about fixed IPs or firewall openings.

flowchart LR
    GITHUB["GitHub"]
    subgraph LOCAL["Local network"]
        direction LR

        subgraph BUILDER1["builder1"]
            FA1["FireActions"]
        end

        subgraph BUILDER2["builder2"]
            FA2["FireActions"]
        end

        subgraph SERVER1["server1"]
            subgraph K8S["Kubernetes"]
                CACHE["GHA Cache"]
                DIST["Distribution"]
            end
        end

        FA1 --> K8S
        FA2 --> K8S
    end

    DOCKERHUB["Docker Hub"]
    
    GITHUB <--> FA1
    GITHUB <--> FA2
    DIST --> DOCKERHUB

    linkStyle 2,3 marker-end:none

Typical problem for me with these homelab setups is that it’s nice and fun to do the setup. Then things work for a while, until some machine breaks and you loose everything. To solve this, I’m using Ansible for the configuration of the machines. This means there’s documentation always on Git and it will help to recreate things in case I loose a computer.

builder1 and 2 are running Ubuntu 26.04. Server1 doubles as desktop machine so that happens to run Mint. Since I just have single node Kubernetes, I’m running microk8s.

All the practical setup was done with Codex (5.6, Medium). I told it on high level what to do and let it sort the practicalities. It wrote and executed the Ansible scripts and debugged them. It is managing the Kubernetes setup via GitOps (with ssh access to the cluster to debug things).