My Interconnected Systems and Tools I Use

I feel like the best way to show that I have a solid understanding of the technology running most of the mordern world's digital systems, I'd walk you through a bit of the tools I use, how it is all set up, and the way I use it to stay on top of things in the digital world.

It is a bit of a web of interconnected systems, so let me start off simple.

Let's start with this site, for example. This site is a static site compiled with Elevnty from API accessed markdown data managed on a self-hosted Ghost installation running on my workstation at home. It is hosted in Cloudflare pages, and is re-compiled and re-deployed automatically any time my Eleventy code is changed or posts/pages are update from my ghost administration panel. My network at home is not public, but I do have an nginx proxy for the ghost API running at the same location as my mailserver that routes API requests during deployments to my machine at home using Wireguard.

This machine, is not a regular machine, and the reason why is I do not work on the machine directly. On this physical machine, I hosts Virtual Machines, and my primary VM, which I use as a workstation, has GPU and USB passthrough, which means the physical hardware is passed directly through to my virtual machine. Seeing as my GPU is outputting directly to the monitor, my workstation VM doesn't feel like a VM at all. What this means is the OS I work on, on a daily basis, is logically separate from the operating system that runs on my machine at the bare-metal level. This machine also has an onboard GPU built into my CPU that has an HDMI output. If I use displayport, I have my workstation, from which I can SSH around, including to the bare-metal host under this VM. From this workstation, I can also open virt-manager, connect to my host's hypervisor, and manage/use other VMs running on my host. For example when I work for a specific company or client, they each get their own VM which I can interact with in a window on my, you bet, VM workstation. From this workstation I can even run a Windows VM, if needed, although the only case I needed to do so lately was for using my tax preparation software, as I could not get it to work directly on Linux using WINE, unfortunately.

So workstation aside, which I also happen to use for gaming (although not nearly as much asI'd like), on my bare-metal host, I also host other services, like Mysql, Postgres, Gitea, Nextcloud, and Grafana. The best part of working this way is I can use a single powerful computer, maintain uptime for the services that require it, and still have an OS/workstation completely independent from the underlying OS. This allows me to manage my host and my workstation independently, which also calls for a lot less reboots of my host seeing as I can reboot and shutdown my workstation OS and interact with it in ways completely unrelated to the other services working on this machine. I also have a laptop, but it is pretty bare bones and mostly gets used if/when I am on the road.

Moving on to the cloud, to keep things logically separated, both my host and workstation VM are connected to the same Wireguard VPN network, and all their traffic is routed through this VPN tunnel to a datacenter in west-Montreal. I'm also extra paranoid and set up an Encrypted DNS resolver and cache on this instance, and all connected peers use this encrypted DNS service to make DNS calls, so that my ISP, or any government either, cannot see the DNS calls made by any of my systems and thereby not know what sites I am visiting. Sure, HTTPS encrypts your HTTP traffic, but ISPs and governments can still see your unecrypted DNS calls...

As they are all connected and attributed internal domain names, it allows me to access services and SSH of my various cloud services internally, as most things are set up only to allow requests from an internal Wireguard peer client IP address. That way I do not even need to have any ports open and do not need to deal with any routing or firewall issues at the networking level. In fact, some of my services are not even open to the outside world at all unless given explicit access through Cloudflare Zero Trust Access Rules. The instance on which I run this VPN is, you bet, also free (thanks, Oracle Free tier)!

Other than domain registrations, the only thing I currently pay for is my mail/proxy+misc server. This is a virtual server hosted on Hetzner, and I might migrate everything over to Oracle later this year if their free tier proves to be enough compute for my personal needs. Hosting your own mail server is not as daunting a task as some might make it out to be. Generally speaking, even if big email providers will blacklist some cloud provider's IP ranges, I have made sure to get around this by using Sendgrid as my outgoing SMTP server for my personal and professional emails, oh and yep, also free!

Sounds pretty complex, so how do I manage to stay on top of all this? Well, for one thing. It's all just straight Linux. Other than cloudflare pages and its connection with Gitlab, which I automatically copy from my self-hosted Gitea, everything I have mentioned is configured and running not only on any Linux, but as someone who looks at things from a system's engineering point of view, probably what I would consider to be the best distribution geared towards managing complex distributed systems: NixOS

Why NixOS?

Declarative configuration-as-code: no more working with config files for seperate services, and trying to deal with the OS level configuration stuff via the command line. NixOS's configuration can live in a git repo. Not only that, but through the use of advanced Nix topics like Flakes and Modules, multiple systems can even share a repo, and share different components of their nix configurations. This means I only have to work in one repo centrally manage all of my system's setups. The code is also reproducible. With nix, if you take the configuration code of one system, copy and rebuild the OS using the copied code from another, that OS and its packages will be set and configured 100% exactly like it was elsewhere. You can even configure services and packages down to the commit level,  and even lock packages to certain commits, meaning you can chose not only which versions or packages you are building from, but also which commit you are pulling from to get an exact codebase stuck at a specific point in time.

Atomic and reliable: Each time you make a change to a system's Nix configuration, that system essentially needs to be rebuilt. This process first evaluates your configuration file against the Nixpkgs github repo, and verifies the logic of your configuration in order to successfully configure the OS and packages/services before switching to this configuration. If the configuration involves kernel changes or module changes related to the hardware, you can also tell it to apply the new configuration only after a reboot, as well. Here's one of the best parts though; nothing is overwritten. The way NixOS is set up, is that every time you rebuild your system, you are creating essentially a new branch of your operating system, and it downloads and compiles everything that is needed for that specific branch, without removing the previous generation of your NixOS build.

So not only is there logic verifying your configuration codebase before making any changes to a running system, it also stores these previous generations and their packages/configurations, giving you an easy way to roll-back to a previous generation if for some reason your new configuration complied, but something was not quite working as intented.

No more trying to deal with distro differences, package manager differences, and trying to coordinate and maintain packages and updates and configurations over time, and also significantly less chances of introducing breaking changes. I haven't gone as far as one could, but realistically if you were so inclined, you could CI/CD your entire OS, including service tests using structured data in order to validate OS and service level changes before they are applied to a production machine.

"Think herd, not pets"? How about "think zen garden, not herd"? I'm claiming that quote, ha.