Being an engineer at heart, each time I see a company write about their tech stack, I brew a fresh cup of coffee, sit back and enjoy reading the newfound little treat.

There’s just something fascinating about getting to know what’s under the hood of other people’s businesses. It’s like gossip, but about software.

A couple of months ago I started working on yet another private analytics service, a project which has gone through numerous iterations, and I feel lucky that thousands of websites have already integrated with it, even though it's still in the early stages.

As a self-funded solo founder, I believe that focusing on automation is how I've been able to provide a reliable service to customers from more than 80 countries, and continue to ship new features on a weekly basis.

I try to leverage the tools I already know best, as I understand how they work, but even more importantly I understand in which cases they won't help me.

That’s why, in the same spirit as Jake Lazaroff’s Tools and Services I Use to Run My SaaS, I thought it’s now my turn to do a short write up of the technologies I’m using to run my startup.

It's important to note that even though the list seems long, and there's lots of unconventional choices here, I actually spend very little time on infrastructure. A couple of hours per month, if at all. But this is not a recipe. What worked for me, may not work for you. Consider your own circumstances, and leverage what you already know best.

Languages

Over the years I have added many programming languages to my toolbelt, but for solo projects I have converged to two in particular that strike a good balance of productivity and reliability.

• Python: Most of the backend code is in Python. Which has enabled me to ship features incredibly fast. Additionally, I use mypy for optional type hints, which helps keep the codebase manageable.

• Typescript: I used to avoid working on the frontend as much as I could. That is until I discovered Typescript about 4 years ago. It just makes the whole experience a lot better, and I now use it for all my projects together with React.

Frameworks and libraries

This list could have been huge, as I stand on the shoulders of giants who have published the vast amount of open-source code which I rely on. But I'd like to highlight only a handful due to their major role in the stack:

• Django: It's like a superpower for solo developers. The longer you work in this industry, the more you appreciate not having to reinvent the wheel for the 100th time. A monolithic framework can get you really, really far. To me, it's about predictable software that's fast in every way that matters. In case you're interested, I talk more about this topic on Choose Boring Technology.
• React: The web app for the dashboards is built using React + Webpack. After using Angular for a long time, I switched to React because it's just a pluggable view layer that doesn't get in the way. I use the fantastic django-react-templatetags to embed the React components in my Django templates.
• NextJS: I use it for the landing pages, documentation and the blog which you are currently reading. It enables me to re-use various React components, and still reap the performance and SEO benefits of a statically generated site.
• Celery: I use it for any kind of background/scheduled tasks. It does have a learning curve for more advanced use-cases, but it's quite reliable once you understand how it works, and more importantly when it fails.
• Bootstrap 4: I built a custom theme on top of Bootstrap. It has saved me a lot of time, and there's lots of documentation around it. That's why I picked it.

Databases

I originally stored all data in a single SQLite database, doing backups meant making a copy of this file to an object storage like S3. At the time, it was more than enough for the small sites I tested Panelbear with. But as I added more features and websites, I needed more specialized software to support those features:

• Clickhouse: I believe this is one of those technologies that over time will become ubiquitous. It's honestly a fantastic piece of software that enabled me to build features that initially seemed impossible on low-cost hardware. I do intend to write a future blog post on some lessons learned from running Clickhouse on Kubernetes. So stay tuned!
• PostgreSQL: My go-to relational database. Sane defaults, battle-tested, and deeply integrated with Django. For Panelbear, I use it for all application data that is not analytics related. For the analytics data, I instead wrote a simple interface for querying Clickhouse within Django.
• Redis: I use it for many things: caching, rate-limiting, as a task queue, and as a key/value store with TTL for various features. Rock-solid, and great documentation.

Deployment

I treat my infrastructure as cattle instead of pets, things like servers and clusters are meant to come and go. So if one server gets "sick", I just replace it with another one. That means everything is described as code in a git repo, and I do not change things by SSH'ing into the servers. You can think of it like a template to clone my entire infrastructure with one command into any AWS region/environment.

This also helps me in case of disaster recovery. I just run a few commands, and some minutes later my stack has been re-created. This was particularly useful when I moved from DigitalOcean, to Linode, and recently to AWS. Everything is described in code, so it's easy to keep track of what components I own, even years later (all companies have some AWS IAM policy or VPC subnet lurking around which was created via clicky-clicky on the UI, and now everyone depends on it).

• Terraform: I manage most of my cloud infrastructure with Terraform. Things like EKS clusters, S3 buckets, roles, and RDS instances are declared in my Terraform manifests. The state is synced to an encrypted S3 bucket to avoid getting in trouble in case something happens to my development laptop.
• Docker: I build everything as Docker images. Even stateful components like Clickhouse or Redis are packaged and shipped as Docker containers to my cluster. It also makes my stack very portable, as I can run it anywhere I can run Docker.
• Kubernetes: Allowed me to simplify the operational aspects tremendously. However, I wouldn’t bindly recommend it to everyone, as I already felt comfortable working with it after having the pleasure of putting down multiple production fires for my employer over the years. I also rely on managed offerings, which helps reduce the burden too.
• GitHub Actions: Normally I’d use CircleCI in the past (which is also great), but for this project I prefer to use GitHub Actions as it removes yet another service which needs to have access to my repositories, and deployment secrets. However, CircleCI has plenty of good features, and I still recommend it.

Infrastructure

I started in a single $5/mo instance in DigitalOcean, then moved to the managed Kubernetes offering as I was reinventing the wheel for a lot of things Kubernetes already gives me out of the box (service discovery, TLS certs, load balancing, log rotation, rollout, scaling, fault-tolerance, among others).

Unfortunately, I had reliability issues with DigitalOcean's Kubernetes offering, even on larger instances. The cluster API would often go down randomly and no longer recover, this disrupted a lot of cluster services including the load balancer, which translated into downtime for me. I had to create a new cluster each time this happened, and while Terraform made it trivial, this was not something that inspired a lot of confidence about their managed service. I suspect their control plane was underprovisioned, which would be kind of understandable given the price tag.

Unfortunately I was not able to resolve the issue after several weeks. That's why I decided to move to Linode, and had exactly 0 problems during the 1.5 month-long honeymoon that followed.

However, I recently moved once again, this time to AWS due to a pretty good deal I received. It also enabled me to use managed services like RDS to offload managing PostgreSQL, which is a big plus. What made all these migrations relatively easy, was that all my infrastructure was described via Terraform and Kubernetes manifests. The migrations essentially consisted of an evening, some tea, and patience. But that's for another post.

• AWS: Predictable, and lots of managed services. However, I use it at my full-time job, so I didn't have to spend too much time figuring things out. The main services I use are EKS, ELB, S3, RDS, IAM and private VPCs. I might also add Cloudfront and Kinesis in the future.
• Cloudflare: I mainly use it for DDoS protection, serving DNS, and offloading edge caching of various static assets (currently shaves off 80% of the egress charges from AWS - their bandwidth pricing is insane!).
• Let’s Encrypt: Free SSL certificate authority. I use cert-manager in my Kubernetes cluster to automatically issue and renew certificates based on my ingress rules.
• Namecheap: My domain name registrar of choice. Allows MFA for login which is an important security feature. Unlike other registrars, they haven't surprised me with an expensive renewal every few years. I like them.

Kubernetes components

The following components automate most of the devops work for me. I use several others too, but some of the main ones I use are:

• ingress-nginx: Rock-solid ingress controller for Kubernetes using NGINX as a reverse proxy, and load balancer. Sits behind the NLB which controls ingress to the cluster nodes.
• cert-manager: Automatically issue/renew TLS certs as defined in my ingress rules.
• external-dns: Synchronizes exposed Kubernetes Services and Ingresses with DNS providers (such as Cloudflare).
• prometheus-operator: Automatically monitors most of my services, and exposes dashboards via Grafana.
• flux: GitOps way to do continuous delivery in Kubernetes. Basically pulls and deploys new Docker images when I release them.

CLI tools

There’s plenty here, but frequently used include:

• kubectl: To interact with the Kubernetes cluster to watch logs, pods and services, SSH into a running container, and so on.
• stern: Multi pod log tailing for Kubernetes. Really handy.
• htop: Interactive system process viewer. Better than “top” if you ask me.
• cURL: Issue HTTP requests locally, inspect headers.
• HTTPie: Like cURL, but simpler for JSON APIs.
• hey: Load testing HTTP endpoints. Gives a nice latency distribution summary.

Monitoring

• Prometheus: Efficient storage of time series data for monitoring. Tracks all the cluster and app metrics. It was a lot cheaper than using Cloudwatch for app metrics.
• Grafana: Nice dashboards for the Prometheus monitoring data. All dashboards are described in JSON files and versioned in the git repo.
• Sentry: Application exception monitoring and aggregation. Notifies when there are unhandled errors with additional metadata.
• Loki: Log aggregation system inspired by Prometheus. It’s bundled with the prometheus-operator, and helps me search logs across the cluster.

Email

• Fastmail: My choice of business email. Good, and reliable.
• Postmark: I use it for transactional emails (email verification, weekly reports, login security alerts, password reset, and so on). Their email delivery rates are great, and the tooling/mobile app is top-notch.

Development

• GitHub: Source code hosting and versioning.
• PyCharm: Probably the best IDE for Python. I can refactor and navigate the entire codebase with ease, and not just individual files. Works really well, even with large, dynamic codebases.
• VS Code: Great for Typescript/React, and as a general purpose code editor.
• Poetry: Python packaging and dependency management with lock files.
• Yarn: Fast JS dependency management with local caching.
• Invoked: I wrap all my codebase tasks in invoked commands. For example inv build will prepare static assets, package the frontend/backend distribution, and build the docker image. This way, I can run locally the same commands that I would run on CI, in case it's needed.

Other

• Panelbear Analytics: Of course what better tool to track Panelbear's website analytics than Panelbear itself :) The benefits of dogfooding are real, as I am my own customer.
• Healthchecks.io: Notifies me via email/whatsapp when a scheduled job doesn't run. It's also a bootstrapped SaaS, and I'm very happy to recommend it as I've used it for several years.
• Trello: I use it to keep track of issues/requests/ideas and what-not.
• Figma: Replaced Sketch as my go-to tool for making quick mockups, banners, and illustrations for the landing pages.