Leveling Up Your Git Server

Leveling Up Your Git Server - Sharing Repos with a Friend

Justin Wernick <[email protected]>
Curly Tail, Curly Braces
2023-06-02

I've previously written about how it's completely possible to set up your own Git server, and regain control over where your data goes. At the end of the previous article, we had a basic Git server, with a single user, and SSH access to push and pull repos. If you haven't already, I recommend reading that article first, because this one builds on it.

It's nice to have a Git server all to yourself, but it's a bit antisocial.

In this article, I'd like to take it to the next level!

I'm going to cover:

• Scripting server setup and changes with Ansible.

• Multiple users, and sharing repos, using Linux groups and file permissions.

• Restricting what users can do when they connect over SSH, and making it easier to use, using Shackle Shell.

Disclaimer: I'm not a site reliability engineering specialist. I'm just a developer running his own Git server to see what he learns from it, and then writing up what works. If you're following along here, please use your own discretion when it comes to things like reliability and security.

Better tooling for server management: Ansible

In the previous article, I shared shell commands that you can type into a terminal on your server to do things like adding users. This is fine for starting out, but isn't very repeatable. The problem is that it's very difficult to keep track of what exactly you did to set up a server. If you ever want to set up something similar on a different server, or make a small change to how things run, it's difficult to do so.

Luckily, we have a standard software engineering approach to the problem of repeatability: do the setup with a script! Check the script into version control (pushed to a Git server maybe?), and you can easily track the changes you made to your server over time.

You can do this with a bunch of shell scripts, but it's a bit easier with a tool specifically designed for setting up servers. This is where Ansible comes in. You create a "Playbook" which describes how to set up your server. When you run your Playbook, Ansible connects to your server over SSH and runs through tasks in the Playbook.

If you point Ansible at localhost, you can use it to set up your dev machine. At a previous employer, we used this to make it easy for new developers to get their machines set up with everything running on their first day.

The Playbooks themselves are YAML files. Each step has a name, and uses one of Ansible's modules to declare something about the server setup. For example, let's look at how we'd make sure that Git is installed on our server and configured the way we want it.

- name: Ensure required packages are installed
  apt:
    name:
      - "git"

- name: Set global git config
  copy:
    src: "files/gitconfig"
    dest: "/etc/gitconfig"
    owner: "root"
    group: "root"
    mode: u=rw,g=r,o=r

The first step makes sure that Git is installed using Ansible's apt module. It takes a list of packages and makes sure they're installed using the Apt package manager.

The second step copies a file that's part of my Ansible Playbook (files/gitconfig in the Playbook) to the server and puts it where Git will look for its config (/etc/gitconfig on the server). While it's at it, it makes sure the file ownership and permissions are all right. I use the Git config to set my default branch to main instead of master, but you can use it to set any global Git config.

For the rest of this article I'm going to share how to do things using Ansible. If you want to use a different tool like Docker, Chef, or even just connecting to the server with SSH the principles will be the same.

The plan for sharing repos: Users and Groups

In the last article, I talked about how to set up your server only for private repos for a single user. One of the big changes I'm introducing here is allowing multiple users, and letting them share repos.

To share repos, we need to introduce the idea of groups. A user can belong to as many groups as they want. Shared repos aren't owned by any particular user, but rather they're owned by a group. Then all users belonging to that group can push and pull to the repo.

How do we manage these groups? We use Linux's built in user management and groups! And we do it with Ansible!

There are two parts to our config here. The first is a variables file where we list all of the users and groups we want.

git_groups:
  - name: "family"
  - name: "worthe-it"

git_users:
  - name: "justin"
    groups:
      - "family"
      - "worthe-it"
  - name: "nina"
    groups:
      - "family"

The second part is our Playbook, with a list of tasks which use these variables to create the users and groups. We can use Ansible's loop to go over each item in the git_groups and git_users variables. For each thing we're looping over, item will refer to the specific user or group we're on in the loop.

In Ansible, stuff inside double curly braces like "{{ item.name }}" is treated as code so that you can do things like use variables in your tasks.

- name: Create git user groups
  group:
    name: "{{ item.name }}"
  loop: "{{ git_groups }}"

- name: Create git user accounts
  user:
    name: "{{ item.name }}"
    groups: "{{ item.groups }}"
  loop: "{{ git_users }}"

While we're at it, we also need to set up SSH access for these users. We have a public key file for each user in our Playbook, and use Ansible to upload that too.

- name: Add ssh keys for users
  authorized_key:
    exclusive: yes
    key: "{{ lookup('file', 'files/ssh_keys/{{ item.name }}.pub') }}"
    user: "{{ item.name }}"
  loop: "{{ git_users }}"

Formalizing a directory structure

It will help to keep things organized if we have a plan for where our repos should actually be. In the last article when we left off we would just put repos in our home directory, but that won't work well for our shared repos.

Instead, I prefer to create a folder /srv/git/, where all of the Git repos will go. This has the advantage that when we add backups to the server (in a future article), then the backups can just backup everything in /srv/git/ and know that all of the Git repos on the server are covered.

- name: Create git file hierarchy
  file:
    path: /srv/git
    state: directory
    owner: root
    group: root

Under, /srv/git/, each user and group will get their own directory. So the user justin will be able to put all of their private repos in /srv/git/justin/.

- name: Create personal git directories
  file:
    path: "/srv/git/{{ item.name }}"
    state: directory
    owner: "{{ item.name }}"
    group: "{{ item.name }}"
    mode: u=rwX,g=,o=
    recurse: yes
  loop: "{{ git_users }}"

For extra convenience and pretty clone URLs, we'll create a symlink in each user's home directory, pointing at /srv/git/. This means that the clone URLs will now look like this:

git clone justin@your-cool-server-ip:git/justin/my-awesome-project.git

- name: Git symlinks for user accounts
  file:
    path: "/home/{{ item.name }}/git"
    src: "/srv/git/"
    state: link
    follow: no
    owner: "{{ item.name }}"
    group: "{{ item.name }}"
  loop: "{{ git_users }}"

The next thing to consider is shared repos. If there is a group called worthe-it, then the shared repos owned by worthe-it all go in /srv/git/worthe-it/.

File permissions are important here. You want everyone in the right group to be able to read and write files, so you need to make sure you use the group flags for read and write access. Then, we also need to set the setgid flag (set group id). This will make sure that when new files and directories are created inside that directory, they will also be owned by the same group. Symbolically, this comes together as g=rwXs.

- name: Create shared git directories
  file:
    path: "/srv/git/{{ item.name }}"
    state: directory
    owner: "root"
    group: "{{ item.name }}"
    mode: u=rwX,g=rwXs,o=
    recurse: yes
  loop: "{{ git_groups }}"

We're ready to create shared repos

Right, so we have some our users, our users are in the right groups. We have directories that are owned by the appropriate group, with the setgid flag set. We're ready to create some repos!

The main difference to how we create our repo is that we need to user the --shared group flag to tell Git that it should generally set file permissions to be based on groups rather than individual users.

git init --bare --shared group git/worthe-it/my-awesome-shared-repo.git

It's also really important that we create this repo in the right place. Remember that it's getting which group it belongs to from the directory that we're putting it in.

Shackle Shell

We now have everything set up so that we can share repos, as long as everyone uses it exactly right. Shared repos have to be put in the right folder, and you have to make sure you use the --shared flag when you create the repo.

Also, opening up your server to new people adds new potential security problems. You might want to share a repo with a friend, without letting that friend run funny programs on your server.

To understand how to restrict this, we need to look at the tools involved. When I run ssh justin@my-git-server, the SSH program is connecting to an SSH daemon running on my server. It will check my credentials, and then run my shell (this is often bash by default). You can then type some more stuff, which will be sent by SSH to your server which will enter them into your shell. The shell would then interpret your keystrokes to, for example, run a program.

If you want to limit what a user can do, one option is to set their default shell to something more limited.

Git actually includes a shell called Git Shell that you can use for this. By default, Git Shell just lets you push and pull any repos you have access to. That's usually a bit too restrictive, so you can extend it with shell scripts.

But I don't like writing shell scripts. I'm pretty sure it's impossible to write secure shell scripts. Also, I want to restrict the directory structure for pushing and pulling to what we expect. That's why I wrote Shackle Shell.

Shackle Shell will let through normal Git push and pulls, but only if the repos are in the right place. It has built in commands for creating new repos in the right place and managing them, so you don't need to write any shell scripts. Basically, where it's very easy to accidentally do something wrong when using Bash, Shackle Shell will help stop you from doing it wrong (even if you're trying to do it wrong on purpose), and help you to fall into the pit of success when using your Git server.

Shackle's built in "extra" functionality is still minimal at the moment. It can create new repos, update the main branch or description of an existing repo, and tell you which repos you have. I plan on extending this functionality in the future.

You can either get Shackle Shell by building it from source using cargo install shackle-shell, or you can download the one I've built and uploaded on Codeberg.

You can download the release, unpack it, and set your user's shell using Ansible. The user creation here should replace the earlier user creation task.

- name: Install Shackle Shell
  unarchive:
    src: "https://codeberg.org/worthe-it/shackle-shell/releases/download/v0.1.1/shackle-shell-x86_64-unknown-linux-gnu.tar.xz"
    remote_src: yes
    dest: "/usr/bin/"
    extra_opts:
      - "--strip=1"
    include:
      - "shackle-shell-x86_64-unknown-linux-gnu/shackle-shell"
    owner: "root"
    group: "root"
    mode: u=rwx,g=rx,o=rx

- name: Create git user accounts
  user:
    name: "{{ item.name }}"
    shell: /usr/bin/shackle-shell
    groups: "{{ item.groups }}"
  loop: "{{ git_users }}"

When you next log in over SSH, you'll see Shackle Shell's prompt instead of your normal Bash shell.

Unlike Git Shell, Shackle also has built in help for its various commands, so you can type help to see what you can do.

  > help

  Usage: <COMMAND>

  Commands:
    init              Create a new repository
    list              List all repositories available
    set-description   Sets the description of a repository, as shown in the CLI listing and web interfaces
    set-branch        Sets the main branch of the repository
    exit              Quit the shell
    git-upload-pack   Server side command required to git fetch from the server
    git-receive-pack  Server side command required by git push to the server
    help              Print this message or the help of the given subcommand(s)

  Options:
    -h, --help  Print help

Then you can use the init command to make a new repo. It will take the group name, the name of your project, and figure out the right path and arguments to pass Git to create your repo.

  > init --group worthe-it awesome-project-idea

  Successfully created "git/worthe-it/awesome-project-idea.git"

Wrapping up, what do we have now?

At this point, we have a Git server, that we can push code to, and pull code from. Like at the end of the last article, we can create as many private repos as we want. Excitingly, we can now have repos that we share with a friend!

We're also running Shackle Shell, which makes it much easier to consistently create repos in the right place. And just in case, it stops your friend from using their new Git server access to run a cryptocurrency miner.

There's still much more ground to cover. If you are doing this, here are some other things to consider:

• How to bootstrap this whole thing. You'll need a user with root permissions for Ansible to run as so it can do all of this setup.

• Managing your various operating system packages, and keeping everything updated. You can do all of this with Ansible too.

• I shared Ansible snippets, but there are various ways of organizing your Playbooks you might want to consider. I organized my Playbooks based on Ansible Roles.

• Backups! Since all of the Git stuff is in /srv/git/, it's a good idea to have some job that backs up /srv/git/ regularly. Without going into too much detail in this article, you can make sure a limited backup user has access to everyone's repos for backups with Access Control Lists, and do incremental backups to cloud storage using something like Duplicity.

• If you want to share your repos publicly, you can use something like CGit to add a web interface.

I'm personally learning a lot by running my own Git server. I also like that I can now share private repos with my wife, without worrying about big Git hosting vendors scraping my private repositories for their AI training.

If you're following along at home, I hope this helped you to get more use out of your own Git server, and that you're enjoying it as much as I am!

If you'd like to share this article on social media, please use this link: https://www.worthe-it.co.za/blog/2023-06-02-leveling-up-your-git-server-sharing-repos.html

Tag: blog

Support

If you get value from these blog articles, consider supporting me on Patreon. Support via Patreon helps to cover hosting, buying computer stuff, and will allow me to spend more time writing articles and open source software.

Related Articles

Automated Syncing with Git

I wanted Dropbox-style syncing of my notes between my computers. However, rather than actually using Dropbox, I wanted to keep my notes in a Git repo so that I can manage it the same way that I manage code that I write. This article shows how I achieved this using Git Sync and Systemd.

How to Train Your Git Server

Have you considered hosting your own Git server? It's easier than you might think. In this article, I go step by step through setting up a simple self-hosted Git server which only supports private repositories for a single person.