Implementing a System Call for OpenBSD

Shout outs to my patrons !

As usual, a huge thanks goes to the people sponsoring me on patreon or github , you can read more about my sponsorship here .

Disclaimer

The content here is from an article I wrote

The content here is based on an assignment I wrote in 2005. This is by no mean a tutorial to encourage you to write system calls and send them to OpenBSD.

A few reminders

Program vs Process

People often use the two words interchangeably but it is important to understand the difference between a program and a process, particularly because the same program may be allowed to use a system call in a process and not in another, but also because the process is part of the syscall API.

A program is an executable which contains a set of instructions that are meant to be executed and do something. It resides as a structured file on the filesystem which enforces restrictions as to who can or cannot execute it.

A process is an instance of that program, running in its own memory-space, with its own privileges.

If we take /bin/ls , it is a program that lists directories and files. When a user executes it, a process is created which will actually run the program with the privileges of that user, in a memory space that’s not shared with other processes.

Kernel and userland

Unix-like systems have an architecture where code is executed in two main areas: the kernel and userland.

The kernel is in charge of providing and limiting access to devices, enforcing restrictions as to what an executing program can do, and providing programs with a virtual memory space in which they can execute.

A program executes in userland and perform operations on memory that’s allocated to it by the kernel during the initialisation of the process. When the program needs to access a device or needs the kernel to perform an operation that it’s not allowed to perform itself, it asks the kernel to trigger a system call. The system call is a function that’s part of the kernel and that runs as part of it on behalf of the process.

System call

A system call is a service provided by the kernel so that a userland process can request the kernel to do something on its behalf, usually something that the userland program is not able or not allowed to do on its own.

From the point of view of a program, it is a somewhat special function that it can call similarly to any other function, but which doesn’t run in the process memory space. A program only knows about the system call interface but doesn’t have access to its implementation, so it can call it, pass parameters to it, obtain a result from it, but not inspect what happens inside the system call as it runs. It can’t debug it.

This comes with side-effects. Performance-wise, a system call switches the execution to kernel which is costly. Then bugs in a system call have a different impact from bugs in a function call: a memory corruption bug may cause the process to terminate, whereas the same memory corruption bug in a system call may cause the system to crash.

There are two sides to a system call:

The system call implementation, which is the actual code of the system call that’s going to run inside the kernel when called, and the system call interface, which is how the system call is meant to be called from a userland application. It is important to differentiate both as, in OpenBSD, the prototype for the system call implementation does not match the prototype for the system call interface as we’ll see shortly.

Implementing system calls for OpenBSD

Prerequisites

Use a privileged account

It is obvious that an unprivileged account is not allowed to alter the kernel as it enforces permissions on the system. For that reason, it is mandatory to use a privileged account at least for installing a mo