TL;DR: This article is a deep dive into the concept of pinning in Rust. I argue that while ownership and transfer of ownership are powerful abstractions, they are somewhat leaky. This should not come as a surprise, as all non-trivial abstractions are leaky! However, knowing where an abstraction is leaky helps us use it more effectively. I further explain why Pin is such an effective construct for dealing with the leak!

If you have any experience with Async Rust, you must have come across Pin and pinning. Pinning is a complex topic and many Rust programmers find it confusing. If you feel uncomfortable around pinning, or like me, have had too many “why” questions about it, this article is for you!

All the articles (e.g., [1], [2], [3]) I have read on the topic of pinning state that the need for pinning is due to the existence of self-referential objects. However, that cannot be the only reason, as we see self-referential objects in other programming languages too, but nothing similar to Rust’s pinning.

Some other factor, unique to Rust, must be at play. In this article, we are aiming to understand this other factor. Along the way, we will answer the following questions:

1. How are mutability, move semantics, and movability related?
2. How does Pin work, and what guarantees does it provide?
3. When do you need to use Pin in your API?
4. How do other languages deal with self-referential objects?

Pin in a nutshell

Pin is a smart pointer. It is often said that Pin “prevents an object from being moved in memory, unless it is address-insensitive”. This article is all about deciphering this sentence, as it is not quite accurate, or rather, does not give the right impression when you first read it. Pin, alone, does not have a mechanism to keep an object in its place in memory, but is the way to express a guarantee about the object’s immovability. You, as the developer, may need to use other language constructs to achieve the stated guarantee.

The following diagram shows the structure of a Pin. We will come back to this diagram after clarifying some concepts, and will have a closer look at the three players in it when discussing Pin’s guarantees.

Movability

If I ask you what is movability, chances are, you’ll think of Rust’s move semantics and transfer of ownership. Here is a brief summary:

Rust manages memory through a system of ownership with a set of rules that the compiler checks. Each value (e.g., an object) in a Rust program is owned by a variable. Ownership of a value can be transferred from one variable to another, for instance in an assignment statement. The Rust compiler calls this moving out of one variable into another. Ownership transfer is necessary for guaranteeing single ownership, a rule that the compiler relies on for ensuring memory safety.

Question: Is ownership transfer the type of move that Pin expresses a guarantee about?

Not really. In the context of pinning, “move” and “movability” mean that the value has moved in the mechanical sense of being located at a new place in memory. In the following, I am going to refer to this as a mechanical move.

Two types of move

Question: Does a “move”, or transfer of ownership of a value, involve a mechanical move?

The short answer is “yes”. The longer answer is “yes, but sometimes only partially”.

Question: Can the location of a value in memory change without a transfer of ownership?

This one is trickier to answer. This is where the boundaries between transfer of ownership, mechanical move, and even mutability get blurred. To answer this question, it is best to look at some examples.

In the first example (playground), we create a vector with capacity 2, then push 4 elements to it. The elements are of type SelfPoint:

// Example 1:

struct SelfPoint {
    data: u32,
    ptr: *const u32,
}

impl SelfPoint {
    fn new(data: u32) -> Self {
        SelfPoint {
            data,
            ptr: std::ptr::null(),
        }
    }

fn init(&mut self) {
        let ptr: *const u32 = &self.data;
        self.ptr = ptr;
    }
}

fn main() {
    let mut v = Vec::with_capacity(2);

for i in 0..4 {
        let x = SelfPoint::new(i);
        v.push(x);
        v.last_mut().expect("vec is empty").init();
        println!("Vector addr & cap: {:p}, {}; heap location: {:p}",
                 &v, v.capacity(), &v[0]);
    }
    ...
}

Each time we push an element to the vector, a move happens. This move involves both a transfer of ownership and a mechanical move (from the stack, to the call stack of push, and then to the heap). After pushing each element to the vector, we call init on it to set the value of ptr to the address of data. Since we initialized the vector with capacity 2, pushing the third element will result in allocating a larger buffer, and moving the elements to the new buffer. The following diagram shows the memory layout of the vector, before and after pushing the third element to it.

Notice how after the third push, the ptr fields of the first two elements of the vector point to memory locations they no longer own. This is exactly the kind of problem that Pin is designed to prevent.

In this example, the memory allocated to the vector on the heap is moved, mechanically, but no explicit transfer of ownership has happened. Deep in the implementation of Vec, the vector’s ptr is updated and mutated, but even this is not something that I can call a transfer of ownership.

What happens if you write a similar program in Python or Java? Surely there too new memory has to be allocated as the vector expands, but will SelfPoint instances be relocated? How about in C++?

In the next example (playground), we initialize the vector with a larger capacity to prevent memory reallocation. We also pass the vector to a function causing a transfer of ownership.

// Example 2:

fn validate(v: Vec<SelfPoint>) {
    for x in &v {
        let data = &x.data as *const u32;
        assert!(data == x.ptr);
    }
    println!("Validated vector at {:p}; buffer at: {:p}", &v, v.as_ptr());
}

fn main() {
    let mut v = Vec::with_capacity(4);

for i in 0..4 {
        let x = SelfPoint::new(i);
        v.push(x);
        v.last_mut().expect("vec empty").init();
    }

println!("Created vector\t at {:p}; buffer at: {:p}", &v, v.as_ptr());
    validate(v);
}

The following diagram shows the memory layout in this case.

In both of the examples, vector v is partially moved to a new location in memory. In the first example, only the heap-allocated memory is moved, without any transfer of ownership. In the second example, only the stack-allocated memory is moved, involving a transfer of ownership.

With these examples in mind, we can now answer the question “Can the location of a value in memory change without a transfer of ownership?”

The answer is yes. In other words, mechanical move is a broader concept than ownership transfer.

Movability and mutability

The mechanical move in Example 1 was caused by a mutation. In fact, there is a very close connection between mutability and movability. More specifically, mutations often entail a mechanical move (see the classic example with mem::swap). However, they are not necessarily accompanied by a transfer of ownership. Transfer of ownership involves marking a memory location as invalid, but in the case of mem::swap for instance, both memory locations remain valid after the operation.

Due to this close connection between mutability and movability, in the description of Pin, we see many references to mutability, despite Pin being all about movability!

Leaky Abstractions

While ownership and transfer of ownership are powerful abstractions, they are somewhat leaky, particularly when it comes to capturing the notion of mechanical moves. This is hinted at in the explanation of ownership in the Rust book. Moreover, as we saw in the use of Vec in Example 1, we may need to study the internal details of a type when deciding whether its manipulation can result in a mechanical move or not.

Having a leaky abstraction or only partially hiding details in a programming language is nothing new. Niklaus Wirth, in his 1974 paper “On the Design of Programming Languages”, writes:

I found a large number of programs perform poorly because of the language’s tendency to hide “what is going on” with the misguided intention of “not bothering the programmer with details”.

In my view, Rust programs are anything but “programs that perform poorly”, but knowing that an abstraction is leaky allows us to use it more effectively.

Address sensitivity

So far, we’ve learned that objects in Rust are movable in a mechanical sense, and that this kind of move is different from ownership transfer.

In some cases, the validity of an object’s invariants and the soundness of its behavior depend on its location in memory. We say that such an object is in an address-sensitive state. Self-referential objects (i.e., objects with pointers to themselves, or to fields in themselves) are the archetype of address-sensitivity. In the examples above, calling init on a SelfPoint instance made it self-referential and address-sensitive.

Pointing vs referencing

The field ptr in SelfPoint is a pointer, not a reference! If we change ptr to be a reference, we get a compiler error (playground)! Moving a truly self-referential (as opposed to a self-pointing!) object violates Rust’s borrow-checking rules.

Strictly speaking, address-sensitivity is a state of an object, not necessarily a property that can be specified via its type (e.g., instances of SelfPointare not address-sensitive before calling init on them). However, it is often a good practice to associate a different type to each state of a stateful entity. Following this practice, a self-referential object becomes address-sensitive early at the beginning of its lifetime, and remains address-sensitive until the end of its lifetime. In other words, once your object is no longer address-sensitive, it is a good practice to consume it and convert it into an instance of another type.

As a result, in practice, address-sensitivity can be specified via types. This allows the compiler to reason about the address-sensitivity of objects at compile time.

Why is pinning needed?

Once an object is in an address-sensitive state, the compiler can guarantee memory-safe interaction with it, only if the object stays in its location in memory. If an object is not address-sensitive, it can freely move around in memory, without violating any of the compiler’s safety guarantees.

To tell the compiler that instances of a type will not be address-sensitive, we implement the Unpin auto trait for that type. By default, all types are Unpin. To announce potential address-sensitivity of instances of a type, we implement !Unpin for it. We will learn more about Unpin below.

To promise to the compiler that an object has a fixed location in memory, or otherwise is address-insensitive, we wrap a pointer (i.e., pinning pointer) to that object (i.e., pointee) in a Pin. As we will see below, Pin restricts mutable access to a pointee that is !Unpin.

In short, pinning (i.e., the use of Pin and Unpin), are the means that you as a developer use to help the compiler provide its memory-safety guarantees, and avoid undefined behavior when it comes to address-sensitive types and objects.

The Unpin Trait

Unpin is a marker trait, and is automatically implemented for almost all types. It is implemented even for type SelfPoint in Example 1. This is because the compiler has no reason to assume that at runtime you are going to set SelfPoint.ptr to point toSelfPoint.data.

To tell the compiler that instances of SelfPoint will be self-referential and address-sensitive, you have to explicitly implement !Unpin for it. The conventional way of doing this is to add a field of type PhantomPinned to your type. PhantomPinned is !Unpin, and a type that contains a PhantomPinned does not get a default Unpin implementation. We change SelfPoint as follows:

// Example 3:

use std::marker::PhantomPinned;

struct SelfPoint {
    data: u32,
    ptr: *const u32,
    _pinned: PhantomPinned,
}

If your type has a field that is !Unpin (e.g., a tokio::time::Sleep field), and you want to make your type Unpin, you have to explicitly implement Unpin for it. Doing so is as simple as writing a single line of code.

impl Unpin for MyType {}

But you have to be very careful, because when you implement Unpin, you are promising to the compiler that instances of your type will be address-insensitive and freely movable.

Pin and its guarantees

Now that we know why pinning is needed, we can take a closer look at Pin, its API, and its guarantees.

The structure of a Pin

As mentioned above, to promise to the compiler that an object either has a fixed location in memory (i.e, is immovable) or is address-insensitive (i.e., is Unpin), we wrap a pointer to it in a Pin. This is shown in the diagram below (repeated from above).

• The object that we want to express the promise about is called the pointee.
• The pointer to pointee is called the pinning pointer.
• The pin itself is a smart pointer that wraps around the pinning pointer, and takes ownership of it.

We say that pin pins the pointee.

Mutability, Movability, and Pin’s guarantees

Once you express your promise about the immovability or address-insensitivity of an object, by pinning it, Pin guarantees that it will maintain your promise. To achieve this, Pin won’t easily give mutable access to the pointee. This is important, because as we saw above, mutable access to a value can result in moving it in memory. For address-sensitive objects, this may violate the invariant of the object, or cause memory-safety issues.

APIs utilizing Pin, with Future::poll being a prominent example, are concerned with integrity and validity of the pointee’s invariants rather than its specific memory location. These APIs need to mutate the pointee, and need to know that doing so won’t result in undefined behaviors or a violation of the pointee’s invariants. To establish this proposition, immovability is used as an overapproximation — a sufficient and easier-to-verify condition that ensures the object’s integrity as it is being mutated.

By relying on immovability, Pin provides a simple and generic API that allows safe interaction with address-sensitive objects in arbitrary contexts without having to make any additional assumptions about the internal invariants of those objects.

Pinning address-insensitive objects

Instances of an Unpin type are promised to be address-insensitive, and therefore allowed to move freely in memory. Pin has a safe API for working with Unpin objects. The following table lists the methods in Pin’s safe API. Those in green require the pointee to be Unpin. The table shows the API from stable Rust 1.77.0.

If the pointee is Unpin, you can easily wrap it in a Pin, using the new method, without any additional restrictions. Moreover, if the pointee is Unpin, you can use the get_mut method to get mutable access to it. Mutating the pointee is safe because it is address-insensitive, and can move and mutate freely.

The methods in blue do not give direct mutable access to pointee, and cannot be used to mutate or move the object, without additional, possibly unsafe, code. Therefore they are safe.

Pinning address-sensitive objects

If instances of a type can enter an address-sensitive state, then that type must be declared !Unpin.

To soundly Pin instances of an !Unpin type, you have to promise that the pointee’s data will not be moved nor have its storage invalidated, until it gets dropped. The compiler cannot check these promises, so when working with !Unpin types, you have to write unsafe code. Pin offers a number of unsafe methods particularly for working with !Unpin types.

To create an instance of Pin in this case, you have to use the unsafe new_unchecked method. The documentation for this method states a few additional promises that you have to make. These promises allow Pin to guarantee that your use of its safe API (colored in blue in the previous table) will be sound.

These additional promises are about the pinning pointer, which is the argument to new_unchecked. Needless to say, the pinning pointer must be Deref. You have to promise that in your implementation of Deref::deref, you do not “move [any data] out of self” (even a partial move violates this promise). You have to make a similar promise about your implementation of DerefMut if you have it. This is because Pin relies on Deref::deref and DerefMut::deref_mut in the implementation of some of its safe methods (e.g., as_ref and as_mut), and can only maintain the immovability promise, if Deref::deref and DerefMut::deref_mut guarantee it.

Note that inside DerefMut::deref_mut, you have mutable access to pointee. A malicious implementation could simply use swap to move a value out of self.

The methods map_unchecked and map_unchecked_mut are consuming methods that, among other things, are particularly useful for converting the pointee to an Unpin type, signaling its transition to an address-insensitive state. As an example, we could map instances of SelfPoint to address-insensitive u32 values holding only the content of the field data.

Putting it all together

We can use Pin and Unpin to fix the problem that we faced in Example 1. Here are the changes we have to make for the fix:

1. Change SelfPoint to be !Unpin as in Example 3.
2. Change init to take Pin<&mut Self> as its receiver. It only makes sense to make an instance of SelfPoint self-referential if we can guarantee that it is immovable. One way to do this is to move instances of SelfPoint to the heap.
3. Change the vector to store pinned references to SelfPoint instances.

Here is the final code, with these changes applied (playground).

// Example 4:

use std::marker::PhantomPinned;
use std::pin::Pin;

struct SelfPoint {
    data: u32,
    ptr: *const u32,
    _pinned: PhantomPinned,
}

impl SelfPoint {
    fn new(data: u32) -> Self {
        SelfPoint {
            data,
            ptr: std::ptr::null(),
            _pinned: PhantomPinned,
        }
    }

fn init(self: Pin<&mut Self>) {
        let ptr: *const u32 = &self.data;
        let this = unsafe { self.get_unchecked_mut() };
        this.ptr = ptr;
    }
}

fn validate(v: Vec<Pin<Box<SelfPoint>>>) {
    for x in &v {
        let data = &x.data as *const u32;
        assert!(data == x.ptr);
    }
    println!("Validated vec:\t{:p}; buffer addr: {:p}; v[0] addr: {:p}",
             &v, v.as_ptr(), v[0]);
}

fn main() {
    let mut v = Vec::with_capacity(2);

for i in 0..4 {
        let mut x = Box::pin(SelfPoint::new(i));
        x.as_mut().init();
        v.push(x);
        println!("{i} Vector addr:\t{:p}; buffer addr: {:p}; v[0] addr: {:p}",
                 &v, &v[0], v[0]);
    }

validate(v);
}

Do I need to use Pin in my API?

When I started digging into pinning, I wanted to be able to comfortably decide whether to use Pin in my APIs or not. This decision becomes relevant when designing generic APIs or traits, where we don’t know the exact types of the objects that will interact with the API.

In these cases, you might want to use Pin, if:

1. Your API is likely to work with address-sensitive types and objects. For instance, types that implement the Future trait are likely to be self-referential, and therefore address-sensitive. APIs that work with a Future need to Pin it. On the other hand, elements stored in a Vec don’t have that tendency, and more often than not are address-insensitive.
2. Your API needs to mutate objects. For instance, Future::poll is called to make progress on a Future object. This requires changing, or mutating, the state of the Future object. On the other hand, in its implementation, Vec does not mutate the elements in its buffer.
3. The safety of your API depends on the object’s integrity and the validity of its invariants. For instance, as a safe method, calls to Future::poll “must never cause undefined behavior (memory corruption, incorrect use of unsafe functions, or the like), regardless of the future’s state.” On the other hand, while most of the Vec’s methods are safe, its contract is not affected by the invariants of the elements in its buffer. The presence of address-sensitive objects in a Vec does not cause undefined behavior when invoking any of its methods.

If your API passes all these checks, you’ll need Pin in your API. Passing these checks implies that your API depends tightly on the inner working of the types that it interacts with. Using Pin hides those details behind an immovability promise. This promise, although possibly more conservative than what you want, is easier to reason about.

Otherwise, even if you have a general purpose API like Vec, you won’t need Pin in your API; and your API will work just fine with pinned objects too.

Before wrapping up this section, I want to emphasize again that Pin does not force the pointee to stay in its location. It merely expresses an immovability promise about it. While all objects in Rust are movable, most objects never leave their original locations in memory throughout their lifetimes. For instance, most heap-allocated objects never move, as we saw in Example 4.

How do other languages deal with self-referential objects?

To answer this question, let’s start with Java, as a representative of garbage-collected languages.

In Java, all objects are heap-allocated. In your programs, you work with references to these objects, and can easily make self-referential objects. Consider the following example, which declares class SelfRef.

// Example 5:
class SelfRef {
   SelfRef self;

void init() {
       this.self = this;
   }
}

You can make instances of SelfRef self-referential by calling init on them. But how about moving instances of SelfRef in memory? Well, the language does not provide you with a mechanism to do that. The notion of movability, in the sense of manually relocating objects in memory, does not exist in Java or similar garbage-collected languages. In these languages, objects reside safely in the heap, each hiding behind a reference. However, this does not mean that objects in these languages never relocate in the memory. The runtime, more specifically the garbage collector, does move objects around. This happens specifically when defragmenting or compacting the heap after removing the garbage collected objects. The garbage collector manages this process by tracking references to ensure their validity.

The same is true for other garbage-collected languages, such as Python. An exception worth mentioning is Golang, which (as opposed to Java and Python) allows pointer types, and pass-by-value. In Golang, you can declare a type similar to SelfPoint in Example 1, create self-referential instances of it, and move them around in memory. The garbage-collector does not release the old memory as long as a pointer to it exists, but you have to be careful with invariants as the objects move.

In C++, as far as I understand, you can move objects around. However, the language allows you to custom-implement move constructors. I suppose, move constructors are where you can ensure that invariants of your type are maintained as your objects move around in memory.

Conclusion

In my quest for understanding Pin, I came to the conclusion that it is a construct that sits at the intersection of a few leaky abstractions, and provides a simple and generic mechanism for expressing invariants about address-sensitive objects without depending on their internal details.

The main leaky abstraction is the concept of moving a value. You want to think of it as a transfer of ownership, but it is more complex than that. Sometimes you don’t have an explicit transfer of ownership, but parts of the value move to another location in memory. Sometimes you have an explicit transfer of ownership, but parts of the value (e.g., the heap-allocated parts) stay in their memory location.

These subtleties become important when working with address-sensitive objects, whose invariants may be violated if their location in memory changes.

It is important to note that it is not the location of these objects that we care about, but their integrity and the validity of their invariants. However, it is generally easier to reason about the movability of an object rather than its invariants. Pin and Unpin are the constructs that we use for expressing movability guarantees.

Despite some unsoundness issues in earlier versions of Pin, in my assessment, Pin and Unpin form a neat and effective abstraction that conceals the complexities of memory management that are leaked via the concepts of ownership and move semantics. I am curious about their origin and evolution. The story is certainly somewhere in the discussions or on Github, if you go looking!

This article is a follow up on my earlier article titled AsyncWrite and a Tale of Four Implementations, which goes into an in-depth discussion of the decisions you have to make when working with async and Futures.

References