POSTS

Raptor CS: Fully Owner Controlled Computing using OpenPOWER

May 17, 2022

This week I am talking to Timothy Pearson of Raptor Engineering. He is behind the Talos II and Blackbird boards for IBM POWER9 CPUs. His major claim is creating the first fully owner controlled general purpose computer in a long while. My view of the Talos II and Blackbird systems is that these boards helped to revitalize the open source ecosystem around POWER more than any other efforts (See also: https://peter.czanik.hu/posts/cult-amiga-sgi-workstations-matter/). Most open source developers I talked to say that coding on a remote server is just work. Doing the same on your local workstation adds an important ingredient: passion. This is why the re-introduction of POWER workstations was a very important step: developers started to improve support for POWER also in their free time, not just in their regular working hours. I asked Tim how the idea of creating their POWER board was born, how Covid affected them and also a bit about their future plans.

Talos II mainboard

Raptor Engineering seems to focus on machine vision. How can a company that’s focused on other activities start to develop a mainboard for POWER CPUs?

So to start with, Raptor Engineering’s public facing website was always more of a way to market technologies we’d already developed for internal purposes to the general public. Raptor Engineering is, and always has been, more of a FPGA HDL/firmware/OS level design company focused on providing those services to those that need them. Sometimes we do engage in internal development projects to showcase certain technology capabilities publicly, and one of those was the open source machine vision system you allude to earlier.

More relevant to the current technology offerings, we’ve always had some degree of focus on security and what we now tend to call “owner control”. This was borne out of several long-running research and development projects, where significant investment was made in technologies that could be years, if not decades, from practical application. As a result of this longer term focus, it was realized early on that two issues had to be addressed: something we call “continuance” (the ability to seamlessly move from one generation of technology to another as needed, with no loss of data) and also the data security / privacy aspects of keeping internal R&D results private and out of the hands of potential competition.

Around 2008-2009 we realized that a fully owner controlled system would meet all of these objectives, specifically a system over which we had full control of the firmware, OS, and application layer, a system where we could modify any aspect of its operation independently as required to support the overall objectives. There’s a true story I like to tell, which dates from the early days of the coreboot port efforts on our side to support the AMD K8 systems in use at the time: the proprietary BIOS would not, for some infurating reason, allow a boot without a CMOS battery and a keyboard connected! When spread across supercomputer racks, this was a perfect opportunity to highlight just what owner control actually meant in terms of benefit for maintainance and overall scalability – the rather obvious resulting boot problem could have been solved with a single line of code, if only there had been code available.

Once we had a completely free stack for K8/Family 10h, we continued on with AMD systems until around 2013/2014, when it became clear that AMD would force the Platform Security Processor (an unauditable low-level AMD-controlled black box that presents a severe security risk) on all CPUs going forward. This kicked off a multi-year evaluation of anything and everything that might be able to replace the x86 systems we had in terms of owner control, overall performance, and ease of administration (ecosystem compatibility). Over those years we evaluated everything from ARM to RISC-V to MIPS, and eventually settled on the then-new OpenPOWER systems as the only solution to actually check all of the boxes. The rest, as they say, is history – the nascent computer ODM arm of Raptor Engineering was spun off as Raptor Computing Systems to allow it to grown into the role demanded of it with the Talos II systems it was bringing to market.

Six years ago I first learned about the Talos plans from the Raptor Engineering website. Now all POWER-related activity is on the Raptor Computing website. How are the two related?

As of right now I am involved with both companies, my role at Raptor Computing Systems being that of CTO. Basically I help ensure RCS brings new owner-controlled, blob-free devices to market, and keep fairly close tabs on available silicon and its limitations (mostly centering around blobs) as a result. On the Raptor Engineering side I still provide specialized consulting services at a low level (typically HDL / firmware), as an example you’ll see my name here and there on projects like OpenBMC, LibreSoC, and more recently the Xen hypervisor port for POWER systems that is just spinning up now.

What do you mean by “fully owner controlled”?

An owner-controlled device is best defined as a tool that answers only to its physical owner, i.e. its owner (and only its owner) has full control over every aspect of its operation. If something is mutable on that device, the owner must be able to make those changes to alter its operation without vendor approval or indeed any vendor involvement at all. This is in stark contrast with the standard PC model, where e.g. Intel or AMD are allowed to make changes on the device but the owner is expressly forbidden to change the device’s operation through various means (legal restrictions, lack of source code, vendor-locked cryptographic signing keys, etc.). In our opinion, such devices never really left the control of the vendor, yet somehow the owner is still legally responsible for the data stored on them – to me, this seems like a rather strange arrangement on which to build an entire modern digital economy and infrastructure.

What are the main differences between the Talos II and Blackbird boards?

The Talos II is a high end full EATX server mainboard, dual socket, designed for 24/7 use in a datacenter or workstation type environment. Blackbird is a much smaller uATX single CPU system with more standard consumer-type interfaces (e.g. audio, HDMI, SATA, etc.) available directly on the mainboard.

You mention that Blackbird is consumer focused. It has three Ethernet ports, and support for remote management. Did you also have servers / appliances in mind?

Yes, there is the ability to use the Blackbird for a home NAS or similar appliance. The remote management sort of comes “for free” due to the POWER9 processor being paired with the AST2500 BMC ASIC and OpenBMC, it is more of a baseline POWER9 feature than anything else.

I know that quite a few Linux distribution maintainers now use Talos II workstations. Who else are your typical users?

In general we see the same overall subset of people that would use Linux on x86, with a strong skew toward those concerned about privacy and security and a notable (expected) cutout of the normal “content consumer” market (gamers, streaming service consumers, etc.). This holds true both for individuals and organizations, though the skew is much less notable among organizations simply because organizations are not generally purchasing desktop / server systems for gaming and media consumption!

For quite a few years POWER9 was the best CPU to run syslog-ng. What are the typical software your users are running on your POWER workstations and servers?

For the most part, the standard software you would see on comparable x86 boxes. That’s always been one of our requirements, that using the POWER system be as close to using a standard PC as possible, except the POWER system isn’t putting digital handcuffs on its owner and potentially exposing their data for monetization (or, for that matter, to more nefarious actors).

Covid affected most families and businesses around the world. How Raptor was / is affected?

We were hit hard by the shutdowns and subsequent inflation, in common with most manufacturers across the world. We did have to cancel some of the more ambitions POWER9 systems under development (Condor) as well as take on increased ecosystem maintainance load. Some of that, along with the continuous rise in cost of parts and rolling shortages of components, is reflected in the price increases and long lead times that have occured over the past couple of years.

Could you explain how Condor compares to Talos II and Blackbird?

Condor was a pre-COVID development project to try to create a high end standard ATX (vs. EATX) desktop board with OpenCAPI brought out to the appropriate connector. With COVID shutdowns, industry adoption of CXL, and more importantly POWER10 requiring closed source binaries, we didn’t see a path to actually bring the product to market post-COVID. The completed designs are sitting in our archives, but no hardware was manufactured.

The latest x86 CPUs now beat POWER9 in many use cases. Which is no wonder, these CPUs are four years old now. Is there something where POWER9 still has an advantage?

Absolutely! POWER9 still does what we originally intended it to do – it gives reasonable performance on a stable, standardized ecosystem, in a familiar PC-style form factor, using standard PC components, while providing full owner control. As a secure computing platform, both on server and desktop, it simply cannot be beat – there is literally nothing else on the market that is both 100% blob free and can be used as a daily driver for basically every task that a Linux x86 machine can be used for.

For example, I’m responding to this interview using a POWER9 workstation with hundreds of Chromium tabs open, media running, Libreoffice in the background, and am even compiling some software. If you didn’t tell me it was a POWER system, I wouldn’t be able to easily tell just from using it, yet at the same time it’s not restricting what I can do with it, attempting to monetize my data, or otherwise waiting for commands from a potentially hostile (at least to my interests) third party.

The other major selling point is that the ISA is both open and standardized. The standardization means that I could migrate this system as-is – no reinstallation – to anything that is POWER ISA 3.0 compliant, which is a huge advantage only available for the “big three” architectures (x86, SBSA ARM, and OpenPOWER). That said, being an open ISA, anyone is also free to create a new compliant CPU if they want to or need to. This neatly avoids the entire problem with x86 and ARM, where various technologies (ME, PSP, TrustZone w/ bootloader locking) could be (and eventually were) unilaterally forced onto all users regardless of the grave issues they introduce for specific use cases.

Also interesting is how the OpenPOWER ISA is governed – a neutral entity (the OpenPOWER Foundation) controls the standards documents that implementations must adhere to, and anyone is free to propose an extension to the ISA. If accepted, it becomes part of the ISA compliance requirements for a future version, and the requisite IP rights are transferred to the Foundation such that anyone is still free to implement that instruction per the specification without needing to go back and license with the entity that proposed the extension. This is a great model in my mind, it should allow the best of both worlds going forward. The standardization and compliance requirements mean we should see the same level of binary support normally expected on x86, while the extension proposal mechanism allows the ISA to morph and adapt in a backward-compatibile way in response to external needs.

The POWER 10 CPUs from IBM are manufactured with the latest technologies allowing higher performance with lower power consumption. Do you have any plans to have a new board with POWER 10 support?

At this time we do not have plans to create a POWER10 system. The reasoning behind this is that somehow, during the COVID19 shutdowns and subsequent Global Foundries issues, IBM ended up placing two binary blobs into the POWER10 system. One is loaded onto the Microsemi OMI to DDR4 memory bridge chip, and the other is loaded into what appears to be a Synopsis IP block located on the POWER10 die itself. Combined, they mean that all data flowing into and out of the POWER10 cores over any kind of high speed interface is subject to inspection and/or modfication by a binary firmware component that is completely unauditable – basically a worst-case scenario that is strangely reminiscent of the Intel Management Engine / AMD Platorm Security Processor (both have a similar level of access to all data on the system, and both are required to use the processor). Our general position is that if IBM considered these components potentially unstable enough to require future firmware updates, the firmware must be open source so that entities and owners outside of IBM can also modify those components to fit their specific needs.

Were IBM to either open source the firmware or produce a device that did not require / allow mutable firmware components in those locations, we would likely reconsider this decision. For now, we continue to work in the background on potential pathways off of POWER9 that retain full compatibility with the existing POWER9 software ecosystem, so stay tuned!

Does “potential pathways off of POWER9” have something to do with the LibreSoc project?

We’re going to remain a bit coy on this, but LibreSoC is definitely one project that would fit that requirement for at least low-end devices. In fact, you can see some of my fingerprints in the LibreSoC GIT history…

Photo credits:

Copyright: Vikings GmbH License: CC BY 4.0