The Joy Of Properly Designed Embedded Systems

The ages-old dream of home automation has never been nearer to reality. Creating an Internet of Things device or even a building-wide collection of networked embedded devices is “easy” thanks to cheap building blocks like the ESP8266 WiFi-enabled microcontroller. Yet for any sizable project, it really helps to have a plan before getting started. But even more importantly, if your plan is subject to change as you work along, it is important to plan for flexibility. Practically, this is going to mean expansion headers and over-the-air (OTA) firmware upgrades are a must.

I’d like to illustrate this using a project I got involved in a few years ago, called BMaC , which grew in complexity and scope practically every month. This had us scrambling to keep up with the changes, while teaching us valuable lessons about how to save time and money by having an adaptable system architecture.

How to Design a Flexible Embedded Ecosystem

A flexible system is one that can adapt to new requirements and incorporate new software and hardware without significant difficulties. Since it’s often impossible to know what the final set of requirements will look like, it’s best to keep options open, just in case they are needed later. Options means features like connectors for expansion modules, a more flexible power supply than strictly necessary, and a convenient means of re-flashing the firmware.

When the project got open-sourced after a number of months, it had expanded from simply collecting temperature data to also monitoring and controling the coffee machines and the air conditioning units with custom hardware, and reacting to logic running on a number of back-end services. With none of this planned beforehand, the development process involved a lot of improvisation and many revisions of the hardware.

As the project is now known, the Building Management and Control (BMaC) system spans sensors for room temperature, air pressure, relative humidity, CO2, motion and soil humidity, with actuator control for valves, coffee machines, Fan Coil Unit (FCU) fans, as well as pumps and similar. While growing the project in scope was not without difficulties, by leaving options open it nevertheless allowed the system to be extended.

The core hardware consists out of ESP8266 microcontrollers (ESP-12E/F modules), which are cheap and plentifully available, while still providing a significant amount of processing power with its single 32-bit core and easy connectivity over WiFi. This allowed us to quickly roll out the hardware and try out iterations without having to run Ethernet cables everywhere.

This diagram covers the environmental controls and monitoring, as well as the back-end services, which store the node configuration (Command & Control), provide an MQTT broker (Mosquitto), store sensor data (InfluxDB) and control the FCUs (ACControl). The white enclosure in the center is the ceiling-mounted BMaC node that contains the ESP8266 microcontroller, as well as a custom PCB.

This PCB provides mounting locations for the various environmental sensors, an external I2C GPIO expander, four channels of variable DC voltage output, and power rails for multiple devices running on 3.3 V, 5 V, and 12 V. The cut-out in the PCB is for the motion sensor, which would connect to the free pin header.

And yes, the footprint for the DC-DC module was wrong, which is why the module ended up flipped on the board. The circuit for the DC voltages underwent some debugging as well, as the emitter follower section didn’t allow voltages to go down to zero. These are more examples of a design being rushed through without full validation.

As mentioned, the initial goal was to just put some sensors around the building and store the data that got reported over MQTT. Next the idea to also read out the EEPROM of the coffee machines was added, which required custom hardware to interface with these machines. That escalated into controlling these machines, followed by the idea to monitor motion in meeting rooms, and finally to also control the AC units.

Obviously using the ceiling-mounted node for the coffee machines wouldn’t be very practical, as all that was needed there was a discreet little box that could be put behind the coffee machines, connecting to the serial port on the back of the machine (!), which would also provide the “coffee node” with power via the 5 V line of the TTL serial connection.

All of these nodes then communicated via the same MQTT broker, with the data from sensors being integrated into the backend services. Thanks to the use of MQTT, any MQTT client was able to communicate with any other client, making for a highly flexible network. With the use of WiFi, it meant that nodes could be installed anywhere in the building, with sensors and actuators placed practically at will.

Reverse-Engineering Devices to Control Them

And then came scope creep — the coffee machine controllers would control the building’s air conditioning. While the coffee machine interfacing was a decidedly ‘for the heck of it’ kind of affair, non-functioning air conditioning would be highly disruptive and unpleasant. On the other hand, the original motivation for rebuilding the AC controls was also that the current system did not work properly, with parts of the room freezing while others were sweltering, mostly due to a lack of granular temperature monitoring and control. So there was much room for improvement.

These requirements meant that we just had to reverse-engineer the entirety of the air conditioning system, figure out a location to mount the new hardware in each room, interface with the air conditioning system, write and expand backend services to support the new functionality, update the firmware on the ESP8266 nodes to do all of the new things, develop custom hardware where needed, and integrate everything into a package that even the average software developer could use in terms of end-user controls.

Since the original node PCB could not accommodate all of the valve controls with already four GPIO pins used for the fan controllers of four FCUs, a GPIO expander would be needed for toggling four relays that would control the 24 VAC thermal expansion valves. Here, the I2C breakout header on the node PCB came to the rescue. To control the FCUs, we connected an external PCB that combined an MCP23008 GPIO expander with an ULN2003.

For the switches that toggle the building sections between cooling and heating, another ESP8266-based board had to be developed that would be able to select the desired mode based on commands from the backend. Since we didn’t need to change that often, a latching relay was the perfect solution. It retains the last state even on power loss, and with two sides, the second side could be used as a single-bit memory cell. This led to this beautiful bodge of a prototype.

A fun feature of this board is that it directly connects to 230 volt mains power using an AC-DC converter module. The entirety of this board would then get stuffed into the location where the original switch was mounted, with a mains power line being run towards it inside the wall.

The fun part of reverse-engineering like this is that you never really know what you may encounter during the process. If we had first done the reverse-engineering, then made a plan and implemented it, we could have skipped many of the frantic changes and planning. The moral of this section thus obviously has to be that it pays to do the reverse-engineering up front where possible, rather than half-way during the development process. But when that’s not possible, start making break-out boards.

What Worked and What Didn’t

With the hardware more or less finished, it got installed over the course of a few weeks on the first three FCUs and the section’s AC mode switch. Installing the hardware turned out to be complicated and error-prone, with many points where installation could have been simplified. This included things like the screw terminals on the ceiling-mounted PCBs, which are a pain to work with while standing on top of a ladder.

Ideally ceiling-based installations would just involve drilling a hole into the suspended ceiling, leading a cable through it and plugging it into the node before attaching the latter to the ceiling. The less labor one has to perform while installing the hardware, the better. Removing the possibility of switching wires during installation is also highly desirable. This means big, clunky connectors that are easy to install without tools.

During development of the base node board, back when it was a simple temperature monitor, I had decided to have an I2C header on the board, just in case we needed it. This turned out to be a lifesaver. It allowed for the easy extension of this hardware to fully control the FCUs before we even knew how they even worked or what we would need. Having expansion interfaces easily available like that is essential. Plan for the unplanned.

Sticking to a single hardware design for all nodes, except for the coffee nodes worked out pretty well, as they could be rapidly deployed after finishing the prototyping phase. This let us use a single, unified firmware image, with modules that could be turned on and off for specific functionality. This kept the code base small and organized, and the build- and update process simple, not having to manage which image fit for which node.

The nodes were also configured for over-the-air (OTA) updates, with the rBoot bootloader switching between two ROM slots. This mean that trying out new firmware images was relatively easy, not requiring one to take down the ceiling-based nodes, or fishing the nodes from behind the coffee machines every time an update was available.

Still, trying out new changes was quite involved, with the OTA firmware image having to be pushed to a HTTP server, then the MQTT trigger command sent, followed by checking and validating that the node had properly updated itself and that the changes actually had the expected effect.

The AC control back end was based on the C&C service, with a closed feedback loop implemented for the FCUs and the temperature sensors. With no time left to write anything fancy, a straight-forward loop was used, which then got debugged over the course of weeks, involving countless OTA and backend updates, prodding at hardware while on top of a ladder, and other assorted, muscle-pain inducing fun.

Where to Go From Here

Looking back on the project, it was a good learning experience. If I had to do it again, knowing what I know today, however? There would have been a set of clear goals, with a list of hardware devices to be developed, along with the firmware and backend requirements. I would also have started on the integration test for the software, using a virtual hardware platform and building. The current version of the BMaC platform has since gained exactly this kind of test, which should make any future development much easier. A few years too late for me, but just in time for you!

The BMaC project is still in flux. Hopefully a new opportunity will arise to test out the changes in a new office environment or similar.