By Remco van Bree and Alex Hartwell

At Lyft Bikes and Scooters we care deeply about our app, but the app isn’t the whole story. If it doesn’t interact with our bikes, stations and scooters out in the real world properly the end user experience falls flat and it doesn’t matter how good the app is. At Lyft we work hard to automate testing of a lot of our software flows, largely through UI tests that mock out server responses. This only gets us so far, which is why we think it’s important to have robust abilities to test things end to end in an exploratory mode, because hardware is messy in ways software isn’t.

This article is a history of how we have thought about testing golden path flows that interact with hardware and how we found an approach we are really happy with.

Stations in a box

Four or five years ago, before Lyft got into bikeshare and acquired Motivate, Motivate had built stations in a box to facilitate end to end testing. They were pretty awesome; they allowed you to test against a real station without the footprint of a real station. All the smart insides were put into a toolbox with a single triangle/dock on top. This had a few problems, but overall this approach was pretty good, since it was real station hardware and software tests were extremely realistic.

Our docked bikeshare stations consist of three components:

1) A kiosk hooked up to a cellular network that controls all the docks at a station with credit card reader and screen to facilitate walk-up rentals

2) A dock that will lock up a bike when it’s not in use

3) A metal triangle at the front of the bike that gets captured by the dock. The bike and dock identify each other through the triangle

It had some major problems though, you needed to have one available; there was only one in the NY office, and it was connected to production servers (with logic to not expose it to the general public). This lack of access was a major blocker to this being the be all end all solution; it just didn’t scale. If you were working from home, or someone else had to do some testing you had to wait. The fact that the station was connected to our production environment was problematic as well; it wasn’t possible to verify features before rolling them out, one couldn’t even run smoke tests on staging before changes went out. We needed something better, so we built it.

Station Simulator

Next we developed the station simulator, a software tool that emulated a full station setup (both hardware and software) on a developer’s machine.

The station simulator was a response to the shortcomings of the station in a box. It solved the scalability and environment problems that the station in a box had. Anyone could download an executable file and set up a simulated station on their computer.

In theory this is great, but in practice it took a lot of work to set it up and maintain. A station is a complex beast with lots of functionality, most of which Mobile and Quality Engineers don’t need to care about in their day to day work. They had a steep learning curve and required you to know a lot about the internal workings of stations and the backend systems supporting them. They weren’t something you could download and just start testing with; you needed to read through a lot of documentation and learn a lot about the underlying system. We relied on a mixture of station simulators and stations in a box for a few years, up until the Lyft acquisition of Motivate.

Python commands

When Lyft acquired Motivate, we began an org-wide effort to unify our two systems: The Motivate bikeshare system, and Lyft’s scooter-share system. This led to a lot of fragmentation in the short term, with sources of truth split between the Motivate systems and Lyft systems. At this point it became even harder for mobile app developers and testers to test rides because our architecture temporarily got a lot more complicated. Moreover, we now also needed to support dockless ebikes and scooters that communicated with our servers directly, which meant that we had a lot more vehicle states to display and test in our apps.

At this point we started relying on one-off python CLIs (command-line interfaces) that simulated webhooks coming from the legacy system. This way we could ignore the complexity of some of our legacy systems and manually send updates to progress ride state while testing.

This approach worked well; it was way more reliable than the previous approaches since it didn’t rely on real hardware or complex simulated hardware. There was no complicated setup; you could just ssh into staging and run commands that make API calls pretending to be the not-yet-unified systems.

The biggest problem was a lack of abstraction; the APIs weren’t necessarily designed in the way you’d think about taking a ride, but in a way that made sense for the integration. It required whoever was controlling the ride to know the internal implementation details of both the legacy bikeshare system and the Lyft bikeshare microservices. This is a pretty big ask for mobile and quality engineers who just want to take a ride and make sure the app handles the client side of things properly.

Additionally, because all of these commands required hard to type and memorize values like user IDs, rideable identifiers and coordinates, it was super hard to quickly execute varieties of scenarios that were tailored to a specific test case (or even switch test users). But regardless, it was easy enough and we relied on this for a while, until unification had come along far enough that it made sense to create something better.

Lyft-dev-cli and lastmile-cli

Lyft had already solved this problem of making end to end exploratory testing easier on the rider and driver side with lyft-dev-cli, an interactive typescript cli that made it easy to take rideshare rides in staging. We knew we wanted something similar and instead of trying to reinvent the wheel, we integrated directly. This CLI is extremely full featured (loading states, multiselect, autocomplete, etc) and, better yet, it runs on the engineer’s local machine so they won’t need ssh access.

This allowed us to build out flows that abstract away the underlying implementation from users, polling “ride state” during transitions and showing engineers valid next steps only after those transitions are finished. No longer did one need to know what the different webhooks look like, just choose “unlock” from the list of options and your bike unlocks. For some ride types, like scooters, there is no user interaction in the CLI necessary for the golden path, it just moves the ride forward for you, 🎨.

Additionally, before the CLI, we generally only tested the most common scenarios (the “golden path”) consistently, since finding the correct data and commands/webhooks for all possible variations was complicated and tedious. For example, if you wanted to test what happens when you park a bike in a no parking zone, you would need to find a no parking zone on the map, open up Google Maps and visually identify the same location, copy the coordinates from Google Maps and then paste that coordinate into the terminal as part of a command. With the CLI all that data gathering is done for you.

This effort was mostly undertaken for the benefit of mobile, server and quality engineers but had the unintended side effect of making testing in staging possible for all team members, including product managers and designers and has helped new features get fast feedback by allowing everyone in the team to play around with really early versions before it is available in production.

Conclusion

When you think about test automation, generally the first thing that comes to mind is automating end to end tests, but when dealing with complex systems, the biggest pain point is often getting systems to a state where you can begin to explore, debug or develop against the feature that you wish to modify, so oftentimes it pays off to invest in test tooling that simplifies state management.

Testing software that interfaces with hardware is really hard, but by defining good boundaries and really thinking about the ergonomics of your tooling, you can make testing a lot more accessible and improve your team’s productivity and velocity

Acknowledgements

We would like to extend huge thanks to the test tooling team for building the foundation for Lastmile CLI. The excellent developer ergonomics of the existing lyft-dev-cli system allowed us to stand on the shoulders of giants, and focus our efforts on building for our micromobility-specific use cases. We would like to especially thank PG Herveou, who helped us refine our designs and supported us through building this.

As always, we are hiring. If you are interested in building the future of micromobility, join our team.