This Is What Rails Testing Is For, Right?

Yes, and no. It is often difficult to gain complete confidence in code changes before deployment. The unit and system tests pass. It's good to go, right?

The reality is that there is no substitute for the real world, i.e. production. What if the data quality is bad or tests are missing? How can you know if the new software will perform well enough to handle production throughput?

Teams with public services sometimes find that they need to deal with “bugwards compatibility” issues. When a bug has existed in production for a while, clients may code in a way that depends on consistent incorrect behavior. Customers often use software in unexpected ways.

Observe Production Changes in Ruby and Rails with Scientist

If production is the best place to gain confidence in a change, then consider observing how code behaves there. This may sound scary at first, as the idea of “testing in production” contradicts classic software engineering practices.

However, the good news is that it’s easy and safe to do so in Ruby and Rails using the Scientist gem. Scientist's name is based on the scientific method of conducting experiments to verify a given hypothesis. In this case, our hypothesis is that the new code does the job.

The reason we can safely take this approach stems from the fact that experiments still use the result of the existing code. New code is only evaluated for observation and comparison purposes, both for accuracy and performance. We mitigate the test coverage concerns discussed earlier by evaluating performance using real-world data and parameters. Experiments typically evaluate a chosen sample rate of requests to minimize the impact on production. However, you can evaluate every request if desired.

Let's now take a quick look at how Scientist works in a branch by abstraction way.

The Branch by Abstraction Pattern in Ruby's Scientist

Scientist's approach begins with the Branch by Abstraction pattern described by Martin Fowler as making “a large-scale change to a software system in a gradual way.”

We introduce an abstraction layer to isolate the code being updated. This layer decides which implementation to use so that the experiment is transparent to the rest of the system. The technique is related to using a feature flag that determines the code path.

The Scientist gem, which originated from Github, implements this pattern using an experiment. The existing code is referred to as the control, and the new implementation is the candidate. Both code paths are run in randomized order, but only the control result is returned to the client.

Using Scientist to Refactor a Ruby Service

Consider a Ruby service that returns the largest prime factor for a given number. Assume that we've identified optimizations to prune the required set of candidates, speeding up the service.

However, service owners want to be sure no bugs were introduced. They also want to observe any performance improvements. Introduce the following code, modifying clients to call this method:

require 'scientist'
 
def largest_prime_factor(number)
  science "prime-factors" do |experiment|
    experiment.use { find_largest_prime_factor(number) }     # old way
    experiment.try { improved_largest_prime_factor(number) } # new way
  end  # returns the control value
end

At this point, only the use (control) expression is invoked. To make the experiment worthwhile, define a custom Experiment class to enable it (100% of the time below) and publish the results (in this case, just logging). Scientist generates fantastic data but it doesn’t do anything with it by default. That part is left up to you.

require 'scientist/experiment'
require 'pp'
 
class MyExperiment
  include Scientist::Experiment
 
  attr_accessor :name
 
  def initialize(name)
    @name = name
  end
 
  def enabled?
    true
  end
 
  def raised(operation, error)
    p "Operation '#{operation}' failed with error '#{error.inspect}'"
    super # will re-raise
  end
 
  def publish(result)
    pp result
  end
end

The results of the experiment will be logged, and we can make improvements over time based on the feedback. Once the new code meets the requirements and confidence is high, accomplish cutover to the new implementation by simply replacing the science code with a delegation to the new implementation.

LabTech Simplifies Scientist Experiments in Ruby on Rails

You can use the LabTech gem in your Rails application to easily configure Scientist and handle the results.

Applications that use AppSignal can use the Appsignal.instrument custom instrumentation helper to track how long Scientist events take to complete. Wrap it around the different experiment code blocks to see events appear in the performance timeline.

Now, going back to LabTech — the web page below simply accepts a number to factor.

Getting started is easy provided you have access to the console. First, add the LabTech gem to your Gemfile and run a bundle install.

gem 'lab_tech'

Tables store results and experiment configuration, so run a database migration.

rails lab_tech:install:migrations db:migrate

The abstraction layer is the same, except the LabTech module is used. The full code is available on GitHub.

def largest_prime_factor(number)
    LabTech.science "prime-factors" do |experiment|
      ...
    end
end

At this point, the experiment is disabled, so use the console to enable it in all cases or for a percentage of the time.

bin/rails console
LabTech.enable "prime-factors"
LabTech.enable "prime-factors", percent: 5

We can now run tests and the experiment will be evaluated. For a textual view of results, use either of the following commands from the Rails console.

LabTech.summarize_results "prime-factors"
LabTech.summarize_errors "prime-factors"

After a few successful runs and one manufactured error, here is an example of what the results summary looks like. There is an overview of successes and failures, as well as an ASCII chart showing performance differences.

--------------------------------------------------------------------------------
Experiment: prime-factors
--------------------------------------------------------------------------------
Earliest results: 2022-04-27T02:42:45Z
Latest result:    2022-05-01T17:27:39Z (5 days)
 
3 of 4 (75.00%) correct
1 of 4 (25.00%) mismatched
 
Median time delta: +0.000s  (90% of observations between +0.000s and +0.000s)
 
Speedups (by percentiles):
      0%  [                         ·         █               ]    +2.4x faster
      5%  [                         ·         █               ]    +2.4x faster
     10%  [                         ·         █               ]    +2.4x faster
     15%  [                         ·         █               ]    +2.4x faster
     20%  [                         ·         █               ]    +2.4x faster
     25%  [                         ·         █               ]    +2.4x faster
     30%  [                         ·         █               ]    +2.4x faster
     35%  [                         ·         █               ]    +2.4x faster
     40%  [                         ·         █               ]    +2.4x faster
     45%  [                         ·         █               ]    +2.4x faster
     50%  [ · · · · · · · · · · · · · · · · · █ · · · · · · · ]    +2.4x faster
     55%  [                         ·         █               ]    +2.4x faster
     60%  [                         ·         █               ]    +2.4x faster
     65%  [                         ·         █               ]    +2.4x faster
     70%  [                         ·                        █]    +6.9x faster
     75%  [                         ·                        █]    +6.9x faster
     80%  [                         ·                        █]    +6.9x faster
     85%  [                         ·                        █]    +6.9x faster
     90%  [                         ·                        █]    +6.9x faster
     95%  [                         ·                        █]    +6.9x faster
    100%  [                         ·                        █]    +6.9x faster
--------------------------------------------------------------------------------

The Blazer gem provides a nice way to analyze the results easily. It is simple to install and allows SQL queries to run against tables. The query here shows that the candidate implementation is significantly faster than the original.

In the example prime factoring service, the speedup in the improved implementation comes from a heuristic that eliminates some possible factors to consider. As we consider higher numbers and find a prime factor, we can stop searching after we get to our target number divided by that factor. The new code path only adds one statement to accomplish this.

We can also see the reduction in execution time using a Blazer query against the LabTech tables.

Use Cases and Limitations of Scientist

Optimal use cases for Scientist include searches, calculations, and code that has no side effects. Code that includes transactional updates or external integrations such as email does not fit cleanly into the model because the capability is run twice (both the old and new implementations).

This is not a trivial limitation, as it does eliminate several use cases. However, there are some workarounds if the experiment is critical to your success. Consider whether the side effects are relevant or if duplication is an issue. For example, it may not matter in some cases whether two emails are sent during the evaluation. Another option is to have the new code determine the result but not persist it. This would prevent any meaningful performance comparisons. However, it would allow you to verify accuracy.

Other limitations stem from Scientist’s focus on return values. In some cases, valid results may exhibit differences over time, whether they simply include timestamps in the response or certain factors vary. In many cases, we can write custom comparison logic in the experiment to verify accuracy beyond basic string comparisons.

Finally, a limitation of LabTech is that it has not been ported to Rails 7 as of the time of writing.

Best Practices for Effective Scientist Experiments in Rails

Consider these items when implementing your experiments:

• In Rails projects, Scientist can either be configured in an initializer or a wrapper like the Rails LabTech gem. Most Rails applications already have a database, so LabTech leverages ActiveRecord to store results.
• To avoid slowing down development and testing, enable your experiment only in staging and production environments.
• To minimize any potential impact on production, only run the experiment on a percentage of requests. LabTech supports this out of the box as an optional parameter when you enable the experiment (it is initially disabled by default). Using pure Scientist, this logic is easy to code in the experiment’s enabled? method.
• Some logic is resource-intensive, so a low sampling rate may be a good place to start. As you gain confidence with the results, ramp up the percentage of requests being evaluated.
• You can add context attributes to get the most from your results. The experiment context can be set to a Symbol-keyed Hash of data that is then made available in published results, e.g.:

experiment.context :user => user

Wrap-Up: Observe and Monitor Your Ruby App with Scientist

In this post, we explored how to use the Scientist gem to change, migrate, and refactor Ruby code in production.

We examined Scientist's origins in the Branch by Abstraction pattern, then dived into refactoring. Next, we saw how LabTech could help with gathering results and your Scientist configuration.

We then touched on some limitations of Scientist before finally outlining a few best practices.

You must observe and monitor what is happening in your system. Integrate Scientist into your development process to make critical changes in your Ruby code with greater confidence.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!