Configuring a Read-Only Web Interface for Orchestrator

In the MySQL ecosystem, orchestrator is the most popular and well-respected high availability and topology management tool, integrating well with other solutions such as ProxySQL . It facilitates automatic (or manual) discovery, refactoring and recovery of a replicated MySQL environment, and comes complete with both command-line ( CLI) and web interfaces for both humans and machines to interact with.

As we all know, humans are prone to errors and as such accidents can happen, particularly when humans and computers interact with each other! Recently, one of these situations related to the web interface of orchestrator during topology refactoring with its drag-and-drop capabilities, where a drop occurred unintentionally and thus had an impact on replication.

When using orchestrator in CLI mode it is possible to set the ReadOnly flag to make the web interface and API unable to make changes on the user’s behalf, which would avoid such accidents. However, what if you want to maintain access to the API for management? Thankfully, there are a number of settings that already exist that when combined together with a reverse proxy (such as NGINX ) make this possible.

Configuration of Orchestrator to Support a Read-Only Web Interface

The configuration of orchestrator is well documented, so if you are unfamiliar with configuring it then its documentation is the best place to start.

In order to make the web interface read-only you will need to use the following configuration options:

AuthenticationMethod
AuthUserHeader

There are a number of choices for the AuthenticationMethod and to achieve the goal of a read-only interface we will set this to:

    "AuthenticationMethod": "proxy"

This will instruct orchestrator to expect authentication to occur via HTTP headers, which brings us to the second setting option, AuthUserHeader . For the purpose of this post, the header name will be set to X-Auth-User , although you are able to set this to whatever makes sense to you, so long as it can be used as a header name.

    "AuthUserHeader": "X-Auth-User"

For the sake of simplicity, we will also set the PowerAuthUsers option to match only the HTTPAuthUser , which allows us to easily revert back to direct full access with "AuthenticationMethod": "basic" should we wish to do so.

    "HTTPAuthUser": "orchestrator",
    "PowerAuthUsers": ["orchestrator"]

There are some other options that we will now set, not relating to authentication, so that we can choose the path via which the web and API interfaces of orchestrator will be accessible in a browser, etc:

    "URLPrefix": "/orchestrator",
    "StatusEndpoint": "/orchestrator/api/status"

To help with configuration of NGINX we will make orchestrator bind to a local address:

    "ListenAddress": "127.0.1.1:3000"

Configuration of NGINX to Support Orchestrator

We will assume that we already have a working NGINX configuration and just need to make the necessary changes to provide access to orchestrator . The ngx_http_core_module enables configuration of a location rule to match the value of the URLPrefix that we chose for orchestrator . Here is the full snippet that we will use:

    location /orchestrator/ {
        auth_basic            "orchestrator";
        auth_basic_user_file  /etc/nginx/orchestrator.htpasswd;
 
        proxy_pass_request_headers on;
        proxy_hide_header Authorization;
 
        set $orchestrator_auth $remote_user;
        if ($uri ~ /web/) {
            set $orchestrator_auth "readonly";
        }
 
        proxy_set_header X-Auth-User $orchestrator_auth;
        proxy_pass http://127.0.1.1:3000;
    }

This configuration instructs NGINX to use:

• The Basic HTTP Authentication Scheme using the contents of /etc/nginx/orchestrator.htpasswd as the source for credentials.
• Forward request headers to orchestrator , except for the Authorization header that would normally be sent as a result of the client authenticating.
• Dynamically set the value of a variable ( $orchestrator_auth chosen for the example) to either the authenticated user, or force it to readonly for requests to the web interface ( /web/ ).
• Set an HTTP header named X-Auth-User (our AuthUserHeader ) to have a value of $orchestrator_auth .
• Pass all requests on to the service at 127.0.1.1:3000 (our ListenAddress ).

Create the auth_basic_user_file

In order to be able to use authentication, it will be necessary to add at least one user to the auth_basic_user_file. The easiest way to manage authentication user files is with the htpaswd tool.

# CentOS
$ sudo yum install httpd-tools
 
# Ubuntu
$ sudo apt-get install apache2-utils

The following commands will create the file ( -c ) and add the power user “orchestrator” and then add another user, “ readonly “, both using the more secure bcrypt encryption: ( -B ).

$ sudo htpasswd -Bbc /etc/nginx/orchestrator.htpasswd orchestrator orchestrator
$ sudo htpasswd -Bb /etc/nginx/orchestrator.htpasswd readonly readonly

Apply Configuration and Test

In order to activate this, we will need to either reload the configuration of each service that was running or start those that were not. We will presume that both orchestrator and NGINX were already running, so apply the configs to the running instances:

# orchestrator
$ sudo /usr/bin/pkill -HUP -x orchestrator
 
# NGINX
$ sudo nginx -t && sudo systemctl reload nginx

We will expect requests going directly to orchestrator to fail, while requests going via NGINX will succeed.

# Testing direct access
$ curl --basic --user orchestrator:orchestrator \
  http://127.0.1.1:3000/orchestrator/api/async-discover/192.168.0.2/3306
{"Code":"ERROR","Message":"Unauthorized","Details":null}
 
# Testing proxied access
$ curl --basic --user orchestrator:orchestrator \
  http://127.0.0.1:/orchestrator/api/async-discover/192.168.0.2/3306
{"Code":"OK","Message":"Asynchronous discovery initiated for Instance: 192.168.0.2:3306","Details":null}

If you were to now use the web interface via NGINX and authenticate using the power user, you would see that the user would be displayed as “ readonly ” and the icon indicating that the interface is in read-only mode would show up.

We Still Have an Issue

This is not 100% fool-proof as there is still a way to circumvent it, although it would require access to the server that is hosting orchestrator and NGINX. However, it prevents accidents from happening as you would have to be rather determined to bypass the reverse proxy to gain write access for the web interface.

I have sent a pull request to add a new option to orchestrator that would simplify the whole process and also reach the 100% read-only goal, but for now, it is necessary to take these extra steps.