Multi Tenant MQTT broker
source link: https://www.hardill.me.uk/wordpress/2024/03/23/multi-tenant-mqtt-broker/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Multi Tenant MQTT broker
I’ve talked about doing hostname based proxying of MQTT in the past as a way to host multiple MQTT brokers with different hostnames behind a single IP address. To make this work I had to use MQTTS (MQTT over TLS) and the clients has to support SNI.
SNI allows a load balancer/reverse proxy to determine which back end broker instance to forward the connection to.
What I’m going to talk about here is multiple groups of users accessing a single broker but each groups messages are partitioned from other groups even if they use the same topics.
This means that any client can connect to the broker via MQTT, MQTTS or MQTT over WebSockets.
For this project I’m going to use the mosquitto broker. I had noticed that the instruction on the test.mosquitto.org test broker had recently changed. In the past it had explicitly blocked subscriptions to # (the global wild card topic) because of the load it places on the broker (test.mosquitto.org handles a large amount of traffic from people just playing, OpenSource collaborative projects or those that make use of it as “free” internet accessible broker). The new instructions mentioned that if you passed a specific username when connecting to the broker it would allow subscriptions to # for 20 seconds, before removing that subscription. I asked Roger Light (mosquitto’s author/maintainer) about how he’d achieved this (as it is none standard behaviour) and he pointed me at the mosquitto develop branch where v2.1.0 is taking shape.
In this branch there are a number of example plugins, one of which provides this 20 seconds subscription behaviour, but also several others. One of these caught my attention called topic-jail.
This plugin would create 2 topic domains, if the client id was set to jailed it would alter all messages from that client with a topic prefix of jailed/ and would also strip that prefix when delivering any messages to that client. All other client ids would have their topics left alone. This looked like it could be modified to solve the multi-tenancy problem.
Multi Tenant Mosquitto
Rather than using the client id I decided to look at using the username the client connected with, or at least part of it. This makes sense because in a multi-tenant environment authentication is always going to be enabled in order to determine which team a user belongs to.
As a starting point I decided to use the following schema for the username: user@team I could then use a pretty simple regex to separate the two parts. That being said I also made the regex configurable to allow for more complex username patterns, the only stipulation is that the regex must only have 1 matching group that equates to the team name.
The following code is the plugin init function that reads the plugin options from the config file at startup and initialises the regex, with either the supplied pattern or the default (^[a-z0-9]+@([a-z0-9]+)$).
int mosquitto_plugin_init(mosquitto_plugin_id_t *identifier, void **user_data, struct mosquitto_opt *opts, int opt_count){int i, found = 0;UNUSED(user_data);mosq_pid = identifier;mosquitto_plugin_set_info(identifier, PLUGIN_NAME, PLUGIN_VERSION);for(i=0; i<opt_count; i++) {if (!strcasecmp(opts[i].key, "regex")) {regcomp(&username_match, opts[i].value, REG_EXTENDED);found = 1;}}if (!found) {regcomp(&username_match, "^[a-z0-9]+@([a-z0-9]+)$", REG_EXTENDED);}int rc;rc = mosquitto_callback_register(mosq_pid, MOSQ_EVT_MESSAGE_IN, callback_message_in, NULL, NULL);if(rc) return rc;rc = mosquitto_callback_register(mosq_pid, MOSQ_EVT_MESSAGE_OUT, callback_message_out, NULL, NULL);if(rc) return rc;rc = mosquitto_callback_register(mosq_pid, MOSQ_EVT_SUBSCRIBE, callback_subscribe, NULL, NULL);if(rc) return rc;rc = mosquitto_callback_register(mosq_pid, MOSQ_EVT_UNSUBSCRIBE, callback_unsubscribe, NULL, NULL);return rc;} |
The init function also sets up the callbacks for incoming messages, outbound messages as well as subscriptions and unsubscription events.
We will need a utility function to apply the regex to the username
static char* get_team( const char *str) {regmatch_t pmatch[2];char *team;size_t size;int result = regexec(&username_match, str, 2, pmatch, 0);if (result == 0) {size = (size_t)(pmatch[1].rm_eo - pmatch[1].rm_so);team = mosquitto_malloc(size+1);strncpy(team, str+pmatch[1].rm_so, size);team[size] = 0;return team;} else {return NULL;}} |
This is used in each of the callback functions to extract the team from the username and then either prepend this to the topic for the MESSAGE_IN, SUBSCRIBE and UNSUBSCRIBE events and to strip the prefix from the MESSAGE_OUT event.
static int callback_message_in(int event, void *event_data, void *userdata){struct mosquitto_evt_message *ed = event_data;char *new_topic;size_t new_topic_len;UNUSED(event);UNUSED(userdata);const char *username = mosquitto_client_username(ed->client);if (!username) {return MOSQ_ERR_SUCCESS;}const char *team = get_team(username);if(!team){/* will only modify the topic of team clients */return MOSQ_ERR_SUCCESS;}/* put the team on front of the topic *//* calculate the length of the new payload */new_topic_len = strlen(team) + sizeof('/') + strlen(ed->topic) + 1;/* Allocate some memory - use* mosquitto_calloc/mosquitto_malloc/mosquitto_strdup when allocating, to* allow the broker to track memory usage */new_topic = mosquitto_calloc(1, new_topic_len);if(new_topic == NULL){return MOSQ_ERR_NOMEM;}/* prepend the team to the topic */snprintf(new_topic, new_topic_len, "%s/%s", team, ed->topic);mosquitto_free((void *)team);/* Assign the new topic to the event data structure. You* must *not* free the original topic, it will be handled by the* broker. */ed->topic = new_topic;return MOSQ_ERR_SUCCESS;} |
All the code can be found here
Configuring and Testing
Once the plugin is built it can be configured like this:
port 1883plugin /path/to/mosquitto_multi_tenant.soplugin_opt_regex ^[a-z0-9]+@([a-z0-9]+)$allow_anonymous falsepassword_file passwd |
This example uses the built in password file for authentication, but it can be paired with one of the other authentication plugins.
We can create a bunch of test users with the mosquitto_passwd command
$ mosquitto_passwd -c passwd adminPassword: Reenter password:$ mosquitto_passwd passwd user@fooPassword: Reenter password:$ mosquitto_passwd passwd user@barPassword: Reenter password: |
You would then connect as the user user@foo using mosquitto_sub as follows:
$ mosquitto_sub -u user@foo -P password -v -t '#' |
The admin user is not a member of any teams so can publish to all topics including to a specific team by prefixing the topic with the team name.
$ mosquitto_pub -u admin -P password -t foo/hello -m world |
This will be delivered to the members of team foo on the topic hello
I have a couple of use cases for this, but it will need some proper testing and there might be some performance improvements (not having to run the regex on every event and finding a way to cache the result somewhere).
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK