How to turn an ASP.NET Core appsettings.json file into a fast-read database

This article describes a way to create a database using ASP.NET Core appsettings.json configuration feature. The big benefit of this approach is the read is blistering fast, something like 1,000 times quicker than a SQL Server database query. The downside is the write is relatively slow (e.g. >1 ms write) so this approach is best for situations where you have lots more reads than writes. I use this approach in an ASP.NET Core applications where certain data are read on every HTTP requests.

TL;DR; – Summary of this article

• There is a way to use ASP.NET Core Configuration feature to create a type of database that has is much faster (~1,000 times faster) read than a typical database, but the write is slower than a database (small database = ~2 times slower, 400kb size database = ~10 times slower).
• A good use for this type of database is where you have lots of reads and rare writes. I created this approach to handle a database query on every HTTP request.
• This article describes the five steps to implement a database using ASP.NET Core Configuration feature.
• There is a library called Net.DistributedFileStoreCache (shortened to FileStore cache) which provides a pre-build version of this approach. You might like to refer to these articles:
  • A .NET distributed cache with a ~25 nanosecond read time!
  • How to change/move databases on a live ASP.NET Core web site, which uses the FileStore cache as a database.

Setting the scene – why did I use an appsettings.json file as a database?

My AuthPermissions.AspNetCore library (shortened to AuthP) provides back-end code for building multi-tenant applications using ASP.NET Core and EF Core, and in version 3 of the AuthP library I added support for sharding. To implement sharding I needed to create a connection string that points to the database server+database on every HTTP request from a tenant user.

Also, there were couple of extra features that I wanted my sharding implementation to support

• Should work with Azure’s SQL Server elastic pools. Azure elastic pooling provides a cost-effective way to have lots of databases (see this document on why this is useful).
• Support geographically placed database servers to improve performance when you have users that are geographically spread out.
• Good security: the connection strings contains Username/Password of the servers, so for security reasons I store the database strings in Azure.

The diagram below shows how the implementation of the sharding is changed to support these three extra features:

The sharding code gets the sharding data for a specific tenant which contains the name of the connection string linked to a database server and the name of the database on that database server. From these two parts it forms the composite connection string needed to access the tenant database. This isn’t that complex procedure, but it does to read in the sharding data (name of connection string and name of the database) on every HTPP request for a tenant user.

I could go with a database access, but I’m really trying to make this library very fast, so I started to look at ASP.NET Core Configuration features i.e. appsettings.json and IOptions because I know that the read of configuration data is really fast because the data is cached by ASP.NET Code configuration code.

Therefore, I created a appsettings.json type file which I could update and then used the Configuration IOptionsSnapshot<T> method to get the latest data my settings file. (see Microsoft Options Pattern docs for more info on this type of read). The diagram below shows the whole process.

This might seem very complex, but it’s:

1. Very fast: something like 1,000 times quicker than using a database access.
2. Secure: The connection string isn’t in any of your code or your claims.
3. Doesn’t use a database: This means your tenant databases won’t be slowed by lots of small queries.

The rest of this article describes the steps needed to create a generic fast-read database by using ASP.NET Core Configuration feature. In the steps I show examples from the sharding feature described above, with links to the code in the AuthP’s open-source repo. That way you have working code examples of how I used this approach.

Steps to turn an appsettings.json file into a fast-read database

NOTE: I refer to the json file which will be used as database as the database json file in this article.

The steps are to implementing a database json file are:

1. Create a json file to act as a database
2. Make sure the database json file isn’t overwritten
3. Register the database json file on startup
4. Write to the database json file within a distributed lock
5. Use IOptionsSnapshot<T> to read the database json file

1. Define a json file to act as a database

The first thing to do is work out what data you need to store the database json file for your application. Once you have decided on the data you need, then you must implement the dats by using class(es) that contain parameters that can be serialized / deserialized to json by the .NET Text.Json library.

For my sharding settings I have a List of the class called DatabaseInformation, which has four properties (all of type string) that define a specific settings of each sharding server+database. The code below shows the type of json the sharding settings file would contain.

NOTE: The name of the section / array used in your database json file must be unique across all the configuration json files.

2. Make sure the database json file isn’t overwritten

A normal appsettings.json file gets overwritten when an application is deployed. But because we want to use json file as a database, then you don’t want the file to overwritten. I do two things to make sure the database json file isn’t overwritten.

First, I use a filename which includes the environment name, e.g. Debug, Staging, Production, so my implementation the filename is $“shardingsettings.{EnvironmentName}.json”. This means that filename used developing the application in Debug mode can’t overwrite your Production database json file.

But the most important thing to do (but easy to forget) is to set the file’s “Copy to Output Director” property to “Do not copy”. This stops the database json file being copied in your deployment. You can manually set this via file properties, but I prefer to add a ItemGroup to the ASP.NET Core .csproj file, as shown below.

3. Register the database json file on startup

There are two parts to registering database json file on startup. They are:

1. Register the database json file to the ASP.NET Core’s Configuration
2. Register your IOptions access via the Configure<T> service

3.1 Register the database json file to the ASP.NET Core’s Configuration

To register your database json file to be part of the Configuration you use the AddJsonFile method. The code below goes in the Program class and registers my shardingsettings file.

You need to think what happens when you first deploy using a database json file. In this case I set the optional parameter to true, which means the application can start without the file. If you use IOptionsSnapshot<T> (see next subsection on IOptionsSnapshot<T>) it will return null if the database json file isn’t there or doesn’t have any json in it, but once you create the file the application will start tracking the file and the IOptionsSnapshot<T> Value will be non-null.

NOTE: The other approach is set the optional parameter to false and ensure that there is a json file exists. But if the optional parameter is false, then if the json file isn’t there, then the application will fail on startup. This means you need to create on startup if no file exists.  

3.2 Register your IOptions access via the Configure<T> service

You must register a Configure<T> service, where T is the class which defines the json content of the database json file, to allow you use the IOptions access to the data inside. This is done by registering a class to a specific part of the configuration setting via a class.

In my shardingsettings file I use a collection of data, so my options class looks like this:

And configured by the code below, which will look for a json array in all the registered json files with the name of ShardingDatabases.

4. Write to the database json file within a distributed lock

To update the data in the database json file you need to read in the current json, add your change and write back out. This read->update->write process is fairly easy to implement – Have a look at my AccessDatabaseInformation class for an example of what this would look like.

While the update part of the code is straightforward, we do need to handle simultaneous updates, because one update could overwrite another update. This type of simultaneous updates is rare, but because they can occur, we need to handle this. This means we need to wrap the update process with some code that would stop other updates from running until the current update has finished.

If you are only running one instance of your ASP.NET Core application, then you could use a .NET lock. But my library is designed with high performance applications where multiple instances of the application are running at the same time (Azure calls this scale out), so I need a distributed lock. I use an excellent library called DistributedLock.  

The DistributedLock library uses a global resource, such as a database, to form a lock across all the running instances. The code below (adapted from the Acquire section of the DistributedLock Readme)

5. Use IOptionsSnapshot<T> to read the database json file

Finally, you can access the information in the database json file via ASP.NET Core’s IOptionsSnapshot<T> method. The code below is a simplified version of the AuthP’s ShardingConnections constructor. When the ShardingConnections service is created it uses the IOptionsSnapshot<T> method to get the data in the database json file, in this case my sharding settings file. (see Microsoft Options Pattern docs for more info).

As I showed in the setting the scene section using the IOptionsSnapshot<T> method in the code below reads in the current sharding settings.

Note that if the sharding settings file doesn’t exist the IOptionsSnapshot<T> Value will be null, and you need to work out in that case. You could return the null, but often the best solution is to create an empty collection or similar. In the AuthP’s sharding settings shown above a new deployment always has a single, default DatabaseInformation, which is formed from the multi-tenant setup information contains.

Conclusions

Creating a fast-read / slow-write database using ASP.NET Core’s Configuration / IOptionsSnapshot<T> might not be the first approach you would think of for creating a database, but in situations where you want a very fast read where the data changes rarely. For instance, the AuthP sharding feature is a very good fit to this approach because it needs two read queries (one to get the sharing data and another to get the database server connection string) on every HTTP tenant user request read with rare changes to the sharding data.

I also created a library called Net.DistributedFileStoreCache (shortened to FileStore cache) which implement a Distributed cache. This library uses the approach that ASP.NET Core Configuration / IOptionsSnapshot<T>  but uses .NET’s FileSystemWatcher class instead to IOptionsSnapshot<T>. The FileStore cache has a ~25 ns. read time and a write time > 1 ms. – see the FileStore cache full performance figures here.

I use the FileStore cache in database mode in the article “How to change / move databases on a live ASP.NET Core web site” because this feature needs multiple reads on every HTTP request. Using FileStore cache removes the extra ~1 ms. that a database might have.

If nothing else you have learnt more about ASP.NET Core’s Configuration / IOptionsSnapshot<T>, and you have learnt a new way to store data with a different performance from a normal database.

Happy coding.