Introduction
Multitenant products have become quite commonplace nowadays. Any B2B product deployed on the cloud needs to have some kind of multitenancy. The exact kind depends on the business case and the product infrastructure and development team structure. You have a bunch of options, each with its reason to exist.
Multitenancy provides isolation of data or processing (or both) of one tenant from others. The purpose of this isolation could have multiple reasons. It could be for better debuggability, providing access to tenants to parts of the data/infrastructure they care about, differently allocating resources to tenants, or for invoicing.
In this post, we talk about the different areas of an application, and how multitenancy manifests itself in each one of them.
The frontend server
Many workloads are frontend heavy. Such as e-commerce solution where each seller can Whitelabel their site. Hosted content-management solutions. If you are working on such a solution, then you have a few problems to deal with.
- Isolation of issues to specific tenants — You might not want high traffic on one site to disrupt others.
- Provision for additional infrastructure for specific tenants — You might want to measure how much each tenant uses, or provide a better experience to paying customers.
- Separate logging, which might need to be exposed to tenants directly.
Usual patterns
- One fat frontend — If you are not working on a frontend heavy workload, this is the best way to go. The one fat frontend means you still can scale (horizontally, or vertically based on what your preferences are).
- One or more virtual machines per tenant — If all your tenants are decent sized to take up space of at least one virtual machine, have a lot of frontend customization, then providing virtual machines per tenant is a decent approach. Logs are automatically partitioned, machine stats are easy to debug per tenant, and billing is easy.
- One or more containers per tenant — If you still need all the isolation that approximately equals a virtual machine, but have many small tenants, this might be a more attractive approach.
In any of these cases, keeping user state out of the frontend is usually a good idea. This is true even without multitenancy.
Another point of interest here is that isolation is often necessary when the assets of each tenant are different. If they are all the same, returns on isolation are lower.
Choose sophistication on the frontend based on requirements; Often, less is good.
Database
The next most important decision would be to decide how to isolate different tenants on the database. There are 3 approaches when it comes to relational databases.
1. Single database per tenant
If your tenants can pay for it, and the situation demands it (legal constraints etc), then this is the safest solution. It also happens to be the costliest. Why?
- 10 databases are 10 times costly (well, roughly) to purchase on the cloud than one.
- Multiple databases mean multiple migrations which increase product release times. Imagine a 1-minute migration taking place for 100 tenants.
- Multiple databases potentially mean different versions of the software on each tenant. It is normal for each tenant to have their team managing their software, which can lead to the addition of new items on the schema (if you are not strict enough), differing software versions per tenant, eventually leading up to multiple deployments of the same software. This is just multiple deployments of the same software in adjacent hardware infrastructure, not multitenancy.
There could be valid reasons to go this way but know that your deployment is getting costly.
Database-per-tenant provides the highest isolation levels in data at the highest cost.
2. Single database, multiple schemas per tenant
There are some minor differences between database-per-tenant and schema-per-tenant.
- Schemas are comparatively cheaper but do not offer the same levels of hardware isolation that database-per-tenant provide
- Database versioning pains still exist
- You can afford to keep the same connection pool. Remember that this brings some complexity within the middleware code. We will need to prefix every query with the schema corresponding to the tenant.
3. Single database with a discriminator column
Here, a column (tenant_id) is present on every table of the database where there is tenant data. This provides the least isolation among the different solutions available. It also is the cheapest possible solution in the medium term.
You cannot have different versions of the software for every tenant if you use this approach (there are ways such as multiple deployments, but then you don’t call it multitenancy anymore).
Consider having a single database with discriminator option if you can. It works well in the long term.
4. Single database with a hierarchical discriminator
This is a slight twist on the regular discriminator column. If you have a lot of reference data that is common to many organisations, then it might make sense to have the ability to inherit that data. In this case, your tenant table will have a hierarchical structure, and your queries will start retrieving data belonging to you and your parent(s).
The case for hierarchical discriminators
Using hierarchical discriminators come with its troubles. Use with caution. At Avni, we were planning to have common metadata for programs that we hoped will be used by many tenants. To enable this, we started with hierarchical discriminators. Soon, this approach proved to be hard to maintain.
- Sharing of data between unrelated organisations is painful — First, we noticed that there were minor changes each tenant wanted on the base metadata. We tried to fix this problem by providing options to override. Providing options to override came with its maintenance problems. The base common metadata was hard to modify because each change needed to be tested on all tenants that use the metadata. Sometimes, it was easy to test, but we had to go to every tenant to verify if the change makes sense to their organisation.
- Slicing of common metadata is hard — The other problem with hierarchical discriminators was to find out the right size to slice the base metadata. We were adding base metadata for multiple health programmes into one big tenant, but our tenants needed only one. We could not implement multiple-inheritance on the metadata, which means a lot of metadata each tenant received was not required for their operations.
Finally, we ended up using plain old discriminator-based multitenancy, with the option to copy over metadata to a tenant where necessary. We still have hierarchical discriminators, but for a different reason. We are also seeing customer that want to run a program through their partners where they decide the metadata. The partners need to see only the transactional data they generate. Hierarchical discriminators make sense in such a scenario.
Different tenants MUST not share common metadata
Middleware design
If you are using discriminator column-based multitenancy, every query needs to be filtered using the tenant id. While it is possible to audit all your queries before deployment, it is an error-prone approach. All the more so when you have your ORM silently running queries on your behalf. The best way forward is to bake multitenancy at the design level of the server. All code assumes that there is just one tenant in the database.
If you use Hibernate, it provides options for switching databases when using database-level or schema-level multitenancy. It is comparatively simple.
Isolate multitenancy functions of your middleware through design
Multitenancy in Avni
We now discuss a concrete implementation — Avni.
Frontend — Avni has a fat frontend. This is because there are no tenant-specific modifications in either our API, admin app or our web-based data entry modules. We maintain the servers ourselves, and we don’t have a scaling problem yet. A fat frontend is sufficient for now.
Database — At the backend, we use a single database separating data using discriminators. This was primarily because our tenants mostly had low volume (10,000–100,000 transactions per annum), having multiple databases or schemas was unnecessary complexity.
We use the row-level security (RLS) feature available in Postgres. Each tenant gets their own Postgres role. All tables have RLS policies applied to ensure data does not leak between tenants. Now all that remains is to ensure that operations are performed using the role assigned to a tenant. The middleware takes care of this.
Middleware — Avni server identifies the user and their tenant from a JWT that comes from the client. This information is set at the thread level in a UserContext. The user context is then used by a tomcat JDBC interceptor to change the role of the database connection. The tomcat JDBC interceptor provides a hook at the connection level. This helps us keep all business logic free from the knowledge of multitenancy.
Before committing to the database, a hibernate JDBC interceptor ensures that all data inserted or updated by a user has their tenant_id. Once the request is serviced, we reset the role on the database connection.
Reports — Reporting servers identify each tenant as its database, with tenant-specific database user. This is not very efficient because Metabase collects database statistics on all the databases it has. This became a bottleneck because the number of tenants keeps increasing, adding to the load of the database. We switched Metabase statistics to manual. Metabase uses JDBC connection pool which provides control over the number of database connection per tenant.
The fat frontend — the web app and the android app are not multitenant aware. Nor are the queries for reporting or the business logic of the server. By containing multitenant behaviour to a small slow-moving area of the system, we can confidently work on newer features easily.
Published On: 03-Feb-2020
Author: Vinay Venu