Configuring Queryable Encryption

Added in version 6.0.1.

Queryable Encryption is a powerful MongoDB feature that allows you to encrypt sensitive fields in your database while still supporting queries on that encrypted data.

This section will guide you through the process of configuring Queryable Encryption in your Django project.

MongoDB requirements

Queryable Encryption can be used with MongoDB replica sets or sharded clusters running version 7.0 or later. Standalone instances are not supported. The Queryable Encryption Compatibility table summarizes which MongoDB server products support Queryable Encryption.

Installation

In addition to Django MongoDB Backend’s regular installation and configuration steps, Queryable Encryption requires installing optional Python dependencies and the Automatic Encryption Shared Library.

To install the optional dependencies, use pip with the encryption extra:

$ pip install 'django-mongodb-backend[encryption]'

Next, download the Automatic Encryption Shared Library. You can choose the latest version, even if it doesn’t match your MongoDB server version. After extracting the archive, configure the crypt_shared_lib_path.

Configuring the DATABASES setting

In addition to the database settings required to use Django MongoDB Backend, Queryable Encryption requires configuring a separate database connection that uses use PyMongo’s AutoEncryptionOpts.

Here’s a sample configuration using a local KMS provider:

from pymongo.encryption_options import AutoEncryptionOpts

DATABASES = {
    "default": {
        "ENGINE": "django_mongodb_backend",
        "HOST": "mongodb+srv://cluster0.example.mongodb.net",
        "NAME": "my_database",
        # ...
    },
    "encrypted": {
        "ENGINE": "django_mongodb_backend",
        "HOST": "mongodb+srv://cluster0.example.mongodb.net",
        "NAME": "encrypted",
        # ...
        "OPTIONS": {
            "auto_encryption_opts": AutoEncryptionOpts(
                key_vault_namespace="encrypted.__keyVault",
                kms_providers={
                    "local": {
                        # Generated by os.urandom(96)
                        "key": (
                            b'-\xc3\x0c\xe3\x93\xc3\x8b\xc0\xf8\x12\xc5#b'
                            b'\x19\xf3\xbc\xccR\xc8\xedI\xda\\ \xfb\x9cB'
                            b'\x7f\xab5\xe7\xb5\xc9x\xb8\xd4d\xba\xdc\x9c'
                            b'\x9a\xdb9J]\xe6\xce\x104p\x079q.=\xeb\x9dK*'
                            b'\x97\xea\xf8\x1e\xc3\xd49K\x18\x81\xc3\x1a"'
                            b'\xdc\x00U\xc4u"X\xe7xy\xa5\xb2\x0e\xbc\xd6+-'
                            b'\x80\x03\xef\xc2\xc4\x9bU'
                        )
                    },
                },
                crypt_shared_lib_path="/path/to/mongo_crypt_shared_v1",
                crypt_shared_lib_required=True,
            )
        },
    },
}

key_vault_namespace specifies where to store the data encryption keys. The database name of the key vault must be the same as in "NAME". The vault’s collection name can be whatever you wish, but by convention, it’s often __keyVault.

Dynamic library path configuration

If you encounter the following error:

Pymongocrypt.errors.MongoCryptError: An existing crypt_shared library is
loaded by the application at [/path/to/mongo_crypt_v1.so], but the current
call to mongocrypt_init() failed to find that same library.

add the directory that contains the shared library to your platform’s dynamic library search path:

Platform	Environment variable
Windows	PATH
macOS	DYLD_FALLBACK_LIBRARY_PATH
Linux	LD_LIBRARY_PATH

Examples:

macOS (bash):

$ export DYLD_FALLBACK_LIBRARY_PATH="/path/to/mongo_crypt_shared:${DYLD_FALLBACK_LIBRARY_PATH}"

Linux (bash):

$ export LD_LIBRARY_PATH="/path/to/mongo_crypt_shared:${LD_LIBRARY_PATH}"

Windows (PowerShell):

$env:Path = "C:\path\to\mongo_crypt_shared" + ";" + $env:Path

Windows (Command Prompt):

set PATH=C:\path\to\mongo_crypt_shared;%PATH%

Notes:

• Set the variable to the directory that contains the shared library file (for example, mongo_crypt_shared_v1.dylib on macOS, mongo_crypt_v1.so on Linux, or mongo_crypt_v1.dll on Windows), not the file itself.

• This environment variable is separate from the crypt_shared_lib_path option: the environment variable points to a directory, while crypt_shared_lib_path is the explicit path to the library file.

Configuring the DATABASE_ROUTERS setting

Similar to configuring the DATABASE_ROUTERS setting for embedded models, Queryable Encryption requires a DATABASE_ROUTERS setting to route database operations to the encrypted database.

The following example shows how to configure a router for the "myapp" application that routes database operations to the encrypted database for all models in that application:

# myapp/routers.py
class EncryptedRouter:
    def allow_migrate(self, db, app_label, model_name=None, **hints):
        if app_label == "myapp":
            return db == "encrypted"
        # Prevent migrations on the encrypted database for other apps
        if db == "encrypted":
            return False
        return None

    def db_for_read(self, model, **hints):
        if model._meta.app_label == "myapp":
            return "encrypted"
        return None

    db_for_write = db_for_read

Then in your Django settings, add the custom database router to the DATABASE_ROUTERS setting:

# settings.py
DATABASE_ROUTERS = [
    "django_mongodb_backend.routers.MongoRouter",
    "myapp.routers.EncryptedRouter",
]

Encrypted fields

Now you can start using encrypted fields in your Django models.

Encrypted fields may be used to protect sensitive data like social security numbers, credit card information, or personal health information. With Queryable Encryption, you can also perform queries on encrypted fields. To use encrypted fields in your models, import the necessary field types from django_mongodb_backend.models and define your models as usual.

Here are models based on the Python Queryable Encryption Tutorial:

# myapp/models.py
from django.db import models
from django_mongodb_backend.models import EmbeddedModel
from django_mongodb_backend.fields import (
    EmbeddedModelField,
    EncryptedCharField,
    EncryptedEmbeddedModelField,
)


class PatientRecord(EmbeddedModel):
    ssn = EncryptedCharField(max_length=11, queries={"queryType": "equality"})
    billing = EncryptedEmbeddedModelField("Billing")
    bill_amount = models.DecimalField(max_digits=10, decimal_places=2)

class Patient(models.Model):
    patient_name = models.CharField(max_length=255)
    patient_id = models.BigIntegerField()
    patient_record = EmbeddedModelField("PatientRecord")

    def __str__(self):
        return f"{self.patient_name} ({self.patient_id})"

class Billing(EmbeddedModel):
    cc_type = models.CharField(max_length=50)
    cc_number = models.CharField(max_length=20)

Migrations

Once you have defined your models, create a migration as usual:

$ python manage.py makemigrations

Then run the migrations with:

$ python manage.py migrate --database encrypted

Warning

Be aware that you cannot add encrypted fields to existing models, nor can you change the definition of an encrypted field, for example, to make it queryable.

Creating encrypted data

Now create and manipulate instances of the data just like any other Django model data. The fields will automatically handle encryption and decryption, ensuring that sensitive data is stored securely in the database.

Here’s an example of creating a new Patient instance with encrypted fields:

>>> from myapp.models import Patient, PatientRecord, Billing
>>> billing = Billing(cc_type="Visa", cc_number="4111111111111111")
>>> record = PatientRecord(ssn="123-45-6789", billing=billing, bill_amount=250.75)
>>> patient = Patient(patient_name="John Doe", patient_id=1001, patient_record=record)
>>> patient.save()

Querying encrypted fields

In order to query encrypted fields, you must include the queries argument. For example, notice PatientRecord's ssn field:

class PatientRecord(EmbeddedModel):
    ssn = EncryptedCharField(max_length=11, queries={"queryType": "equality"})

You can perform a equality query just like you would on a non-encrypted field:

>>> patient = Patient.objects.get(patient_record__ssn="123-45-6789")
>>> patient.patient_name
'John Doe'

Configuring the Key Management Service (KMS)

A local KMS provider with a hardcoded key is suitable for local development and testing, but production environment, you should securely store and manage your encryption keys.

To use Queryable Encryption, you must configure a Key Management Service (KMS) to store and manage the encryption keys used to encrypt and decrypt data.

There are two primary configuration points:

1. The kms_providers parameter of AutoEncryptionOpts (see the kms_providers parameter in AutoEncryptionOpts for the available providers (aws, azure, gcp, etc.) and provider options).

2. The KMS_CREDENTIALS inner option of DATABASES. The keys for each provider are documented under the master_key parameter of create_data_key().

Here’s an example of KMS configuration with aws:

from pymongo.encryption_options import AutoEncryptionOpts

DATABASES = {
    "encrypted": {
        # ...
        "OPTIONS": {
            "auto_encryption_opts": AutoEncryptionOpts(
                # ...
                kms_providers={
                    "aws": {
                        "accessKeyId": "your-access-key-id",
                        "secretAccessKey": "your-secret-access-key",
                    },
                },
            ),
        },
        "KMS_CREDENTIALS": {
            "aws": {
                "key": "...",  # Amazon Resource Name
                "region": "...",  # AWS region
            },
        },
    },
}

Configuring the encrypted_fields_map option

Required configuration

As described here, ensure django_mongodb_backend is listed in INSTALLED_APPS to enable the showencryptedfieldsmap command.

Encryption keys are created when you run migrations for models that have encrypted fields.

To see the encrypted fields map for your models (which includes the encryption key IDs), run the showencryptedfieldsmap command:

$ python manage.py showencryptedfieldsmap --database encrypted

In a production environment, it’s recommended to include this map in your settings to protect against a malicious server advertising a false encrypted fields map:

from bson import json_util
from pymongo.encryption_options import AutoEncryptionOpts

DATABASES = {
    "encrypted": {
        # ...
        "OPTIONS": {
            "auto_encryption_opts": AutoEncryptionOpts(
                # ...
                encrypted_fields_map=json_util.loads(
                    """{
                    "encrypt_patient": {
                        "fields": [
                             {
                                 "bsonType": "string",
                                 "path": "patient_record.ssn",
                                 "keyId": {
                                      "$binary": {
                                          "base64": "2MA29LaARIOqymYHGmi2mQ==",
                                          "subType": "04"
                                      }
                                 },
                                 "queries": {
                                     "queryType": "equality"
                                 }
                             },
                        ]
                    }}"""
                ),
            ),
        },
    },
}