MongoDB Powered DocTypes in Frappe Framework

In this tutorial, we will learn how to use MongoDB as a datasource for our DocTypes, using the power of Virtual DocTypes.

 · 7 min read

Prerequisites

This tutorial assumes you know the basics of Frappe Framework and MongoDB. You should have a working Frappe Bench in order to follow along. Just create a new site, create a new custom app and install it on the site. We will write all of our controller code in this custom Frappe app.

The Normal DocTypes

Normal DocTypes in Frappe are analogous to models in Django or similar frameworks. When you create a new DocType, a new database table in created in the MariaDB database and Frappe manages the table and CRUD operations for you. When you create a new document of a DocType from desk, a new row is inserted in the corresponding database table, behind the scenes. Same is true when you update or delete a document, the Framework takes care of handling the database actions for you. So, we can say, the backend for a DocType is basically a table in the MariaDB (SQL) database.

But what if we want the DocType to be powered by some other source of data? What if we don't want the default behavior of a database table being created and managed for us? In that case, we want to control the source of data for a DocType. This is where Virtual DocTypes come into action.

Virtual DocTypes

Virtual DocTypes were introduced in Version 13 of Frappe Framework. Basically, it gives you the power to control the complete "backend" of a DocType. In contrast to normal DocTypes, Virtual DocTypes do not have a corresponding table in the site database. You as a developer have to write code to control the data source of the DocType. The data can live anywhere: a CSV file, a file-based database like SQLite, a full-blown database like Cassandra or MongoDB. In this tutorial, we will learn how to use MongoDB as the backend for a DocType.

Setting up MongoDB

For this tutorial, I am going to use a free-forever MongoDB Atlas instance (MongoDB on the cloud). You can spin-up a local instance or create a free one here, to follow along.

Once you have a working MongoDB database server, copy the connection string and we are ready to go!

Installing PyMongo

PyMongo is the official Python driver to connect and work with the MongoDB database in Python.

Let's install the PyMongo Python Package on our Frappe Bench using the below command:

bench pip install "pymongo[srv]"

Once the installation is complete, we can open our site's Python console and test the driver by connecting to our MongoDB instance.

We can open the site console using the below command:

bench --site <site_name> console

Let's connect to the database and insert a few test records. We are going to store information about cars in our MongoDB collection. We will name the database cars-database and the collection can be named cars.

from pymongo import MongoClient

CONNECTION_STRING = "<your-connection-string>"

client = MongoClient(CONNECTION_STRING)
db = client['cars-database'] # Get the database
cars = db['cars'] # Get the collection

# Insert some test records
cars.insert_many([
    {"make": "BMW", "model": "M4", "year": 2019, "name": "ccdeb89695"},
    {"make": "Renault", "model": "Kwid", "year": 2022, "name": "79c1eb5a4b"}
])

If you set up the database correctly and the PyMongo installation succeeded, the above code should execute without any errors.

Creating a Virtual DocType

Go to the DocType list and create a new DocType. Let's name it Car and set the module to our custom app (in my case My Awesome App). Check the 'Is Virtual' checkbox near the top of the DocType form to make this a Virtual DocType:

DocType Form View

Rest of the things are similar to any normal DocType. We will create 3 fields for the Car doctype as shown below:

  1. Make - Data - Mandatory
  2. Model - Data - Mandatory
  3. Year - Int - Non-Negative

Hit Save and let's navigate to Car List. It is empty, as expected:

Empty Car List View

You will not be able to create a new document, yet, because the functionality does not exist. We need to write the code to populate the list view, create a new document and so on. Let the fun begin!

The Generated Boilerplate Code

Open up your custom app in the code editor of your choice and look for the newly generated files for the Car DocType. As with any normal DocType, 4 files are generated: car.py, car.js, test_car.py and car.json.

Open the car.py file. You will notice that some boilerplate code for implementing the Virtual DocType Car has already been generated for you:

class Car(Document):
    def db_insert(self):
        pass

    def load_from_db(self):
        pass

    def db_update(self):
        pass

    @staticmethod
    def get_list(args):
        pass

    @staticmethod
    def get_count(args):
        pass

    @staticmethod
    def get_stats(args):
        pass

These are the methods which must exist on a controller class for a Virtual DocType. Let's start by getting the list view working. The general idea is to fetch all the cars from our MongoDB cars collection and return it from the get_list static method:

class Car(Document):
    @staticmethod
    def get_list(args):
        cars = get_cars_collection()
        cars_list = []

        for car in cars.find():
            cars_list.append({
                **car,
                "_id": str(car["_id"])
            })

        return cars_list

    ...

# Utility function to get the db collection
def get_cars_collection():
    client = MongoClient(CONNECTION_STRING)
    db = client['cars-database']
    cars = db.cars
    return cars

Here, we are fetching all the cars and returning them as a list of Python dictionaries. Notice how we are converting the _id field to str before returning. This needs to be done because the _id field is of type: bson.ObjectId, which is not JSON serialisable.

Let's go back to list view and refresh:

Working List View

And voilà! List view is getting the data from our MongoDB collection!

Trying to open any of the documents will throw an error since we have not written the code to handle it.

Getting a particular document

The method which we need to implement to populate the form view is load_from_db:

def load_from_db(self):
    cars = get_cars_collection()
    d = cars.find_one({"name": self.name})
    super(Document, self).__init__(d)

Here we are using the name field of the doctype to fetch a particular document from our cars collection. If you recall, we added the name field in the test records we inserted while testing our PyMongo client too. We will use the name as our primary key instead of the _id field generated by MongoDB. The name field will be automatically generated for us by Frappe Framework as we will see in the next section.

You can now open the Car list view and try to open individual documents. It will work as expected.

Creating a new document

Now, let's implement the functionality to create a new document for the Car DocType. Easy, peasy, lemon-squeezy:

def db_insert(self, *args, **kwargs):
    cars = get_cars_collection()
    d = self.get_valid_dict()
    cars.insert_one(d)

The get_valid_dict method returns a dictionary of validated values for the document. Let's inspect its contents:

{'creation': '2022-11-27 13:28:48.065820',
 'docstatus': 0,
 'idx': 0,
 'make': 'Honda',
 'model': 'Dezire',
 'modified': '2022-11-27 13:28:48.065820',
 'modified_by': 'Administrator',
 'name': 'c1f463ae7b',
 'owner': 'Administrator',
 'year': 2022}

Here we have a variety of fields along with the ones we created (make, model and year). You can choose to store only the fields you want in your MongoDB instance or store all of the above. Storing all the key-value returned from the get_valid_dict() method can have its own benefits like automatic modified and creation timestamps. I choose to store all the above fields as they are useful meta-data stuff that the framework uses to enhance your doctype.

Let's try and create a new Car now:

New Car Document

Fill in the details and hit save. You should be able to see the newly created car in your MongoDB Atlas Collection dashboard:

Newly Added Record in MongoDB Atlas Dashboard

Updating a document

We will need to implement the db_update method in order to make the update action work. It is fairly straight-forward as well:

def db_update(self):
    cars = get_cars_collection()
    updated_values = self.get_valid_dict()
    cars.update_one({"name": self.name}, {"$set": updated_values})

Here we are using the get_valid_dict method again to get the required key-value pairs. You can now try editing a document and saving the changes. The changes should be synced in MongoDB now!

Deleting a document

In order to make the delete document functionality work, we will implement the delete method as shown below:

def delete(self):
    cars = get_cars_collection()
    cars.delete_one({"name": self.name})

Getting count of documents

The list view shows the total number of documents. We can supply that number by returning it from the get_count static method:

@staticmethod
def get_count(args):
    return get_cars_collection().count_documents({})

Additional Resources

You can find the complete source code for the Car DocType controller code here. I have also added some sample code to get a few list view filters working too.

The documentation for Virtual DocTypes is a great place to start. It contains a very good example of using a JSON file as your doctype's backend.

The official PyMongo docs are a good place to find examples of using the PyMongo driver.

After Rushabh's idea about a generic base class in the comments, I implemented an simple abstract base class that can easily turn any DocType controller (of a Virtual DocType) into a MongoDB powered document. You can find the code here.

Using the base class, it becomes as easy as implementing just one method:

In the end

Once we are done with the implementation, things like frappe.get_doc and DocType REST APIs work out of the box! Isn't it magical?!

As you can see, how easy it is to use MongoDB as a data source for our DocTypes. You can follow the same pattern to connect any other database like Cassandra or even Firebase Firestore. This makes DocTypes even more powerful. Go ahead, create DocTypes powered by your favorite database!


Md Hussain Nagaria

Hussain Nagaria is a Product Engineer at Frappe. He is very passionate about teaching technical content. He also does product design, writing and marketing. So, you can say, he is a company of one at Frappe!

Add a comment
Ctrl+Enter to add comment

A
Alaa Alsalehi 1 month ago

I love the concept of virtual doc it covers an extensive area of situations I think there is a bug there in the Virtual doc feature when it combined with the child table feature thank you for the great article Md Hussain

A
avc 2 months ago

Yeah! This is a really amazing feature. Things like linking virtual doctype from other doctype, or fetching fields are still pretty dark holes to us.

Thank you!

Hussain Nagaria 2 months ago

Good Idea @Rushabh! Implemented a basic base class that makes it very easy to turn Virtual DocTypes into MongoDB Powered Documents. Added the code and link in the "Additional Resources" section above.

A
Abdo Saeed 2 months ago

Great job, i think we can create a mongo db connector and let's all virtual doctypes running across it

Rushabh Mehta 2 months ago

@Hussain, you can create a base class that can be re-used across DocTypes?