Prerequisites
This tutorial assumes you know the basics of Frappe Framework and MongoDB. You should have a working Frappe Bench in order to follow along. Just create a new site, create a new custom app and install it on the site. We will write all of our controller code in this custom Frappe app.
The Normal DocTypes
Normal DocTypes in Frappe are analogous to models in Django or similar frameworks. When you create a new DocType, a new database table in created in the MariaDB database and Frappe manages the table and CRUD operations for you. When you create a new document of a DocType from desk, a new row is inserted in the corresponding database table, behind the scenes. Same is true when you update or delete a document, the Framework takes care of handling the database actions for you. So, we can say, the backend for a DocType is basically a table in the MariaDB (SQL) database.
But what if we want the DocType to be powered by some other source of data? What if we don't want the default behavior of a database table being created and managed for us? In that case, we want to control the source of data for a DocType. This is where Virtual DocTypes come into action.
Virtual DocTypes
Virtual DocTypes were introduced in Version 13 of Frappe Framework. Basically, it gives you the power to control the complete "backend" of a DocType. In contrast to normal DocTypes, Virtual DocTypes do not have a corresponding table in the site database. You as a developer have to write code to control the data source of the DocType. The data can live anywhere: a CSV file, a file-based database like SQLite, a full-blown database like Cassandra or MongoDB. In this tutorial, we will learn how to use MongoDB as the backend for a DocType.
Setting up MongoDB
For this tutorial, I am going to use a free-forever MongoDB Atlas instance (MongoDB on the cloud). You can spin-up a local instance or create a free one here, to follow along.
Once you have a working MongoDB database server, copy the connection string and we are ready to go!
Installing PyMongo
PyMongo is the official Python driver to connect and work with the MongoDB database in Python.
Let's install the PyMongo Python Package on our Frappe Bench using the below command:
bench pip install "pymongo[srv]"
Once the installation is complete, we can open our site's Python console and test the driver by connecting to our MongoDB instance.
We can open the site console using the below command:
bench --site <site_name> console
Let's connect to the database and insert a few test records. We are going to store information about cars in our MongoDB collection. We will name the database cars-database and the collection can be named cars.
from pymongo import MongoClient
CONNECTION_STRING = "<your-connection-string>"
client = MongoClient(CONNECTION_STRING)
db = client['cars-database'] # Get the database
cars = db['cars'] # Get the collection
# Insert some test records
cars.insert_many([
{"make": "BMW", "model": "M4", "year": 2019, "name": "ccdeb89695"},
{"make": "Renault", "model": "Kwid", "year": 2022, "name": "79c1eb5a4b"}
])
If you set up the database correctly and the PyMongo installation succeeded, the above code should execute without any errors.
Creating a Virtual DocType
Go to the DocType list and create a new DocType. Let's name it Car and set the module to our custom app (in my case My Awesome App). Check the 'Is Virtual' checkbox near the top of the DocType form to make this a Virtual DocType:
Rest of the things are similar to any normal DocType. We will create 3 fields for the Car doctype as shown below:
- Make - Data - Mandatory
- Model - Data - Mandatory
- Year - Int - Non-Negative
Hit Save and let's navigate to Car List. It is empty, as expected:
You will not be able to create a new document, yet, because the functionality does not exist. We need to write the code to populate the list view, create a new document and so on. Let the fun begin!
The Generated Boilerplate Code
Open up your custom app in the code editor of your choice and look for the newly generated files for the Car DocType. As with any normal DocType, 4 files are generated: car.py, car.js, test_car.py and car.json.
Open the car.py file. You will notice that some boilerplate code for implementing the Virtual DocType Car has already been generated for you:
class Car(Document):
def db_insert(self):
pass
def load_from_db(self):
pass
def db_update(self):
pass
@staticmethod
def get_list(args):
pass
@staticmethod
def get_count(args):
pass
@staticmethod
def get_stats(args):
pass
These are the methods which must exist on a controller class for a Virtual DocType. Let's start by getting the list view working. The general idea is to fetch all the cars from our MongoDB cars collection and return it from the get_list static method:
class Car(Document):
@staticmethod
def get_list(args):
cars = get_cars_collection()
cars_list = []
for car in cars.find():
cars_list.append({
**car,
"_id": str(car["_id"])
})
return cars_list
...
# Utility function to get the db collection
def get_cars_collection():
client = MongoClient(CONNECTION_STRING)
db = client['cars-database']
cars = db.cars
return cars
Here, we are fetching all the cars and returning them as a list of Python dictionaries. Notice how we are converting the _id field to str before returning. This needs to be done because the _id field is of type: bson.ObjectId, which is not JSON serialisable.
Let's go back to list view and refresh:
And voilà! List view is getting the data from our MongoDB collection!
Trying to open any of the documents will throw an error since we have not written the code to handle it.
Getting a particular document
The method which we need to implement to populate the form view is load_from_db:
def load_from_db(self):
cars = get_cars_collection()
d = cars.find_one({"name": self.name})
super(Document, self).__init__(d)
Here we are using the name field of the doctype to fetch a particular document from our cars collection. If you recall, we added the name field in the test records we inserted while testing our PyMongo client too. We will use the name as our primary key instead of the _id field generated by MongoDB. The name field will be automatically generated for us by Frappe Framework as we will see in the next section.
You can now open the Car list view and try to open individual documents. It will work as expected.
Creating a new document
Now, let's implement the functionality to create a new document for the Car DocType. Easy, peasy, lemon-squeezy:
def db_insert(self, *args, **kwargs):
cars = get_cars_collection()
d = self.get_valid_dict()
cars.insert_one(d)
The get_valid_dict method returns a dictionary of validated values for the document. Let's inspect its contents:
{'creation': '2022-11-27 13:28:48.065820',
'docstatus': 0,
'idx': 0,
'make': 'Honda',
'model': 'Dezire',
'modified': '2022-11-27 13:28:48.065820',
'modified_by': 'Administrator',
'name': 'c1f463ae7b',
'owner': 'Administrator',
'year': 2022}
Here we have a variety of fields along with the ones we created (make, model and year). You can choose to store only the fields you want in your MongoDB instance or store all of the above. Storing all the key-value returned from the get_valid_dict() method can have its own benefits like automatic modified and creation timestamps. I choose to store all the above fields as they are useful meta-data stuff that the framework uses to enhance your doctype.
Let's try and create a new Car now:
Fill in the details and hit save. You should be able to see the newly created car in your MongoDB Atlas Collection dashboard:
Updating a document
We will need to implement the db_update method in order to make the update action work. It is fairly straight-forward as well:
def db_update(self):
cars = get_cars_collection()
updated_values = self.get_valid_dict()
cars.update_one({"name": self.name}, {"$set": updated_values})
Here we are using the get_valid_dict method again to get the required key-value pairs. You can now try editing a document and saving the changes. The changes should be synced in MongoDB now!
Deleting a document
In order to make the delete document functionality work, we will implement the delete method as shown below:
def delete(self):
cars = get_cars_collection()
cars.delete_one({"name": self.name})
Getting count of documents
The list view shows the total number of documents. We can supply that number by returning it from the get_count static method:
@staticmethod
def get_count(args):
return get_cars_collection().count_documents({})
Additional Resources
You can find the complete source code for the Car DocType controller code here. I have also added some sample code to get a few list view filters working too.
The documentation for Virtual DocTypes is a great place to start. It contains a very good example of using a JSON file as your doctype's backend.
The official PyMongo docs are a good place to find examples of using the PyMongo driver.
After Rushabh's idea about a generic base class in the comments, I implemented an simple abstract base class that can easily turn any DocType controller (of a Virtual DocType) into a MongoDB powered document. You can find the code here.
Using the base class, it becomes as easy as implementing just one method:
In the end
Once we are done with the implementation, things like frappe.get_doc and DocType REST APIs work out of the box! Isn't it magical?!
As you can see, how easy it is to use MongoDB as a data source for our DocTypes. You can follow the same pattern to connect any other database like Cassandra or even Firebase Firestore. This makes DocTypes even more powerful. Go ahead, create DocTypes powered by your favorite database!
