The Story So Far

In part I, we built the storefront of our online eBook store using Bulma CSS framework and Frappe DocType views:

In this part, we will implement the eBook purchase (payment gateway integration) and delivery mechanism. We will learn how to integrate the Razorpay payment gateway in our Frappe site from scratch and also learn a lot about programmatically sending emails from our Frappe instance.

Resources

2. The Completed Source Code of Part 1: link
3. The Completed Source Code of Part 2: link
4. The Razorpay Documentation: link

Go and read Part I before reading this post, if you haven't yet, because this post picks up right where Part I left.

The Delivery Mechanism

We want the eBook to be delivered via email after a successful purchase. The asset file linked to the eBook DocType will be sent as an attachment. The email will also contain some details about the purchase like book name, cover image and author. Let's get to work!

Email Setup

Feel free to skip this section if you already have outgoing email setup on your Frappe instance. But you may miss out on learning a faster and cooler way to test emails on your local setup!

In order to send emails from your Frappe instance, you need to setup an Email Account. If your site is hosted on Frappe Cloud, you can use the Email Delivery Service app by Frappe to start sending emails without any configuration. Just install the app from FC Marketplace and done!

For local testing of emails, I use mailtrap.io. You can create an account and get started for free. Once you have created an account, navigate to your Sandbox inbox and note down the credentials for SMTP:

Insert the credentials in your site_config.json file as given below:

{
    "mail_server": "smtp.mailtrap.io",
    "mail_login": "<Username from mailtrap>",
    "mail_password": "<Password from mailtrap>",
    "mail_port": "2525",
    "auto_email_id": "hussain@localsetup.com"
}

If you have an email account already setup in your Frappe site, you can skip the above step.

The sendmail function

Frappe Framework has this awesome utility function to send emails from your Frappe site named sendmail. It is a must know and hence let's learn how to use it with some simple examples first. You can find its documentation here.

The simplest use of the sendmail function is:

frappe.sendmail(["faris@frappe.io"])

The only required argument for this function is a list of recipients, everything else is optional. You can check the Email Queue list to see if the email got queued for sending. Faris will receive an email with the subject "No Subject" and body "No Message". Why? Well, we didn't provide any subject or message, so what more can we expect. Let's make it better:

frappe.sendmail(
    ["faris@frappe.io"],
    subject="Important Notification"
    message="# Hello Bro!",
    as_markdown=True
)

This time I have added a subject line as well as a message. Setting the as_markdown parameter to True will render the message (content of the email) as markdown (to HTML in our email body). You can also pass a HTML string directly as message.

Attaching Files

To attach files to the outgoing email, we can pass a list of dictionaries as the attachments to the sendmail utility:

frappe.sendmail(
    ["faris@frappe.io"],
    subject="Your Frappe Cloud Invoice is Here",
    attachments=[{"file_url": "/files/invioce.pdf"}]
)

Notice how we are passing a list of dictionaries. The dictionary should have the file_url of the File we want to attach to the email.

Using Jinja Templates

We can also use a Jinja template for the email content (message). In order to do so, we can create a HTML file in our app's templates/emails directory and write our Jinja HTML code inside it. This template will be rendered and used as our email content.

When sending an email using the frappe.sendmail function, we can provide the template file name like so:

frappe.sendmail(
    ["ankush@frappe.io"],
    template="my_awesome_template",
    args={"username": "NagariaHussain"}
)

This will render the template located at templates/emails/my_awesome_template.html. You can pass a dictionary with the data you want to use in your jinja template via the args parameter. Now, you will be able to use username in your template:

<h1>{{ username }}</h1>

Sending The eBook via Email

Let's combine all we have learned about sending emails and implement the eBook delivery logic. I will add a new method in our eBook controller class to send the eBook to a given recipient (email address):

class eBook:
    ...
    def send_via_email(self, recipient):
        author_name = frappe.db.get_value("Author", self.author, "full_name")
        args = {
            "name": self.name,
            "cover_image": self.cover_image,
            "author_name": author_name,
        }
        asset_attachments = [{"file_url": self.asset_file}]

        frappe.sendmail(
            [recipient],
            subject="[Star EBook Store] Your eBook has arrived!",
            attachments=asset_attachments,
            template="ebook_delivery",
            args=args,
        )

The contents of templates/emails/ebook_delivery.html is:

<div>
    <h2>{{ name }}</h2>
    <p>by {{ author_name }}</p>
    <img alt="book cover image" src="{{ cover_image }}" />

    <p>Please find attached your awesome eBook!</p>
</div>

Now, let's open up our python console and try calling our send_via_email method:

bench --site abc.localhost console

ebook = frappe.get_doc("eBook", "Creative Confidence")
ebook.send_via_email("ankush@frappe.io")
frappe.db.commit()

You can now check your inbox in mailtrap dashboard. You will find your delivery right in your inbox:

Yay! It works! If the email does not show up for you, check the Email Queue list on your site.

The Purchase Mechanism

As promised, we will implement online payments via Razorpay. Razorpay has good documentation and is fairly easy to setup. You can also use Stripe if you want, the core concepts will be similar.

Razorpay Setup

We are going to setup Razorpay as our payment gateway. We will follow this guide from Razorpay docs to setup standard checkout on our eBook store. We could have also used the Frappe Payments app, but it uses web form for payments, and I wanted a smooth in-web page experience. As a bonus, we get to go over the complete flow of integrating a Payment gateway in Frappe Framework, yippee!

The general flow of getting the Razorpay payment gateway working on our store is as follows:

1. When the user clicks the "Buy" button on our eBook details page, we will make an API call to the backend to fetch Razorpay order details (defined in next step).
2. In the backend, we generate a Razorpay Order record, store some information about it in a doctype and send the Order ID and Key ID (required by Razorpay client library) to our client side (our store web portal).
3. Once the API call has returned the required details about the order, we will use Razorpay JavaScript client to open the gateway and proceed with the Payment. Razorpay will take control from this point.
4. On a successful payment, we will display an alert to the user.
5. Behind the scenes, we will listen to the web hooks from Razorpay for successful payment and trigger the delivery of the eBook.

Sounds complicated? Well, it isn't that complicated. Let's do this step-by-step, together.

Obtaining Razorpay Credentials

Head over to Razorpay signup page and create a free account. Now, visit the dashboard and navigate to the Settings Page from the sidebar. Open up the API Keys tab and copy the credentials (Key ID and Key Secret):

We will create a new DocType to store these credentials on our site. This will be a Single DocType (learn more here), let's call it Store Razorpay Settings:

This DocType has 2 fields:

1. Key ID: Data
2. Key Secret: Password

Copy-paste the test credentials we obtained from the Razorpay dashboard to these fields:

Setting Up Razorpay Python Client

In order to interact with the Razorpay API from our Frappe backend, we are going to use the official Razorpay Python Client. Let's install it on our bench using pip:

$ bench pip install razorpay

I will also add this package to our custom app's requirements.txt file:

This will make sure the razorpay package is automatically installed along with our app.

Now, we can try out the client by creating a test order. I will use the Python console to do so:

import razorpay
from frappe.utils.password import get_decrypted_password

key_id = frappe.db.get_single_value("Store Razorpay Settings", "key_id")
key_secret = get_decrypted_password(
    "Store Razorpay Settings", "Store Razorpay Settings", "key_secret"
)

client = razorpay.Client(auth=(key_id, key_secret))
order_data = {"amount": 50000, "currency": "INR"} # in paisa
razorpay_order = client.order.create(data=order_data)

print(razorpay_order)

In the above code snippet, first I am fetching the credentials from the database. Notice how I am using the get_decrypted_password to, well, get the decrypted value of the key_secret password field.

Next, we use the fetched credentials to create an instance of the Razorpay Client. We create a new Razorpay Order by passing the amount and currency as the data. If everything is setup correctly, a dictionary like the one below will be printed to the console:

{'amount': 50000,
 'amount_due': 50000,
 'amount_paid': 0,
 'attempts': 0,
 'created_at': 1673682347,
 'currency': 'INR',
 'entity': 'order',
 'id': 'order_L3ynWUbZjR6SZr',
 'notes': [],
 'offer_id': None,
 'receipt': None,
 'status': 'created'}

Yay! We are good to go now!

Tracking eBook Orders

We will need to track the orders placed on our online store. We should be able to track the below information:

1. eBook: The eBook for which the order was placed
2. Customer Email: The email to which the order (eBook) will be delivered
3. Order Amount: How much did the customer pay?
4. Status: The status of the Order. This can have one of the three values: "Pending", "Paid" and "Delivered", which are pretty self-explanatory.
5. Razorpay Order ID: Returned by Razorpay Client when we create a new order, will be used to track the payment status.
6. Razorpay Payment ID: Will be returned by Razorpay on a successful payment (more about later)

You know how we will store this information, right? New DocType Time!

What you see above, is the new Form Builder about which I was talking about in Part I. You can easily build the DocType form by dragging and dropping fields from the sidebar! New and exciting stuff is happening in Frappe Framework, making it even more powerful and a delight to use.

We will use the eBook Order DocType we just created to track the payment and delivery of an eBook on our store.

API To Create Order

As I briefed previously, our client side will call an API function to create a new order for an eBook when a user clicks the "Buy" button. Let's implement that API now. Create a new file named api.py in the root directory of our custom app:

Now, we will add the following code to this file:

import frappe
import razorpay

from frappe.utils.password import get_decrypted_password

@frappe.whitelist(allow_guest=True)
def create_ebook_order(ebook_name):
    # Fetch price of this ebook
    ebook_price_inr = frappe.db.get_value("eBook", ebook_name, "price_inr")

    # Create Razorpay Order
    order_data = {"amount": ebook_price_inr * 100, "currency": "INR"}  # convert to paisa
    client = get_razorpay_client()
    razorpay_order = client.order.create(data=order_data)

    // Create and insert new ebook order document
    frappe.get_doc(
        {
            "doctype": "eBook Order",
            "ebook": ebook_name,
            "razorpay_order_id": razorpay_order["id"],
            "order_amount": ebook_price_inr,
        }
    ).insert(ignore_permissions=True)

    return {
        "key_id": client.auth[0],
        "order_id": razorpay_order["id"],
    } # will be used in client side

def get_razorpay_client():
    key_id = frappe.db.get_single_value("Store Razorpay Settings", "key_id")
    key_secret = get_decrypted_password(
        "Store Razorpay Settings", "Store Razorpay Settings", "key_secret"
    )

    # create razorpay client and return
    return razorpay.Client(auth=(key_id, key_secret))

The important piece here is the create_ebook_order function. This gets the name of the eBook as a parameter. We have added the @frappe.whitlist() wrapper to allow calling of this API from the frontend. Setting allow_guests=True makes sure that guest (non-logged in) users can also call this API. I have decided to not require authentication by the user at any step, but you could require user authentication before placing an order.

Here is what the code is basically doing:

1. Fetching the price of the given eBook (ebook_name) from the database
2. Using the fetched price to create a new Razorpay Order
3. Create a new eBook Order document with the required values
4. Return the key_id and order_id in a dictionary (we will use these in the next section)

Note that the default/initial status of the created eBook Order in this step is "Pending". This implies that the payment has not been completed yet. The status, customer email and payment ID field will be updated once the payment is successful. Feel free to try to call this method by providing an eBook name from your Python console.

Client Setup

This is where things start to get exciting!

Including The Razorpay JavaScript Client

According to the Razorpay Integration docs, we can include the Razorpay JavaScript Client by adding the below script tag in our HTML <head>:

<script src="https://checkout.razorpay.com/v1/checkout.js"></script>

To keep things clean and efficient, we will add a new block in our base.html template to include script tags. Open up the templates/store/base.html file and add the following lines inside <head> tag:

<head>
        ...
    {%- block head_script_tags -%} {%- endblock -%}
    ...
</head>

We don't want to load the client on every page, but just the eBook details page (ebook.html). Now, open the ebook.html file and add the following code at the top of the file:

{% block head_script_tags %}
<script src="https://checkout.razorpay.com/v1/checkout.js"></script>
{% endblock %}

Click To Pay

If you recall, in the first part, we added a Download button to our ebook details page. I have changed its text to Buy and given it an id of "buy-button", so we can get it by ID in our script:

The general outline of what we want to happen when the button is clicked is:

<script>
    const buyButton = document.getElementById("buy-button")

    buyButton.onclick = async function (e) {
        const options = await getRazorpayOrderOptions()
        const razorpayOrder = createRazorpayOrder(options)
        razorpayOrder.open()

        e.preventDefault()
    }

    async function getRazorpayOrderOptions() { ... }
    function createRazorpayOrder(options) { ... }
</script>

I have added the above code in a new script tag just before our {% endblock %} line in the ebook.html file. The script is doing the following:

1. Attaching a click handler to the Buy button
2. On button click, the getRazorpayOrderOptions async function is invoked, which calls our backend API and returns the required details (more on this in a second)
3. Creating and Opening a new Razorpay order using the provided order details (fetched in step 2)

Here is the body of the getRazorpayOrderOptions function:

const headers = {
    Accept: 'application/json',
    'Content-Type': 'application/json; charset=utf-8',
    'X-Frappe-CSRF-Token': '{{ csrf_token }}' # Note: Will be passed in the context by `get_context`
}

const response = await fetch("/api/method/star_ebook_store.api.create_ebook_order", {
    method: 'POST',
    body: JSON.stringify({
        ebook_name: '{{ doc.name }}'
    }),
    headers
})

if (response.ok) {
    const orderData = (await response.json()).message // frappe wraps the response inside "message" key

    const options = {
        "key": orderData.key_id,
        "order_id": orderData.order_id,
        "handler": (res) => {
            alert("Payment Success! Your eBook is on the way!")
        }
    }

    return options
} else {
    // EXERCISE: Better handling 
    alert("Something went wrong")
    throw Error("")
}

Nothing fancy here, we are just doing a fetch to our backend API method (create_ebook_order function we wrote in the previous step) and returning the fetched data in a new JavaScript object. The "handler" function will be called by Razorpay on successful completion of the payment. We are just showing an alert with a message.

But for the above function to work, we will need to provide the csrf_token as context. It is a one line change, open up ebook.py file and add the following line to the get_context method:

The csrf_token is required for security purposes. It prevents cross-site request forgery (CSRF) attacks. You can google search to learn more about this if you are interested.

The body of the createRazorpayOrder function:

const razorpayOrder = new Razorpay(options)
return razorpayOrder

The Razorpay class comes from the script tag we included a few steps ago.

That's it! Go ahead, click the Buy button now:

I am using a special test UPI ID to do a successful test transaction (more methods to test here). We get the success message! If you check your transaction page in the Razorpay dashboard, you can clearly see the successful payment transaction:

Let's also check the latest eBook Order document created on our site:

Wait, wut?! The user just made a successful payment and the order is still pending. Why didn't it get marked "Paid"? Why are the customer_email and razorpay_payment_id fields empty? Why? Why?

Oh! We haven't handled successful payments yet. In the frontend, we just showed an alert. What we could have done is told our backend to process the order in the handler. But what if the user's internet connection drops? What if he closes the tab before we could talk to our backend? The solution to this problem is webhooks.

Setting Up Webhooks

Services like Stripe, Razorpay and many more have the provision to send HTTP requests to developer-defined endpoints on the occurrence of some event, for instance, when a payment is successful.

So, we can tell Razorpay to call an endpoint on our site on a successful payment and then we can process the order. Razorpay will send all the details related to the payment in the webhook's (request's) body, which we can use to process the payment according to our requirements.

For Razorpay to be able to call our endpoint, it needs to be accessible on the public internet. But how can it reach our localhost, our development environment? When we deploy our custom app on a production site, we have its URL, which can be used as is to listen to webhooks. But in case of our local setup, we will need to somehow "tunnel" our localhost to the internet. Worry not! We can use ngrok for exposing our local Frappe site to the internet and get a public URL which we can then give to Razorpay to send webhooks to.

Frappe has built-in support for ngrok:

$ bench --site <site-name> ngrok

You should see the below output:

We now have a public URL for our local site. How cool is that! Try visiting the public URL and you should see your Frappe site's login page. You can also share this URL with your teammates to show something off directly from your local machine.

Now that we have a public URL, let's write a new API function (endpoint) to handle the webhooks that will be sent by Razorpay (in the api.py file):

@frappe.whitelist(allow_guest=True)
def handle_razorpay_webhook():
    form_dict = frappe.local.form_dict
    payload = frappe.request.get_data()

    verify_webhook_signature(payload)  # for security purposes

    # Get payment details (check Razorpay docs for object structure)
    payment_entity = form_dict["payload"]["payment"]["entity"]
    razorpay_order_id = payment_entity["order_id"]
    razorpay_payment_id = payment_entity["id"]
    customer_email = payment_entity["email"]
    event = form_dict.get("event")

    # Process the order
    ebook_order = frappe.get_doc("eBook Order", {"razorpay_order_id": razorpay_order_id})
    if event == "payment.captured" and ebook_order.status != "Paid":
        ebook_order.update(
            {
                "razorpay_payment_id": razorpay_payment_id,
                "status": "Paid",
                "customer_email": customer_email,
            }
        ) # Mark as paid and set payment_id and customer_email
        ebook_order.save(ignore_permissions=True)

frappe.local.form_dict returns the JSON body contained in the webhook request. frappe.request.get_data() returns the HTTP request body (payload), which we are using to verify the signature (ignore this for now, we will come back to this). We are basically doing the following:

1. Extract the payment details from the data sent with the webhook
2. Get the eBook Order record associated with the order_id (which is unique for every record)
3. If the type of the event (sent by Razorpay) is payment.captured (successful payment), we mark the order Paid and update the fields razorpay_payment_id and customer_email

Now that we have the handler function in place, let's ask Razorpay to call it when a payment is captured.

Setting Up Webhooks In Razorpay

Head over to the docs here for a complete guide on setting up webhooks. But this will be as easy as opening the Webhooks tab in the Settings page and filling a form:

The Webhook URL will be:

<YOUR-PUBLIC-URL>/api/method/star_ebook_store.api.handle_razorpay_webhook

For security purposes, you can optionally provide a secret, which can be used to verify if the request was actually sent by Razorpay. For now, just enter some random string and copy it somewhere. We will implement verification in the next step.

Signature Verification

As you may have noticed in the implementation of the handle_razorpay_webhook function, we had a line:

verify_webhook_signature(payload)  # for security purposes

But what does this call do? Here is its implementation:

def verify_webhook_signature(payload):
    signature = frappe.get_request_header("X-Razorpay-Signature")
    webhook_secret = get_decrypted_password(
        "Store Razorpay Settings", "Store Razorpay Settings", "webhook_secret"
    )

    client = get_razorpay_client()
    client.utility.verify_webhook_signature(payload.decode(), signature, webhook_secret)

I have added a new webhook_secret (Password) field in our Store Razorpay Settings DocType to store the webhook secret. This must be the same secret we entered in the Razorpay dashboard while configuring webhooks.

Razorpay Python Client has a utility method verify_webhook_signature that can be used to easily verify the webhook request.

That's it! Let's try buying an ebook again:

Hurray! The order got processed this time. You can also inspect the ngrok logs to see the request body that Razorpay sent.

Wiring Payment & Delivery

This is the only thing left now! You can do this in multiple ways. But I will write the delivery logic right in the eBook Order controller (ebook_order.py) like so:

class eBookOrder(Document):
    def on_update(self):
        if self.status == "Paid":
            self.deliver_ebook()

    def deliver_ebook(self):
        ebook_doc = frappe.get_doc("eBook", self.ebook)

        try:
            ebook_doc.send_via_email(self.customer_email)
            self.status = "Delivered"
        except:
            ebook_doc.log_error("Ebook Delivery Error")

Remember the send_via_email method we implemented in the first section? I am just using the on_update life cycle hook to trigger the eBook delivery when the status is Paid.

Done.

Conclusion

If you are with me till now, I would love to know your thoughts. Feel free to drop a "Hi" @ hussain@frappe.io. Since this post became too long, I have decided to create a Part III! In the next part, we will add interactivity using AlpineJS and deploy our site to Frappe Cloud! Spoiler: We will also add a confetti animation! See you there!