Nodejs with Python Deployment using Heroku and Travis-CI

A Note about CI with deployment

I’m a huge fan of continuous integration/continuous deployment(CI/CD). The amount of one time investment to set it up is definitely worth it if you are planning on deploying your application. Travis-CI is a free platform that allows you to set up CI and automatically deploy to Heroku if the build is successful. With Travis-CI there’s no more testing your Heroku deploys by just pushing to Heroku.

If you are reading the article, you don’t need me to tell you about the power Python. By the end of this article you too, should be able to access the power of Python from your NodeJS application.

Let’s go ahead and get into it and I’ll explain how to add Python to your Nodejs deployment.

This project assumes that you have deployment experience and have at least used Travis-CI with Heroku. Here are some helpful links to get you up to speed if you do not have a Nodejs application deployed to Heroku using Travis-CI yet:

Setting up deployment for a node app:

Heroku with Travis-CI deployment-option 1

Heroku with Travis-CI deployment-option2

Encrypting secrets if you encounter validation issues on build:

securing of Travis-CI environment variables

Travis-CI authentication with Heroku

If you still have problems with validation try to encrypt the key directly without referencing it with $(heroku auth:token):

travis encrypt <paste heroku token here> — add deploy.api_key — pro

Once you have a Nodejs app deployed to Heroku using Travis-CI, we can begin to add the pieces necessary to bring in Python to the deployment.

Assuming You Already Have A Nodejs App Deployed To Heroku With Travis

add buildpacks on your application’s Heroku dashboard in application settings

• it is key here to add the Nodejs buildpack first so it is at index 1. The buildpacks are in order. You can also check from the Heroku CLI with:

heroku buildpacks 

• If the buildpacks are out of order, you can remove them from the Heroku dashboard then add them again. You can also set the index from Heroku CLI with:

heroku buildpacks:set — index 1 heroku/nodejs 

• Make sure you have a Python buildpack added as well. You may need to create an Aptfile which we will discuss later. If so, add this buildpack https://github.com/heroku/heroku-buildpack-apt for the Aptfile

In your Root Folder

add a runtime.txt file

• in it, add the Python run time environment, example:

python-3.8.6

• If this does not exist, Travis-CI will automatically assume we are using python 2.7.

add a Procfile

• with a node app, Heroku will automatically run node server. Now that we have multiple Heroku buildpacks, we need to explicitly state this in our Procfile with

web: node server

add an Aptfile

• an Aptfile is only necessary when there are dependencies which are neither python nor node that is being used. For example, one of the libraries for my python machine learning model was using libsndfile so I had to add libsndfile1 libsndfile-dev libasound2-dev to my Aptfile like this:

libsndfile1 
libsndfile-dev 
libasound2-dev

• Typically Travis-CI’s build log will tell you if something is missing.

add jobs and include in .travis.yml. set the language to Python, set the version. This version and the runtime.txt version does not need to be exactly the same from my experience but at some point issues could arise if the versions get too far apart. Add a script config option and pip install the requirements.

You will now be able to use Python with your Nodejs deployed application.

Below are some helpful tips to get your Python scripts running from Nodejs:

Since Python machine learning is quite common and the combination of all the machine learning files can get pretty large, you may run into deployed size constraints when deploying to Heroku.

If the slug size is too big, one option is to change tensorflow to tensorflow-cpu in your Python requirements.txt. Tensorflow automatically tries to use the GPU and is much larger than tensorflow-cpu. Your model may be slower but at least you can get it deployed. I was able to shrink my slug size by 200mb doing this.

Running python scripts from our node app can be accomplished by using child_process exec or spawn. To await the results of the python scripts, wrap the exec or spawn in util.promisify.

const util = require(‘util’) 
const exec = util.promisify(require(‘child_process’).exec)

That’s it! Thanks for reading and good luck with your deployment. Feel free to contact me if you have any further questions!

Linkedin/in/yinsong1

Finally here! A deployable boilerplate for a NodeJS application that can run python scripts

https://github.com/songniy/DeployingNodePython