Install and configure a full software stack for a Flask app: Apache, Gunicorn, MongoDB, Redis

Tue 28 February 2017

A few days ago, I had to deploy a new Flask application for a customer and here are some notes which could be useful for somebody, as well. I will show how everything is configured, even if here, in this tutorial, I have just a minimal Flask app, in one file: app.py

The software stack used: Python 3.x, Flask, Ubuntu 14.04, Apache, Gunicorn, MongoDB (via mongoengine), Redis(used for Flask-Cache)

Python/Flask environment setup

First we will create a user to run our application

# adduser deploy
  • login as that user
# su - deploy

Let's prepare the environment to run our Flask app

$ pwd
/home/deploy/
$ mkdir apps
$ mkdir apps/domain
$ cd apps/domain
  • create a python virtual environment
$ pyvenv-3.4 venv
$ source venv/bin/activate
$ pip install -r requirements.txt
$ deactivate

Apache setup

The customer has a VPS with Plesk installed, so, the place for Apache config file for a specific domain is different from a standard Apache installation. For a standard installation and configuration please check my previous post: Deploy your Flask Web Application on Ubuntu 16.04 with Apache, Gunicorn and systemd

  • configure Apache to serve our Flask app:
# a2enmod proxy proxy_ajp proxy_http rewrite deflate headers proxy_balancer proxy_connect proxy_html
# service apache2 restart
  • create our Apache config file specific to our domain:
# cd /var/www/vhosts/system/domain.com/conf
# cat /var/www/vhosts/system/domain.com/conf/vhost.conf
<Proxy *>
    Order deny,allow
      Allow from all
</Proxy>
ProxyPreserveHost On
<Location "/">
      ProxyPass "http://127.0.0.1:5000/"
      ProxyPassReverse "http://127.0.0.1:5000/"
</Location>

Restart Apache

# service apache2 restart

Testing Flask application

  • go to the folder which contains the Flask app
$ cd /home/user/apps/domain
  • start the Flask app (just for testing, using python interpreter)
$ python app.py

Let's check to see it running on: http://www.domain.com

Works, so, now let's create also a Gunicorn config file and run our aplication in production using Gunicorn.

$ vim /home/user/apps/domain/gunicorn.conf
accesslog = "/home/deploy/apps/domain/logs/gunicorn_access.log"
errorlog = "/home/deploy/apps/domain/logs/gunicorn_error.log"
  • and we need to create also the directory for Gunicorn's log files
$ mkdir logs

So, the directory looks like:

$ ls -ltr
total 16
-rw-rw-r-- 1 deploy deploy  206 Feb 28 10:19 app.py
drwxrwxr-x 5 deploy deploy 4096 Feb 28 21:47 venv
-rw-rw-r-- 1 deploy deploy  142 Feb 28 21:48 gunicorn.conf
drwxrwxr-x 2 deploy deploy 4096 Feb 28 21:48 logs

Now, is time to run our Flask application in production using Gunicorn

(venv) deploy@myhost:~/apps/domain$ gunicorn -c gunicorn.conf -b 0.0.0.0:5000 app:app

Let's check again to see it running on: http://www.domain.com ==> Works

Good - it seems that up to now everything is configured and running.

One additional thing: you perhaps want to have your Flask application surviving at reboot, and also to start/stop it using service command. Ubuntu 14.04 is using Upstart, so we have to create one more file, let's call it /etc/init/myapp.conf with the following content (all errors will go to /var/log/upstart/myapp.log):

description "myapp"

start on (filesystem)
stop on runlevel [016]

respawn
setuid deploy
setgid deploy
chdir /home/deploy/apps/domain/

exec /home/deploy/apps/domain/venv/bin/gunicorn -c /home/deploy/apps/domain/gunicorn.conf -b 0.0.0.0:5000 app:app

And now we should be able to use service myapp start to start the application or service myapp stop to stop the application. For a similar start-up script for Systemd you can check my previous tutorial: Deploy your Flask Web Application on Ubuntu 16.04 with Apache, Gunicorn and systemd.

Install and configure Redis

Redis is an open source (BSD licensed), in-memory data structure store which can be used as a database, cache and message broker. We are using it in our Flask application for caching.

  • update our repos/packages
# apt-get update
  • install Redis
# apt-get install redis-server
  • check if Redis is running
# netstat -ant | grep 6379
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN
# service redis-server status
redis-server is running
  • is it working?
# redis-cli
127.0.0.1:6379> ping
PONG
  • do a quick performance check
# redis-benchmark -q -n 1000 -c 10 -P 5
PING_INLINE: 200000.00 requests per second
PING_BULK: 249999.98 requests per second
SET: 333333.34 requests per second
GET: 200000.00 requests per second
INCR: 200000.00 requests per second
LPUSH: 333333.34 requests per second
LPOP: 333333.34 requests per second
SADD: 499999.97 requests per second
SPOP: 499999.97 requests per second
LPUSH (needed to benchmark LRANGE): 249999.98 requests per second
LRANGE_100 (first 100 elements): 71428.57 requests per second
LRANGE_300 (first 300 elements): 26315.79 requests per second
LRANGE_500 (first 450 elements): 15384.62 requests per second
LRANGE_600 (first 600 elements): 9708.74 requests per second
MSET (10 keys): 142857.14 requests per second

In default installation (on Ubuntu 14.04) Redis is running only on localhost but it doesn't have any authorization mechanism turned on. So, we need to take care of that first. We have to edit the config file

/etc/redis/redis.conf
  • make sure is listening on localhost:
bind 127.0.0.1
  • make sure you add AUTH (foobared is the password here, be sure you have a good one)
requirepass foobared
# service redis-server restart
Stopping redis-server: redis-server.
Starting redis-server: redis-server.
# service redis-server status
redis-server is running
  • AUTH was added so, let's check to see if is working without password
# redis-cli
127.0.0.1:6379> ping
(error) NOAUTH Authentication required.
127.0.0.1:6379>

===> is not working without AUTH, GOOD

Let’s try using AUTH command

# redis-cli
127.0.0.1:6379> ping
(error) NOAUTH Authentication required.
127.0.0.1:6379> AUTH foobared
OK
127.0.0.1:6379> ping
PONG
127.0.0.1:6379>

===> now works. So, any Redis client should first authenticate using AUTH command, in order to proceed.

Install and configure MongoDB

MongoDB is a NoSQL database that offers a high performance, high availability, and automatic scaling enterprise database. Data is stored in a "document" structure in JSON format (in MongoDB called BSON).

  • update our repos/packages
# apt-get update
  • let’s install MongoDB
# apt-get install mongodb
  • check if it is running
# service mongodb status
mongodb start/running, process 6405
  • check if is listening only on localhost
# netstat -ant | grep 27017
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN
  • show some statistics (press CTRL+C to stop it)
# mongostat
  • show some statistics (10 rows, every 2 Seconds)
# mongostat --rowcount 10 2
  • let’s use MongoDB command iine
# mongo
MongoDB shell version: 2.4.9
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
> show dbs
local     0.078125GB
>
  • get out of mongo client: type exit

To create an admin user we have to login to Mongo server and use admin database:

# mongo
MongoDB shell version: 2.4.9
connecting to: test
> use admin
switched to db admin

NOTE: the installed MongoDB version is 2.4.9, and the syntax for user management changed for 2.6.0. Please check MongoDB's documentation.

Create a User Administrator

>db.addUser( { user: "root",
              pwd: "root_password",
              roles: [ "userAdminAnyDatabase" ] } )

Let’s add a user with read/write permissions, which has dbAdmin role as well

> use admin
switched to db admin
> db.addUser( {user:"admin", pwd:"admin_pass", roles:[ "readWrite", "dbAdmin"] } )
{
     "user" : "admin",
     "pwd" : "bfe79169688f26a49508d70ce4c0b01c",
     "roles" : [
          "readWrite",
          "dbAdmin"
     ],
     "_id" : ObjectId("58b5ce37c339fdf8b9638d30")
}
  • show all available users to be sure it was created
> show users
{
     "_id" : ObjectId("58b5ce37c339fdf8b9638d30"),
     "user" : "admin",
     "pwd" : "bfe79169688f26a49508d70ce4c0b01c",
     "roles" : [
          "readWrite",
          "dbAdmin"
     ]
}
>

If you want/need to add a user with only read access:

> db.addUser( {user:"readuser", pwd:"readuserpass", roles:[ "read"] } )

I will add only one more user for our Flask app (with read/write access):

> db.addUser( {user:"flask_user", pwd:"flask_user_pass", roles:[ "readWrite"] } )
> db.addUser( {user:"flask_user", pwd:"flask_user_pass", roles:[ "readWrite"] } )
{
     "user" : "flask_user",
     "pwd" : "7e80bce849898d7c85fe007147dc5c94",
     "roles" : [
          "readWrite"
     ],
     "_id" : ObjectId("58b5d011c339fdf8b9638d31")
}
  • and now we should have two users in our database, let's check if we are right
> show users
{
     "_id" : ObjectId("58b5ce37c339fdf8b9638d30"),
     "user" : "admin",
     "pwd" : "bfe79169688f26a49508d70ce4c0b01c",
     "roles" : [
          "readWrite",
          "dbAdmin"
     ]
}
{
     "_id" : ObjectId("58b5d011c339fdf8b9638d31"),
     "user" : "flask_user",
     "pwd" : "7e80bce849898d7c85fe007147dc5c94",
     "roles" : [
          "readWrite"
     ]
}
>

Now, the only thing left is to restart mongo with auth flag. In order to do that we have to change a line in Mongo's config file.

# vim /etc/mongodb.conf

and change the line #auth = true to

auth = true
  • restart mongodb
# service mongodb restart
mongodb stop/waiting
mongodb start/running, process 7354

Let’s test to see if everything works:

# mongo
MongoDB shell version: 2.4.9
connecting to: test
> use admin
switched to db admin
> show dbs
Tue Feb 28 20:36:39.328 listDatabases failed:{ "ok" : 0, "errmsg" : "unauthorized" } at src/mongo/shell/mongo.js:46
>

==> right, we are not authorised!!!

Everything should be installed and configured by now.