EC2 Deployment Part 3:





Overview

In this installment we will finally use nginx, uWSGI and Django together on our ec2 instance to illustrate how a simple website can be deployed.

Configuration of nginx

Previously, in part 1 of this series we installed nginx and verified that we could access our EC2 instance via port 80. We were able to confirm this by visiting the site on our browsers and seeing the default nginx page. To go beyond that default behavior we will need to further configure nginx.

The primary nginx configuration file should be located at /etc/nginx/nginx.conf. This configuration file was used earlier when we started the server and used our browser to investigate port 80. Rather than modifying this main configuration file it is common to create "modular" configuration files that specify how nginx should proceed for a particular website. Here we will create one of the modular configuration files for our simple site that instructs nginx to listen to port 8000 of our instance for incoming requests.

We are using port 8000 rather than 80 for testing purposes. When you visit the site in a browser you will need to specify you wish to hit port 8000 by appending :8000 to the IP address.

When installed on the Amazon Linux (and other Centos like Linux systems) nginx is configured to look into the /etc/nginx/conf.d/ directory for these modular configuration files. Note that if you are doing this on an Ubuntu/Debian OS this will be different and you will probably want the /etc/nginx/sites-enabled/ directory instead.

Instead of putting our configuration file into the /etc/nginx/conf.d/ directory directly we will house the file in our Django project and then create a symbolic link from the /etc/nginx/conf.d/ to it. Create the configuration file in the main project directory (~/testProj/), I named mine testProj_nginx.conf. Into this file we will copy the contents of the config file from the uWSGI tutorial and modify it appropriately.

# testProj_nginx.conf

# the upstream component nginx needs to connect to
upstream django {
         server unix:/home/ec2-user/testProj/testProj.sock; # for a file socket
         #server 127.0.0.1:8001; # for a web port socket (we'll use this first)                     
}

# configuration of the server
server {
    listen      8000; # the port your site will be served on
    server_name <instance IP>; # IP of ec2 instance
    charset     utf-8;

    # max upload size
    client_max_body_size 75M;

    # Django media
    location /media  {
        alias /home/ec2-user/testProj/media; # your Django project's media files
    }

    # Django static
    location /static {
        alias /home/ec2-user/testProj/static; # your Django project's static files
    }

    # Finally, send all non-media requests to uWSGI
    location / {
        uwsgi_pass  django;
        # we will create the following uwsgi_params file soon in our project
        include     /home/ec2-user/testProj/uwsgi_params;
    }
}

The upstream django block indicates a Unix socket that can be used by nginx to forward requests to uWSGI (which then will interact with the Django project). You can see in the server block that the port on which we want nginx to listen and the server IP address are listed (change <instance IP> to the public IP for your EC2 instance). Next are the location blocks that are used by nginx to decide how the incoming requests should be handled based on their URI. See this article for a deeper explanation of how nginx uses these blocks to route requests.

For our purposes we want nginx to serve as a reverse proxy server. It will return the static media content, that is the content within the static and media folders of a Django project, on it's own; hence the location /media and location /static blocks that direct the server to the actual location of the files.

When nginx processes a more complex request, that doesn't pertain to the static/media content, it forwards it to uWSGI through the upstream django Unix socket that we indicated at the top of test_project.conf. That socket is created when we run uWSGI. We need now need to create the uwsgi_params file within our Django project, as indicated by the path above. The content of the file can be copied from your nginx installation at /etc/nginx/uwsgi_params or from this listing:

# uwsgi_params

uwsgi_param  QUERY_STRING       $query_string;
uwsgi_param  REQUEST_METHOD     $request_method;
uwsgi_param  CONTENT_TYPE       $content_type;
uwsgi_param  CONTENT_LENGTH     $content_length;

uwsgi_param  REQUEST_URI        $request_uri;
uwsgi_param  PATH_INFO          $document_uri;
uwsgi_param  DOCUMENT_ROOT      $document_root;
uwsgi_param  SERVER_PROTOCOL    $server_protocol;
uwsgi_param  REQUEST_SCHEME     $scheme;
uwsgi_param  HTTPS              $https if_not_empty;

uwsgi_param  REMOTE_ADDR        $remote_addr;
uwsgi_param  REMOTE_PORT        $remote_port;
uwsgi_param  SERVER_PORT        $server_port;
uwsgi_param  SERVER_NAME        $server_name;

Testing nginx and uWSGI together

With these files in place we should now be able to perform a test to ensure that nginx is listening on port 8000 and communicating with uWSGI. First we will use our local browser to send a request to <instance IP>:8000. Then nginx should forward that request to uWSGI which will handle the request as we have specified in test.py.

First first restart nginx, if you get an error here check for typos in testProj_nginx.conf.

$ sudo service nginx stop
$ sudo service nginx stop

Now we can start uWSGI, run it from you home directory.

$ cd ~
$ uwsgi --socket testProj/testProj.sock --wsgi-file test.py

Note: We don't need to create the socket testProj.sock file within the Django project. We only do that because we configured nginx to find it there in testProj_nginx.conf. We configured it that we since we know that eventually we will be running the actual Django app and we might as well contain all the associated files within the project directory.

When I run this command and then go to <instance IP>:8000 in my browser I get a "502 Bad Gateway" error. Here is a quote from the best practices section of the uWSGI documentation that suggests what our issue may be:

" If you plan to use UNIX sockets (as opposed to TCP), remember they are standard filesystem objects. This means they have permissions and as such your webserver must have write access to them. "

This issue is discussed in the other tutorials and you can see for yourself if you go to /var/logs/nginx/error.logs (you may need to use $ sudo su before you can access the file). At the bottom of the log you should see something like this.

018/09/08 14:21:44 [crit] 9130#0: *3 connect() to unix:/home/ec2-user/testProj/testProj.sock
failed\ (13: Permission denied) while connecting to upstream, ...

That error indicates that when nginx tried to access the Unix socket, which is the file ~/testProj/testProj.sock it had insufficient permissions. nginx needs to be able to write to the socket file.

So the next question should be: what are the permission settings for that socket? Unfortunately the default setting for uwsgi is to delete the socket when the process concludes but if we run uwsgi as a background process we can examine what the permission settings of the testProj.sock file are.

 $ uwsgi --socket testProj/testProj.sock --wsgi-file test.py &
   ...
 $ ll testProj/
 total 16
 -rw-r--r-- 1 ec2-user ec2-user    0 Sep  6 16:22 db.sqlite3
 -rwxrwxr-x 1 ec2-user ec2-user  540 Sep  6 16:20 manage.py
 drwxrwxr-x 2 ec2-user ec2-user   74 Sep  7 23:03 testProj
 -rw-r--r-- 1 ec2-user ec2-user 1059 Sep  8 13:42 testProj_nginx.conf
 srwxrwxr-x 1 ec2-user ec2-user    0 Sep  8 14:37 testProj.sock
 -rw-rw-r-- 1 ec2-user ec2-user  147 Sep  8 14:18 test.py
 -rw-r--r-- 1 ec2-user ec2-user  664 Sep  8 14:05 uwsgi_params

So the socket file has its permission settings as 775, i.e. read, write and execute for owners and members of the owner's group but only read and execute permissions for all other users. It says the we, "ec2-user", are the owner of the file, that makes sense since we ran the uWSGI process that created it, and our user group, also named "ec2-user", is the relevant group. We can see if we run the id command that the nginx worker process, that will be trying to use the socket, is not a part of our group.

$ id nginx
uid=997(nginx) gid=995(nginx) groups=995(nginx)

It is not surprising then that nginx can not use the socket, it won't have write permissions.

There are several solutions to this problem that are presented in the various tutorials I have referenced so far. Before we move onto any of them you need to change the permissions of your home folder to allow any user read and execute permissions:

$ chmod +rx /home/ec2-user

Once that is done there are two possible solutions for dealing with the socket permissions.

The simple solution is to force uWSGI to make the socket file permissions less restrictive. Do this by adding another argument to our command:

$ uwsgi --socket testProj/testProj.sock --wsgi-file test.py  --chmod-socket=666

This creates the socket with 666 permissions so anyone, including the nginx process can read and write to it. If you run this command you should be able to access port 80 with your browser and see the "Hello World" text that comes from test.py.

A more permanent solution, suggested here, is to simply add the nginx worker process to our user group, "ec2-user". Adding a user to a group can be done with a Unix command

$ sudo usermod -a -G ec2-user nginx

Now the default permissions for the socket file created by uWSGI will permit nginx to read and write since it is in the owner group (you will need to stop and restart nginx for the changes to take effect). We can now execute the earlier uWSGI command and should be able to access port 8000 with our browser to see the "hello world" response.

Running nginx, uWSGI and our Django application together

Before we go any further we can make a configuration file for uWSGI so that we can stop typing all of the arguments into the command line every time we want to run it. We will simultaneously introduce arguments that direct uWSGI to run the Django project rather than the test.py script. Following the uWSGI tutorial we can create the file testProj_uwsgi.ini within our main Django project directory

$ touch /home/ec2-user/testProj/testProj_uwsgi.ini

and then we can copy and modify the file contents from the tutorial

# testProj_uwsgi.ini
[uwsgi]

## Django-related settings
# the base directory (full path)
chdir           = /home/ec2-user/testProj
# Django's wsgi file
module          = testProj.wsgi
# the virtualenv (full path)
venv            = /home/ec2-user/.virtualenvs/djangoEnv

## process-related settings
# master
master          = true
# maximum number of worker processes
processes       = 10
# the socket (use the full path to be safe)
socket          = /home/ec2-user/testProj/testProj.sock
# ... with appropriate permissions - may be needed
# chmod-socket    = 664
# clear environment on exit
vacuum          = true

You can see that the arguments we were passing to uWSGI through the command line previously are represented here. First we indicate which directory uWSGI should run from with chdir. Next we specify the same module file, testProj.uwsgi that we used in Part 2 of this tutorial series when running our Django project with uWSGI alone. After that we again need to specify which virtual environment we are using, just like we did before (this time, unless you have the virtualenv active the VIRTUAL_ENV environment variable will not be available so just use the absolute path instead).

Further down we again specify the socket we will be using. You can use all of the same arguments that we used earlier for the command line , e.g. chmod-socket. We don't need that one now since we added the nginx worker process to our user group. The final argument, vacuum, will instruct uWSGI to delete the socket files upon exit when set to true.

We can now run our entire stack; the nginx webserver, uWSGI and the Django project, on port 8000.

$ cd ~/testProj/
$ sudo service nginx start
$ uwsgi --ini testProj_uwsgi.ini

When you visit <instance IP>:8000 you should see the Django splash page which indicates that everything is working correctly.

Conclusion

This concludes this tutorial series on deploying a Django project from an AWS EC2 instance.