Building a website for my wife

Building a website for my wife.

My wife had a small work related project which involved classifying time-slots in a particular way. After modeling things using Zoho Creator (shoutout: its rather good for many things), we realized that it didn't quite fit what we were looking for,
with the major pain point being the price per record (we expect to have a lot of records)

So.. Being a Devops guy, and being that I've been meaning to learn about this django thing and something
about front end web development which I'd last done when registering a site with CERN (really!). To achieve
this, I went to my friendly home linux computer and fired up
python and started playing. What did I learn?

1) Django is asininely powerful, but beyond the simple examples its not always obvious how to wield the awesomeness.
2) The Django examples use functional views, whereas nearly all best practices recommend using class based views.
3) I didn't see any unit tests in the django example.
4) It would be nice if the examples demonstrated extending templates, etc.
5) OMG its butt ugly (I didn't discover bootstrap until a week in)
6) Start with the ElasticBeanstalk canned example, then everything is easy.

Obviously, this means that I really ought to contribute and improve the examples. Perhaps next month, I've
a site to build this month.

After playing around for a bit, I had an awfully ugly, but usable website. I even learned how
to write unit tests against Django to help keep my stuff somewhat rational (they saved me later)

All good, then I started looking up best practices because ugly. What did I learn? I was doing nearly
everything wrong. Refactor here, flip to class based views (mostly), etc. For those who don't
believe in unit tests, this is where they totally save your ass. I was able to refactor, and once
things passed my tests it all worked. Sometimes good practices really do help.

Then it became time to think about how to deploy the dang thing. Having personal interest
in playing with AWS, I started there. I then walked through the elastic beanstalk django sample
and figured out how to make a django site with elastic beanstalk, make it talk SSL, added a mysql
database, and squared that up.

I took a deep breath, then walked through the Elastic Beanstalk demo, then added an RDS
database to it, then converted it to use SSL. I found the
following very handy:

how to act like your env on the EB host:
http://stackoverflow.com/questions/21764319/django-admin-py-and-python-path-on-ec2-amazon-beanstalk

how to get django talking https on elastic beanstalk:
https://rickchristianson.wordpress.com/2013/10/31/getting-a-django-app-to-use-https-on-aws-elastic-beanstalk/


I started down the path of a self-signed certificate, which I'm recording how to do
for posterity. I needed this to feed Elastic Beanstalk since it doesn't yet support
ACM certificates.

openssl genrsa -out my-private-key.pem 2048
openssl req -sha256 -new -key my-private-key.pem -out csr.pem
openssl x509 -req -days 365 -in csr.pem -signkey my-private-key.pem -out my-certificate.pem
aws iam upload-server-certificate --server-certificate-name rvu-server-cert   \
   --certificate-body file://my-certificate.pem --private-key file://my-private-key.pem

Update the ELB with that. Which gives you a cert that you can use in the EB control panel for the Elastic Load Balancer so you can enable SSL.

Next, I got a valid certificate from AWS ACM so that my site wouldn't give SSL warnings.

I went and created a certificate using AWS ACM.  Sadly, ELB (ElasticBeanstalk doesn't yet seem to see ACM certificates, even when they are created. I then did this to assign it to my ELB:

1) Go to the AWS console, EC2->Load Balancers
2) Click on my load balancer
3) Click on the listeners tab, then edit.
4) Click change next to the certificate
5) Select "Change the certificate source "Choose an existing certificate from AWS Certificate Manager (ACM)"
6) Click Save,
7) Click Save again,
8) Click Close

Sadly, I don't see any good way for me to save this configuration in EB. I'm not
sure what it will do on next deployment. Once done, don't mess with the Network/ELB configuration
as it seems to want to force changing the certificate.



I found that using an if statement linked to an environment
(or json loaded config struct from s3) switching ssl on/off
made going from local test to remote ssl usage much easier.
I did this:

if configuration['django']['use_ssl']:
    SECURE_CONTENT_TYPE_NOSNIFF = True
    SECURE_BROWSER_XSS_FILTER = True
    SECURE_SSL_REDIRECT = True
    SECURE_FRAME_DENY = True
    SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
    SESSION_COOKIE_SECURE = True
    CSRF_COOKIE_SECURE = True
    CSRF_COOKIE_HTTPONLY = True
    X_FRAME_OPTIONS = "DENY"
    SECURE_HSTS_SECONDS = 3600
    SECURE_HSTS_INCLUDE_SUBDOMAINS = True

Ok, now I've seen that this is a plausible way to deploy my app. Time to
move it into the cloud.

I started off launching my dataplane, in this case an RDS MariaDB instance. Two
reasons for the selection over PostgresSQL: most of my database is static with joins,
so the result caches help substantially. Next, it was a few tenths of a cent cheaper in RDS.

The ElasticBeanstalk environments are able to manage a database for you, I wanted to do it on my own to play with the UI
and to get my data imported up front. Not much here other than needing to remember to set your security group inbound rules
so that appropriate ip addresses and/or subnets can access the database. Since I didn't
have any hosts in my VPC nor any subnets (EB creates its own), I elected to remove
the permissions RDS helpfully provided to permit my home computer to the unreachable
non-public database I created, then permitted the ip range of my VPC. Later, I'll
narrow this down to just my application's back-end security group when it is created.

I then started over and tried to import my app into elastic beanstalk. Mostly painless, but, a few pain points:

Least favorite error, right after I created my environment:

Environment health has transitioned from Severe to Degraded. Command is executing on all instances. Command failed on all instances. Unable to assume role "arn:aws:iam::ACCOUNT_ID:role/aws-elasticbeanstalk-ec2-role". Verify that the role exists and is configured correctly

AUGH!

What did I learn here?

1) The eb command line doesn't create appropriate IAM roles. Thats sad, I'd really like idiot mode.
2) Since I'd completely cleaned up my account, I coudln't fully crib the example without restarting it. So I did,
then worked backwards.

The fix:

I needed was two roles:

1) aws-elasticbeanstalk-ec2-role:

I used the policy template  AWSElasticBeanstalkService.
More importantly, I needed the following trust relationship:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "elasticbeanstalk.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "elasticbeanstalk"
        }
      }
    }
  ]
}

Only the first clause had been created by default, which was the cause of the tool.

Since I pull some configuration information from S3, I added an additional policy:  AmazonS3ReadOnlyAccess


I then told EB to rebuild my environment because I didn't want to get a new external dns name which I'd
happily handed to my dns server and was waiting for the 24h refresh I've set on my zone. Eb promptly
got rather upset because it couldn't delete the security group I'd permitted to my database. Sigh. Hand fix
permissions, and repeat through to success. Penalty me for doing it right and not just permitting
all IPs in my VPC like I'd started off doing.

Anyhow, database created, and user created for my app, so I can happily move
on to more important things like having web pages to show for my trouble.

2) aws-elasticbeanstalk-service-role

I used two policy templates: AWSElasticBeanstalkEnhancedHealth and AWSElasticBeanstalkService
and created the following trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "elasticbeanstalk.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "elasticbeanstalk"
        }
      }
    }
  ]
}

From there, off to the races.

Next step was playing configuration games. I didn't immediately bump into the environment variable stuffing that ElasticBeanstalk
has (very useful!), and started by pulling my configuration from s3 into a local file on disk and having my settings.py load it. This also let me easily differentiate between test and prod databases/users, so I've not yet seen fit to fully
change this.

Courtesy of both the AWS example and https://realpython.com/blog/python/deploying-a-django-app-to-aws-elastic-beanstalk/

I created the following modifications to my eb local files to make elastic beanstalk work
for me:

.ebextensions:

file 03_apache.config

container_commands:
  01_setup_apache:
    command: "cp .ebextensions/enable_mod_deflate.conf /etc/httpd/conf.d/enable_mod_deflate.conf"

file enable_mod_deflate.conf

# mod_deflate configuration
<IfModule mod_deflate.c>
  # Restrict compression to these MIME types
  AddOutputFilterByType DEFLATE text/plain
  AddOutputFilterByType DEFLATE text/html
  AddOutputFilterByType DEFLATE application/xhtml+xml
  AddOutputFilterByType DEFLATE text/xml
  AddOutputFilterByType DEFLATE application/xml
  AddOutputFilterByType DEFLATE application/xml+rss
  AddOutputFilterByType DEFLATE application/x-javascript
  AddOutputFilterByType DEFLATE text/javascript
  AddOutputFilterByType DEFLATE text/css
  # Level of compression (Highest 9 - Lowest 1)
  DeflateCompressionLevel 9
  # Netscape 4.x has some problems.
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  # Netscape 4.06-4.08 have some more problems
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  # MSIE masquerades as Netscape, but it is fine
  BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
<IfModule mod_headers.c>
  # Make sure proxies don't deliver the wrong content
  Header append Vary User-Agent env=!dont-vary
</IfModule>
</IfModule>

file db-migrate.config

container_commands:
  01_migrate:
    command: "django-admin.py migrate"
    leader_only: true
  02wsgipass:
    command: 'echo "WSGIPassAuthorization On" >> ../wsgi.conf'
  99customize:
    command: "scripts/customize.sh"
option_settings:
  aws:elasticbeanstalk:application:environment:
    DJANGO_SETTINGS_MODULE: myapp.settings

file scripts/customize.sh

#!/bin/bash
if [ ! -d /etc/myapp ]; then
  sudo mkdir /etc/myapp
  sudo chmod +rx /etc/myapp
fi
# mybucket is created in us-east-1, the region must match the location
# of the bucket.
aws s3 --region us-east-1 cp s3://mybucket/etc/myapp/config.json /tmp/config.json && sudo mv /tmp/config.json /etc/myapp