My current set of tools for deploying my application to production includes Packer and Terraform. I didn't write all of the code for the deployments, but I've been over most of it now.

When trying to upgrade my server from Ubuntu 20.01 to 22.01, I ran into some problems with conflicting versions of dependencies (related to postgres, mostly). My first instinct was to create an EC2 instance, then walk through each step manually, but I realized I didn't even have to spin up an instance if I could use Docker.

I'm not much of a Docker user, but I've used it a few times professionally. Mostly other people have done the hard work of creating a docker image, and I've run it for development. So I thought this was a great opportunity to try using it myself.

I started by spinning up a docker image for the target Ubuntu version, named jammy:

docker pull ubuntu:jammy
docker create -it --name jelly ubuntu:jammy
docker start jelly
docker attach jelly

Then running through the scripts from my packer build manually:

echo "deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main"> /etc/apt/sources.list.d/pgdg.list
apt-get update
apt-get install -y wget gnupg
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -

export DEBIAN_FRONTEND=noninteractive
apt update && apt upgrade -y && apt auto-remove -y && apt update

apt-get install -y unzip libpq-dev postgresql-client-14 nginx curl awscli logrotate

This is where I get the errors about conflicting versions or versions not available. Did you catch it?

echo "deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main"> /etc/apt/sources.list.d/pgdg.list

The bionic in there refers to the version of Ubuntu that we are requesting the dependencies for. It should be jammy!

This was a great example where Docker saved me some time (and a few pennies) from not having to spin up a cloud instance. And there really wasn't much to learn about Docker itself to dive in. So my advice to you is to give Docker a try for a simple use case like this. Despite being used in massive (sometimes very complicated) build chains, Docker is a relatively simple technology to get started with if you already have some familiarity with the Linux version you are planning to use in your container.

Maybe someday I'll start using it for building a small project...

This is a short post to tell you to create more bash aliases.

I have had one for my devops setup for a while because I have to set several environment variables to get just about anything done. More recently, though, I set up a few more for my main Django project and even this blog. For the blog it's a simple as entering the directory and activating the environment:

alias pmlstart='cd ~/pml_blog && source env/bin/activate'

For the Django project, it took me forever to remember how to start my local postgres instance (is it service start, or start service, or...?), so I finally just made that part of an alias for when I start working on the project:

alias webstart="cd ~/website && source env/bin/activate && sudo service postgresql start && source env_vars.sh"

Since writing this, I added that last piece to it to load environment variables into my environment so I don't have to remember to edit my activate file when I need to recreate my environment locally. Highly recommended!

Are there bash commands you use frequently together? Make an alias!

There are a few code formatting tools I like to use in just about any Python (and Django) project: black, isort, and flake8. These all serve slightly different purposes, and there are alternatives to each. A full discussion/comparison of code formatting tools in Python is beyond the scope of this post*, but in brief:

• Black is great because it automatically formats my code with a well recognized style. I have my editor (currently VSCode) set to run black when I save a file. This makes my code consistent, and is great in teams I've worked in because it means we don't have to worry about styling in code reviews.
• isort is great for sorting imports. Even though I tend to do a bit of this sorting as I go, I inevitably swap an import or leave one out of place. This organizes them so I don't have to.
• flake8 catches minor (and sometimes not so minor) code issues like unused variables.

There are a few ways to configure these libraries, but for now, I have them in their own individual config files. These should all be placed at the root of your Django project.

For black, I use the default config. In some other projects, I've extended the line length, but currently I'm leaving black as default.

For isort, isort.cfg is the filename and mine looks like this:

[settings]
skip_glob=env/*
extend_skip_glob=**/migrations
profile=black

Where env is the name of your environment. The extended skip for migrations isn't necessary, but I wanted to avoid changing my migration files. The black profile is nice to avoid conflicts between black and isort. Skipping your environment is likely required. When I've left this out (or accidentally run isort without the config) isort runs against the libaries in my environment and ends up breaking them by creating circular dependencies. One way to make sure you aren't going to do this is to run isort --check-only . to make sure it's only running against your files before actually running it.

For flake8 .flake8 is the filename and mine looks like this:

[flake8]
exclude =
    migrations
    __pycache__
    manage.py
    settings.py
    env
    .pytest_cache
    .vscode

These exclude all of the files in the folders migrations, __pycache__ and env (where env is the name of your environment). You may want to exclude more, or not exclude some of these, but I've found this to work for me.

Finally, on other projects I've used all of these with pre-commit, which is a library that uses git hooks to prevent you from committing code that doesn't use the standards you and your team have set, such as black, isort, and flake8, or some other combination of code formatting tools. It also allows you to use a single command to run all three tools.

Unfortunately, I can't figure out how to use pre-commit in my current setup with my python projects in WSL and my git client (Fork) in Windows. Perhaps this is a sign I should start using the git CLI more. But if you know of a good way to use precommit with this setup, let me know!

*between writing and publishing this I read James Bennett covering similar ground. He goes a bit deeper on some of the tools I mention (and other related tools), so I wanted to link this for further reading.

I found the documentation for presigned URLS on AWS using boto to be insufficient, so here is how I formed the request.

I wanted to have a file private in S3, but expose it to a user for download if they were authorized to access it. I found the easiest way to do that was leave the object in S3 as private, but use the AWS API to generate a pre-signed URL with a low timeout (a minute or so). Unfortunately, I found the documentation not so straightforward and cobbled together a few Stack Overflow answers to come up with exactly what I needed. In particular, I needed to add the ‘ResponseContentType’ to get the file to download correctly, and needed to specify my AWS credentials.

I’m using Django to serve up the download via a get request:

class GetDownloadURL(APIView):

    def get(self, request):
        # Get the service client.
        session = boto3.session.Session(profile_name="AWSUserName")
        s3 = session.client("s3")

        # Generate the URL to get 'key-name' from 'bucket-name'
        url = s3.generate_presigned_url(
            ClientMethod="get_object",
            Params={
                "Bucket": "your-s3-bucket",
                "Key": "SampleDLZip.zip",
                "ResponseContentType": "application/zip",
            },
            ExpiresIn=100,
        )

        return Response(url)

Notes: You will need to update the Key and Bucket params to match what you have in S3. Depending how you have set up your AWS credentials, you may be able to omit the ‘profile_name="AWSUserName"’ parameter. I prefer to be explicit in my config because I’ve run into issues when I have used the default in the past.

I am creating a Django application using PostgreSQL (PSQL) for my database and was nearly finished with the API when I discovered some strange behavior. After successfully testing the API in the Django app, I decided to run some basic queries on the database. I received the following error for nearly every field in the app:

    select MaxSceneKey from game_progress_gameplaykeys;
    ERROR:  column "maxscenekey" does not exist
    LINE 1: select MaxSceneKey from game_progress_gameplaykeys;
                   ^
    HINT:  Perhaps you meant to reference the column "game_progress_gameplaykeys.MaxSceneKey".

I was getting the same result for every field in the table that I tried (and when I try to include the table name as the hint suggests), except for ‘user_id’ and ‘objective'.

I confirmed that the fields existed using \d+ game_progress_gameplaykeys, tried changing some of their field types, and even upgraded from Postgres 9.5 to 10.5 (I was planning to do this anyway).

After a bunch of searching, I found the issue:

“All identifiers (including column names) that are not double-quoted are folded to lower case in PostgreSQL.” from https://stackoverflow.com/questions/20878932/are-postgresql-column-names-case-sensitive

I created camelCase field names in my Django app based on what the field names were previously in my application (written in C#).

I decided to fix this (for now) by fixing my models to all use snake_case and using https://github.com/vbabiy/djangorestframework-camel-case to switch the keys from camelCase to snake_case when they come into the API. One issue solved!

While debugging that issue, I decided to update my laptop’s code + postgres version since I hadn’t worked on it in a while and wanted to see if the issue was just on my desktop. When I reinstalled PSQL, I couldn’t seem to log into it using the user I was creating. Using the postgres user was fine, though.

I finally figured out the issue was that PSQL was running on port 5433, not 5432 (the default). After that, I was puzzling over what could be running on 5432 since ‘netstat’ and ‘lsof’ revealed nothing else running on my WSL Ubuntu VM. As I was searching around, I saw someone mention that really only PSQL should be running on that port, and I realized I had installed PSQL on Windows on that machine before I moved over to WSL. I uninstalled that, switched back to 5432 in Linux, restarted PSQL, and boom, good to go.

While I was debugging that issue, I learned some good information about PSQL along the way:

/etc/postgresql/10/main/postgresql.conf allows you to set and check the port that PSQL is running on.

/etc/postgresql/10/main/pg_hba.conf allows you to set different security protocols for connections to PSQL. Notable for local development: set the local connection lines to ‘trust’ so you don’t have to enter a password when logging in.

Note: you need to restart the PSQL server for either of these changes to take effect. Note 2: MORE IMPORTANT NOTE: Don’t use trust anywhere other than a local version of PSQL. Ever.

These are the lines I had to change to get that to work (may be different in versions of PSQL other than 10.5):

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

While trying to get more familiar with Django, I started watching talks from DjangoCon from the last few years. I can’t seem to find the talk, but one of them had a list of great Python/Django talks, which inspired me to create me own list (with some definite overlap).

I have found that revisiting talks like these make me reconsider some design problems that I have recently worked through, so I want to keep a list and rewatch these periodically. I will likely add to this list in the future.

• Greg Ward - How to Write Reusable Code

• You Don’t Need That! - Christopher Neugebauer (About Design Patterns in Python)

• Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code

• Raymond Hettinger - Transforming Code into Beautiful, Idiomatic Python

• Raymond Hettinger - Python's Class Development Toolkit

• Loop like a native: while, for, iterators, generators

These two go together (from DjangoCon2015): - Jane Austen on Pep 8 - Lacey Williams Henschel - The Other Hard Problem: Lessons and Advice on Naming Things by Jacob Burch

Because I feel much more comfortable in a terminal than on the Windows command line (or in Powershell), I’ve really been enjoying Windows Subsystem for Linux (WSL). In fact, I use it almost exclusively for accessing the server I run this blog from. WSL is essentially a Linux VM of only a terminal shell in Windows (with no GUI access to Linux) and no lag (like you get in most VMs).

When I created my Grocery List Flask App, I began by using WSL. However, I ran into an issue that prevented me from seeing a locally hosted version of the API in Windows, so I switched to the Windows command line for that app.

Recently, I’ve been developing a Django application (more on that in a future post), and I ran into a similar issue. Between posting the issue with localhost on WSL and starting this new app, there was a response that I had been meaning to check out. I found that for Django and PostgreSQL, making sure everything was running from localhost (or 0.0.0.0) instead of 127.0.0.x, seemed to fix any issues I had. PSQL gave me some issues just running within WSL, but I found that I just need to add '-h localhost' to get it to run.

Below are the commands I used to get Conda, Django, and PSQL all set up on my PC and then again my laptop. This works for Django 2.0, PSQL 9.5, and Conda 4.5.9.

Installation Instructions

Edit: I originally had installation instructions in here for PSQL 9.5. If you want 9.5 in Ubuntu, good news! You already have it. To install the newest version of PSQL, you should uninstall that version first, then install the new version from here

Install Conda (need to restart after installing for it to recognize ‘conda’ commands)

#create environment
conda create --name NameOfEnvironment
#activate environment
source/conda activate NameOfEnvironment
#install Django
conda install -c anaconda django
#install psycopg2, to interface with PSQL
conda install -c anaconda psycopg2

If you get ‘permission denied’, or it hangs, just rerun the install command that failed. Not sure why, but that fixed things for me.

#Remove PSQL from Ubuntu:
sudo apt-get --purge remove postgresql\*
#Then run this to make sure you didn’t miss any:
dpkg -l | grep postgres

#Install PSQL 10 using instructions here: https://www.postgresql.org/download/linux/ubuntu/ I have copied them here for convenience, but please double check that they have not changed
#Create the file /etc/apt/sources.list.d/pgdg.list and add the following line:
#deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main

#Then exeucte the following three commands
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get install postgresql-10

sudo service postgresql start    
sudo -i -u postgres -h localhost
createuser --interactive
       psql user: local_user
       y to superuser

#create the database
createdb local_db
#log into local_db
psql -d local_db

#privileges for Django to modify tables.
GRANT ALL PRIVILEGES ON DATABASE local_db TO local_user;

ALTER USER local_user WITH PASSWORD 'password';

'\q' to quit interactive console.
'exit' to leave postgres as postgres user.

#one line command to log in as the user to check tables during development.
psql -h localhost -d local_db -U local_user

python manage.py makemigrations
python manage.py migrate

Now log back in to PSQL using the line above, then enter '\dt' and you should see tables like django_admin_log, django_content_type, django_migrations, and django_sessions. Your PSQL DB is now connected to your Django app!

#optional for now, but allows you to ensure db connection works by storing credentials for the superuser you create.
python manage.py createsuperuser

#command to run the server. go to localhost:8000 in your web browser to view!
python manage.py runserver 0.0.0.0:8000

I used this post for reference