Programming My Life - Django

Programming My Life

Django Tasks - Cronjobs vs Celery

Like my previous vs post, this isn't a showdown. Instead, this post is about matching the solution to the problem. The tl;dr is that Celery is great at what it does, but it requires more infrastructure and setup than I needed to solve my problem.

The Problem to Solve

I needed to sync data from my database to Canvas, a learning management system where instructors store grades. If I had a year to do this, I would send the data directly to Canvas as it comes to my server. Unfortunately, I had a much more limited time frame to get this done, and that solution would require a lot more testing and robust error handling in cases where the data does not go through.

Before starting, I knew Celery was a common solution for task based workflows in Django, but I wanted to see what other options there were and how they stack up. I did some research and posted this to see if there was anything I missed in my research, which got some helpful responses.

Based on those responses, I started looking into setting up cron jobs to sync the data. I decided that Celery seems to be a great product that handles a lot of complexity for tasks (long and short running), but it was more effort to set up and maintain than I needed.

Management Commands and Cron Jobs

I created management commands to nail down the code while I looked into libraries for setting up cron jobs with Django. Several were no longer supported, and I didn't find any that I felt confident in. I began looking into how to set the cron jobs up myself in the packer scripts I already had for deployments. Fortunately, it was pretty easy. I created a shell script for each cron job that activates the proper environment on the server (to get the appropriate context to connect to my database and Django application) then runs a management command. These are copied by packer onto the server instance. I referenced those scripts in cronjob.sh which packer executes while setting up the server to set up the cron jobs. It looks like this:

...
(crontab -l 2>/dev/null; echo '11 * * * * /opt/website/cron_canvas_user_sync.sh >> /opt/website/logs/cron_users.log 2>&1') | crontab -
...

Explaining cron syntax is outside the scope of this post, mostly because I don't entirely understand it. Crontab Guru was incredibly helpful in setting these up.

The only (minor) difficulty was having to split my packer script into two so that the cron jobs wouldn't be set up on my staging server.

This has worked great! Initially I had three separate cron jobs running:

One to sync account names on our service with those in Canvas.
One for quiz grades.
One for game progress grades.

The first has since been scoped down a bit, but I still need it for a specific case. I recently combined the last two because they were happening simultaneously and sending data to the same endpoint. I hadn't realized that when I first created them.

In the future, I'd like to set up our server to send the grades directly to Canvas when they come in, but for now, this is working well.

Another alternative to these solutions is Django Q2, which Jeff Triplett has written about. It wasn't as established when I was considering what to adopt, but it's worth considering now.

[TIL] Using Fixtures with pytest `parametrize`

Published: Thu 08 February 2024
By Andrew Mshar

In Django.

tags: python django testing til

For longer than I care to admit, I was running my end to end tests in only Firefox. I thought it would be trivial to add a fixture to have my tests run using multiple browsers. Unfortunately, there isn't an easy way in pytest to use fixtures as parametrized values …

[TIL] Mocking Chained Calls

At work last week, I was asked to write tests for code that reaches out to Twilio's verify service. Here is what that code looks like (from the Twilio docs):

client = Client(account_sid, auth_token)

verification = client.verify \
                     .v2 \
                     .services('VAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX') \
                     .verifications \
                     .create(to='+15017122661', channel='sms')

I was able to …

Strategies for a Long Running Side Project

Published: Tue 19 December 2023
By Andrew Mshar

In Django.

tags: django python

At DjangoConUS 2023, I gave a lighting talk with this title. It was a lot of fun, and I'd suggest you try to give a lightning talk if you haven't before. It's a great way to try giving a talk at a conference without having to commit to a full …
Using SQLite for a Django application on fly.io

Published: Mon 06 November 2023
By Andrew Mshar

In Django.

tags: django python

I'd like to run a few small apps I've been developing for learning purposes (to do list, etc.) on a cheap or free platform as a service (PaaS) provider so I don't have to worry about a full deployment setup like I have on my main project. These projects aren't …
Moving from Django Microframework to Django

Published: Mon 30 October 2023
By Andrew Mshar

In Django.

tags: django

At DjangoConUS 2019, Carlton Gibson gave a talk called Using Django as a Micro-Framework, where he demonstrated a single file implementation of "Hello, World!" in Django.

At the sprints during DjangoConUS 2023, Paolo Melchiorre showed off an even more minimal version that sparked some great discussion. Specifically, the usage of …
Testing Stripe Integration in Django

Published: Wed 12 July 2023
By Andrew Mshar

In Django.

tags: stripe python django testing

I have been using Stripe to process payments for my company AstroVenture (see more context in a previous post here) for a couple of years now, and it has worked great. One thing that took me quite a while to figure out was how to test that we were properly …
Lessons Learned Teaching Undergraduate Astronomy with a Video Game - Testing

Published: Thu 29 June 2023
By Andrew Mshar

In Django.

tags: python django testing

This is the fourth and final installment of the series breaking down my talk from DjangoConUS 2022. The first entry covered background information about the project, the second was about using Django Rest Framework, and the third was about infrastructure and deployment.

Before diving in, I'd like to emphasize that …
Lessons Learned Teaching Undergraduate Astronomy with a Video Game - Infrastructure and Deployment

Published: Thu 22 June 2023
By Andrew Mshar

In Django.

tags: python django devops

This is the third installment of the series breaking down my talk from DjangoConUS 2022. The first entry covered background information about the project and the second was about using Django Rest Framework.

First, some important context: if you are a devops engineer, or have a lot of experience with …
Lessons Learned Teaching Undergraduate Astronomy with a Video Game - Django vs Django Rest Framework (DRF)

Published: Thu 15 June 2023
By Andrew Mshar

In Django.

tags: python django django-rest-framework

If you've ended up here from somewhere outside of this blog, and are looking for an exhaustive comparison of these two libraries, I regret to inform you, this isn't that. If you're here for the next installment of the series breaking down my talk from DjangoConUS 2022, welcome back!

This …