Programming My Life
  1. Moving from Django Microframework to Django

    At DjangoConUS 2019, Carlton Gibson gave a talk called Using Django as a Micro-Framework, where he demonstrated a single file implementation of "Hello, World!" in Django.

    At the sprints during DjangoConUS 2023, Paolo Melchiorre showed off an even more minimal version that sparked some great discussion. Specifically, the usage of lambda in this initial version of Paolo's effort made me realize that using the original function based version, you could build from that single file to what an early Django project typically looks like. After a great discussion over lunch with Eric Matthes about this, I decided to write this post to show just that: going from a single file to a project where a user has run startproject and startapp.

    I hope this demystifies some of the 'magic' behind Django and we can learn some things in the process. Note that we are using Django 4.2.X for this example.

    Getting Started

    I forked Will Vincent's repo for django-microframework and have made a few modifications to get us started. First, I renamed hello_django1.py to hello_django.py because we only need one file, and this one was closer to what I wanted to build from. In my fork, the main branch will have the final version of this code, but I've also created branches that start at the beginning and mark important steps in our journey. Here is the first branch.

    Writing Blog Posts

    We are going to be writing a blog, so let's start with some 'blog posts' in hello_django.py:

    from django.conf import settings
    from django.core.handlers.wsgi import WSGIHandler
    from django.core.management import execute_from_command_line
    from django.http import HttpResponse
    from django.urls import path
    
    settings.configure(
        ROOT_URLCONF=__name__,
        DEBUG=True,
    )
    
    def hello_world(request):
        return HttpResponse("Hello, Django!")
    
    def second_post(request):
        return HttpResponse("This is my second blog post. I'm still figuring this whole thing out.")
    
    def third_post(request):
        return HttpResponse(
            "This is my third blog post. I'm feeling more comfortable with this whole thing, and I'm going to try to write a bit more. "
        )
    
    def fourth_post(request):
        return HttpResponse(
            "Hot dogs are not a sandwich. In this blog post I will first define the meaning of sandwich, then I will ???? which will conclusively prove that hot dogs are indeed not sandwiches."
        )
    
    urlpatterns = [
        path("hello-world", hello_world),
        path("second-post", second_post),
        path("third-post", third_post),
        path("fourth-post", fourth_post),
    ]
    
    application = WSGIHandler()
    
    if __name__ == "__main__":
        execute_from_command_line()
    

    Here is the branch for that code.

    Now that we have those, hello_django.py is getting a bit long, and we need to focus on our writing. What can we do to simplify our code so we can do that? Let's move some of the code from hello_django.py into a few other files.

    manage.py

    First, we can move the last couple of lines (and the associated import) from hello_django.py to manage.py:

    from django.core.management import execute_from_command_line
    
    if __name__ == "__main__":
        execute_from_command_line()
    

    But that yields:

    django.core.exceptions.ImproperlyConfigured: Requested setting DEBUG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
    

    So we define DJANGO_SETTINGS_MODULE and now we have a working manage.py:

    from django.core.management import execute_from_command_line
    import os
    
    if __name__ == "__main__":
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hello_django.settings")
        execute_from_command_line()
    

    I'd like to pause here to highlight that a lot of magic misdirection happens when we run python manage.py runserver. In order to understand what is happening, though, we only need to go to the Django source code for execute_from_command_line(). This function handles whatever command you call after python manage.py, such as runserver, makemigrations, and migrate. Walking through these is a post (maybe a series) in and of itself, but at least now we know where to find what happens when we call a command using python manage.py.

    Back to our Blog

    We still have a few more lines that we can move out of hello_django.py to really focus on our posts. Let's move everything that isn't the posts themselves into their own files starting from the top, moving settings.configure() into settings.py:

    from django.conf import settings
    
    settings.configure(
        ROOT_URLCONF="urls",
        DEBUG=True,
    )
    

    Now, we can match the change we made to the location of our URLs in settings.py by creating urls.py:

    from django.urls import path
    from hello_django import hello_world, second_post, third_post, fourth_post
    
    urlpatterns = [
        path("hello-world", hello_world),
        path("second-post", second_post),
        path("third-post", third_post),
        path("fourth-post", fourth_post),
    ]
    

    Finally, let's create wsgi.py:

    from django.core.handlers.wsgi import WSGIHandler
    
    application = WSGIHandler()
    

    Awesome. Now hello_django.py contains only the blog posts.

    But we've done something else here, too.

    Let's take one final step to see what else we've accomplished. We can make a folder called config (the name isn't important, so you can call this what you want) and move our new files there: wsgi.py, settings.py, and urls.py.

    Now we have to match our paths so we update manage.py:

    from django.core.management import execute_from_command_line
    import os
    
    if __name__ == "__main__":
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.settings")
        execute_from_command_line()
    

    and settings.py:

    from django.conf import settings
    
    settings.configure(
        ROOT_URLCONF="config.urls",
        DEBUG=True,
    )
    

    Here is a link to the third branch with the current state of our code.

    We've just done (almost) everything that happens when you create a new django project and execute:

    django-admin startproject config .

    Two notable exceptions:

    First, we have not created asgi.py, which gets created with new projects as of Django 3.0. This isn't an asynchronous project, so we can leave it out.

    Second, you may notice our manage.py has less code than the version that Django creates and that we set our DJANGO_SETTINGS_MODULE environment variable in manage.py rather than in wsgi.py. I'm choosing to leave these as is for this exercise to simplify moving from the original code to our destination. I wanted to mention it in case it confuses any readers looking at those files in their own projects, but I don't want to slow us down by diving into every difference between this project and a project where you've run startproject.

    After startproject

    You may notice that hello_django.py now looks an awful lot like views.py that you're likely familiar with from any other Django project, so let's rename it to reflect that.

    Since we're going to want to write longer blog posts than we currently have, it doesn't make much sense to hard code our blog entries. Let's store those in a database. For that, we'll need to set up models.py and add DATABASES to settings.py:

    from pathlib import Path
    from django.conf import settings
    
    # Build paths inside the project like this: BASE_DIR / 'subdir'.
    BASE_DIR = Path(__file__).resolve().parent.parent
    
    settings.configure(
        ROOT_URLCONF="config.urls",
        DEBUG=True,
        DATABASES={
            "default": {
                "ENGINE": "django.db.backends.sqlite3",
                "NAME": BASE_DIR / "db.sqlite3",
            }
        },
    )
    

    For the sake of simplicity, here is a minimal models.py:

    from django.db import models
    
    class Post(models.Model):
        body = models.TextField()
    

    We've got our model set up, but how can we move our posts into our database? Well, to start with, we can set up our Django admin panel for this project. This requires several changes. First, I added only the required settings here by adding django.contrib.admin to INSTALLED_APPS. then I added each app/middleware entry to address an error message until python manage.py runserver successfully ran without errors.

    After those additions, here is settings.py:

    from pathlib import Path
    from django.conf import settings
    
    # Build paths inside the project like this: BASE_DIR / 'subdir'.
    BASE_DIR = Path(__file__).resolve().parent.parent
    
    settings.configure(
        ROOT_URLCONF="config.urls",
        DEBUG=True,
        DATABASES={
            "default": {
                "ENGINE": "django.db.backends.sqlite3",
                "NAME": BASE_DIR / "db.sqlite3",
            }
        },
        INSTALLED_APPS=[
            "django.contrib.admin",
            "django.contrib.auth",
            "django.contrib.contenttypes",
            "django.contrib.messages",
            "django.contrib.sessions",
        ],
        MIDDLEWARE=[
            "django.contrib.sessions.middleware.SessionMiddleware",
            "django.contrib.auth.middleware.AuthenticationMiddleware",
            "django.contrib.messages.middleware.MessageMiddleware",
        ],
        SECRET_KEY="django-insecure-4@r)20xhni#*+xcl*8u0t9b)85djrm+3(^0(7pzbb#7+l337*r",
        TEMPLATES=[
            {
                "BACKEND": "django.template.backends.django.DjangoTemplates",
                "DIRS": [],
                "APP_DIRS": True,
                "OPTIONS": {
                    "context_processors": [
                        "django.template.context_processors.debug",
                        "django.template.context_processors.request",
                        "django.contrib.auth.context_processors.auth",
                        "django.contrib.messages.context_processors.messages",
                    ],
                },
            },
        ],
    )
    

    Installed Apps and Middleware

    Before we get to the admin, I want to focus on INSTALLED_APPS and MIDDLEWARE. Similar to running commands with manage.py, these hide a lot of complexity. The upside is that this is where a lot of the 'batteries' are stored that people refer to when saying Django is a batteries included framework. To elaborate, let's start with django.contrib.admin in INSTALLED_APPS. While this is a string in our file, it's actually a reference to code in Django itself. Similarly, you can find the other INSTALLED_APPS by navigating the code there.

    Similarly, the strings in the MIDDLEWARE list all refer to classes in the Django codebase that you can navigate to by seeing each element in the string (separated by a period) as a folder in the Django codebase (until the last which is the PascalCase class name). Example: django.contrib.sessions.middleware.SessionMiddleware.

    Admin

    We are now ready for admin.py:

    from django.contrib import admin
    from .models import Post
    
    admin.site.register(Post)
    

    startapp

    We've hit a bit of a stopping point because we won't be able to see our new model in the admin until we've created and run the migration. In order to do that, we need an installed application for python manage.py makemigrations to pick up the new model. We could run python manage.py startapp to create these files, but we're already most of the way there, so let's create the folder ourselves (call it blog) and move models.py, admin.py, and views.py into it.

    We need to update the views import, and add the admin route in urls.py:

    from django.contrib import admin
    from django.urls import path
    
    from blog.views import hello_world, second_post, third_post, fourth_post
    
    urlpatterns = [
        path("hello-world", hello_world),
        path("second-post", second_post),
        path("third-post", third_post),
        path("fourth-post", fourth_post),
        path("admin/", admin.site.urls),
    ]
    

    Make and run migrations:

    python manage.py makemigrations
    python manage.py migrate
    

    We have a functioning admin!

    But, in order to log in, we'll have to create a super user:

    python manage.py createsuperuser
    

    Ok, now we can move our blog posts into the database. Go to your local admin and log in. Click 'Add' down by 'Posts', and add some text to the field labeled 'Body'. Click 'Save and Add Another' and repeat at least four times.

    After we've added all four posts, we can update views.py:

    from django.http import HttpResponse
    from blog.models import Post
    
    def hello_world(request):
        return HttpResponse(Post.objects.all().get(id=1).body)
    
    def second_post(request):
        return HttpResponse(Post.objects.all().get(id=2).body)
    
    def third_post(request):
        return HttpResponse(Post.objects.all().get(id=3).body)
    
    def fourth_post(request):
        return HttpResponse(Post.objects.all().get(id=4).body)
    

    Run python manage.py runserver and navigate to one of the post URLs (here's our first one), and you can see that we're now getting the text from our database. Hooray!

    We're almost done, but since we're good software developers, we should add a test for this project in tests.py:

    from django.test import TestCase
    from .models import Post
    
    class BlogTests(TestCase):
        @classmethod
        def setUpTestData(cls):
            cls.post = Post.objects.create(
                body="post text",
            )
    
        def test_post_model(self):
            self.assertEqual(self.post.body, "post text")
    

    Run the test.

    python manage.py test blog.tests
    

    Finally, for the sake of completion, and to get rid of that warning you've probably been seeing if you're coding along, we'll create blog/apps.py:

    from django.apps import AppConfig
    
    class BlogConfig(AppConfig):
        default_auto_field = "django.db.models.BigAutoField"
        name = "blog"
    

    And that's it! Our project now matches a project that has run python manage.py startapp blog.

    What have we learned?

    • When we run django-admin startproject config ., we are creating wsgi.py, settings.py, urls.py, and manage.py in the folder config. Now we know what these do and contain.
    • If we want to know more about what a manage.py command does, we know we can look into the Django source code.
    • We can also visit the source code to better understand what any section of INSTALLED_APPS and MIDDLEWARE do.
    • When we run python manage.py startapp appname, the files apps.py, models.py, admin.py, tests.py and views.py are created in the folder appname.
  2. Aliases, pip, and dotfiles

    Since my last post on aliases, I've added a few new aliases. Two are from Boost Your Django DX, which I'd highly recommend. The first is a straight cut from the book:

    alias pip="python -m pip"
    

    Using python -m pip instead of pip ensures you are using the version of pip for your local python. I've definitely had times where using it instead of just pip fixed issues for me.

    The second is:

    alias newenv="python3.11 -m venv --prompt . .venv && .venv/bin/pip install -U pip setuptools wheel"
    

    This isn't recommended as an alias in the book, but the pattern is recommended. I set this up as an alias because I've been playing around with a few small projects lately, which has caused me to create and recreate environments repeatedly. This makes it much simpler. The big win for me here is that I really don't like another recommended pattern I see in other places which is python3.11 -m venv venv. The problem with this is that your prompt always looks like:

    (venv)computername:~/home
    

    How do you know which environment is active? I rarely have issues related to this since I usually use separate terminals for separate projects, but it provides me peace of mind to know I'm in the correct environment. This brings me to my third new alias:

    alias act="source .venv/bin/activate"
    

    Again, I just got tired of typing all of that. And by using --prompt in the previous alias, I can have my prompt match the project while the virtual environment folder name is consistent across all projects. This allows me to run act in any folder where I have a virtual environment and activate it!

    In addition to adding these to my development desktop, I also created a dotfiles repo. Now, whenever I set up a new machine (or use my laptop for travel), I can have the same aliases everywhere. Here are two great examples of dotfiles repos: Ned Batchelder and Matt Layman.

    Finally, another tip regarding pip. A pip tip, if you will: Set up pip to require an active virtual environment. There are several ways to do this, mentioned in this blog post. If you want to save a click, execute pip config set global.require-virtualenv True. It will set the config file for you automatically. If you want to read more, click through.

    The common use case this prevents is accidentally installing a bunch of packages (usually with pip install -r requirements.txt) to your global pip. However, I just ran into another nice use case for this today. I had moved a virtual environment from a subfolder to the main folder of a project (I was experimenting with different ways to create the project so I had a few projects in a single repo). When I tried to use the environment again, pip told me there was no virtual environment active even though I had activated it. I'm not exactly sure what caused the issue, but I wouldn't be surprised if it would have caused more issues down the line. Instead, I just deleted the virtual environment, remade and activated it with my handy new aliases!

  3. Testing Stripe Integration in Django

    I have been using Stripe to process payments for my company AstroVenture (see more context in a previous post here) for a couple of years now, and it has worked great. One thing that took me quite a while to figure out was how to test that we were properly authenticated with the Stripe API without executing a payment.

    I've had two instances that caused issues with purchasing:

    1. One release, I broke the server's local environment so things seemed to be working, but our API keys were not read in correctly. Most things seemed to work, but purchases did not.
    2. Another time I made changes to help secure our servers, but one of the rules prevented most outbound traffic including to Stripe's API!

    I wanted a way to test for scenarios like this, but I couldn't find any good solutions until recently. I was searching the API docs for Stripe when I realized I could use any stripe API method that requires authentication to ensure that we are able to access Stripe. If we don't supply the keys or something else goes wrong, the test (below) will fail with an error. I only run this as part of my end to end tests which you can read more about here.

    I added an endpoint to my Django application that looks like the following (I removed the authentication I do at this endpoint since it isn't the point of the example, and of course, don't put your API key directly in your code, read it from a secret store or somewhere else secure):

    class StripeTest(APIView):
        """
        This endpoint exists so that we can run an end to end test
        for the environment to ensure that we are able to access 
        the Stripe API after we have deployed. 
        """
        def get(self, request):
                stripe.api_key = key
                try:
                    data = stripe.Customer.list(limit=3)["data"]
                    return Response()
                except Exception as e:
                    return Response(str(e), status=status.HTTP_400_BAD_REQUEST)
    

    And in my end to end tests I have the following:

    def test_validate_stripe_keys():
        """
        By calling the Stripe API and not receiving an error,
        we validate that the keys are properly loaded and that
        we can access the API. We only care if we get an error,
        so we ignore the return value.
        """
        response = requests.get(
            f"{url}stripe-test-api-is-working/"
        )
        assert response.status_code == status.HTTP_200_OK
    

    Do you have a better way of testing that you can access Stripe after a deployment? Let me know!

  4. Lessons Learned Teaching Undergraduate Astronomy with a Video Game - Testing

    This is the fourth and final installment of the series breaking down my talk from DjangoConUS 2022. The first entry covered background information about the project, the second was about using Django Rest Framework, and the third was about infrastructure and deployment.

    Before diving in, I'd like to emphasize that my testing philosophy isn't the end state for a project. It's a guiding principal for getting started and staying motivated with testing. If you have an established application with a large team, you likely already have rules and processes to follow for testing. What I want to discuss here is how I think about testing as I'm building a project. This isn't limited to small/side projects, but it's much more important earlier in a project's lifecycle.

    So what is my philosophy? Test enough to give you confidence to change and deploy your code.

    For me, this does not include test driven development (TDD) or a specific coverage number. Why not test driven development? I've tried it, and it doesn't match my preferred way to work in most projects which is to develop my code then write tests. If you find TDD works better for you, definitely do that! I do some development by writing tests first, but I generally write at least some code before I write my tests. And while I do strive for a high coverage percentage, I use that metric more as a guide to see where I might need more tests rather than a bar to meet. That is, I'm more interested in coverage of a given file than I am an overall number.

    Ok so why this philosophy? For me, I find that without tests (either at all, or even with limited tests for a specific section of the code), I'm much less confident about changing code. This means that I take a lot longer to write or change code in untested sections of the codebase. Usually, I have to take extra time to think through edge cases since I can't be confident those were covered previously and, I don't know what they are because they aren't written into any tests.

    There are several terms specific to testing that I'd like to explicitly define since some folks use different definitions and terminology for different classes of tests. I want to make sure I'm clear about what I mean when I'm using these terms.

    First, unit tests test a 'unit' of code, generally meaning as little code as possible (oftentimes a single function) so that you can be sure you are testing each section without depending on other sections, which can introduce complexity into tests.

    Integration tests bring together (integrate) two or more 'units' to test their functionality when combined.

    End to end tests cover the full functionality of some part of the application. For example, testing a student sending gameplay data to an endpoint and receiving a response, or someone purchasing our game.

    Our test suite has a large number of unit tests, almost no integration tests, and a few end to end tests. I'll explain my reasoning for each of these.

    First, unit tests. I really like having a good suite of unit tests for two reasons:

    1. Almost every time I write some code, then write unit tests, I find that I refactor the code as I'm writing the tests. Sometimes this is just a little bit of cleanup. Other times, it's a bigger rewrite. But almost every time, writing unit tests gets me to think about my code a little bit differently, and I uncover something that I want to improve.
    2. When I'm making a change to some code I haven't touched in a while, I know that my unit tests will tell me if I broke something. This gives me more confidence to dive in than if I didn't have them.

    Unit tests take some time and effort to write and maintain, but I'll take that overhead on any project for the confidence that they give. One other great use of unit tests is covering a bugfix. Sometimes, I'll find a bug, write a fix, then write a test (or two) that covers the case for that bug so we can avoid it in the future.

    Why no integration tests?

    I think the functionality is covered better by end to end tests for this project. For other projects I've worked on, I've had a much larger suite of integration tests and fewer end to end tests. This is highly dependent on what your application does. Generally, for larger projects, you'll want more integration tests to ensure parts of your code work together without having to run a larger suite of end to end tests for every change.

    End to end tests can take a long time to write, a long time to run (relative to unit tests), and are more difficult to maintain. That's why I recommend these only cover the most important parts of your code. Even though these were the most difficult to write and maintain, these give me the most confidence when deploying my code. I know that something can still be wrong if these pass, but I at least know the most important parts of the site are mostly working. I wish I would have written these earlier in the project since my first few deployments were much more stressful without them and required some manual testing.

    For our end to end tests, I use Selenium. I've heard a lot of good things about Playwright, and I'm hoping to have some time to look into it, but I haven't investigated it enough yet to recommend it myself. I also use pytest instead of the built in testing system in Django. I don't have a strong opinion about pytest versus the test runner in Django, but I've used pytest a good bit professionally (with and without Django), so I find it easier to get started with. I also like the features it has for running tests (like flags for running just the last failed test, --lf, stopping when hitting a failure --x, etc.) as well as fixtures and parametrized inputs. I'd recommend you give pytest a try if you haven't already. You don't need to worry about the more advanced features when starting with it and you can build up your knowledge as you go.

    Speaking of fixtures, I tend to create a factory for each data type that I need (when I need it for a test) with Factory Boy and make that a fixture if I need to use it in multiple tests. I'll move those fixtures out to a conftest file (this is a file particular to pytest) if I find a need for them across multiple test files. That way, I'm not populating conftest with a large number of fixtures that aren't used or are only used in one or two places, making it easier to read. If you'd like me to write more about how I use fixtures, let me know!

    Some key advice: write tests early!

    This doesn't mean you need 100% coverage on day 1. Or even 10%. But having a test, any test, makes it much easier to write the next one. So start with something as simple as making sure your pages load or if your application is an API, that your endpoints return the correct basic response to a good request. Then, as you are building out your application, keep asking yourself, what sections of the code worry me the most to change, or what do I worry about when we deploy? And write tests in those areas to alleviate your stress. Also, try to get some tests covering your most important code in place as early as you can. In the long run, it'll speed up your development and increase your confidence in deploying your code.

  5. Update for Notarizing Mac Unity Apps

    I recently received an email from Apple stating that they are migrating users from altool to notarytool to notarize applications. Apple says altool will no longer work for notarization starting November 1 2023, but that applications notarized before that will still behave as normal. I previously wrote about notarizing a Unity app on MacOS, so I wanted to add to that post since the linked scripts will be out of date by the end of the year.

    What I'm about to write about is covered by Apple here, but I ended up submitting (and waiting and submitting and waiting) several times before figuring out the proper incantations, so I'm gathering the information here.

    First, in the script for notarizing your application, you'll need to change --username and --asc-provider to --apple-id and --team-id respectively. Then you'll need to remove --primary-bundle-id (and the value associated with that flag) and --file. For that last one, remove ONLY --file and leave the location of the file you want to notarize. Finally, swap altool --notarize-app for notarytool submit and add --wait (the wait flag is optional, but I'll explain it below).

    So you should go from this in the old script: xcrun altool --notarize-app --username "$USERNAME" --password "$PASSWORD" --asc-provider "$TEAM_ID" --primary-bundle-id "$BUNDLE_ID" --file "$SIGNING_FOLDER/$APP_NAME.zip"

    to xcrun notarytool submit --wait --apple-id "$USERNAME" --password "$PASSWORD" --team-id "$TEAM_ID" "$SIGNING_FOLDER/$APP_NAME.zip"

    The new tool has a nice progress text in the terminal for the upload status that looks like XX% (YY MB of ZZ MB). The wait flag is nice because it means you can skip the second script (from the link in the previous blog post) for checking the status of the file after you've uploaded it. If you use the wait flag, after the file finishes uploading, you'll see 'In progress ........', while it waits to be notarized. Much nicer than a hanging terminal session with no info!

  6. Lessons Learned Teaching Undergraduate Astronomy with a Video Game - Infrastructure and Deployment

    This is the third installment of the series breaking down my talk from DjangoConUS 2022. The first entry covered background information about the project and the second was about using Django Rest Framework.

    First, some important context: if you are a devops engineer, or have a lot of experience with AWS/GCP/Azure, this post may not be for you. This post is aimed at folks who would prefer to write Django code than deal with the intracacies of deployment.

    With that being said, this is the section of the talk that made me want to write this series of posts. Specifically, I realized that I focused a lot on the infrastructure setup of this project, which I want to outline here. However, I wish I would have spent more time focusing on what I think the goal of any successful deployment strategy should be for a Django project regardless of the infrastructure:

    Repeatability and confidence in deployments.

    There are a lot of ways to get to this point. And on day 1 (or even 100 if it's a side project), you likely won't be there. But starting with good process documentation, and moving that toward making your deployments consistent and repeatable is a massive boost in confidence that increases the time you can spend working on your application rather than its infrastructure. It can also significantly lower your stress when it comes to deploying. Importantly, this is all independent of what services you use to deploy your application!

    At AstroVenture, I chose Amazon Web Services (AWS) for our infrastructure early on in founding the company because I had experience with it, and we received free credits for a limited time. At that time, I manually created an EC2 instance for the server (and RDS for the database) and manually installed all of the packages I needed to run the server. I manually installed the app and 'deployments' were done by pulling from GitHub and restarting the gunicorn workers. The 'backup strategy', was the in depth document I wrote with step by step instructions about how I did all of that. I tested it by recreating the server a second time for our production environment and using the first as a test/staging server.

    In the event of a catastrophic server issue, I was likely to be down for several hours, if not an entire day. But having that document gave me the confidence that it wouldn't be more than that, and that I wouldn't have to stress that I would miss a step during that recovery. So if you aren't sure how you would bring your servers back in the event of everything going down, I'd highly recommend going through this exercise for whatever service you are using. It can at least alleviate some of the stress of a deployment going wrong.

    From there, I hired a friend who had more infrastructure experience to write the Packer and Terraform scripts we use now, and to help me make a few architecture decisions that allow us to have ~zero downtime deployments and scaling. I was already using a load balancer, but we added an autoscaling group so that we can spin up new instances if we need.

    The Packer scripts create the server image with the application and all of its dependencies, so if we ever have an issue where we need to redeploy an old stable version, we can do that directly from the image instead of having to recreate it. Luckily, we haven't had to do that yet. We use the Terraform scripts to provision an entirely new server and wait until it is live before swapping it with the previous server (and then terminating that one). There are other tools that handle automating infrastructure and application building that others might prefer, but these have worked well (in combination with some good old fashioned bash scripts) for us.

    We also have end to end tests (more on this in a post coming soon), which I run after every deployment to make sure that the most important parts of the site are functioning correctly.

    What if you don't have a friend with devops experience that can help you, and you don't have that experience yourself?

    There are a number of options of Platform as a Service (PaaS) offerings from companies like Render and Fly.io that a lot of folks in the Django community are using. I'm hoping to try these in the near future along with Django Simple Deploy. So while I can't give specific recommendations for these platforms, I can tell you that the goal: Repeatability and confidence in deployments is even easier to achieve on these platforms than it is on AWS (or GCP or Azure). They handle a lot of the work that our Packer and Terraform scripts do so that you can focus on your application code. The tradeoff with these services is that they can be a little bit (to a lot if you scale very large) more expensive than equivalent 'bare metal' servers from AWS, GCP, or Azure. But they can also be cheaper starting out, and the added price can be worth it while you get started.

    No matter what tools you use for hosting and deploying your code, if you are reluctant to deploy because of something in your process you aren't confident about, I strongly recommend you look into ways to address that issue. I found that it was a big relief to stop worrying about deploying once I was able to address the more manual parts of our process.

    Finally, remember, you don't have to do this all at once, and you don't have to be at the point of continuous integration of your code to feel confident with your deployments. Take small steps and work toward the goal of feeling confident deploying. It'll make coding a lot more fun!

  7. Lessons Learned Teaching Undergraduate Astronomy with a Video Game - Django vs Django Rest Framework (DRF)

    If you've ended up here from somewhere outside of this blog, and are looking for an exhaustive comparison of these two libraries, I regret to inform you, this isn't that. If you're here for the next installment of the series breaking down my talk from DjangoConUS 2022, welcome back!

    This section of the talk outlines my project's usage of Django Rest Framework (DRF). For some added context, I wanted to start this project using DRF, but ran into some difficulties because the data coming from the game was largely string based and so it wasn't able to be neatly serialized into the types we had in the database. As a result, I ended up using DRF, but I wasn't able to use some of the built in features like viewsets and generics.

    If you are new to Django, or have never used DRF, one major reason to consider DRF is if you are interested in returning JSON from your application rather than integrating with the Django frontend. Vanilla Django does a great job with integrating the frontend (templates) and backend code. But if you need to send data to a frontend that's not written in Django, or you are developing an API, DRF has a lot of tools to help.

    With that out of the way, there is one bit of this section of the talk that I'd like to amend. In that part, I say that if your data model requires a lot of changes to get from hitting your endpoints to your database (like ours did), that you should consider falling back to vanilla Django (versus powering forward with Django Rest Framework).

    What I should have said, is that you can still use DRF, but if you feel like you are fighting with the viewsets or mixins or serializers, you can fall back to the basic APIView, which allows much more prescriptive code rather than the classes built into DRF, which are powerful, but less flexible. If you are more comfortable with vanilla Django, feel free to head back that way, but DRF is capable of handling scenarios where your incoming data don't match your models.

    If you are familiar with DRF, you may be thinking 'isn't that the point of the serializer?' and you would be correct. But, handling too much complexity in the serializer can lead to worse performance. Before I was able to rewrite my incoming data, I tried (with some help from a developer more experienced with DRF) to rewrite our endpoints to use DRF and ended up with more queries than I had before! I needed to update the incoming data before I could improve performance. Otherwise, I needed too many queries to gather everything and return it the way it needed to be to match what our game was expecting. I'm hoping to write up more about how I determined the number of queries, and how I keep my codebase from increasing that number. If that's interesting to you, let me know!

    This gets us to the core message I want to convey: Don't feel like you have to choose only one way of doing things in a Django project. Or that you can't evolve your codebase in the future.

    Yes, consistency can help readability in a project. But just because some of your endpoints use viewsets doesn't mean that every endpoint needs to use viewsets. Use the right tool for the job. It's ok to mix DRF generic views with viewsets, or DRF viewsets with Django class based views if that better matches what you are trying to accomplish for a given route. And you can always evolve your code to better match the libraries you are using as you learn more about them.

    One piece of advice from this section of the talk I still find myself coming back to is to really focus on understanding serializers and getting to know the inner workings of DRF (by this I mean what it's doing generally behind the scenes to process requests, not necessarily understanding every line of code in the library). After getting over the initial hump of learning about DRF, it can be easy to write some quick views that work. But it can be harder to modify them if you don't understand some of the basics of what DRF is doing under the hood.

    Working with DRF, when I send in data and get an unexpected error, the issue is very often with (what I implemented in) the serializer. Whether that's fields that are missing* (or are not in the model or are misspelled) or I'm trying to do something that isn't possible with a specific serializer class (usually with a serializer that is more specific than what I need). Occasionally, I'll run into an error with a misconfigured URL, or a view that's using the wrong viewset or generic view, but most often, it has to do with the serializer.

    *In fact, just today, I hit an error in the admin locally because I forgot to update my serializer after updating a model field's name.

    So how do we get to know DRF (and serializers) better? If you don't have an app, build one! Either use the DRF Tutorial or some other tutorial to build an app with DRF. There are many good ones out there (Real Python has quite a few). If you have an app already, try using DRF to rebuild endpoints you already have and see how DRF handles them. Or consider adding a few new endpoints. Think about extra data a user may want, or combining data from multiple tables. Anything different from what the endpoints are currently doing will improve your understanding. Even if these changes aren't optimal, you'll see how changes in the data affect the viewsets/generic views and serializers, which will help your understanding a whole lot more than reading another blog post...

    I recently rewrote the majority of the endpoints in my application in DRF, and I love how little code there is in my views.py file now! It's also great that I have fewer queries and less processing to do. A lot of that has to do with the aforementioned update to the incoming data, but it's still great to see.

    If you use Django and haven't used DRF (or haven't used it in a while), I highly recommend trying it. There is no need to commit to rewriting your whole application. It can be helpful to simplify endpoints that best match DRF's built in generics or viewsets. And remember that with APIView you can learn as you go and make things a bit more explicit before diving in entirely with generics and viewsets.

  8. AstroVenture and University of Mars

    In this post, I'd like to provide the context for the project I'm currently writing about on this blog. I'm also using this as an opportunity to start a series of blog posts breaking down my recent talk from DjangoConUS 2022 about what I learned related to Django while building this project.

    I've realized since giving the talk there are a few things I'd like to change in/add to it. So instead of a blog post summarizing the talk, I'm opting to revisit the talk and discuss those changes/additions (as well as highlighting the advice I feel holds up). The beginning of the talk covers some of the content of this post. The other sections of the talk will be covered in dedicated posts in the coming weeks.

    In 2018, I founded AstroVenture with my former colleagues from Penn State University. When I worked there from 2010-2014, we built a videogame that teaches introductory astronomy to undergraduate students that we later named 'University of Mars'. We founded the company in order to sell that game to anyone who wants to learn astronomy. If you'd like to check it out, the first quarter of the game is free as a demo at the link above. If you'd like to play more than that, contact me! There is also a video at the site if you just want to see some gameplay to get a better idea of how it works.

    In the nearly 5 years since we founded the company, this has been a side project that I've worked on actively. It is the project that I really dove into and learned Django with. I'd previously had some professional experience with Django, but I had a lot of other responsibilities on that project, so I didn't have time to dig into Django itself as much as I would have liked. The game was built with Unity3D, which I also learned as I built this project.

    The game uses a story to engage students and set up discussions for each topic. The gameplay loop for each lesson is for the game characters to start discussing the topic of the lesson (ranging from gravity to dark energy), then the student explores some type of interactive minigame on the topic, followed by a quiz. Then we usually lead into the next section's discussion and repeat.

    The servers that I manage for the company handle the website, which has the functionality you would expect from a Django website: login, download the game, view content related to the game (mostly in the form of our static 'encyclopedia' pages that accompany the game). Additionally, those servers also track student progress as they play through the game in order for instructors to give credit to students for finishing the different sections of the game. Currently, instructors can download a CSV file with player progress and quiz scores, but I'm working on integrating with Canvas and Blackboard to make grade syncing more direct.

    The backend code for this player tracking was the first API that I ever wrote and was originally written in PHP hosted in a VPS. When I founded the company, I rewrote the API in Django and moved the hosting to AWS. Since my time was limited, I mostly ported the previous API to Django, faults and all. Fortunately, I had some time to dedicate to this recently and I've just finished updating it to be more efficient and use more of Django Rest Framework's built in features. As for the infrastructure, I chose AWS because I had some experience with it, and I had access to free credits there for a limited time. I'll cover more about the infrastructure in a future post.

    We use Sentry for error handling and Stripe for payment processing, and I highly recommend both.

    The front end of the website is in a separate repo written in Typescript[1] with a bit of jQuery sprinkled in. It also uses Bootstrap for CSS. I chose to not use Django for the front end because I thought I might have someone else working on it who wouldn't have familiarity with Django. If I were to start again today, I might do it differently and have everything in one repo. I'm not exactly sure what I'd settle on, though I'd investigate HTMX, Alpine.js, and Tailwind for CSS as I've heard good things about those.

    If any of these technology choices are interesting to you, I'm happy to discuss them more. See the contact links on the about page, or at the bottom of this page.

    In the upcoming posts, I'll talk more about the other sections of the talk: infrastructure, testing, and Django vs DRF.

    1 - This is almost worth a blog post in itself. When I chose to use Typescript, it was because I was frustrated with JavaScript's lack of typing leading to errors (usually with JSON objects) and not being able to easily debug them in larger codebases. I think Typescript is good, but it was also difficult to find good resources (responses on StackOverflow etc.) dealing with Typescript outside of React or Angular, which made getting started a lot more difficult for me.

  9. [TIL] Django Admin Permissions, etc.

    So far in my main project (more on that to come), the only users of the admin have been myself and my cofounder, so I've made us both superusers. Recently, I had a request to allow an instructor to view and update data from their own courses. This would require me to give them admin access, but in a highly limited view. The good news is that there is a lot of information on how to limit admin access in Django! The bad news is that a lot of it applies to entire models or other use cases that were much less specific than mine. Three major things I learned about:

    First, I'm not sure if this is the best or only way to do this, but I created a group (I could not get this to work as permissions for only one user) and limited the permissions to just a few models (and only the required permissions). I'd be interested if there is a way to do this per individual, but I think I might need the group settings sooner rather later anyway.

    Second, for those models they are able to use, I needed to limit what they could see to only their students. I was able to do that by adding to the relevant models in admin.py:

        def get_queryset(self, request):
            qs = super().get_queryset(request)
    
            if request.user.is_superuser:
                return qs
            else:
                return qs.filter(institution=their_institution)
    

    This returns all data for superusers, but only returns whatever you filter on to other users. In my case, I only have two levels of admin user, so this simple filter works well. You'll have to fill in the filter depending on your model.

    Third, and this was most difficult to track down, I needed to make sure they couldn't see data outside of their scope when adding new students to their course. For the users model, I was able to add a filter to the user admin like above. But for the Course model, I would still see all courses even when using get_queryset for the CourseAdmin as above. I'm not sure why this is. To fix this, I had to use:

        def formfield_for_foreignkey(self, db_field, request, **kwargs):
            if not request.user.is_superuser and db_field.name == "course":
                kwargs["queryset"] = Course.objects.filter(institution=institution)
            return super().formfield_for_foreignkey(db_field, request, **kwargs)
    

    This overwrites the data returned for a foreign key in a dropdown on the 'Add' page, and was exactly what I needed!

Page 1 / 3 »