This is the fourth and final installment of the series breaking down my talk from DjangoConUS 2022. The first entry covered background information about the project, the second was about using Django Rest Framework, and the third was about infrastructure and deployment.
Before diving in, I'd like to emphasize that my testing philosophy isn't the end state for a project. It's a guiding principal for getting started and staying motivated with testing. If you have an established application with a large team, you likely already have rules and processes to follow for testing. What I want to discuss here is how I think about testing as I'm building a project. This isn't limited to small/side projects, but it's much more important earlier in a project's lifecycle.
So what is my philosophy? Test enough to give you confidence to change and deploy your code.
For me, this does not include test driven development (TDD) or a specific coverage number. Why not test driven development? I've tried it, and it doesn't match my preferred way to work in most projects which is to develop my code then write tests. If you find TDD works better for you, definitely do that! I do some development by writing tests first, but I generally write at least some code before I write my tests. And while I do strive for a high coverage percentage, I use that metric more as a guide to see where I might need more tests rather than a bar to meet. That is, I'm more interested in coverage of a given file than I am an overall number.
Ok so why this philosophy? For me, I find that without tests (either at all, or even with limited tests for a specific section of the code), I'm much less confident about changing code. This means that I take a lot longer to write or change code in untested sections of the codebase. Usually, I have to take extra time to think through edge cases since I can't be confident those were covered previously and, I don't know what they are because they aren't written into any tests.
There are several terms specific to testing that I'd like to explicitly define since some folks use different definitions and terminology for different classes of tests. I want to make sure I'm clear about what I mean when I'm using these terms.
First, unit tests test a 'unit' of code, generally meaning as little code as possible (oftentimes a single function) so that you can be sure you are testing each section without depending on other sections, which can introduce complexity into tests.
Integration tests bring together (integrate) two or more 'units' to test their functionality when combined.
End to end tests cover the full functionality of some part of the application. For example, testing a student sending gameplay data to an endpoint and receiving a response, or someone purchasing our game.
Our test suite has a large number of unit tests, almost no integration tests, and a few end to end tests. I'll explain my reasoning for each of these.
First, unit tests. I really like having a good suite of unit tests for two reasons:
- Almost every time I write some code, then write unit tests, I find that I refactor the code as I'm writing the tests. Sometimes this is just a little bit of cleanup. Other times, it's a bigger rewrite. But almost every time, writing unit tests gets me to think about my code a little bit differently, and I uncover something that I want to improve.
- When I'm making a change to some code I haven't touched in a while, I know that my unit tests will tell me if I broke something. This gives me more confidence to dive in than if I didn't have them.
Unit tests take some time and effort to write and maintain, but I'll take that overhead on any project for the confidence that they give. One other great use of unit tests is covering a bugfix. Sometimes, I'll find a bug, write a fix, then write a test (or two) that covers the case for that bug so we can avoid it in the future.
Why no integration tests?
I think the functionality is covered better by end to end tests for this project. For other projects I've worked on, I've had a much larger suite of integration tests and fewer end to end tests. This is highly dependent on what your application does. Generally, for larger projects, you'll want more integration tests to ensure parts of your code work together without having to run a larger suite of end to end tests for every change.
End to end tests can take a long time to write, a long time to run (relative to unit tests), and are more difficult to maintain. That's why I recommend these only cover the most important parts of your code. Even though these were the most difficult to write and maintain, these give me the most confidence when deploying my code. I know that something can still be wrong if these pass, but I at least know the most important parts of the site are mostly working. I wish I would have written these earlier in the project since my first few deployments were much more stressful without them and required some manual testing.
For our end to end tests, I use Selenium. I've heard a lot of good things about Playwright, and I'm hoping to have some time to look into it, but I haven't investigated it enough yet to recommend it myself. I also use pytest instead of the built in testing system in Django. I don't have a strong opinion about pytest versus the test runner in Django, but I've used pytest a good bit professionally (with and without Django), so I find it easier to get started with. I also like the features it has for running tests (like flags for running just the last failed test, --lf, stopping when hitting a failure --x, etc.) as well as fixtures and parametrized inputs. I'd recommend you give pytest a try if you haven't already. You don't need to worry about the more advanced features when starting with it and you can build up your knowledge as you go.
Speaking of fixtures, I tend to create a factory for each data type that I need (when I need it for a test) with Factory Boy and make that a fixture if I need to use it in multiple tests. I'll move those fixtures out to a conftest file (this is a file particular to pytest) if I find a need for them across multiple test files. That way, I'm not populating conftest with a large number of fixtures that aren't used or are only used in one or two places, making it easier to read. If you'd like me to write more about how I use fixtures, let me know!
Some key advice: write tests early!
This doesn't mean you need 100% coverage on day 1. Or even 10%. But having a test, any test, makes it much easier to write the next one. So start with something as simple as making sure your pages load or if your application is an API, that your endpoints return the correct basic response to a good request. Then, as you are building out your application, keep asking yourself, what sections of the code worry me the most to change, or what do I worry about when we deploy? And write tests in those areas to alleviate your stress. Also, try to get some tests covering your most important code in place as early as you can. In the long run, it'll speed up your development and increase your confidence in deploying your code.