Fixing an occasionally failing test

Since we started the Django project I'm currently working on, about 20 months ago, I've been evangelising the joy of thoroughly writing tests for the code.

For quite some time, we've had a single test that would fail once in a while. But only rarely, so nobody bothered trying to fix it – we would just run the tests again and move on with our lives. But yesterday I ended up spending half the day trying to figure out the root cause of this. Below is the pull request description I wrote for my fix to this:

Replace "smith" job title with "cheesemonger" in tests

It turns out that Factory Boy, which we use to populate test data with sort of meaningful values, sometimes leads us to create companies with the name "Smith Inc.".

In the test called test_match_job_ad_no_matches() we make sure that no matches occur for the economist job.

But once in a while, rarely, the economist job would end up having a company with the name "Smith Inc." and this would result in a match for the agent looking for jobs with the job title "smith".

By changing the job title from "smith" to "cheesemonger" everywhere in these tests, I hope we will no longer run into this problem. But who knows: Maybe Factory Boy has more surprises for us :)

Update, a few days later

A friend pointed out that it's possible to control the seed of Factory Boy. And thus make your tests print out the seed used, in order to be able to reproduce failed tests with a certain seed. Neat!