Test dark visitors update and improve code/test_files/robots.json

[glyn]

          I wonder if it's worth reworking this file so that it has less duplication of fields and so that it exercises the consolidation logic thoroughly if testing of dark visitors update is added?

Originally posted by @glyn in #51 (comment)

The tests should be expanded to cover the logic for the dark visitors update, suitably refactored so that the dark visitors input is parameterised. The robots.json test fixture should also be reworked to:

1. ensure good coverage of the dark visitors logic.
2. reduce duplication.

(This issue is a follow-on to PR 51, should that be merged.)