-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Restructure 'seeding' to be Knex based (#1230)
https://eaflood.atlassian.net/browse/WATER-4575 We have been working on replicating how the [water-abstraction-import](https://github.com/DEFRA/water-abstraction-import) service imports a licence. See [Import Licence versions](#1195) for example. This is in readiness for moving to ReSP managing licences rather than NALD. What it exposed was that our test data setup needed expanding. The import relies on 'real' reference data being present. So, it makes testing easier if this is the case in our test database when writing unit tests. We started with [create wrapper for licence-version-purpose-condition table](#1181) which added the ability to seed licence-purpose condition types. This was then expanded by [Add new test reference data seeders](#1206) which added some more reference data, for example, purposes. The issue we've had is that the existing changes are straddling two different mechanisms. We've included them as a seed file in our [Knex seeding capabilities](https://knexjs.org/guide/migrations.html#seed-files). This is reference data, so why not seed it?! And we are seeding it whenever we perform a database clean. Our intent is to have real reference data when unit testing, and until we've removed calls to `DatabaseSupport.clean()` from all our tests, we need to do this each time. That has led to the logic and data itself living in the `test/` folder. There is a conceptual problem with this: something we depend on 'for real' lives in our `test/` folder. Plus, the functionality is split into 2 places. It also means our `test/support/seeders` have a mix of types. Original 'unit test seeders' that were intended to help move bulky test setup away from within the test itself, with 'knex seeders', things we'll also call when we run `npm run seed` against the non-test DB. So, on reflection, we felt the need to carry out a little restructuring. _All_ the 'seeding' functionality is moved to `db/seeds` but with the ability to still call it from `clean()` until such time as we have updated all tests and can drop the command. While we're doing this housekeeping, we also update our existing user seed to match the conventions that have been introduced while also re-implementing some existing ones, for example, preferring **Objection.js** to raw SQL queries. The end result is that we can seed both the real and test databases with reference data, even when a database already has existing records, and this functionality will live in one place and be built on **Knex**. But there's more! Having made these changes to ensure both the real and test databases are correctly populated with reference data, it no longer made any sense to be adding new reference records within our tests. So, we updated the relevant test helpers to provide a `select()` in place of the `add()` method. This means you can select the details of a reference record without having to first create or query for it. We then went through the affected tests and 'test-seeders' and swapped calls to `Helper.add()` with `Helper.select()`. We also performed some housekeeping on some of the tests we touched.
- Loading branch information
1 parent
8a21e90
commit e8619c5
Showing
112 changed files
with
2,359 additions
and
2,289 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| 'use strict' | ||
|
|
||
| const { db } = require('../db.js') | ||
|
|
||
| const { timestampForPostgres } = require('../../app/lib/general.lib.js') | ||
| const { data: regions } = require('./data/regions.js') | ||
| const RegionModel = require('../../app/models/region.model.js') | ||
|
|
||
| const ServerConfig = require('../../config/server.config.js') | ||
|
|
||
| async function seed () { | ||
| for (const region of regions) { | ||
| const exists = await _exists(region) | ||
|
|
||
| if (exists) { | ||
| await _update(region) | ||
| } else { | ||
| await _insert(region) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| async function _applyTestFlag (region, id) { | ||
| if (region.name !== 'Test') { | ||
| return null | ||
| } | ||
|
|
||
| return db('regions').withSchema('water').update('isTest', true).where('regionId', id) | ||
| } | ||
|
|
||
| async function _exists (region) { | ||
| const { chargeRegionId, naldRegionId } = region | ||
|
|
||
| const result = await RegionModel.query() | ||
| .select('id') | ||
| .where('chargeRegionId', chargeRegionId) | ||
| .andWhere('naldRegionId', naldRegionId) | ||
| .limit(1) | ||
| .first() | ||
|
|
||
| return !!result | ||
| } | ||
|
|
||
| async function _insert (region) { | ||
| // The Test region is only intended to be seeded in our non-production environments | ||
| if (region.name === 'Test' && ServerConfig.environment === 'production') { | ||
| return | ||
| } | ||
|
|
||
| const result = await RegionModel.query().insert(region) | ||
|
|
||
| return _applyTestFlag(region, result.id) | ||
| } | ||
|
|
||
| async function _update (region) { | ||
| const { chargeRegionId, displayName, naldRegionId, name } = region | ||
|
|
||
| return RegionModel.query() | ||
| .patch({ displayName, name, updatedAt: timestampForPostgres() }) | ||
| .where('chargeRegionId', chargeRegionId) | ||
| .andWhere('naldRegionId', naldRegionId) | ||
| } | ||
|
|
||
| module.exports = { | ||
| seed | ||
| } |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| 'use strict' | ||
|
|
||
| const { timestampForPostgres } = require('../../app/lib/general.lib.js') | ||
| const { data: purposes } = require('./data/purposes.js') | ||
| const PurposeModel = require('../../app/models/purpose.model.js') | ||
|
|
||
| async function seed () { | ||
| for (const purpose of purposes) { | ||
| await _upsert(purpose) | ||
| } | ||
| } | ||
|
|
||
| async function _upsert (purpose) { | ||
| return PurposeModel.query() | ||
| .insert({ ...purpose, updatedAt: timestampForPostgres() }) | ||
| .onConflict('legacyId') | ||
| .merge(['description', 'lossFactor', 'twoPartTariff', 'updatedAt']) | ||
| } | ||
|
|
||
| module.exports = { | ||
| seed | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| 'use strict' | ||
|
|
||
| const { timestampForPostgres } = require('../../app/lib/general.lib.js') | ||
| const { data: primaryPurposes } = require('./data/primary-purposes.js') | ||
| const PrimaryPurposeModel = require('../../app/models/primary-purpose.model.js') | ||
|
|
||
| async function seed () { | ||
| for (const purpose of primaryPurposes) { | ||
| await _upsert(purpose) | ||
| } | ||
| } | ||
|
|
||
| async function _upsert (primaryPurpose) { | ||
| return PrimaryPurposeModel.query() | ||
| .insert({ ...primaryPurpose, updatedAt: timestampForPostgres() }) | ||
| .onConflict('legacyId') | ||
| .merge(['description', 'updatedAt']) | ||
| } | ||
|
|
||
| module.exports = { | ||
| seed | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| 'use strict' | ||
|
|
||
| const { timestampForPostgres } = require('../../app/lib/general.lib.js') | ||
| const { data: secondaryPurposes } = require('./data/secondary-purposes.js') | ||
| const SecondaryPurposeModel = require('../../app/models/secondary-purpose.model.js') | ||
|
|
||
| async function seed () { | ||
| for (const purpose of secondaryPurposes) { | ||
| await _upsert(purpose) | ||
| } | ||
| } | ||
|
|
||
| async function _upsert (secondaryPurpose) { | ||
| return SecondaryPurposeModel.query() | ||
| .insert({ ...secondaryPurpose, updatedAt: timestampForPostgres() }) | ||
| .onConflict('legacyId') | ||
| .merge(['description', 'updatedAt']) | ||
| } | ||
|
|
||
| module.exports = { | ||
| seed | ||
| } |
22 changes: 22 additions & 0 deletions
22
db/seeds/05-licence-version-purpose-condition-types.seed.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| 'use strict' | ||
|
|
||
| const { timestampForPostgres } = require('../../app/lib/general.lib.js') | ||
| const { data: licenceVersionPurposeConditionTypes } = require('./data/licence-version-purpose-condition-types.js') | ||
| const LicenceVersionPurposeConditionTypeModel = require('../../app/models/licence-version-purpose-condition-type.model.js') | ||
|
|
||
| async function seed () { | ||
| for (const licenceVersionPurposeConditionType of licenceVersionPurposeConditionTypes) { | ||
| await _upsert(licenceVersionPurposeConditionType) | ||
| } | ||
| } | ||
|
|
||
| async function _upsert (licenceVersionPurposeConditionType) { | ||
| return LicenceVersionPurposeConditionTypeModel.query() | ||
| .insert({ ...licenceVersionPurposeConditionType, updatedAt: timestampForPostgres() }) | ||
| .onConflict(['code', 'subcode']) | ||
| .merge(['description', 'displayTitle', 'subcodeDescription', 'updatedAt']) | ||
| } | ||
|
|
||
| module.exports = { | ||
| seed | ||
| } |
Oops, something went wrong.