Add Licence Version Purpose Point model #1288
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them. Users select these points as part of the return requirements setup journey we've built. We extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. We don't know why the previous team never opted to extract licence points to their own table. But we are now [importing them as part of the licence import](DEFRA/water-abstraction-import#1009). This change adds the model and supporting changes to make the new table available to us.
Cruikshanks
added a commit
that referenced
this pull request
Aug 28, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS **Context** We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. Thanks to [this change in water-abstraction-import we are now importing them](DEFRA/water-abstraction-import#1009) to a [new table in the `water` schema](DEFRA/water-abstraction-service#2626). And with [Add Licence Version Purpose Point model](#1288) we enabled access to the data via [Objection.js](https://vincit.github.io/objection.js/). **Change** Depending on the detail in a 'point' it needs to be described in 3 different ways. - when a single grid reference as a _point_ - when two grid references as a _reach_ - when four grid references as an _area_ > A point can only have 1, 2 or 4 grid references, never 3! We already have so many places in this repo where we need to work this out that we have a general helper method (`generateAbstractionPointDetail()` in `app/lib/general.lib.js`) to do it. This will become defunct, however, when we start pulling the data using our new `LicenceVersionPurposePointModel`. We already have a pattern for models that contain data that needs to be interpreted in a certain way, for example, contact name in `ContactModel` and licence holder in `LicenceModel`. The slight difference is we now have _two_ models that need the same ability. So, this change will introduce a new `BasePointModel` that the existing `ReturnRequirementPointModel` and `LicenceVersionPurposePointModel` will extend. On the new `BasePointModel` we'll add the Objection.js instance function `$describe()` that will do the work of interpreting the point data and determining what description to use. > In later changes we'll start updating the repo to use the new helper and model rather than the old one.
Cruikshanks
added a commit
that referenced
this pull request
Aug 29, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS **Context** We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. Thanks to [this change in water-abstraction-import we are now importing them](DEFRA/water-abstraction-import#1009) to a [new table in the `water` schema](DEFRA/water-abstraction-service#2626). And with [Add Licence Version Purpose Point model](#1288) we enabled access to the data via [Objection.js](https://vincit.github.io/objection.js/). **Change** Depending on the detail in a 'point', it needs to be described in 3 different ways. - when a single grid reference as a _point_ - when two grid references as a _reach_ - when four grid references as an _area_ > A point can only have 1, 2 or 4 grid references, never 3! We already have so many places in this repo where we need to work this out that we have a general helper method (`generateAbstractionPointDetail()` in `app/lib/general.lib.js`). This will become defunct, however, when we start pulling the data using our new `LicenceVersionPurposePointModel`. We already have a pattern for models that contain data that needs to be interpreted in a certain way, such as the contact name in `ContactModel` and the licence holder in `LicenceModel`. The slight difference is that we now have _two_ models that need the same ability. So, this change will introduce a new `BasePointModel` that the existing `ReturnRequirementPointModel` and `LicenceVersionPurposePointModel` will extend. On the new `BasePointModel`, we'll add the Objection.js instance function `$describe()`, which will interpret the point data and determine what description to use. > In later changes, we'll start updating the repo to use the new helper and model rather than the old one.
Cruikshanks
added a commit
that referenced
this pull request
Sep 2, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. In [Add Licence Version Purpose Point model](#1288) to allow us to access the data. The in [Add base point model with a describe helper](#1289) we extended the models to hand how to describe a point. With all this in place, we can now get back to updating the return requirements setup journey to source 'points' from the new table. This will fix the issue we currently have with it and get it working for all licences again. The fix requires - updating the 'use abstraction data' journey to pull points from the new table - updating the 'manual journey' to pull points from the new table We'll also update the 'copy from existing' journey to use the new features of the `BasePointModel` rather than the general lib helper.
Cruikshanks
added a commit
that referenced
this pull request
Sep 4, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. In [Add Licence Version Purpose Point model](#1288) to allow us to access the data. The in [Add base point model with a describe helper](#1289) we extended the models to hand how to describe a point. With all this in place, we can now get back to updating the return requirements setup journey to source 'points' from the new table. This will fix the issue we currently have with it and get it working for all licences again. The fix requires - updating the 'use abstraction data' journey to pull points from the new table - updating the 'manual journey' to pull points from the new table We'll also update the 'copy from existing' journey to use the new features of the `BasePointModel` rather than the general lib helper.
Cruikshanks
added a commit
that referenced
this pull request
Sep 5, 2024
https://eaflood.atlassian.net/browse/WATER-4645 https://eaflood.atlassian.net/browse/WATER-4635 We recently added 2 new models connected to two new tables to support managing return requirements and viewing licence history. - [Add Licence Version Purpose Point model](#1288) - [Add Mod Log model](#1264) When we did though we forgot to make them available to the acceptance test loader!
Cruikshanks
added a commit
that referenced
this pull request
Sep 5, 2024
https://eaflood.atlassian.net/browse/WATER-4645 https://eaflood.atlassian.net/browse/WATER-4635 We recently added two new models connected to two new tables to support managing return requirements and viewing licence history. - [Add Licence Version Purpose Point model](#1288) - [Add Mod Log model](#1264) When we did, though, we forgot to make them available to the acceptance test loader! Plus, as we are adding them to the test loader, we also need to ensure they are included in the teardown.
Cruikshanks
added a commit
that referenced
this pull request
Sep 6, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. So, we needed another source for points. In [Start importing licence version points to 'water'](DEFRA/water-abstraction-import#1009) we updated the **water-abstraction-import** to start importing licence version purpose points to a new table. We added the model to access the data in [Add Licence Version Purpose Point model](#1288). In [Add base point model with a describe helper](#1289), we extended the models to show how to describe a point. With all this in place, we can now update the return requirements setup journey to source 'points' from the new table. This will fix the issue we currently have with it and get it working again for all licences. The fix requires - updating the 'use abstraction data' journey to pull points from the new table - updating the 'manual journey' to pull points from the new table We'll also update the 'copy from existing' journey to use the new features of the `BasePointModel` rather than the general lib helper.
Cruikshanks
added a commit
that referenced
this pull request
Sep 24, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS In [Add Licence Version Purpose Point model](#1288) we added our second model that represented a table in the `water` schema that hold point information. When we added `ReturnRequirementPointModel` it was to represent points linked to a return requirement. In the setup journey for new return requirements you select the points to associate from those against the licence. We were extracting this information from the `permit.licence` table until we realised that [water-abstraction-import](https://github.com/DEFRA/water-abstraction-import) does not populate it when a licence is 'dead'. We need to be able to create new return requirements even for dead licences in order to correct historic information. So, this source being defunct we made changes to the import to populate a new `water.licence_version_purpose_points` from which we could then derive a licence's points. That worked great for returns setup, so we moved to updating the view licence summary page to use the new source. It also was effected by the issue that point information was no longer available if the licence was dead. The page includes information about the water source for a licence. It was then we realised that the water source is derived from a licence's points! What we'd built for return requirements and licence version purposes was missing this information. We investigated the NALD data properly and came away with some realisations - NALD maintains a central list of 'points' (we knew this, but like addresses felt this was unnecessary) - Each point holds a lot more data than just the National Grid References for its location - Each NALD point is linked to a NALD source, and these too hold extra information There was extra information users were maintaining against each point that we were missing. This information explained why points were centrally managed and other records were linked to them. This realisation meant we needed to completely re-think how we should maintain points in WRLS going forward. So, in [the water-abstraction-service](DEFRA/water-abstraction-service#2638) we've created a migration to add new `sources` and `points` tables. We'also amended the existing `licence_version_purpose_points` and `return_requirement_points` to make it possible to link them to `points`. (Their `ngr_*` fields plus things like `description` are now defunct but to avoid breaking changes we'll need to drop them in a future change.) In [water-abstraction-import](DEFRA/water-abstraction-import#1028) we've added a new job to ensure `points` gets populated as part of the import. We've also incorporated populating and linking `licence_version_purpose_points` and `return_requirement_points` as part of the new job. With those changes in place, this change updates everything around the existing point models, as well as adding the new ones. The water sources are reference data, so as part of adding `SourceModel` we include a seeder. The existing models and their tests are updated to represent the fact they are now just `many-to-many` models. This means we make `points` a top level property of `ReturnRequirementModel` and `LicenceVersionPurposeModel`. The logic for describing a point is copied to `PointModel`. We'll delete `BasePointModel` in the next change, where we update the existing functionality to use the new structure.
Cruikshanks
added a commit
that referenced
this pull request
Sep 25, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS In [Add Licence Version Purpose Point model](#1288), we added our second model, which represented a table in the `water` schema that holds point information. When we added `ReturnRequirementPointModel,` it was to represent points linked to a return requirement. In the setup journey for new return requirements, you select the points to associate with those against the licence. We were extracting this information from the `permit.licence` table until we realised that [water-abstraction-import](https://github.com/DEFRA/water-abstraction-import) does not populate it when a licence is 'dead'. We need to be able to create new return requirements, even for dead licences, to correct historical information. So, this source being defunct, we made changes to the import to populate a new `water.licence_version_purpose_points` from which we could then derive a licence's points. That worked great for returns setup, so we also updated the view licence summary page to use `water.licence_version_purpose_points`. The issue that point information was not available if the licence was dead affected it. The page also includes information about a licence's water source. It was then that we realised that the water source is derived from a licence's points! What we'd built for return requirements and licence version purposes was missing this information. We investigated the NALD data properly and came up with some new realisations. - NALD maintains a central list of 'points' (we knew this, but like addresses felt this was unnecessary) - Each point holds a lot more data than just the National Grid References for its location - Each NALD point is linked to a NALD source, and these, too, hold extra information There was extra information users were maintaining against each point that we were missing. This information explained why points were centrally managed and linked to other records. This realisation meant we needed to completely re-think how we should maintain points in WRLS going forward. So, in [the water-abstraction-service](DEFRA/water-abstraction-service#2638), we've created a migration to add new `sources` and `points` tables. We also amended the existing `licence_version_purpose_points` and `return_requirement_points` to make it possible to link them to `points`. (Their `ngr_*` fields plus things like `description` are now defunct, but to avoid breaking changes, we'll need to drop them in a future change.) In [water-abstraction-import](DEFRA/water-abstraction-import#1028), we've added a new job to ensure `points` get populated as part of the import. We've also incorporated populating and linking `licence_version_purpose_points` and `return_requirement_points` into the new job. With those changes in place, this change updates everything around the existing point models and adds new ones. The water sources are reference data, so we include a seeder when adding `SourceModel`. We update the existing models and their tests to reflect the fact that they are now just `many-to-many` models. This means we make `points` a top-level property of `ReturnRequirementModel` and `LicenceVersionPurposeModel`. We then go through _all_ existing logic and update it to use the new points data structure.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://eaflood.atlassian.net/browse/WATER-4645
We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we Created a return-requirement-points table and updated the import to import them.
Users select these points as part of the return requirements setup journey we've built. We extract them from the JSON blob stored in the
permit.licencetable. The problem we've encountered is the import service only populates the points for licences that.Otherwise,
permit.licenceis not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this.For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to.
We don't know why the previous team never opted to extract licence points to their own table. But we are now importing them as part of the licence import. This change adds the model and supporting changes to make the new table available to us.