Import return requirement points #933
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Cruikshanks
added a commit
that referenced
this pull request
Aug 27, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS for managing them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](#933) to import them. These points are selected by users as part of the return requirements setup journey we've built. Currently, we extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that - Have not ended - Have a current licence version Else, `permit.licence` is not populated with the points data our journey relies on causing it to throw an error. There are also places like the view licence page that are affected by this. It is perfectly valid, for example, that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state that we can see what points the licence was linked to. Why the previous team never opted to extract licence points to their own table we don't know. But this change imports licence version points to a table so we can stop depending on the `permit.licence` table and its incomplete data.
Cruikshanks
added a commit
to DEFRA/water-abstraction-service
that referenced
this pull request
Aug 27, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them. Users select these points as part of the return requirements setup journey we've built. We extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. We don't know why the previous team never opted to extract licence points to their own table. But this change adds a new migration to create the table.
Cruikshanks
added a commit
to DEFRA/water-abstraction-service
that referenced
this pull request
Aug 27, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them. Users select these points as part of the return requirements setup journey we've built. We extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. We don't know why the previous team never opted to extract licence points to their own table. But this change adds a new migration to create the table.
Cruikshanks
added a commit
that referenced
this pull request
Aug 27, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](#933) to import them. Users select these points as part of the return requirements setup journey we've built. We extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. We don't know why the previous team never opted to extract licence points to their own table. But this change imports licence version purpose points to a table so we can stop depending on the `permit.licence` table and its incomplete data.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Aug 27, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them. Users select these points as part of the return requirements setup journey we've built. We extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. We don't know why the previous team never opted to extract licence points to their own table. But we are now [importing them as part of the licence import](DEFRA/water-abstraction-import#1009). This change adds the model and supporting changes to make the new table available to us.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Aug 27, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them. Users select these points as part of the return requirements setup journey we've built. We extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. We don't know why the previous team never opted to extract licence points to their own table. But we are now [importing them as part of the licence import](DEFRA/water-abstraction-import#1009). This change adds the model and supporting changes to make the new table available to us.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Aug 28, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS **Context** We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. Thanks to [this change in water-abstraction-import we are now importing them](DEFRA/water-abstraction-import#1009) to a [new table in the `water` schema](DEFRA/water-abstraction-service#2626). And with [Add Licence Version Purpose Point model](#1288) we enabled access to the data via [Objection.js](https://vincit.github.io/objection.js/). **Change** Depending on the detail in a 'point' it needs to be described in 3 different ways. - when a single grid reference as a _point_ - when two grid references as a _reach_ - when four grid references as an _area_ > A point can only have 1, 2 or 4 grid references, never 3! We already have so many places in this repo where we need to work this out that we have a general helper method (`generateAbstractionPointDetail()` in `app/lib/general.lib.js`) to do it. This will become defunct, however, when we start pulling the data using our new `LicenceVersionPurposePointModel`. We already have a pattern for models that contain data that needs to be interpreted in a certain way, for example, contact name in `ContactModel` and licence holder in `LicenceModel`. The slight difference is we now have _two_ models that need the same ability. So, this change will introduce a new `BasePointModel` that the existing `ReturnRequirementPointModel` and `LicenceVersionPurposePointModel` will extend. On the new `BasePointModel` we'll add the Objection.js instance function `$describe()` that will do the work of interpreting the point data and determining what description to use. > In later changes we'll start updating the repo to use the new helper and model rather than the old one.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Aug 29, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS **Context** We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. Thanks to [this change in water-abstraction-import we are now importing them](DEFRA/water-abstraction-import#1009) to a [new table in the `water` schema](DEFRA/water-abstraction-service#2626). And with [Add Licence Version Purpose Point model](#1288) we enabled access to the data via [Objection.js](https://vincit.github.io/objection.js/). **Change** Depending on the detail in a 'point', it needs to be described in 3 different ways. - when a single grid reference as a _point_ - when two grid references as a _reach_ - when four grid references as an _area_ > A point can only have 1, 2 or 4 grid references, never 3! We already have so many places in this repo where we need to work this out that we have a general helper method (`generateAbstractionPointDetail()` in `app/lib/general.lib.js`). This will become defunct, however, when we start pulling the data using our new `LicenceVersionPurposePointModel`. We already have a pattern for models that contain data that needs to be interpreted in a certain way, such as the contact name in `ContactModel` and the licence holder in `LicenceModel`. The slight difference is that we now have _two_ models that need the same ability. So, this change will introduce a new `BasePointModel` that the existing `ReturnRequirementPointModel` and `LicenceVersionPurposePointModel` will extend. On the new `BasePointModel`, we'll add the Objection.js instance function `$describe()`, which will interpret the point data and determine what description to use. > In later changes, we'll start updating the repo to use the new helper and model rather than the old one.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Sep 2, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. In [Add Licence Version Purpose Point model](#1288) to allow us to access the data. The in [Add base point model with a describe helper](#1289) we extended the models to hand how to describe a point. With all this in place, we can now get back to updating the return requirements setup journey to source 'points' from the new table. This will fix the issue we currently have with it and get it working for all licences again. The fix requires - updating the 'use abstraction data' journey to pull points from the new table - updating the 'manual journey' to pull points from the new table We'll also update the 'copy from existing' journey to use the new features of the `BasePointModel` rather than the general lib helper.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Sep 4, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. In [Add Licence Version Purpose Point model](#1288) to allow us to access the data. The in [Add base point model with a describe helper](#1289) we extended the models to hand how to describe a point. With all this in place, we can now get back to updating the return requirements setup journey to source 'points' from the new table. This will fix the issue we currently have with it and get it working for all licences again. The fix requires - updating the 'use abstraction data' journey to pull points from the new table - updating the 'manual journey' to pull points from the new table We'll also update the 'copy from existing' journey to use the new features of the `BasePointModel` rather than the general lib helper.
Cruikshanks
added a commit
to DEFRA/water-abstraction-acceptance-tests
that referenced
this pull request
Sep 6, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import these point details. Users select these points as part of the return requirements setup journey we've built. We were extracting them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. So, we needed another source for points. In [Start importing licence version points to 'water'](DEFRA/water-abstraction-import#1009) we updated the **water-abstraction-import** to start importing licence version purpose points to a new table. Then in [Update Return req set up journey to use new points](DEFRA/water-abstraction-system#1301) we updated **water-abstraction-system** to use this new source in the journey. These changes have fixed the problem but broken our acceptance tests! We need to be loading these new `water.licence_version_purpose_points` instead of the `permit.licence` table. In this change we update the fixtures that depend on populating the `permit.licence` table to instead populate our new table.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Sep 6, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. So, we needed another source for points. In [Start importing licence version points to 'water'](DEFRA/water-abstraction-import#1009) we updated the **water-abstraction-import** to start importing licence version purpose points to a new table. We added the model to access the data in [Add Licence Version Purpose Point model](#1288). In [Add base point model with a describe helper](#1289), we extended the models to show how to describe a point. With all this in place, we can now update the return requirements setup journey to source 'points' from the new table. This will fix the issue we currently have with it and get it working again for all licences. The fix requires - updating the 'use abstraction data' journey to pull points from the new table - updating the 'manual journey' to pull points from the new table We'll also update the 'copy from existing' journey to use the new features of the `BasePointModel` rather than the general lib helper.
Cruikshanks
added a commit
to DEFRA/water-abstraction-acceptance-tests
that referenced
this pull request
Sep 8, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import these point details. Users select these points as part of the return requirements setup journey we've built. We were extracting them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. So, we needed another source for points. In [Start importing licence version points to 'water'](DEFRA/water-abstraction-import#1009) we updated the **water-abstraction-import** to start importing licence version purpose points to a new table. Then, in [Update Return req set up journey to use new points](DEFRA/water-abstraction-system#1301), we updated **water-abstraction-system** to use this new source in the journey. These changes have fixed the problem but broken our acceptance tests! We must load the new `water.licence_version_purpose_points` table instead of the `permit.licence` table. In this change, we update the fixtures that depend on populating the `permit.licence` table to populate our new table instead.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://eaflood.atlassian.net/browse/WATER-4475
As part of our plan to replace NALD we need to import the abstraction point detail. This PR adds a script that will get the information from the import tables and add it to the newly created return-requirement-points table.