Start importing licence version points to 'water' #1009
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS for managing them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](#933) to import them. These points are selected by users as part of the return requirements setup journey we've built. Currently, we extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that - Have not ended - Have a current licence version Else, `permit.licence` is not populated with the points data our journey relies on causing it to throw an error. There are also places like the view licence page that are affected by this. It is perfectly valid, for example, that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state that we can see what points the licence was linked to. Why the previous team never opted to extract licence points to their own table we don't know. But this change imports licence version points to a table so we can stop depending on the `permit.licence` table and its incomplete data.
When we amended the import to bring in return requirement points we were able to tag on another query to those already being run as part of the 'charging-data' job. Unfortunately, for 'reasons', the previous team made the licence import job much more complicated. It is broken down into various 'jobs', which typically - queue up a bunch of jobs, for example, licences to import - process each individual licence import job We're in the middle of migrating the licence import to [water-abstraction-system](https://github.com/DEFRA/water-abstraction-system) which will do away with all this needless complexity. But that won't 'land' for some time and we need points now. This is a half-way house. Adding this licence-based query to the 'charging-data' job didn't seem right. But replicating how things are done in the 'licence-import' job seemed overkill. So, we've amended the job process in licence import to queue up a new 'import-points' job when the last licence has been imported. But the job is just running a single query to create or update the points in one go.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Aug 27, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them. Users select these points as part of the return requirements setup journey we've built. We extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. We don't know why the previous team never opted to extract licence points to their own table. But we are now [importing them as part of the licence import](DEFRA/water-abstraction-import#1009). This change adds the model and supporting changes to make the new table available to us.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Aug 27, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them. Users select these points as part of the return requirements setup journey we've built. We extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. We don't know why the previous team never opted to extract licence points to their own table. But we are now [importing them as part of the licence import](DEFRA/water-abstraction-import#1009). This change adds the model and supporting changes to make the new table available to us.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Aug 28, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS **Context** We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. The problem we've encountered is the import service only populates the points for licences that. - Have not ended - Have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. Thanks to [this change in water-abstraction-import we are now importing them](DEFRA/water-abstraction-import#1009) to a [new table in the `water` schema](DEFRA/water-abstraction-service#2626). And with [Add Licence Version Purpose Point model](#1288) we enabled access to the data via [Objection.js](https://vincit.github.io/objection.js/). **Change** Depending on the detail in a 'point' it needs to be described in 3 different ways. - when a single grid reference as a _point_ - when two grid references as a _reach_ - when four grid references as an _area_ > A point can only have 1, 2 or 4 grid references, never 3! We already have so many places in this repo where we need to work this out that we have a general helper method (`generateAbstractionPointDetail()` in `app/lib/general.lib.js`) to do it. This will become defunct, however, when we start pulling the data using our new `LicenceVersionPurposePointModel`. We already have a pattern for models that contain data that needs to be interpreted in a certain way, for example, contact name in `ContactModel` and licence holder in `LicenceModel`. The slight difference is we now have _two_ models that need the same ability. So, this change will introduce a new `BasePointModel` that the existing `ReturnRequirementPointModel` and `LicenceVersionPurposePointModel` will extend. On the new `BasePointModel` we'll add the Objection.js instance function `$describe()` that will do the work of interpreting the point data and determining what description to use. > In later changes we'll start updating the repo to use the new helper and model rather than the old one.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Aug 29, 2024
https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS **Context** We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this. For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to. Thanks to [this change in water-abstraction-import we are now importing them](DEFRA/water-abstraction-import#1009) to a [new table in the `water` schema](DEFRA/water-abstraction-service#2626). And with [Add Licence Version Purpose Point model](#1288) we enabled access to the data via [Objection.js](https://vincit.github.io/objection.js/). **Change** Depending on the detail in a 'point', it needs to be described in 3 different ways. - when a single grid reference as a _point_ - when two grid references as a _reach_ - when four grid references as an _area_ > A point can only have 1, 2 or 4 grid references, never 3! We already have so many places in this repo where we need to work this out that we have a general helper method (`generateAbstractionPointDetail()` in `app/lib/general.lib.js`). This will become defunct, however, when we start pulling the data using our new `LicenceVersionPurposePointModel`. We already have a pattern for models that contain data that needs to be interpreted in a certain way, such as the contact name in `ContactModel` and the licence holder in `LicenceModel`. The slight difference is that we now have _two_ models that need the same ability. So, this change will introduce a new `BasePointModel` that the existing `ReturnRequirementPointModel` and `LicenceVersionPurposePointModel` will extend. On the new `BasePointModel`, we'll add the Objection.js instance function `$describe()`, which will interpret the point data and determine what description to use. > In later changes, we'll start updating the repo to use the new helper and model rather than the old one.
Cruikshanks
added a commit
to DEFRA/water-abstraction-acceptance-tests
that referenced
this pull request
Sep 6, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import these point details. Users select these points as part of the return requirements setup journey we've built. We were extracting them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. So, we needed another source for points. In [Start importing licence version points to 'water'](DEFRA/water-abstraction-import#1009) we updated the **water-abstraction-import** to start importing licence version purpose points to a new table. Then in [Update Return req set up journey to use new points](DEFRA/water-abstraction-system#1301) we updated **water-abstraction-system** to use this new source in the journey. These changes have fixed the problem but broken our acceptance tests! We need to be loading these new `water.licence_version_purpose_points` instead of the `permit.licence` table. In this change we update the fixtures that depend on populating the `permit.licence` table to instead populate our new table.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Sep 6, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import them point details. Users select these points as part of the return requirements setup journey we've built. We currently extract them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. So, we needed another source for points. In [Start importing licence version points to 'water'](DEFRA/water-abstraction-import#1009) we updated the **water-abstraction-import** to start importing licence version purpose points to a new table. We added the model to access the data in [Add Licence Version Purpose Point model](#1288). In [Add base point model with a describe helper](#1289), we extended the models to show how to describe a point. With all this in place, we can now update the return requirements setup journey to source 'points' from the new table. This will fix the issue we currently have with it and get it working again for all licences. The fix requires - updating the 'use abstraction data' journey to pull points from the new table - updating the 'manual journey' to pull points from the new table We'll also update the 'copy from existing' journey to use the new features of the `BasePointModel` rather than the general lib helper.
Cruikshanks
added a commit
to DEFRA/water-abstraction-acceptance-tests
that referenced
this pull request
Sep 8, 2024
https://eaflood.atlassian.net/browse/WATER-4600 https://eaflood.atlassian.net/browse/WATER-4645 > Part of the work to migrate return versions from NALD to WRLS We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this, we [Created a return-requirement-points table](DEFRA/water-abstraction-service#2540) and [updated the import](DEFRA/water-abstraction-import#933) to import these point details. Users select these points as part of the return requirements setup journey we've built. We were extracting them from the JSON blob stored in the `permit.licence` table. However, the import service only populates the points for licences. - That have not ended - That have a current licence version Otherwise, `permit.licence` is not populated with the points data our journey relies on, causing it to throw an error. So, we needed another source for points. In [Start importing licence version points to 'water'](DEFRA/water-abstraction-import#1009) we updated the **water-abstraction-import** to start importing licence version purpose points to a new table. Then, in [Update Return req set up journey to use new points](DEFRA/water-abstraction-system#1301), we updated **water-abstraction-system** to use this new source in the journey. These changes have fixed the problem but broken our acceptance tests! We must load the new `water.licence_version_purpose_points` table instead of the `permit.licence` table. In this change, we update the fixtures that depend on populating the `permit.licence` table to populate our new table instead.
Cruikshanks
added a commit
that referenced
this pull request
Sep 24, 2024
https://eaflood.atlassian.net/browse/WATER-4645 In [Start importing licence version points to 'water'](#1009) we added a step to the **licence-import** job to start importing licence points to a new table we'd created: `water.licence_version_purpose_points`. However, we've since encountered two issues with that original approach. - We've found the legacy UI also display details about sources, which are connected to points. Our original solution didn't cater for this which means we cannot switch wholly to it - Updating the `licence_version_purpose_points` table happens as the last step of the **licence-import** job. But we've found it is often failing because of bad licence data, which means our new step is never getting called So, in this change we are creating a whole new job to handle points. Part of the change is NALD points will now sit in their own table. `return_requirement_points` and `licence_version_purpose_points` will link to it rather than holding their own copies of the licence. The job job will be responsible for updating all three.
Cruikshanks
added a commit
that referenced
this pull request
Sep 24, 2024
https://eaflood.atlassian.net/browse/WATER-4645 In [Start importing licence version points to 'water'](#1009), we added a step to the **licence-import** job to start importing licence points to a new table we'd created: `water.licence_version_purpose_points`. However, we've since encountered two issues with that original approach. - We've found the legacy UI also displays details about sources, which are connected to points. Our original solution didn't cater for this, which means we cannot switch wholly to it - Updating the `licence_version_purpose_points` table happens as the last step of the **licence-import** job. But we've found it is often failing because of bad licence data, which means our new step is never getting called So, in this change, we are creating a whole new job to handle points. Part of the change is NALD points will now sit in their own table. `return_requirement_points` and `licence_version_purpose_points` will link to it rather than holding their own copies of the licence. The job will be responsible for updating all three.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://eaflood.atlassian.net/browse/WATER-4645
We've been extending and amending the import of return versions from NALD to WRLS as part of our work to switch from NALD to WRLS to manage them. To support this we Created a return-requirement-points table and updated the import to import them.
Users select these points as part of the return requirements setup journey we've built. We extract them from the JSON blob stored in the
permit.licencetable. The problem we've encountered is the import service only populates the points for licences that.Otherwise,
permit.licenceis not populated with the points data our journey relies on, causing it to throw an error. Places like the view licence page are also affected by this.For example, it is perfectly valid that we have an 'ended' licence that we need to correct the historic return versions. And no matter the state, we can see what points the licence was linked to.
We don't know why the previous team never opted to extract licence points to their own table. But this change imports licence version purpose points to a table so we can stop depending on the
permit.licencetable and its incomplete data.