Move service status page from UI to system #54
Labels
assimilate
Moving functionality from the old service into wabs-system
Comments
Cruikshanks
added
the
assimilate
StuAA78
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Oct 12, 2022
* Add views to system DEFRA/water-abstraction-team#54 The first step to moving the service status page from the UI to our new system is to implement views. We do this using https://github.com/DEFRA/hapi-web-boilerplate as a basis to work from. Once implemented, we add a simple `/hello-world` page, based on the example in `hapi-web-boilerplate`. Building assets is done in the exact same way as the boilerplate app: - `npm run build` calls a bash script `bin/build`. This script is intended to run whatever build stages are needed, but at present it only runs `npm run build:css`. - `npm run build:css` uses `node-sass` to compile the sass stylesheet currently in `client/sass/application.scss` (which simply links to the `govuk-frontend` stylesheet) to standard css in `app/public/build`. Note that we update `.gitignore` so that this folder isn't checked in. - A `postinstall` script is added to `package.json` so that `npm run build` is automatically run after an `npm ci` or `npm install`. A few points to bear in mind for potential future changes: - Is `client/sass/application.scss` where we want our sass stylesheet to live? - We've followed the model of the boilerplate app and serve a static file for `/robots.txt` (currently living in `app/public/static/robots.txt`). In the existing UI we have a `no-robots` plugin which serves this up on request without needing a static file to be stored. Do we want to do this instead? - We probably want to double-check the `ErrorPagesPlugin` to ensure it correctly logs 500 errors; for now we've simply copied the error format in `hapi-web-boilerplate`, adapting it to use our own logger. - Google Analytics hasn't yet been brought over from `hapi-web-boilerplate`.
Beckyrose200
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Oct 13, 2022
DEFRA/water-abstraction-team#54 Here we have the addition of a status page to the service repo. This allows us to see the different status of the service extensions (virus checker, database etc)
Jozzey
pushed a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Oct 19, 2022
DEFRA/water-abstraction-team#54 Here we have the addition of a status page to the service repo. This allows us to see the different status of the service extensions (virus checker, database etc). This change just creates the initial holding page. We will add the additional functionality in separate PRs
Jozzey
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Oct 19, 2022
DEFRA/water-abstraction-team#54 We are going to create a summary list section for each app and we are also going to remove the tables from shared apps like virus scanner
Cruikshanks
added a commit
to DEFRA/water-abstraction-ui
that referenced
this issue
Oct 25, 2022
DEFRA/water-abstraction-team#54 This is in support of the new service-status page we're building in the [water-abstraction-system repo](https://github.com/DEFRA/water-abstraction-system). We want to be able to show both the version and commit hash for each service in that page. The existing `/status` page does include the `package.json` version. But that endpoint is also used for the AWS load balancer health check so is heavily hit. We don't want to make it do more work and get the commit hash as well. Instead, we add a new `/health/` path and a new `/info` endpoint to return both the version and commit hash.
Cruikshanks
added a commit
to DEFRA/water-abstraction-ui
that referenced
this issue
Oct 25, 2022
DEFRA/water-abstraction-team#54 This supports the new service-status page we're building in the [water-abstraction-system repo](https://github.com/DEFRA/water-abstraction-system). We want to be able to show both the version and commit hash for each service on that page. The existing `/status` page does include the `package.json` version. But that endpoint is also used for the AWS load balancer health check so is heavily hit. We don't want to make it do more work getting the commit hash. Instead, we add a new `/health/` path and a new `/info` endpoint to return both the version and commit hash. ** Notes - We want the endpoint available in both internal and external apps. So, we add the functionality to the shared folder
Cruikshanks
pushed a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Nov 1, 2022
DEFRA/water-abstraction-team#54 We are going to create a summary list section for each app and we are also going to remove the tables from shared apps like virus scanner
Cruikshanks
pushed a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Nov 1, 2022
DEFRA/water-abstraction-team#54 We are going to create a summary list section for each app and we are also going to remove the tables from shared apps like virus scanner
Cruikshanks
pushed a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Nov 1, 2022
DEFRA/water-abstraction-team#54 We are going to create a summary list section for each app and we are also going to remove the tables from shared apps like virus scanner
Cruikshanks
pushed a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Nov 1, 2022
DEFRA/water-abstraction-team#54 We are going to create a summary list section for each app and we are also going to remove the tables from shared apps like virus scanner
Cruikshanks
pushed a commit
to DEFRA/water-abstraction-ui
that referenced
this issue
Nov 3, 2022
https://eaflood.atlassian.net/browse/WATER-3787 DEFRA/water-abstraction-team#54 Going forward, we intend all new functionality to be built in the [water-abstraction-system](https://github.com/DEFRA/water-abstraction-system) repo. We even intend to _try_ and migrate any existing functionality we touch. In a perfect world, we'll end up with all the existing repos becoming defunct and a clean, performant service all in one repo. To get there though, we still need to work with the existing services and the existing environments. In those, the only apps exposed are the internal and eternal UI which are based on this repo. So, for any requests we want the water-abstraction-system to handle we need to 'proxy' them via the water-abstraction-ui. So, this change adds the ability for the UI to proxy all requests sent to `/system` to the water-abstraction-system app behind the scenes. One of the challenges we hit was assets. We realised when we requested a view, assets would also be requested. So, included in the change is the ability for the UI to proxy asset requests for the system as well.
Cruikshanks
pushed a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Nov 3, 2022
DEFRA/water-abstraction-team#54 We are going to create a summary list section for each app and we are also going to remove the tables from shared apps like virus scanner
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Nov 6, 2022
DEFRA/water-abstraction-team#54 We wanted to make the layout for each repo consistent and relate things like jobs to the service they belong to. We then decided we'd report on 'apps' rather than just 'repos', as that is what is running on a server and which 'the service' is dependent on. We tidied up the external services section, including adding the Charge Module API which we'd, though not in the existing /service-status page, should be. We also got them to return meaningful information. All the calls for info are now robust. Even if all the other apps and external services were down the page would still display. All this work meant some major refactoring of the `ServiceStatusService()`, for example, a single method for making HTTP requests to the other apps and services. All of this is tested. To do this we needed to bring in [Nock](https://github.com/nock/nock) to mock our HTTP requests to the other services and [proxyquire](https://github.com/thlorenz/proxyquire) to allow us to stub our calls to `child_process.exec()`. ** Thoughts on the current state Looking at the tests we see a copious amount of work needed to mock web requests and system calls. We also have a range of scenarios. In our opinion, they are telling us that our service is doing _too_ much. It is collecting data from different sources in different ways, which means it definitely doesn't just have a single responsibility. It needs breaking up, which in turn means we can break up the tests and hopefully make them a damn sight less scary. We didn't do those changes in this PR though, because we'd already clocked up more than 1000 changes, well beyond our working target of 100. So, we've agreed to merge and tackle the refactoring as a separate exercise. ** Rebuilding the package-lock.json Because of how long this PR had been around and the fact it adds new dependencies, we ended up having major merge issues in the `package-lock.json`. We went for a 'delete-and-rebuild' approach but we know we shouldn't have 😳 . Next time we'll endeavour to use `npm install --package-lock-only` which we [recently learned](https://tkdodo.eu/blog/solving-conflicts-in-package-lock-json) is a command added to npm to help rectify the package-lock.json in these 'merge-hell' situations. Co-authored-by: Stuart Adair <[email protected]> Co-authored-by: Rebecca Ransome <[email protected]> Co-authored-by: Alan Cruikshanks <[email protected]>
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Nov 10, 2022
DEFRA/water-abstraction-team#54 Our status page depends on sending requests to the other running apps and 3rd party services. All is fine in our Vagrant box but when we deployed to our AWS environments the requests just seemed to hang. After some investigation inspiration hit! Our AWS environments require all requests to go through a proxy server and without support for this, our requests are getting stuck. We tackled this same problem in [Replace existing proxy config with global-agent](DEFRA/sroc-charging-module-api#572) in the sroc-charging-module-api. In the Charging Module, we used [global-agent](https://github.com/gajus/global-agent). This intercepts all http/https requests and adds the proxy details them. Perfect! The problem is it was last updated more than a year ago, the build is marked as failing, test coverage is low, and issues and PR's seem to be going unanswered. Our other issue is it does not recognise the lowercase versions of standard proxy env vars (`http_proxy`, `https_proxy` and `no_proxy`). It's a shame because it works seemlessly in the background. The folks behind the request package we do use, [Got](https://github.com/sindresorhus/got), recommend a couple. We liked [hpagent](https://github.com/delvedor/hpagent/). The example was simple, it's being regularly updated, supports [keep-alive connections](https://en.wikipedia.org/wiki/HTTP_persistent_connection), and even uses [standardjs](https://standardjs.com/)! So, this change brings in support for working behind a proxy using **hpagent**. As part of this we'll be refactoring the work of making HTTP requests into its own service. We'll also solve one of the other things we noted; the requests don't timeout if there is a problem making a connection.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Nov 10, 2022
DEFRA/water-abstraction-team#54 Our status page depends on sending requests to the other running apps and 3rd party services. All is fine in our Vagrant box but when we deployed to our AWS environments the requests just seemed to hang. After some investigation inspiration hit! Our AWS environments require all requests to go through a proxy server and without support for this, our requests are getting stuck. We tackled this same problem in [Replace existing proxy config with global-agent](DEFRA/sroc-charging-module-api#572) in the sroc-charging-module-api. In the Charging Module, we used [global-agent](https://github.com/gajus/global-agent). This intercepts all http/https requests and adds the proxy details them. Perfect! The problem is it was last updated more than a year ago, the build is marked as failing, test coverage is low, and issues and PR's seem to be going unanswered. Our other issue is it does not recognise the lowercase versions of standard proxy env vars (`http_proxy`, `https_proxy` and `no_proxy`). It's a shame because it works seemlessly in the background. The folks behind the request package we do use, [Got](https://github.com/sindresorhus/got), recommend a couple. We liked [hpagent](https://github.com/delvedor/hpagent/). The example was simple, it's being regularly updated, supports [keep-alive connections](https://en.wikipedia.org/wiki/HTTP_persistent_connection), and even uses [standardjs](https://standardjs.com/)! So, this change brings in support for working behind a proxy using **hpagent**. As part of this we'll be refactoring the work of making HTTP requests into its own service. We'll also solve one of the other things we noted; the requests don't timeout if there is a problem making a connection.
Cruikshanks
added a commit
to DEFRA/water-abstraction-ui
that referenced
this issue
Nov 30, 2022
DEFRA/water-abstraction-team#54 https://eaflood.atlassian.net/browse/WATER-3787 In a previous change we [Create system proxy endpoint](#2229). We need this to be able to send requests to the [water-abstraction-system](https://github.com/DEFRA/water-abstraction-system) when using our AWS environments. That all seemed to work, until we changed an endpoint in **water-absraction-system** to require a query param to be passed through. Our proxy isn't doing this so now **water-abstraction-system** errors because of the missing param. This change fixes the issue.
Cruikshanks
added a commit
to DEFRA/water-abstraction-ui
that referenced
this issue
Dec 1, 2022
DEFRA/water-abstraction-team#54 https://eaflood.atlassian.net/browse/WATER-3787 In a previous change we [Create system proxy endpoint](#2229). We need this to be able to send requests to the [water-abstraction-system](https://github.com/DEFRA/water-abstraction-system) when using our AWS environments. That all seemed to work until we changed an endpoint in **water-abstraction-system** to require a query param to be passed through. Our proxy isn't doing this so now **water-abstraction-system** errors because of the missing param. This change fixes the issue by switching to [h2o2](https://github.com/hapijs/h2o2) which "adds proxying functionality to a hapi server." When we first implemented our proxy, we did it ourselves. The issue with the query params would require us to continue to expand that functionality, and we haven't even got to `POST`, `PUT` and `DELETE` requests yet! So, before we went down that route we investigated what options were available and found this plugin. The documentation and current examples all direct you to use this if you need Hapi to act like a reverse/pass-through proxy server. We should have done this from the outset, but we've at least stopped before we expanded our own functionality further.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Dec 12, 2022
DEFRA/water-abstraction-team#54 Our status page depends on sending requests to the other running apps and 3rd party services. All is fine in our Vagrant box but when we deployed to our AWS environments the requests just seemed to hang. After some investigation inspiration hit! Our AWS environments require all requests to go through a proxy server and without support for this, our requests are getting stuck. We tackled this same problem in [Replace existing proxy config with global-agent](DEFRA/sroc-charging-module-api#572) in the sroc-charging-module-api. In the Charging Module, we used [global-agent](https://github.com/gajus/global-agent). This intercepts all http/https requests and adds the proxy details them. Perfect! The problem is it was last updated more than a year ago, the build is marked as failing, test coverage is low, and issues and PR's seem to be going unanswered. Our other issue is it does not recognise the lowercase versions of standard proxy env vars (`http_proxy`, `https_proxy` and `no_proxy`). It's a shame because it works seemlessly in the background. The folks behind the request package we do use, [Got](https://github.com/sindresorhus/got), recommend a couple. We liked [hpagent](https://github.com/delvedor/hpagent/). The example was simple, it's being regularly updated, supports [keep-alive connections](https://en.wikipedia.org/wiki/HTTP_persistent_connection), and even uses [standardjs](https://standardjs.com/)! So, this change brings in support for working behind a proxy using **hpagent**. As part of this we'll be refactoring the work of making HTTP requests into its own service. We'll also solve one of the other things we noted; the requests don't timeout if there is a problem making a connection.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Dec 16, 2022
DEFRA/water-abstraction-team#54 Bit of housekeeping. Adding a new /service-status page was one of our initial tasks with this repo, intended to help us understand just how we'd go about fitting this app into our existing service. Now we're adding more and more services it's time to do a little tidying. This moves the `/service-status` page to the `/health` path. We also move services used by the health pages into a `health/` folder.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Dec 16, 2022
DEFRA/water-abstraction-team#54 Bit of housekeeping. Adding a new /service-status page was one of our initial tasks with this repo, intended to help us understand just how we'd go about fitting this app into our existing service. Now we're adding more and more services it's time to do a little tidying. So, this change - moves the `/service-status` page to the `/health` path and renames it '/info` - moves services used by the health pages into a `health/` folder. - replaces the individual route files with a new `health.routes.js` - adds a test for our `/health/info` route (we never had one for service-status - move the `/robots.txt` route into `root.routes.js` as it is on the root!
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Dec 19, 2022
DEFRA/water-abstraction-team#54 We recently added `RequestLib` to bring all our HTTP request logic into one place. It also has support for working behind proxy services. This change updates `InfoService` to use our new module and to get requests working in our AWS environment.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this issue
Dec 19, 2022
DEFRA/water-abstraction-team#54 We recently added `RequestLib` to bring all our HTTP request logic into one place. It also has support for working behind proxy services. This change updates `InfoService` to use our new module and to get requests working in our AWS environment.
This was referenced Sep 8, 2023
|
This issue has now been resolved. The new status page can either be accessed locally from the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The current WABS UI features a
/service-statuspageThis is the ideal first 'UI' function to assimilate into water-abstraction-system.
It's not a user feature, so no one will shout if we break it! There is no authentication, we don't have to worry about that topic yet. But it does involve
Once we have those figured out, the current service page does checks that involve
By performing the same checks we'll have figured out how system will integrate with the current service whilst things slowly get assimilated.
The text was updated successfully, but these errors were encountered: