Create endpoint to generate fake data #347

StuAA78 · 2023-08-09T09:50:46Z

https://eaflood.atlassian.net/browse/WATER-4070
https://eaflood.atlassian.net/browse/WATER-4073

We are looking to re-design the bill runs page in the internal service due to repeated issues with the existing one timing out. It is simply trying to return too much data when displaying annual bill runs.

To properly test the new design our UAT team hit a blocker; how do we demonstrate a large bill run in the prototype? They were able to manually create a bill run or 2. But to properly test this they needed 100's of licences represented and thousands of bills. They also had to be 'realistic' because previous testing had highlighted users got distracted by inaccuracies in the test data.

So, we have created a new feature in the app which allows the delivery team to 'mock' existing records. Currently, it only supports mocking bill runs but this change is done in a way it should be easy to add additional entity types if needed.

To mock a bill run, a team member would first identify one in the environment they are connected to that they would like to mock (the endpoint is only available in non-prod environments). They then extract the ID from the URL and make a request in their browser to

https://[non-prod-environment-domain]/system/data/mock/bill-run/96187453-7243-4f45-a6a7-d0c5f530cbac

When called in our local development environment drop /system from the URL

The system will then use the existing bill run as a template, mock things like invoice and licence holder details and obfuscate other data before returning the mocked bill run as JSON. The format returned is based on what the prototype is already using so it can be copied and pasted right in.

app/services/data/mock/generate-bill-run.service.js

https://eaflood.atlassian.net/browse/WATER-4070 https://eaflood.atlassian.net/browse/WATER-4073 Whilst working on [Create endpoint to generate fake data](#347) we added a number of formatters. These take data from the DB and re-format them to match how the UI displays the values. We believe we are going to need these formatters in future changes we'll be making, the first of which is a re-designed bill run page. So, to help prepare for that we are adding those formatters to our `BasePresenter` so they are available to any future presenters we add.

https://eaflood.atlassian.net/browse/WATER-4070 https://eaflood.atlassian.net/browse/WATER-4073 Whilst working on [Create endpoint to generate fake data](#347) we added a number of formatters. These take DB data and re-format it to match how the UI displays the values. We will need these formatters in future changes we'll be making, the first of which is a re-designed bill run page. So, to help prepare for that we are adding those formatters to our `BasePresenter` so they are available to any future presenters we add.

For prototype and test purposes we need to be able to generate realistic-looking fake data. We therefore creat an endpoint to do this.

This adds the initial version of the service that will generate our 'mock' bill run. We start with finding the billing match that matches the ID provided as well as handling what happens if there is no match.

'Cause I am lazy and can't remember what I was doing.

The service is just picking random from arrays of fixed values. We feel there is very little we need to cover so the tests just ensure that the service is 'working'.

app/services/data/mock/generate-bill-run.service.js

+  billingInvoices.forEach((billingInvoice) => {
+    const { address, name } = GenerateMockDataService.go()
+
+    billingInvoice.accountAddress = address


StuAA78

All looking good, just a few small comments (it won't let me Request Changes since it's my own PR)

StuAA78 · 2023-08-11T12:24:17Z

app/controllers/data/data.controller.js

@@ -44,6 +56,7 @@ async function tearDown (_request, h) {

 module.exports = {
  exportDb,
+  mockData: mock,


Do we want to rename our mock function to be mockData instead of aliasing it in the exports?

StuAA78 · 2023-08-11T12:26:53Z

app/presenters/data/mock-bill-run.presenter.js

+ * @module MockBillRunPresenter
+ */
+
+const { convertPenceToPounds, formatAbstractionPeriod, formatLongDate, formatNumberAsMoney } = require('../base.presenter.js')


This line is just a little too long and should probably be split up.

StuAA78 · 2023-08-11T12:30:37Z

app/presenters/data/mock-bill-run.presenter.js

+    transactionFile,
+    billRunNumber,
+    financialYear: `${fromFinancialYearEnding} to ${toFinancialYearEnding}`,
+    debit: formatNumberAsMoney(convertPenceToPounds(netTotal)),


We do formatNumberAsMoney(convertPenceToPounds()) in several places. Do we want to break it out into a function of its own, _formatPenceAsPounds() or something?

StuAA78 · 2023-08-11T12:36:19Z

app/presenters/data/mock-bill-run.presenter.js

+      abstractionPeriodStartDay: startDay,
+      abstractionPeriodStartMonth: startMonth,
+      abstractionPeriodEndDay: endDay,
+      abstractionPeriodEndMonth: endMonth,


Do we need to alias these seeing as we're passing them to a function and not using them directly? Or do we want to keep that cos if we don't alias them, our call to the function would be too big for one line?

StuAA78 · 2023-08-11T12:38:26Z

app/services/data/mock/generate-bill-run.service.js

+  return _response(billingBatch)
+}
+
+async function _fetchBillingBatch (id) {


Not 100% certain this needs to be async seeing as it returns a query, and it's the returned query that we await; but not going to quibble over it!

StuAA78 · 2023-08-11T12:43:24Z

app/services/data/mock/generate-bill-run.service.js

+const GenerateMockDataService = require('./generate-mock-data.service.js')
+const MockBillRunPresenter = require('../../../presenters/data/mock-bill-run.presenter.js')
+
+async function go (id) {


Do we need some brief docs here?

StuAA78 · 2023-08-11T12:44:54Z

app/services/data/mock/generate-bill-run.service.js

+ * @param {*} transactions
+ * @returns


I think I'd want to see us either flesh out @param with the type and @returns with some details, or remove them completely.

Cruikshanks

Paired on this with @StuAA78 . Admittedly, the later changes were done just by me so ideally would be confirmed as all good before merging.

But he's had the audacity to take time off (!!) and it would be unfair to ask the rest of the team to review this massive change cold.

So, making the executive decision to approve and merge-away! This is not a feature that is intended for use in production anyway 😬

https://eaflood.atlassian.net/browse/WATER-4152 We are currently working on the SROC two part tariff bill. The first step of the process in the UI is to review how the returns have been matched. Our superstar design team are working on an improved design for the returns review screen but to do that they need data. Enter our mock data service! We previously [Create endpoint to generate fake data](#347) which was focused on a bill run. The design team would love it if we could do the same for the data the current returns review page uses. The page relies on data in the `water.billing_volumes` table. So, first step in generating the mock data is to add that model.

https://eaflood.atlassian.net/browse/WATER-4152 We are currently working on the SROC two-part tariff bill. The first step of the process in the UI is to review how the returns have been matched. Our superstar design team are working on an improved design for the returns review screen but to do that they need data. Enter our mock data service! We previously [Create endpoint to generate fake data](#347) which was focused on a bill run. The design team would love it if we could do the same for the data the current returns review page uses. The page relies on data in the `water.billing_volumes` table. So, the first step in generating the mock data is to add that model.

https://eaflood.atlassian.net/browse/WATER-4351 The project comes with an inbuilt mechanism to configure routes to not be available in protected environments, for example, production. You can see this in `app/routes/data.routes.js`. Both `/data/seed` and `/data/tear-down` are configured to be `excludeFromProd`. How this works is during startup our `app/plugins/router.plugin.js` calls `app/services/plugins/filter-routes.service.js` which handles filtering out those routes that should not be added when running in a protected environment. To be honest, `excludeFromProd` is a bad name because it could be any environment we add to this ```javascript function _protectedEnvironment (environment) { return ['prd'].includes(environment) } ``` When this was first implemented `pre` was included so we could confirm routes were not available prior to shipping to production. However, in [Create endpoint to generate fake data](#347) we took it out so we could get this fake data generator to work. It needed to be in `pre` because it generated data based on real bill runs. Clearly we weren't on the ball because we didn't even bother to update the comments which still exist ```javascript * So, we use this service to ensure any endpoint that is still being worked on is not available when the API is * running in production. We also include pre-production in our protected environments so we can test and ensure * an endpoint does not get registered as part of our release testing and sign-off. ``` Since that time the fake data generator is no more and we need to be able to test again that routes (and therefore pages) are not accessible in production. Hence, we need to add `pre` back into our list of protected environments.

https://eaflood.atlassian.net/browse/WATER-4351 The project comes with an inbuilt mechanism to configure routes to not be available in protected environments, for example, production. You can see this in `app/routes/data.routes.js`. Both `/data/seed` and `/data/tear-down` are configured to be `excludeFromProd`. How this works is during startup our `app/plugins/router.plugin.js` calls `app/services/plugins/filter-routes.service.js` which handles filtering out those routes that should not be added when running in a protected environment. To be honest, `excludeFromProd` is a bad name because it could be any environment we add to this ```javascript function _protectedEnvironment (environment) { return ['prd'].includes(environment) } ``` When this was first implemented `pre` was included so we could confirm routes were not available before shipping to production. However, in [Create endpoint to generate fake data](#347) we took it out so we could get this fake data generator to work. It needed to be in `pre` because it generated data based on real bill runs. Clearly, we weren't on the ball because we didn't even bother to update the comments which still exist ```javascript * So, we use this service to ensure any endpoint that is still being worked on is not available when the API is * running in production. We also include pre-production in our protected environments so we can test and ensure * an endpoint does not get registered as part of our release testing and sign-off. ``` Since that time the fake data generator is no more and we need to be able to test again that routes (and therefore pages) are not accessible in production. Hence, we need to add `pre` back into our list of protected environments.

StuAA78 self-assigned this Aug 9, 2023

StuAA78 added the enhancement New feature or request label Aug 9, 2023

github-advanced-security bot found potential problems Aug 10, 2023

View reviewed changes

app/services/data/mock/generate-bill-run.service.js Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Aug 10, 2023

View reviewed changes

app/services/data/mock/generate-bill-run.service.js Fixed Show fixed Hide fixed

Cruikshanks mentioned this pull request Aug 10, 2023

Add new data formatters to base presenter #350

Merged

StuAA78 and others added 12 commits August 10, 2023 12:19

Create endpoint to generate fake data

aeae4a0

For prototype and test purposes we need to be able to generate realistic-looking fake data. We therefore creat an endpoint to do this.

Set up initial controller, routes and service

db27e6b

Add GenerateBillRunService

aec3fcc

This adds the initial version of the service that will generate our 'mock' bill run. We start with finding the billing match that matches the ID provided as well as handling what happens if there is no match.

wip

bde7218

WIP

e7eb8e0

Masking

04235e0

WIP

2fa3dc9

'Cause I am lazy and can't remember what I was doing.

Move functionality to new presenter

c42da46

Completed presenter for endpoint with tests

c59bdbe

Additional changes we'd missed to the presenter

7cb8a23

Add tests for GenerateBillRun service

af7547e

Add test for GenerateMockDataService

20c1896

The service is just picking random from arrays of fixed values. We feel there is very little we need to cover so the tests just ensure that the service is 'working'.

Cruikshanks force-pushed the fake-data-endpoint branch from a3aba94 to 20c1896 Compare August 11, 2023 11:05

github-advanced-security bot found potential problems Aug 11, 2023

View reviewed changes

Complete error handling todo's

81c85e7

Cruikshanks marked this pull request as ready for review August 11, 2023 11:34

StuAA78 commented Aug 11, 2023

View reviewed changes

Cruikshanks added 3 commits August 15, 2023 13:25

Merge branch 'main' into fake-data-endpoint

92e3bab

Merge branch 'main' into fake-data-endpoint

6a5764c

Fix up code following rebase

12ee968

Cruikshanks approved these changes Aug 15, 2023

View reviewed changes

Cruikshanks merged commit 20d1e8b into main Aug 15, 2023

Cruikshanks deleted the fake-data-endpoint branch August 15, 2023 13:34

Cruikshanks mentioned this pull request Sep 29, 2023

Add new bill run volume model #440

Merged

Cruikshanks mentioned this pull request Oct 4, 2023

Implement mock return review #447

Closed

Cruikshanks mentioned this pull request Feb 8, 2024

Add pre back into protected environments #716

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create endpoint to generate fake data #347

Create endpoint to generate fake data #347

StuAA78 commented Aug 9, 2023 •

edited by Cruikshanks

Loading

StuAA78 left a comment

StuAA78 Aug 11, 2023

StuAA78 Aug 11, 2023

StuAA78 Aug 11, 2023

StuAA78 Aug 11, 2023

StuAA78 Aug 11, 2023

StuAA78 Aug 11, 2023

StuAA78 Aug 11, 2023

Cruikshanks left a comment

Create endpoint to generate fake data #347

Create endpoint to generate fake data #347

Conversation

StuAA78 commented Aug 9, 2023 • edited by Cruikshanks Loading

StuAA78 left a comment

Choose a reason for hiding this comment

StuAA78 Aug 11, 2023

Choose a reason for hiding this comment

StuAA78 Aug 11, 2023

Choose a reason for hiding this comment

StuAA78 Aug 11, 2023

Choose a reason for hiding this comment

StuAA78 Aug 11, 2023

Choose a reason for hiding this comment

StuAA78 Aug 11, 2023

Choose a reason for hiding this comment

StuAA78 Aug 11, 2023

Choose a reason for hiding this comment

StuAA78 Aug 11, 2023

Choose a reason for hiding this comment

Cruikshanks left a comment

Choose a reason for hiding this comment

StuAA78 commented Aug 9, 2023 •

edited by Cruikshanks

Loading