Extract table results to a file #193
Conversation
https://eaflood.atlassian.net/jira/software/c/projects/WATER/boards/96?modal=detail&selectedIssue=WATER-3964 This pull request is just a small part of a larger project that involves exporting all our database schemas, converting them into CSV files, and uploading them to our Amazon S3 bucket. Its primary focus is exporting the billing charge categories table to a file. Converting it to a CSV standard first. To expedite the export process and see the output sooner, we are using a vertical slicing approach, rather than a horizontal one, which means exporting a single table at a time from each schema.
Beckyrose200
force-pushed
the
extract-table-to-file
branch
from
d97deb3 to
47ee54c
Compare
There was a problem hiding this comment.
This is a big step towards our slice of cake!
I've added some stuff to think about for this PR. I've not gone into the tests because my suggestions will probably result in changes to them.
The top-level takeaway is we just need to review where 'work' is being done, and if there is somewhere else that might be better instead.
config/server.config.js
Outdated
| }, | ||
| temporaryFilePath: process.env.TMPDIR || '/tmp/' |
There was a problem hiding this comment.
This is typically the location of the tmp directory on a Linux-based system. But clearly, I failed to provide enough clues about the next step to research. 🤦
If OS's have default locations for where they put temporary files, is there a way we can get the OS to tell us where that place is?
There is, and it's built into Node.js
I would opt for using that, rather than having it as a config value we maintain.
There was a problem hiding this comment.
Sorted! (I think)
app/services/db-export/fetch-billing-charge-categories-column-names.service.js
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
This is a big step towards our slice of cake!
I've added some stuff to think about for this PR. I've not gone into the tests because my suggestions will probably result in changes to them.
The top-level takeaway is we just need to review where 'work' is being done, and if there is somewhere else that might be better instead.
There was a problem hiding this comment.
Great stuff! Just a few things to consider. Also a couple of notes for @Cruikshanks as there are a couple of things that could go either way since they're not currently covered by our coding standards.
| * @param {Object} columnNames - An object containing the column names. | ||
| * @returns {String} - A string representing the formatted header for the column names | ||
| */ | ||
| function _generateHeader (columnNames) { |
There was a problem hiding this comment.
More a thought than a suggestion so feel free to ignore -- once we've used Object.keys(columnNames) to get an array of column names, we're basically transforming that array into CSV just like we do with each line of the table data. So I don't know if it's worth a refactor to have a general _transformRowToCsv() function that can be used both to generate the header and to convert each line to CSV?
…criptions, and code refactor
Beckyrose200
force-pushed
the
extract-table-to-file
branch
from
e91b14f to
31e0911
Compare
Beckyrose200
force-pushed
the
extract-table-to-file
branch
from
2c4e1a5 to
73ca9ad
Compare
There was a problem hiding this comment.
I've made 2 suggestions for app/services/db-export/fetch-billing-charge-categories.service.js which will have knock effects on other services and the tests.
So, I've just focused on that service for now. Ping me if you have questions about what I've suggested.
app/services/db-export/fetch-billing-charge-categories.service.js
Outdated
Show resolved
Hide resolved
app/services/db-export/fetch-billing-charge-categories.service.js
Outdated
Show resolved
Hide resolved
| */ | ||
| async function go (data) { | ||
| try { | ||
| await fs.writeFile(_filenameWithPath('Billing Charge Categories Table Export.csv'), data) |
There was a problem hiding this comment.
I wouldn't put spaces in a filename. I'd write it more like this: billing_charge_categories_table_export.csv. Personally I think it's a good habit to get into. What do you guys think @Cruikshanks @StuAA78
Sorry @Beckyrose200
There was a problem hiding this comment.
https://eaflood.atlassian.net/browse/WATER-3964
This pull request is just a small part of a larger project that involves exporting all our database schemas, converting them into CSV files, and uploading them to our Amazon S3 bucket.
Its primary focus is exporting the billing charge categories table to a file. Converting it to a CSV standard first.
To expedite the export process and see the output sooner, we are using a vertical slicing approach, rather than a horizontal one, which means exporting a single table at a time from each schema.