diff --git a/.eslintrc b/.eslintrc index da0a550f..25775f44 100644 --- a/.eslintrc +++ b/.eslintrc @@ -4,7 +4,7 @@ "es6": true }, "parserOptions": { - "ecmaVersion": 8 + "ecmaVersion": 9 }, "extends": "eslint:recommended" } diff --git a/README.md b/README.md index b6c55f4e..47ab3b09 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # json2csv -Converts json into csv with column titles and proper line endings. +Converts JSON into CSV with column titles and proper line endings. Can be used as a module and from the command line. [![npm version][npm-badge]][npm-badge-url] @@ -9,7 +9,7 @@ Can be used as a module and from the command line. See the [CHANGELOG] for details about the latest release. -## Features +Features - Fast and lightweight - Scalable to infinitely large datasets (using stream processing) @@ -45,7 +45,9 @@ By default, the above script will get the latest release of json2csv. You can al ``` -## Command Line Interface +## Usage + +### Command Line Interface `json2csv` can be called from the command line if installed globally (using the `-g` flag). @@ -85,104 +87,308 @@ Use `-p` to show the result as a table in the console. Any option passed through the config file `-c` will be overriden if a specific flag is passed as well. For example, the fields option of the config will be overriden if the fields flag `-f` is used. -### CLI examples +For more details, you can check some of our CLI usage [examples](docs/cli-examples.md) or our [test suite](test/CLI.js). + +## Javascript module -All examples use this example [input file](https://github.com/zemirco/json2csv/blob/master/test/fixtures/json/default.json). +`json2csv` can also be use programatically from you javascript codebase. -#### Input file and specify fields +The programatic APIs take a configuration object very similar to the CLI options. All APIs take the exact same options. -```sh -$ json2csv -i input.json -f carModel,price,color -carModel,price,color -"Audi",10000,"blue" -"BMW",15000,"red" -"Mercedes",20000,"yellow" -"Porsche",30000,"green" +- `fields` - Array of Objects/Strings. Defaults to toplevel JSON attributes. See example below. +- `ndjson` - Boolean, indicates that the data is in NDJSON format. Only effective when using the streaming API and not in object mode. +- `transforms` - Array of transforms. A transform is a function that receives a data recod and returns a transformed record. Transforms are executed in order before converting the data record into a CSV row. See bellow for more details. +- `formatters` - Object where the each key is a Javascript data type and its associated value is a formatters for the given type. A formatter is a function that receives the raw js value of a given type and formats it as a valid CSV cell. Supported types are the types returned by `typeof` i.e. `undefined`, `boolean`, `number`, `bigint`, `string`, `symbol`, `function` and `object`. +- `defaultValue` - Default value to use when missing data. Defaults to `` if not specified. (Overridden by `fields[].default`) +- `delimiter` - String, delimiter of columns. Defaults to `,` if not specified. +- `eol` - String, overrides the default OS line ending (i.e. `\n` on Unix and `\r\n` on Windows). +- `header` - Boolean, determines whether or not CSV file will contain a title column. Defaults to `true` if not specified. +- `includeEmptyRows` - Boolean, includes empty rows. Defaults to `false`. +- `withBOM` - Boolean, with BOM character. Defaults to `false`. + +### Transforms + +json2csv supports transforms. A transform is a function that receives a data recod and returns a transformed record. + +#### Custom transforms + +```js +function doNothing(item) { + // apply tranformations or create new object + return transformedItem; +} +``` +or using ES6 +```js +const doNothing = (item) => { + // apply tranformations or create new object + return transformedItem; +} ``` -#### Input file, specify fields and use pretty logging +For example, let's add a line counter to our CSV, capitalize the car field and change the price to be in Ks (1000s). +```js +function addCounter() { + let counter = 1; + return (item) => ({ counter: counter++, ...item, car: item.car.toUpperCase(), price: item.price / 1000 }); +} +``` +Then you can add `addCounter()` to the `transforms` array. +The reason to wrap the actual transform in a factory function is so the counter always starts with one and you can reuse it. But it's nor strictly necessary. -```sh -$ json2csv -i input.json -f carModel,price,color -p +#### Built-in transforms + +There is a number of built-in transform provider by the library. + +```js +const { transforms: { unwind, flatten } } = require('json2csv'); ``` -![Screenshot](https://s3.amazonaws.com/zeMirco/github/json2csv/json2csv-pretty.png) +##### Unwind -#### Generating CSV containing only specific fields +The unwind transform deconstructs an array field from the input item to output a row for each element. Is's similar to MongoDB's $unwind aggregation. -```sh -$ json2csv -i input.json -f carModel,price,color -o out.csv -$ cat out.csv -carModel,price,color -"Audi",10000,"blue" -"BMW",15000,"red" -"Mercedes",20000,"yellow" -"Porsche",30000,"green" +The transform needs to be instantiated and takes an options object as arguments containing: +- `paths` - Array of String, list the paths to the fields to be unwound. It's mandatory and should not be empty. +- `blankOut` - Boolean, unwind using blank values instead of repeating data. Defaults to `false`. + +```js +// Default +unwind({ paths: ['fieldToUnwind'] }); + +// Blanking out repeated data +unwind({ paths: ['fieldToUnwind'], blankOut: true }); ``` -Same result will be obtained passing the fields config as a file. +##### Flatten +Flatten nested javascript objects into a single level object. -```sh -$ json2csv -i input.json -c fieldsConfig.json -o out.csv +The transform needs to be instantiated and takes an options object as arguments containing: +- `objects` - Boolean, whether to flatten JSON objects or not. Defaults to `true`. +- `arrays`- Boolean, whether to flatten Arrays or not. Defaults to `false`. +- `separator` - String, separator to use between nested JSON keys when flattening a field. Defaults to `.`. + +```js +// Default +flatten(); + +// Custom separator '__' +flatten({ separator: '_' }); + +// Flatten only arrays +flatten({ objects: false, arrays: true }); ``` -where the file `fieldsConfig.json` contains -```json -[ - "carModel", - "price", - "color" -] +### Formatters + +json2csv supports formatters. A formatter is a function that receives the raw js value of a given type and formats it as a valid CSV cell. Supported types are the types returned by `typeof` i.e. `undefined`, `boolean`, `number`, `bigint`, `string`, `symbol`, `function` and `object`. + +There is a special type of formatter that only applies to the CSV headers if they are present. This is the `header` formatter and by default it uses the `string` formatter. + +Pay special attention to the `string` formatter since other formatters like the `headers` or `object` formatters, rely on the `string` formatter for the stringification. + +#### Custom formatters + +```js +function formatType(itemOfType) { + // format object + return formattedItem; +} +``` +or using ES6 +```js +const formatType = (itemOfType) => { + // apply tranformations or create new object + return itemOfType; +} ``` -#### Read input from stdin +For example, let's format functions as their name or 'unkwown'. -```sh -$ json2csv -f price -[{"price":1000},{"price":2000}] +```js +const functionNameFormatter = (item) => item.name || 'unkown'; ``` -Hit Enter and afterwards CTRL + D to end reading from stdin. The terminal should show +Then you can add `{ function: functionNameFormatter }` to the `formatters` object. +A less trivial example would be to ensure that string cells never take more than 20 characters. +```js +const stringFixedFormatter = (stringLength, elipsis = '...') => (item) => item.length <= stringLength ? item : `${item.slice(0, stringLength - elipsis.length)}${elipsis}`; ``` -price -1000 -2000 + +Then you can add `{ string: stringFixedFormatter(20) }` to the `formatters` object. +Or `stringFixedFormatter(20, '')` to don't use ellipsis and just clip the text. +As with the sample transform in the previous section, the reason to wrap the actual formatter in a factory function is so it can be parameterized easily. + +Keep in mind that the above example doesn't quote or escape the string which is problematic. A more realistic example could use our built-in string formated to do the quoting and escaping like: + +```js +const { formatters: { string: defaulStringFormatter } } = require('json2csv'); + +const stringFixedFormatter = (stringLength, elipsis = '...', stringFormatter = defaulStringFormatter()) => (item) => item.length <= stringLength ? item : stringFormatter(`${item.slice(0, stringLength - elipsis.length)}${elipsis})`; ``` -#### Appending to existing CSV +#### Built-in formatters -Sometimes you want to add some additional rows with the same columns. -This is how you can do that. +There is a number of built-in transform provider by the library. -```sh -# Initial creation of csv with headings -$ json2csv -i test.json -f name,version > test.csv -# Append additional rows -$ json2csv -i test.json -f name,version --no-header >> test.csv +```js +const { formatters: { + default: defaultFormatter, + number: numberFormatter, + string: stringFormatter, + stringQuoteOnlyIfNecessary: stringQuoteOnlyIfNecessaryFormatter, + stringExcel: stringExcelFormatter, + symbol: symbolFormatter, + object: objectFormatter, +} } = require('json2csv'); ``` -## Javascript module +##### Default +Just rely on standard Javascript strignification. +This is the default formatter for `undefined`, `boolean`, `number` and `bigint` elements. -`json2csv` can also be use programatically from you javascript codebase. +It's not a factory but the formatter itself. -### Available Options +```js +{ + undefined: defaultFormatter, + boolean: defaultFormatter, + number: defaultFormatter, + bigint: defaultFormatter, +} +``` -The programatic APIs take a configuration object very equivalent to the CLI options. +##### Number +Format numbers with a fixed amount of decimals -- `fields` - Array of Objects/Strings. Defaults to toplevel JSON attributes. See example below. -- `ndjson` - Only effective on the streaming API. Indicates that data coming through the stream is NDJSON. -- `transforms` - Array of transforms to be applied to each data item. A transform is simply a function that receives a data item and returns the transformed item. -- `defaultValue` - String, default value to use when missing data. Defaults to `` if not specified. (Overridden by `fields[].default`) -- `quote` - String, quote around cell values and column names. Defaults to `"` if not specified. -- `escapedQuote` - String, the value to replace escaped quotes in strings. Defaults to 2x`quotes` (for example `""`) if not specified. -- `delimiter` - String, delimiter of columns. Defaults to `,` if not specified. -- `eol` - String, overrides the default OS line ending (i.e. `\n` on Unix and `\r\n` on Windows). -- `excelStrings` - Boolean, converts string data into normalized Excel style data. -- `header` - Boolean, determines whether or not CSV file will contain a title column. Defaults to `true` if not specified. -- `includeEmptyRows` - Boolean, includes empty rows. Defaults to `false`. -- `withBOM` - Boolean, with BOM character. Defaults to `false`. +The formatter needs to be instantiated and takes an options object as arguments containing: +- `separator` - String, separator to use between integer and decimal digits. Defaults to `.`. It's crucial that the decimal separator is not the same character as the CSV delimiter or the result CSV will be incorrect. +- `decimals` - Number, amount of decimals to keep. Defaults to all the available decimals. + +```js +{ + // 2 decimals + number: numberFormatter(), + + // 3 decimals + number: numberFormatter(3) +} +``` + +##### String + +Format strings quoting them and escaping illegal characters if needed. + +The formatter needs to be instantiated and takes an options object as arguments containing: +- `quote` - String, quote around cell values and column names. Defaults to `"`. +- `escapedQuote` - String, the value to replace escaped quotes in strings. Defaults to 2x`quotes` (for example `""`). + +This is the default for `string` elements. + +```js +{ + // Uses '"' as quote and '""' as escaped quote + string: stringFormatter(), + + // Use single quotes `'` as quotes and `''` as escaped quote + string: stringFormatter({ quote: '\'' }), + + // Never use quotes + string: stringFormatter({ quote: '' }), + + // Use '\"' as escaped quotes + string: stringFormatter({ escapedQuote: '\"' }), +} +``` + +##### String Quote Only Necessary + +The default string formatter quote all strings. This is consistent but it is not mandatory according to the CSV standard. This formatter only quote strings if they don't contain quotes (by default `"`), the CSV separator character (by default `,`) or the end-of-line (by default `\n` or `\r\n` depending on you operating system). + +The formatter needs to be instantiated and takes an options object as arguments containing: +- `quote` - String, quote around cell values and column names. Defaults to `"`. +- `escapedQuote` - String, the value to replace escaped quotes in strings. Defaults to 2x`quotes` (for example `""`). +- `eol` - String, overrides the default OS line ending (i.e. `\n` on Unix and `\r\n` on Windows). Ensure that you use the same `eol` here as in the json2csv options. + +```js +{ + // Uses '"' as quote, '""' as escaped quote and your OS eol + string: stringQuoteOnlyIfNecessaryFormatter(), + + // Use single quotes `'` as quotes, `''` as escaped quote and your OS eol + string: stringQuoteOnlyIfNecessaryFormatter({ quote: '\'' }), + + // Never use quotes + string: stringQuoteOnlyIfNecessaryFormatter({ quote: '' }), + + // Use '\"' as escaped quotes + string: stringQuoteOnlyIfNecessaryFormatter({ escapedQuote: '\"' }), + + // Use linux EOL regardless of your OS + string: stringQuoteOnlyIfNecessaryFormatter({ eol: '\n' }), +} +``` + +##### String Excel + +Converts string data into normalized Excel style data after formatting it using the given string formatter. + +The formatter needs to be instantiated and takes an options object as arguments containing: +- `stringFormatter` - Boolean, whether to flatten JSON objects or not. Defaults to our built-in `stringFormatter`. + +```js +{ + // Uses the default string formatter + string: stringExcelFormatter(), + + // Uses custom string formatter + string: stringExcelFormatter(myStringFormatter()), +} +``` + +##### Symbol + +Format the symbol as its string value and then use the given string formatter i.e. `Symbol('My Symbol')` is formatted as `"My Symbol"`. + +The formatter needs to be instantiated and takes an options object as arguments containing: +- `stringFormatter` - Boolean, whether to flatten JSON objects or not. Defaults to our built-in `stringFormatter`. + + +This is the default for `symbol` elements. + +```js +{ + // Uses the default string formatter + symbol: symbolFormatter(), + + // Uses custom string formatter + // You rarely need to this since the symbol formatter will use the string formatter that you set. + symbol: symbolFormatter(myStringFormatter()), +} +``` + +##### Object + +Format the object using `JSON.stringify` and then the given string formatter. +Some object types likes `Date` or Mongo's `ObjectId` are automatically quoted by `JSON.stringify`. This formatter, remove those quotes and uses the given string formatter for correct quoting and escaping. + +The formatter needs to be instantiated and takes an options object as arguments containing: +- `stringFormatter` - Boolean, whether to flatten JSON objects or not. Defaults to our built-in `stringFormatter`. + +This is the default for `function` and `object` elements. `functions` are formatted as empty ``. + +```js +{ + // Uses the default string formatter + object: objectFormatter(), + + // Uses custom string formatter + // You rarely need to this since the object formatter will use the string formatter that you set. + object: objectFormatter(myStringFormatter()), +} +``` ### json2csv parser (Synchronous API) @@ -222,13 +428,13 @@ Both of the methods above load the entire JSON in memory and do the whole proces ### json2csv async parser (Streaming API) -The synchronous API has the downside of loading the entire JSON array in memory and blocking javascript's event loop while processing the data. This means that your server won't be able to process more request or your UI will become irresponsive while data is being processed. For those reasons, is rarely a good reason to use it unless your data is very small or your application doesn't do anything else. +The synchronous API has the downside of loading the entire JSON array in memory and blocking javascript's event loop while processing the data. This means that your server won't be able to process more request or your UI will become irresponsive while data is being processed. For those reasons, it is rarely a good reason to use it unless your data is very small or your application doesn't do anything else. The async parser process the data as a non-blocking stream. This approach ensures a consistent memory footprint and avoid blocking javascript's event loop. Thus, it's better suited for large datasets or system with high concurrency. One very important difference between the asynchronous and the synchronous APIs is that using the asynchronous API json objects are processed one by one. In practice, this means that only the fields in the first object of the array are automatically detected and other fields are just ignored. To avoid this, it's advisable to ensure that all the objects contain exactly the same fields or provide the list of fields using the `fields` option. -The async API uses takes a second options arguments that's directly passed to the underlying streams and accept the same options as the standard [Node.js streams](https://nodejs.org/api/stream.html#stream_new_stream_duplex_options). +The async API takes a second options arguments that is directly passed to the underlying streams and accepts the same options as the standard [Node.js streams](https://nodejs.org/api/stream.html#stream_new_stream_duplex_options). Instances of `AsyncParser` expose three objects: * *input:* Which allows to push more data @@ -353,486 +559,80 @@ const json2csv = new Transform(opts, transformOpts); const processor = input.pipe(json2csv).pipe(output); ``` -### Data transforms - -json2csv supports data transforms. A transform is simply a function that receives a data item and returns the transformed item. - - -#### Custom transforms - -```js -function (item) { - // apply tranformations or create new object - return transformedItem; -} -``` -or using ES6 -```js -(item) => { - // apply tranformations or create new object - return transformedItem; -} -``` - -For example, let's add a line counter to our CSV, capitalize the car field and change the price to be in Ks (1000s). -```js -let counter = 1; -(item) => ({ counter: counter++, ...item, car: item.car.toUpperCase(), price: item.price / 1000 }); -``` - -#### Built-in transforms - -There is a number of built-in transform provider by the library. - -```js -const { transforms: { unwind, flatten } } = require('json2csv'); -``` -##### Unwind - -The unwind transform deconstructs an array field from the input item to output a row for each element. Is's similar to MongoDB's $unwind aggregation. - -The transform needs to be instantiated and takes an options object as arguments containing: -- `paths` - Array of String, list the paths to the fields to be unwound. It's mandatory and should not be empty. -- `blankOut` - Boolean, unwind using blank values instead of repeating data. Defaults to `false`. +## Upgrading -```js -// Default -unwind({ paths: ['fieldToUnwind'] }); +### Upgrading from 5.X to 6.X -// Blanking out repeated data -unwind({ paths: ['fieldToUnwind'], blankOut: true }); -``` +The CLI hasn't changed at all. -##### Flatten -Flatten nested javascript objects into a single level object. +In the javascript Javascript modules, `formatters` are introduced and the `quote`, `escapedQuote` and `excelStrings` options are removed. -The transform needs to be instantiated and takes an options object as arguments containing: -- `objects` - Boolean, whether to flatten JSON objects or not. Defaults to `true`. -- `arrays`- Boolean, whether to flatten Arrays or not. Defaults to `false`. -- `separator` - String, separator to use between nested JSON keys when flattening a field. Defaults to `.`. - -```js -// Default -flatten(); - -// Custom separator '__' -flatten({ separator: '_' }); - -// Flatten only arrays -flatten({ objects: false, arrays: true }); -``` - -### Javascript module examples - -#### Example `fields` option -```js -{ - fields: [ - // Supports pathname -> pathvalue - 'simplepath', // equivalent to {value:'simplepath'} - 'path.to.value' // also equivalent to {value:'path.to.value'} - - // Supports label -> simple path - { - label: 'some label', // Optional, column will be labeled 'path.to.something' if not defined) - value: 'path.to.something', // data.path.to.something - default: 'NULL' // default if value is not found (Optional, overrides `defaultValue` for column) - }, - - // Supports label -> derived value - { - label: 'some label', // Optional, column will be labeled with the function name or empty if the function is anonymous - value: (row, field) => row[field.label].toLowerCase() ||field.default, - default: 'NULL' // default if value function returns null or undefined - }, - - // Supports label -> derived value - { - value: (row) => row.arrayField.join(',') - }, - - // Supports label -> derived value - { - value: (row) => `"${row.arrayField.join(',')}"` - }, - ] -} -``` - -#### Example 1 +Custom `quote` and `escapedQuote` are applied by setting the properties in the `string` formatter. ```js const { Parser } = require('json2csv'); - -const myCars = [ - { - "car": "Audi", - "price": 40000, - "color": "blue" - }, { - "car": "BMW", - "price": 35000, - "color": "black" - }, { - "car": "Porsche", - "price": 60000, - "color": "green" - } -]; - -const json2csvParser = new Parser(); -const csv = json2csvParser.parse(myCars); - -console.log(csv); -``` - -will output to console - -``` -"car", "price", "color" -"Audi", 40000, "blue" -"BMW", 35000, "black" -"Porsche", 60000, "green" +const json2csvParser = new Parser({ quote: '\'', escapedQuote: '\\\'' }); +const csv = json2csvParser.parse(myData); ``` -#### Example 2 - -You can choose which fields to include in the CSV. - +should be replaced by ```js -const { Parser } = require('json2csv'); -const fields = ['car', 'color']; - -const json2csvParser = new Parser({ fields }); -const csv = json2csvParser.parse(myCars); - -console.log(csv); -``` - -will output to console - -``` -"car", "color" -"Audi", "blue" -"BMW", "black" -"Porsche", "green" +const { Parser, formatter: { string: stringFormatter } } = require('json2csv'); +const json2csvParser = new Parser({ + formatters: { + string: stringFormatter({ quote: '\'', escapedQuote: '\\\'' })), + } +}); +const csv = json2csvParser.parse(myData); ``` -#### Example 3 - -You can choose custom column names for the exported file. +`excelStrings` can be used by using the `stringExcel` formatter. ```js const { Parser } = require('json2csv'); - -const fields = [{ - label: 'Car Name', - value: 'car' -},{ - label: 'Price USD', - value: 'price' -}]; - -const json2csvParser = new Parser({ fields }); -const csv = json2csvParser.parse(myCars); - -console.log(csv); -``` - -will output to console - -``` -"Car Name", "Price USD" -"Audi", 40000 -"BMW", 35000 -"Porsche", 60000 +const json2csvParser = new Parser({ quote: '\'', escapedQuote: '\\\'', excelStrings: true }); +const csv = json2csvParser.parse(myData); ``` -#### Example 4 - -You can also specify nested properties using dot notation. - +should be replaced by ```js -const { Parser } = require('json2csv'); - -const myCars = [ - { - "car": { "make": "Audi", "model": "A3" }, - "price": 40000, - "color": "blue" - }, { - "car": { "make": "BMW", "model": "F20" }, - "price": 35000, - "color": "black" - }, { - "car": { "make": "Porsche", "model": "9PA AF1" }, - "price": 60000, - "color": "green" +const { Parser, formatter: { stringExcel: stringExcelFormatter } } = require('json2csv'); +const json2csvParser = new Parser({ + formatters: { + string: stringExcelFormatter(stringFormatter({ quote: '\'', escapedQuote: '\\\'' }))), } -]; - -const fields = ['car.make', 'car.model', 'price', 'color']; - -const json2csvParser = new Parser({ fields }); -const csv = json2csvParser.parse(myCars); - -console.log(csv); -``` - -will output to console - -``` -"car.make", "car.model", "price", "color" -"Audi", "A3", 40000, "blue" -"BMW", "F20", 35000, "black" -"Porsche", "9PA AF1", 60000, "green" -``` - -#### Example 5 - -Use a custom delimiter to create tsv files using the delimiter option: - -```js -const { Parser } = require('json2csv'); - -const json2csvParser = new Parser({ delimiter: '\t' }); -const tsv = json2csvParser.parse(myCars); - -console.log(tsv); +}); +const csv = json2csvParser.parse(myData); ``` -will output to console +### Upgrading from 4.X to 5.X -``` -"car" "price" "color" -"Audi" 10000 "blue" -"BMW" 15000 "red" -"Mercedes" 20000 "yellow" -"Porsche" 30000 "green" -``` +In the CLI, the config file option, `-c`, used to be a list of fields and now it's expected to be a full configuration object. -If no delimiter is specified, the default `,` is used. +The `stringify` option hass been removed. -#### Example 6 +`doubleQuote` has been renamed to `escapedQuote`. -You can choose custom quotation marks. +In the javascript Javascript modules, `transforms` are introduced and all the `unwind` and `flatten` -related options has been moved to their own transforms. +What used to be ```js const { Parser } = require('json2csv'); - -const json2csvParser = new Parser({ quote: '' }); -const csv = json2csvParser.parse(myCars); - -console.log(csv); -``` - -will output to console - -``` -car, price, color -Audi, 40000, blue -BMW", 35000, black -Porsche", 60000, green +const json2csvParser = new Parser({ unwind: paths, unwindBlank: true, flatten: true, flattenSeparator: '__' }); +const csv = json2csvParser.parse(myData); ``` -#### Example 7 - -You can unwind arrays similar to MongoDB's $unwind operation using the `unwind` transform. - +should be replaced by ```js -const { Parser, transforms: { unwind } } = require('json2csv'); - -const myCars = [ - { - "carModel": "Audi", - "price": 0, - "colors": ["blue","green","yellow"] - }, { - "carModel": "BMW", - "price": 15000, - "colors": ["red","blue"] - }, { - "carModel": "Mercedes", - "price": 20000, - "colors": "yellow" - }, { - "carModel": "Porsche", - "price": 30000, - "colors": ["green","teal","aqua"] - } -]; - -const fields = ['carModel', 'price', 'colors']; -const transforms = [unwind({ paths: ['colors'] })]; - -const json2csvParser = new Parser({ fields, transforms }); -const csv = json2csvParser.parse(myCars); - -console.log(csv); -``` - -will output to console - -``` -"carModel","price","colors" -"Audi",0,"blue" -"Audi",0,"green" -"Audi",0,"yellow" -"BMW",15000,"red" -"BMW",15000,"blue" -"Mercedes",20000,"yellow" -"Porsche",30000,"green" -"Porsche",30000,"teal" -"Porsche",30000,"aqua" -``` - -#### Example 8 - -You can also unwind arrays multiple times or with nested objects. - -```js -const { Parser, transforms: { unwind } } = require('json2csv'); - -const myCars = [ - { - "carModel": "BMW", - "price": 15000, - "items": [ - { - "name": "airbag", - "color": "white" - }, { - "name": "dashboard", - "color": "black" - } - ] - }, { - "carModel": "Porsche", - "price": 30000, - "items": [ - { - "name": "airbag", - "items": [ - { - "position": "left", - "color": "white" - }, { - "position": "right", - "color": "gray" - } - ] - }, { - "name": "dashboard", - "items": [ - { - "position": "left", - "color": "gray" - }, { - "position": "right", - "color": "black" - } - ] - } - ] - } -]; - -const fields = ['carModel', 'price', 'items.name', 'items.color', 'items.items.position', 'items.items.color']; -const transforms = [unwind({ paths: ['items', 'items.items'] })]; -const json2csvParser = new Parser({ fields, transforms }); -const csv = json2csvParser.parse(myCars); - -console.log(csv); -``` - -will output to console - -``` -"carModel","price","items.name","items.color","items.items.position","items.items.color" -"BMW",15000,"airbag","white",, -"BMW",15000,"dashboard","black",, -"Porsche",30000,"airbag",,"left","white" -"Porsche",30000,"airbag",,"right","gray" -"Porsche",30000,"dashboard",,"left","gray" -"Porsche",30000,"dashboard",,"right","black" -``` - -#### Example 9 - -You can also unwind arrays blanking the repeated fields. - -```js -const { Parser, transforms: { unwind } } = require('json2csv'); - -const myCars = [ - { - "carModel": "BMW", - "price": 15000, - "items": [ - { - "name": "airbag", - "color": "white" - }, { - "name": "dashboard", - "color": "black" - } - ] - }, { - "carModel": "Porsche", - "price": 30000, - "items": [ - { - "name": "airbag", - "items": [ - { - "position": "left", - "color": "white" - }, { - "position": "right", - "color": "gray" - } - ] - }, { - "name": "dashboard", - "items": [ - { - "position": "left", - "color": "gray" - }, { - "position": "right", - "color": "black" - } - ] - } - ] - } -]; - -const fields = ['carModel', 'price', 'items.name', 'items.color', 'items.items.position', 'items.items.color']; -const transforms = [unwind({ paths: ['items', 'items.items'], blankOut: true })]; - -const json2csvParser = new Parser({ fields, transforms }); -const csv = json2csvParser.parse(myCars); - -console.log(csv); -``` - -will output to console - -``` -"carModel","price","items.name","items.color","items.items.position","items.items.color" -"BMW",15000,"airbag","white",, -,,"dashboard","black",, -"Porsche",30000,"airbag",,"left","white" -,,,,"right","gray" -,,"dashboard",,"left","gray" -,,,,"right","black" +const { Parser, transform: { unwind, flatten } } = require('json2csv'); +const json2csvParser = new Parser({ transforms: [unwind({ paths, blankOut: true }), flatten('__')] }); +const csv = json2csvParser.parse(myData); ``` -### Migrations +You can se the documentation for json2csv v4.X.X [here](https://github.com/zemirco/json2csv/blob/v4/README.md). -#### Migrating from 3.X to 4.X +### Upgrading from 3.X to 4.X What in 3.X used to be ```js @@ -856,32 +656,6 @@ const csv = json2csv.parse(myData, { fields: myFields, unwind: paths, ... }); Please note that many of the configuration parameters have been slightly renamed. Please check one by one that all your parameters are correct. You can se the documentation for json2csv 3.11.5 [here](https://github.com/zemirco/json2csv/blob/v3.11.5/README.md). -#### Migrating from 4.X to 5.X - -In the CLI, the config file option, `-c`, used to be a list of fields and now it's expected to be a full configuration object. - -The `stringify` option hass been removed. - -`doubleQuote` has been renamed to `escapedQuote`. - -The `unwind` and `flatten` -related options has been moved to their own transforms. - -What used to be -```js -const { Parser } = require('json2csv'); -const json2csvParser = new Parser({ unwind: paths, unwindBlank: true, flatten: true, flattenSeparator: '__' }); -const csv = json2csvParser.parse(myData); -``` - -should be replaced by -```js -const { Parser, transforms: { unwind, flatten } } = require('json2csv'); -const json2csvParser = new Parser({ transforms: [unwind({ paths, blankOut: true }), flatten('__')] }); -const csv = json2csvParser.parse(myData); -``` - -You can se the documentation for json2csv v4.X.X [here](https://github.com/zemirco/json2csv/blob/v4/README.md). - ## Known Gotchas ### Excel support @@ -917,7 +691,17 @@ Excel can display Unicode correctly (just setting the `withBOM` option to true). PowerShell do some estrange double quote escaping escaping which results on each line of the CSV missing the first and last quote if outputting the result directly to stdout. Instead of that, it's advisable that you write the result directly to a file. -## Building +## Development + +### Pulling the repo + +After you clone the repository you just need to install the required packages for development by runnning following command under json2csv dir. + +```sh +$ npm install +``` + +### Building json2csv is packaged using `rollup`. You can generate the packages running: @@ -932,7 +716,7 @@ which generates 3 files under the `dist folder`: When you use packaging tools like webpack and such, they know which version to use depending on your configuration. -## Testing +### Linting & Testing Run the folowing command to check the code style. @@ -946,15 +730,9 @@ Run the following command to run the tests and return coverage $ npm run test-with-coverage ``` -## Contributors - -After you clone the repository you just need to install the required packages for development by runnning following command under json2csv dir. - -```sh -$ npm install -``` +### Contributing changes -Before making any pull request please ensure sure that your code is formatted, test are passing and test coverage haven't decreased. (See [Testing](#testing)) +Before making any pull request please ensure sure that your code is formatted, test are passing and test coverage haven't decreased. ## License diff --git a/bin/json2csv.js b/bin/json2csv.js index c41aa37f..da472f66 100755 --- a/bin/json2csv.js +++ b/bin/json2csv.js @@ -16,6 +16,7 @@ const readFile = promisify(readFileOrig); const writeFile = promisify(writeFileOrig); const { unwind, flatten } = json2csv.transforms; +const { string: stringFormatterCtr, stringExcel: stringExcelFormatter } = json2csv.formatters; const JSON2CSVParser = json2csv.Parser; const Json2csvTransform = json2csv.Transform; @@ -149,18 +150,24 @@ async function processStream(config, opts) { separator: config.flattenSeparator })); } + + const stringFormatter = stringFormatterCtr({ + quote: config.quote, + escapedQuote: config.escapedQuote, + }); + const formatters = { + string: config.excelStrings ? stringExcelFormatter({ stringFormatter }) : stringFormatter + }; const opts = { transforms, + formatters, fields: config.fields ? (Array.isArray(config.fields) ? config.fields : config.fields.split(',')) : config.fields, defaultValue: config.defaultValue, - quote: config.quote, - escapedQuote: config.escapedQuote, delimiter: config.delimiter, eol: config.eol, - excelStrings: config.excelStrings, header: config.header, includeEmptyRows: config.includeEmptyRows, withBOM: config.withBom diff --git a/docs/cli-examples.md b/docs/cli-examples.md new file mode 100644 index 00000000..5379931a --- /dev/null +++ b/docs/cli-examples.md @@ -0,0 +1,77 @@ +# CLI examples + +All examples use this example [input file](https://github.com/zemirco/json2csv/blob/master/test/fixtures/json/default.json). + +## Input file and specify fields + +```sh +$ json2csv -i input.json -f carModel,price,color +carModel,price,color +"Audi",10000,"blue" +"BMW",15000,"red" +"Mercedes",20000,"yellow" +"Porsche",30000,"green" +``` + +## Input file, specify fields and use pretty logging + +```sh +$ json2csv -i input.json -f carModel,price,color -p +``` + +![Screenshot](https://s3.amazonaws.com/zeMirco/github/json2csv/json2csv-pretty.png) + +## Generating CSV containing only specific fields + +```sh +$ json2csv -i input.json -f carModel,price,color -o out.csv +$ cat out.csv +carModel,price,color +"Audi",10000,"blue" +"BMW",15000,"red" +"Mercedes",20000,"yellow" +"Porsche",30000,"green" +``` + +Same result will be obtained passing the fields config as a file. + +```sh +$ json2csv -i input.json -c fieldsConfig.json -o out.csv +``` + +where the file `fieldsConfig.json` contains + +```json +[ + "carModel", + "price", + "color" +] +``` + +## Read input from stdin + +```sh +$ json2csv -f price +[{"price":1000},{"price":2000}] +``` + +Hit Enter and afterwards CTRL + D to end reading from stdin. The terminal should show + +```sh +price +1000 +2000 +``` + +## Appending to existing CSV + +Sometimes you want to add some additional rows with the same columns. +This is how you can do that. + +```sh +# Initial creation of csv with headings +$ json2csv -i test.json -f name,version > test.csv +# Append additional rows +$ json2csv -i test.json -f name,version --no-header >> test.csv +``` diff --git a/docs/parser-examples.md b/docs/parser-examples.md new file mode 100644 index 00000000..b3a6cc5f --- /dev/null +++ b/docs/parser-examples.md @@ -0,0 +1,439 @@ +# Javascript module examples + +Most of the examples in this section use the same input data: + +```js +const myCars = [ + { + "car": "Audi", + "price": 40000, + "color": "blue" + }, { + "car": "BMW", + "price": 35000, + "color": "black" + }, { + "car": "Porsche", + "price": 60000, + "color": "green" + } +]; +``` + +## Example `fields` option +```js +{ + fields: [ + // Supports pathname -> pathvalue + 'simplepath', // equivalent to {value:'simplepath'} + 'path.to.value' // also equivalent to {value:'path.to.value'} + + // Supports label -> simple path + { + label: 'some label', // Optional, column will be labeled 'path.to.something' if not defined) + value: 'path.to.something', // data.path.to.something + default: 'NULL' // default if value is not found (Optional, overrides `defaultValue` for column) + }, + + // Supports label -> derived value + { + label: 'some label', // Optional, column will be labeled with the function name or empty if the function is anonymous + value: (row, field) => row[field.label].toLowerCase() ||field.default, + default: 'NULL' // default if value function returns null or undefined + }, + + // Supports label -> derived value + { + value: (row) => row.arrayField.join(',') + }, + + // Supports label -> derived value + { + value: (row) => `"${row.arrayField.join(',')}"` + }, + ] +} +``` + +## Default parsing + +```js +const { Parser } = require('json2csv'); + +const json2csvParser = new Parser(); +const csv = json2csvParser.parse(myCars); + +console.log(csv); +``` + +will output to console + +``` +"car","price","color" +"Audi",40000,"blue" +"BMW",35000,"black" +"Porsche",60000,"green" +``` + +## Specify fields to parse + +```js +const { Parser } = require('json2csv'); +const fields = ['car', 'color']; + +const json2csvParser = new Parser({ fields }); +const csv = json2csvParser.parse(myCars); + +console.log(csv); +``` + +will output to console + +``` +"car","color" +"Audi","blue" +"BMW","black" +"Porsche","green" +``` + +## Use custom headers + +```js +const { Parser } = require('json2csv'); + +const fields = [{ + label: 'Car Name', + value: 'car' +},{ + label: 'Price USD', + value: 'price' +}]; + +const json2csvParser = new Parser({ fields }); +const csv = json2csvParser.parse(myCars); + +console.log(csv); +``` + +will output to console + +``` +"Car Name","Price USD" +"Audi",40000 +"BMW",35000 +"Porsche",60000 +``` + +## Parse nested properties + +You can specify nested properties using dot notation. + +```js +const { Parser } = require('json2csv'); + +const myCars = [ + { + "car": { "make": "Audi", "model": "A3" }, + "price": 40000, + "color": "blue" + }, { + "car": { "make": "BMW", "model": "F20" }, + "price": 35000, + "color": "black" + }, { + "car": { "make": "Porsche", "model": "9PA AF1" }, + "price": 60000, + "color": "green" + } +]; + +const fields = ['car.make', 'car.model', 'price', 'color']; + +const json2csvParser = new Parser({ fields }); +const csv = json2csvParser.parse(myCars); + +console.log(csv); +``` + +will output to console + +``` +"car.make", "car.model", "price", "color" +"Audi", "A3", 40000, "blue" +"BMW", "F20", 35000, "black" +"Porsche", "9PA AF1", 60000, "green" +``` + +## Use a custom delimiter + +For example, to create tsv files + +```js +const { Parser } = require('json2csv'); + +const json2csvParser = new Parser({ delimiter: '\t' }); +const tsv = json2csvParser.parse(myCars); + +console.log(tsv); +``` + +will output to console + +``` +"car" "price" "color" +"Audi" 10000 "blue" +"BMW" 15000 "red" +"Mercedes" 20000 "yellow" +"Porsche" 30000 "green" +``` + +If no delimiter is specified, the default `,` is used. + +## Use custom formatting + +For example, you could use `*` as quotes and format numbers to always have 2 decimals and use `,` as separator. +To avoid conflict between the number separator and the CSV delimiter, we can use a custom delimiter again. + +```js +const { Parser, formatters: {string: stringFormatter, number: numberFormatter } } = require('json2csv'); + +const json2csvParser = new Parser({ + delimiter: ';', + formatters: { + string: stringFormatter({ quote: '*' }), + number: numberFormatter({ separator: ',', decimals: 2 }), + }, +}); +const csv = json2csvParser.parse(myCars); + +console.log(csv); +``` + +will output to console + +``` +*car*;*price*;*color* +*Audi*;40000,00;*blue* +*BMW*;35000,00;*black* +*Porsche*;60000,00;*green* +``` + +## Format the headers differently + +For example, you can not quote the headers. + +```js +const { Parser } = require('json2csv'); + +const json2csvParser = new Parser({ + formatters: { + header: stringFormatter({ quote: '' }, + }, +}); +const csv = json2csvParser.parse(myCars); + +console.log(csv); +``` + +will output to console + +``` +car, price, color +"Audi",40000,"blue" +"BMW",35000,"black" +"Porsche",60000,"green" +``` + +## Unwind arrays + +You can unwind arrays similar to MongoDB's $unwind operation using the `unwind` transform. + +```js +const { Parser, transforms: { unwind } } = require('json2csv'); + +const myCars = [ + { + "carModel": "Audi", + "price": 0, + "colors": ["blue","green","yellow"] + }, { + "carModel": "BMW", + "price": 15000, + "colors": ["red","blue"] + }, { + "carModel": "Mercedes", + "price": 20000, + "colors": "yellow" + }, { + "carModel": "Porsche", + "price": 30000, + "colors": ["green","teal","aqua"] + } +]; + +const fields = ['carModel', 'price', 'colors']; +const transforms = [unwind({ paths: ['colors'] })]; + +const json2csvParser = new Parser({ fields, transforms }); +const csv = json2csvParser.parse(myCars); + +console.log(csv); +``` + +will output to console + +``` +"carModel","price","colors" +"Audi",0,"blue" +"Audi",0,"green" +"Audi",0,"yellow" +"BMW",15000,"red" +"BMW",15000,"blue" +"Mercedes",20000,"yellow" +"Porsche",30000,"green" +"Porsche",30000,"teal" +"Porsche",30000,"aqua" +``` + +## Unwind of nested arrays + +You can also unwind arrays multiple times or with nested objects. + +```js +const { Parser, transforms: { unwind } } = require('json2csv'); + +const myCars = [ + { + "carModel": "BMW", + "price": 15000, + "items": [ + { + "name": "airbag", + "color": "white" + }, { + "name": "dashboard", + "color": "black" + } + ] + }, { + "carModel": "Porsche", + "price": 30000, + "items": [ + { + "name": "airbag", + "items": [ + { + "position": "left", + "color": "white" + }, { + "position": "right", + "color": "gray" + } + ] + }, { + "name": "dashboard", + "items": [ + { + "position": "left", + "color": "gray" + }, { + "position": "right", + "color": "black" + } + ] + } + ] + } +]; + +const fields = ['carModel', 'price', 'items.name', 'items.color', 'items.items.position', 'items.items.color']; +const transforms = [unwind({ paths: ['items', 'items.items'] })]; +const json2csvParser = new Parser({ fields, transforms }); +const csv = json2csvParser.parse(myCars); + +console.log(csv); +``` + +will output to console + +``` +"carModel","price","items.name","items.color","items.items.position","items.items.color" +"BMW",15000,"airbag","white",, +"BMW",15000,"dashboard","black",, +"Porsche",30000,"airbag",,"left","white" +"Porsche",30000,"airbag",,"right","gray" +"Porsche",30000,"dashboard",,"left","gray" +"Porsche",30000,"dashboard",,"right","black" +``` + +## Unwind array blanking the repeated fields + +You can also unwind arrays blanking the repeated fields. + +```js +const { Parser, transforms: { unwind } } = require('json2csv'); + +const myCars = [ + { + "carModel": "BMW", + "price": 15000, + "items": [ + { + "name": "airbag", + "color": "white" + }, { + "name": "dashboard", + "color": "black" + } + ] + }, { + "carModel": "Porsche", + "price": 30000, + "items": [ + { + "name": "airbag", + "items": [ + { + "position": "left", + "color": "white" + }, { + "position": "right", + "color": "gray" + } + ] + }, { + "name": "dashboard", + "items": [ + { + "position": "left", + "color": "gray" + }, { + "position": "right", + "color": "black" + } + ] + } + ] + } +]; + +const fields = ['carModel', 'price', 'items.name', 'items.color', 'items.items.position', 'items.items.color']; +const transforms = [unwind({ paths: ['items', 'items.items'], blankOut: true })]; + +const json2csvParser = new Parser({ fields, transforms }); +const csv = json2csvParser.parse(myCars); + +console.log(csv); +``` + +will output to console + +``` +"carModel","price","items.name","items.color","items.items.position","items.items.color" +"BMW",15000,"airbag","white",, +,,"dashboard","black",, +"Porsche",30000,"airbag",,"left","white" +,,,,"right","gray" +,,"dashboard",,"left","gray" +,,,,"right","black" +``` diff --git a/lib/JSON2CSVBase.js b/lib/JSON2CSVBase.js index 708054ef..a6a323fa 100644 --- a/lib/JSON2CSVBase.js +++ b/lib/JSON2CSVBase.js @@ -3,6 +3,11 @@ const os = require('os'); const lodashGet = require('lodash.get'); const { getProp, fastJoin, flattenReducer } = require('./utils'); +const defaultFormatter = require('./formatters/default'); +const numberFormatterCtor = require('./formatters/number') +const stringFormatterCtor = require('./formatters/string'); +const symbolFormatterCtor = require('./formatters/symbol'); +const objectFormatterCtor = require('./formatters/object'); class JSON2CSVBase { constructor(opts) { @@ -20,14 +25,28 @@ class JSON2CSVBase { processedOpts.transforms = !Array.isArray(processedOpts.transforms) ? (processedOpts.transforms ? [processedOpts.transforms] : []) : processedOpts.transforms + + const stringFormatter = (processedOpts.formatters && processedOpts.formatters['string']) || stringFormatterCtor(); + const objectFormatter = objectFormatterCtor({ stringFormatter }); + const defaultFormatters = { + header: stringFormatter, + undefined: defaultFormatter, + boolean: defaultFormatter, + number: numberFormatterCtor(), + bigint: defaultFormatter, + string: stringFormatter, + symbol: symbolFormatterCtor({ stringFormatter }), + function: objectFormatter, + object: objectFormatter + }; + + processedOpts.formatters = { + ...defaultFormatters, + ...processedOpts.formatters, + }; + processedOpts.delimiter = processedOpts.delimiter || ','; processedOpts.eol = processedOpts.eol || os.EOL; - processedOpts.quote = typeof processedOpts.quote === 'string' - ? processedOpts.quote - : '"'; - processedOpts.escapedQuote = typeof processedOpts.escapedQuote === 'string' - ? processedOpts.escapedQuote - : `${processedOpts.quote}${processedOpts.quote}`; processedOpts.header = processedOpts.header !== false; processedOpts.includeEmptyRows = processedOpts.includeEmptyRows || false; processedOpts.withBOM = processedOpts.withBOM || false; @@ -93,7 +112,7 @@ class JSON2CSVBase { */ getHeader() { return fastJoin( - this.opts.fields.map(fieldInfo => this.processValue(fieldInfo.label)), + this.opts.fields.map(fieldInfo => this.opts.formatters.header(fieldInfo.label)), this.opts.delimiter ); } @@ -122,7 +141,7 @@ class JSON2CSVBase { const processedRow = this.opts.fields.map(fieldInfo => this.processCell(row, fieldInfo)); - if (!this.opts.includeEmptyRows && processedRow.every(field => field === undefined)) { + if (!this.opts.includeEmptyRows && processedRow.every(field => field === '')) { return undefined; } @@ -150,36 +169,7 @@ class JSON2CSVBase { * @returns {String} Value stringified and processed */ processValue(value) { - if (value === null || value === undefined) { - return undefined; - } - - const valueType = typeof value; - if (valueType !== 'boolean' && valueType !== 'number' && valueType !== 'string') { - value = JSON.stringify(value); - - if (value === undefined) { - return undefined; - } - - if (value[0] === '"') { - value = value.replace(/^"(.+)"$/,'$1'); - } - } - - if (typeof value === 'string') { - if(value.includes(this.opts.quote)) { - value = value.replace(new RegExp(this.opts.quote, 'g'), this.opts.escapedQuote); - } - - value = `${this.opts.quote}${value}${this.opts.quote}`; - - if (this.opts.excelStrings) { - value = `"="${value}""`; - } - } - - return value; + return this.opts.formatters[typeof value](value); } } diff --git a/lib/formatters/default.js b/lib/formatters/default.js new file mode 100644 index 00000000..4cf925eb --- /dev/null +++ b/lib/formatters/default.js @@ -0,0 +1,7 @@ +function defaultFormatter(value) { + if (value === null || value === undefined) return ''; + + return `${value}`; +} + +module.exports = defaultFormatter; diff --git a/lib/formatters/number.js b/lib/formatters/number.js new file mode 100644 index 00000000..98b81ad9 --- /dev/null +++ b/lib/formatters/number.js @@ -0,0 +1,26 @@ +function toFixedDecimals(value, decimals) { + return value.toFixed(decimals); +} + +function replaceSeparator(value, separator) { + return value.replace('.', separator); +} + + +function numberFormatter(opts = {}) { + if (opts.separator) { + if (opts.decimals) { + return (value) => replaceSeparator(toFixedDecimals(value, opts.decimals), opts.separator); + } + + return (value) => replaceSeparator(value.toString(), opts.separator); + } + + if (opts.decimals) { + return (value) => toFixedDecimals(value, opts.decimals); + } + + return (value) => value.toString(); +} + +module.exports = numberFormatter; diff --git a/lib/formatters/object.js b/lib/formatters/object.js new file mode 100644 index 00000000..4e3af224 --- /dev/null +++ b/lib/formatters/object.js @@ -0,0 +1,17 @@ +const defaulStringFormatter = require('./string'); + +function objectFormatter(opts = { stringFormatter: defaulStringFormatter() }) { + return (value) => { + if (value === null) return ''; + + value = JSON.stringify(value); + + if (value === undefined) return ''; + + if (value[0] === '"') value = value.replace(/^"(.+)"$/,'$1'); + + return opts.stringFormatter(value); + } +} + +module.exports = objectFormatter; diff --git a/lib/formatters/string.js b/lib/formatters/string.js new file mode 100644 index 00000000..98a59690 --- /dev/null +++ b/lib/formatters/string.js @@ -0,0 +1,18 @@ +function stringFormatter(opts = {}) { + const quote = typeof opts.quote === 'string' ? opts.quote : '"'; + const escapedQuote = typeof opts.escapedQuote === 'string' ? opts.escapedQuote : `${quote}${quote}`; + + if (!quote) { + return (value) => value; + } + + return (value) => { + if(value.includes(quote)) { + value = value.replace(new RegExp(quote, 'g'), escapedQuote); + } + + return `${quote}${value}${quote}`; + } +} + +module.exports = stringFormatter; diff --git a/lib/formatters/stringExcel.js b/lib/formatters/stringExcel.js new file mode 100644 index 00000000..3b59bb20 --- /dev/null +++ b/lib/formatters/stringExcel.js @@ -0,0 +1,7 @@ +const defaulStringFormatter = require('./string'); + +function stringExcel(opts = { stringFormatter: defaulStringFormatter() }) { + return (value) => `"="${opts.stringFormatter(value)}""`; +} + +module.exports = stringExcel; diff --git a/lib/formatters/stringQuoteOnlyIfNecessary.js b/lib/formatters/stringQuoteOnlyIfNecessary.js new file mode 100644 index 00000000..234d193a --- /dev/null +++ b/lib/formatters/stringQuoteOnlyIfNecessary.js @@ -0,0 +1,21 @@ +const os = require('os'); +const defaulStringFormatter = require('./string'); + +function stringQuoteOnlyIfNecessaryFormatter(opts = {}) { + const quote = typeof opts.quote === 'string' ? opts.quote : '"'; + const escapedQuote = typeof opts.escapedQuote === 'string' ? opts.escapedQuote : `${quote}${quote}`; + const separator = typeof opts.separator === 'string' ? opts.separator : ','; + const eol = typeof opts.eol === 'string' ? opts.escapedQeoluote : os.EOL; + + const stringFormatter = defaulStringFormatter({ quote, escapedQuote }); + + return (value) => { + if([quote, separator, eol].some(char => value.includes(char))) { + return stringFormatter(value); + } + + return value; + } +} + +module.exports = stringQuoteOnlyIfNecessaryFormatter; diff --git a/lib/formatters/symbol.js b/lib/formatters/symbol.js new file mode 100644 index 00000000..85105b36 --- /dev/null +++ b/lib/formatters/symbol.js @@ -0,0 +1,7 @@ +const defaulStringFormatter = require('./string'); + +function symbolFormatter(opts = { stringFormatter: defaulStringFormatter() }) { + return (value) => opts.stringFormatter((value.toString().slice(7,-1))); +} + +module.exports = symbolFormatter; diff --git a/lib/json2csv.js b/lib/json2csv.js index 7fe5ba4f..9a5fc002 100644 --- a/lib/json2csv.js +++ b/lib/json2csv.js @@ -4,9 +4,20 @@ const { Readable } = require('stream'); const JSON2CSVParser = require('./JSON2CSVParser'); const JSON2CSVAsyncParser = require('./JSON2CSVAsyncParser'); const JSON2CSVTransform = require('./JSON2CSVTransform'); + +// Transforms const flatten = require('./transforms/flatten'); const unwind = require('./transforms/unwind'); +// Formatters +const defaultFormatter = require('./formatters/default'); +const number = require('./formatters/number'); +const string = require('./formatters/string'); +const stringQuoteOnlyIfNecessary = require('./formatters/stringQuoteOnlyIfNecessary'); +const stringExcel = require('./formatters/stringExcel'); +const symbol = require('./formatters/symbol'); +const object = require('./formatters/object'); + module.exports.Parser = JSON2CSVParser; module.exports.AsyncParser = JSON2CSVAsyncParser; module.exports.Transform = JSON2CSVTransform; @@ -41,4 +52,14 @@ module.exports.parseAsync = (data, opts, transformOpts) => { module.exports.transforms = { flatten, unwind, -}; \ No newline at end of file +}; + +module.exports.formatters = { + default: defaultFormatter, + number, + string, + stringQuoteOnlyIfNecessary, + stringExcel, + symbol, + object, +}; diff --git a/lib/utils.js b/lib/utils.js index de22b554..0a473224 100644 --- a/lib/utils.js +++ b/lib/utils.js @@ -25,10 +25,6 @@ function flattenReducer(acc, arr) { function fastJoin(arr, separator) { let isFirst = true; return arr.reduce((acc, elem) => { - if (elem === null || elem === undefined) { - elem = ''; - } - if (isFirst) { isFirst = false; return `${elem}`; diff --git a/test/CLI.js b/test/CLI.js index 83cd7a3c..7a5532df 100644 --- a/test/CLI.js +++ b/test/CLI.js @@ -48,7 +48,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { }); testRunner.add('should handle ndjson', (t) => { - const opts = '--fields carModel,price,color,transmission --ndjson'; + const opts = '--fields carModel,price,color,manual --ndjson'; exec(`${cli} -i "${getFixturePath('/json/ndjson.json')}" ${opts}`, (err, stdout, stderr) => { t.notOk(stderr); @@ -59,7 +59,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { }); testRunner.add('should error on invalid ndjson input path without streaming', (t) => { - const opts = '--fields carModel,price,color,transmission --ndjson --no-streaming'; + const opts = '--fields carModel,price,color,manual --ndjson --no-streaming'; exec(`${cli} -i "${getFixturePath('/json2/ndjsonInvalid.json')}" ${opts}`, (err, stdout, stderr) => { t.ok(stderr.includes('Invalid input file.')); @@ -68,7 +68,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { }); testRunner.add('should error on invalid ndjson input data', (t) => { - const opts = '--fields carModel,price,color,transmission --ndjson'; + const opts = '--fields carModel,price,color,manual --ndjson'; exec(`${cli} -i "${getFixturePath('/json/ndjsonInvalid.json')}" ${opts}`, (err, stdout, stderr) => { t.ok(stderr.includes('Invalid JSON')); @@ -77,7 +77,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { }); testRunner.add('should handle ndjson without streaming', (t) => { - const opts = '--fields carModel,price,color,transmission --ndjson --no-streaming'; + const opts = '--fields carModel,price,color,manual --ndjson --no-streaming'; exec(`${cli} -i "${getFixturePath('/json/ndjson.json')}" ${opts}`, (err, stdout, stderr) => { t.notOk(stderr); @@ -174,7 +174,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { }); testRunner.add('should parse json to csv using custom fields', (t) => { - const opts = '--fields carModel,price,color,transmission'; + const opts = '--fields carModel,price,color,manual'; exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, (err, stdout, stderr) => { t.notOk(stderr); @@ -307,92 +307,6 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { }); }); - // Quote - - testRunner.add('should use a custom quote when \'quote\' property is present', (t) => { - const opts = '--fields carModel,price --quote "\'"'; - - exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.withSimpleQuotes); - t.end(); - }); - }); - - testRunner.add('should be able to don\'t output quotes when setting \'quote\' to empty string', (t) => { - const opts = '--fields carModel,price --quote ""'; - - exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.withoutQuotes); - t.end(); - }); - }); - - testRunner.add('should escape quotes when setting \'quote\' property is present', (t) => { - const opts = '--fields carModel,color --quote "\'"'; - - exec(`${cli} -i "${getFixturePath('/json/escapeCustomQuotes.json')}" ${opts}`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.escapeCustomQuotes); - t.end(); - }); - }); - - testRunner.add('should not escape \'"\' when setting \'quote\' set to something else', (t) => { - const opts = '--quote "\'"'; - - exec(`${cli} -i "${getFixturePath('/json/escapedQuotes.json')}" ${opts}`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.escapedQuotesUnescaped); - t.end(); - }); - }); - - // Escaped Quote - - testRunner.add('should escape quotes with double quotes', (t) => { - exec(`${cli} -i "${getFixturePath('/json/quotes.json')}"`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.quotes); - t.end(); - }); - }); - - testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end', (t) => { - exec(`${cli} -i "${getFixturePath('/json/backslashAtEnd.json')}"`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.backslashAtEnd); - t.end(); - }); - }); - - testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end, and its not the last column', (t) => { - exec(`${cli} -i "${getFixturePath('/json/backslashAtEndInMiddleColumn.json')}"`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.backslashAtEndInMiddleColumn); - t.end(); - }); - }); - - testRunner.add('should escape quotes with value in \'escapedQuote\'', (t) => { - const opts = '--fields "a string" --escaped-quote "*"'; - - exec(`${cli} -i "${getFixturePath('/json/escapedQuotes.json')}" ${opts}`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.escapedQuotes); - t.end(); - }); - }); - // Delimiter testRunner.add('should use a custom delimiter when \'delimiter\' property is defined', (t) => { @@ -430,69 +344,10 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { }); }); - // Excell - - testRunner.add('should format strings to force excel to view the values as strings', (t) => { - const opts = '--fields carModel,price,color --excel-strings'; - - exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.excelStrings); - t.end(); - }); - }); - - // Escaping and preserving values - - testRunner.add('should parse JSON values with trailing backslashes', (t) => { - const opts = '--fields carModel,price,color'; - - exec(`${cli} -i "${getFixturePath('/json/trailingBackslash.json')}" ${opts}`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.trailingBackslash); - t.end(); - }); - }); - - testRunner.add('should escape " when preceeded by \\', (t) => { - exec(`${cli} -i "${getFixturePath('/json/escapeDoubleBackslashedEscapedQuote.json')}"`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.escapeDoubleBackslashedEscapedQuote); - t.end(); - }); - }); - - testRunner.add('should preserve new lines in values', (t) => { - const opts = '--eol "\r\n"'; - - exec(`${cli} -i "${getFixturePath('/json/escapeEOL.json')}" ${opts}`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, [ - '"a string"', - '"with a \u2028description\\n and\na new line"', - '"with a \u2029\u2028description and\r\nanother new line"' - ].join('\r\n')); - t.end(); - }); - }); - - testRunner.add('should preserve tabs in values', (t) => { - exec(`${cli} -i "${getFixturePath('/json/escapeTab.json')}"`, (err, stdout, stderr) => { - t.notOk(stderr); - const csv = stdout; - t.equal(csv, csvFixtures.escapeTab); - t.end(); - }); - }); - // Header testRunner.add('should parse json to csv without column title', (t) => { - const opts = '--fields carModel,price,color,transmission --no-header'; + const opts = '--fields carModel,price,color,manual --no-header'; exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, (err, stdout, stderr) => { t.notOk(stderr); @@ -555,7 +410,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { // BOM testRunner.add('should add BOM character', (t) => { - const opts = '--fields carModel,price,color,transmission --with-bom'; + const opts = '--fields carModel,price,color,manual --with-bom'; exec(`${cli} -i "${getFixturePath('/json/specialCharacters.json')}" ${opts}`, (err, stdout, stderr) => { t.notOk(stderr); @@ -630,7 +485,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { testRunner.add('should output to file', (t) => { const outputPath = getFixturePath('/results/default.csv'); - const opts = `-o "${outputPath}" --fields carModel,price,color,transmission`; + const opts = `-o "${outputPath}" --fields carModel,price,color,manual`; exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, async (err, stdout, stderr) => { t.notOk(stderr); @@ -647,7 +502,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { testRunner.add('should output to file without streaming', (t) => { const outputPath = getFixturePath('/results/default.csv'); - const opts = `-o ${outputPath} --fields carModel,price,color,transmission --no-streaming`; + const opts = `-o ${outputPath} --fields carModel,price,color,manual --no-streaming`; exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, async (err, stdout, stderr) => { t.notOk(stderr); @@ -663,7 +518,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { testRunner.add('should error on invalid output file path', (t) => { const outputPath = getFixturePath('/results2/default.csv'); - const opts = `-o "${outputPath}" --fields carModel,price,color,transmission`; + const opts = `-o "${outputPath}" --fields carModel,price,color,manual`; exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, (err, stdout, stderr) => { t.ok(stderr.includes('Invalid output file.')); @@ -673,7 +528,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { testRunner.add('should error on invalid output file path without streaming', (t) => { const outputPath = getFixturePath('/results2/default.csv'); - const opts = `-o "${outputPath}" --fields carModel,price,color,transmission --no-streaming`; + const opts = `-o "${outputPath}" --fields carModel,price,color,manual --no-streaming`; exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, (err, stdout, stderr) => { t.ok(stderr.includes('Invalid output file.')); @@ -830,4 +685,152 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { t.end(); }); }); + + // Formatters + + + // String Quote + + testRunner.add('should use a custom quote when \'quote\' property is present', (t) => { + const opts = '--fields carModel,price --quote "\'"'; + + exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.withSimpleQuotes); + t.end(); + }); + }); + + testRunner.add('should be able to don\'t output quotes when setting \'quote\' to empty string', (t) => { + const opts = '--fields carModel,price --quote ""'; + + exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.withoutQuotes); + t.end(); + }); + }); + + testRunner.add('should escape quotes when setting \'quote\' property is present', (t) => { + const opts = '--fields carModel,color --quote "\'"'; + + exec(`${cli} -i "${getFixturePath('/json/escapeCustomQuotes.json')}" ${opts}`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.escapeCustomQuotes); + t.end(); + }); + }); + + testRunner.add('should not escape \'"\' when setting \'quote\' set to something else', (t) => { + const opts = '--quote "\'"'; + + exec(`${cli} -i "${getFixturePath('/json/escapedQuotes.json')}" ${opts}`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.escapedQuotesUnescaped); + t.end(); + }); + }); + + // String Escaped Quote + + testRunner.add('should escape quotes with double quotes', (t) => { + exec(`${cli} -i "${getFixturePath('/json/quotes.json')}"`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.quotes); + t.end(); + }); + }); + + testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end', (t) => { + exec(`${cli} -i "${getFixturePath('/json/backslashAtEnd.json')}"`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.backslashAtEnd); + t.end(); + }); + }); + + testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end, and its not the last column', (t) => { + exec(`${cli} -i "${getFixturePath('/json/backslashAtEndInMiddleColumn.json')}"`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.backslashAtEndInMiddleColumn); + t.end(); + }); + }); + + testRunner.add('should escape quotes with value in \'escapedQuote\'', (t) => { + const opts = '--fields "a string" --escaped-quote "*"'; + + exec(`${cli} -i "${getFixturePath('/json/escapedQuotes.json')}" ${opts}`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.escapedQuotes); + t.end(); + }); + }); + + // String Excel + + testRunner.add('should format strings to force excel to view the values as strings', (t) => { + const opts = '--fields carModel,price,color --excel-strings'; + + exec(`${cli} -i "${getFixturePath('/json/default.json')}" ${opts}`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.excelStrings); + t.end(); + }); + }); + + // String Escaping and preserving values + + testRunner.add('should parse JSON values with trailing backslashes', (t) => { + const opts = '--fields carModel,price,color'; + + exec(`${cli} -i "${getFixturePath('/json/trailingBackslash.json')}" ${opts}`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.trailingBackslash); + t.end(); + }); + }); + + testRunner.add('should escape " when preceeded by \\', (t) => { + exec(`${cli} -i "${getFixturePath('/json/escapeDoubleBackslashedEscapedQuote.json')}"`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.escapeDoubleBackslashedEscapedQuote); + t.end(); + }); + }); + + testRunner.add('should preserve new lines in values', (t) => { + const opts = '--eol "\r\n"'; + + exec(`${cli} -i "${getFixturePath('/json/escapeEOL.json')}" ${opts}`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, [ + '"a string"', + '"with a \u2028description\\n and\na new line"', + '"with a \u2029\u2028description and\r\nanother new line"' + ].join('\r\n')); + t.end(); + }); + }); + + testRunner.add('should preserve tabs in values', (t) => { + exec(`${cli} -i "${getFixturePath('/json/escapeTab.json')}"`, (err, stdout, stderr) => { + t.notOk(stderr); + const csv = stdout; + t.equal(csv, csvFixtures.escapeTab); + t.end(); + }); + }); }; diff --git a/test/JSON2CSVAsyncParser.js b/test/JSON2CSVAsyncParser.js index 3a0d82a7..f4766d71 100644 --- a/test/JSON2CSVAsyncParser.js +++ b/test/JSON2CSVAsyncParser.js @@ -1,7 +1,12 @@ 'use strict'; const { Readable, Transform, Writable } = require('stream'); -const { AsyncParser, parseAsync, transforms: { flatten, unwind } } = require('../lib/json2csv'); +const { + parseAsync, + AsyncParser, + transforms: { flatten, unwind }, + formatters: { number: numberFormatter, string: stringFormatter, stringExcel: stringExcelFormatter, stringQuoteOnlyIfNecessary: stringQuoteOnlyIfNecessaryFormatter }, +} = require('../lib/json2csv'); module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) => { testRunner.add('should should error async if invalid opts are passed using parseAsync method', async (t) => { @@ -21,14 +26,14 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should parse in-memory json array to csv, infer the fields automatically and not modify the opts passed using parseAsync method', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; try { const csv = await parseAsync(inMemoryJsonFixtures.default, opts); t.ok(typeof csv === 'string'); t.equal(csv, csvFixtures.default); - t.deepEqual(opts, { fields: ['carModel', 'price', 'color', 'transmission'] }); + t.deepEqual(opts, { fields: ['carModel', 'price', 'color', 'manual'] }); } catch(err) { t.fail(err.message); } @@ -38,14 +43,14 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should parse in-memory json object to csv, infer the fields automatically and not modify the opts passed using parseAsync method', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; try { const csv = await parseAsync({ "carModel": "Audi", "price": 0, "color": "blue" }, opts); t.ok(typeof csv === 'string'); - t.equal(csv, '"carModel","price","color","transmission"\n"Audi",0,"blue",'); - t.deepEqual(opts, { fields: ['carModel', 'price', 'color', 'transmission'] }); + t.equal(csv, '"carModel","price","color","manual"\n"Audi",0,"blue",'); + t.deepEqual(opts, { fields: ['carModel', 'price', 'color', 'manual'] }); } catch(err) { t.fail(err.message); } @@ -55,14 +60,14 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should parse streaming json to csv, infer the fields automatically and not modify the opts passed using parseAsync method', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; try { const csv = await parseAsync(jsonFixtures.default(), opts); t.ok(typeof csv === 'string'); t.equal(csv, csvFixtures.default); - t.deepEqual(opts, { fields: ['carModel', 'price', 'color', 'transmission'] }); + t.deepEqual(opts, { fields: ['carModel', 'price', 'color', 'manual'] }); } catch(err) { t.fail(err.message); } @@ -72,7 +77,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should handle object mode with default input', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const transformOpts = { readableObjectMode: true, writableObjectMode: true }; const parser = new AsyncParser(opts, transformOpts); @@ -98,7 +103,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = input.push(null); const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const transformOpts = { readableObjectMode: true, writableObjectMode: true }; const parser = new AsyncParser(opts, transformOpts); @@ -115,7 +120,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should handle ndjson', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'], + fields: ['carModel', 'price', 'color', 'manual'], ndjson: true }; const parser = new AsyncParser(opts); @@ -132,7 +137,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should error on invalid ndjson input data', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'], + fields: ['carModel', 'price', 'color', 'manual'], ndjson: true }; const parser = new AsyncParser(opts); @@ -183,7 +188,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should error on invalid json input data', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const parser = new AsyncParser(opts); @@ -279,7 +284,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should parse json to csv using custom fields', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const parser = new AsyncParser(opts); @@ -576,146 +581,6 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = t.end(); }); - // Quote - - testRunner.add('should use a custom quote when \'quote\' property is present', async (t) => { - const opts = { - fields: ['carModel', 'price'], - quote: '\'' - }; - const parser = new AsyncParser(opts); - - try { - const csv = await parser.fromInput(jsonFixtures.default()).promise(); - t.equal(csv, csvFixtures.withSimpleQuotes); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - testRunner.add('should be able to don\'t output quotes when setting \'quote\' to empty string', async (t) => { - const opts = { - fields: ['carModel', 'price'], - quote: '' - }; - const parser = new AsyncParser(opts); - - try { - const csv = await parser.fromInput(jsonFixtures.default()).promise(); - t.equal(csv, csvFixtures.withoutQuotes); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - testRunner.add('should escape quotes when setting \'quote\' property is present', async (t) => { - const opts = { - fields: ['carModel', 'color'], - quote: '\'' - }; - const parser = new AsyncParser(opts); - - try { - const csv = await parser.fromInput(jsonFixtures.escapeCustomQuotes()).promise(); - t.equal(csv, csvFixtures.escapeCustomQuotes); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - testRunner.add('should not escape \'"\' when setting \'quote\' set to something else', async (t) => { - const opts = { - quote: '\'' - }; - const parser = new AsyncParser(opts); - - try { - const csv = await parser.fromInput(jsonFixtures.escapedQuotes()).promise(); - t.equal(csv, csvFixtures.escapedQuotesUnescaped); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - // Escaped Quote - - testRunner.add('should escape quotes with double quotes', async (t) => { - const parser = new AsyncParser(); - try { - const csv = await parser.fromInput(jsonFixtures.quotes()).promise(); - t.equal(csv, csvFixtures.quotes); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end', async (t) => { - const parser = new AsyncParser(); - try { - const csv = await parser.fromInput(jsonFixtures.backslashAtEnd()).promise(); - t.equal(csv, csvFixtures.backslashAtEnd); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end, and its not the last column', async (t) => { - const parser = new AsyncParser(); - try { - const csv = await parser.fromInput(jsonFixtures.backslashAtEndInMiddleColumn()).promise(); - t.equal(csv, csvFixtures.backslashAtEndInMiddleColumn); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - testRunner.add('should escape quotes with value in \'escapedQuote\'', async (t) => { - const opts = { - fields: ['a string'], - escapedQuote: '*' - }; - const parser = new AsyncParser(opts); - - try { - const csv = await parser.fromInput(jsonFixtures.escapedQuotes()).promise(); - t.equal(csv, csvFixtures.escapedQuotes); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - testRunner.add('should escape quotes before new line with value in \'escapedQuote\'', async (t) => { - const opts = { - fields: ['a string'] - }; - const parser = new AsyncParser(opts); - - try { - const csv = await parser.fromInput(jsonFixtures.backslashBeforeNewLine()).promise(); - t.equal(csv, csvFixtures.backslashBeforeNewLine); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - // Delimiter testRunner.add('should use a custom delimiter when \'delimiter\' property is defined', async (t) => { @@ -768,93 +633,12 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = t.end(); }); - // Excell - - testRunner.add('should format strings to force excel to view the values as strings', async (t) => { - const opts = { - fields: ['carModel', 'price', 'color'], - excelStrings:true - }; - const parser = new AsyncParser(opts); - - try { - const csv = await parser.fromInput(jsonFixtures.default()).promise(); - t.equal(csv, csvFixtures.excelStrings); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - // Escaping and preserving values - - testRunner.add('should parse JSON values with trailing backslashes', async (t) => { - const opts = { - fields: ['carModel', 'price', 'color'] - }; - const parser = new AsyncParser(opts); - - try { - const csv = await parser.fromInput(jsonFixtures.trailingBackslash()).promise(); - t.equal(csv, csvFixtures.trailingBackslash); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - testRunner.add('should escape " when preceeded by \\', async (t) => { - const parser = new AsyncParser(); - try { - const csv = await parser.fromInput(jsonFixtures.escapeDoubleBackslashedEscapedQuote()).promise(); - t.equal(csv, csvFixtures.escapeDoubleBackslashedEscapedQuote); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - testRunner.add('should preserve new lines in values', async (t) => { - const opts = { - eol: '\r\n' - }; - const parser = new AsyncParser(opts); - - try { - const csv = await parser.fromInput(jsonFixtures.escapeEOL()).promise(); - t.equal(csv, [ - '"a string"', - '"with a \u2028description\\n and\na new line"', - '"with a \u2029\u2028description and\r\nanother new line"' - ].join('\r\n')); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - - testRunner.add('should preserve tabs in values', async (t) => { - const parser = new AsyncParser(); - try { - const csv = await parser.fromInput(jsonFixtures.escapeTab()).promise(); - t.equal(csv, csvFixtures.escapeTab); - } catch(err) { - t.fail(err.message); - } - - t.end(); - }); - // Header testRunner.add('should parse json to csv without column title', async (t) => { const opts = { header: false, - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const parser = new AsyncParser(opts); @@ -970,7 +754,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should add BOM character', async (t) => { const opts = { withBOM: true, - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const parser = new AsyncParser(opts); @@ -1003,7 +787,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should use custom transforms if configured', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const myTransform = new Transform({ transform(chunk, encoding, callback) { @@ -1051,7 +835,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should use custom output if configured', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const memoryOutput = new Writable({ write(chunk, enc, cb) { @@ -1111,7 +895,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should catch errors even if ret option is set to false', async (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const parser = new AsyncParser(opts); @@ -1266,7 +1050,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = model: row.carModel, price: row.price / 1000, color: row.color, - transmission: row.transmission || 'automatic', + manual: row.manual || 'automatic', })], }; @@ -1281,4 +1065,363 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = t.end(); }); + + // Formatters + + + + // Number + + testRunner.add('should used a custom separator when \'decimals\' is passed to the number formatter', async (t) => { + const opts = { + formatters: { + number: numberFormatter({ decimals: 2 }) + } + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.numberFormatter()).promise(); + t.equal(csv, csvFixtures.numberFixedDecimals); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should used a custom separator when \'separator\' is passed to the number formatter', async (t) => { + const opts = { + delimiter: ';', + formatters: { + number: numberFormatter({ separator: ',' }) + } + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.numberFormatter()).promise(); + t.equal(csv, csvFixtures.numberCustomSeparator); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should used a custom separator and fixed number of decimals when \'separator\' and \'decimals\' are passed to the number formatter', async (t) => { + const opts = { + delimiter: ';', + formatters: { + number: numberFormatter({ separator: ',', decimals: 2 }) + } + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.numberFormatter()).promise(); + t.equal(csv, csvFixtures.numberFixedDecimalsAndCustomSeparator); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + // Symbol + + testRunner.add('should format Symbol by its name', async (t) => { + const data = [{ test: Symbol('test1') }, { test: Symbol('test2') }]; + const input = new Readable({ objectMode: true }); + input._read = () => {}; + data.forEach(item => input.push(item)); + input.push(null); + + const transformOpts = { readableObjectMode: true, writableObjectMode: true }; + + const parser = new AsyncParser({}, transformOpts); + + try { + const csv = await parser.fromInput(input).promise(); + t.equal(csv, '"test"\n"test1"\n"test2"'); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + // String Quote + + testRunner.add('should use a custom quote when \'quote\' property is present', async (t) => { + const opts = { + fields: ['carModel', 'price'], + formatters: { + string: stringFormatter({ quote: '\'' }) + } + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.default()).promise(); + t.equal(csv, csvFixtures.withSimpleQuotes); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should be able to don\'t output quotes when setting \'quote\' to empty string', async (t) => { + const opts = { + fields: ['carModel', 'price'], + formatters: { + string: stringFormatter({ quote: '' }) + } + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.default()).promise(); + t.equal(csv, csvFixtures.withoutQuotes); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should escape quotes when setting \'quote\' property is present', async (t) => { + const opts = { + fields: ['carModel', 'color'], + formatters: { + string: stringFormatter({ quote: '\'' }) + } + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.escapeCustomQuotes()).promise(); + t.equal(csv, csvFixtures.escapeCustomQuotes); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should not escape \'"\' when setting \'quote\' set to something else', async (t) => { + const opts = { + formatters: { + string: stringFormatter({ quote: '\'' }) + } + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.escapedQuotes()).promise(); + t.equal(csv, csvFixtures.escapedQuotesUnescaped); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + // String Escaped Quote + + testRunner.add('should escape quotes with double quotes', async (t) => { + const parser = new AsyncParser(); + try { + const csv = await parser.fromInput(jsonFixtures.quotes()).promise(); + t.equal(csv, csvFixtures.quotes); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end', async (t) => { + const parser = new AsyncParser(); + try { + const csv = await parser.fromInput(jsonFixtures.backslashAtEnd()).promise(); + t.equal(csv, csvFixtures.backslashAtEnd); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end, and its not the last column', async (t) => { + const parser = new AsyncParser(); + try { + const csv = await parser.fromInput(jsonFixtures.backslashAtEndInMiddleColumn()).promise(); + t.equal(csv, csvFixtures.backslashAtEndInMiddleColumn); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should escape quotes with value in \'escapedQuote\'', async (t) => { + const opts = { + fields: ['a string'], + formatters: { + string: stringFormatter({ escapedQuote: '*' }) + } + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.escapedQuotes()).promise(); + t.equal(csv, csvFixtures.escapedQuotes); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should escape quotes before new line with value in \'escapedQuote\'', async (t) => { + const opts = { + fields: ['a string'] + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.backslashBeforeNewLine()).promise(); + t.equal(csv, csvFixtures.backslashBeforeNewLine); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + // String Quote Only if Necessary + + testRunner.add('should quote only if necessary if using stringQuoteOnlyIfNecessary formatter', async (t) => { + const opts = { + formatters: { + string: stringQuoteOnlyIfNecessaryFormatter() + } + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.quoteOnlyIfNecessary()).promise(); + t.equal(csv, csvFixtures.quoteOnlyIfNecessary); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + // String Excel + + testRunner.add('should format strings to force excel to view the values as strings', async (t) => { + const opts = { + fields: ['carModel', 'price', 'color'], + formatters: { + string: stringExcelFormatter() + } + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.default()).promise(); + t.equal(csv, csvFixtures.excelStrings); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + // String Escaping and preserving values + + testRunner.add('should parse JSON values with trailing backslashes', async (t) => { + const opts = { + fields: ['carModel', 'price', 'color'] + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.trailingBackslash()).promise(); + t.equal(csv, csvFixtures.trailingBackslash); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should escape " when preceeded by \\', async (t) => { + const parser = new AsyncParser(); + try { + const csv = await parser.fromInput(jsonFixtures.escapeDoubleBackslashedEscapedQuote()).promise(); + t.equal(csv, csvFixtures.escapeDoubleBackslashedEscapedQuote); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should preserve new lines in values', async (t) => { + const opts = { + eol: '\r\n' + }; + const parser = new AsyncParser(opts); + + try { + const csv = await parser.fromInput(jsonFixtures.escapeEOL()).promise(); + t.equal(csv, [ + '"a string"', + '"with a \u2028description\\n and\na new line"', + '"with a \u2029\u2028description and\r\nanother new line"' + ].join('\r\n')); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + testRunner.add('should preserve tabs in values', async (t) => { + const parser = new AsyncParser(); + try { + const csv = await parser.fromInput(jsonFixtures.escapeTab()).promise(); + t.equal(csv, csvFixtures.escapeTab); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); + + // Headers + + testRunner.add('should format headers based on the headers formatter', async (t) => { + const opts = { + fields: ['carModel', 'price', 'color', 'manual'], + formatters: { + header: stringFormatter({ quote: '' }) + } + }; + + const parser = new AsyncParser(opts); + try { + const csv = await parser.fromInput(jsonFixtures.default()).promise(); + t.equal(csv, csvFixtures.customHeaderQuotes); + } catch(err) { + t.fail(err.message); + } + + t.end(); + }); }; diff --git a/test/JSON2CSVParser.js b/test/JSON2CSVParser.js index ac0b2f12..de1620e3 100644 --- a/test/JSON2CSVParser.js +++ b/test/JSON2CSVParser.js @@ -1,6 +1,11 @@ 'use strict'; -const { parse, Parser: Json2csvParser, transforms: { flatten, unwind } } = require('../lib/json2csv'); +const { + parse, + Parser: Json2csvParser, + transforms: { flatten, unwind }, + formatters: { number: numberFormatter, string: stringFormatter, stringExcel: stringExcelFormatter, stringQuoteOnlyIfNecessary: stringQuoteOnlyIfNecessaryFormatter }, +} = require('../lib/json2csv'); module.exports = (testRunner, jsonFixtures, csvFixtures) => { testRunner.add('should parse json to csv, infer the fields automatically and not modify the opts passed using parse method', (t) => { @@ -110,7 +115,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { testRunner.add('should parse json to csv using custom fields', (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const parser = new Json2csvParser(opts); @@ -356,110 +361,6 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { t.end(); }); - // Quote - - testRunner.add('should use a custom quote when \'quote\' property is present', (t) => { - const opts = { - fields: ['carModel', 'price'], - quote: '\'' - }; - - const parser = new Json2csvParser(opts); - const csv = parser.parse(jsonFixtures.default); - - t.equal(csv, csvFixtures.withSimpleQuotes); - t.end(); - }); - - testRunner.add('should be able to don\'t output quotes when setting \'quote\' to empty string', (t) => { - const opts = { - fields: ['carModel', 'price'], - quote: '' - }; - - const parser = new Json2csvParser(opts); - const csv = parser.parse(jsonFixtures.default); - - t.equal(csv, csvFixtures.withoutQuotes); - t.end(); - }); - - testRunner.add('should escape quotes when setting \'quote\' property is present', (t) => { - const opts = { - fields: ['carModel', 'color'], - quote: '\'' - }; - - const parser = new Json2csvParser(opts); - const csv = parser.parse(jsonFixtures.escapeCustomQuotes); - - t.equal(csv, csvFixtures.escapeCustomQuotes); - t.end(); - }); - - testRunner.add('should not escape \'"\' when setting \'quote\' set to something else', (t) => { - const opts = { - quote: '\'' - }; - - const parser = new Json2csvParser(opts); - const csv = parser.parse(jsonFixtures.escapedQuotes); - - t.equal(csv, csvFixtures.escapedQuotesUnescaped); - t.end(); - }); - - // Escaped Quote - - testRunner.add('should escape quotes with double quotes', (t) => { - const parser = new Json2csvParser(); - const csv = parser.parse(jsonFixtures.quotes); - - t.equal(csv, csvFixtures.quotes); - t.end(); - }); - - testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end', (t) => { - const parser = new Json2csvParser(); - const csv = parser.parse(jsonFixtures.backslashAtEnd); - - t.equal(csv, csvFixtures.backslashAtEnd); - t.end(); - }); - - testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end, and its not the last column', (t) => { - const parser = new Json2csvParser(); - const csv = parser.parse(jsonFixtures.backslashAtEndInMiddleColumn); - - t.equal(csv, csvFixtures.backslashAtEndInMiddleColumn); - t.end(); - }); - - testRunner.add('should escape quotes with value in \'escapedQuote\'', (t) => { - const opts = { - fields: ['a string'], - escapedQuote: '*' - }; - - const parser = new Json2csvParser(opts); - const csv = parser.parse(jsonFixtures.escapedQuotes); - - t.equal(csv, csvFixtures.escapedQuotes); - t.end(); - }); - - testRunner.add('should escape quotes before new line with value in \'escapedQuote\'', (t) => { - const opts = { - fields: ['a string'] - }; - - const parser = new Json2csvParser(opts); - const csv = parser.parse(jsonFixtures.backslashBeforeNewLine); - - t.equal(csv, csvFixtures.backslashBeforeNewLine); - t.end(); - }); - // Delimiter testRunner.add('should use a custom delimiter when \'delimiter\' property is defined', (t) => { @@ -500,73 +401,12 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { t.end(); }); - // Excell - - testRunner.add('should format strings to force excel to view the values as strings', (t) => { - const opts = { - fields: ['carModel', 'price', 'color'], - excelStrings:true - }; - - const parser = new Json2csvParser(opts); - const csv = parser.parse(jsonFixtures.default); - - t.equal(csv, csvFixtures.excelStrings); - t.end(); - }); - - // Escaping and preserving values - - testRunner.add('should parse JSON values with trailing backslashes', (t) => { - const opts = { - fields: ['carModel', 'price', 'color'] - }; - - const parser = new Json2csvParser(opts); - const csv = parser.parse(jsonFixtures.trailingBackslash); - - t.equal(csv, csvFixtures.trailingBackslash); - t.end(); - }); - - testRunner.add('should escape " when preceeded by \\', (t) => { - const parser = new Json2csvParser(); - const csv = parser.parse(jsonFixtures.escapeDoubleBackslashedEscapedQuote); - - t.equal(csv, csvFixtures.escapeDoubleBackslashedEscapedQuote); - t.end(); - }); - - testRunner.add('should preserve new lines in values', (t) => { - const opts = { - eol: '\r\n' - }; - - const parser = new Json2csvParser(opts); - const csv = parser.parse(jsonFixtures.escapeEOL); - - t.equal(csv, [ - '"a string"', - '"with a \u2028description\\n and\na new line"', - '"with a \u2029\u2028description and\r\nanother new line"' - ].join('\r\n')); - t.end(); - }); - - testRunner.add('should preserve tabs in values', (t) => { - const parser = new Json2csvParser(); - const csv = parser.parse(jsonFixtures.escapeTab); - - t.equal(csv, csvFixtures.escapeTab); - t.end(); - }); - // Header testRunner.add('should parse json to csv without column title', (t) => { const opts = { header: false, - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const parser = new Json2csvParser(opts); @@ -654,7 +494,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { testRunner.add('should add BOM character', (t) => { const opts = { withBOM: true, - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const parser = new Json2csvParser(opts); @@ -787,7 +627,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { model: row.carModel, price: row.price / 1000, color: row.color, - transmission: row.transmission || 'automatic', + manual: row.manual || 'automatic', })], }; @@ -797,4 +637,277 @@ module.exports = (testRunner, jsonFixtures, csvFixtures) => { t.equal(csv, csvFixtures.defaultCustomTransform); t.end(); }); + + // Formatters + + // undefined + // boolean + + // Number + + testRunner.add('should used a custom separator when \'decimals\' is passed to the number formatter', (t) => { + const opts = { + formatters: { + number: numberFormatter({ decimals: 2 }) + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.numberFormatter); + + t.equal(csv, csvFixtures.numberFixedDecimals); + t.end(); + }); + + testRunner.add('should used a custom separator when \'separator\' is passed to the number formatter', (t) => { + const opts = { + delimiter: ';', + formatters: { + number: numberFormatter({ separator: ',' }) + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.numberFormatter); + + t.equal(csv, csvFixtures.numberCustomSeparator); + t.end(); + }); + + testRunner.add('should used a custom separator and fixed number of decimals when \'separator\' and \'decimals\' are passed to the number formatter', (t) => { + const opts = { + delimiter: ';', + formatters: { + number: numberFormatter({ separator: ',', decimals: 2 }) + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.numberFormatter); + + t.equal(csv, csvFixtures.numberFixedDecimalsAndCustomSeparator); + t.end(); + }); + + // Symbol + + testRunner.add('should format Symbol by its name', (t) => { + const parser = new Json2csvParser(); + const csv = parser.parse([{ test: Symbol('test1') }, { test: Symbol('test2') }]); + + t.equal(csv, '"test"\n"test1"\n"test2"'); + t.end(); + }); + + // function + // object + + // String Quote + + testRunner.add('should use a custom quote when \'quote\' property is present', (t) => { + const opts = { + fields: ['carModel', 'price'], + formatters: { + string: stringFormatter({ quote: '\'' }) + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.default); + + t.equal(csv, csvFixtures.withSimpleQuotes); + t.end(); + }); + + testRunner.add('should be able to don\'t output quotes when setting \'quote\' to empty string', (t) => { + const opts = { + fields: ['carModel', 'price'], + formatters: { + string: stringFormatter({ quote: '' }) + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.default); + + t.equal(csv, csvFixtures.withoutQuotes); + t.end(); + }); + + testRunner.add('should escape quotes when setting \'quote\' property is present', (t) => { + const opts = { + fields: ['carModel', 'color'], + formatters: { + string: stringFormatter({ quote: '\'' }) + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.escapeCustomQuotes); + + t.equal(csv, csvFixtures.escapeCustomQuotes); + t.end(); + }); + + testRunner.add('should not escape \'"\' when setting \'quote\' set to something else', (t) => { + const opts = { + formatters: { + string: stringFormatter({ quote: '\'' }) + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.escapedQuotes); + + t.equal(csv, csvFixtures.escapedQuotesUnescaped); + t.end(); + }); + + // String Escaped Quote + + testRunner.add('should escape quotes with double quotes', (t) => { + const parser = new Json2csvParser(); + const csv = parser.parse(jsonFixtures.quotes); + + t.equal(csv, csvFixtures.quotes); + t.end(); + }); + + testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end', (t) => { + const parser = new Json2csvParser(); + const csv = parser.parse(jsonFixtures.backslashAtEnd); + + t.equal(csv, csvFixtures.backslashAtEnd); + t.end(); + }); + + testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end, and its not the last column', (t) => { + const parser = new Json2csvParser(); + const csv = parser.parse(jsonFixtures.backslashAtEndInMiddleColumn); + + t.equal(csv, csvFixtures.backslashAtEndInMiddleColumn); + t.end(); + }); + + testRunner.add('should escape quotes with value in \'escapedQuote\'', (t) => { + const opts = { + fields: ['a string'], + formatters: { + string: stringFormatter({ escapedQuote: '*' }) + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.escapedQuotes); + + t.equal(csv, csvFixtures.escapedQuotes); + t.end(); + }); + + testRunner.add('should escape quotes before new line with value in \'escapedQuote\'', (t) => { + const opts = { + fields: ['a string'] + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.backslashBeforeNewLine); + + t.equal(csv, csvFixtures.backslashBeforeNewLine); + t.end(); + }); + + // String Quote Only if Necessary + + testRunner.add('should quote only if necessary if using stringQuoteOnlyIfNecessary formatter', (t) => { + const opts = { + formatters: { + string: stringQuoteOnlyIfNecessaryFormatter() + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.quoteOnlyIfNecessary); + + t.equal(csv, csvFixtures.quoteOnlyIfNecessary); + t.end(); + }); + + // String Excel + + testRunner.add('should format strings to force excel to view the values as strings', (t) => { + const opts = { + fields: ['carModel', 'price', 'color'], + formatters: { + string: stringExcelFormatter() + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.default); + + t.equal(csv, csvFixtures.excelStrings); + t.end(); + }); + + // String Escaping and preserving values + + testRunner.add('should parse JSON values with trailing backslashes', (t) => { + const opts = { + fields: ['carModel', 'price', 'color'] + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.trailingBackslash); + + t.equal(csv, csvFixtures.trailingBackslash); + t.end(); + }); + + testRunner.add('should escape " when preceeded by \\', (t) => { + const parser = new Json2csvParser(); + const csv = parser.parse(jsonFixtures.escapeDoubleBackslashedEscapedQuote); + + t.equal(csv, csvFixtures.escapeDoubleBackslashedEscapedQuote); + t.end(); + }); + + testRunner.add('should preserve new lines in values', (t) => { + const opts = { + eol: '\r\n' + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.escapeEOL); + + t.equal(csv, [ + '"a string"', + '"with a \u2028description\\n and\na new line"', + '"with a \u2029\u2028description and\r\nanother new line"' + ].join('\r\n')); + t.end(); + }); + + testRunner.add('should preserve tabs in values', (t) => { + const parser = new Json2csvParser(); + const csv = parser.parse(jsonFixtures.escapeTab); + + t.equal(csv, csvFixtures.escapeTab); + t.end(); + }); + + // Headers + + testRunner.add('should format headers based on the headers formatter', (t) => { + const opts = { + formatters: { + header: stringFormatter({ quote: '' }) + } + }; + + const parser = new Json2csvParser(opts); + const csv = parser.parse(jsonFixtures.default); + + t.equal(csv, csvFixtures.customHeaderQuotes); + t.end(); + }); }; diff --git a/test/JSON2CSVTransform.js b/test/JSON2CSVTransform.js index b1240283..27ac201f 100644 --- a/test/JSON2CSVTransform.js +++ b/test/JSON2CSVTransform.js @@ -1,7 +1,11 @@ 'use strict'; const { Readable } = require('stream'); -const { Transform: Json2csvTransform, transforms: { flatten, unwind } } = require('../lib/json2csv'); +const { + Transform: Json2csvTransform, + transforms: { flatten, unwind }, + formatters: { number: numberFormatter, string: stringFormatter, stringExcel: stringExcelFormatter, stringQuoteOnlyIfNecessary: stringQuoteOnlyIfNecessaryFormatter }, +} = require('../lib/json2csv'); module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) => { testRunner.add('should handle object mode', (t) => { @@ -11,7 +15,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = input.push(null); const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const transformOpts = { objectMode: true }; @@ -33,7 +37,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should handle ndjson', (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'], + fields: ['carModel', 'price', 'color', 'manual'], ndjson: true }; @@ -55,7 +59,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should error on invalid ndjson input data', (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'], + fields: ['carModel', 'price', 'color', 'manual'], ndjson: true }; @@ -113,7 +117,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should error on invalid json input data', (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const transform = new Json2csvTransform(opts); @@ -233,7 +237,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = testRunner.add('should parse json to csv using custom fields', (t) => { const opts = { - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price', 'color', 'manual'] }; const transform = new Json2csvTransform(opts); @@ -590,12 +594,12 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - // Quote + // Delimiter - testRunner.add('should use a custom quote when \'quote\' property is present', (t) => { + testRunner.add('should use a custom delimiter when \'delimiter\' property is defined', (t) => { const opts = { - fields: ['carModel', 'price'], - quote: '\'' + fields: ['carModel', 'price', 'color'], + delimiter: '\t' }; const transform = new Json2csvTransform(opts); @@ -605,7 +609,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.withSimpleQuotes); + t.equal(csv, csvFixtures.tsv); t.end(); }) .on('error', err => { @@ -614,20 +618,17 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should be able to don\'t output quotes when setting \'quote\' to empty string', (t) => { - const opts = { - fields: ['carModel', 'price'], - quote: '' - }; + testRunner.add('should remove last delimiter |@|', (t) => { + const opts = { delimiter: '|@|' }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.default().pipe(transform); + const processor = jsonFixtures.delimiter().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.withoutQuotes); + t.equal(csv, csvFixtures.delimiter); t.end(); }) .on('error', err => { @@ -636,20 +637,22 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should escape quotes when setting \'quote\' property is present', (t) => { + // EOL + + testRunner.add('should use a custom eol character when \'eol\' property is present', (t) => { const opts = { - fields: ['carModel', 'color'], - quote: '\'' + fields: ['carModel', 'price', 'color'], + eol: '\r\n' }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.escapeCustomQuotes().pipe(transform); + const processor = jsonFixtures.default().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.escapeCustomQuotes); + t.equal(csv, csvFixtures.eol); t.end(); }) .on('error', err => { @@ -658,19 +661,22 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should not escape \'"\' when setting \'quote\' set to something else', (t) => { + // Header + + testRunner.add('should parse json to csv without column title', (t) => { const opts = { - quote: '\'' + header: false, + fields: ['carModel', 'price', 'color', 'manual'] }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.escapedQuotes().pipe(transform); + const processor = jsonFixtures.default().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.escapedQuotesUnescaped); + t.equal(csv, csvFixtures.withoutHeader); t.end(); }) .on('error', err => { @@ -679,17 +685,17 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - // Escaped Quote + // Include empty rows - testRunner.add('should escape quotes with double quotes', (t) => { + testRunner.add('should not include empty rows when options.includeEmptyRows is not specified', (t) => { const transform = new Json2csvTransform(); - const processor = jsonFixtures.quotes().pipe(transform); + const processor = jsonFixtures.emptyRow().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.quotes); + t.equal(csv, csvFixtures.emptyRowNotIncluded); t.end(); }) .on('error', err => { @@ -698,15 +704,19 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end', (t) => { - const transform = new Json2csvTransform(); - const processor = jsonFixtures.backslashAtEnd().pipe(transform); + testRunner.add('should include empty rows when options.includeEmptyRows is true', (t) => { + const opts = { + includeEmptyRows: true + }; + + const transform = new Json2csvTransform(opts); + const processor = jsonFixtures.emptyRow().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.backslashAtEnd); + t.equal(csv, csvFixtures.emptyRow); t.end(); }) .on('error', err => { @@ -715,15 +725,19 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end, and its not the last column', (t) => { - const transform = new Json2csvTransform(); - const processor = jsonFixtures.backslashAtEndInMiddleColumn().pipe(transform); + testRunner.add('should not include empty rows when options.includeEmptyRows is false', (t) => { + const opts = { + includeEmptyRows: false, + }; + + const transform = new Json2csvTransform(opts); + const processor = jsonFixtures.emptyRow().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.backslashAtEndInMiddleColumn); + t.equal(csv, csvFixtures.emptyRowNotIncluded); t.end(); }) .on('error', err => { @@ -732,20 +746,32 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should escape quotes with value in \'escapedQuote\'', (t) => { + testRunner.add('should include empty rows when options.includeEmptyRows is true, with default values', (t) => { const opts = { - fields: ['a string'], - escapedQuote: '*' + fields: [ + { + value: 'carModel' + }, + { + value: 'price', + default: 1 + }, + { + value: 'color' + } + ], + defaultValue: 'NULL', + includeEmptyRows: true, }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.escapedQuotes().pipe(transform); + const processor = jsonFixtures.emptyRow().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.escapedQuotes); + t.equal(csv, csvFixtures.emptyRowDefaultValues); t.end(); }) .on('error', err => { @@ -754,19 +780,25 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should escape quotes before new line with value in \'escapedQuote\'', (t) => { + testRunner.add('should parse data:[null] to csv with only column title, despite options.includeEmptyRows', (t) => { + const input = new Readable(); + input._read = () => {}; + input.push(JSON.stringify([null])); + input.push(null); + const opts = { - fields: ['a string'] + fields: ['carModel', 'price', 'color'], + includeEmptyRows: true, }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.backslashBeforeNewLine().pipe(transform); + const processor = input.pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.backslashBeforeNewLine); + t.equal(csv, csvFixtures.emptyObject); t.end(); }) .on('error', err => { @@ -775,22 +807,25 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - // Delimiter + // BOM - testRunner.add('should use a custom delimiter when \'delimiter\' property is defined', (t) => { + testRunner.add('should add BOM character', (t) => { const opts = { - fields: ['carModel', 'price', 'color'], - delimiter: '\t' + withBOM: true, + fields: ['carModel', 'price', 'color', 'manual'] }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.default().pipe(transform); + const processor = jsonFixtures.specialCharacters().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.tsv); + // Compare csv length to check if the BOM character is present + t.equal(csv[0], '\ufeff'); + t.equal(csv.length, csvFixtures.default.length + 1); + t.equal(csv.length, csvFixtures.withBOM.length); t.end(); }) .on('error', err => { @@ -799,17 +834,22 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should remove last delimiter |@|', (t) => { - const opts = { delimiter: '|@|' }; + // Transform + + testRunner.add('should unwind all unwindable fields using the unwind transform', (t) => { + const opts = { + fields: ['carModel', 'price', 'extras.items.name', 'extras.items.color', 'extras.items.items.position', 'extras.items.items.color'], + transforms: [unwind()], + }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.delimiter().pipe(transform); + const processor = jsonFixtures.unwind2().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.delimiter); + t.equal(csv, csvFixtures.unwind2); t.end(); }) .on('error', err => { @@ -818,22 +858,20 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - // EOL - - testRunner.add('should use a custom eol character when \'eol\' property is present', (t) => { + testRunner.add('should support unwinding specific fields using the unwind transform', (t) => { const opts = { - fields: ['carModel', 'price', 'color'], - eol: '\r\n' + fields: ['carModel', 'price', 'colors'], + transforms: [unwind({ paths: ['colors'] })], }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.default().pipe(transform); + const processor = jsonFixtures.unwind().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.eol); + t.equal(csv, csvFixtures.unwind); t.end(); }) .on('error', err => { @@ -842,22 +880,20 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - // Excell - - testRunner.add('should format strings to force excel to view the values as strings', (t) => { + testRunner.add('should support multi-level unwind using the unwind transform', (t) => { const opts = { - fields: ['carModel', 'price', 'color'], - excelStrings:true + fields: ['carModel', 'price', 'extras.items.name', 'extras.items.color', 'extras.items.items.position', 'extras.items.items.color'], + transforms: [unwind({ paths: ['extras.items', 'extras.items.items'] })], }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.default().pipe(transform); + const processor = jsonFixtures.unwind2().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.excelStrings); + t.equal(csv, csvFixtures.unwind2); t.end(); }) .on('error', err => { @@ -866,21 +902,20 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - // Escaping and preserving values - - testRunner.add('should parse JSON values with trailing backslashes', (t) => { + testRunner.add('should support unwind and blank out repeated data using the unwind transform', (t) => { const opts = { - fields: ['carModel', 'price', 'color'] + fields: ['carModel', 'price', 'extras.items.name', 'extras.items.color', 'extras.items.items.position', 'extras.items.items.color'], + transforms: [unwind({ paths: ['extras.items', 'extras.items.items'], blankOut: true })], }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.trailingBackslash().pipe(transform); + const processor = jsonFixtures.unwind2().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.trailingBackslash); + t.equal(csv, csvFixtures.unwind2Blank); t.end(); }) .on('error', err => { @@ -889,15 +924,19 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should escape " when preceeded by \\', (t) => { - const transform = new Json2csvTransform(); - const processor = jsonFixtures.escapeDoubleBackslashedEscapedQuote().pipe(transform); + testRunner.add('should support flattening deep JSON using the flatten transform', (t) => { + const opts = { + transforms: [flatten()], + }; + + const transform = new Json2csvTransform(opts); + const processor = jsonFixtures.deepJSON().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.escapeDoubleBackslashedEscapedQuote); + t.equal(csv, csvFixtures.flattenedDeepJSON); t.end(); }) .on('error', err => { @@ -906,23 +945,19 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should preserve new lines in values', (t) => { + testRunner.add('should support flattening JSON with nested arrays using the flatten transform', (t) => { const opts = { - eol: '\r\n' + transforms: [flatten({ arrays: true })], }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.escapeEOL().pipe(transform); + const processor = jsonFixtures.flattenArrays().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, [ - '"a string"', - '"with a \u2028description\\n and\na new line"', - '"with a \u2029\u2028description and\r\nanother new line"' - ].join('\r\n')); + t.equal(csv, csvFixtures.flattenedArrays); t.end(); }) .on('error', err => { @@ -931,15 +966,19 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should preserve tabs in values', (t) => { - const transform = new Json2csvTransform(); - const processor = jsonFixtures.escapeTab().pipe(transform); + testRunner.add('should support custom flatten separator using the flatten transform', (t) => { + const opts = { + transforms: [flatten({ separator: '__' })], + }; + + const transform = new Json2csvTransform(opts); + const processor = jsonFixtures.deepJSON().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.escapeTab); + t.equal(csv, csvFixtures.flattenedCustomSeparatorDeepJSON); t.end(); }) .on('error', err => { @@ -948,22 +987,19 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - // Header - - testRunner.add('should parse json to csv without column title', (t) => { + testRunner.add('should support multiple transforms and honor the order in which they are declared', (t) => { const opts = { - header: false, - fields: ['carModel', 'price', 'color', 'transmission'] + transforms: [unwind({ paths: ['items'] }), flatten()], }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.default().pipe(transform); + const processor = jsonFixtures.unwindAndFlatten().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.withoutHeader); + t.equal(csv, csvFixtures.unwindAndFlatten); t.end(); }) .on('error', err => { @@ -972,17 +1008,24 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - // Include empty rows + testRunner.add('should support custom transforms', (t) => { + const opts = { + transforms: [row => ({ + model: row.carModel, + price: row.price / 1000, + color: row.color, + manual: row.manual || 'automatic', + })], + }; - testRunner.add('should not include empty rows when options.includeEmptyRows is not specified', (t) => { - const transform = new Json2csvTransform(); - const processor = jsonFixtures.emptyRow().pipe(transform); + const transform = new Json2csvTransform(opts); + const processor = jsonFixtures.default().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.emptyRowNotIncluded); + t.equal(csv, csvFixtures.defaultCustomTransform); t.end(); }) .on('error', err => { @@ -991,19 +1034,24 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should include empty rows when options.includeEmptyRows is true', (t) => { + // Formatters + + // Number + + testRunner.add('should used a custom separator when \'decimals\' is passed to the number formatter', (t) => { const opts = { - includeEmptyRows: true + formatters: { + number: numberFormatter({ decimals: 2 }) + } }; - const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.emptyRow().pipe(transform); + const processor = jsonFixtures.numberFormatter().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.emptyRow); + t.equal(csv, csvFixtures.numberFixedDecimals); t.end(); }) .on('error', err => { @@ -1012,19 +1060,21 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should not include empty rows when options.includeEmptyRows is false', (t) => { + testRunner.add('should used a custom separator when \'separator\' is passed to the number formatter', (t) => { const opts = { - includeEmptyRows: false, + delimiter: ';', + formatters: { + number: numberFormatter({ separator: ',' }) + } }; - const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.emptyRow().pipe(transform); + const processor = jsonFixtures.numberFormatter().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.emptyRowNotIncluded); + t.equal(csv, csvFixtures.numberCustomSeparator); t.end(); }) .on('error', err => { @@ -1033,32 +1083,21 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should include empty rows when options.includeEmptyRows is true, with default values', (t) => { + testRunner.add('should used a custom separator and fixed number of decimals when \'separator\' and \'decimals\' are passed to the number formatter', (t) => { const opts = { - fields: [ - { - value: 'carModel' - }, - { - value: 'price', - default: 1 - }, - { - value: 'color' - } - ], - defaultValue: 'NULL', - includeEmptyRows: true, + delimiter: ';', + formatters: { + number: numberFormatter({ separator: ',', decimals: 2 }) + } }; - const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.emptyRow().pipe(transform); + const processor = jsonFixtures.numberFormatter().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.emptyRowDefaultValues); + t.equal(csv, csvFixtures.numberFixedDecimalsAndCustomSeparator); t.end(); }) .on('error', err => { @@ -1067,25 +1106,25 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should parse data:[null] to csv with only column title, despite options.includeEmptyRows', (t) => { - const input = new Readable(); + // Symbol + + testRunner.add('should format Symbol by its name', async (t) => { + const data = [{ test: Symbol('test1') }, { test: Symbol('test2') }]; + const input = new Readable({ objectMode: true }); input._read = () => {}; - input.push(JSON.stringify([null])); + data.forEach(item => input.push(item)); input.push(null); - const opts = { - fields: ['carModel', 'price', 'color'], - includeEmptyRows: true, - }; + const transformOpts = { readableObjectMode: true, writableObjectMode: true }; - const transform = new Json2csvTransform(opts); + const transform = new Json2csvTransform({}, transformOpts); const processor = input.pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.emptyObject); + t.equal(csv, '"test"\n"test1"\n"test2"'); t.end(); }) .on('error', err => { @@ -1094,25 +1133,24 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - // BOM + // String Quote - testRunner.add('should add BOM character', (t) => { + testRunner.add('should use a custom quote when \'quote\' property is present', (t) => { const opts = { - withBOM: true, - fields: ['carModel', 'price', 'color', 'transmission'] + fields: ['carModel', 'price'], + formatters: { + string: stringFormatter({ quote: '\'' }) + } }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.specialCharacters().pipe(transform); + const processor = jsonFixtures.default().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - // Compare csv length to check if the BOM character is present - t.equal(csv[0], '\ufeff'); - t.equal(csv.length, csvFixtures.default.length + 1); - t.equal(csv.length, csvFixtures.withBOM.length); + t.equal(csv, csvFixtures.withSimpleQuotes); t.end(); }) .on('error', err => { @@ -1121,22 +1159,46 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - // Transform + testRunner.add('should be able to don\'t output quotes when setting \'quote\' to empty string', (t) => { + const opts = { + fields: ['carModel', 'price'], + formatters: { + string: stringFormatter({ quote: '' }) + } + }; - testRunner.add('should unwind all unwindable fields using the unwind transform', (t) => { + const transform = new Json2csvTransform(opts); + const processor = jsonFixtures.default().pipe(transform); + + let csv = ''; + processor + .on('data', chunk => (csv += chunk.toString())) + .on('end', () => { + t.equal(csv, csvFixtures.withoutQuotes); + t.end(); + }) + .on('error', err => { + t.fail(err.message); + t.end(); + }); + }); + + testRunner.add('should escape quotes when setting \'quote\' property is present', (t) => { const opts = { - fields: ['carModel', 'price', 'extras.items.name', 'extras.items.color', 'extras.items.items.position', 'extras.items.items.color'], - transforms: [unwind()], + fields: ['carModel', 'color'], + formatters: { + string: stringFormatter({ quote: '\'' }) + } }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.unwind2().pipe(transform); + const processor = jsonFixtures.escapeCustomQuotes().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.unwind2); + t.equal(csv, csvFixtures.escapeCustomQuotes); t.end(); }) .on('error', err => { @@ -1145,20 +1207,21 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should support unwinding specific fields using the unwind transform', (t) => { + testRunner.add('should not escape \'"\' when setting \'quote\' set to something else', (t) => { const opts = { - fields: ['carModel', 'price', 'colors'], - transforms: [unwind({ paths: ['colors'] })], + formatters: { + string: stringFormatter({ quote: '\'' }) + } }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.unwind().pipe(transform); + const processor = jsonFixtures.escapedQuotes().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.unwind); + t.equal(csv, csvFixtures.escapedQuotesUnescaped); t.end(); }) .on('error', err => { @@ -1167,20 +1230,75 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should support multi-level unwind using the unwind transform', (t) => { + // String Escaped Quote + + testRunner.add('should escape quotes with double quotes', (t) => { + const transform = new Json2csvTransform(); + const processor = jsonFixtures.quotes().pipe(transform); + + let csv = ''; + processor + .on('data', chunk => (csv += chunk.toString())) + .on('end', () => { + t.equal(csv, csvFixtures.quotes); + t.end(); + }) + .on('error', err => { + t.fail(err.message); + t.end(); + }); + }); + + testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end', (t) => { + const transform = new Json2csvTransform(); + const processor = jsonFixtures.backslashAtEnd().pipe(transform); + + let csv = ''; + processor + .on('data', chunk => (csv += chunk.toString())) + .on('end', () => { + t.equal(csv, csvFixtures.backslashAtEnd); + t.end(); + }) + .on('error', err => { + t.fail(err.message); + t.end(); + }); + }); + + testRunner.add('should not escape quotes with double quotes, when there is a backslash in the end, and its not the last column', (t) => { + const transform = new Json2csvTransform(); + const processor = jsonFixtures.backslashAtEndInMiddleColumn().pipe(transform); + + let csv = ''; + processor + .on('data', chunk => (csv += chunk.toString())) + .on('end', () => { + t.equal(csv, csvFixtures.backslashAtEndInMiddleColumn); + t.end(); + }) + .on('error', err => { + t.fail(err.message); + t.end(); + }); + }); + + testRunner.add('should escape quotes with value in \'escapedQuote\'', (t) => { const opts = { - fields: ['carModel', 'price', 'extras.items.name', 'extras.items.color', 'extras.items.items.position', 'extras.items.items.color'], - transforms: [unwind({ paths: ['extras.items', 'extras.items.items'] })], + fields: ['a string'], + formatters: { + string: stringFormatter({ escapedQuote: '*' }) + } }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.unwind2().pipe(transform); + const processor = jsonFixtures.escapedQuotes().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.unwind2); + t.equal(csv, csvFixtures.escapedQuotes); t.end(); }) .on('error', err => { @@ -1189,20 +1307,19 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should support unwind and blank out repeated data using the unwind transform', (t) => { + testRunner.add('should escape quotes before new line with value in \'escapedQuote\'', (t) => { const opts = { - fields: ['carModel', 'price', 'extras.items.name', 'extras.items.color', 'extras.items.items.position', 'extras.items.items.color'], - transforms: [unwind({ paths: ['extras.items', 'extras.items.items'], blankOut: true })], + fields: ['a string'] }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.unwind2().pipe(transform); + const processor = jsonFixtures.backslashBeforeNewLine().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.unwind2Blank); + t.equal(csv, csvFixtures.backslashBeforeNewLine); t.end(); }) .on('error', err => { @@ -1211,19 +1328,23 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should support flattening deep JSON using the flatten transform', (t) => { + // String Quote Only if Necessary + + testRunner.add('should quote only if necessary if using stringQuoteOnlyIfNecessary formatter', async (t) => { const opts = { - transforms: [flatten()], + formatters: { + string: stringQuoteOnlyIfNecessaryFormatter() + } }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.deepJSON().pipe(transform); + const processor = jsonFixtures.quoteOnlyIfNecessary().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.flattenedDeepJSON); + t.equal(csv, csvFixtures.quoteOnlyIfNecessary); t.end(); }) .on('error', err => { @@ -1232,19 +1353,24 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should support flattening JSON with nested arrays using the flatten transform', (t) => { + // String Excel + + testRunner.add('should format strings to force excel to view the values as strings', (t) => { const opts = { - transforms: [flatten({ arrays: true })], + fields: ['carModel', 'price', 'color'], + formatters: { + string: stringExcelFormatter() + } }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.flattenArrays().pipe(transform); + const processor = jsonFixtures.default().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.flattenedArrays); + t.equal(csv, csvFixtures.excelStrings); t.end(); }) .on('error', err => { @@ -1253,19 +1379,21 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should support custom flatten separator using the flatten transform', (t) => { + // String Escaping and preserving values + + testRunner.add('should parse JSON values with trailing backslashes', (t) => { const opts = { - transforms: [flatten({ separator: '__' })], + fields: ['carModel', 'price', 'color'] }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.deepJSON().pipe(transform); + const processor = jsonFixtures.trailingBackslash().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.flattenedCustomSeparatorDeepJSON); + t.equal(csv, csvFixtures.trailingBackslash); t.end(); }) .on('error', err => { @@ -1274,19 +1402,40 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should support multiple transforms and honor the order in which they are declared', (t) => { + testRunner.add('should escape " when preceeded by \\', (t) => { + const transform = new Json2csvTransform(); + const processor = jsonFixtures.escapeDoubleBackslashedEscapedQuote().pipe(transform); + + let csv = ''; + processor + .on('data', chunk => (csv += chunk.toString())) + .on('end', () => { + t.equal(csv, csvFixtures.escapeDoubleBackslashedEscapedQuote); + t.end(); + }) + .on('error', err => { + t.fail(err.message); + t.end(); + }); + }); + + testRunner.add('should preserve new lines in values', (t) => { const opts = { - transforms: [unwind({ paths: ['items'] }), flatten()], + eol: '\r\n' }; const transform = new Json2csvTransform(opts); - const processor = jsonFixtures.unwindAndFlatten().pipe(transform); + const processor = jsonFixtures.escapeEOL().pipe(transform); let csv = ''; processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.unwindAndFlatten); + t.equal(csv, [ + '"a string"', + '"with a \u2028description\\n and\na new line"', + '"with a \u2029\u2028description and\r\nanother new line"' + ].join('\r\n')); t.end(); }) .on('error', err => { @@ -1295,14 +1444,31 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = }); }); - testRunner.add('should support custom transforms', (t) => { + testRunner.add('should preserve tabs in values', (t) => { + const transform = new Json2csvTransform(); + const processor = jsonFixtures.escapeTab().pipe(transform); + + let csv = ''; + processor + .on('data', chunk => (csv += chunk.toString())) + .on('end', () => { + t.equal(csv, csvFixtures.escapeTab); + t.end(); + }) + .on('error', err => { + t.fail(err.message); + t.end(); + }); + }); + + // Headers + + testRunner.add('should format headers based on the headers formatter', async (t) => { const opts = { - transforms: [row => ({ - model: row.carModel, - price: row.price / 1000, - color: row.color, - transmission: row.transmission || 'automatic', - })], + fields: ['carModel', 'price', 'color', 'manual'], + formatters: { + header: stringFormatter({ quote: '' }) + } }; const transform = new Json2csvTransform(opts); @@ -1312,7 +1478,7 @@ module.exports = (testRunner, jsonFixtures, csvFixtures, inMemoryJsonFixtures) = processor .on('data', chunk => (csv += chunk.toString())) .on('end', () => { - t.equal(csv, csvFixtures.defaultCustomTransform); + t.equal(csv, csvFixtures.customHeaderQuotes); t.end(); }) .on('error', err => { diff --git a/test/fixtures/csv/customHeaderQuotes.csv b/test/fixtures/csv/customHeaderQuotes.csv new file mode 100644 index 00000000..db523f65 --- /dev/null +++ b/test/fixtures/csv/customHeaderQuotes.csv @@ -0,0 +1,5 @@ +carModel,price,color,manual +"Audi",0,"blue", +"BMW",15000,"red",true +"Mercedes",20000,"yellow", +"Porsche",30000,"green", \ No newline at end of file diff --git a/test/fixtures/csv/default.csv b/test/fixtures/csv/default.csv index 5356fb4b..ab6c55e2 100644 --- a/test/fixtures/csv/default.csv +++ b/test/fixtures/csv/default.csv @@ -1,5 +1,5 @@ -"carModel","price","color","transmission" +"carModel","price","color","manual" "Audi",0,"blue", -"BMW",15000,"red","manual" +"BMW",15000,"red",true "Mercedes",20000,"yellow", "Porsche",30000,"green", \ No newline at end of file diff --git a/test/fixtures/csv/defaultCustomTransform.csv b/test/fixtures/csv/defaultCustomTransform.csv index 23da3c96..95ba9b4a 100644 --- a/test/fixtures/csv/defaultCustomTransform.csv +++ b/test/fixtures/csv/defaultCustomTransform.csv @@ -1,5 +1,5 @@ -"model","price","color","transmission" +"model","price","color","manual" "Audi",0,"blue","automatic" -"BMW",15,"red","manual" +"BMW",15,"red",true "Mercedes",20,"yellow","automatic" "Porsche",30,"green","automatic" \ No newline at end of file diff --git a/test/fixtures/csv/ndjson.csv b/test/fixtures/csv/ndjson.csv index 5356fb4b..ab6c55e2 100644 --- a/test/fixtures/csv/ndjson.csv +++ b/test/fixtures/csv/ndjson.csv @@ -1,5 +1,5 @@ -"carModel","price","color","transmission" +"carModel","price","color","manual" "Audi",0,"blue", -"BMW",15000,"red","manual" +"BMW",15000,"red",true "Mercedes",20000,"yellow", "Porsche",30000,"green", \ No newline at end of file diff --git a/test/fixtures/csv/numberCustomSeparator.csv b/test/fixtures/csv/numberCustomSeparator.csv new file mode 100644 index 00000000..bc452672 --- /dev/null +++ b/test/fixtures/csv/numberCustomSeparator.csv @@ -0,0 +1,2 @@ +"number1";"number2";"number3" +0,12341234;2,1;65 \ No newline at end of file diff --git a/test/fixtures/csv/numberFixedDecimals.csv b/test/fixtures/csv/numberFixedDecimals.csv new file mode 100644 index 00000000..470e8046 --- /dev/null +++ b/test/fixtures/csv/numberFixedDecimals.csv @@ -0,0 +1,2 @@ +"number1","number2","number3" +0.12,2.10,65.00 \ No newline at end of file diff --git a/test/fixtures/csv/numberFixedDecimalsAndCustomSeparator.csv b/test/fixtures/csv/numberFixedDecimalsAndCustomSeparator.csv new file mode 100644 index 00000000..1ae4bd85 --- /dev/null +++ b/test/fixtures/csv/numberFixedDecimalsAndCustomSeparator.csv @@ -0,0 +1,2 @@ +"number1";"number2";"number3" +0,12;2,10;65,00 \ No newline at end of file diff --git a/test/fixtures/csv/quoteOnlyIfNecessary.csv b/test/fixtures/csv/quoteOnlyIfNecessary.csv new file mode 100644 index 00000000..aa9a4c07 --- /dev/null +++ b/test/fixtures/csv/quoteOnlyIfNecessary.csv @@ -0,0 +1,6 @@ +a string +with a description +"with a description and ""quotes""" +"with a description, and a separator" +"with a description + and a new line" \ No newline at end of file diff --git a/test/fixtures/csv/withBOM.csv b/test/fixtures/csv/withBOM.csv index b9077d55..3a01336f 100644 --- a/test/fixtures/csv/withBOM.csv +++ b/test/fixtures/csv/withBOM.csv @@ -1,5 +1,5 @@ -"carModel","price","color","transmission" +"carModel","price","color","manual" "Audi",0,"blue", -"BMW",15000,"red","manual" +"BMW",15000,"red",true "Mercedes",20000,"yellow", "Citroën",30000,"green", \ No newline at end of file diff --git a/test/fixtures/csv/withoutHeader.csv b/test/fixtures/csv/withoutHeader.csv index a57a6e66..5d0bff7b 100644 --- a/test/fixtures/csv/withoutHeader.csv +++ b/test/fixtures/csv/withoutHeader.csv @@ -1,4 +1,4 @@ "Audi",0,"blue", -"BMW",15000,"red","manual" +"BMW",15000,"red",true "Mercedes",20000,"yellow", "Porsche",30000,"green", \ No newline at end of file diff --git a/test/fixtures/json/default.json b/test/fixtures/json/default.json index 75dcdd2b..82dccba0 100644 --- a/test/fixtures/json/default.json +++ b/test/fixtures/json/default.json @@ -1,6 +1,6 @@ [ { "carModel": "Audi", "price": 0, "color": "blue" }, - { "carModel": "BMW", "price": 15000, "color": "red", "transmission": "manual" }, + { "carModel": "BMW", "price": 15000, "color": "red", "manual": true }, { "carModel": "Mercedes", "price": 20000, "color": "yellow" }, { "carModel": "Porsche", "price": 30000, "color": "green" } ] diff --git a/test/fixtures/json/defaultInvalid.json b/test/fixtures/json/defaultInvalid.json index c30f6c47..222fe48d 100644 --- a/test/fixtures/json/defaultInvalid.json +++ b/test/fixtures/json/defaultInvalid.json @@ -1,6 +1,6 @@ [ { "carModel": "Audi", "price": 0, "color": "blue" }, - { "carModel": "BMW", "price": 15000, "color": "red", "transmission": "manual" }, + { "carModel": "BMW", "price": 15000, "color": "red", "manual": true }, { "carModel": "Mercedes", "price": 20000, "color": "yellow", { "carModel": "Porsche", "price": 30000, "color": "green" } ] diff --git a/test/fixtures/json/ndjson.json b/test/fixtures/json/ndjson.json index 5c4c11a0..6a6137ef 100644 --- a/test/fixtures/json/ndjson.json +++ b/test/fixtures/json/ndjson.json @@ -1,4 +1,4 @@ { "carModel": "Audi", "price": 0, "color": "blue" } -{ "carModel": "BMW", "price": 15000, "color": "red", "transmission": "manual" } +{ "carModel": "BMW", "price": 15000, "color": "red", "manual": true } { "carModel": "Mercedes", "price": 20000, "color": "yellow" } { "carModel": "Porsche", "price": 30000, "color": "green" } \ No newline at end of file diff --git a/test/fixtures/json/ndjsonInvalid.json b/test/fixtures/json/ndjsonInvalid.json index ae563c9d..22e2e730 100644 --- a/test/fixtures/json/ndjsonInvalid.json +++ b/test/fixtures/json/ndjsonInvalid.json @@ -1,4 +1,4 @@ { "carModel": "Audi", "price": 0, "color": "blue" } -{ "carModel": "BMW", "price": 15000, "color": "red", "transmission": "manual" } +{ "carModel": "BMW", "price": 15000, "color": "red", "manual": true } { "carModel": "Mercedes", "price": 20000, "color": "yellow" { "carModel": "Porsche", "price": 30000, "color": "green" } \ No newline at end of file diff --git a/test/fixtures/json/numberFormatter.json b/test/fixtures/json/numberFormatter.json new file mode 100644 index 00000000..d6acdd5c --- /dev/null +++ b/test/fixtures/json/numberFormatter.json @@ -0,0 +1,3 @@ +[ + { "number1": 0.12341234, "number2": 2.1, "number3": 65} +] \ No newline at end of file diff --git a/test/fixtures/json/quoteOnlyIfNecessary.json b/test/fixtures/json/quoteOnlyIfNecessary.json new file mode 100644 index 00000000..ff2d6516 --- /dev/null +++ b/test/fixtures/json/quoteOnlyIfNecessary.json @@ -0,0 +1,6 @@ +[ + {"a string": "with a description"}, + {"a string": "with a description and \"quotes\""}, + {"a string": "with a description, and a separator"}, + {"a string": "with a description\n and a new line"} +] \ No newline at end of file diff --git a/test/fixtures/json/specialCharacters.json b/test/fixtures/json/specialCharacters.json index e369e19f..063f9aee 100644 --- a/test/fixtures/json/specialCharacters.json +++ b/test/fixtures/json/specialCharacters.json @@ -8,7 +8,7 @@ "carModel": "BMW", "price": 15000, "color": "red", - "transmission": "manual" + "manual": true }, { "carModel": "Mercedes",