Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError: Invalid host defined options - Cannot Use TypeScript, JavaScript, ESM, and TS-Node #4594

Closed
loganknecht opened this issue Mar 3, 2021 · 16 comments
Labels
type: question support question

Comments

@loganknecht
Copy link

loganknecht commented Mar 3, 2021

Hello!

I tried reaching out on Gitter, but wasn't able to elicit any help.

I am in the process of trying to migrate my team's code base to TypeScript in small chunks. Which means I need to have JavaScript and TypeScript supported while this is performed. Additionally I am trying to align the standards for that move.

Is there anyone who can provide insight on how to get mocha to run for a typescript project that has Javascript in it using ESM module support?

Also quick disclaimer: This is not the palantir related to Peter Thiel :/, My team mates just really like Lord of the Rings and chose a poor name for our project.


I have a directory structure like so
All my test files have a .js extension on them

.
├── .mocharc.json
├── package.json
├── register.ts
├── src
│   └── ...
├── test
│   ├── runner.js
│   ├── tsconfig.json
│   └── ...
├── tsconfig.json
└── webpack.config.js

I have a package.json file with these contents

{
    "name": "palantir-api",
    // ...
    "scripts": {
        "build": "yarn transpile && webpack",
        "bumpversion": "yarn version --no-git-tag-version --no-commit-hooks",
        "dev": "export DB_HOST=localhost && nodemon ./src/app.js",
        "format": "prettier --write \"./**/*.md\" \"./**/*.json\" \"./**/*.js\" \"./**/*.ts\" \"./**/*.yaml\" \"./**/*.yml\"",
        "open_api": "redoc-cli bundle ./api/palantir_api.yaml --output ./docs/index.html",
        "preopen_api": "node -e \"console.log(require('./package.json').version)\" > api/version.yaml",
        "postbumpversion": "node -e \"console.log(require('./package.json').version)\" > VERSION",
        "prod": "node dist/api.bundle.js",
        "runner": "node --loader ts-node/esm ./test/runner.js",
        "start": "nodemon --exec babel-node ./src/app.js",
        "start_db_server": "pg_ctl -D /usr/local/var/postgres start",
        "stop_db_server": "pg_ctl -D /usr/local/var/postgres stop",
        "transpile": "tsc",
        "test": "yarn runner --t system",
        "unit_test": "nyc yarn runner --t unit"
    },
    // ...
    "dependencies": {
        "@babel/core": "^7.10.4",
        "@babel/node": "^7.10.4",
        "@babel/plugin-proposal-class-properties": "^7.10.4",
        "@babel/preset-env": "^7.10.4",
        "@hapi/joi": "^17.1.1",
        "@types/dotenv": "^8.2.0",
        "@types/express-xml-bodyparser": "^0.3.1",
        "bcrypt": "^5.0.0",
        "cors": "^2.8.5",
        "express": "^4.17.1",
        "express-validator": "^6.6.1",
        "express-winston": "^4.0.3",
        "express-xml-bodyparser": "^0.3.0",
        "jsonwebtoken": "^8.5.1",
        "nodemon": "^2.0.4",
        "pg": "^8.2.1",
        "serve-favicon": "^2.5.0",
        "trim-request-body": "^1.0.1",
        "uuid": "^8.2.0",
        "winston": "^3.3.3"
    },
    "devDependencies": {
        "@babel/plugin-transform-runtime": "^7.10.4",
        "@babel/polyfill": "^7.10.4",
        "@babel/preset-react": "^7.10.4",
        "@types/pg": "^7.14.10",
        "babel-loader": "^8.1.0",
        "babel-plugin-syntax-dynamic-import": "^6.18.0",
        "babel-plugin-transform-runtime": "^6.23.0",
        "chai": "^4.2.0",
        "chai-http": "^4.3.0",
        "esm": "^3.2.25",
        "faker": "^5.1.0",
        "mocha": "^8.0.1",
        "mocha-junit-reporter": "^2.0.0",
        "mocha-multi-reporters": "^1.1.7",
        "nyc": "^15.1.0",
        "palantir-test-data-helpers": "^0.0.4",
        "prettier": "^2.2.1",
        "redoc-cli": "^0.9.8",
        "reify": "^0.20.12",
        "sinon": "^9.0.2",
        "ts-node-dev": "1.1.1",
        "typescript": "4.1.3",
        "webpack": "^4.43.0",
        "webpack-cli": "^3.3.12",
        "webpack-node-externals": "^1.7.2",
        "yargs": "^15.4.0"
    },
    "type": "module"
}

I have a default tsconfig.json file with these contents

{
    "compilerOptions": {
        "allowJs": true,
        "allowSyntheticDefaultImports": true,
        "baseUrl": ".",
        "esModuleInterop": true,
        "forceConsistentCasingInFileNames": true,
        "lib": ["dom", "es6"],
        "module": "es6",
        "moduleResolution": "node",
        "noImplicitAny": true,
        "outDir": "./typescript_build",
        "paths": {
            "*": ["./node_modules/@types", "./custom_types"]
        },
        "resolveJsonModule": true,
        "skipLibCheck": true,
        "strict": true,
        "strictPropertyInitialization": false,
        "target": "es6"
    }
}

Which is inherited by the test/tsconfig.json and looks like this

{
    "extends": "../tsconfig.json",
    "compilerOptions": {
        "baseUrl": ".",
        "experimentalDecorators": true,
        "isolatedModules": false,
        "paths": {
            "*": ["../node_modules/@types", "../custom_types"]
        },
        "resolveJsonModule": true
    },
    "exclude": ["../node_modules"],
    "include": ["./**/*.ts", "../custom_types"]
}

And then I have a .mocharc.json configured with these options

{
    "extension": ["js", "ts"],
    "resolveJsonModule": true,
    "require": ["./register.ts", "esm", "ts-node/register"]
}

And finally I have a register.ts file with these configurations

import * as tsNode from "ts-node";

import testTSConfig from "./test/tsconfig.json";

tsNode.register({
    files: true,
    project: "./test/tsconfig.json"
});

Additionally there is a webpack file here:

import path from "path";
import nodeExternals from "webpack-node-externals";

module.exports = {
    entry: "./typescript_build/src/app.js",
    output: {
        path: path.resolve(__dirname, "dist"),
        filename: "api.bundle.js"
    },
    target: "node",
    mode: "production",
    node: {
        // Need this when working with express, otherwise the build fails
        __dirname: false, // if you don't put this is, __dirname
        __filename: false // and __filename return blank or /
    },
    externals: [nodeExternals()], // Need this to avoid error when working with Express
    module: {
        rules: [
            {
                test: /(\.js[\S]{0,1})$/i,
                exclude: /node_modules/,
                loader: "babel-loader",
                query: {
                    presets: ["@babel/preset-react", "@babel/preset-env"],
                    plugins: ["@babel/proposal-class-properties"]
                }
            }
        ]
    }
};

When I run this command

env TS_NODE_PROJECT=test/tsconfig.json \
yarn run mocha --config ./.mocharc.json --test unit

I am provided this output

 ✘ lknecht  ~/Repositories/palantir-api   do_not_use/prototyping_mocha_typescript-dev ±✚  ⬡ v12.17.0  env TS_NODE_PROJECT=test/tsconfig.json \
yarn run mocha --config ./.mocharc.json --test system --full-trace

yarn run v1.22.10
$ /Users/lknecht/Repositories/palantir-api/node_modules/.bin/mocha --config ./.mocharc.json --test system --full-trace

TypeError: Invalid host defined options
    at formattedImport (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/esm-utils.js:6:23)
    at Object.exports.requireOrImport (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/esm-utils.js:23:14)
    at Object.exports.loadFilesAsync (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/esm-utils.js:33:34)
    at Mocha.loadFilesAsync (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/mocha.js:431:19)
    at singleRun (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/cli/run-helpers.js:125:15)
    at exports.runMocha (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/cli/run-helpers.js:190:10)
    at Object.exports.handler (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/cli/run.js:362:11)
    at /Users/lknecht/Repositories/palantir-api/node_modules/mocha/node_modules/yargs/lib/command.js:241:49
    at process.runNextTicks [as _tickCallback] (internal/process/task_queues.js:62:5)
    at /Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:34535
    at /Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:34176
    at process.<anonymous> (/Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:34506)
    at Function.<anonymous> (/Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:296856)
    at Function.<anonymous> (/Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:296555)
    at Function.<anonymous> (/Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:284879)
    at Object.apply (/Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:199341)
    at internal/main/run_main_module.js:17:47
error Command failed with exit code 1.

I have been testing different configurations non-stop trying to get

  • mocha
  • javascript
  • typescript
  • ecma script modules
  • etc

to all play nicely.

By default I CAN get the api to stand itself up, but the second mocha gets involved this blows up! :(

Can anyone provide any guidance?

@loganknecht loganknecht added the type: question support question label Mar 3, 2021
@loganknecht loganknecht changed the title Cannot Use TypeScript, JavaScript, ESM, and TS-Node TypeError: Invalid host defined options - Cannot Use TypeScript, JavaScript, ESM, and TS-Node Mar 3, 2021
@juergba
Copy link
Contributor

juergba commented Mar 4, 2021

I can't really help with that, but there are two details (?) which are/could be wrong.
You are mixing Node's ESM implementation with the esm module. The results are unpredictable and IMO you should remove the --require esm.
Then resolveJsonModule is not a Mocha option.

@loganknecht
Copy link
Author

@juergba Thanks for the advice!

I changed the .mocharc.json to be

{
    "extension": ["js", "ts"],
    "require": ["./register.ts", "ts-node/register"]
}

Then when I run it I get

lknecht  ~/Repositories/palantir-api   do_not_use/prototyping_mocha_typescript-dev ±  ⬡ v12.17.0  env TS_NODE_PROJECT=test/tsconfig.json \
yarn run mocha --config ./.mocharc.json --test unit

yarn run v1.22.10
$ /Users/lknecht/Repositories/palantir-api/node_modules/.bin/mocha --config ./.mocharc.json --test unit

✖ ERROR: /Users/lknecht/Repositories/palantir-api/register.ts:1
import * as tsNode from 'ts-node';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at wrapSafe (internal/modules/cjs/loader.js:1054:16)
    at Module._compile (internal/modules/cjs/loader.js:1102:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)
    at Module.load (internal/modules/cjs/loader.js:986:32)
    at Function.Module._load (internal/modules/cjs/loader.js:879:14)
    at Module.require (internal/modules/cjs/loader.js:1026:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at exports.requireOrImport (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/esm-utils.js:20:12)
    at exports.handleRequires (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/cli/run-helpers.js:94:34)
    at async /Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/cli/run.js:341:25
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I know that this is the issue with ESM not being used. But I have it configured in my package.json by using the line "type": "module". This leads me to believe I need to have esm in the .mochars.json

When I add ESM back in it gives me this output

{
    "extension": ["js", "ts"],
    "require": ["./register.ts", "esm", "ts-node/register"]
}
 ✘ lknecht  ~/Repositories/palantir-api   do_not_use/prototyping_mocha_typescript-dev ±  ⬡ v12.17.0  env TS_NODE_PROJECT=test/tsconfig.json \
yarn run mocha --config ./.mocharc.json --test unit

yarn run v1.22.10
$ /Users/lknecht/Repositories/palantir-api/node_modules/.bin/mocha --config ./.mocharc.json --test unit

TypeError: Invalid host defined options
    at formattedImport (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/esm-utils.js:6:23)
    at Object.exports.requireOrImport (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/esm-utils.js:23:14)
    at Object.exports.loadFilesAsync (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/esm-utils.js:33:34)
    at Mocha.loadFilesAsync (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/mocha.js:431:19)
    at singleRun (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/cli/run-helpers.js:125:15)
    at exports.runMocha (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/cli/run-helpers.js:190:10)
    at Object.exports.handler (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/cli/run.js:362:11)
    at /Users/lknecht/Repositories/palantir-api/node_modules/mocha/node_modules/yargs/lib/command.js:241:49
    at process.runNextTicks [as _tickCallback] (internal/process/task_queues.js:62:5)
    at /Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:34535
    at /Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:34176
    at process.<anonymous> (/Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:34506)
    at Function.<anonymous> (/Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:296856)
    at Function.<anonymous> (/Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:296555)
    at Function.<anonymous> (/Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:284879)
    at Object.apply (/Users/lknecht/Repositories/palantir-api/node_modules/esm/esm.js:1:199341)
    at internal/main/run_main_module.js:17:47
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Do you have any guidance on these configurations?

@juergba
Copy link
Contributor

juergba commented Mar 5, 2021

I strongly advise against using the esm package, and certainly not both at the same time. esm was excellent when we were young and beautiful, now you should go with Node's native ESM implementation.

see ts-node/esm, maybe @cspotcode can give you a hint.

import * as tsNode from 'ts-node'; I don't know wether ts-node is a CommonJS module. If so, then only default imports are supported.

@cspotcode
Copy link
Contributor

cspotcode commented Mar 5, 2021

On node, you have 2x options:

  • Write native ECMAScript module syntax and run it via node's CommonJS support
  • Write native ECMAScript module syntax and run it via node's new native ESM support

You should commit to one or the other, and then I can give you a configuration that will work. The first option -- using node's CommonJS support -- is by far the recommended option.

If you absolutely must use the second option -- native ESM support -- then do the following:

Remove the esm module
Stop requiring ts-node/register
Run mocha like this:
TS_NODE_PROJECT=absolutepathtotsconfigfile "NODE_OPTIONS=--loader ts-node/esm" mocha
TS_NODE_PROJECT is optional if you want it to load $PWD/tsconfig.json because that file will be auto-detected.
You can get rid of register.ts
Upgrade your node version. Invalid host defined options is a confusing node error which may be improved in a newer version.

@cspotcode
Copy link
Contributor

I've shared a complete, working example here, with logs to prove that it works.
TypeStrong/ts-node#1268

@loganknecht
Copy link
Author

@cspotcode I am committing to working with node's new native ESM support

I will look at the example and report back!

@juergba
Copy link
Contributor

juergba commented Mar 5, 2021

@cspotcode thanks for your support.

@loganknecht
Copy link
Author

Hrm...

I really appreciate the repo to compare against.

I have updated my tslint.config to

{
    "compilerOptions": {
        "allowJs": true,
        "allowSyntheticDefaultImports": true,
        "baseUrl": ".",
        "esModuleInterop": true,
        "forceConsistentCasingInFileNames": true,
        "module": "ES2020",
        "moduleResolution": "node",
        "noImplicitAny": true,
        "outDir": "./typescript_build",
        "paths": {
            "*": ["./node_modules/@types", "./custom_types"]
        },
        "resolveJsonModule": true,
        "skipLibCheck": true,
        "strict": true,
        "strictPropertyInitialization": false,
        "target": "ES2020"
    }
}

I have changed my .mocharcjson to this

{
    "extension": ["js", "ts"],
    "spec": ["test/**.{js,ts}"],
    "loader": ["ts-node/esm"]
}

And now when I run I get the dreaded ESM ERROR for my tests files that are .js 🤔

 ✘ lknecht  ~/Repositories/palantir-api   do_not_use/prototyping_mocha_typescript-dev ±  ⬡ v12.17.0  env TS_NODE_PROJECT=test/tsconfig.json \
yarn run mocha --config ./.mocharc.json --test unit

yarn run v1.22.10
$ /Users/lknecht/Repositories/palantir-api/node_modules/.bin/mocha --config ./.mocharc.json --test unit
(node:60415) ExperimentalWarning: The ESM module loader is experimental.
(node:60415) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
{"message":"Something caused the test to crash.","level":"error"}
{"code":"ERR_REQUIRE_ESM","level":"error"}
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /Users/lknecht/Repositories/palantir-api/test/unit/helper_controller.test.js
require() of ES modules is not supported.
require() of /Users/lknecht/Repositories/palantir-api/test/unit/helper_controller.test.js from /Users/lknecht/Repositories/palantir-api/node_modules/ts-node/dist/index.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead change the requiring code to use import(), or remove "type": "module" from /Users/lknecht/Repositories/palantir-api/package.json.

    at createErrRequireEsm (/Users/lknecht/Repositories/palantir-api/node_modules/ts-node/dist-raw/node-cjs-loader-utils.js:86:15)
    at assertScriptCanLoadAsCJSImpl (/Users/lknecht/Repositories/palantir-api/node_modules/ts-node/dist-raw/node-cjs-loader-utils.js:19:11)
    at assertScriptCanLoadAsCJS (/Users/lknecht/Repositories/palantir-api/node_modules/ts-node/src/index.ts:31:3)
    at Object.require.extensions.<computed> [as .js] (/Users/lknecht/Repositories/palantir-api/node_modules/ts-node/src/index.ts:1048:7)
    at Module.load (internal/modules/cjs/loader.js:986:32)
    at Function.Module._load (internal/modules/cjs/loader.js:879:14)
    at Module.require (internal/modules/cjs/loader.js:1026:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at /Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/mocha.js:394:36
    at Array.forEach (<anonymous>)
    at Mocha.loadFiles (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/mocha.js:391:14)
    at Mocha.run (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/mocha.js:970:10)
    at main (file:///Users/lknecht/Repositories/palantir-api/test/runner.js:246:15)
    at file:///Users/lknecht/Repositories/palantir-api/test/runner.js:283:1
    at ModuleJob.run (internal/modules/esm/module_job.js:110:37)
    at Loader.import (internal/modules/esm/loader.js:179:24)
    at Object.exports.loadFilesAsync (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/esm-utils.js:33:20)
    at singleRun (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/cli/run-helpers.js:125:3)
    at Object.exports.handler (/Users/lknecht/Repositories/palantir-api/node_modules/mocha/lib/cli/run.js:362:5) {
  code: 'ERR_REQUIRE_ESM',
  level: 'error',
  [Symbol(level)]: 'error',
  [Symbol(message)]: '{"code":"ERR_REQUIRE_ESM","level":"error"}'
}

I will continue you to try and figure out where this is coming from

@cspotcode and @juergba I really appreciate all the input. And lol at the young and beautiful above 😂

@cspotcode
Copy link
Contributor

What is in helper_controller.test.js?

@cspotcode
Copy link
Contributor

Eh, nevermind, that error is happening because mocha is trying to require() an ESM file instead of import() it. I would use that stack trace to figure out exactly where and why.

mocha requires files and detects certain errors to know if it should subsequently import them. ts-node must mimic the error that node throws in this situation. It's possible there's an incompatibility between mocha and our mimicry that needs to be addressed.

@loganknecht
Copy link
Author

@cspotcode

This is the contents of helper_controller.test.js

import chai from 'chai';
import sinon from 'sinon';
import { v4 as uuid } from 'uuid';

import { NewStubs } from './mock.js';
import ControllerHelper from '../../src/utilities/controller_helper.js';
import { dbTable } from '../../src/utilities/data_helper.js';

describe('Controller Helper', () => {
    let controller, dbConnectorStub, resStub;

    const testProject = { name: 'test.quick.fox', projectid: uuid() };
    const projectIdParam = { name: 'projectid', value: testProject.projectid };

    const suiteid = uuid();
    const suiteIdParam = { name: 'suiteid', value: suiteid };

    const testrunid = uuid();
    const testrunIdParam = { name: 'testrunid', value: testrunid };

    beforeEach(() => {
        let stubs = NewStubs();

        // Assign the stubs
        dbConnectorStub = stubs.dbConnector;
        resStub = stubs.res;

        // instantiate the controller we are testing with the connector stub and joi schema validator
        controller = new ControllerHelper(dbConnectorStub);
    });

    describe('getall', () => {
        const testData = [
            { title: 'when empty', rows: [] },
            { title: 'when 1 project is available', rows: [testProject] },
        ];

        testData.forEach((test) => {
            it('retrieve all projects ' + test.title, async () => {
                // Arrange
                dbConnectorStub.getAll.returns({ rows: test.rows });

                // Act
                const projectRows = await controller.listProjects();

                // Assert the call on dbConnector.getAll
                sinon.assert.calledOnce(dbConnectorStub.getAll);
                sinon.assert.calledWith(dbConnectorStub.getAll, 'project');

                // Assert payload returned
                chai.assert.equal(projectRows, test.rows);
            });
        });
    });

    describe('deleteProject', () => {
        it('Delete Project with no suites', async () => {
            // Arrange
            dbConnectorStub.getAll.returns({ rows: [testProject] });

            dbConnectorStub.getWithParams.withArgs(dbTable.suite, projectIdParam).returns({ rows: [] });

            dbConnectorStub.delete.returns({});

            // Act
            await controller.deleteProject(testProject.projectid);

            // Assert delete project
            sinon.assert.calledOnceWithExactly(dbConnectorStub.delete, dbTable.project, projectIdParam);

            // Assert suite delete was not invoked
            // not sure about this one
            sinon.assert.neverCalledWith(dbConnectorStub.delete, dbTable.suite, suiteIdParam);
        });

        it('Delete Project with one suite', async () => {
            // Arrange getAll projects
            dbConnectorStub.getAll.returns({ rows: [testProject] });

            // Arrange get suites for project
            dbConnectorStub.getWithParams
                .withArgs(dbTable.suite, projectIdParam)
                .returns({ rows: [{ suiteid: suiteid }] });

            // Arrange get testrunID for suite
            dbConnectorStub.getWithParams.withArgs(dbTable.testrun, suiteIdParam).returns({ rows: [] });

            dbConnectorStub.delete.returns({});

            // Act
            await controller.deleteProject(testProject.projectid);

            // Assert

            sinon.assert.calledWithMatch(dbConnectorStub.getWithParams, dbTable.suite, projectIdParam);

            sinon.assert.calledWithMatch(dbConnectorStub.delete, dbTable.project, projectIdParam);

            sinon.assert.neverCalledWithMatch(dbConnectorStub.delete, dbTable.testresult, testrunIdParam);

            sinon.assert.calledWithMatch(dbConnectorStub.delete, dbTable.test, suiteIdParam);

            sinon.assert.calledWithMatch(dbConnectorStub.delete, dbTable.testrun, suiteIdParam);

            sinon.assert.calledWithMatch(dbConnectorStub.delete, dbTable.suite, suiteIdParam);

            sinon.assert.callCount(dbConnectorStub.delete, 4);
        });

        it('Delete Project with one suite and testrunID', async () => {
            // Arrange getAll projects
            dbConnectorStub.getAll.returns({ rows: [testProject] });

            // Arrange get suites for project
            dbConnectorStub.getWithParams
                .withArgs(dbTable.suite, projectIdParam)
                .returns({ rows: [{ suiteid: suiteid }] });

            // Arrange get testrunID for suite
            dbConnectorStub.getWithParams
                .withArgs(dbTable.testrun, suiteIdParam)
                .returns({ rows: [{ testrunid: testrunid }] });

            dbConnectorStub.delete.returns({});

            // Act
            await controller.deleteProject(testProject.projectid);

            // Assert

            sinon.assert.calledWithMatch(dbConnectorStub.getWithParams, dbTable.suite, projectIdParam);

            sinon.assert.calledWithMatch(dbConnectorStub.delete, dbTable.project, projectIdParam);

            sinon.assert.calledWithMatch(dbConnectorStub.delete, dbTable.testresult, testrunIdParam);

            sinon.assert.calledWithMatch(dbConnectorStub.delete, dbTable.test, suiteIdParam);

            sinon.assert.calledWithMatch(dbConnectorStub.delete, dbTable.testrun, suiteIdParam);

            sinon.assert.calledWithMatch(dbConnectorStub.delete, dbTable.suite, suiteIdParam);

            sinon.assert.callCount(dbConnectorStub.delete, 5);
        });
    });
});

I don't think that's the issue though.

It appears as though my team has created a runner for mocha in runner.js and I think that's where it's getting munged.

This is what's it's runner.js

import chai from 'chai';
import chaiHttp from 'chai-http';
import dotenv from 'dotenv';
import fs from 'fs';
import Mocha from 'mocha';
import path from 'path';
import yargs from 'yargs';

import { logger } from '../src/winston/config.ts';

dotenv.config();
chai.use(chaiHttp);

const argv = yargs
    .usage('To run locally : runner.sh [options]')
    .usage('To push results: runner.sh [options] [push options]')
    .option('test', {
        description: 'type of test',
        alias: 't',
        type: 'string',
        choices: ['unit', 'system'],
    })
    .option('reporterURL', {
        description: 'the URL to submit reports to',
        alias: 'u',
        type: 'string',
        requiresArg: true,
        group: 'push',
    })
    .option('suiteID', {
        description: 'the SuiteID in palantir mapped to the run',
        alias: 's',
        type: 'string',
        requiresArg: true,
        group: 'push',
    })
    .option('suitesTitle', {
        description: 'the testSuitesTitle for mocha-junit-reporter',
        alias: 'x',
        type: 'string',
        requiresArg: true,
    })
    .option('commitSHA', {
        description: 'the commit sha associated to this run',
        alias: 'c',
        type: 'string',
        requiresArg: true,
        default: null,
        group: 'push',
    })
    .option('reporterType', {
        description: 'the mocha reporter preset type',
        alias: 'r',
        type: 'string',
        default: 'multi-junit',
    })
    .option('reportPath', {
        hidden: true,
        default: 'test-result.xml',
    })
    .wrap(120)
    .demandOption(['t'])
    .check((argv, options) => {
        if (
            argv.commitSHA === null &&
            process.env.CI_COMMIT_SHORT_SHA !== null &&
            process.env.CI_COMMIT_SHORT_SHA !== undefined
        ) {
            argv.commitSHA = process.env.CI_COMMIT_SHORT_SHA;
        }

        if (!argv.reporterURL && !argv.suiteID) {
            return true;
        } else if (argv.reporterURL && argv.suiteID) {
            return true;
        }

        throw new Error('Missing or extra values');
    })
    .help()
    .alias('help', 'h').argv;

async function WaitForAPI() {
    let server = process.env.API_HOST + ':' + process.env.API_PORT;

    // In pipeline and when talking to deployments, the API_PORT is left blank
    // hence it is modified
    if (process.env.API_PORT === undefined || process.env.API_PORT === '') {
        server = process.env.API_HOST;
    }

    logger.info('Targeting: ' + server);

    let result = false;
    for (let i = 0; i < 15; i++) {
        result = await chai
            .request(server)
            .get('/api/info')
            .then((res) => {
                logger.info('/api/info ' + res.status);
                if (res.status !== 200) {
                    return false;
                }
                return true;
            })
            .catch((err) => {
                logger.error(err);
                return false;
            });

        if (result) {
            break;
        }

        await new Promise((resolve) => setTimeout(resolve, 1000));
    }

    if (!result) {
        logger.error('Could not reach the service in time');
        return null;
    }

    result = false;
    for (let i = 0; i < 15; i++) {
        result = await chai
            .request(server)
            .get('/api/init')
            .then((res) => {
                logger.info('/api/init ' + res.status);

                if (res.status !== 200) {
                    return false;
                }
                return true;
            })
            .catch((err) => {
                logger.error(err);
                return false;
            });

        if (result) {
            break;
        }

        await new Promise((resolve) => setTimeout(resolve, 1000));
    }

    if (!result) {
        logger.error('DB was not initialized properly in time');
        return null;
    }

    return server;
}

function NewMocha(mochaType) {
    let mocha;
    switch (mochaType) {
        case 'xunit':
            mocha = new Mocha({
                fullTrace: true,
                reporter: 'xunit',
                reporterOptions: {
                    suiteName: argv.suitesTitle,
                    output: process.env.REPORT_PATH,
                    properties: {
                        COMMIT_SHA: argv.commitSHA,
                    },
                },
            });
            argv.reportPath = process.env.REPORT_PATH;
            break;
        case 'multi-xunit':
            mocha = new Mocha({
                fullTrace: true,
                reporter: 'mocha-multi-reporters',
                reporterOptions: {
                    reporterEnabled: 'xunit,spec',
                    xunitReporterOptions: {
                        output: process.env.REPORT_PATH,
                        suiteName: argv.suitesTitle,
                    },
                },
            });
            argv.reportPath = process.env.REPORT_PATH;
            break;
        case 'multi-junit':
            mocha = new Mocha({
                fullTrace: true,
                reporter: 'mocha-multi-reporters',
                reporterOptions: {
                    reporterEnabled: 'mocha-junit-reporter,spec',
                    mochaJunitReporterReporterOptions: {
                        mochaFile: process.env.REPORT_PATH,
                        rootSuiteTitle: 'The root!',
                        testsuitesTitle: argv.suitesTitle,
                        properties: {
                            COMMIT_SHA: argv.commitSHA,
                        },
                    },
                },
            });
            argv.reportPath = process.env.REPORT_PATH;
            break;
        default:
            throw new Error('Not a mocha configuration supported');
    }

    return mocha;
}

async function main() {
    if (!argv.suitesTitle) {
        argv.suitesTitle = process.env.SUITES_TITLE_UNIT;
        if (argv.t === 'system') {
            argv.suitesTitle = process.env.SUITES_TITLE_SYSTEM;
        }
    }

    if (argv.t === 'system') {
        let serv = await WaitForAPI();
        if (serv === null) {
            process.exit(1);
        }

        logger.info(serv);
    }

    // Instantiate a Mocha instance.
    var mocha = NewMocha(argv.reporterType);

    var testDir = 'test/' + argv.test;

    // Add each .js file to the mocha instance
    fs.readdirSync(testDir)
        .filter(function (file) {
            // Only keep the .js files
            return path.extname(file) === '.js';
        })
        .forEach(function (file) {
            mocha.addFile(path.join(testDir, file));
        });

    try {
        // Run the tests.
        mocha.run(function (failures) {
            process.exitCode = failures ? 1 : 0; // exit with non-zero status if there were failures

            if (argv.reporterURL) {
                const content = fs.readFileSync(argv.reportPath, { encoding: 'utf8' });
                let req = chai
                    .request(argv.reporterURL)
                    .post('/api/suites/' + argv.suiteID + '/test-runs')
                    .send(content)
                    .set('Content-Type', 'application/xml;charset=utf-8');

                if (argv.commitSHA !== null) {
                    req = req.set('commitsha', argv.commitSHA);
                }

                req.then((res) => {
                    if (res.status === 200) {
                        logger.info('successfully posted result to palantir');
                        logger.info(res.text);
                    } else {
                        logger.warn('Issues posting the results. Status Code: ' + res.status);
                        logger.warn(res.text);
                    }
                }).catch((err) => {
                    logger.error('failed to submit result to palantir');
                    logger.error(err);
                });
            }
        });
    } catch (error) {
        logger.error('Something caused the test to crash.');
        logger.error(error);
        console.log(error);
        process.exit(1);
    }
}

main();

Do you know if there needs to be anything done when programmatically using Mocha for ESM?
I am still tracing this down ❤️

@loganknecht
Copy link
Author

loganknecht commented Mar 5, 2021

I can confirm that if I remove the runner.js this works as anticipated and I am fixing bugs.

Just not sure why the runner.js isn't working 🤔

@cspotcode
Copy link
Contributor

In general, ts-node/esm should be able to transparently hook into any mocha invocation, or any node process, and "do it's thing" as if node natively supported .ts files. Of course, if any step of the chain fails to specify --loader ts-node/esm then there's nothing we can do because we're not installed into that node process. That bit is your team's responsibility to get right.

NODE_OPTIONS is a great way to fulfill that responsibility. It ensures we are installed into every node process that spawns, assuming env vars are passed through to child processes. Pass NODE_OPTIONS="--loader ts-node/esm" and then every node process which receives that env var will install our ESM loader hook.

Unfortunately, there is a bug in node that makes node&mocha incompatible with this approach. IIRC, the mocha team can choose to implement a workaround if they want.
#4267
nodejs/node#33226

@juergba
Copy link
Contributor

juergba commented Mar 5, 2021

Do you know if there needs to be anything done when programmatically using Mocha for ESM?

You have to explicitely load the test files asynchronously with loadFilesAsync, before calling mocha.run().

@thorsent
Copy link

thorsent commented Mar 6, 2021

FWIW, I have nearly the same exact setup as @loganknecht and it's been working fine for months. Today after updating node modules I started getting the same error. In node_modules only the "esm" and "escalade" libraries had been touched, despite neither of them having had a version upgrade. I've been racking my brain trying to figure out what went wrong and then found this issue which was opened only 2 days ago.

Could be a coincidence. It's one of those, "but I didn't change anything and now I have to learn node internals" problems...

Thanks for all the advice above about switching to modern node esm support.

Edit: after spending a few hours down the rabbit hole of converting my project to native esm support (changing all the imports, fixing typescript errors, dealing with modules that could only be required, etc) I got stuck, backed everything out and then the problem mysteriously disappeared. My mocha tests are now back to running normal though I didn't change a thing.

@loganknecht
Copy link
Author

loganknecht commented Mar 8, 2021

@juergba and @cspotcode There we go! 👏 👏 👏 👏 👏

The final trick was the requirement of calling loadFilesAsync!

Prior to this change. The previous version called mocha.run after calling addFile and that caused errors to be thrown!

Thank you so much for your time, your patience, the example project, and everything in between!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: question support question
Projects
None yet
Development

No branches or pull requests

4 participants