Registering CQs

Registering CQs (Deprecated)

Shortcuts

I want to...

Borrow some code from another project, for us to maintain
Add or update a production dependency
Add a test or build dependency

Dealing with Intellectual Property

This is an effort to have a central place to guide committers who need to register CQs for the project, instead of having these instructions in some PR. There are many cases to cover, and the goal is not to re-write the Eclipse Foundation documentation on this subject. Consult the .pdf below to confirm in which scenario a specific case falls under.

Eclipse Foundation Legal Process Poster(.pdf)

Note in many (probably most) cases it's not necessary to register a CQ for a contribution/PR; e.g. completely original content written by a committer or someone from the same company, containing no cryptography, the whole thing being under the project's license and introducing no new dependencies (prereq or workswith).

Outside of a few simple cases like the one above, many contributions will end-up falling under the "Everything else" blue box on page 1 of the above .pdf.

Case: 3rd party project code copied/forked from another project, into Eclipse Theia (maintained by us)

Code/content is copied from the main VS Code repo

This special case for code copied from VS Code is on hold, pending a decision by the foundation. For now register CQs like for any other project.

The Foundation's IP team has suggested that it would make more sense for us to (occasionally?) submit the whole VS Code repo for review in a CQ, rather than having piecemeal CQs for copying various bits, as we have been doing. Since a CQ has to be for a specific version, we started with the latest release as of writing. It's TBD how often we would submit a new one to cover newer versions of VS Code.

TL;DR: As much as possible we should copy from VS Code version(s) below. When we do, no extra CQ is required, but we still have to wait for the corresponding CQ, listed in the table, to be approved before we can merge copied code.

note: it's still required to give proper credit for the copied content

note2: this covers only the main vscode repo, not anything else: microsoft/vscode

VS Code version	CQ	Approved?
1.36.1	https://dev.eclipse.org/ipzilla/show_bug.cgi?id=20620	no

We need to clarify with the Foundation what we should do in the case where the content we want to copy is only part of a newer VS Code version; probably either create a piecemeal CQ or submit that new version as a whole, and update the table above.

TODO: document in details how to obtain the attachment for future CQs for the whole of VS Code. See here for the short version.

Code/content is copied from other projects/repos

Assuming the upstream project is not hosted at the Eclipse Foundation, etc. (see .pdf for details about when it's not necessary to register a CQ) and that the license is an approved one:

This case corresponds to this, from the PDF:

• Page 2 -> Everything else -> project is not Eclipse Foundation one -> license is approved (EPL1, Apache2, ...).

1. navigate to our project page and log-in
2. On the right side of the page, under "Intellectual Property", chose "Create a Contribution Questionnaire"
3. type of contribution: "Contribution of code to be maintained by an Eclipse Foundation project"
4. Fill-in first page:

• name: 'code copied from project <project> <version/release>'
• descr: e.g.: "we want to copy content from project <project>, version: <version> for Theia to use. This code will be modified.
• reason: select "New Feature but not initial contribution"
• contribution record: URL of the upstream repo we copied from
• Authors: original authors if known. e.g. paste one copyright notice

5. Click continue and submit CQ
6. We then need to attach the copied code (from original project) to the CQ, as a zip or tar archive. It should contain only the copied file(s) as well as file(s) containing potentially relevant license info. e.g. if we copied file foo.ts from directory src/node and there is not relevant license-related file in src, we could create the following archive to attach to the PR:

license.txt
src/node/foo.ts

Case: new or updated NPM production dependency

[WIP] - New "ECD Theia Intellectual Property Clearance, Approach (Experimental)"

The foundation has proposed a new, less rigid process, where we will be able, in some/most cases, to self-check, for license compatibility of 3rd party dependencies, without the need for a CQ.

If we are not able to complete the license check on our own or have doubts, we can register a CQ to ask the Foundation's IP Team for help. Worst case, we can follow the old process - see Fallback: old process section below.

See below for the wording provided by the Foundation: ECD Theia Intellectual Property Clearance Approach (Experimental). This can be referred-to for the principles. Here we'll try to apply this, in a practical way.

Tools needed

ClearlyDefined web tool

Eclipse Foundation approved "Third Party Content licenses" list.

synp: tool used to create a package-lock.json, needed by ClearlyDefined, from our yarn.lock:

npm install -g synp

HOWTO

1. Quick assessment considering only one or a few new/updated production dependencies + their own dependencies, in isolation from Theia

Here's an example where [email protected] was added as a dependency

mkdir /tmp/a; cd /tmp/a
npm init -y
npm install [email protected] --save-dep
rm -rf node_modules/ package-lock.json
yarn install --production
synp -s yarn.lock 

Then proceed to step 3 below, with the analysis of the obtained package-lock.json

2. Deeper assessment considering the whole repo's production dependencies In this case we consider the new/updated dependency(ies), along with all others in the whole repo.

Starting with the baseline (e.g. PR) to analyse, we want to install production dependencies for the repo and generate a package-lock.json from our yarn.lock. For example, taken from a recent PR:

# Checkout PR
git checkout --track upstream/env-methods

# clean state
git clean -fdx

# install only prod dependencies
yarn install --production

# create package-lock.json from yarn.lock, required by ClearlyDefined
synp -s yarn.lock

Then proceed to step 3 below, with the analysis of the obtained package-lock.json

3. Analyse packages licenses

Using a file explorer, drag-and-drop the generated package-lock.json into the ClearlyDefined workspace. Sort packages by license:

Going down the list, check the license of all packages against the Foundation's approved list. A few quick rules-of-thumb, when encountering a mix of licenses for a given component:

• a "mix" of approved licenses is ok: e.g. (MIT AND Apache 2), (MIT OR Apache 2)
• an approved license "OR" and non-approved one is also ok. e.g. GPL2 or MIT
• an approved license "AND" a non-approved one is NOT OK. e.g. GPL2 AND MIT

Special case: license reported as "NOASSERTION": Towards the end of the sorted list, there might be some packages reported having license "NOASSERTION". This means the license is undetermined and that we need to dig a bit deeper to find the applicable license(s) ourselves.

To make our life easier, we have this spreadsheet, where we can document "NOASSERTION" packages that we have investigated in the past. So check there, and if a "NOASSERTION" package of the same name and version is already investigated, you can re-use the conclusion. Else proceed with the checks below and add this new package/version to the spreadsheet.

Investigating "NOASSERTION" packages: For each one, we need to:

• click on the package's version, besides its name, to open the NPM page for it and note if a license is mentioned
• click on the "github" link to be taken to the component's GH repo.
• check for a license file: Note any license mentioned there
• check the README for license info: Note any license mentioned there
• check the package.json: Note any license mentioned there
• If still not sure, check some source code files: Note any license mentioned there
• Add this package/version to the spreadsheet . If required request write permission through Google Docs or ping @marcdumais-work.

Note: If we can't determine the license for one of the packages using the strategy above, we need to ask for help from the Foundation.

Let's see some real live examples. Here's the list of packages reported as "NOASSERTION", found for the main Theia repo at the time of writing:

And short videos showing how to quickly determine their license:

Package: error 7.0.2 :

Package: regenerator-transform 0.10.1 :

Package: string-template 0.2.1 :

Package: umd-compat-loader 2.1.1 :

Conclusion for the NOASSERTION examples above: all approved licenses

If all licenses are approved: Assuming we have no reason to think that this information is incorrect or incomplete, we're done. else: open a CQ and ask for help, about what to do with the non-approved license(s) or other issue(es) noticed.

Fallback: old process

e.g. a PR adds a new production (i.e. non-"dev-dependency") npm dependency in one of Eclipse Theia's package.json, or updates the version of such a dependency.

Current Way: for production/runtime dependencies, it's necessary to analyse license compatibility of the dependencies themselves (as defined in our various package.json files) and as well as their own recursive runtime dependencies.

1. navigate to our project page and log-in
2. On the right side of the page, under "Intellectual Property", chose "Create a Contribution Questionnaire"
3. type of contribution: Third-Party Code Request
4. Fill-in first page:

• Name and Version of the Library: Theia npm node click "Continue"

• Due Diligence Type: Type A
• description: This is a request for all of the production node.js/NPM dependencies for Eclipse Theia
• cryptography: A few of the npm dependencies look like they might be cryptography-related: bcrypt-pbkdf,crypt, cryptiles,crypto-browserify,crypto-random-string,https-browserify
• license: --- Other licenses ---
• Other licenses: MIT AND BSD-3-Clause AND ISC AND Apache-2.0 AND BSD-2-Clause AND Zlib AND X11 AND (BSD-3-Clause OR AFL-2.1) AND CC-By-4.0 AND CC
• Distribution: Both Source and Binary
• Modified: Unmodified (Usual Case)

6. Click continue and submit CQ
7. We then need to add an attachment to the CQ. See instructions below to generate this archive

Here are some rough instructions on how to generate the archive/zip to be attached to the CQ:

git clone https://github.com/theia-ide/theia.git && cd theia
yarn install --production

# find node_modules folders other than the main one in the root
npmlist=`find . -name node_modules | grep -v "^./node_modules"`

# Find the extensions where a Debug Adapter is Downloaded/used (defined in an "adapters" section in package.json)
grep "\"adapters" `find packages -name package.json`
# if there are changes, update list below

# list of the path where the Adapters are saved:
# note: this assumes each DA is in "unzipped" form. It it's not the case, we need to unzip it/them manually so it/they will be analysed correctly when submitted as part of the CQ.
DAlist="./packages/debug-nodejs/download ./packages/java-debug/download"

# Find the extensions where a Language Server is Downloaded/used under that extension (vs through the normal yarn install,
# that installs under the root node_modules) Such LS can be defined under the "ls" section in package.json
grep "\"ls" `find packages -name package.json`
# if there are changes, update list below
# note: only npm modules are of interest here. If the LS is not a npm module, it should be registered in its own, different CQ. e.g. jdt.ls
# note: "typescript-language-server" is already taken into account since it's in the root node_modules

# list of the path where the Language Servers are saved:
LSlist=""

# result: root npm_modules folder as well as any other we find elsewhere in the repo. Also we include the 
# Debug Adapters that are used as plugins:
tar czvf theia-npm-prod-dependencies.tar.gz ./node_modules $npmlist $DAlist $LSlist

TBD, clarifying this topic with the Eclipse Foundation. We may have a simplified way to do this in the near future.

Case: new or updated test/build dependency (e.g. devDependencies)

These are dependencies that are not distributed, but used only to build and test our project. They map to the "devDependencies" field in our various package.json files.

For these, the effort should be lower, since "These are processed as Workswith; and as a result, no IP review is involved."

More info here

1. navigate to our project page and log-in
2. On the right side of the page, under "Intellectual Property", chose "Create a Contribution Questionnaire"
3. type of contribution: "Third-Party Code Request"
4. Fill-in first page:

• name: 'Build- and test-only NPM dependencies for Eclipse Theia'

5. go to next page

• Due Diligence: "Type A"
• Description: This is an umbrella CQ for build and test dependencies

Then for each dev-dependency we seek approval-for, add an entry like this:

name: <dev-dependency name>
url: <repo URL>
license: <license>
version: <version> and later versions

• Cryptography: answer depending on case
• license: dependencies license(s). if more than 1, chose "Other licenses" and list them in the next field.
• Distribution: "both source and binary"?
• Modified: Unmodified

6. Click continue and submit CQ
7. For 'build/test' CQ, no source code needs be attached. If the tool still asks for an attachment, you can attach something to it will be satisfied. It can be an empty archive or an Eclipse logo, for example

Such CQs registered so far:

• https://dev.eclipse.org/ipzilla/show_bug.cgi?id=18832

References

ECD Theia Intellectual Property Clearance Approach (Experimental)

Project licensed Content

CQs required based on Due Diligence Process (contributions >1K LOCS, Content to move to Eclipse, Initial Contributions, etc.)

Third Party Content (Modified)

Modifications are performed under the third party license; CQ required.

Third Party Dependencies (Non-Modified)

The Project will use tooling to run and analyze the applicable license(s) of third party content required for project releases. The applicable license(s) must conform to the White List as per [1].

Should the project be unclear of applicable licensing and/or the license is not contained in the White List, the project will raise an IP Request (CQ) seeking assistance from the IP Team.

Please note “No Assertion” and/or “Unknown” license information is not sufficient license terms. Further, WTFPL will be distributed subject to the MIT.

The project must perform this exercise each time a new dependency is introduced and/or when the dependency version rev is greater than a service revision. The project may use the output of Yarn Lock, package-lock.json, npm-shrinkwarp.json to confirm applicable licensing of javascript third party content. Package-lock.json and npm-shrinkwrap.json may be used in conjunction with ClearyDefined [2]. For non javascript content, the project may use scan code [3].

PRIOR TO RELEASE:

The project will enter necessary CQs to receive final IP clearance prior to a project release. This clearance will be a gate to releases.

[1] https://www.eclipse.org/legal/licenses.php#approved

[2] https://clearlydefined.io/workspace

[3] https://github.com/nexB/scancode-toolkit