Merge pull request #294 from open-contracting/210-mapping-updates

210 mapping updates

docs/cost/index.md

@@ -33,43 +33,34 @@ You can read more about the Infrastructure Data Standard in [CoST Guidance Note
 
 ```
 
-## Mapping to IDS and from OCDS
-
 <!-- Note - mappings come from https://docs.google.com/spreadsheets/d/1tpXKCrNY1vUEPo6O1j-GPhxgSna7CZ5uwz_eTNLEOr8/edit#gid=2054628701 -->
 
-The following mapping tables describe:
+The following tables document two mappings:
 
-* How each element of the CoST Infrastructure Data Standard can be represented as **structured data** using the [Open Contracting for Infrastructure Data Standard](../projects/index), in the 'Mapping to OC4IDS' column.
+* The **CoST IDS to OC4IDS** mapping describes how to represent each element of the CoST IDS as structured data using OC4IDS. Use this mapping if you already collect data according to the CoST IDS and you want to publish your data using OC4IDS, or if you want to make sure that your OC4IDS publication conforms to the CoST IDS.
+* The **OCDS to OC4IDS** mapping describes how to use OCDS data to populate the sections of an OC4IDS file which relate to the CoST IDS. Use this mapping if you have access to OCDS data on infrastructure contracting processes and you want to create a summary by project in OC4IDS format, or if you want to check which CoST IDS elements your OCDS data covers.
 
-* How existing OCDS data can be used to populate project-level and contracting process summary data, in the 'Mapping from OCDS' column.
+The organization of the mapping tables reflects the structure of the CoST IDS, which is described in [Getting Started](../projects/index).
 
-```eval_rst
-.. admonition:: Mapping from OCDS
-    :class: Note
-
-    .. markdown::
-
-        Some mappings use fields from [OCDS extensions](https://standard.open-contracting.org/1.1/en/extensions/). In these cases, the names of extensions are noted in parentheses; where possible, alternative mappings are provided that use only fields from the core OCDS schema.
-
-```
+## CoST IDS to OC4IDS Mapping
 
 ### Project level
 
-#### Project identification
+#### Identification
 
 ```eval_rst
 .. csv-table-no-translate::
    :header-rows: 1
-   :widths: 20 20 30 30
+   :widths: 20 20 20 30
    :file: ../../build/current_lang/project-level-identification.csv
 ```
 
-#### Project preparation
+#### Preparation
 
 ```eval_rst
 .. csv-table-no-translate::
    :header-rows: 1
-   :widths: 20 20 30 30
+   :widths: 20 20 20 30
    :file: ../../build/current_lang/project-level-preparation.csv
 ```
 
@@ -78,7 +69,7 @@ The following mapping tables describe:
 ```eval_rst
 .. csv-table-no-translate::
    :header-rows: 1
-   :widths: 20 20 30 30
+   :widths: 20 20 20 30
    :file: ../../build/current_lang/project-level-completion.csv
 ```
 
@@ -89,12 +80,112 @@ The following mapping tables describe:
 ```eval_rst
 .. csv-table-no-translate::
    :header-rows: 1
-   :widths: 20 20 30 30
+   :widths: 20 20 20 30
    :file: ../../build/current_lang/process-level-procurement.csv
 ```
 
 #### Implementation
 
+```eval_rst
+.. csv-table-no-translate::
+   :header-rows: 1
+   :widths: 20 20 20 30
+   :file: ../../build/current_lang/process-level-implementation.csv
+```
+
+## OCDS to OC4IDS Mapping
+
+### Guidance
+
+#### Command-line tool and reference implementation
+
+OCDS Kit's [convert-to-oc4ids](https://ocdskit.readthedocs.io/en/latest/cli/ocds.html#convert-to-oc4ids) command  is a command-line tool and reference implementation for converting OCDS data to OC4IDS format.
+
+`convert-to-oc4ids` covers most mappings in the following categories:
+
+* project-level identification
+* project-level preparation
+* process-level procurement
+
+However, `convert-to-oc4ids` does not cover all mappings, nor does it perform currency conversions. Mappings which `convert-to-oc4ids` does not cover are shown in *italics*.
+
+#### Mapping codelists
+
+Mappings that depend on the specific classification or codelist used in the OCDS data are not documented in detail, as they may differ by publisher. For example, mapping to the OC4IDS projectSector codelist.
+
+#### Alternative mappings
+
+Some mappings offer optional alternatives in case the primary mapping isn't available. For example, for OCDS data in which `planning.project.title` isn't available, you can optionally set the project `title` based on the `tender.title`.
+
+In order to provide analysts with additional context, alternative mappings may copy additional fields which don't appear in OC4IDS schema. You should remove these fields if you plan to publish your OC4IDS data.
+
+#### OCDS extensions
+
+Some mappings use fields from [OCDS extensions](https://standard.open-contracting.org/latest/en/guidance/map/extensions/#extensions). In these cases, the names of extensions are noted in parentheses; where possible, alternative mappings are provided that use only fields from the core OCDS schema.
+
+#### Handling conflicts and duplicates
+
+Implementations of the mapping should give consideration to:
+
+* OCDS data that contains fields that differ between contracting processes but map to a single field in OC4IDS: for example, where `planning.project.title` differs for two contracting processes that relate to the same project, but OC4IDS has a single `title` field at the project level.
+* OCDS data that contains multiple `Organization` objects with the same `.role` that map to a single field in OC4IDS: for example, where a contracting process has two `Organization`s with the 'procuringEntity' role, but OC4IDS has a single `.summary.tender.procuringEntity` field at the contract level.
+* Checking for duplicates when copying objects from OCDS. For example, checking whether an `Organization` object has already been copied before copying it again.
+* Handling identifier conflicts when copying objects from OCDS. For example, where two contracting processes both contain a ``Document`` with the same `.id`.
+
+Read the `convert-to-oc4ids` [transformation notes](https://ocdskit.readthedocs.io/en/latest/cli/ocds.html#transformation-notes) to learn about how OCDS Kit handles the above scenarios.
+
+#### Handling multiple currencies
+
+Some mappings involve converting values in OCDS, which may be in different currencies, to a base currency.
+
+Implementations which include multiple currencies should give consideration to [value dating](https://en.wikipedia.org/wiki/Value_date). One approach is to use the compiled release's `date`.
+
+### Mapping
+
+#### Project level
+
+##### Identification
+
+```eval_rst
+.. csv-table-no-translate::
+   :header-rows: 1
+   :widths: 20 20 20 30
+   :file: ../../build/current_lang/project-level-identification.csv
+```
+
+##### Preparation
+
+```eval_rst
+.. csv-table-no-translate::
+   :header-rows: 1
+   :widths: 20 20 20 30
+   :file: ../../build/current_lang/project-level-preparation.csv
+```
+
+##### Project completion
+
+```eval_rst
+.. csv-table-no-translate::
+   :header-rows: 1
+   :widths: 20 20 20 30
+   :file: ../../build/current_lang/project-level-completion.csv
+```
+
+#### Process level
+
+The mappings in this section relate to the `contractingProcesses` section of the OC4IDS schema, unless otherwise specified.
+
+##### Procurement
+
+```eval_rst
+.. csv-table-no-translate::
+   :header-rows: 1
+   :widths: 20 20 20 30
+   :file: ../../build/current_lang/process-level-procurement.csv
+```
+
+##### Implementation
+
 Disclosures in the implementation section of the CoST IDS relate to changes to a contract's value, duration or scope that were made after the contract was awarded.
 
 If OCDS data is available, these changes can be determined by comparing the most recent OCDS release to a compiled release created from all prior releases (to better understand these concepts, refer to the [OCDS documentation](https://standard.open-contracting.org/1.1/en/getting_started/releases_and_records/)). The specific fields to monitor for changes between releases are described in the mapping table below.
@@ -104,7 +195,7 @@ In some cases, OCDS data may include an explanation of changes in the relevant `
 ```eval_rst
 .. csv-table-no-translate::
    :header-rows: 1
-   :widths: 20 20 30 30
+   :widths: 20 20 20 30
    :file: ../../build/current_lang/process-level-implementation.csv
 ```
 

docs/examples/example.json

@@ -26,6 +26,13 @@
         "transport",
         "transport.road"
       ],
+      "additionalClassifications": [
+        {
+          "scheme": "COFOG",
+          "id": "04.5.1",
+          "description": "Road transport (CS)"
+        }
+      ],
       "type": "expansion",
       "relatedProjects": [
         {
@@ -574,7 +581,7 @@
               },
               {
                 "id": "a1b1c1-construction-completion",
-                "documentType": "comletionCertificate",
+                "documentType": "completionCertificate",
                 "title": "Completion certificate for construction at M75 J4-5 upgrade",
                 "description": "Completion certificate for the construction upgrading motorway M75 Junctions 4-5.",
                 "url": "https://example.com/Published/a1b1c1-construction-completion.html",

docs/reference/changelog.md

@@ -4,6 +4,7 @@
 
 ### Documentation
 
+* [#210](https://github.com/open-contracting/infrastructure/issues/210) - update project identification mapping for sector, replace reference to Budget and projects extension with Projects extension. 
 * [#217](https://github.com/open-contracting/infrastructure/issues/217) - remove repeated 'OCDS:' in mapping documentation.
 * [#268](https://github.com/open-contracting/infrastructure/issues/268), [#269](https://github.com/open-contracting/infrastructure/issues/269) replace 'finalAudit' with 'technicalAuditReport' and 'financialAuditReport' in mapping.
 

mapping/process-level-implementation.csv

@@ -1,7 +1,7 @@
-CoST IDS element,CoST IDS draft definition,Mapping to OC for Infrastructure,Mapping from OCDS
-Variation to contract price,Difference between the price at contract award and the current projected price,"Contracting process: For each variation, publish an entry with a ``.date`` and ``.description`` in ``.summary.modifications`` with ``.type`` of 'value'. Provide the contract value before the variation in ``.oldContractValue`` and the contract value after the variation in ``.newContractValue``",Monitor ``contracts.value`` for changes. Copy ``contracts.amendments.description`` to ``.description`` and ``release.date`` to ``.date``
-Escalation of contract price,"Escalation to date of the price of materials, labour, equipment etc. due to fluctuations in inflation, currency etc.","Contracting process: For each escalation, publish an entry with a ``.date`` and ``.description`` in ``.summary.modifications`` with ``.type`` of 'value'. Provide the contract value before the escalation in ``.oldContractValue`` and the contract value after the escalation in ``.newContractValue``",Monitor ``contracts.value`` for changes. Copy ``contracts.amendments.description`` to ``.description`` and ``release.date`` to ``.date``
-Variation to contract duration,Difference between original duration at contract award and the current projected duration in weeks.,"Contracting process: For each variation, publish an entry with a ``.date`` and ``.description`` in ``.summary.modifications`` with ``.type`` of 'duration'. Provide the contract period before the variation in ``.oldContractPeriod`` and the contract period after the variation in ``.newContractPeriod``",Monitor ``contracts.period`` for changes. Copy ``contracts.amendments.description`` to ``.description`` and ``release.date`` to ``.date``
-Variation to contract scope,Any changes between original scope at contract award and the current scope,"Contracting process: For each variation, publish an entry with a ``.date`` and ``.description`` in ``.summary.modifications`` with ``.type`` of 'scope'","Monitor ``contracts.description``, ``contracts.items`` and ``contracts.implementation.milestones`` for changes. Copy ``contracts.amendments.description`` to ``.description`` and ``release.date`` to ``.date``"
-Reasons for price changes,"Summary of reasons for primary changes (e.g. variations) that then lead to changes in contract price, major price fluctuations and accumulative increase or decrease in price.","Contracting process: For each variation, provide a ``.rationale`` in ``.summary.modifications``.",Copy ``contracts.amendments.rationale`` to ``.rationale``
-Reasons for scope and duration changes,Summary of reasons for primary changes (e.g. variations) that then lead to changes in the scope and duration.,"Contracting process: For each variation, provide a ``.rationale`` in ``.summary.modifications``.",Copy ``contracts.amendments.rationale`` to ``.rationale``
+CoST IDS element,CoST IDS draft definition,Mapping to OC4IDS,Mapping from OCDS
+Variation to contract price,Difference between the price at contract award and the current projected price,"Contracting process: For each variation, publish an entry with a ``.date`` and ``.description`` in ``.summary.modifications`` with ``.type`` of 'value'. Provide the contract value before the variation in ``.oldContractValue`` and the contract value after the variation in ``.newContractValue``",*Monitor* ``contracts.value`` *for changes. Copy* ``contracts.amendments.description`` *to* ``.description`` *and* ``release.date`` *to* ``.date``.
+Escalation of contract price,"Escalation to date of the price of materials, labour, equipment etc. due to fluctuations in inflation, currency etc.","Contracting process: For each escalation, publish an entry with a ``.date`` and ``.description`` in ``.summary.modifications`` with ``.type`` of 'value'. Provide the contract value before the escalation in ``.oldContractValue`` and the contract value after the escalation in ``.newContractValue``",*Monitor* ``contracts.value`` *for changes. Copy* ``contracts.amendments.description`` *to* ``.description`` *and* ``release.date`` *to* ``.date``.
+Variation to contract duration,Difference between original duration at contract award and the current projected duration in weeks.,"Contracting process: For each variation, publish an entry with a ``.date`` and ``.description`` in ``.summary.modifications`` with ``.type`` of 'duration'. Provide the contract period before the variation in ``.oldContractPeriod`` and the contract period after the variation in ``.newContractPeriod``",*Monitor* ``contracts.period`` *for changes. Copy* ``contracts.amendments.description`` *to* ``.description`` *and* ``release.date`` *to* ``.date``.
+Variation to contract scope,Any changes between original scope at contract award and the current scope,"Contracting process: For each variation, publish an entry with a ``.date`` and ``.description`` in ``.summary.modifications`` with ``.type`` of 'scope'","*Monitor* ``contracts.description``, ``contracts.items`` *and* ``contracts.implementation.milestones`` *for changes. Copy* ``contracts.amendments.description`` *to* ``.description`` *and* ``release.date`` *to* ``.date``."
+Reasons for price changes,"Summary of reasons for primary changes (e.g. variations) that then lead to changes in contract price, major price fluctuations and accumulative increase or decrease in price.","Contracting process: For each variation, provide a ``.rationale`` in ``.summary.modifications``.",*Copy* ``contracts.amendments.rationale`` *to* ``.rationale``.
+Reasons for scope and duration changes,Summary of reasons for primary changes (e.g. variations) that then lead to changes in the scope and duration.,"Contracting process: For each variation, provide a ``.rationale`` in ``.summary.modifications``.",*Copy* ``contracts.amendments.rationale`` *to* ``.rationale``.