guidance/map/translations: Rewrite with simpler structure

- Clarify that this page is for sources with distinct fields for each language. If a field mixes languages, this page is not relevant.
- Clarify that translated text must have the same content across translations.
- Instead of a long list of repetitive field names, summarize the patterns.

Author: jpmckinney

docs/_static/i18n.csv

@@ -1,15 +1,41 @@
 # Translations
 
-If you need to publish data in multiple languages, you can publish full-file translations. That is, separate OCDS releases for each language.
+If your data sources have separate data elements for different language versions of the same content, you can publish each language as a separate OCDS dataset.
 
-You can publish the values of free-text fields – like titles and descriptions – in any language, but you need to ensure that the values of `id` fields are consistent across languages so that users can find the translations of objects. You ought to set the `language` field to the language used in free-text fields.
+```{admonition} What does "same content" mean?
+Two texts have the same content if they describe the same thing: for example, "United Kingdom" in English and "Royaume-Uni" in French. On the other hand, if your contracting process has different contact points for different language speakers, that content is *not* the same. Your OCDS data should therefore contain one contact point for each, using the [Additional Contact Points](https://extensions.open-contracting.org/en/extensions/additionalContactPoint/master/) extension.
+```
 
-For example, for data published in English and Spanish:
+## What you can translate
+
+You can publish the values of these fields in any language:
+
+% STARTLIST
+- `description`, in any location
+- `legalName`, in any location
+- `name`, in any location
+- `rationale`, in any location
+- `statusDetails`, in any location
+- `title`, in any location
+- `parties/address/locality`
+- `parties/address/region`
+- `parties/address/streetAddress`
+- `planning/budget/project`
+- `tender/awardCriteriaDetails`
+- `tender/exclusionGrounds`
+- `tender/procurementMethodDetails`
+- `tender/procurementMethodRationale`
+- `tender/selectionCriteria`
+- `tender/submissionMethodDetails`
+% ENDLIST
+
+When publishing OCDS data in different languages, remember to set the `language` field. For example:
 
 `````{tab-set}
 
 ````{tab-item} English
-```json
+```{code-block} json
+:emphasize-lines: 8,11
 {
   "ocid": "ocds-213czf-000-00001",
   "id": "1",
@@ -20,14 +46,15 @@ For example, for data published in English and Spanish:
   "language": "en",
   "tender": {
     "id": "1",
-    "title": "Purchase of office supplies",
+    "title": "Purchase of office supplies"
   }
 }
 ```
 ````
 
 ````{tab-item} Spanish
-```json
+```{code-block} json
+:emphasize-lines: 8,11
 {
   "ocid": "ocds-213czf-000-00001",
   "id": "1",
@@ -38,20 +65,30 @@ For example, for data published in English and Spanish:
   "language": "es",
   "tender": {
     "id": "1",
-    "title": "Compra de material de oficina",
+    "title": "Compra de material de oficina"
   }
 }
 ```
 ````
 
 `````
 
-In order for your data to be interoperable and compatible with OCDS tools and methodologies, you cannot translate field names (keys) or codes from codelists. For example, the name of the `tag` field cannot be translated and its items need to be codes from the [release tag codelist](../../schema/codelists.md#release-tag), like 'tender':
+## What you cannot translate
+
+The names of fields, and the values of all fields not listed above, need to be the same across your translated OCDS datasets, in order to support interoperability, which is the purpose of standardization. These cover:
+
+- **Identifiers**, like `ocid`, `id`, etc.
+- **Codes**, like release tags, identifier schemes and milestone codes
+- **Formatted values**, like URLs, dates, email addresses, telephone numbers and postal codes
+- **Non-text fields**, like numbers and booleans
+
+For example, the name of the `tag` field needs to be "tag", and its value needs to be a list of codes from the [release tag codelist](../../schema/codelists.md#release-tag).
 
 `````{tab-set}
 
-````{tab-item} Valid data
-```json
+````{tab-item} Valid
+```{code-block} json
+:emphasize-lines: 5-7
 {
   "ocid": "ocds-213czf-000-00001",
   "id": "1",
@@ -62,14 +99,15 @@ In order for your data to be interoperable and compatible with OCDS tools and me
   "language": "es",
   "tender": {
     "id": "1",
-    "title": "Compra de material de oficina",
+    "title": "Compra de material de oficina"
   }
 }
 ```
 ````
 
-````{tab-item} Invalid data (translated field name)
-```json
+````{tab-item} Invalid (incorrect name)
+```{code-block} json
+:emphasize-lines: 5
 {
   "ocid": "ocds-213czf-000-00001",
   "id": "1",
@@ -80,14 +118,15 @@ In order for your data to be interoperable and compatible with OCDS tools and me
   "language": "es",
   "tender": {
     "id": "1",
-    "title": "Compra de material de oficina",
+    "title": "Compra de material de oficina"
   }
 }
 ```
 ````
 
-````{tab-item} Invalid data (translated code)
-```json
+````{tab-item} Invalid (incorrect value)
+```{code-block} json
+:emphasize-lines: 6
 {
   "ocid": "ocds-213czf-000-00001",
   "id": "1",
@@ -98,39 +137,30 @@ In order for your data to be interoperable and compatible with OCDS tools and me
   "language": "es",
   "tender": {
     "id": "1",
-    "title": "Compra de material de oficina",
+    "title": "Compra de material de oficina"
   }
 }
 ```
 ````
 
 `````
 
-The fields whose values can be translated are listed in the [internationalization lookup table](#internationalization-lookup-table).
+## Translating headers in spreadsheets
 
-## Translating headers in spreadsheets/CSVs
+To ease access for non-English speakers, instead of using field *names* as column headers (which are always in English), you can use field *titles*, which are translated in [OCDS translations](localization.md#translating-the-standard).
 
-In order to ease access for non-English speakers, instead of using field *names* as column headers (which are always in English), you can use field *titles*.
-
-The titles are currently available in English, Spanish and French. If you would like to translate the titles to your own language, please [contact the OCDS Helpdesk](mailto:[email protected]).
-
-For example, this CSV excerpt uses field titles from the Spanish translation of OCDS:
+For example, this CSV excerpt uses field titles from the Spanish translation:
 
 | ID de Entrega | Fecha de entrega     |
 | ------------- | -------------------- |
 | 1             | 2024-01-01T00:00:00Z |
 
-You can use [Flatten Tool](https://flatten-tool.readthedocs.io/en/latest/) to generate files with translated field titles. For example, this command converts an OCDS release package to XLSX format, using field titles from the schema:
+You can use [Flatten Tool](https://flatten-tool.readthedocs.io/en/latest/) to generate files with field titles. For example, this command converts an OCDS release package to XLSX format, using field titles from the release schema:
 
 ```bash
-flatten-tool flatten -s release-schema.json -f xlsx --use-titles --root-id=id --root-list-path=releases release_package.json
+flatten-tool flatten release_package.json --output-format=xlsx --use-titles --schema=release-schema.json --root-id=ocid --root-list-path=releases
 ```
 
-## Internationalization lookup table
-
-Use the following table to check whether a field can be translated. You can download the table as a [CSV spreadsheet](../../_static/i18n.csv).
-
-```{csv-table}
-:file: ../../_static/i18n.csv
-:header-rows: 1
+```{note}
+Field titles are available in English, Spanish and French. To translate titles to another language, [contact the Data Support Team](mailto:[email protected]).
 ```

docs/guidance/map/translations.md

@@ -7,6 +7,7 @@
 import re
 import sys
 import warnings
+from collections import defaultdict
 from contextlib import contextmanager
 from copy import deepcopy
 from glob import glob
@@ -509,14 +510,14 @@ def missing_changelog(ignore_base):
 @cli.command()
 def pre_commit():
     """
-    Update derivative schema files, and generate a CSV file of translatable fields.
+    Update derivative schema files, and generate a CSV file of multilingual fields.
 
     \b
     - meta-schema.json
     - dereferenced-release-schema.json
     - versioned-release-validation-schema.json
     """
-    nontranslatable = {
+    nonmultilingual = {
         # Identifiers.
         'amendsReleaseID', 'id', 'identifier', 'ocid', 'relatedItems', 'releaseID',
         # Missing format properties. https://github.com/open-contracting/standard/issues/881
@@ -528,28 +529,47 @@ def pre_commit():
     }
 
     release_schema = json_load('release-schema.json')
-
     jsonref_release_schema = json_load('release-schema.json', jsonref, merge_props=True)
 
-    with (basedir / 'docs' / '_static' / 'i18n.csv').open('w') as f:
-        writer = csv.writer(f, lineterminator='\n')
-        writer.writerow(['path', 'title'])
-        for field in get_schema_fields(jsonref_release_schema):
-            if (
-                # Output dereferenced properties, not definitions.
-                not field.definition_pointer_components
-                # Skip deprecated fields.
-                and not field.deprecated
-                # If a field can be a non-string, it is not translatable.
-                and not any(t in field.schema['type'] for t in ('boolean', 'integer', 'number', 'object'))
-                # If a field's value is constrained to a codelist or format, it is not translatable.
-                and not any(prop in field.schema for prop in ('codelist', 'format'))
-                # If an array can contain non-strings, it is not translatable.
-                and not ('array' in field.schema['type'] and 'object' in field.schema['items']['type'])
-                # Specific exceptions.
-                and not field.path_components[-1] in nontranslatable
-            ):
-                writer.writerow([field.path.replace('.', '/'), field.schema['title']])
+    counts = defaultdict(list)
+    for field in get_schema_fields(jsonref_release_schema):
+        name = field.path_components[-1]
+        # Skip definitions (output dereferenced properties only). Skip deprecated fields.
+        if field.definition_pointer_components or field.deprecated:
+            continue
+        multilingual = (
+            # If a field can be a non-string, it is not multilingual.
+            not any(t in field.schema['type'] for t in ('boolean', 'integer', 'number', 'object'))
+            # If a field's value is constrained to a codelist or format, it is not multilingual.
+            and not any(prop in field.schema for prop in ('codelist', 'format'))
+            # If an array can contain non-strings, it is not multilingual.
+            and not ('array' in field.schema['type'] and 'object' in field.schema['items']['type'])
+            # Specific exceptions.
+            and name not in nonmultilingual
+        )
+        field.sep = '/'
+        if name in counts and bool(counts[name]) ^ multilingual:
+            if multilingual:
+                raise Exception(f'{name} is multilingual at {field.path}, but not elsewhere')
+            else:
+                raise Exception(f'{name} is multilingual at {" & ".join(counts[name])}, but not at {field.path}')
+        if multilingual:
+            counts[name].append(field.path)
+        else:
+            counts[name] = []
+
+    bulletlist = [
+        '% STARTLIST',
+        *sorted([f'- `{name}`, in any location' for name, paths in counts.items() if len(paths) > 1]),
+        *sorted([f'- `{paths[0]}`' for _, paths in counts.items() if len(paths) == 1]),
+        '% ENDLIST',
+    ]
+
+    path = basedir / 'docs' / 'guidance' / 'map' / 'translations.md'
+    with path.open() as f:
+        contents = f.read()
+    with path.open('w') as f:
+        f.write(re.sub(r'% STARTLIST.+% ENDLIST', '\n'.join(bulletlist), contents, flags=re.DOTALL))
 
     json_dump('meta-schema.json', get_metaschema())
     json_dump('dereferenced-release-schema.json', jsonref_release_schema)