fixes typos in IRIs, resolves #81; updates terms

The following typos in IRIs were fixed using the new SHACL shapes from
previous commit:
- dpv:expiry relation instead of dpv:hasExpiry relation in consent
- dpv:hasConsequenceOn was used as a parent even though it was proposed.
  The term has been promoted to accepted status
- Typos in Technical measures where Crypto- was mistyped as Cryto-

Errors in labels:
- MaintainCreditCheckingDatabase
- MaintainCreditRatingDatabase

The following terms were updated:
- GDPR's legal bases where text has been added from Art.6 and the parent
  terms have been aligned with main spec's legal bases (including
  creation of new terms to match granularity)
- Anonymisation and Pseudonymisation have been changed to be types of
  Deidentification techniques (as the grouping parent concept) to
  distinguish them following discussions in #15
- DPV-LEGAL has laws and DPAs for USA from contributions by @JonathanBowker

documentation-generator/002_parse_csv_to_rdf.py

@@ -693,17 +693,17 @@ def serialize_graph(graph, filepath):
     graph.namespace_manager.bind(prefix, namespace)
 proposed = []
 Location_schema = namedtuple('Legal_Location', (
-    'Term', 'Label', 'ParentTerm', 'Alpha2', 'Alpha3', 'Numeric', 'M49',
+    'term', 'Label', 'Parentterm', 'Alpha2', 'Alpha3', 'Numeric', 'M49',
     'broader', 'narrower', 'created', 'modified', 
     'status', 'contributors', 'resolution'))
 concepts = extract_terms_from_csv(
     f'{IMPORT_CSV_PATH}/legal_Locations.csv', Location_schema)
 for row in concepts:
     if row.status not in VOCAB_TERM_ACCEPT:
-        proposed.append(row.Term)
+        proposed.append(row.term)
         continue
-    term = BASE[row.Term]
-    parent = DPV[row.ParentTerm.replace("dpv:", "")]
+    term = BASE[row.term]
+    parent = DPV[row.Parentterm.replace("dpv:", "")]
     graph.add((term, RDF.type, DPV.Concept))
     graph.add((term, RDF.type, SKOS.Concept))
     graph.add((term, DPV.isInstanceOf, parent))
@@ -819,7 +819,7 @@ def serialize_graph(graph, filepath):
     f'{IMPORT_CSV_PATH}/legal_Authorities.csv', Location_schema)
 for row in concepts:
     if row.status not in VOCAB_TERM_ACCEPT:
-        proposed.append(row.Term)
+        proposed.append(row.term)
         continue
     term = BASE[row.term]
     graph.add((term, RDF.type, DPV.Concept))
@@ -871,7 +871,7 @@ def serialize_graph(graph, filepath):
     f'{IMPORT_CSV_PATH}/legal_EU_EEA.csv', Location_schema)
 for row in concepts:
     if row.status not in VOCAB_TERM_ACCEPT:
-        proposed.append(row.Term)
+        proposed.append(row.term)
         continue
     term = BASE[row.term]
     graph.add((term, RDF.type, DPV.Concept))
@@ -930,7 +930,7 @@ def serialize_graph(graph, filepath):
     f'{IMPORT_CSV_PATH}/legal_EU_Adequacy.csv', Location_schema)
 for row in concepts:
     if row.status not in VOCAB_TERM_ACCEPT:
-        proposed.append(row.Term)
+        proposed.append(row.term)
         continue
     term = BASE[row.term]
     graph.add((term, RDF.type, DPV.Concept))

documentation-generator/801_generate_releases.sh

@@ -38,8 +38,8 @@ zip -q releases/dpv-owl.zip -r dpv-owl
 echo "generated releases/dpv-owl.zip"
 
 # 4. dpv-csv.zip - DPV + Extensions as CSV files
-cp documentation-generator/vocab_csv/dpv_terms_discussion.xlsx releases/dpv.xlsx
-echo "generated releases/dpv.xlsx"
+zip -q releases/dpv-xlsx.zip documentation-generator/vocab_csv/*.xlsx
+echo "generated releases/dpv-xlsx.zip"
 
 # 5. dpv-json.zip - DPV + Extensions as JSON files
 # TODO

documentation-generator/logs/changelog.txt

@@ -1,35 +1,3 @@
-Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_date at 0x7f4e95382ac0>
-Traceback (most recent call last):
-  File "/home/harsh/.pyenv/versions/dpv/lib/python3.11/site-packages/rdflib/term.py", line 2084, in _castLexicalToPython
-    return conv_func(lexical)  # type: ignore[arg-type]
-           ^^^^^^^^^^^^^^^^^^
-  File "/home/harsh/.pyenv/versions/dpv/lib/python3.11/site-packages/isodate/isodates.py", line 203, in parse_date
-    raise ISO8601Error('Unrecognised ISO 8601 date format: %r' % datestring)
-isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: ''
-Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_date at 0x7f4e95382ac0>
-Traceback (most recent call last):
-  File "/home/harsh/.pyenv/versions/dpv/lib/python3.11/site-packages/rdflib/term.py", line 2084, in _castLexicalToPython
-    return conv_func(lexical)  # type: ignore[arg-type]
-           ^^^^^^^^^^^^^^^^^^
-  File "/home/harsh/.pyenv/versions/dpv/lib/python3.11/site-packages/isodate/isodates.py", line 203, in parse_date
-    raise ISO8601Error('Unrecognised ISO 8601 date format: %r' % datestring)
-isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: ''
-Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_date at 0x7f4e95382ac0>
-Traceback (most recent call last):
-  File "/home/harsh/.pyenv/versions/dpv/lib/python3.11/site-packages/rdflib/term.py", line 2084, in _castLexicalToPython
-    return conv_func(lexical)  # type: ignore[arg-type]
-           ^^^^^^^^^^^^^^^^^^
-  File "/home/harsh/.pyenv/versions/dpv/lib/python3.11/site-packages/isodate/isodates.py", line 203, in parse_date
-    raise ISO8601Error('Unrecognised ISO 8601 date format: %r' % datestring)
-isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: ''
-Failed to convert Literal lexical form to value. Datatype=http://www.w3.org/2001/XMLSchema#date, Converter=<function parse_date at 0x7f4e95382ac0>
-Traceback (most recent call last):
-  File "/home/harsh/.pyenv/versions/dpv/lib/python3.11/site-packages/rdflib/term.py", line 2084, in _castLexicalToPython
-    return conv_func(lexical)  # type: ignore[arg-type]
-           ^^^^^^^^^^^^^^^^^^
-  File "/home/harsh/.pyenv/versions/dpv/lib/python3.11/site-packages/isodate/isodates.py", line 203, in parse_date
-    raise ISO8601Error('Unrecognised ISO 8601 date format: %r' % datestring)
-isodate.isoerror.ISO8601Error: Unrecognised ISO 8601 date format: ''
 
 --- DPV --- 
 MODULE: base
@@ -39,7 +7,9 @@ MODULE: consent_status
 added: 0 ; removed: 0
 ---
 MODULE: consent
-added: 0 ; removed: 0
+added: 0 ; removed: 1
+Concepts Removed
+https://w3id.org/dpv#expiry
 ---
 MODULE: consent_types
 added: 0 ; removed: 0
@@ -60,49 +30,22 @@ MODULE: entities_organisation
 added: 0 ; removed: 0
 ---
 MODULE: entities
-added: 1 ; removed: 0
-
-Concepts Added
-https://w3id.org/dpv#isRepresentativeFor
+added: 0 ; removed: 0
 ---
 MODULE: jurisdiction
-added: 1 ; removed: 1
-Concepts Removed
-https://w3id.org/dpv#PubliclyLocation
-
-Concepts Added
-https://w3id.org/dpv#PublicLocation
+added: 0 ; removed: 0
 ---
 MODULE: legal_basis
 added: 0 ; removed: 0
 ---
 MODULE: organisational_measures
-added: 1 ; removed: 5
-Concepts Removed
-https://w3id.org/dpv#ReviewImpactAssessmentConformance
-https://w3id.org/dpv#ReviewImpactAssessmentAdequacy
-https://w3id.org/dpv#SecurityAssessments
-https://w3id.org/dpv#DataProcessingRecords
-https://w3id.org/dpv#CybersecurityAssessments
-
-Concepts Added
-https://w3id.org/dpv#CybersecurityAssessment
+added: 0 ; removed: 0
 ---
 MODULE: personal_data
-added: 3 ; removed: 0
-
-Concepts Added
-https://w3id.org/dpv#IncorrectData
-https://w3id.org/dpv#UnverifiedData
-https://w3id.org/dpv#VerifiedData
+added: 0 ; removed: 0
 ---
 MODULE: processing_context
-added: 3 ; removed: 0
-
-Concepts Added
-https://w3id.org/dpv#DataController
-https://w3id.org/dpv#DataSubject
-https://w3id.org/dpv#ThirdParty
+added: 0 ; removed: 0
 ---
 MODULE: processing_scale
 added: 0 ; removed: 0
@@ -111,32 +54,7 @@ MODULE: processing
 added: 0 ; removed: 0
 ---
 MODULE: purposes
-added: 12 ; removed: 10
-Concepts Removed
-https://w3id.org/dpv#IdentifyRectifyImpairments
-https://w3id.org/dpv#CreateEventRecommendations
-https://w3id.org/dpv#HumanResources
-https://w3id.org/dpv#RegistrationAuthentication
-https://w3id.org/dpv#ServicePersonalization
-https://w3id.org/dpv#CreatePersonalisedRecommendations
-https://w3id.org/dpv#ServiceRecordManagement
-https://w3id.org/dpv#Payment
-https://w3id.org/dpv#ServiceOptimization
-https://w3id.org/dpv#CreateProductRecommendations
-
-Concepts Added
-https://w3id.org/dpv#RepairImpairments
-https://w3id.org/dpv#FulfilmentOfContractualObligation
-https://w3id.org/dpv#PaymentManagement
-https://w3id.org/dpv#FulfilmentOfObligation
-https://w3id.org/dpv#ServicePersonalisation
-https://w3id.org/dpv#ServiceRegistration
-https://w3id.org/dpv#ServiceOptimisation
-https://w3id.org/dpv#SearchFunctionalities
-https://w3id.org/dpv#ProvideProductRecommendations
-https://w3id.org/dpv#ProvidePersonalisedRecommendations
-https://w3id.org/dpv#EstablishContractualAgreement
-https://w3id.org/dpv#ProvideEventRecommendations
+added: 0 ; removed: 0
 ---
 MODULE: risk
 added: 0 ; removed: 0
@@ -145,7 +63,10 @@ MODULE: status
 added: 0 ; removed: 0
 ---
 MODULE: technical_measures
-added: 0 ; removed: 0
+added: 0 ; removed: 2
+Concepts Removed
+https://w3id.org/dpv#DataAnonymisationTechnique
+https://w3id.org/dpv#CrytographicMethods
 ---
 MODULE: technical_organisational_measures
 added: 0 ; removed: 0
@@ -156,7 +77,21 @@ added: 0 ; removed: 0
 
 --- DPV-GDPR --- 
 MODULE: legal_basis
-added: 0 ; removed: 0
+added: 12 ; removed: 0
+
+Concepts Added
+https://w3id.org/dpv#VitalInterestOfNaturalPerson
+https://w3id.org/dpv/dpv-gdpr#A6-1-f-controller
+https://w3id.org/dpv#LegitimateInterestOfThirdParty
+https://w3id.org/dpv#LegitimateInterestOfController
+https://w3id.org/dpv/dpv-gdpr#A6-1-d-data-subject
+https://w3id.org/dpv/dpv-gdpr#A6-1-f-third-party
+https://w3id.org/dpv/dpv-gdpr#A6-1-d-natual-person
+https://w3id.org/dpv#ContractPerformance
+https://w3id.org/dpv/dpv-gdpr#A6-1-b-enter-into-contract
+https://w3id.org/dpv#VitalInterestOfDataSubject
+https://w3id.org/dpv#EnterIntoContract
+https://w3id.org/dpv/dpv-gdpr#A6-1-b-contract-performance
 ---
 MODULE: legal_basis_special
 added: 0 ; removed: 0
@@ -165,39 +100,13 @@ MODULE: legal_basis_data_transfer
 added: 0 ; removed: 0
 ---
 MODULE: rights
-added: 6 ; removed: 0
-
-Concepts Added
-https://w3id.org/dpv/dpv-gdpr#IndirectDataCollectionNotice
-https://w3id.org/dpv/dpv-gdpr#SARNotice
-http://www.w3.org/ns/dcat#Resource
-https://w3id.org/dpv/dpv-gdpr#RightsRecipientsNotice
-https://w3id.org/dpv/dpv-gdpr#DirectDataCollectionNotice
-https://w3id.org/dpv#RightFulfilmentNotice
+added: 0 ; removed: 0
 ---
 MODULE: data_transfers
 added: 0 ; removed: 0
 ---
 MODULE: dpia
-added: 16 ; removed: 0
-
-Concepts Added
-http://purl.org/dc/terms/isVersionOf
-http://purl.org/dc/terms/title
-http://purl.org/dc/terms/created
-http://purl.org/dc/terms/coverage
-http://purl.org/dc/terms/hasPart
-http://purl.org/dc/terms/subject
-http://purl.org/dc/terms/description
-http://purl.org/dc/terms/isPartOf
-http://purl.org/dc/terms/dateSubmitted
-http://purl.org/dc/terms/dateAccepted
-http://purl.org/dc/terms/valid
-http://purl.org/dc/terms/temporal
-http://purl.org/dc/terms/conformsTo
-https://w3id.org/dpv#hasStatus
-http://purl.org/dc/terms/modified
-http://purl.org/dc/terms/identifier
+added: 0 ; removed: 0
 ---
 MODULE: compliance
 added: 0 ; removed: 0
@@ -210,7 +119,21 @@ added: 0 ; removed: 0
 
 --- DPV-LEGAL --- 
 MODULE: authorities
-added: 0 ; removed: 0
+added: 12 ; removed: 0
+
+Concepts Added
+https://w3id.org/dpv/dpv-legal#US-NV
+https://w3id.org/dpv/dpv-legal#DPA-US-NV
+https://w3id.org/dpv/dpv-legal#US-CT
+https://w3id.org/dpv/dpv-legal#DPA-US-VC
+https://w3id.org/dpv/dpv-legal#US-CO-CPA
+https://w3id.org/dpv/dpv-legal#US-VC
+https://w3id.org/dpv/dpv-legal#US-VA-VCDPA
+https://w3id.org/dpv/dpv-legal#US-CO
+https://w3id.org/dpv/dpv-legal#DPA-US-CO
+https://w3id.org/dpv/dpv-legal#US-NV-NPICICA
+https://w3id.org/dpv/dpv-legal#DPA-US-CT
+https://w3id.org/dpv/dpv-legal#US-CT-CTPA
 ---
 MODULE: eu_adequacy
 added: 0 ; removed: 0
@@ -219,7 +142,20 @@ MODULE: eu_eea
 added: 0 ; removed: 0
 ---
 MODULE: laws
-added: 0 ; removed: 0
+added: 9 ; removed: 1
+Concepts Removed
+https://w3id.org/dpv/dpv-legal#GB-DPA-2018
+
+Concepts Added
+https://w3id.org/dpv/dpv-legal#US-CT
+https://w3id.org/dpv/dpv-legal#US-VA-VCDPA
+https://w3id.org/dpv/dpv-legal#US-VA
+https://w3id.org/dpv/dpv-legal#US-CO-CPA
+https://w3id.org/dpv/dpv-legal#US-NV
+https://w3id.org/dpv/dpv-legal#US-NV-NPICICA
+https://w3id.org/dpv/dpv-legal#GB-DPA
+https://w3id.org/dpv/dpv-legal#US-CO
+https://w3id.org/dpv/dpv-legal#US-CT-CTPA
 ---
 MODULE: locations
 added: 0 ; removed: 0

documentation-generator/logs/releases.txt

@@ -1,5 +1,5 @@
 generated releases/dpv.zip
 generated releases/dpv-skos.zip
 generated releases/dpv-owl.zip
-generated releases/dpv.xlsx
+generated releases/dpv-xlsx.zip
 generated releases/dpv-nace.zip