documents: simplify subjects structure #3278

jma · 2023-03-08T13:03:52Z

Uses entity for subjects and imported subjects.
Uses entity for genreForm and imported genreForm.
Uses entities as subject subdivisions.
Removes non working entity links such as timestamps, places, etc.
Removes entity links for imported version of subjects and genreForm.
Removes subject factory.

Why are you opening this PR?

Which task/US does it implement?
Which issue does it fix?

Dependencies

My PR depends on the following rero-ils-ui's PR(s):

rero/rero-ils-ui#

How to test?

What command should I have to run to test your PR?
What should I test through the UI?

rerowep · 2023-03-13T09:03:59Z

rero_ils/modules/documents/dojson/contrib/marc21tojson/rero/model.py

                build_string_from_subfields(
                    value,
                    subfield_code_per_tag[creator_tag_key]),
                '.', '.'
-            )
+            ) + '. ' + subject['authorized_access_point']


Don't understand why we have to add subject['authorized_access_point'] again.
If we really have to add it it is preferable to work with f strings.

@rerowep in this case (600t, 610t, 611t), it seems some subfields value must be prepend to subject term.

If we really have to add it it is preferable to work with f strings.

I do not see a simple way to do this. Can you provide an example?

rerowep · 2023-03-13T09:09:46Z

rero_ils/modules/documents/jsonschemas/documents/document_contribution_local-v0.0.1.json

@@ -0,0 +1,59 @@
+{
+  "title": "Local Entity",


Maybe better to name Local Contribution?

The idea is to express the fact that a contribution is an entity. The professional should "see" that an entity can be a subfield of different document fields.

rerowep · 2023-03-13T09:12:24Z

rero_ils/modules/documents/jsonschemas/documents/document_genre_form-v0.0.1.json

-                  }
-                ]
-              }
+      "title": "Entities",


File name and title not matching ?

No but should it be?

rerowep · 2023-03-13T09:12:42Z

rero_ils/modules/documents/jsonschemas/documents/document_genre_form_local-v0.0.1.json

@@ -0,0 +1,60 @@
+{
+  "title": "Local Entity",


File name and title not matching ?

No but should it be?

rerowep · 2023-03-13T09:13:07Z

rero_ils/modules/documents/jsonschemas/documents/document_subjects-v0.0.1.json

-            },
-            "place_subdivisions": {
-              "$ref": "#/definitions/place_subdivisions"
+      "title": "Entities",


File name and title not matching ?

No but should it be?

rerowep · 2023-03-13T09:18:10Z

rero_ils/modules/documents/views.py

-        # print(str(err))
-        return 'Subject parsing error !'
+    entity = subject_data['entity']
+    if language is None:


Normally we don't need this test, because entity.get(... will do the job, but it is more clear this way.

zannkukai · 2023-03-13T15:21:16Z

rero_ils/modules/documents/dojson/contrib/marc21tojson/rero/model.py

                build_string_from_subfields(
                    value,
                    subfield_code_per_tag[creator_tag_key]),
                '.', '.'
-            )
+            ) + '. ' + subject['authorized_access_point']


@rerowep in this case (600t, 610t, 611t), it seems some subfields value must be prepend to subject term.

zannkukai · 2023-03-13T15:23:55Z

rero_ils/modules/documents/dojson/contrib/marc21tojson/utils.py

+        'z': 'bf:Place'
+    }
+    for tag, val in value.items():
+        if tag in subdivisions.keys():


COSMETIC : you could use if tag in subdivisions: . Using this notation on a dict, will check into dict.keys()... but it's maybe less human readable.

zannkukai · 2023-03-13T15:33:58Z

rero_ils/modules/documents/utils_mef.py

+                for subject in [
+                    res.get('entity') for res in doc.get(self.name, [])
+                ]:


COSMETIC : maybe add a default value for res.get('entity', {}). It should prevent any error if entity isn't present (I know it's required into JSON schema... but 2 security systems are always better than only one)

zannkukai · 2023-03-13T15:37:40Z

tests/api/documents/test_documents_dumpers.py

+        'entity': {
+            '$ref': 'n/a',
+            'pid': 'n/a',
+            'type': DocumentSubjectType.PERSON


We could use EntityType class and remove DocumentSubjectType as we already removed subjectFactory

PascalRepond · 2023-03-14T07:17:38Z

rero_ils/modules/documents/jsonschemas/documents/document-v0.0.1.json

+      "items": {
+        "type": "object",
+        "title": "Entities",
+        "description": "Topic (including genre/form), place, temporal, person, family or corporate body (including conferences). Always create a link to IdRef or GND, if possible.",


"Imported topic, place, temporal, person, family or corporate body (including conferences), used for information. Do not enter data manually."

PascalRepond · 2023-03-14T07:19:07Z

rero_ils/modules/documents/jsonschemas/documents/document-v0.0.1.json

+      "items": {
+        "type": "object",
+        "title": "Entities",
+        "description": "Topic (including genre/form), place, temporal, person, family or corporate body (including conferences). Always create a link to IdRef or GND, if possible.",


"Imported genre or form, used for information. Do not enter data manually."

PascalRepond · 2023-03-14T07:29:36Z

rero_ils/modules/documents/jsonschemas/documents/document-v0.0.1.json

-              "cssClass": "editor-title"
-            }
+      "title": "Genres, forms (imported)",
+      "description": "Genre of form from imported bibliographic data",


"Genre or form imported from external data"

PascalRepond · 2023-03-14T07:30:09Z

rero_ils/modules/documents/jsonschemas/documents/document-v0.0.1.json

-              "cssClass": "editor-title"
-            }
+      "title": "Subjects (imported)",
+      "description": "Subject from imported bibliographic data",


"Subject imported from external data"

PascalRepond · 2023-03-14T07:33:23Z

rero_ils/modules/documents/jsonschemas/documents/document_entity_local-v0.0.1.json

+    },
+    "source": {
+      "title": "Source",
+      "description": "Source of the subject.",


"Source catalog where the subject was imported from."

PascalRepond · 2023-03-14T07:34:20Z

rero_ils/modules/documents/jsonschemas/documents/document_entity_local-v0.0.1.json

+      "items": {
+        "type": "object",
+        "title": "Entities",
+        "description": "Topic (including genre/form), place, temporal, person, family or corporate body (including conferences). Always create a link to IdRef or GND, if possible.",


"Topic (including genre/form), place, temporal, person, family or corporate body (including conferences). Always create a link to IdRef or GND, if possible."

Seems to be a copy from the original description.

PascalRepond · 2023-03-14T07:37:12Z

rero_ils/modules/documents/jsonschemas/documents/document_entity_local-v0.0.1.json

+        "bf:Topic",
+        "bf:Concept",
+        "bf:Place",
+        "bf:Temporal"


bf:Person and bf:Organisation?

No as subdivision does not allows Person nor Organisation.

so default should not be bf:Person

PascalRepond · 2023-03-14T07:38:51Z

rero_ils/modules/documents/jsonschemas/documents/document_genre_form-v0.0.1.json

-                ]
-              }
+      "title": "Entities",
+      "description": "Topic (including genre/form), place, temporal, person, family or corporate body (including conferences). Always create a link to IdRef or GND, if possible.",


"Genre or form of the document. Always create a link to IdRef or GND, if possible."

PascalRepond · 2023-03-14T07:45:53Z

rero_ils/modules/documents/jsonschemas/documents/document_genre_form_local-v0.0.1.json

+    "source": {
+      "title": "Source",
+      "description": "Source of the subject.",
+      "type": "string",
+      "minLength": 3,
+      "form": {
+        "templateOptions": {
+          "itemCssClass": "col-lg-6"
+        }


Not needed.

* Uses entity for subjects and imported subjects. * Uses entity for genreForm and imported genreForm. * Uses entities as subject subdivisions. * Removes non working entity links such as timestamps, places, etc. * Removes entity links for imported version of subjects and genreForm. * Removes subject factory. Co-Authored-by: Johnny Mariéthoz <[email protected]> Co-Authored-by: Renaud Michotte <[email protected]>

github-actions bot added dev: fixtures Fixtures data used for ils.test and ilsdev.test f: data About data model, importation, transformation, exportation of data, specific for bibliographic data f: data migration Data migration from a legacy system or a previous version f: search labels Mar 8, 2023

jma force-pushed the maj-subjects branch 3 times, most recently from 3e373c3 to aac5d59 Compare March 9, 2023 15:56

jma changed the title ~~in progress~~ documents: simplify subjects structure Mar 9, 2023

jma force-pushed the maj-subjects branch from aac5d59 to 3a2fd5d Compare March 9, 2023 15:58

jma marked this pull request as ready for review March 9, 2023 18:36

jma requested review from rerowep, zannkukai and lauren-d March 9, 2023 18:36

rerowep reviewed Mar 13, 2023

View reviewed changes

zannkukai assigned jma and zannkukai Mar 13, 2023

zannkukai approved these changes Mar 13, 2023

View reviewed changes

jma force-pushed the maj-subjects branch from 3a2fd5d to f14fc60 Compare March 14, 2023 06:56

jma requested a review from PascalRepond March 14, 2023 06:56

github-actions bot added the dev: DB Database structure (models.py) label Mar 14, 2023

PascalRepond reviewed Mar 14, 2023

View reviewed changes

jma requested a review from rerowep March 14, 2023 12:54

rerowep approved these changes Mar 15, 2023

View reviewed changes

jma force-pushed the maj-subjects branch from f14fc60 to ce334a6 Compare March 16, 2023 13:16

jma force-pushed the maj-subjects branch from ce334a6 to fc5d8ba Compare March 16, 2023 13:18

jma requested a review from PascalRepond March 16, 2023 13:20

PascalRepond approved these changes Mar 16, 2023

View reviewed changes

jma merged commit 97b3635 into rero:US-2434-entity Mar 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

documents: simplify subjects structure #3278

documents: simplify subjects structure #3278

jma commented Mar 8, 2023 •

edited

Loading

rerowep Mar 13, 2023

zannkukai Mar 13, 2023

jma Mar 14, 2023

rerowep Mar 13, 2023

jma Mar 14, 2023

rerowep Mar 13, 2023

jma Mar 14, 2023

rerowep Mar 13, 2023

jma Mar 14, 2023

rerowep Mar 13, 2023

jma Mar 14, 2023

rerowep Mar 13, 2023

zannkukai Mar 13, 2023

zannkukai Mar 13, 2023

zannkukai Mar 13, 2023

zannkukai Mar 13, 2023

PascalRepond Mar 14, 2023

PascalRepond Mar 14, 2023

PascalRepond Mar 14, 2023

PascalRepond Mar 14, 2023

PascalRepond Mar 14, 2023

PascalRepond Mar 14, 2023

jma Mar 16, 2023 •

edited

Loading

PascalRepond Mar 14, 2023

jma Mar 16, 2023

PascalRepond Mar 16, 2023

PascalRepond Mar 14, 2023

PascalRepond Mar 14, 2023

documents: simplify subjects structure #3278

documents: simplify subjects structure #3278

Conversation

jma commented Mar 8, 2023 • edited Loading

Why are you opening this PR?

Dependencies

How to test?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jma Mar 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jma commented Mar 8, 2023 •

edited

Loading

jma Mar 16, 2023 •

edited

Loading