WIP Prod for legacy #3384

Co-authored-by: Alessio Deiana <[email protected]> Signed-off-by: Alessio Deiana <[email protected]>

* Improves the standard Journal tokenizer to keep page-end when indexing, as well as adding new journal,volume pairs. * Amends the search_engine to specially handle searches in the journal field with page ranges to make use of the new tokenized strings. This fixes issues seen in BibRank and other search engine clients that passes a valid full page range and obtains no result (because it was not indexed!). Co-authored-by: Alessio Deiana <[email protected]>

* For certain journal search queries with INSPIRE style page ranges the user is now presented with an alternative result without the page-end part of the query, should the original query return no results. Co-authored-by: Alessio Deiana <[email protected]>

* Session is now correctly initialized before use.

* Now handles correctly repeatable affiliations

* Optimised ticketing system sync, significantly reduces server polling load.

* Introduces a new parameter in the datacacher that defaults to 30s. This is the mininum time the data is cached. That means that calling recreate_cache_if_needed() does nothing if it was called less than 30s ago.

* When bibrank -w citation fails because we lost too many citations, displays in the log for each record how many citations they lost.

* Clicking on a name on a paper now brings correctly to the author page instead of a person search page. Solves some minor ambiguity problems.

* BibTasks are typically loading large quantities of data from the database. This is not easily cacheable, therefore query_cache_type is locally disabled. Reported-by: Thorsten Schwander <[email protected]> Signed-off-by: Samuele Kaplun <[email protected]>

* Silences "request data read error" IOError seen when the client closes a connection upon POSTing. * Clean up exception raising in general. Signed-off-by: Samuele Kaplun <[email protected]>

* Now /author/manage_profile/ correctly displays an error message when visited without the expected argument. Signed-off-by: Samuele Carli <[email protected]> Reviewed-by: Samuele Kaplun <[email protected]>

* Adds % to keyword lookup.

* If the client sends invalid header containing '/n', send back a 400 message instead of creating an exception

* Extracts transform links out of the HTMLWasher class to allow easy use in other context Co-authored-by: Jan Aage Lavik <[email protected]>

Co-authored-by: Jan Aage Lavik <[email protected]> Co-authored-by: Alessio Deiana <[email protected]>

* Adds a try..except around __write and raises an exception in case of GET so that we interrupt the processing of request (prevents useless computation) Signed-off-by: Alessio Deiana <[email protected]>

* Since try...excepts are all over the place, we need extra code to silence this exception.

* Gives priority to site/dist-packages when importing HTMLParser. In this way, when on Python-2.7 and HTMLParser is installed from pypi, it will use the latter. Signed-off-by: Samuele Kaplun <[email protected]> Reviewed-by: Javier Martin Montull <[email protected]> Reviewed-by: Jan Aage Lavik <[email protected]>

* The assertion message was generating an exception due to a bad string replacement. Signed-off-by: Carli Samuele <[email protected]>

* The system crashed in case something goes wrong and the orcid was not assignet to a profile yet when the import was called. Now this is handled gracefully. Signed-off-by: Carli Samuele <[email protected]>

* Updates the run_shell_command to only escape_shell_args when there are actual args to escape. Otherwise errors related to Python string formatting may happen. Signed-off-by: Jan Aage Lavik <[email protected]>

* Fixes UnboundLocalError when number of server retries reach it's maximum and the code is trying to call a variable that is not defined. * Fixes a few PEP8 issues. Signed-off-by: Jan Aage Lavik <[email protected]>

* When 'login_method' is missing from user preferences, the exception is now gracefully raised and execution continues using default values. Co-authored-by: Samuele Kaplun <[email protected]>

* Do not report KeyboardInterrupt exception. (fixes inveniosoftware#1701) Signed-off-by: Jan Aage Lavik <[email protected]>

* Reverts some changes made in f5969cf to fix the 'keywords' tab, as exceptions were raised for bad use of parameters. * Adds desired additions made in f5969cf instead in BibClassify module instead of having it in search_engine. Signed-off-by: Jan Aage Lavik <[email protected]> Conflicts: modules/websearch/lib/search_engine.py

* Adds a readable representation of BibFormatObject so that it can be used in debugging etc. Signed-off-by: Jan Aage Lavik <[email protected]>

* When bibrank -w citation fails because we lost too many citations, displays in the log for each record how many citations they lost. * Abort bibrank task when the loss for a record is greater than x. (x is set in the citation.cfg) Conflicts: modules/bibrank/lib/bibrank_citation_indexer.py

* When creating a BibFormatObject, if the recid passed is not an int cast it and alert the admin

* ClientDisconnected is trigger volontarely. It should never be reported. * KeyboardInturrupt is most likely called when a script is run from the command line and interrupted voluntarely thus not requiring alerting either.

* When creating a BibFormatObject, if the recid passed is not an int or a long cast it and alert the admin. * Also convert to int when it is taken from the marcxml.

Conflicts: modules/bibrank/lib/bibrank_citation_indexer.py

* The differ was using the cites dictionary twice when checking for the overall citation balances in bibrank -w citation.

* Updates the per-record citation/reference loss checker to not abort updating when too many are lost and insteads warns admins' to check this record manually. If too many references are lost overall, execution is still aborted. * If loss is detected a ticket is sent to "CitationLoss" queue for CFG_INSPIRE_SITE. Signed-off-by: Jan Aage Lavik <[email protected]> Reviewed-by: Javier Martin Montull <[email protected]>

* Removes underused (and undocumented) fulltext_status and introduces new export_fulltext flag in sitemap.cfg to fully enable/disable exporting of visible fulltext files to crawlers.

Conflicts: modules/docextract/lib/refextract_config.py

* Refactoring of book search. Signed-off-by: Alessio Deiana <[email protected]> Tested-by: Tibor Simko <[email protected]> Conflicts: modules/docextract/lib/refextract_config.py

* In 13774cd it was introduced a doi tag in the EndNote export that was not matching the EndNote format. This is now fixed. Signed-off-by: Samuele Kaplun <[email protected]> Reported-by: George Alverson <[email protected]>

* get_pretty_traceback would display the line of the end of the scope instead of the line the exception appeared in.

* Adds LIGO report numbers to the kb. Conflicts: modules/docextract/lib/refextract_regression_tests.py

* Fixes a bug when setting the primary profile. The merge_primary_profile was initizialized properly. Reviewed-by: Petros Ioannidis <[email protected]>

* Do not trigger arxiv PDF checking based on modification timestamp of records in INSPIRE, rather check for modified records on arXiv. Signed-off-by: Samuele Kaplun <[email protected]>

* Skip records that are not in selected collections (in citation.cfg)

Signed-off-by: Alessio Deiana <[email protected]> Conflicts: modules/docextract/lib/refextract_config.py

* Upon parsing MARCXML in input, bibupload now calls wash_for_xml() in order to remove very broken UTF8 characters that are not valid within XML. Co-authored-by: Alessio Deiana <[email protected]> Signed-off-by: Samuele Kaplun <[email protected]>

* BibFormat would error when call format_record() with marcxml containning no 001 tag. Conflicts: modules/bibformat/lib/bibformat_engine.py

* Updates the authorlist conversion to allow multiple $u affiliations and properly parse INSPIRE IDs. Signed-off-by: Jan Aage Lavik <[email protected]>

* Replace m='r' with a simple star query, less precise but way faster. Signed-off-by: Alessio Deiana <[email protected]> Acked-by: Samuele Carli <[email protected]>

* Adds the filename of the extracted files in the 999C6f field. Signed-off-by: Alessio Deiana <[email protected]>

* Changes captcha JS source to https Signed-off-by: Javier Martin Montull <[email protected]>

* Disable the garbage collection at the end of the request. The connection to the client would remain open (and probably the flushing wouldn't happen either) till gc.collect() finished.

* Prevents the output of debug information when the content-type is not text/html. This prevents from breaking json and latex.

* Prevents to store None in the citation weights cache.

This reverts commit adc292c.

* This commit complements 3004c20 patch to also check if AFS PATTERN is not empty. Signed-off-by: Jan Aage Lavik <[email protected]>

* Fixes get_papers_affected_since() to consider timestamps in an inclusive manner. Signed-off-by: Samuele Kaplun <[email protected]> Reviewed-by: Samuele Carli <[email protected]>

* Adds parameter --starting-date. * Stores the last harvest and reuses it as starting point if no starting date is provided. Co-authored-by: Samuele Kaplun <[email protected]> Signed-off-by: Samuele Kaplun <[email protected]>

* Fixes the following bug: DATE AFTER XXXX is interpreted * year .GT. xxxx in Spires * year .GE. xxxx in Inspires * Fixes unit tests for date after/before queries. * Deletes a duplicate unit test for 'find date 1976-04'. * Adds unit tests for 'find date < 1978-10'. * Adds unit tests for 'find date > 1978-10'. * Adds unit tests for 'find date 23 Sep 2010'. * Adds unit tests for 'find (date 23 Sep 2010)'. The parenthesis after the date is an edge case for the regexp used by th method 'convert_date'. * Fixes method '_convert_spires_date_after_to_invenio_span_query'. * Fixes method '_convert_spires_date_before_to_invenio_span_query'. * Fixes regexp used by '_convert_all_dates'. * Fixes indentation for kwalitee.

* Dumps important information to a pickled file in /tmp-shared/ called /tmp/orcid_dictionary_problem_[TIMESTAMP].cPickle in case of protocol error. Reviewed-by: Laura Rueda <[email protected]> Acked-by: Jan Aage Lavik <[email protected]>

* We now set a priority on which title form is the most important. References will be given priority over anything else while Reference will ranked last with Bibliography in between. That means that if "Bibliography" is found in the text, we assume it is the reference section and we don't look for a section titles "Reference".

* Mocks the query to arxiv that happened during regression tests.

* For testing, use jalavik repo that supports reponame != invenio. * NOTE Remove when upstream invenio-devscripts is updated. Signed-off-by: Jan Aage Lavik <[email protected]>

* Ignores references from superseeded records. Superseeded records have the tag 78502 defined.

* Accept recid=None in format_record() parameters.

* Allows to index all records when collections = ""

* Computes affiliations during inveniocfg --load-demo-records

* Updates tests to match the new journal index.

This reverts commit f9c1185.

* Changes ->9999 to ->9999-01-01

* Fixes detection of superseeded records when doing the citations catchup in the citation indexing phase.

* Renames the label 'HepData' to 'Data' in the details page. * Renames the url 'record/123/hepdata' to 'record/123/data' Signed-off-by: Federico Poli <[email protected]> Conflicts: modules/websearch/lib/search_engine.py

* Expect 9999-01-01 instead of 9999 in unit tests.

* string_partition (python 2.4 compatibility) was replaced by python's string.partition in the standard_library. Signed-off-by: Minn Soe <[email protected]>

* Default directory is now under /opt/invenio/var/cache.

* In author pages check that self.person_id is an integer before querying for the author tables.

Signed-off-by: Samuele Kaplun <[email protected]>

* Does not filter anymore by collection:citeable in the counts displayed next to the tab title.

This reverts commit 89c9319.

This reverts commit e6f5e06.

* Removes the recid from the information dictionary when computing citations so that references done via 999C50 are ignored for record not in the selected collections.

* INSPIRE Datasets were mistakenly shown as external publications. Signed-off-by: Samuele Carli <[email protected]> Signed-off-by: Artem Tsikiridis <[email protected]>

* Replaces the SPIRES operators `<=` and `>=` with the Invenio operators before the equal signs removal and before replacing the date before/after operators * Adds unit tests to the query parser Signed-off-by: Federico Poli <[email protected]>

* Show an error page when committing already committed tickets instead of crashing miserably. Signed-off-by: Artem Tsikiridis <[email protected]> Signed-off-by: Samuele Carli <[email protected]>

Signed-off-by: Artem Tsikiridis<[email protected]>

* We already open a ticket so remove the exception reporting.

* Refactors the regression test `BibIndexVirtualIndexQueueTableTest`, in order to keep the queue table clean even if a test fails. Signed-off-by: Federico Poli <[email protected]>

* Disables the filtering step that uses `hstRECORD` table: set always `force_all_indexes=True` when calling `find_affected_records_for_index` * Fixes `find_affected_records_for_index` when called with `recIDs=[]` * Fixes some regression test that relied on the utilization of `hstRECORD` in BibIndex Signed-off-by: Federico Poli <[email protected]>

Signed-off-by: Jan Aage Lavik <[email protected]>

Signed-off-by: Artem Tsikiridis <[email protected]>

* Order linkbacks by id when they have the same insert date in test_get_approved_latest_added_linkback so that tests pass reliably.

* Use bibxxx ids in the self cites tables when bibauthorid is disabled.

* Adds `.drone.yml` for Drone continuous integration. Signed-off-by: Federico Poli <[email protected]> Co-authored-by: Alessio Deiana <[email protected]>

* When using the -a/--all CLI parameter of BibCheck, actually performs to reset of the last_run timestamp only upon execution of the bibtask and not when it is scheduled. * Improves bibtask verbosity. Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Mateusz Susik <[email protected]>

* Adds the ability for tasks to transfer arbitrary data to BibTasklets specified via the `--post-process=` argument via a new task parameter `post_process_params`. Reviewed-by: Jan Aage Lavik <[email protected]> Signed-off-by: Georgios Papoutsakis <[email protected]>

* Adds sending of added/modified record IDs during webcoll run to `post-process` tasklet via a new task parameter `post-process_params`. Amended-by: Jan Aage Lavik <[email protected]> Signed-off-by: Georgios Papoutsakis <[email protected]>

Signed-off-by: Jan Aage Lavik <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

* Avoids crashing in invenio.wsgi during pre-loading of data structures. In this way, even in case of bugs in one of the calls, the site can still load. Signed-off-by: Samuele Kaplun <[email protected]>

* Filters first and ignores updates for extraction of plots, fulltext authors, references etc. Signed-off-by: Jan Aage Lavik <[email protected]>

Signed-off-by: Jan Aage Lavik <[email protected]>

* Amends the several regular expressions spread around the code that deal with arXiv identifiers so that the upcoming 5-digits based identifiers are also supported. Signed-off-by: Samuele Kaplun <[email protected]>

* Since the move to RT4, text key is not returned by RT.

* Fixes bug introduced in previous OPS commit where all insert files were not correctly handled because the wrong extension was looked for. Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

* Introduces support for selfcites family of operators in the special SPIRES syntax. Signed-off-by: Samuele Kaplun <[email protected]>

This reverts commit f8f2fc6. commit 83c131a addresses the underlying reason

Signed-off-by: Jan Aage Lavik <[email protected]>

* Handles Proceedings records when assigning texkeys. Signed-off-by: Javier Martin Montull <[email protected]>

* When having an FFT explicitly specifying a PDF file, in case CFG_INSPIRE_SITE = 1, check that the downloaded PDF is actually a PDF and not e.g. an HTML error page. Signed-off-by: Samuele Kaplun <[email protected]>

* Harvests PDFs for new records in the system that didn't came through regular harvesting and can therefore refer to old records on arXiv.org. Signed-off-by: Samuele Kaplun <[email protected]> Co-authored-by: Javier Martin Montull <[email protected]>

* Fixes an exception that will happen if CFG_ACCESS_CONTROL_LEVEL_SITE is configured to something else than 1 or 2 when page_not_authorized is triggered. Signed-off-by: Jan Aage Lavik <[email protected]>

* Store the earliest date among last time webcoll was run and the last bibindex `last_update` as the `last_run`, instead of the last time the rule was initiated. This prevents a race condition where records inserted between a webcoll run and the beginning of the following bibcheck ran were never considered by bibcheck. (resolves inveniosoftware#2387) Reported-by: Samuele Kaplun <[email protected]> Co-authored-by: Samuele Kaplun <[email protected]>

* In case a record is malformed (e.g. because of a missing subfield code, captures the corresponding exceptions and correctly register the record as invalid (thus issuing a ticket). Signed-off-by: Samuele Kaplun <[email protected]>

* If both BibRank and BibIndex are using Solr to index fulltext files when BibRank is updating the corresponding last_updated column at the end of a run, also takes care to update the corresponding last_updated column in bibindex. Signed-off-by: Samuele Kaplun <[email protected]>

* Fixes infinite loop in beautify_range_list() function that was triggered in particular conditions. Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Mateusz Susik <[email protected]>

* Improves bibtask polling by checking if the actual bibtask process that was just spawned is indeed dead before bailing out. Signed-off-by: Samuele Kaplun <[email protected]>

* Fixes outdated_caches() query that, by filtering out record not modified since the last bibreformat run, was implicitly skipping those records that were, by whatever reason, not reformatted in the past. Signed-off-by: Samuele Kaplun <[email protected]>

* Improves reporting of HTTP error codes in ways that can help crawlers correctly analyzing the available pages. Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

* Fixes the index repair workflow that had been broken in some previous patch. Now an index is repaired as first step, before analyzing whether is worth to update it or not. Signed-off-by: Samuele Kaplun <[email protected]>

* Makes any index having 'author' in the name, to fully consider BAIs. Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

This reverts commit 1e0d02a.

* Enforces proper UTF8 washing (lower-casing, stripping of accents) for the whole author tokenizer. Signed-off-by: Samuele Kaplun <[email protected]>

* Tries alternate patterns by default when matching a citation (e.g. useful when the citation includes ending page, but the cited record does not.) Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Jan Aage Lavik <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

* Uses phrase journal index instead of word journal index to avoid truncation. Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Jan Aage Lavik <[email protected]>

* Adds possibility to insert into holdingpen through robotupload. Signed-off-by: Javier Martin Montull <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

* Improves the referencecount tokenizer by using get_record and directly count the effective number of references rather than the number of subfields. Signed-off-by: Samuele Kaplun <[email protected]>

* In case of DB connection errors, sleeps 30s to account for when the database is simply restarted (e.g. after unattended upgrades). Signed-off-by: Samuele Kaplun <[email protected]>

* Sorry Thorsten... I know what you would say... :-) BTW I couldn't say to you goodbye! Signed-off-by: Samuele Kaplun <[email protected]>

* Improves the authorlist conversion to also ignore lowercase "undefined" in values. Signed-off-by: Jan Aage Lavik <[email protected]>

Signed-off-by: Jan Aage Lavik <[email protected]> Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Jan Aage Lavik <[email protected]>

* Finally proper fix the fix of the fix for the emergency fix. (in particular "'foo' in bar", where bar is an instance of an exception, does not do what one would expect, but simply returns False, even if str(bar) actually contains 'foo'). Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

* When BibUpload spots that two records are duplicates (e.g. because a DOI is being added to record 1 while it was already existing for record 2) creates a ticket in Asana with sensible links and prefixing the title of the link in case any of the involved records has in a way or another a 773__m field for Erratum Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

* Fixes crash when calculating next starting time for a rule, in case webcoll as not run, or bibindex has not run for all indexes. Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Samuele Kaplun <[email protected]>

Signed-off-by: Thorsten Schwander <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP Prod for legacy #3384

WIP Prod for legacy #3384

Commits on Jul 21, 2015