Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ogc 508 replace elastic search by postgres v3 #1559

Open
wants to merge 52 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
6ff58c9
Introduce postgres search views and integration (start over from v2 n…
Tschuppi81 Oct 24, 2024
18658fb
Introduce postgres search views and integration (start over from v2 n…
Tschuppi81 Oct 24, 2024
d0aad65
Put only upcoming events to the top of search results (instead of all…
Tschuppi81 Oct 25, 2024
0dd6e57
Revert several changes
Tschuppi81 Oct 25, 2024
0e165e3
store static search property data in separate column. Determine publi…
Tschuppi81 Oct 25, 2024
a3d2a4b
Fix linter issues
Tschuppi81 Oct 25, 2024
54d9860
Fix fsi search views and add tests
Tschuppi81 Oct 25, 2024
2e3816d
fix tests
Tschuppi81 Oct 25, 2024
f4b87d0
Adds to index data and filter for it during search
Tschuppi81 Oct 28, 2024
333f57a
Exclude members from seeing non es_public documents
Tschuppi81 Oct 29, 2024
7e652e4
Ensure member finds documents with access level 'member'
Tschuppi81 Oct 29, 2024
b021ef4
Resolve mypy issues
Tschuppi81 Oct 29, 2024
f75fccf
Fix statement for members
Tschuppi81 Oct 30, 2024
e2f9e00
Removed unused import
Tschuppi81 Oct 30, 2024
037d6ce
Fix member / manager filters
Tschuppi81 Oct 31, 2024
41b2714
Ignore mypy truthy-function and unreachable
Tschuppi81 Nov 4, 2024
0eef94f
Extend search tests
Tschuppi81 Nov 4, 2024
8e50ff5
Revert renaming
Tschuppi81 Nov 11, 2024
e11f317
Test for instance to identify events
Tschuppi81 Nov 11, 2024
1ed1d1d
Renaming variables
Tschuppi81 Nov 11, 2024
b168a85
Update src/onegov/org/models/search.py
Tschuppi81 Nov 12, 2024
39313d0
Update src/onegov/org/models/search.py
Tschuppi81 Nov 12, 2024
14bf8f2
Performance: use jsonb instead of json
Tschuppi81 Nov 14, 2024
c0f95ed
Rework search suggestions
Tschuppi81 Nov 18, 2024
2bdd14c
Fix static number of priorities for properties
Tschuppi81 Nov 25, 2024
8ab1dee
Merge master
Tschuppi81 Dec 2, 2024
d7f0958
mend
Tschuppi81 Dec 2, 2024
7c82279
Filter search models removing base classes
Tschuppi81 Dec 3, 2024
b13f212
Cleanup
Tschuppi81 Dec 3, 2024
d34584f
Also compare table name
Tschuppi81 Dec 3, 2024
2db81c5
Index only searchable base classes removing duplicates in search results
Tschuppi81 Dec 13, 2024
324ebbe
Update src/onegov/search/integration.py
Tschuppi81 Dec 16, 2024
e449cd7
Update src/onegov/search/utils.py
Tschuppi81 Dec 16, 2024
11e93ff
Fix syntax
Tschuppi81 Dec 16, 2024
5191975
Revert filter for base models for search
Tschuppi81 Dec 17, 2024
50ac000
Rework test
Tschuppi81 Dec 17, 2024
cad0655
Merge master
Tschuppi81 Dec 17, 2024
1f142d9
Move weighted tsvector to indexing step
Tschuppi81 Dec 23, 2024
d3912d3
Adjust search
Tschuppi81 Dec 24, 2024
127bc26
Adds a time decay function to search results
Tschuppi81 Dec 24, 2024
a68d9fd
Merge branch 'master' into ogc-508-replace-elastic-search-by-postgres-v3
Tschuppi81 Dec 24, 2024
775c48e
Fix test
Tschuppi81 Dec 24, 2024
61f3268
Resolve merge conflicts
Tschuppi81 Jan 17, 2025
95abc04
Town6 is now in h5 tag, simplify strip
Tschuppi81 Jan 17, 2025
90255e1
Merge branch 'master' into ogc-508-replace-elastic-search-by-postgres-v3
Tschuppi81 Mar 6, 2025
467f441
Rework reindexing storing all indices in own table
Tschuppi81 Mar 6, 2025
384f0f8
Remove unnecessary upgrade task
Tschuppi81 Mar 10, 2025
c4ffe6f
Rework search, searching over single table
Tschuppi81 Mar 11, 2025
9b0799c
Fix not all entries got indexed properly
Tschuppi81 Mar 11, 2025
9692cf6
Rework cli command for single search table
Tschuppi81 Mar 11, 2025
fbf7779
Calculate weighted vector only once
Tschuppi81 Mar 11, 2025
c99b7b4
Rework hashtag search
Tschuppi81 Mar 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions src/onegov/search/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
from langdetect.utils.lang_profile import LangProfile
from onegov.core.orm import find_models


from typing import Any, Generic, TypeVar, TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Callable, Iterator, Sequence
Expand Down Expand Up @@ -43,7 +42,9 @@ def searchable_sqlalchemy_models(
)


def filter_non_base_models(models: 'set[type[T]]') -> 'set[type[T]]':
def filter_non_base_models(
models: 'set[type[T]]'
) -> 'set[type[T]]':
""" Remove model classes that are base classes of other models in the set.
Args: models: set of model classes to filter
Returns: set: Model classes that are not base classes of any other model
Expand All @@ -55,7 +56,8 @@ def filter_non_base_models(models: 'set[type[T]]') -> 'set[type[T]]':
for model in models:
is_base = False
for other_model in models:
if model is not other_model and issubclass(other_model, model):
if (model is not other_model and issubclass(other_model, model)
and model.__tablename__ == other_model.__tablename__): # type:ignore[attr-defined]
is_base = True
break

Expand Down
45 changes: 43 additions & 2 deletions tests/onegov/search/test_utils.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from onegov.search import ORMSearchable, Searchable
from onegov.search import utils
from sqlalchemy import Column, Integer, Text
from sqlalchemy import Column, Integer, Text, ForeignKey
from sqlalchemy.ext.declarative import declarative_base


Expand Down Expand Up @@ -29,7 +29,6 @@ class D(Bar, Searchable):


def test_get_searchable_sqlalchemy_models_inheritance():

Base = declarative_base()

class Page(Base, Searchable):
Expand All @@ -42,11 +41,53 @@ class Topic(Page):
class News(Page):
pass

class B(Base):
id = Column(Integer, primary_key=True)
__tablename__ = 'b'

assert list(utils.searchable_sqlalchemy_models(Base)) == [
Page, Topic, News
]


def test_filter_non_base_models():
Base = declarative_base()

class Page(Base, Searchable):
id = Column(Integer, primary_key=True)
__tablename__ = 'pages'

class Topic(Page):
pass

class News(Page):
pass

assert utils.filter_non_base_models({Page, Topic, News}) == {Topic, News}

class A(Base, Searchable):
id = Column(Integer, primary_key=True)
__tablename__ = 'a'

class AA(A):
pass

class B(Base, Searchable):
id = Column(Integer, primary_key=True)
__tablename__ = 'b'

class C(Base, Searchable):
id = Column(Integer, primary_key=True)
__tablename__ = 'c'

class CC(C):
id_2 = Column(Integer, primary_key=True)
c_id = Column(Integer, ForeignKey('c.id'))
__tablename__ = 'cc'

assert utils.filter_non_base_models({A, AA, B, C, CC}) == {AA, B, C, CC}


def test_related_types():

Base = declarative_base()
Expand Down
Loading