Search updates

[nsheff]

Brainstorming some ideas for bringing the search to the next level?

• Number of GEO entries: do we have enough?
• Re-indexing, and automated re-indexing
• add metadata description to index
• Only 11-14 results are showing. Enable paging? Adjust cutoff?
• Show similarity score on results page

[nleroy917]

Re: pagination, I am a bit stumped since the qdrant.search() API doesn't return a count. A workaround is to just set the limit to an incredibly high value and count what get's returned

count = len(qdrant.search(
    collection_name=(
         query.collection_name or DEFAULT_QDRANT_COLLECTION_NAME
    ),
    query_vector=query_vec,
     limit=1e99, # get everything above the threshold
     offset=offset,
     score_threshold=score_threshold,
))

This would be really non-performant, however. I guess we could still paginate but adopt an "infinite scroll" type scenario. Just let them keep paging, incrementing the offset by the previous value + limit, until results are of length 0. With a low enough score_threshold you could in theory just page forever.

[nleroy917]

New UI:

Settings modal

[nsheff]

Re: pagination, I am a bit stumped since the qdrant.search() API doesn't return a count. A workaround is to just set the limit to an incredibly high value and count what get's returned

No -- use the limit and offset of the search. So, by default you search with limit = 100 and offset = 0. You display these results with a "page 2" button. If the user clicks "page 2", then you repeat the search with limit = 100 and offset = 100.

Offset = (page-1) * limit
limit is always 100.

There is no "number of results", in this case.

[nleroy917]

I feel pretty good about the updates here. Maybe some UI/UX/styling things could be done, but the functionality is basically there.

[nleroy917]

Two things should be completed:

1. A GitHub action should be created to run once a day (or week) to index the PEPs
2. We should augment how we are mining descriptions of all PEPs (@khoroshevskyi mentioned he might be able to do this)

[nsheff]

Some search results that come up for me, when I click on them, it says "not found".

I think this may just reflect that the re-indexing is still needed, is that right?

[nleroy917]

This is related to pepkit/pepembed#2

@khoroshevskyi it would be nice to get this stuff fleshed out prior to the latest release of PEPhub

[nleroy917]

Seems like this still needs to occur

[nleroy917]

Still open because of the indexing strategy we are still using.

[nleroy917]

I'll close this since really the re-indexing is a pepembed issue and being worked on over here: pepkit/pepembed#2