Fix an issue where we were not properly filtering projects #5309

ericholscher · 2019-02-19T02:01:18Z

There were some edge cases where the public queryset method on Project’s and Version’s could return an empty list. This lead to not properly filtering the queryset in the docsearch API. Properly raise an exception in this case, so that we don’t return search results that aren’t properly filtered.

agjohnson · 2019-02-19T07:27:27Z

This PR looks to break some tests around search.

humitos

Changes look good for now.

Although, we will need to find a better fix probably at self.get_all_projects to return the projects we need under different authentication circumstances or filter this list of project from corporate after checking for permissions.

humitos · 2019-02-19T10:47:05Z

readthedocs/search/api.py

@@ -65,6 +65,10 @@ def get_queryset(self):
        kwargs = {'filter_by_user': False}
        kwargs['projects_list'] = [p.slug for p in self.get_all_projects()]
        kwargs['versions_list'] = self.request.query_params.get('version')
+        if not kwargs['projects_list']:


We want two check two things here, 1) if the key exists on the dictionary and 2) if the value of the key contains something. if not kwargs.get('projects_list'): gives us these two checks

We set the key 2 lines above, I don't know how it wouldn't exist.

I got confused with the test failing, but it was a typo:

if not kwargs['projects_list']: raise ValidationError("Unable to find a project to search") > if not kwargs['version_list']: E KeyError: 'version_list'

humitos · 2019-02-19T10:47:56Z

readthedocs/search/documents.py

@@ -81,9 +81,9 @@ def faceted_search(
        }

        filters = {}
-        if projects_list:
+        if projects_list is not None:


I think that we want to check that the value here is a list (via isinstance).

I think it may also be a queryset. An empty list doesn't help us here either, so perhaps we just keep this with the existing check?

humitos · 2019-02-19T11:13:44Z

I went ahead an pushed my suggested checkings for the lists. I think that should fix the tests --feel free to revert the commit if you think that the checking is not valid.

ericholscher · 2019-02-19T11:18:04Z

readthedocs/search/documents.py

@@ -81,9 +81,9 @@ def faceted_search(
        }

        filters = {}
-        if projects_list:
+        if isinstance(projects_list) and projects_list:


Doesn't isinstance need 2 arguments? What is it checking for an instance of?

🤦‍♂️

You are right.

humitos · 2019-02-19T11:24:38Z

I QAed this and I get a validation error when using the search,

["Unable to find a project to search"]

Corporate:
- on a Private version accessed by a Shared Link
- on a Public version
Community:
- on any Public version

Summarizing, with this PR ES search does not work :/

I suppose that get_all_projects returns [] for non logged in users, and that's the problem.

ericholscher · 2019-02-19T11:29:16Z

It works fine for me locally (after fixing the variable name :D)

ericholscher · 2019-02-19T11:29:34Z

readthedocs/search/api.py

@@ -65,6 +65,10 @@ def get_queryset(self):
        kwargs = {'filter_by_user': False}
        kwargs['projects_list'] = [p.slug for p in self.get_all_projects()]
        kwargs['versions_list'] = self.request.query_params.get('version')
+        if not kwargs.get('project_list'):


Suggested change

if not kwargs.get('project_list'):

if not kwargs.get('projects_list'):

ericholscher · 2019-02-19T11:33:35Z

readthedocs/search/documents.py

@@ -81,9 +81,9 @@ def faceted_search(
        }

        filters = {}
-        if projects_list:
+        if isinstance(projects_list, list) and projects_list:


These feel like the wrong solution. I think we should just go back to the original logic. An empty list doesn't work when passed into ES either, so really we want to be checking for truthyness for it to be evaluated in a meaningful way.

This checks that projects_list is effectively a list and it's not empty (and projects_list).

humitos · 2019-02-19T11:38:36Z

readthedocs/search/documents.py

            filters['project'] = projects_list
-        if versions_list:
+        if isinstance(versions_list, (list, str)) and versions_list:


This one was tricky because of the _list suffix. versions_list can be a list or just a latest string, for example. I was expecting it to behave as projects_list that it's a list of one element in that case.

humitos · 2019-02-19T11:43:20Z

OK, I made a lot of mistakes on this PR. I will revert all my changes and just fix the typo that you had originally. I'm sorry for the noise here.

There were some edge cases where the `public` queryset method on Project’s and Version’s could return an empty list. This lead to not properly filtering the queryset in the docsearch API. Properly raise an exception in this case, so that we don’t return search results that aren’t properly filtered.

humitos · 2019-02-19T11:49:25Z

QAed this again with the latest push (your changes plus the typo fix) and it works properly in corporate (ES does not run when accessing via Shared Link) and community.

agjohnson added the Bug A bug label Feb 19, 2019

humitos reviewed Feb 19, 2019

View reviewed changes

ericholscher commented Feb 19, 2019

View reviewed changes

humitos reviewed Feb 19, 2019

View reviewed changes

humitos force-pushed the fix-empty-project-lists branch from 26fc020 to b000284 Compare February 19, 2019 11:44

ericholscher added 2 commits February 19, 2019 09:07

Fix test by making version active so it's returned

732786c

Clean up docstring

d0b5eb6

humitos approved these changes Feb 19, 2019

View reviewed changes

ericholscher merged commit 5978ec9 into master Feb 19, 2019

delete-merged-branch bot deleted the fix-empty-project-lists branch February 19, 2019 16:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix an issue where we were not properly filtering projects #5309

Fix an issue where we were not properly filtering projects #5309

ericholscher commented Feb 19, 2019

agjohnson commented Feb 19, 2019

humitos left a comment

humitos Feb 19, 2019

ericholscher Feb 19, 2019

humitos Feb 19, 2019

humitos Feb 19, 2019

ericholscher Feb 19, 2019

humitos commented Feb 19, 2019

ericholscher Feb 19, 2019

humitos Feb 19, 2019

humitos commented Feb 19, 2019

ericholscher commented Feb 19, 2019

ericholscher Feb 19, 2019

ericholscher Feb 19, 2019

humitos Feb 19, 2019

humitos Feb 19, 2019

humitos commented Feb 19, 2019

humitos commented Feb 19, 2019

	if not kwargs.get('project_list'):
	if not kwargs.get('projects_list'):

Fix an issue where we were not properly filtering projects #5309

Fix an issue where we were not properly filtering projects #5309

Conversation

ericholscher commented Feb 19, 2019

agjohnson commented Feb 19, 2019

humitos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Feb 19, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Feb 19, 2019

ericholscher commented Feb 19, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Feb 19, 2019

humitos commented Feb 19, 2019