Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

having trouble with searching using filter! #595

Closed
meisnate12 opened this issue Oct 31, 2020 · 22 comments · Fixed by #693
Closed

having trouble with searching using filter! #595

meisnate12 opened this issue Oct 31, 2020 · 22 comments · Fixed by #693

Comments

@meisnate12
Copy link
Contributor

I'm trying to use LibrarySection.search to search using ! in the search filter and when I add the ! after actor, country, director, genre, and writer no matter what value I give it, it just returns every movie in my library. studio! and year! seemed to work but i haven't tested any other filters other then the ones mentioned in this post. what i ran to test is belowis below

def test(filters):
    print(str(filters) + ": " + str(len(plex.Library.search(**filters))))

test({'actor': 'Will Smith'})
test({'actor!': 'Will Smith'})
test({'country': 'USA'})
test({'country!': 'USA'})
test({'director': 'Steven Spielberg'})
test({'director!': 'Steven Spielberg'})
test({'genre': 'Romance'})
test({'genre!': 'Romance'})
test({'genre': 'Comedy'})
test({'genre!': 'Comedy'})
test({'studio': 'Pixar'})
test({'studio!': 'Pixar'})
test({'year': '1990'})
test({'year!': '1990'})
test({'writer': 'Quentin Tarantino'})
test({'writer!': 'Quentin Tarantino'})

Output

{'actor': 'Will Smith'}: 21
{'actor!': 'Will Smith'}: 1780
{'country': 'USA'}: 1489
{'country!': 'USA'}: 1780
{'director': 'Steven Spielberg'}: 16
{'director!': 'Steven Spielberg'}: 1780
{'genre': 'Romance'}: 107
{'genre!': 'Romance'}: 1780
{'genre': 'Comedy'}: 603
{'genre!': 'Comedy'}: 1780
{'studio': 'Pixar'}: 9
{'studio!': 'Pixar'}: 1771
{'year': '1990'}: 15
{'year!': '1990'}: 1765
{'writer': 'Quentin Tarantino'}: 12
{'writer!': 'Quentin Tarantino'}: 1780

Am i doing something wrong or is this feature not working correctly?

@blacktwin
Copy link
Collaborator

What did you expect to see by appending ! to the search key?

@meisnate12
Copy link
Contributor Author

meisnate12 commented Oct 31, 2020

I expect to see the inverse search i.e.

{'studio': 'Pixar'}: 9          # 9 movies with Pixar
{'studio!': 'Pixar'}: 1771      # 1771/1780 movies without Pixar

this is what i expect to see whereas with most of the filters it just returns all movies instead of the inverse i.e.

{'writer': 'Quentin Tarantino'}: 12      # 12 movies with Quentin Tarantino as the writer
{'writer!': 'Quentin Tarantino'}: 1780   # 1780 movies which is my entire library including the 
                                         # 12 with Quentin Tarantino as the writer

In LibrarySection.search is says "**kwargs (dict) – Any of the available filters for the current library section. Partial string matches allowed. Multiple matches OR together. Negative filtering also possible, just add an exclamation mark to the end of filter name, e.g. resolution!=1x1."

@blacktwin
Copy link
Collaborator

print(len(plex.library.section('Movies').all()))
print(len(plex.library.section('Movies').search(**{'studio': 'Pixar'})))
print(len(plex.library.section('Movies').search(**{'studio!': 'Pixar'})))
1163
10
1153

PlexAPI version 4.1.2

@meisnate12
Copy link
Contributor Author

yes studio and year work correctly but all the other i tested do not

@meisnate12
Copy link
Contributor Author

meisnate12 commented Oct 31, 2020

try it with an actor, country, writer, director, or genre

@meisnate12
Copy link
Contributor Author

also i just tried network and network! for tv libraries and neither filter worked they both just returned my entire library no matter what i set their values too

@blacktwin
Copy link
Collaborator

network is not a valid filter. You would still use studio for show libraries. Can you please provide your plexapi version? I've confirmed your findings.

Can you edit this area, to this:

    def _cleanSearchFilter(self, category, value, libtype=None):
        # check a few things before we begin
        categories = [x.key for x in self.filterFields()]
        booleanFilters = [x.key for x in self.filterFields() if x.type == 'boolean']
        if category.endswith('!'):
            if category[:-1] not in categories:
                raise BadRequest('Unknown filter category: %s' % category[:-1])
            category = category[:-1] # <--- add this line
        elif category not in categories:
            raise BadRequest('Unknown filter category: %s' % category)
        if category in booleanFilters:
            return '1' if value else '0'
        if not isinstance(value, (list, tuple)):
            value = [value]

Then run your tests again. Let me know.

@meisnate12
Copy link
Contributor Author

when i use studio on show libraries i get

plexapi.exceptions.BadRequest: Unknown filter category: studio

but not when i use network also on the LibrarySection.search it does say that network is a valid tv filter and studio is not for tv. In fact, it says studio is for music even though I've confirmed it works for movies

ill try and update the code and get back to you shortly

@meisnate12
Copy link
Contributor Author

my api version is 4.1.2 and I'm not sure how to test your change i edited the library.py file that was in pythons library but I don't think the change was actually being used since non of my prints were working either

@blacktwin
Copy link
Collaborator

With edit:

print(len(plex.library.section('Movies').all()))
def test(filters):
    print(str(filters) + ": " + str(len(plex.library.section('Movies').search(**filters))))

test({'actor': 'Will Smith'})
test({'actor!': 'Will Smith'})
test({'country': 'USA'})
test({'country!': 'USA'})
test({'director': 'Steven Spielberg'})
test({'director!': 'Steven Spielberg'})
test({'genre': 'Romance'})
test({'genre!': 'Romance'})
test({'genre': 'Comedy'})
test({'genre!': 'Comedy'})
test({'studio': 'Pixar'})
test({'studio!': 'Pixar'})
test({'year': '1990'})
test({'year!': '1990'})
test({'writer': 'Quentin Tarantino'})
test({'writer!': 'Quentin Tarantino'})

Output

1163
{'actor': 'Will Smith'}: 7
{'actor!': 'Will Smith'}: 1156
{'country': 'USA'}: 869
{'country!': 'USA'}: 294
{'director': 'Steven Spielberg'}: 12
{'director!': 'Steven Spielberg'}: 1151
{'genre': 'Romance'}: 113
{'genre!': 'Romance'}: 1050
{'genre': 'Comedy'}: 414
{'genre!': 'Comedy'}: 749
{'studio': 'Pixar'}: 10
{'studio!': 'Pixar'}: 1153
{'year': '1990'}: 30
{'year!': '1990'}: 1133
{'writer': 'Quentin Tarantino'}: 7
{'writer!': 'Quentin Tarantino'}: 1156

Without edit:

1163
{'actor': 'Will Smith'}: 7
{'actor!': 'Will Smith'}: 1163
{'country': 'USA'}: 869
{'country!': 'USA'}: 1163
{'director': 'Steven Spielberg'}: 12
{'director!': 'Steven Spielberg'}: 1163
{'genre': 'Romance'}: 113
{'genre!': 'Romance'}: 1163
{'genre': 'Comedy'}: 414
{'genre!': 'Comedy'}: 1163
{'studio': 'Pixar'}: 10
{'studio!': 'Pixar'}: 1153
{'year': '1990'}: 30
{'year!': '1990'}: 1133
{'writer': 'Quentin Tarantino'}: 7
{'writer!': 'Quentin Tarantino'}: 1163

@meisnate12
Copy link
Contributor Author

That's awesome looks like it's working! did you get a chance to look at network/studio search for tv libraries it was broke not only inversely but the normal one as far as I can tell.
Both network and network! We're returning all shows in the library no matter what value was input

@blacktwin
Copy link
Collaborator

blacktwin commented Oct 31, 2020

Network should not be a valid filter. You can see for yourself when filtering in the webUI. Pull out your browser Inspect tool and look at the network tab for the API call.

There was a somewhat recent change to this project. The filters are no longer hard coded and instead pull from Plex. If network is not throwing an error the I'd assume you aren't using then latest updates from this project.

Also, check out #402

@meisnate12
Copy link
Contributor Author

I'm using 4.1.2 to be honest I'm not sure how to use your git version over what I installed with pip install I'm very new to python even if I'm not new to programming but it looks like you fixed the network problem before I even brought it up. I have to wait till the next release that I can pull with pip install but I'm glad these issues seemed to have been fixed. Do you have an idea of when the next release would be or is there any place i can check to see whne the next one is coming?

@blacktwin
Copy link
Collaborator

You can try one of these.
pip install --upgrade --force-reinstall plexapi reinstall all packages even if they are already up-to-date
pip install -I plexapi Ignore the installed packages (reinstalling instead)
pip install git+https://github.com/pkkid/python-plexapi.git pull from github

@meisnate12
Copy link
Contributor Author

so I finally got this all working and yes adding category = category[:-1] did fix the problem the inverse search problem I did notice another issue I tried searching using show.genre it returned nothing

print(len(plex.library.section('Movies').all()))
def test(filters):
    print(str(filters) + ": " + str(len(plex.library.section('Movies').search(**filters))))

test({'show.genre': 'Romance'})
test({'show.genre!': 'Romance'})
test({'show.genre': 'Comedy'})
test({'show.genre!': 'Comedy'})

Output

366
{'show.genre': 'Romance'}: 0
{'show.genre!': 'Romance'}: 366
{'show.genre': 'Comedy'}: 0
{'show.genre!': 'Comedy'}: 366

@blacktwin
Copy link
Collaborator

show.genre wouldn't work in a movie library, but I am seeing this with my TV Shows library. I'm looking into it. Thanks for your patience and testing. I think your test example would be a great unit test to add.

@meisnate12
Copy link
Contributor Author

my bad i ran it using my TV library when i was copy pasting into here i copied your code from earlier and forgot to change that to TV Shows lol but yea no problem im glad to help improve this where i can. If you need me to test any more just let me know

@blacktwin
Copy link
Collaborator

blacktwin commented Nov 6, 2020

Ok same area as before:

   def _cleanSearchFilter(self, category, value, libtype=None):
        # check a few things before we begin
        categories = [x.key for x in self.filterFields()]
        booleanFilters = [x.key for x in self.filterFields() if x.type == 'boolean']
        if category.endswith('!'):
            if category[:-1] not in categories:
                raise BadRequest('Unknown filter category: %s' % category[:-1])
            category = category[:-1]  # <--- add this line
        elif category not in categories:
            raise BadRequest('Unknown filter category: %s' % category)
        if category in booleanFilters:
            return '1' if value else '0'
        if not isinstance(value, (list, tuple)):
            value = [value]
        # convert list of values to list of keys or ids
        result = set()
        category = category.split('.')[1] if '.' in category else category # <--- add this line
        choices = self.listChoices(category, libtype)
        lookup = {c.title.lower(): unquote(unquote(c.key)) for c in choices}
        allowed = set(c.key for c in choices)
        for item in value: .... 
def test(filters):
    print(str(filters) + ": " + str(len(plex.library.section('TV Shows').search(**filters))))

test({'show.genre': 'Romance'})
test({'show.genre!': 'Romance'})
test({'show.genre': 'Comedy'})
test({'show.genre!': 'Comedy'})

output with edit:

{'show.genre': 'Romance'}: 5
{'show.genre!': 'Romance'}: 287
{'show.genre': 'Comedy'}: 127
{'show.genre!': 'Comedy'}: 165

I knew this seems familiar #392. Might be able to close two issues at once since the other issue was in regards to the libtype.key type of filtering.

@meisnate12
Copy link
Contributor Author

yea that change fixed it for me and i didn't realize you could could use operators while fetching Items that going to be very useful to me in the future

@blacktwin blacktwin mentioned this issue Nov 8, 2020
7 tasks
@meisnate12
Copy link
Contributor Author

I've been using the fix here for a while but I just installed the new Beta version of Plex with the new TV Agent and it seems to have completely broken filters. The filterFields method under the class LibrarySection no longer returns any filters. The PlexAPI endpoint '/library/sections/%s/filters?includeMeta=1' no longer returns the Meta Information when using the attribute includeMeta=1, which is what seems to be the root of the issue here.

If I need to make a new issue I'm happy to.

What the normal branch returns

<?xml version="1.0" encoding="UTF-8"?>
<MediaContainer size="19" allowSync="0" art="/:/resources/movie-fanart.jpg" content="secondary" identifier="com.plexapp.plugins.library" mediaTagPrefix="/system/bundle/media/flags/" mediaTagVersion="1614092584" thumb="/:/resources/movie.png" title1="Movies" viewGroup="secondary" viewMode="65592">
<Meta>
<Type key="/library/sections/18/all?type=1" type="movie" title="Movies" active="0">
<Filter filter="genre" filterType="string" key="/library/sections/18/genre" title="Genre" type="filter" />
<Filter filter="year" filterType="integer" key="/library/sections/18/year" title="Year" type="filter" />
<Filter filter="decade" filterType="integer" key="/library/sections/18/decade" title="Decade" type="filter" />
<Filter filter="contentRating" filterType="string" key="/library/sections/18/contentRating" title="Content Rating" type="filter" />
<Filter filter="collection" filterType="string" key="/library/sections/18/collection" title="Collection" type="filter" />
<Filter filter="director" filterType="string" key="/library/sections/18/director" title="Director" type="filter" />
<Filter filter="actor" filterType="string" key="/library/sections/18/actor" title="Actor" type="filter" />
<Filter filter="writer" filterType="string" key="/library/sections/18/writer" title="Writer" type="filter" />
<Filter filter="producer" filterType="string" key="/library/sections/18/producer" title="Producer" type="filter" />
<Filter filter="country" filterType="string" key="/library/sections/18/country" title="Country" type="filter" />
<Filter filter="studio" filterType="string" key="/library/sections/18/studio" title="Studio" type="filter" />
<Filter filter="resolution" filterType="string" key="/library/sections/18/resolution" title="Resolution" type="filter" />
<Filter filter="hdr" filterType="boolean" key="/library/sections/18/hdr" title="HDR" type="filter" />
<Filter filter="unwatched" filterType="boolean" key="/library/sections/18/unwatched" title="Unplayed" type="filter" />
<Filter filter="inProgress" filterType="boolean" key="/library/sections/18/inProgress" title="In Progress" type="filter" />
<Filter filter="unmatched" filterType="boolean" key="/library/sections/18/unmatched" title="Unmatched" type="filter" />
<Filter filter="audioLanguage" filterType="string" key="/library/sections/18/audioLanguage" title="Audio Language" type="filter" />
<Filter filter="subtitleLanguage" filterType="string" key="/library/sections/18/subtitleLanguage" title="Subtitle Language" type="filter" />
<Filter filter="label" filterType="string" key="/library/sections/18/label" title="Labels" type="filter" />
<Sort default="asc" defaultDirection="asc" descKey="titleSort:desc" firstCharacterKey="/library/sections/18/firstCharacter" key="titleSort" title="Title" />
<Sort defaultDirection="desc" descKey="originallyAvailableAt:desc" key="originallyAvailableAt" title="Release Date" />
<Sort defaultDirection="desc" descKey="rating:desc" key="rating" title="Critic Rating" />
<Sort defaultDirection="desc" descKey="audienceRating:desc" key="audienceRating" title="Audience Rating" />
<Sort defaultDirection="desc" descKey="duration:desc" key="duration" title="Duration" />
<Sort defaultDirection="desc" descKey="addedAt:desc" key="addedAt" title="Date Added" />
<Sort defaultDirection="desc" descKey="lastViewedAt:desc" key="lastViewedAt" title="Date Viewed" />
<Sort defaultDirection="asc" descKey="mediaHeight:desc" key="mediaHeight" title="Resolution" />
<Field key="title" title="Title" type="string" />
<Field key="studio" title="Studio" type="string" />
<Field key="userRating" subType="rating" title="Rating" type="integer" />
<Field key="contentRating" title="Content Rating" type="tag" />
<Field key="year" subType="year" title="Year" type="integer" />
<Field key="decade" subType="decade" title="Decade" type="integer" />
<Field key="originallyAvailableAt" title="Release Date" type="date" />
<Field key="duration" subType="duration" title="Duration" type="integer" />
<Field key="unmatched" title="Unmatched" type="boolean" />
<Field key="duplicate" title="Duplicate" type="boolean" />
<Field key="genre" title="Genre" type="tag" />
<Field key="collection" title="Collection" type="tag" />
<Field key="director" title="Director" type="tag" />
<Field key="writer" title="Writer" type="tag" />
<Field key="producer" title="Producer" type="tag" />
<Field key="actor" title="Actor" type="tag" />
<Field key="country" title="Country" type="tag" />
<Field key="addedAt" title="Date Added" type="date" />
<Field key="viewCount" title="Plays" type="integer" />
<Field key="lastViewedAt" title="Last Played" type="date" />
<Field key="unwatched" title="Unplayed" type="boolean" />
<Field key="resolution" title="Resolution" type="resolution" />
<Field key="hdr" subType="hdr" title="HDR" type="boolean" />
<Field key="mediaSize" subType="fileSize" title="File Size" type="integer" />
<Field key="mediaBitrate" subType="bitrate" title="Bitrate" type="integer" />
<Field key="subtitleLanguage" title="Subtitle Language" type="subtitleLanguage" />
<Field key="audioLanguage" title="Audio Language" type="audioLanguage" />
<Field key="inProgress" title="In Progress" type="boolean" />
<Field key="trash" title="Trash" type="boolean" />
<Field key="label" title="Label" type="tag" />
</Type>
<Type key="/library/sections/18/folder" type="folder" title="Folders" active="0">
</Type>
<FieldType type="tag">
<Operator key="=" title="is" />
<Operator key="!=" title="is not" />
</FieldType>
<FieldType type="integer">
<Operator key="=" title="is" />
<Operator key="!=" title="is not" />
<Operator key="&gt;&gt;=" title="is greater than" />
<Operator key="&lt;&lt;=" title="is less than" />
</FieldType>
<FieldType type="string">
<Operator key="=" title="contains" />
<Operator key="!=" title="does not contain" />
<Operator key="==" title="is" />
<Operator key="!==" title="is not" />
<Operator key="&lt;=" title="begins with" />
<Operator key="&gt;=" title="ends with" />
</FieldType>
<FieldType type="boolean">
<Operator key="=" title="is true" />
<Operator key="!=" title="is false" />
</FieldType>
<FieldType type="date">
<Operator key="&lt;&lt;=" title="is before" />
<Operator key="&gt;&gt;=" title="is after" />
</FieldType>
<FieldType type="subtitleLanguage">
<Operator key="=" title="is" />
<Operator key="!=" title="is not" />
</FieldType>
<FieldType type="audioLanguage">
<Operator key="=" title="is" />
<Operator key="!=" title="is not" />
</FieldType>
<FieldType type="resolution">
<Operator key="=" title="is" />
</FieldType>
</Meta>
<Directory filter="genre" filterType="string" key="/library/sections/18/genre" title="Genre" type="filter" />
<Directory filter="year" filterType="integer" key="/library/sections/18/year" title="Year" type="filter" />
<Directory filter="decade" filterType="integer" key="/library/sections/18/decade" title="Decade" type="filter" />
<Directory filter="contentRating" filterType="string" key="/library/sections/18/contentRating" title="Content Rating" type="filter" />
<Directory filter="collection" filterType="string" key="/library/sections/18/collection" title="Collection" type="filter" />
<Directory filter="director" filterType="string" key="/library/sections/18/director" title="Director" type="filter" />
<Directory filter="actor" filterType="string" key="/library/sections/18/actor" title="Actor" type="filter" />
<Directory filter="writer" filterType="string" key="/library/sections/18/writer" title="Writer" type="filter" />
<Directory filter="producer" filterType="string" key="/library/sections/18/producer" title="Producer" type="filter" />
<Directory filter="country" filterType="string" key="/library/sections/18/country" title="Country" type="filter" />
<Directory filter="studio" filterType="string" key="/library/sections/18/studio" title="Studio" type="filter" />
<Directory filter="resolution" filterType="string" key="/library/sections/18/resolution" title="Resolution" type="filter" />
<Directory filter="hdr" filterType="boolean" key="/library/sections/18/hdr" title="HDR" type="filter" />
<Directory filter="unwatched" filterType="boolean" key="/library/sections/18/unwatched" title="Unplayed" type="filter" />
<Directory filter="inProgress" filterType="boolean" key="/library/sections/18/inProgress" title="In Progress" type="filter" />
<Directory filter="unmatched" filterType="boolean" key="/library/sections/18/unmatched" title="Unmatched" type="filter" />
<Directory filter="audioLanguage" filterType="string" key="/library/sections/18/audioLanguage" title="Audio Language" type="filter" />
<Directory filter="subtitleLanguage" filterType="string" key="/library/sections/18/subtitleLanguage" title="Subtitle Language" type="filter" />
<Directory filter="label" filterType="string" key="/library/sections/18/label" title="Labels" type="filter" />
</MediaContainer>

what the new endpoint returns

<?xml version="1.0" encoding="UTF-8"?>
<MediaContainer size="19" allowSync="0" art="/:/resources/movie-fanart.jpg" content="secondary" identifier="com.plexapp.plugins.library" mediaTagPrefix="/system/bundle/media/flags/" mediaTagVersion="1614701937" thumb="/:/resources/movie.png" title1="Movies" viewGroup="secondary" viewMode="65592">
<Directory filter="genre" filterType="string" key="/library/sections/18/genre" title="Genre" type="filter" />
<Directory filter="year" filterType="integer" key="/library/sections/18/year" title="Year" type="filter" />
<Directory filter="decade" filterType="integer" key="/library/sections/18/decade" title="Decade" type="filter" />
<Directory filter="contentRating" filterType="string" key="/library/sections/18/contentRating" title="Content Rating" type="filter" />
<Directory filter="collection" filterType="string" key="/library/sections/18/collection" title="Collection" type="filter" />
<Directory filter="director" filterType="string" key="/library/sections/18/director" title="Director" type="filter" />
<Directory filter="actor" filterType="string" key="/library/sections/18/actor" title="Actor" type="filter" />
<Directory filter="writer" filterType="string" key="/library/sections/18/writer" title="Writer" type="filter" />
<Directory filter="producer" filterType="string" key="/library/sections/18/producer" title="Producer" type="filter" />
<Directory filter="country" filterType="string" key="/library/sections/18/country" title="Country" type="filter" />
<Directory filter="studio" filterType="string" key="/library/sections/18/studio" title="Studio" type="filter" />
<Directory filter="resolution" filterType="string" key="/library/sections/18/resolution" title="Resolution" type="filter" />
<Directory filter="hdr" filterType="boolean" key="/library/sections/18/hdr" title="HDR" type="filter" />
<Directory filter="unwatched" filterType="boolean" key="/library/sections/18/unwatched" title="Unplayed" type="filter" />
<Directory filter="inProgress" filterType="boolean" key="/library/sections/18/inProgress" title="In Progress" type="filter" />
<Directory filter="unmatched" filterType="boolean" key="/library/sections/18/unmatched" title="Unmatched" type="filter" />
<Directory filter="audioLanguage" filterType="string" key="/library/sections/18/audioLanguage" title="Audio Language" type="filter" />
<Directory filter="subtitleLanguage" filterType="string" key="/library/sections/18/subtitleLanguage" title="Subtitle Language" type="filter" />
<Directory filter="label" filterType="string" key="/library/sections/18/label" title="Labels" type="filter" />
</MediaContainer>

@blacktwin
Copy link
Collaborator

blacktwin commented Mar 3, 2021

I've confirmed this on my end as well. We'll have to see if this is only reflected in the Beta or will be present in the main release. Version 1.22.0.4136 to be specific.

I'll have to update that key in that method from /library/sections/{sectionID}/filters?includeMeta=1 to /library/sections/{sectionID}/all?includeMeta=1

Can you confirm that this new key /library/sections/{sectionID}/all?includeMeta=1 works in the previous stable release @meisnate12 if you have one up still?

@meisnate12
Copy link
Contributor Author

i do still have the current stable release up and yes switching the endpoint did still allow it to work in the current stable release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants