Error in content provider

[derrgoo]

Using aura mod skin 2.1

This will show up randomly.

• Add-on Version:7.1.0
• Kodi Version:nexus
• Kodi GUI Language:English
• Operating System:shield pro
• Operating System Language:English

Steps to Reproduce it does it randomly.

[inb4after]

For more context, see SerpentDrago/skin.auramod#293 (comment).

[MoojMidge]

@derrgoo - follow the issue template and post a log if you think this a problem with this plugin.

All your screenshot shows is that the search has failed. The most likely reason for this happening randomly is because you are either getting blocked after some time and/or you are using up all your API access quota.

@inb4after - what is your skin trying to do? Is it some kind of video info or trailer search? How is it initiated? Automatically initiated searches are probably not a great idea.

If someone has their own API keys setup then by default 100 searches will fully exhaust their daily API access quota.

If they don't have their own API keys, then performing automatically initiated searches can effect everyone who uses the plugin without API keys. Best case everyone gets temporarily blocked for the day, worst case access gets permanently revoked and everyone is forced to use their own API keys again.

If you want to do this automatically it is probably better for you to provide your own API keys for your users to use. This plugin allows for this but I have no experience using it in this manner. Let me know if you want to pursue this further.

Also these plugin urls don't have valid search parameter values:
https://github.com/SerpentDrago/skin.auramod/blob/bd30a1c72b878e90deaa51e64a68326401d968bb/1080i/Includes_Defs.xml#L379

You are probably looking for &search_type=video&hide_folders=true rather than &search_type=list

Edit:

This is due to how the Youtube addon handles pagination on the Video Info screen.

It doesn't seem to be able to handle searching with dynamic paths as well as TMDBHelper does, as the page number changes when you load the screen and hit next/previous page using a window property.

This is not correct and I'm reasonably certain there is nothing to fix in this plugin.

The real reason for the reported problem can only be determined if a log is provided, but it is most likely a network issue or temporary block due to API access protection.

[inb4after]

@MoojMidge

Thanks for your quick response.

I don't think it's related to API keys or using up too many searches, but rather the part you mentioned about using a dynamic URI as container content.

In the URI we use a window property that is set to 1 (the widget page number) when Home.xml loads:

https://github.com/SerpentDrago/skin.auramod/blob/52213fa866224c2a3594be950b1b9a843145f5b4/1080i/Home.xml#L25

https://github.com/SerpentDrago/skin.auramod/blob/52213fa866224c2a3594be950b1b9a843145f5b4/1080i/Includes_Defs.xml#L447

The skin has next page/previous page buttons that increase/decrease the widget page number using addon functions from script.skin.helper.service:

https://github.com/SerpentDrago/skin.auramod/blob/52213fa866224c2a3594be950b1b9a843145f5b4/1080i/Includes_Object.xml#L8313

As a side-note, I tried only initializing the window property where it is needed in DialogVideoInfo.xml but that caused more errors than not pre-initializing the window property before you enter Video Info screen.

From testing, the path handler often searches the same path multiple times or searches a valid path, but returns that it is invalid as can be seen in the error above. This is evident because even if the widget page number updates, the same list of results is returned in the container.

The same system is implemented with TMDBHelper lists, and that addon does handle dynamic paths differently and works as intended.

I would recommend, if you can, to set up a quick reproduction environment using the latest Omega version of AuraMOD, as I've left the behavior in so it can easily be tested/diagnosed.

In addition to this, there could be a regex issue that's the cause of all of this (or contributing on top of another bug), because if I do not use an invalid search_type parameter, a dialog will always open asking to choose the page to jump to:

plugin.video.youtube/resources/lib/youtube_plugin/kodion/abstract_provider.py

Line 55 in aeaba81

self.register_path(r''.join((

plugin.video.youtube/resources/lib/youtube_plugin/kodion/abstract_provider.py

Line 201 in aeaba81

def on_goto_page(provider, context, re_match):

[MoojMidge]

I don't have the ability to create a test setup to reproduce something like this. If you have seen this happen then I think the best way forward is to send through a log of your existing installation that can provide details of what you have observed.

Some comments in the meantime:

In addition to this, there could be a regex issue that's the cause of all of this (or contributing on top of another bug), because if I do not use an invalid search_type parameter, a dialog will always open asking to choose the page to jump to:

The regexp used defines the path and query requirements. If the url isn't being matched, that is not an issue with the regexp, that is a problem with the url.

If you are getting a dialog opening prompting for a page number, then that means the url being used has not provided a valid page number. The requirements for the page number, from the regexp is /kodion/goto_page/[0-9]+/path

Without a log can't see what url you are trying to use or how it is malformed. Perhaps the window property that is being set by SHS is zero indexed or is otherwise empty under some conditions? This plugin expects page numbers to start from 1.

Aside from changing the search_type parameter, would also suggest including incognito=true as otherwise you will flood the plugin search history with these requests i.e. &search_type=video&hide_folders=true&incognito=true.

From what I can see in the skin includes you are also limiting the number of items being displayed. This plugin does not support that being dynamically changed from a plugin url, but I can add this in if you like, using &items_per_page=10 as an example to limit items to 10.

The reason why this is important is because the page number that is used needs to be converted to a (next) page token for use in the YouTube API request, that is based on the total results in a request, how many items per page are shown, and the current number of results that have been retrieved.

As a side-note, I tried only initializing the window property where it is needed in DialogVideoInfo.xml but that caused more errors than not pre-initializing the window property before you enter Video Info screen.

What errors?

searches a valid path, but returns that it is invalid as can be seen in the error above.

If the url being used is valid, then the only reason why the error shown above can occur is if the YouTube API request failed. Basically only four possibilities for this for a search:

• Network issues
• Temporarily blocked because personal or developer supplied API key is not being used
• Temporarily blocked because quota allowance for shared API key has been exhausted
• Temporarily blocked because quota allowance for personal or developer supplied API key has been exhausted

Every search takes up 100 quota units. You can only do 100 per day using default quota allowance. If you are preloading a search everytime you scroll through or open an item in a widget you can burn through this in no time at all.

From testing, the path handler often searches the same path multiple times
...
This is evident because even if the widget page number updates, the same list of results is returned in the container.

The path handler for pagination wasn't really designed for what you are doing. It was primarily intended for direct loading from within this plugin or bookmarks/favourites.

The use of the same path multiple times, that you may be seeing, could be because the plugin, when loading a specific page of results, will request the data, cache it, open a new window, then load the cached data into the plugin listing of the new window.

From what you are describing, you are pre-loading the listing based on parent window properties but not displaying it until later? This can cause problems, and why TMDbHelper as an example uses a service monitor for things like this.

More specifically for this plugin you are essentially throwing away the previously cached data, then reloading it when the listing is actually displayed. I think I can detect when this is happening and disable the caching that occurs so the data only gets loaded once.

However if you are seeing the content in the container not updating then there is a more fundamental problem with the way you are updating the plugin urls. Again a log would identify what is happening, but if the content does not update, then that is because the plugin url has not been updated or the request has otherwise failed.

[inb4after]

I think I was incorrect regarding the regex and the URI issues. When I first tested the feature without initializing the widget page property, it would search for invalid paths such as /kodion/goto_page//search (as you correctly stated), because the search was done as soon as the Video Info window opened, but before the window property was actually set:

https://github.com/SerpentDrago/skin.auramod/blob/52213fa866224c2a3594be950b1b9a843145f5b4/1080i/DialogVideoInfo.xml#L11

You could tell this was occurring because the addon would fallback to opening a dialog to select the page manually (with the number 1 preloaded in that dialog). Therefore I had to use the Home.xml workaround to pre-initialize the Youtube widget page property.

This does not actually run the search, it just makes sure we have a page number to use in the URI, instead of it sometimes being empty. I might do this for all of the Video Info lists, because sometimes even TMDBHelper fails to load the container because of an empty widget page property.

Maybe the above issue Error in content provider is related to API quota, as I've not seen it since updating my testing environment to Omega, but I'll try to reproduce it by visiting many items and will post a log here.

Thank you for adding the new parameters and I have now switched to testing with your fork.

If it is that the API quota is being reached too quickly, I think it is because when I change the widget page property from 1 to 2 using the next page button, the addon will run a search for the current page again and then a second search for the next page.

The result of this is that the contents of any page above page 1 are the exact same as page 1 (the search that ran first is the container contents) and so your comment above does appear to be relevant in this case:

However if you are seeing the content in the container not updating then there is a more fundamental problem with the way you are updating the plugin urls. Again a log would identify what is happening, but if the content does not update, then that is because the plugin url has not been updated or the request has otherwise failed.

Here is the log extract showing this happening:

https://paste.kodi.tv/ebugahefuf.kodi

The first entry is when I first open the Video Info page. Then I hit next page button to go to page 2, then page 3. Then I hit the previous page button to go back to page 2 and then page 1 in sequence. At no point does the content of the container change, it is always page 1 content that shows as the container reloads.

[MoojMidge]

If you are using the test version from my repository then I think things will have now been narrowed down to 2 different issues:

1. Requests failing for whatever reason
2. Container not updating which is shown in your log

Because searches have a big API quota impact, the search result are cached for 10 minutes. They are cached based on the parameters used for the search which includes the page token parameter I referred to previously, but not the page parameter because that isn't actually used for the request itself.

The problem occurs because the pagination can apply generally to various plugin requests, some of which use the page token parameter, while others use the page parameter.

Because the pagination used for jumping within the plugin occurs within an existing plugin listing, the plugin already knows what type of parameter to use, and updates them accordingly.

When the pagination is done externally the initial parameters provided are whatever is in the plugin url, and because there is no page token parameter, it is not being updated to create a new page of search results and instead the old cached results are used.

The easiest way to workaround this is to provide a dummy page_token query parameter e.g. any of the following should work:

plugin://plugin.video.youtube/kodion/goto_page/1/search?q=deadpool&search_type=video&hide_folders=true&incognito=true&items_per_page=10&page_token=dummy
plugin://plugin.video.youtube/kodion/goto_page/2/search?q=deadpool&search_type=video&hide_folders=true&incognito=true&items_per_page=10&page_token=2
plugin://plugin.video.youtube/kodion/goto_page/3/search?q=deadpool&search_type=video&hide_folders=true&incognito=true&items_per_page=10&page_token=new

[MoojMidge]

Therefore I had to use the Home.xml workaround to pre-initialize the Youtube widget page property.

This does not actually run the search, it just makes sure we have a page number to use in the URI, instead of it sometimes being empty. I might do this for all of the Video Info lists, because sometimes even TMDBHelper fails to load the container because of an empty widget page property.

My skinning knowledge is not great, but I think you will find that that the way you have done things will in fact run a search as soon as the info dialog is opened with the updated windows properties, regardless of whether the widget container is shown.

Same will happen with the TMDbHelper widgets. I am not sure if this is intentional to avoid delays in loading the widgets when the containers are actually visible, but for the YouTube plugin it can result in many search requests where the results are never even shown.

[inb4after]

@MoojMidge

Thanks for the explanation, the dummy page_token has fixed the issue.

I think the skin has always used a hidden loading mechanism for this page, in order to reduce the delay as you said.

I will try to change the containers so that they only load when the user opens the Cast & More page and see what the performance actually looks like.

[MoojMidge]

All good. I'll create a new official beta release shortly that includes the other referenced fixes.

[inb4after]

Just testing some features of the addon and found a pagination issue on your test version.

If I hide shorts using the addon setting, my first page only has 3 items, while I've kept the items_per_page at 10.

It seems the filtering of items is done after the page limit, instead of before?

[MoojMidge]

It seems the filtering of items is done after the page limit, instead of before?

Yes that's right. YouTube does not provide the ability to distinguish a video from a short when using their V3 Data API.

For some types of listings additional requests are made to try and ensure that listings are filled up to the page limit, but it is not practical for all types of listing due to quota usage.

Would you want the ability to override the plugin setting from the plugin url?

[inb4after]

Yes, that would be great.

[MoojMidge]

item_filter query parameter added

Usage details are in commit message MoojMidge@3da06e1

[MoojMidge]

Closing this. If the origial issue re-occurs try and get a log and I can have a look again

[inb4after]

@MoojMidge

Is there any way to hide the plugin's built-in next page item using a query parameter?

[MoojMidge]

Added ?hide_next_page=true

MoojMidge@7ce4f89