No Json for BlizzardNews and AnimeUltime

[Bockiii]

Describe the bug
Selecting the json output of the BlizzardNews or the AnimeUltime bridge, retrieves no output. All other feeds of those bridges work.

To Reproduce
Steps to reproduce the behavior:
Go to the html view of both bridges, see that content is retrieved. Then look at the mrss feed (content is retrieved), then look at the json feed (no content)

Expected behavior
Json works like with every other bridge

Additional context
Happens on my own instance and a few public instances i've tested.

I dont think the bridge maintainers can do something, this is probably a bug with the json feed producer. So @em92 maybe you.

[em92]

Hey, @Niehztog !
In BlizzardBridge tags are returned with invalid utf-8 characters. So json feed cannot be generated

array(7) {
  ["version"]=>
  string(30) "https://jsonfeed.org/version/1"
  ["title"]=>
  string(13) "Blizzard News"
  ["home_page_url"]=>
  string(31) "https://news.blizzard.com/en-us"
  ["feed_url"]=>
  string(80) "http://localhost:82/?action=display&bridge=BlizzardNews&locale=en-us&format=Json"
  ["icon"]=>
  string(177) "https://blznews.akamaized.net/images/favicon-cb34a003c6f2f637ee8f4f7b406f3b9b120b918c04cabec7f03a760e708977ea9689a1c638f4396def8dce7b202cd007eae91946cc3c4a578aa8b5694226cfc6.ico"
  ["favicon"]=>
  string(177) "https://blznews.akamaized.net/images/favicon-cb34a003c6f2f637ee8f4f7b406f3b9b120b918c04cabec7f03a760e708977ea9689a1c638f4396def8dce7b202cd007eae91946cc3c4a578aa8b5694226cfc6.ico"
  ["items"]=>
  array(1) {
    [0]=>
    array(7) {
      ["id"]=>
      string(170) "https://news.blizzard.com/en-us/blizzard/23724530/play-vanguard-s-arthur-kingsley-now-in-black-ops-cold-war-and-warzone-with-digital-pre-purchase-of-call-of-duty-vanguard"
      ["title"]=>
      string(123) "Play Vanguard’s Arthur Kingsley now in Black Ops Cold War and Warzone with digital pre-purchase of Call of Duty: Vanguard"
      ["date_modified"]=>
      string(25) "2021-09-27T17:05:00+00:00"
      ["url"]=>
      string(170) "https://news.blizzard.com/en-us/blizzard/23724530/play-vanguard-s-arthur-kingsley-now-in-black-ops-cold-war-and-warzone-with-digital-pre-purchase-of-call-of-duty-vanguard"
      ["content_text"]=>
      string(208) "Call of Duty: Vanguard pre-purchase rewards deliver the ultimate value, with Arthur Kingsley arriving today in Black Ops Cold War and Warzone, the next evolution in the “Night Raid” Mastercraft, and more."
      ["attachments"]=>
      array(1) {
        [0]=>
        array(2) {
          ["url"]=>
          string(84) "https://bnetcmsus-a.akamaihd.net/cms/blog_thumbnail/b2/B21TPXAZBEZG1632761225223.jpg"
          ["mime_type"]=>
          string(10) "image/jpeg"
        }
      }
      ["tags"]=>
      array(1) {
        [0]=>
        string(23) "Call of Duty�: Vanguard"
      }
    }
  }
}

[em92]

Pardon me. There is a feed item, that has tag with invalid utf8 characters.

[em92]

Hey, @ORelio! Same as for Niehtzog.

array(7) {
  ["version"]=>
  string(30) "https://jsonfeed.org/version/1"
  ["title"]=>
  string(39) "Latest Everything - Anime-Ultime Bridge"
  ["home_page_url"]=>
  string(28) "http://www.anime-ultime.net/"
  ["feed_url"]=>
  string(72) "http://localhost:82/?action=display&bridge=AnimeUltime&type=&format=Json"
  ["icon"]=>
  string(40) "http://www.anime-ultime.net//favicon.ico"
  ["favicon"]=>
  string(40) "http://www.anime-ultime.net//favicon.ico"
  ["items"]=>
  array(1) {
    [0]=>
    array(7) {
      ["id"]=>
      string(73) "http://www.anime-ultime.net/file-0-1/5781/Tomica-Hero--Rescue-Fire-vostfr"
      ["title"]=>
      string(41) "Tomica Hero : Rescue Fire Tokusatsu 1 - 3"
      ["author"]=>
      array(1) {
        ["name"]=>
        string(19) "Henshin no Densetsu"
      }
      ["date_modified"]=>
      string(25) "2021-09-22T00:00:00+00:00"
      ["url"]=>
      string(73) "http://www.anime-ultime.net/file-0-1/5781/Tomica-Hero--Rescue-Fire-vostfr"
      ["content_html"]=>
      string(1108) "<div class="titre">Episode (03/50) :</div><br />           <img title="Tomica-Hero--Rescue-Fire" alt="Tomica-Hero--Rescue-Fire" src="http://www.anime-ultime.net/images/img52693.png" /><br /><br /><br />           <strong><u>Note : Tomica Hero: Rescue Fire est une série de 2009 basée sur la gamme de jouets Takara Tomy’s Tomica.</u></strong><br />  <br />  <br />  <br />  <strong><u>Synopsis</u></strong> : � voir. <br />  <br />  <br />  <br />  <strong><span style="color: #FF8000">Genre : Tokusatsu<br />  Création : Takara Tomy<br />  Scénario : Takashi Yamada, Yoshimi Narita, Yasushi Hirano, Itaru Era, Mitsutaka Hirota, Mio Inoue<br />  Réalisation : Akira Iwamoto, Hideki Oka, Kenzo Maihara, Junya Morita, Daishi Matsunaga, Toshimitsu Chimura, Yuichi Abe, Yoka Kusano<br />  Diffusion : 4 Avril 2009 / 27 Mars 2010<br />  Acteurs : Sho Kubo, Yu Kawada, Yu Nakamura, Kazuki Fukuyama, Masanori Mizuno, Masataka Fujishige, Shizuka Nakamura<br />  Nombres d&#039;épisodes : 51</span></strong><br />  <br />  <br />  <br />  <strong><em><u>Sources : Wikipédia</u></em></strong><br /><br /><br />"
      ["attachments"]=>
      array(1) {
        [0]=>
        array(2) {
          ["url"]=>
          string(47) "http://www.anime-ultime.net/images/img52693.png"
          ["mime_type"]=>
          string(9) "image/png"
        }
      }
    }
  }

But in your case invalid character is in content_html, between words "Synopsis" and "voir"

[Bockiii]

Great find!

Should we add something like this to the main bridge abstract and give it as an option to bridge writers?

https://github.com/neitanod/forceutf8

It's the top answer from this stackoverflow question

[ORelio]

Indeed, that site (Anime-Ultime) is latin1, not utf-8. I'll check if I can play with utf8_encode to solve this but @Bockiii's solution is even better. Would be nice to have, I encountered in the past some sites with broken encoding (html header saying utf-8 but content in latin1) so the ability to auto-fix encoding would be awesome 💯

[Bockiii]

Flickr seems to have the same problem.

[Niehztog]

Hello @em92
I think I found the bug which led to invalid utf8 characters in feed items tags (categories). Problem was that with const SETTING_FIX_ENCODING = true; utf8_decode was invoked twice for items categories. My pull request fixes that. In my test scenarious Json outout of BlizzardNewsBridge rendered fine with this fix.

Since AnimeUltimeBridge does not use XPathAbstract but merely inherits from BridgeAbstract, my pull request does not affect the issues there (seem to be two separate issues).

[em92]

Hey, @Niehztog! GJ on your part!

[dvikan]

Fixed in #2297

[Bockiii]

The blizzard-part is actually not solved. Also, the problem is still lingering to come up again since we didnt align on the "enforceUTF8" topic yet.

I'm not savvy enough to integrate the forceutf8 as an option into all abstracts, so maybe one of you can do this @em92 @dvikan .

[dvikan]

Fixed animeultime. Unable to reproduce the blizzardnews bug.

[Bockiii]

It was because of a character in the feed at that point

string(23) "Call of Duty�: Vanguard"

So the question is, do we just ignore it or also force utf8 in the blizzard bridge. And do we force utf8 in general?

[dvikan]

If they usually return text in utf8 it's a bug on their part. The json standard requires utf8. We can just strip off non-utf8 byte sequences in JsonFormat which I think we should do. Perhaps log it when it happens.

[dvikan]

Fixed.