Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom mappings are read with the wrong encoding on Windows #100

Closed
DexLuther opened this issue Jan 19, 2022 · 10 comments
Closed

Custom mappings are read with the wrong encoding on Windows #100

DexLuther opened this issue Jan 19, 2022 · 10 comments

Comments

@DexLuther
Copy link

Tried to deal with the list of shows it spit out in the Failed list. After creating the custom mappings list, Plexanisync starts and then closes immediately. The only thing that is spit out in either logs (the normal or debug one) is this:

PlexAniSync - version: 1.3.16
[MAPPING] Custom map file found: custom_mappings.yaml

Not very helpful. Have no idea what the problem is.

Ran my custom mappings through a quick online YAML validator. Found a copy/paste error where one of the sections was missing a Seasons section. Fixed that, but still doesn't work.

The only information I can find is in the pycache and having a look at the custom_mappings.cpython-39.pyc file in notpad++. The jumbled mess seems to say that the custom mapping file isn't valid. Doesn't say why though. Passing it through the online YAML validator says it's valid, so at least I know it's structured correctly now.

This is my custom_mappings.yaml file (updated):

entries:
---
  - title: "Blue Exorcist"
    seasons:
      - season: 2
        anilist-id: 21861

  - title: "Dr. Stone"
    seasons:
      - season: 2
        anilist-id: 113936

  - title: "Full Metal Panic!"
    seasons:
      - season: 1
        anilist-id: 71
      - season: 2
        anilist-id: 72
      - season: 3
        anilist-id: 73
      - season: 4
        anilist-id: 21451

  - title: "Haganai: I Don't Have Many Friends"
    seasons:
      - season: 1
        anilist-id: 10719
      - season: 2
        anilist-id: 14967

  - title: "Heaven's Lost Property"
    seasons:
      - season: 1
        anilist-id: 5958
      - season: 2
        anilist-id: 8407

  - title: "High School DxD"
    seasons:
      - season: 1
        anilist-id: 11617
      - season: 2
        anilist-id: 15451
      - season: 3
        anilist-id: 20745
      - season: 4
        anilist-id: 97767 

  - title: "Is This A Zombie?"
    seasons:
      - season: 1
        anilist-id: 8841
      - season: 2
        anilist-id: 10790

  - title: Jormungand
    seasons:
      - season: 1
        anilist-id: 12413
      - season: 2
        anilist-id: 13331

  - title: "Love, Chunibyo & Other Delusions!"
    synonyms:
      - "Chuunibyou demo Koi ga Shitai!"
      - "Chūnibyō demo Koi ga Shitai!"
    seasons:
      - season: 1
        anilist-id: 14741
      - season: 2
        anilist-id: 18671

  - title: "Miss Kobayashi's Dragon Maid"
    synonyms:
      - "Kobayashi-san Chi no Maidragon"
    seasons:
      - season: 1
        anilist-id: 21776
      - season: 2
        anilist-id: 107717

  - title: "My Teen Romantic Comedy SNAFU"
    synonyms:
      - "Yahari Ore no Seishun Love Come wa Machigatteiru."
    seasons:
      - season: 1
        anilist-id: 14813
      - season: 2
        anilist-id: 20698
      - season: 3
        anilist-id: 108489

  - title: Noragami
    seasons:
      - season: 1
        anilist-id: 20447
      - season: 2
        anilist-id: 21128

  - title: "Queen's Blade"
    seasons:
      - season: 1
        anilist-id: 4719
      - season: 2
        anilist-id: 6633

  - title: "Rosario + Vampire"
    seasons:
      - season: 1
        anilist-id: 2993
      - season: 2
        anilist-id: 4214

  - title: "School Rumble"
    seasons:
      - season: 1
        anilist-id: 24
      - season: 2
        anilist-id: 846

  - title: "The Testament of Sister New Devil"
    seasons:
      - season: 1
        anilist-id: 20678
      - season: 2
        anilist-id: 21110

  - title: "To Love-Ru"
    seasons:
      - season: 1
        anilist-id: 3455
      - season: 2
        anilist-id: 9181
      - season: 3
        anilist-id: 13663
      - season: 4
        anilist-id: 20995

  - title: VanDread
    seasons:
      - season: 1
        anilist-id: 180
      - season: 2
        anilist-id: 181

  - title: 'Yamada-kun and the Seven Witches (2015)'
    synonyms:
      - "Yamada-kun and the Seven Witches"
      - "Yamada-kun to 7-nin no Majo"
    seasons:
      - season: 1
        anilist-id: 20966

  - title: "Saekano: How to Raise a Boring Girlfriend"
    seasons:
      - season: 1
        anilist-id: 20657
      - season: 2
        anilist-id: 21180

  - title: "Is It Wrong to Try to Pick Up Girls in a Dungeon?"
    synonyms:
      - "Dungeon ni Deai wo Motomeru no wa Machigatteiru Darou ka"
    seasons:
      - season: 1
        anilist-id: 20920
      - season: 2
        anilist-id: 101167
      - season: 3
        anilist-id: 112124
      - season: 4
        anilist-id: 129196
@kaiserbh
Copy link
Contributor

kaiserbh commented Jan 19, 2022

Hmm could be encoding, but for easy test would be download this https://github.com/RickDB/PlexAniSync-Custom-Mappings/blob/main/series-tvdb.en.yaml and use it as the custom mapping for now. If it run successfully then try editing it and put the values you have.

@DexLuther
Copy link
Author

DexLuther commented Jan 19, 2022

Hmm could be encoding, but for easy test would be download this https://github.com/RickDB/PlexAniSync-Custom-Mappings/blob/main/series-tvdb.en.yaml and use it as the custom mapping for now. If it run successfully then try editing it and put the values you have.

Tried this, but the same thing happens. Opens then closes immediately. Log file shows nothing other than the version and that the custom mapping was found/loaded.

The most information seems to come from the Pycache Custom mapping file:

`a

!tçaL�  ã                �   @   sp   d d�l Z d d�l�Z�d d�l�Z�d d�l�m�Z�� d d�l�m�Z�� d d�l�Z�d d�l�m	Z	� e� 

d�¡�Z�d�Z e�G d�d�„ d�ƒ�ƒ�Z
d d
„ Z�d�S )�é N)�Ú�List)�Ú dataclass)�Ú�YamaleErrorÚ�PlexAniSyncz;C:\Python39\Scripts\PlexAniSync-1.3.16\custom_mappings.yamlc � @ s& e Z�d Z�U e�e�d�< e�e�d�< e�e�d�< d�S )�Ú�AnilistCustomMappingÚ�seasonZ�anime_idÚ�startN)�Ú�__name__Ú
__module__Ú __qualname__Ú�intÚ�__annotations__© r� r� ú9C:\Python39\Scripts\PlexAniSync-1.3.16\custom_mappings.pyr� � s�
�����r� c � C sz� i } t j� �t�¡�s$t� �d�t�› ��¡�� ��nRt� �d�t�› ��¡�� t�j�d�d�d���}�t�j�t�d�d���}�z�t�  |�|�¡�� W n� t yÀ� }�� zHt� �d�¡�� |�j D ] }�|�j D ]�}�t� �|�› d���¡�� qŠq€t� �d�¡�� W Y d }�~�n d }�~�0 0 |�d � d � d � D ]¢}�t�|�d�� ƒ�}�|� �d g ¡�}�g } |�d � D ]T} | d�� }�| d�� } |  �d�d�¡�} t� �d�|�› d�|�› d�| › d�| › ��¡�� |  �t�|�| | ƒ�¡�� qú| | |� �¡ < |�D ]�}�| | |� �¡ < ��qqÒ| S )�Nz%[MAPPING] Custom map file not found: z![MAPPING] Custom map file found: z�./custom_mappings_schema.yamlÚ�ruamel)�Ú�parserz#Custom Mappings validation failed!
Ú�
é� r� Ú�entriesÚ�titleÚ�synonymsZ�seasonsr� z
anilist-idr� z)[MAPPING] Adding custom mapping | title: z� | season: z� | anilist id: z
| start: )�Ú�osÚ�pathÚ�isfileÚ MAPPING_FILEÚ�loggerÚ�infoÚ�yamaleZ�make_schemaZ make_dataÚ�validater� Ú�errorÚ�resultsÚ�errorsÚ�sysÚ�exitÚ�strÚ�getÚ�appendr� Ú�lower)�Ú�custom_mappingsZ�schemaZ
file_mappingsÚ�eÚ�resultr� Z
file_entryZ series_titler� Z�series_mappingsZ�file_seasonr� Z
anilist_idr� Z�synonymr� r� r� �read_custom_mappings� s< ��� ���������������


��� ��� � ��� ����� ����ÿ���� �����r+ )�r� Ú�loggingr" Ú�typingr� Z�dataclassesr� r� Z�yamale.yamale_errorr� Ú getLoggerr� r� r� r+ r� r� r� r� Ú�� s� ������ � ��� �
�������`

Seems almost corrupted, but that's no surprise. On line 20 it says "Not Validated" and shows it's being checked against the shema.

I don't know why it would be different, and when comparing it between everything, I don't see where the problem would be. I'll include it here just to be sure:

entries: list(include('entry'), min=1)
---
entry:
  title: str()
  synonyms: list(str(), required=False)
  seasons: list(include('season'), min=1)

season:
  season: int(min=1)
  anilist-id: int(min=1)
  start: int(required=False)

EDIT:

As soon as I clicked "update" to post the comment I think I may have solved it.

The Shema looking for entry, but the example and file you linked don't have those at all.

entries: list(include('entry'), min=1)
---
entry:

But even the example file starts like this:

entries:
  - title: The Rising of the Shield Hero
    seasons:

@kaiserbh
Copy link
Contributor

Hmm could be encoding, but for easy test would be download this https://github.com/RickDB/PlexAniSync-Custom-Mappings/blob/main/series-tvdb.en.yaml and use it as the custom mapping for now. If it run successfully then try editing it and put the values you have.

Tried this, but the same thing happens. Opens then closes immediately. Log file shows nothing other than the version and that the custom mapping was found/loaded.

The most information seems to come from the Pycache Custom mapping file:

`a

!tçaL�  ã                �   @   sp   d d�l Z d d�l�Z�d d�l�Z�d d�l�m�Z�� d d�l�m�Z�� d d�l�Z�d d�l�m	Z	� e� 

d�¡�Z�d�Z e�G d�d�„ d�ƒ�ƒ�Z
d d
„ Z�d�S )�é N)�Ú�List)�Ú dataclass)�Ú�YamaleErrorÚ�PlexAniSyncz;C:\Python39\Scripts\PlexAniSync-1.3.16\custom_mappings.yamlc � @ s& e Z�d Z�U e�e�d�< e�e�d�< e�e�d�< d�S )�Ú�AnilistCustomMappingÚ�seasonZ�anime_idÚ�startN)�Ú�__name__Ú
__module__Ú __qualname__Ú�intÚ�__annotations__© r� r� ú9C:\Python39\Scripts\PlexAniSync-1.3.16\custom_mappings.pyr� � s�
�����r� c � C sz� i } t j� �t�¡�s$t� �d�t�› ��¡�� ��nRt� �d�t�› ��¡�� t�j�d�d�d���}�t�j�t�d�d���}�z�t�  |�|�¡�� W n� t yÀ� }�� zHt� �d�¡�� |�j D ] }�|�j D ]�}�t� �|�› d���¡�� qŠq€t� �d�¡�� W Y d }�~�n d }�~�0 0 |�d � d � d � D ]¢}�t�|�d�� ƒ�}�|� �d g ¡�}�g } |�d � D ]T} | d�� }�| d�� } |  �d�d�¡�} t� �d�|�› d�|�› d�| › d�| › ��¡�� |  �t�|�| | ƒ�¡�� qú| | |� �¡ < |�D ]�}�| | |� �¡ < ��qqÒ| S )�Nz%[MAPPING] Custom map file not found: z![MAPPING] Custom map file found: z�./custom_mappings_schema.yamlÚ�ruamel)�Ú�parserz#Custom Mappings validation failed!
Ú�
é� r� Ú�entriesÚ�titleÚ�synonymsZ�seasonsr� z
anilist-idr� z)[MAPPING] Adding custom mapping | title: z� | season: z� | anilist id: z
| start: )�Ú�osÚ�pathÚ�isfileÚ MAPPING_FILEÚ�loggerÚ�infoÚ�yamaleZ�make_schemaZ make_dataÚ�validater� Ú�errorÚ�resultsÚ�errorsÚ�sysÚ�exitÚ�strÚ�getÚ�appendr� Ú�lower)�Ú�custom_mappingsZ�schemaZ
file_mappingsÚ�eÚ�resultr� Z
file_entryZ series_titler� Z�series_mappingsZ�file_seasonr� Z
anilist_idr� Z�synonymr� r� r� �read_custom_mappings� s< ��� ���������������


��� ��� � ��� ����� ����ÿ���� �����r+ )�r� Ú�loggingr" Ú�typingr� Z�dataclassesr� r� Z�yamale.yamale_errorr� Ú getLoggerr� r� r� r+ r� r� r� r� Ú�� s� ������ � ��� �
�������`

Seems almost corrupted, but that's no surprise. On line 20 it says "Not Validated" and shows it's being checked against the shema.

I don't know why it would be different, and when comparing it between everything, I don't see where the problem would be. I'll include it here just to be sure:

entries: list(include('entry'), min=1)
---
entry:
  title: str()
  synonyms: list(str(), required=False)
  seasons: list(include('season'), min=1)

season:
  season: int(min=1)
  anilist-id: int(min=1)
  start: int(required=False)

Hmm, that's weird 🤔 okay just to make sure this only happens when you are using the custom_mappings? Have you tried it without custom mapping and it works?

@DexLuther
Copy link
Author

DexLuther commented Jan 19, 2022

Hmm, that's weird 🤔 okay just to make sure this only happens when you are using the custom_mappings? Have you tried it without custom mapping and it works?

That would be correct. Notepad++ shows all related files being encoded in UTF-8.

If I rename custom_mappings.yaml to custom_mappings.yaml.bak it'll run fine, and then spit out a failed matches file.

@DexLuther
Copy link
Author

Update:
I tried running PlexAniSync manually from the command line, and got this output:

C:\Python39\Scripts\PlexAniSync-1.3.16>python PlexAniSync.py
2022-01-19 00:37:33 PlexAniSync - version: 1.3.16
2022-01-19 00:37:33 [MAPPING] Custom map file found: custom_mappings.yaml
Traceback (most recent call last):
  File "C:\Python39\Scripts\PlexAniSync-1.3.16\PlexAniSync.py", line 131, in <module>
    start()
  File "C:\Python39\Scripts\PlexAniSync-1.3.16\PlexAniSync.py", line 84, in start
    anilist.CUSTOM_MAPPINGS = read_custom_mappings()
  File "C:\Python39\Scripts\PlexAniSync-1.3.16\custom_mappings.py", line 29, in read_custom_mappings
    schema = yamale.make_schema('./custom_mappings_schema.yaml', parser='ruamel')
  File "C:\Python39\lib\site-packages\yamale\yamale.py", line 10, in make_schema
    raw_schemas = readers.parse_yaml(path, parser, content=content)
  File "C:\Python39\lib\site-packages\yamale\readers\yaml_reader.py", line 35, in parse_yaml
    return parse(f)
  File "C:\Python39\lib\site-packages\yamale\readers\yaml_reader.py", line 17, in _ruamel
    return list(yaml.load_all(f))
  File "C:\Python39\lib\site-packages\ruamel\yaml\main.py", line 460, in load_all
    while constructor.check_data():
  File "C:\Python39\lib\site-packages\ruamel\yaml\constructor.py", line 108, in check_data
    return self.composer.check_node()
  File "_ruamel_yaml.pyx", line 687, in _ruamel_yaml.CParser.check_node
  File "_ruamel_yaml.pyx", line 904, in _ruamel_yaml.CParser._parse_next_event
ruamel.yaml.parser.ParserError: did not find expected <document start>
  in "./custom_mappings_schema.yaml", line 7, column 1

Line 7 col 1 in custom_mappings_schema.yaml is just an empty line, so I thought the easiest solution was to just remove it.

The resulting error:

C:\Python39\Scripts\PlexAniSync-1.3.16>python PlexAniSync.py
2022-01-19 00:49:36 PlexAniSync - version: 1.3.16
2022-01-19 00:49:36 [MAPPING] Custom map file found: custom_mappings.yaml
Traceback (most recent call last):
  File "C:\Python39\Scripts\PlexAniSync-1.3.16\PlexAniSync.py", line 131, in <module>
    start()
  File "C:\Python39\Scripts\PlexAniSync-1.3.16\PlexAniSync.py", line 84, in start
    anilist.CUSTOM_MAPPINGS = read_custom_mappings()
  File "C:\Python39\Scripts\PlexAniSync-1.3.16\custom_mappings.py", line 32, in read_custom_mappings
    file_mappings = yamale.make_data(MAPPING_FILE, parser='ruamel')
  File "C:\Python39\lib\site-packages\yamale\yamale.py", line 29, in make_data
    raw_data = readers.parse_yaml(path, parser, content=content)
  File "C:\Python39\lib\site-packages\yamale\readers\yaml_reader.py", line 35, in parse_yaml
    return parse(f)
  File "C:\Python39\lib\site-packages\yamale\readers\yaml_reader.py", line 17, in _ruamel
    return list(yaml.load_all(f))
  File "C:\Python39\lib\site-packages\ruamel\yaml\main.py", line 460, in load_all
    while constructor.check_data():
  File "C:\Python39\lib\site-packages\ruamel\yaml\constructor.py", line 108, in check_data
    return self.composer.check_node()
  File "_ruamel_yaml.pyx", line 687, in _ruamel_yaml.CParser.check_node
  File "_ruamel_yaml.pyx", line 902, in _ruamel_yaml.CParser._parse_next_event
  File "_ruamel_yaml.pyx", line 911, in _ruamel_yaml.input_handler
  File "C:\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 1273: character maps to <undefined>

I replaced the empty line, but no closer to figuring out why custom mapping isn't working.

@reconman
Copy link
Collaborator

You're looking in the wrong spot.

Default encoding seems to be Windows 1252 instead of UTF-8, that's the latest error. There's currently no way to set the encoding manually in Yamale. Have to file a feature request later.

Also, the --- in the mapping file should not be there.

@reconman
Copy link
Collaborator

reconman commented Jan 19, 2022

I think removing the line with Chūnibyō in it will help with the encoding issue. And like I said, delete the ---.

Also, please revert the schema file to how it was before.

@DexLuther
Copy link
Author

DexLuther commented Jan 19, 2022

I think removing the line with Chūnibyō in it will help with the encoding issue. And like I said, delete the ---.

Also, please revert the schema file to how it was before.

Did everything you said and ran it again:

2022-01-19 01:20:05 PlexAniSync - version: 1.3.16
2022-01-19 01:20:05 [MAPPING] Custom map file found: custom_mappings.yaml
2022-01-19 01:20:05 Custom Mappings validation failed!

2022-01-19 01:20:05 entry: Unexpected element

2022-01-19 01:20:05 entries: 'None' is not a list.

Removed entry from the custom mappings, and now everything works!

Weird because 'Chūnibyō' was taken from here: https://github.com/RickDB/PlexAniSync-Custom-Mappings/blob/7b9d7098ad0914bfaa76d93946f95dc98ab8240e/series-tvdb.en.yaml#L386

Well everything works now. Thanks for all the help!

@reconman
Copy link
Collaborator

reconman commented Feb 6, 2022

Reopening to keep track of this issue in this repository.

I've reported the bug in the Yamale repo: 23andMe/Yamale#189

@reconman reconman reopened this Feb 6, 2022
@reconman reconman changed the title Custom Mapping Not Validating. Custom mappings are read with the wrong encoding on Windows Feb 6, 2022
@reconman
Copy link
Collaborator

Fixed with v1.3.18.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants