!shaped-import-statblock gives bad value for field: Challenge

[Jimwaters]

Using !shaped-import-statblock I get bad value for field: Challenge for a monster that previously worked.

Even removing the space in front of XP [which fixed my new Monster import issue] this statlock still gives the error emssage

Omou
Medium humanoid (mongrelfolk), neutral evil
Armor Class 17
Hit Points 136 (21d8 + 42)
Speed 40 ft.
STR 13 (+1)
DEX 18 (+4)
CON 15 (+2)
INT 16 (+3)
WIS 17 (+3)
CHA 12 (+1)
Saving Throws Str +4, Dex +7, Int +6, Wis +6 Skills Arcana +6, Insight +6, Perception +6, Stealth +7
Damage Resistance psychic Condition Immunities charmed, frightened Senses passive Perception 16
Languages Common
Challenge 10 (5,900XP)
Darkness Breeds Darkness. Omou has advantage on ability checks and attack rolls against characters with the Touched by the Mists story award. Similarly, such characters have disadvantage on any saving throw made against Omou.
Innate Spellcasting (Psionics). Omou's spellcasting ability is Wisdom (spell save DC 15, +7 to hit with spell attacks). He can innately cast the following spells, requiring no components:
At will: mage hand (the hand is invisible)
3/day each: featherfall, jump, see invisibility, shield
1/day each: phantasmal killer, plane shift
Legendary Resistance (3/day). If Omou fails a saving throw, he succeeds on it instead. Mimicry. The mongrelfolk can mimic any sounds it has heard, including voices. A creature that hears the sounds can tell they are imitations with a successful DC 12 Wisdom (Insight) check.
Obsessive Punishment. If Omou deals damage to a target with the Touched by the Mists story award, Omou deals an additional 1d6 damage for each dark gift the character possesses, to a maximum of 4d6 additional damage.
Probing Telepathy. If a creature communicates telepathically with Omou, he learns the creature's greatest desires if he can see the creature. If the target has the Touched by the Mists story award, Omou learns them without the need to communicate telepathically.
Psychic Defense. While Omou is wearing no armor and wielding no shield, its AC includes its Wisdom modifier.
Standing Leap. The mongrelfolk's long jump is up to 20 feet and its high jump is up to 10 feet, with or without a running start.
Actions
Multiattack. Omou makes three melee attacks.
Unarmed Strike. Melee Weapon Attack: +8 to hit, reach 5 ft., one target. Hit: 11 (2d6 + 4) bludgeoning, piercing, or slashing damage (fists, beak, or claws) plus 13 (3d8) psychic damage. This is a magic weapon attack.
Telekinetic Ray. If the target is a creature, it must succeed on a DC 16 Strength saving throw or Omou moves it up to 30 feet in any direction. It is restrained by the ray's telekinetic grip until the start of Omou's next turn or until Omou is incapacitated. If the target is an object weighing 300 pounds or less that isn't being worn or carried, it is moved up to 30 feet in any direction. Omou can also exert fine control on objects with this ray, such as manipulating a simple tool or opening a door or a container.
Mind Blast (Recharge 5-6). Omou magically emits psychic energy in a 60-foot cone. Each creature in that area must succeed on a DC 15 Intelligence saving throw or take 22 (4d8 + 4) psychic damage and be stunned for 1 minute. A creature can repeat the saving throw at the end of each of its turns, ending the effect on itself on a success.
Legendary Actions
Omou can take 3 Legendary Actions, choosing from the options below. Only one legendary action option can be used at a time, and only at the end of another creature's turn. Omou regains spent Legendary Actions at the start of her turn. Omou can't use the same legendary action twice in consecutive rounds.
*
Keen Eyes. Omou makes a Wisdom (Perception) check.
Punishing Dash. Omou moves his speed and makes an unarmed strike. This movement does not provoke opportunity attacks.
Endless Battery (Costs 2 actions). Omou makes an unarmed strike against every creature within 5 feet of him.
Psychic Drain (Costs 3 actions). Any creature stunned by Omou's mind blast takes 10 (3d6) psychic damage. Omou regains hit points equal to the damage the creatures take.
Lair Actions
On initiative count 20 (losing initiative ties), Omou takes a lair action to cause one of the following effects; Omou can't use the same effect two rounds in a row:

• All of the paintings in the room shriek in anger. Any creature except Omou that can hear the paintings must succeed on a DC 15 Wisdom saving throw or be frightened.
• Part of the ceiling collapses above one creature that Omou can see within 120 feet of him. The creature must succeed on a DC 15 Dexterity saving throw or take 10 (3d6) bludgeoning damage and be knocked prone and buried. The buried target is restrained and unable to breathe or stand up. A creature can take an action to make a DC 10 Strength check, ending the buried state on a success.
• A strong wind emanates from the huge carving of Esmae and blows around Omou. Each creature within 60 feet of Omou must succeed on a DC 15 Strength saving throw or be pushed 15 feet away from Omou and knocked prone. Gases and vapors are dispersed by the wind, and unprotected flames are extinguished. Protected flames, such as lanterns, have a 50 percent chance of being extinguished.

[Jimwaters]

Note: This statblock worked on 28th Feb, with a space before the XP but niether works now.

So worked in, 6.4.6 or 7.0.0

[symposion]

Hi Jim,

I cannot reproduce your Challenge/XP issue at all. There is a problem with the lair actions when I try and paste the statblock from github because the bullet points come out as an HTML list - I will update the parser to be able to deal with this better but I don't think it's your original issue. Are you able to grab the original statblock text (not from the GM notes field, but from wherever you got it originally), save it as a text file, and attach it to this issue as a file? That way I can get a more accurate picture of exactly what you're pasting in to the GM notes field, without it getting modified by various websites (Roll20 and github) in between.

[Jimwaters]

The original for this was here: https://app.roll20.net/forum/post/4683557/5e-shaped-companion-v6-plus/?pageforid=4704915#post-4704915 with my answer a few posts further down.

The one in this issue came from the character after processing, so I assumed would be perfect!

[Jimwaters]

Oddly going back to the source works
Omou.txt

[Jimwaters]

But if I then use the output stored in the Character GM Notes I get the error.

So the ACTUAL issue is:- The character GMNotes cannot be used to import character using this command due to the failure noted. Then again would only be using this to test....

[symposion]

Oh wow, this was a subtle one. Due to the way the parser works, the change I made to allow parentheses in trait names was causing it to read the XP portion as part of the name of a subsequent trait rather than part of the Challenge value. I'll need to mess about with this for a little while. Thanks for reporting - I'd never have found this!

[symposion]

Urgh... and the reason why this hasn't showed up anywhere else... is because SOMEHOW, there was an invisible newline character after "Darkness" in the trait name "Darkness Breeds Darkness". This only happened for the version of the text pasted from the forum, the version you attached in the text file didn't display the problem. I really hate the Roll20 textarea/forum editor, it sucks harder than the largest vacuum cleaner ever invented.

[Jimwaters]

So this gets added when the api script is writing to the character? I don't think it is the Roll20 forum....

[symposion]

No, it's get added somewhere between the Roll20 forum and the GM notes field. I copy and pasted the text from the forum, and then fixed a couple of minor errors (as discussed at the time), and then ran the import. The text that comes to my script from the GM notes of the character already contains the "invisible" newline character.

The Roll20 forum and the GM notes field use the same Javascript Rich Text Editor control. It's extremely buggy and very badly designed; even Roll20 acknowledge that it's a PoS and would like to replace it. Either when you copy out of the forum, or when you paste back into the GM notes field, it's doing something stupid.

[Jimwaters]

I don't understand.

Pasting the contents of the TXT file into a TOKEN and running the command - all is ok. The CHARACTER then created has the same basic text in it. Pasting the text from the CHARACTER to a new TOKEN and running the command again causes the error message. I don't see how the forums are involved in this?

[symposion]

Sorry, there are two problems. One of which I will fix, and the other I won't.

1. The problem you've just described (which errors on the lair actions, yes?) Is due to the fact that somewhere along the line Roll20/your browser/your operating system's clipboard converts the bullets for the lair actions (and regional effects) into an HTML list using <ul> Although the formatted text looks the same, the actual text that gets sent to the parser is substantially different. I've put in a fix that I will release in a day or two that will deal with this
2. The original problem that you had with the Challenge/XP field was due to an invisible newline in one of the trait names, introduced somewhere between the forum and the GM notes editor. This is what I was talking about in my last few responses. I'm not going to do anything to fix this because it's basically impossible - only a human reading the statblock can tell that this newline is a mistake and should be ignored.

[Jimwaters]

1. No the only issue was with the Challenge Rating. This is an issue you found as a result of trying to test this - and will no doubt help when/if people email, or post, statblocks.

2. No forum was involved in this.

[symposion]

Ok I think I finally understand what's going on here. The forum was involved originally - you yourself linked to it: #486 (comment)

That's where the statblock originally came from, and pasting that text from the forum post into a GM notes field adds the spurious invisible character.

Unfortunately, because the Redact JS Rich Text Editor is also used for all the text areas in the main application, you can trigger exactly the same behaviour by copying and pasting from the GM notes field of a character. Here's what happens in that case:

1. You paste Omou.txt into the GM notes of a token
2. You run statblock import
3. During statblock import, the parser sanitises the text and then writes it back to the original token and the new character
4. The actual text as stored by Roll20 is, at this point, correct. If you copy and paste from the token's GM Notes field (which is always in "editor" mode) you will get the right result
5. When you copy text from the character GM Notes, however, you are copying from the Redact editor in "display" mode. This is the same code that Roll20 uses to display forum posts, which is why you get the same bug in both cases.
6. Copying into a clipboard on a modern OS is complicated. It stores a reference rather than a determinate piece of text; when you come to paste it, the application that is receiving the text communicates with the original provider (or a proxy for it) and asks about supported formats. As a result, you will get a different result if you paste into a window that only supports plain text vs if you paste into a window that supports HTML. When you copy from the "display" version of the Redact editor (the character GM Notes) into a target that will accept HTML formatting, you get a formatted version of the statblock; when you paste into a text window you get the original text that I set on the character GM Notes.
7. The problem is that the formatted version of the text is utterly mangled, thanks to the fact that the Redact editor Sucks Ass. It wraps the text and inserts newline characters all over the place, which breaks the statblock parser.

At some point Roll20 are going to replace the editor with something that actually works. Until then you have two options:

1. Don't paste from the Character notes - only from the token
2. Always paste via a plain text editor

I would recommend (2) as an integral part of any workflow involving statblock parsing. The parser is a sensitive beast that can only work by making a series of assumptions about the exact layout of the statblock. You need to be 100% sure that the text you are feeding it is what you think it is, without extraneous additions/formatting/etc. The only way to do that is to paste your text via a plain text editor.

Sorry it took so long to track this down for you. This is a good lesson for us both - I should have pinned you down to a specific reproduction path generating a specific error right at the beginning, but in the absence of the exact error and the exact steps I started making assumptions and then everything got confused! Thanks for sticking with me to get to the bottom it!

[Jimwaters]

The problem started when I had a statblock that did not work. This is because statblocks are invariably for 'monsters' that do not already exist, and are open to design issues in both abilities and format. So it is amazing that the API does such a good job in processing them. In this case the text XP did not exist in the statblock, and in trying to find the format issue I copied from a 'known working' source - which hit this roll20 issue. Further testing I did to identify the issue simply confused the issue.

If we had a 'schema' for the statblock I could have checked against that rather than look at the examples I 'knew worked'.

We need a FAQ that has, in huge letters, ALWAYS COPY ANY STATBLOCK VIA A SIMPLE TEXT EDITOR!

Always happy to help, within the limits of my abilities. You put in a huge amount of effort to provide a tool that saves me (us) a great amount of time so putting a little back in is the least I can do.

[symposion]

Hey Jim,
That's a good idea. I will try and put together a sample statblock and distribute it with the script for people to look at. I will also emphasise the simple text thing in the documentation as much as I can!

[symposion]

I'm going to leave this issue open to cover the list-changing behaviour until that fix goes live, btw.