Can't save text with emoji 😿

[connor-baer]

The Doxter field does not support text that contains emoji.

When attempting to save the entry, the following error is returned:

Internal server error
SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xF0\x9F\x8D\xBB D...' for column 'field_markdown' at row 1 The SQL being executed was: UPDATE `craft_content` SET `elementId`=311, `siteId`=1, `title`='April 12, 2017', `field_markdown`='### 🍻 Done', `dateUpdated`='2017-04-12 12:11:07' WHERE `id`='512'

Any chance you could add support for emoji in the Doxter field? Pretty please? 🙏

[selvinortiz]

@connor-baer I've been wanting to look into this for a while😉

We'll explore adding support for it👍

[DavidStrada]

That's a good Idea/Addition I think Will look into it. Thanks for letting us know 👍

[connor-baer]

Fantastic! 🙌

I did a bit more testing and research. Native Craft fields don't support emoji either. I found this StackExchange answer that's quite insightful, but inconclusive.

Since this is foremost an issue with Craft itself, I've filed an issue there as well: craftcms/cms#1627

[connor-baer]

Native rich text fields in Craft 3 now support emoji, it was fixed in craftcms/cms@3c5323e.

I guess this could be inspiration for Doxter? Changing the code at /src/fields/DoxterField.php to the following...

/**
 * @param DoxterData            $value
 * @param ElementInterface|null $element
 *
 * @return mixed
 */
public function serializeValue($value, ElementInterface $element = null) {

    /** @var DoxterData|null $value */
    if (!$value) {
        return null;
    }

    // Get the raw value
    $value = $value->getRaw();

    if (Craft::$app->getDb()->getIsMysql()) {
        // Encode any 4-byte UTF-8 characters.
        $value = StringHelper::encodeMb4($value);
    }

    return $value;
}

...allows a Doxter field containing emoji to be saved. However, the emoji is then returned as an HTML entity, e.g. 🙌 → 🙌.

That's how far my limited PHP skills got me. 😬

[selvinortiz]

@connor-baer Thanks a bunch for the investigative work and your notes, really helpful.

We'll update you when we decide on plan of attack!

[selvinortiz]

@connor-baer Thanks to your research, we were able to make progress on this front. We believe we have a possible solution for Doxter to be able to save emojis (and any other unicode chars), render properly when markdown is parsed, and render properly within the editor.

This last part is the trickiest (as you pointed out above) and the solution would be to decode those html entities into their corresponding emoji character. However, due to the nature of the Doxter editor having to allow plain text (markdown) and also arbitrary html, we have to be very selective about where to apply the decoding.

I'll work with my team and get a solution in place and will update you when we have something that works.

[connor-baer]

@selvinortiz Glad my contribution was helpful!

I'm pleasantly surprised how quickly you're working on a solution! Thank you!

[connor-baer]

Tested and it works. 🎉

Thank you for implementing this!