Migrate to WordPress.com corrupts emojis

[mgozdis]

Context and steps to reproduce

When migrating to WordPress.com from various hosts, we often experience corrupt emojis. We are not certain on a root cause, however, typically a self-hosted site is defaulting to utf8mb4/utf8mb4_unicode_520_ci since that is what core has used since 2016.

WordPress.com UTF-8 sites use utf8mb4/utf8mb4_general_ci. While those values are being forced for the DB_CHARSET and DB_COLLATE constants in the site's meta, we've also noticed that various collation and charset variables are still set to latin1/latin1_swedish_ci as shown here:

When a self-hosted site with emojis in post titles, comments, etc. is migrated with Migrate to WordPress.com, they all become corrupt and stored as replacement characters/mojibake. Here is a recent example of a wp_comments table with this issue where all of the question marks should be emojis:

Steps to reproduce:

1. Create a self-hosted site that uses utf8mb4/utf8mb4_unicode_520_ci by default like WordPress core (Noting that sites on JN seem to migrate fine as it uses utf8mb4/utf8mb4_general_ci same as Dotcom)
2. Create a post with emojis in the title
3. Migrate the self-hosted site to WordPress.com with Migrate to WordPress.com
4. See data corruption with emojis

Site owner impact

Fewer than 20% of the total website/platform users

Severity

Moderate

What other impact(s) does this issue have?

Platform revenue

If a workaround is available, please outline it here.

Workarounds are to use an alternative migration plugin such as AIOWPM or manually import the database. When this happens, HEs/users essentially need to migrate the site twice.

Platform

Atomic