fix: emojis sent from web not rendering

CONTEXT.md

@@ -140,6 +140,19 @@ A **Message Action** is the active mode on a Message in the Room view. The three
 | **Edit**  | A Message Action where a single Message is being edited by the current user             | Editing          |
 | **React** | A Message Action where a single Message is the target of a reaction picker              | Reacting         |
 
+## Emojis
+
+A **Reaction** and the frequently used emojis table store an emoji by _name_, never as a glyph. A name that stops resolving does not degrade to the old picture — it renders as literal `:shortname:` text — which is why names are only ever added to the resolvable set, not removed. The name travels in two forms, colon-wrapped and bare; see the ambiguity flagged below. The dataset is generated; see [emojis](docs/emojis.md) for how.
+
+| Term                 | Definition                                                                                                                                   | Aliases to avoid          |
+| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
+| **Shortname**        | The colon-wrapped `:name:` token Message text and a Reaction's `emoji` field hold; the only form `useShortnameToUnicode` resolves            | Emoji code, emoji id      |
+| **Listed Name**      | The single Shortname per listed emoji that the picker shows (`emojisByCategory`) and that search returns; unlisted emoji have none           | Canonical name, primary   |
+| **Alias**            | Any other Shortname resolving to the same emoji; searchable, but search answers with the Listed Name (`water_wave` finds `ocean`)            | Synonym, alternate name   |
+| **Legacy Shortname** | A hand-maintained Shortname the generated dataset does not carry, kept resolvable by fallback because an older client could have stored it   | Deprecated name, old name |
+| **Pinned Shortname** | A Shortname held at the glyph a previous release resolved, applied at generation time, for when upstream reassigns the name to another emoji | Override, frozen name     |
+| **Custom Emoji**     | A Workspace-uploaded image emoji, stored by name plus file extension rather than resolving to unicode                                        | Custom reaction, sticker  |
+
 ## Users & Roles
 
 | Term            | Definition                                                                         | Aliases to avoid        |
@@ -269,5 +282,8 @@ A **Message Action** is the active mode on a Message in the Room view. The three
 - **"Preview"** is overloaded. **Message Preview** (`isPreview`) is a Message rendered outside its Room (search, pinned, share, notifications). `PreviewContent` is a different concept: the compact body of a Thread Message shown in the parent Room. Disambiguate when either could be meant.
 - **"Muted"** is overloaded: a User can be muted in a Room (a moderator action that removes send permission, recorded by `user-muted`/`mute_unmute` System Messages) OR be an **Ignored User** (a per-viewer filter that hides their Messages behind an Ignored Message placeholder, stored in `room.ignored`). Muting is a room permission; ignoring is a personal filter. Different concepts — keep them apart.
 - **"Reply"** is overloaded: **Reply Broadcast** is the action available to non-authorized users in a Broadcast Room; replying in a **Thread** is navigation into the Thread view. Neither is a **Message Action** — there is no "reply" Message Action.
+- **"Shortname"** travels in two forms and only one resolves. In-app APIs pass the **bare** name — `IEmoji`, the `emojisByCategory` and `aliasesByEmojiName` keys, `DEFAULT_EMOJIS`, `searchEmojiNames`, the frequently used table's `content`, and the name `setReaction` sends. Message text, a Reaction's `emoji` field, and the `shortnameToUnicodeMap` keys are **colon-wrapped**. `formatShortnameToUnicode` matches only the colon-wrapped form, so a bare name must be wrapped before resolving and a stored one stripped before it is looked up by name.
+- **"Pinned"** is overloaded: **Pinned** is a Message Flag (a Message pinned in a Room); a **Pinned Shortname** is an emoji name held at an older glyph by `scripts/pinned-shortnames.js`. Nothing connects them — say which one you mean.
+- **"Alias"** is overloaded. Every glossary table here has an _Aliases to avoid_ column: words **not** to use. An emoji **Alias** is the opposite — a first-class Shortname that resolves and is searchable, just not the **Listed Name** search answers with. Do not read the emoji sense as a term to avoid.
 - **"Status" vs "flags"** — a Message has exactly one delivery **Status** (Sent, Temp, Error). **Pinned** and **Starred** are independent **Message Flags**, not statuses; do not group them with delivery states.
 - **"Interaction" retired** — the selection-plus-action state was once an "interaction" concept; the canonical term is now **Message Action State**. Use **Message Action**, not "interaction", for which Message is selected and how. (Selection is not separate — it lives inside the active Message Action.)

app/containers/EmojiPicker/EmojiCategory.tsx

@@ -5,7 +5,7 @@ import { type ICustomEmojis, type IEmoji } from '../../definitions/IEmoji';
 import scrollPersistTaps from '../../lib/methods/helpers/scrollPersistTaps';
 import { PressableEmoji } from './PressableEmoji';
 import { EMOJI_BUTTON_SIZE } from './styles';
-import { emojisByCategory } from '../../lib/constants/emojis';
+import { emojisByCategory } from '../../lib/constants/emojis/data';
 import { useAppSelector } from '../../lib/hooks/useAppSelector';
 import { useFrequentlyUsedEmoji } from '../../lib/hooks/useFrequentlyUsedEmoji';
 import { type IEmojiCategoryProps, type TEmojiCategory } from './interfaces';

app/containers/EmojiPicker/index.tsx

@@ -6,7 +6,7 @@ import { useSafeAreaInsets } from 'react-native-safe-area-context';
 import EmojiCategory from './EmojiCategory';
 import Footer from './Footer';
 import styles from './styles';
-import { categories } from '../../lib/constants/emojis';
+import { categories } from '../../lib/constants/emojis/categories';
 import { type IEmoji } from '../../definitions';
 import { addFrequentlyUsed } from '../../lib/methods/emojis';
 import { type IEmojiPickerProps, EventTypes } from './interfaces';

app/containers/EmojiPicker/interfaces.ts

@@ -1,6 +1,6 @@
 import { type ImageStyle, type StyleProp, type TextInputProps } from 'react-native';
 
-import { type emojisByCategory } from '../../lib/constants/emojis';
+import { type emojisByCategory } from '../../lib/constants/emojis/data';
 import { type ICustomEmoji, type IEmoji } from '../../definitions';
 
 export enum EventTypes {

app/containers/MessageComposer/MessageComposer.tsx

@@ -23,7 +23,7 @@ import { sanitizeLikeString } from '../../lib/database/utils';
 import { generateTriggerId } from '../../lib/methods/actions';
 import { runSlashCommand } from '../../lib/services/restApi';
 import log from '../../lib/methods/helpers/log';
-import { prepareQuoteMessage, insertEmojiAtCursor } from './helpers';
+import { prepareQuoteMessage, insertEmojiAtCursor, lastGlyphLength } from './helpers';
 import useShortnameToUnicode from '../../lib/hooks/useShortnameToUnicode';
 import { useCloseKeyboardWhenOrientationChanges } from './hooks/useCloseKeyboardWhenOrientationChanges';
 import { useEmojiKeyboard } from './hooks/useEmojiKeyboard';
@@ -186,14 +186,10 @@ export const MessageComposer = ({
 
 		switch (eventType) {
 			case EventTypes.BACKSPACE_PRESSED:
-				const emojiRegex = /\u00a9|\u00ae|[\u2000-\u3300]|\ud83c[\ud000-\udfff]|\ud83d[\ud000-\udfff]|\ud83e[\ud000-\udfff]/;
-				let charsToRemove = 1;
-				const lastEmoji = text.substr(cursor > 0 ? cursor - 2 : text.length - 2, cursor > 0 ? cursor : text.length);
-				// Check if last character is an emoji
-				if (emojiRegex.test(lastEmoji)) charsToRemove = 2;
-				newText =
-					text.substr(0, (cursor > 0 ? cursor : text.length) - charsToRemove) + text.substr(cursor > 0 ? cursor : text.length);
-				newCursor = cursor - charsToRemove;
+				const deleteAt = cursor > 0 ? cursor : text.length;
+				const charsToRemove = lastGlyphLength(text, deleteAt);
+				newText = text.substr(0, deleteAt - charsToRemove) + text.substr(deleteAt);
+				newCursor = deleteAt - charsToRemove;
 				composerInputComponentRef.current.setInput(newText, { start: newCursor, end: newCursor });
 				break;
 			case EventTypes.EMOJI_PRESSED:

app/containers/MessageComposer/helpers/index.ts

@@ -3,3 +3,4 @@ export * from './forceJpgExtension';
 export * from './getMentionRegexp';
 export * from './prepareQuoteMessage';
 export * from './insertEmojiAtCursor';
+export * from './lastGlyphLength';

app/containers/MessageComposer/helpers/lastGlyphLength.test.ts

@@ -0,0 +1,50 @@
+import { lastGlyphLength } from './lastGlyphLength';
+
+const at = (text: string) => lastGlyphLength(text, text.length);
+
+describe('lastGlyphLength', () => {
+	it('counts a plain character as one code unit', () => {
+		expect(at('ab')).toBe(1);
+	});
+
+	it('counts a surrogate pair as one glyph', () => {
+		expect(at('\u{1F6B2}')).toBe(2);
+	});
+
+	it('counts a surrogate pair followed by a variation selector as one glyph', () => {
+		// What emojibase emits for :bike:, and what the old two-code-unit window missed.
+		expect(at('\u{1F6B2}️')).toBe(3);
+	});
+
+	it('counts a variation selector on a BMP base as one glyph', () => {
+		expect(at('❤️')).toBe(2);
+	});
+
+	it('counts a skin tone modifier as part of the glyph', () => {
+		expect(at('\u{1F44D}\u{1F3FD}')).toBe(4);
+	});
+
+	it('counts a ZWJ sequence as one glyph', () => {
+		expect(at('\u{1F468}‍\u{1F469}‍\u{1F467}‍\u{1F466}')).toBe(11);
+	});
+
+	it('counts a ZWJ sequence ending in a variation selector as one glyph', () => {
+		expect(at('\u{1F93C}‍♂️')).toBe(5);
+	});
+
+	it('counts a keycap as one glyph', () => {
+		expect(at('1️⃣')).toBe(3);
+	});
+
+	it('counts a flag as one glyph', () => {
+		expect(at('\u{1F1E7}\u{1F1F7}')).toBe(4);
+	});
+
+	it('leaves text before the glyph alone', () => {
+		expect(lastGlyphLength('hi \u{1F6B2}️ there', 6)).toBe(3);
+	});
+
+	it('returns zero at the start of the text', () => {
+		expect(lastGlyphLength('abc', 0)).toBe(0);
+	});
+});

app/containers/MessageComposer/helpers/lastGlyphLength.ts

@@ -0,0 +1,57 @@
+const ZWJ = 0x200d;
+const VARIATION_SELECTOR_16 = 0xfe0f;
+const VARIATION_SELECTOR_15 = 0xfe0e;
+const COMBINING_ENCLOSING_KEYCAP = 0x20e3;
+
+const isSkinTone = (codePoint: number) => codePoint >= 0x1f3fb && codePoint <= 0x1f3ff;
+const isRegionalIndicator = (codePoint: number) => codePoint >= 0x1f1e6 && codePoint <= 0x1f1ff;
+const isAttachedToWhatPrecedesIt = (codePoint: number) =>
+	codePoint === VARIATION_SELECTOR_16 ||
+	codePoint === VARIATION_SELECTOR_15 ||
+	codePoint === COMBINING_ENCLOSING_KEYCAP ||
+	isSkinTone(codePoint);
+
+const codePointBefore = (text: string, index: number) => {
+	const low = text.charCodeAt(index - 1);
+	if (low >= 0xdc00 && low <= 0xdfff && index >= 2) {
+		const high = text.charCodeAt(index - 2);
+		if (high >= 0xd800 && high <= 0xdbff) {
+			return { codePoint: text.codePointAt(index - 2) as number, size: 2 };
+		}
+	}
+	return { codePoint: low, size: 1 };
+};
+
+// How many UTF-16 code units the character ending at `end` occupies, counting a whole emoji
+// sequence as one. Backspace deletes that many, so a tap removes the glyph rather than an
+// invisible modifier — emojibase emits fully-qualified sequences, so most emoji end in U+FE0F.
+export const lastGlyphLength = (text: string, end: number): number => {
+	if (end <= 0) {
+		return 0;
+	}
+	let index = end;
+	let length = 0;
+	while (index > 0) {
+		const { codePoint, size } = codePointBefore(text, index);
+		index -= size;
+		length += size;
+
+		if (isAttachedToWhatPrecedesIt(codePoint)) {
+			continue;
+		}
+		if (index >= 1 && text.charCodeAt(index - 1) === ZWJ) {
+			index -= 1;
+			length += 1;
+			continue;
+		}
+		// Flags are a pair of regional indicators with nothing joining them.
+		if (isRegionalIndicator(codePoint) && index >= 2) {
+			const previous = codePointBefore(text, index);
+			if (isRegionalIndicator(previous.codePoint)) {
+				length += previous.size;
+			}
+		}
+		break;
+	}
+	return length;
+};

app/containers/MessageComposer/hooks/useAutocomplete.test.tsx

@@ -0,0 +1,33 @@
+import { renderHook, waitFor } from '@testing-library/react-native';
+
+import { useAutocomplete } from './useAutocomplete';
+import { type IAutocompleteEmoji } from '../interfaces';
+
+jest.mock('../../../lib/database', () => ({
+	__esModule: true,
+	default: { active: { get: jest.fn(() => ({ query: jest.fn(() => ({ fetch: jest.fn().mockResolvedValue([]) })) })) } }
+}));
+jest.mock('../../../lib/methods/search', () => ({ searchLocal: jest.fn(), searchRemote: jest.fn() }));
+jest.mock('../../../lib/services/restApi', () => ({ getCommandPreview: jest.fn(), getListCannedResponse: jest.fn() }));
+jest.mock('../../../lib/hooks/usePermissions', () => ({ usePermissions: () => [false, false] }));
+jest.mock('../../../lib/methods/helpers/log', () => ({ __esModule: true, default: jest.fn() }));
+
+const names = async (text: string) => {
+	const { result } = renderHook(() => useAutocomplete({ text, type: ':', rid: 'rid', accessibilityFocusOnInput: () => null }));
+	await waitFor(() => expect(result.current.some(item => item.type === ':')).toBe(true));
+	return (result.current as IAutocompleteEmoji[]).map(item => item.emoji);
+};
+
+describe('useAutocomplete emoji suggestions', () => {
+	it('suggests an emoji by its listed shortname', async () => {
+		expect(await names('ocean')).toContain('ocean');
+	});
+
+	it('suggests an emoji typed by an alias, returning the listed shortname', async () => {
+		expect(await names('water_wave')).toContain('ocean');
+	});
+
+	it('matches case insensitively, like the emoji picker search', async () => {
+		expect(await names('Ocean')).toContain('ocean');
+	});
+});

app/containers/MessageComposer/hooks/useAutocomplete.ts

@@ -8,9 +8,9 @@ import {
 	type TAutocompleteType
 } from '../interfaces';
 import { searchLocal, searchRemote, type TSearch } from '../../../lib/methods/search';
+import { searchEmojiNames } from '../../../lib/methods/emojis';
 import { sanitizeLikeString } from '../../../lib/database/utils';
 import database from '../../../lib/database';
-import { emojis } from '../../../lib/constants/emojis';
 import { type ICustomEmoji } from '../../../definitions';
 import { getCommandPreview, getListCannedResponse } from '../../../lib/services/restApi';
 import log from '../../../lib/methods/helpers/log';
@@ -144,7 +144,7 @@ export const useAutocomplete = ({
 				if (type === ':') {
 					const customEmojis = await getCustomEmojis(text);
 					if (ignore) return;
-					const filteredStandardEmojis = emojis.filter(emoji => emoji.indexOf(text) !== -1).slice(0, MENTIONS_COUNT_TO_DISPLAY);
+					const filteredStandardEmojis = searchEmojiNames(text).slice(0, MENTIONS_COUNT_TO_DISPLAY);
 					let mergedEmojis: IAutocompleteEmoji[] = customEmojis.map(emoji => ({
 						id: emoji.name,
 						emoji,

app/containers/markdown/__snapshots__/Markdown.test.tsx.snap

@@ -715,7 +715,7 @@ exports[`Story Snapshots: Emoji should match snapshot 1`] = `
           }
         >
 
-          👍
+          👍️
         </Text>
       </Text>
     </Text>
@@ -1005,7 +1005,7 @@ exports[`Story Snapshots: Emoji should match snapshot 1`] = `
           ]
         }
       >
-        👍
+        👍️
       </Text>
       <View
         style={
@@ -9315,7 +9315,7 @@ exports[`Story Snapshots: Preview should match snapshot 1`] = `
     @rocket.cat @name1 @all @here @unknown #general #unknown
   </Text>
   <Text
-    accessibilityLabel="Testing: 😃 👍 :marioparty:"
+    accessibilityLabel="Testing: 😃 👍️ :marioparty:"
     numberOfLines={1}
     style={
       [
@@ -9336,9 +9336,9 @@ exports[`Story Snapshots: Preview should match snapshot 1`] = `
         },
       ]
     }
-    testID="markdown-preview-Testing: 😃 👍 :marioparty:"
+    testID="markdown-preview-Testing: 😃 👍️ :marioparty:"
   >
-    Testing: 😃 👍 :marioparty:
+    Testing: 😃 👍️ :marioparty:
   </Text>
   <Text
     accessibilityLabel="Fallback from new md to old"

app/lib/constants/emojis/data.test.ts

@@ -0,0 +1,105 @@
+import { aliasesByEmojiName, emojisByCategory, shortnameToUnicodeMap } from './data';
+import { emojis } from './emojis';
+import { legacyShortnameToUnicodeMap } from './legacyShortnamesMap';
+import pinnedShortnames from '../../../../scripts/pinned-shortnames';
+
+const bare = (unicode: string) => unicode.replace(/\uFE0F/g, '');
+const resolve = (name: string) => shortnameToUnicodeMap[`:${name}:`] ?? legacyShortnameToUnicodeMap[`:${name}:`];
+
+// The only seven glyphs that changed against the pre-emojibase map, all repairs of a missing joiner.
+const JOINER_REPAIRS: [string, string][] = [
+	[':kiss_mm:', '👨\u200D❤\uFE0F\u200D💋\u200D👨'],
+	[':couplekiss_mm:', '👨\u200D❤\uFE0F\u200D💋\u200D👨'],
+	[':kiss_ww:', '👩\u200D❤\uFE0F\u200D💋\u200D👩'],
+	[':couplekiss_ww:', '👩\u200D❤\uFE0F\u200D💋\u200D👩'],
+	[':kiss_woman_man:', '👩\u200D❤\uFE0F\u200D💋\u200D👨'],
+	[':men_wrestling:', '🤼\u200D♂\uFE0F'],
+	[':women_wrestling:', '🤼\u200D♀\uFE0F']
+];
+
+// The pins, spelled out so editing scripts/pinned-shortnames.js fails here and not only in the picker.
+const PINNED_GLYPHS: [string, string][] = [
+	[':beetle:', '🐞'],
+	[':man_in_tuxedo:', '🤵'],
+	[':man_in_tuxedo_tone1:', '🤵🏻'],
+	[':man_in_tuxedo_tone2:', '🤵🏼'],
+	[':man_in_tuxedo_tone3:', '🤵🏽'],
+	[':man_in_tuxedo_tone4:', '🤵🏾'],
+	[':man_in_tuxedo_tone5:', '🤵🏿']
+];
+
+const UNLISTED_COMPONENTS = [
+	...Array.from({ length: 26 }, (_, i) => `regional_indicator_${String.fromCharCode(97 + i)}`),
+	'digit_zero',
+	'digit_one',
+	'digit_two',
+	'digit_three',
+	'digit_four',
+	'digit_five',
+	'digit_six',
+	'digit_seven',
+	'digit_eight',
+	'digit_nine',
+	'asterisk_symbol',
+	'pound_symbol'
+];
+
+describe('emoji data', () => {
+	it('resolves every listed emoji to a unicode character', () => {
+		const unresolved = emojis.filter(name => !shortnameToUnicodeMap[`:${name}:`]);
+		expect(unresolved).toEqual([]);
+	});
+
+	it('lists every emoji exactly once', () => {
+		expect(emojis.length).toBe(new Set(emojis).size);
+	});
+
+	it('keys aliases by a listed emoji name', () => {
+		const listed = new Set(emojis);
+		expect(Object.keys(aliasesByEmojiName).filter(name => !listed.has(name))).toEqual([]);
+	});
+
+	it('resolves every alias to the same emoji as the name it belongs to', () => {
+		const mismatched = Object.keys(aliasesByEmojiName).filter(name => {
+			const expected = bare(shortnameToUnicodeMap[`:${name}:`]);
+			return aliasesByEmojiName[name].some(alias => bare(resolve(alias) ?? '') !== expected);
+		});
+		expect(mismatched).toEqual([]);
+	});
+
+	it('has no category with an empty emoji list', () => {
+		expect(
+			Object.keys(emojisByCategory).filter(key => emojisByCategory[key as keyof typeof emojisByCategory].length === 0)
+		).toEqual([]);
+	});
+
+	it('holds every pinned shortname at its pinned glyph', () => {
+		expect(Object.entries(pinnedShortnames)).toEqual(PINNED_GLYPHS);
+		const drifted = Object.keys(pinnedShortnames).filter(
+			shortname => shortnameToUnicodeMap[shortname] !== pinnedShortnames[shortname as keyof typeof pinnedShortnames]
+		);
+		expect(drifted).toEqual([]);
+	});
+
+	it('holds every repaired joiner glyph at the value this branch decided on', () => {
+		const drifted = JOINER_REPAIRS.filter(([shortname, unicode]) => shortnameToUnicodeMap[shortname] !== unicode);
+		expect(drifted).toEqual([]);
+	});
+
+	it('never puts a variation selector before a skin tone modifier', () => {
+		const illFormed = [shortnameToUnicodeMap, legacyShortnameToUnicodeMap].flatMap(map =>
+			Object.keys(map).filter(key => /\uFE0F[\u{1F3FB}-\u{1F3FF}]/u.test(map[key]))
+		);
+		expect(illFormed).toEqual([]);
+	});
+
+	it('resolves every unlisted component without listing it', () => {
+		expect(UNLISTED_COMPONENTS.filter(name => !resolve(name))).toEqual([]);
+		const listed = new Set(emojis);
+		expect(UNLISTED_COMPONENTS.filter(name => listed.has(name))).toEqual([]);
+	});
+
+	it('keeps legacy shortnames out of the current map', () => {
+		expect(Object.keys(legacyShortnameToUnicodeMap).filter(key => key in shortnameToUnicodeMap)).toEqual([]);
+	});
+});