DEVELOPMENT.md

Installation

Before PR merge, you should use this installation step.

# Install from source
 npm i 'hans00/transformers.js#merge'

# or prerelease maintained by BRICKS
npm i @fugood/transformers

Basic Polyfills & Fixes

react-native-transformers-example/polyfills.js

Lines 2 to 16 in a0060d1

	import 'text-encoding-polyfill'; // support TextEncoder
	import { Buffer } from 'buffer';
	import XRegExp from 'xregexp';
	global.Buffer = Buffer;
	// replace default RegExp to support unicode
	const nativeRegExp = global.RegExp;
	const newRegExp = (...args) => {
	global.RegExp = nativeRegExp;
	const result = XRegExp(...args);
	global.RegExp = newRegExp;
	return result;
	};
	global.RegExp = newRegExp;

react-native-transformers-example/babel.config.js

Lines 4 to 5 in 74fd925

	'@babel/plugin-proposal-export-namespace-from',
	'babel-plugin-transform-import-meta',

Alternative for babel-plugin-transform-import-meta you could do patch-package

diff --git a/node_modules/@fugood/transformers/src/env.js b/node_modules/@fugood/transformers/src/env.js
index d2699da..b9cd563 100644
--- a/node_modules/@xenova/transformers/src/env.js
+++ b/node_modules/@xenova/transformers/src/env.js
@@ -24,7 +24,6 @@
 
 import fs from 'fs';
 import path from 'path';
-import url from 'url';
 import { Buffer } from 'buffer';
 
 import { ONNX } from './backends/onnx.js';
@@ -44,7 +43,7 @@ let localPath = './';
 if (IS_REACT_NATIVE) {
     localPath = fs.DocumentDirectoryPath;
 } else if (RUNNING_LOCALLY) {
-    localPath = path.dirname(path.dirname(url.fileURLToPath(import.meta.url)));
+    localPath = path.dirname(path.dirname(__filename));
 }
 
 // Only used for environments with access to file system

Android

• Add largeHeap to android/app/src/main/AndroidManifest.xml

<application
  android:name=".MainApplication"
  android:label="@string/app_name"
  android:icon="@mipmap/ic_launcher"
  android:roundIcon="@mipmap/ic_launcher_round"
  android:allowBackup="false"
  android:theme="@style/AppTheme"
+  android:largeHeap="true">

Onnxruntime

Version

Should ensure onnxruntime-common and onnxruntime-react-native version are same. You could using resolution (yarn) or overrides (npm) to force them.

For example.

react-native-transformers-example/package.json

Lines 71 to 77 in 16550ba

	"resolutions": {
	"jpeg-js": "^0.4.4",
	"node-fetch": "^2.6.7",
	"xml2js": "^0.5.0",
	"onnxruntime-common": "1.17.0",
	"onnxruntime-react-native": "1.17.0"
	}

ONNX Runtime patch work for LM models

If you use onnxruntime-react-native<1.17.0, you should patch the library.

• Allow the creation of boolean tensors from ByteBuffer (It included in v1.16.1, Required for language model)
• Allows the creation and extraction of zero length tensors (It included in v1.15, Required for LLM KV cache)
• [js/rn] Support create boolean tensor (It included in v1.17.0, Required for language model)
• [js/rn] Support load external data (It included in v1.18.0, Required for large model)

Performance Improvement

Image Process

Way 1: GCanvas

https://github.com/hans00/react-native-transformers-example/tree/gcanvas

May make app unstable. Should more reuse canvas instance

• @flyskywhy/react-native-browser-polyfill
• @flyskywhy/react-native-gcanvas

Known Issue

• The offscreen canvas too small will not get full decoded image data.
• The canvas too large will crash.
• Create too many canvas may cause crash (include refresh app).

Way 2: Skia

Stable, but slightly slower than gcanvas

react-native-transformers-example/polyfills.js

Line 1 in a0060d1

import 'react-native-skia-offscreencanvas/polyfill';