Implement Tokens in Swift and Kotlin

[w11wo]

Implemented tokens in Swift and Kotlin API. Tested on iOS and Android.

[csukuangfj]

I suggest that we define

const char *_tokens;
const char *const*tokens;

where _tokens is your current tokens

And the new char **tokens is a pointer array.
tokens[0] contains the address of the first token in
_tokens. tokens[1] contains the address of the second
token in _tokens.

In this way, it simplifies users' life as they only need to iterate
char **tokens.

for (int32 i = 0; i != count; ++i) {
  const char*t = tokens[i];
  // process this token
}

[ductranminh]

@csukuangfj I added const char *const *tokensArr; for easy to implement, and I also tested

[csukuangfj]

Please also add code to free it to avoid memory leak.

[ductranminh]

@csukuangfj sorry, what do you want to free? This line is just to initialize null for pointer incase there is no token from SherpaOnnxOnlineRecognizerResult

[csukuangfj]

Please change

sherpa-onnx/sherpa-onnx/c-api/c-api.cc

Lines 179 to 182 in 48038a7

	void DestroyOnlineRecognizerResult(const SherpaOnnxOnlineRecognizerResult *r) {
	delete[] r->text;
	delete r;
	}

to

void DestroyOnlineRecognizerResult(const SherpaOnnxOnlineRecognizerResult *r) {
  delete[] r->text;
  delete[] r->json;
  delete[] r->tokensArr;
  delete r;
}

---

By the way, please change the variable names:

• totalLength -> total_length
• tokensTemp -> tokens_temp
• tokensArr -> tokens_arr
  to follow the code convention in sherpa-onnx.

[csukuangfj]

Suggested change

	r->tokensArr = const_cast<const char *const *>(tokensTemp);
	r->tokensArr = tokensTemp;

[ductranminh]

I'm done

[csukuangfj]

Please also delete

delete[] r->timestamps;

[csukuangfj]

Suggested change

	tokens_temp[i] = const_cast<char*>(r->tokens) + pos;
	tokens_temp[i] = r->tokens + pos;

[ductranminh]

The reason for this casting const_cast<char*> is build error:

error: assigning to 'char *' from 'const char *' discards qualifiers
tokens_temp[i] = r->tokens + pos;

[csukuangfj]

Suggested change

	int pos = 0;
	int32_t pos = 0;

[csukuangfj]

Please update the comment. It is not separated by \0 for timestamps.

[ductranminh]

fixed

[csukuangfj]

Thanks for your contribution!