Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions website/src/_data/docs.js
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,17 @@ export default [
["Structured outputs", "/docs/pages/openai-compatible-api/#structured-outputs"]
]
},
{
title: "SDKs",
description: "Embed mesh clients and local serving into Rust, Node.js, JVM/Android, and Swift apps.",
links: [
["SDK overview", "/docs/pages/sdk/"],
["Rust", "/docs/pages/sdk-rust/"],
["Node.js & Electron", "/docs/pages/sdk-node/"],
["Java / Kotlin / Android", "/docs/pages/sdk-kotlin/"],
["Swift & Apple platforms", "/docs/pages/sdk-swift/"]
]
},
{
title: "Plugins",
description: "Extend mesh-llm with managed plugin processes, MCP tools, and HTTP bindings.",
Expand Down
143 changes: 143 additions & 0 deletions website/src/docs/pages/sdk-kotlin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
---
title: Java, Kotlin, and Android SDK
---

# Java, Kotlin, and Android SDK

The Android/JVM SDK is published as `ai.meshllm:meshllm-android`. The public API is authored in Kotlin and uses coroutines and `Flow`; Java applications can consume the same AAR, while the examples here use Kotlin because it matches the native API most closely.

## Install

Add the GitHub Packages Maven registry and the SDK dependency:

```kotlin
repositories {
maven {
url = uri("https://maven.pkg.github.com/Mesh-LLM/mesh-llm")
credentials {
username = providers.gradleProperty("gpr.user")
.orElse(System.getenv("GITHUB_ACTOR")).get()
password = providers.gradleProperty("gpr.key")
.orElse(System.getenv("GITHUB_TOKEN")).get()
}
}
}

dependencies {
implementation("ai.meshllm:meshllm-android:0.72.1")
}
```

For local development, build the AAR with `./gradlew assembleAar` from `sdk/kotlin`.

## Connect as a client

```kotlin
import ai.meshllm.ChatMessage
import ai.meshllm.ChatRequest
import ai.meshllm.Client
import ai.meshllm.Event
import ai.meshllm.InviteToken
import kotlinx.coroutines.flow.collect
import uniffi.mesh_ffi.generateOwnerKeypairHex

val client = Client(
InviteToken(System.getenv("MESH_INVITE_TOKEN")),
generateOwnerKeypairHex(),
)

client.start()
try {
val models = client.inference.listModels()
client.inference.chatFlow(
ChatRequest(
models.first().id,
listOf(ChatMessage("user", "Say hello from Kotlin.")),
),
).collect { event ->
if (event is Event.TokenDelta) print(event.delta)
if (event is Event.Completed) println()
}
} finally {
client.stop()
}
```

For public mesh discovery, use `Client.connectPublic(ownerKeypair, PublicMeshQuery(...))`. Keep the owner keypair in app storage so reconnects retain the same identity.

## JVM local serving

Local serving is currently supported for validated JVM targets with a matching native runtime. Resolve a bundled or downloadable runtime first:

```kotlin
import ai.meshllm.NativeRuntime
import ai.meshllm.NativeRuntimeResolveOptions
import java.io.File

val runtime = NativeRuntime.resolve(
NativeRuntimeResolveOptions(
artifactDir = System.getenv("MESHLLM_NATIVE_RUNTIME_ARTIFACT_DIR")?.let(::File),
allowDownload = System.getenv("MESH_SDK_RUNTIME_ALLOW_DOWNLOAD") == "1",
),
)
println("using ${runtime.nativeRuntimeId} from ${runtime.path}")
```

Then use `Node` for model management and serving:

```kotlin
import ai.meshllm.ChatMessage
import ai.meshllm.ChatRequest
import ai.meshllm.DevicePolicy
import ai.meshllm.InviteToken
import ai.meshllm.LoadModelOptions
import ai.meshllm.Node
import ai.meshllm.UnloadModelOptions
import uniffi.mesh_ffi.generateOwnerKeypairHex

val node = Node(
InviteToken(System.getenv("MESH_INVITE_TOKEN")),
generateOwnerKeypairHex(),
)

node.start()
try {
val modelRef = System.getenv("MESH_SDK_MODEL_REF") ?: "Qwen2.5-3B-Instruct-Q4_K_M"
node.models.download(modelRef)
val served = node.serving.load(modelRef, LoadModelOptions(DevicePolicy.Auto))
node.inference.chatFlow(
ChatRequest(
served.modelId,
listOf(ChatMessage("user", "Say hello from JVM serving.")),
),
).collect { event ->
if (event is Event.TokenDelta) print(event.delta)
if (event is Event.Completed) println()
}
node.serving.unloadModel(
served.modelId,
UnloadModelOptions(drainTimeoutMs = 1_000UL, force = false),
)
} finally {
node.stop()
}
```

Catch `MeshException.ServingUnsupported` and show an actionable message when the selected JVM/native artifact does not provide local serving.

## Android support

Android apps can use the SDK for mesh inference and model-management APIs. Local serving is not currently advertised for Android, so an Android app should use `Client` to reach a serving node or display the typed unsupported error rather than trying to load a desktop runtime into the APK.

When Android serving is validated, the packaging contract will need an ABI-specific native runtime, app-compatible storage, lifecycle handling, and a verified memory budget. Keep those concerns behind the SDK rather than copying desktop runtime paths into the app.

## Console assets

JVM/Android packages may include the console as resources. Extract packaged resources before starting the static console server:

```kotlin
val console = node.startConsole(ConsoleAssets.packagedOptions(port = 3131u.toUShort()))
println(console.url())
```

Treat the console as optional and bind it only to an interface your app can protect.
129 changes: 129 additions & 0 deletions website/src/docs/pages/sdk-node.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
---
title: Node.js and Electron SDK
---

# Node.js and Electron SDK

Use [`@meshllm/sdk`](https://www.npmjs.com/package/@meshllm/sdk) in Node.js services, desktop apps, and Electron applications. The package uses a native N-API addon and the same embedded serving path as the Swift and Kotlin SDKs.

## Install

```json
{
"dependencies": {
"@meshllm/sdk": "0.72.1"
}
}
```

When building from the repository, build the addon first:

```bash
cd sdk/node
npm run build:native
```

## Connect as a client

```js
const { Client, generateOwnerKeypairHex } = require('@meshllm/sdk')

const client = Client.create({
ownerKeypairHex: generateOwnerKeypairHex(),
inviteToken: process.env.MESH_INVITE_TOKEN
})

await client.start()
try {
const models = await client.inference.listModels()
const result = await client.inference.chat({
model: models[0].id,
messages: [{ role: 'user', content: 'Say hello from Node.js.' }]
})
console.log(result.content)
} finally {
await client.stop()
}
```

The current package expects an invite token selected by the app or service. Use the same `Client` lifecycle for public or private meshes; only the token source changes.

## Embed local serving

Serving needs a verified native runtime artifact. Bundle one with the app or explicitly allow the SDK to download a compatible release runtime:

```js
const { resolveNativeRuntime } = require('@meshllm/sdk')

const runtime = await resolveNativeRuntime({
artifactDir: process.env.MESHLLM_NATIVE_RUNTIME_ARTIFACT_DIR,
allowDownload: process.env.MESH_SDK_RUNTIME_ALLOW_DOWNLOAD === '1',
onProgress: event => console.log(event)
})
console.log(`using ${runtime.nativeRuntimeId} from ${runtime.path}`)
```

Create a `Node`, download or locate the model, load it through the serving API, and unload it during shutdown:

```js
const {
Node,
generateOwnerKeypairHex,
resolveNativeRuntime
} = require('@meshllm/sdk')

await resolveNativeRuntime({
artifactDir: process.env.MESHLLM_NATIVE_RUNTIME_ARTIFACT_DIR,
allowDownload: process.env.MESH_SDK_RUNTIME_ALLOW_DOWNLOAD === '1'
})

const modelRef = process.env.MESH_SDK_MODEL_REF || 'Qwen2.5-3B-Instruct-Q4_K_M'
const node = Node.create({
ownerKeypairHex: generateOwnerKeypairHex(),
inviteToken: process.env.MESH_INVITE_TOKEN,
servingEnabled: true,
cacheDir: process.env.MESH_SDK_CACHE_DIR,
runtimeDir: process.env.MESH_SDK_RUNTIME_DIR
})

await node.start()
try {
await node.models.download(modelRef)
const served = await node.serving.load(modelRef, { devicePolicy: 'auto' })
const result = await node.inference.chat({
model: served.modelId,
messages: [{ role: 'user', content: 'Say hello from local serving.' }]
})
console.log(result.content)
await node.serving.unloadModel(served.modelId, { drainTimeoutMs: 1000 })
} finally {
await node.stop()
}
```

Use `serving.unloadInstance(instanceId)` when the load call returns an instance id and the application manages multiple copies of a model.

## Electron packaging

Package these items for each target platform and architecture:

- `native/<platform>-<arch>/mesh_llm_nodejs.node`
- the matching `meshllm-native-runtime-*` directory
- optional `console/` assets when the app calls `node.startConsole()`

Resolve the runtime from the packaged artifact directory rather than relying on a writable global install. Keep the runtime cache and model cache in Electron's app-specific data directory.

## Optional console

When console assets are included in the npm package:

```js
const consoleHandle = await node.startConsole({ port: 0 })
console.log(consoleHandle.url)
```

For a repository checkout, pass an explicit asset directory such as `crates/mesh-llm-ui/dist`. Do not expose the console on all interfaces unless the app has its own authentication and network policy.

## Runtime and errors

The Node API exposes `Client` for remote inference and `Node` for serving, model management, status, reconnect, and console hosting. Handle runtime resolution failures, unavailable endpoints, model download failures, and serving errors as application errors. Persist the owner keypair instead of generating a new one each time the Electron window opens.
Loading
Loading