Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wasi-nn]Add a new target for llama.cpp as a wasi-nn backend #3709

Merged
merged 2 commits into from
Sep 10, 2024

Conversation

lum1n0us
Copy link
Collaborator

@lum1n0us lum1n0us commented Aug 14, 2024

Minimum support:

  • accept (WasmEdge) customized model parameters. metadata.
  • Target wasmedge-ggml examples
    • basic
    • chatml
    • gemma
    • llama
    • qwen

In future. To support if required.

  • Target wasmedge-ggml examples
    • command-r. (>70G memory requirement)
    • embedding. (embedding mode)
    • grammar. (use the grammar option to constrain the model to generate the JSON output)
    • llama-stream. (new APIS compute_single, get_output_single, fini_single)
    • llava. (image representation)
    • llava-base64-stream. (image representation)
    • multimodel. (image representation)
  • Target llamaedge

@lum1n0us lum1n0us changed the title Add a new target for llama.cpp as a wasi-nn backend into cmake framework [wasi-nn]Add a new target for llama.cpp as a wasi-nn backend into cmake framework Aug 14, 2024
@lum1n0us lum1n0us force-pushed the feat/wasi_nn_llamacpp branch 3 times, most recently from fd1cd38 to 2976066 Compare August 16, 2024 02:03
@lum1n0us lum1n0us force-pushed the feat/wasi_nn_llamacpp branch from 2976066 to 4dc980f Compare August 22, 2024 08:46
@lum1n0us lum1n0us changed the title [wasi-nn]Add a new target for llama.cpp as a wasi-nn backend into cmake framework [wasi-nn]Add a new target for llama.cpp as a wasi-nn backend Aug 22, 2024
@lum1n0us lum1n0us force-pushed the feat/wasi_nn_llamacpp branch 2 times, most recently from 7b74f29 to 96d4f87 Compare August 23, 2024 07:16
@lum1n0us lum1n0us force-pushed the feat/wasi_nn_llamacpp branch 3 times, most recently from b30a5b5 to 860cb5d Compare September 4, 2024 04:17
@lum1n0us
Copy link
Collaborator Author

lum1n0us commented Sep 5, 2024

core/iwasm/libraries/wasi-nn/test/Dockerfile.wasi-nn-smoke needs more work. Will update it in another PR.

@lum1n0us lum1n0us force-pushed the feat/wasi_nn_llamacpp branch from 860cb5d to f893c7f Compare September 6, 2024 00:25
@lum1n0us lum1n0us marked this pull request as ready for review September 6, 2024 00:25
@lum1n0us lum1n0us force-pushed the feat/wasi_nn_llamacpp branch 4 times, most recently from 22174e2 to 2dcb65b Compare September 7, 2024 06:52
core/iwasm/libraries/wasi-nn/src/wasi_nn.c Outdated Show resolved Hide resolved
core/iwasm/libraries/wasi-nn/src/wasi_nn_llamacpp.c Outdated Show resolved Hide resolved
core/iwasm/libraries/wasi-nn/src/wasi_nn_llamacpp.c Outdated Show resolved Hide resolved
core/iwasm/libraries/wasi-nn/src/wasi_nn_llamacpp.h Outdated Show resolved Hide resolved
@lum1n0us lum1n0us force-pushed the feat/wasi_nn_llamacpp branch from 2dcb65b to 10965ba Compare September 9, 2024 10:02
@wenyongh wenyongh merged commit 0599351 into bytecodealliance:main Sep 10, 2024
388 checks passed
@lum1n0us lum1n0us deleted the feat/wasi_nn_llamacpp branch November 20, 2024 05:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants