Skip to content

feat: add Tinfoil private inference provider - #62

Merged
ilblackdragon merged 16 commits into
nearai:mainfrom
lwlee2608:feat/tinfoil-provider
Feb 18, 2026
Merged

ilblackdragon merged 16 commits into
nearai:mainfrom
lwlee2608:feat/tinfoil-provider

Conversation

@lwlee2608

@lwlee2608 lwlee2608 commented Feb 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add a dedicated LLM_BACKEND=tinfoil provider for Tinfoil private inference
  • Tinfoil is not compatible with the openai_compatible backend because rig-core 0.30.0 defaults to the OpenAI Responses API (/v1/responses), which Tinfoil does not support — it only implements the Chat Completions API (/v1/chat/completions), returning 403 "shim: path not allowed"
  • Rather than changing openai_compatible to use Chat Completions (which would break users expecting the Responses API), this adds a dedicated provider that explicitly uses rig's .completions_api() client
  • This lays the groundwork for integrating Tinfoil's privacy wrapper client (enclave attestation, TLS certificate pinning) once their Rust SDK is available — currently only Go and Python clients exist. The provider can be swapped to use the Tinfoil Rust client without changing the LlmProvider interface

Configuration

LLM_BACKEND=tinfoil
TINFOIL_API_KEY=tk_...
TINFOIL_MODEL=kimi-k2-5   # optional, default

Files changed

  • src/config.rs — Tinfoil variant in LlmBackend, TinfoilConfig struct, env var resolution
  • src/llm/mod.rs — create_tinfoil_provider() using rig's Chat Completions client
  • src/setup/wizard.rs — Add tinfoil: None to wizard's LlmConfig initializer

Test plan

  • Verified LLM_BACKEND=tinfoil with kimi-k2-5 model responds successfully
  • All 747 existing tests pass
  • CI passes

Add a dedicated Tinfoil LLM backend (`LLM_BACKEND=tinfoil`) for
Tinfoil's private inference service (https://tinfoil.sh).

The existing `openai_compatible` backend cannot be used with Tinfoil
because rig-core 0.30.0 defaults to the OpenAI Responses API
(`/v1/responses`), which Tinfoil does not support — it only implements
the Chat Completions API (`/v1/chat/completions`), returning 403
"shim: path not allowed" when hit on the responses endpoint.

Rather than changing `openai_compatible` to use Chat Completions (which
would break users expecting the Responses API), this adds a dedicated
provider that explicitly uses rig's `.completions_api()` client.

This also lays the groundwork for integrating Tinfoil's privacy wrapper
client (enclave attestation, TLS certificate pinning) once their Rust
SDK is available. The provider implementation can be swapped to use the
Tinfoil Rust client without changing the LlmProvider interface.

Configuration:
  LLM_BACKEND=tinfoil
  TINFOIL_API_KEY=tk_...
  TINFOIL_MODEL=kimi-k2-5   # optional, default
Copilot AI review requested due to automatic review settings February 13, 2026 08:05
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello @lwlee2608, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces Tinfoil as a new, dedicated LLM backend provider, enabling users to leverage its private inference capabilities. The integration addresses Tinfoil's specific API requirements by using the Chat Completions API, differentiating it from other OpenAI-compatible services. The changes encompass adding necessary configuration structures, environment variable parsing, and a client adapter to facilitate interaction with Tinfoil's services, while also preparing the codebase for future SDK enhancements.

Highlights

  • New LLM Backend: Introduced a dedicated Tinfoil LLM backend provider for private inference, distinct from the existing openai_compatible backend due to API differences.
  • Configuration: Implemented specific configuration for Tinfoil, allowing users to set LLM_BACKEND=tinfoil, TINFOIL_API_KEY, and optionally TINFOIL_MODEL (defaults to kimi-k2-5) via environment variables.
  • API Compatibility: Ensured the Tinfoil integration explicitly uses the Chat Completions API (/v1/chat/completions) via rig's client, as Tinfoil does not support the OpenAI Responses API (/v1/responses).
  • Future-proofing: Laid the groundwork for integrating Tinfoil's privacy wrapper client (enclave attestation, TLS certificate pinning) once their Rust SDK becomes available, allowing for a seamless swap without altering the LlmProvider interface.
Changelog
  • src/config.rs
    • Added Tinfoil variant to the LlmBackend enum.
    • Updated FromStr and Display implementations for LlmBackend to recognize and represent Tinfoil.
    • Defined a new TinfoilConfig struct to hold the API key and model for Tinfoil.
    • Included a tinfoil field in the main LlmConfig struct.
    • Implemented logic within LlmConfig::new to parse TINFOIL_API_KEY and TINFOIL_MODEL environment variables when LLM_BACKEND is set to Tinfoil.
  • src/llm/mod.rs
    • Extended the create_llm_provider function to instantiate a Tinfoil provider when LlmBackend::Tinfoil is selected.
    • Added the create_tinfoil_provider function, which constructs a rig::providers::openai::Client configured with Tinfoil's base URL and API key, specifically utilizing its completions_api().
  • src/setup/wizard.rs
    • Modified the LlmConfig initialization within the setup wizard to include tinfoil: None.
Activity
  • The author verified that LLM_BACKEND=tinfoil with the kimi-k2-5 model successfully responded.
  • All 747 existing tests passed.
  • CI pass status is pending.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for the Tinfoil private inference provider. The implementation is solid, adding the necessary configuration, provider creation logic, and updating the relevant enums and structs. The changes align well with the existing architecture for LLM providers. I have one minor suggestion to enhance configuration consistency.

Comment thread src/config.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds support for Tinfoil, a private inference provider, as a new LLM backend option. Tinfoil is specifically designed for privacy-focused model inference and requires the Chat Completions API rather than the Responses API that rig-core 0.30.0 defaults to for OpenAI-compatible endpoints. This necessitates a dedicated provider implementation rather than using the generic openai_compatible backend.

Changes:

  • Added Tinfoil variant to LlmBackend enum with corresponding FromStr and Display implementations
  • Created TinfoilConfig struct and environment variable resolution logic (TINFOIL_API_KEY, TINFOIL_MODEL)
  • Implemented create_tinfoil_provider() function that explicitly uses rig's Chat Completions API client via .completions_api()
  • Updated setup wizard to initialize tinfoil: None in LlmConfig construction

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
src/config.rs Adds Tinfoil backend enum variant, configuration struct, and environment variable resolution with "kimi-k2-5" as default model
src/llm/mod.rs Implements Tinfoil provider creation using OpenAI client with explicit Chat Completions API and hardcoded base URL
src/setup/wizard.rs Initializes tinfoil field to None in wizard's LlmConfig construction

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

serrrfirat
serrrfirat previously approved these changes Feb 13, 2026

@serrrfirat serrrfirat left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #62 Review: feat: add Tinfoil private inference provider

Author: lwlee2608 | Reviewed: 2026-02-13T12:36:00+04:00

Verdict: APPROVE ✅

Clean, well-structured PR that follows established patterns from other provider integrations (OpenRouter, OpenAI-compatible). No significant issues found.

Findings

P3 - Minor: Hardcoded base URL (informational)

File: src/llm/mod.rs:144

const TINFOIL_BASE_URL: &str = "https://inference.tinfoil.sh/v1";

The base URL is hardcoded. Other providers like OpenAI-compatible allow configurable endpoints. For a specific provider like Tinfoil this is acceptable, but consider adding an optional TINFOIL_BASE_URL env override for flexibility (e.g., self-hosted instances or staging).

P3 - Minor: Default model choice

File: src/config.rs:434

.unwrap_or_else(|| "kimi-k2-5".to_string());

Default model is kimi-k2-5. This is fine but worth confirming it's a valid model on Tinfoil's inference API.

Positives

  • Follows the exact same pattern as other provider implementations
  • Proper use of SecretString for API key handling
  • Good error messages with provider context
  • Config parsing correctly gated behind LlmBackend::Tinfoil check
  • Wizard properly initialized with tinfoil: None
  • Uses the existing RigAdapter via OpenAI-compatible client — clean reuse

Security Assessment

  • ✅ API key stored in SecretString, accessed via expose_secret()
  • ✅ No secrets logged (only model name in tracing::info)
  • ✅ HTTPS endpoint hardcoded
  • ✅ No user input passed unsanitized

Copilot AI review requested due to automatic review settings February 13, 2026 14:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI review requested due to automatic review settings February 14, 2026 12:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@serrrfirat

Copy link
Copy Markdown
Collaborator

@copilot review again

@tribendu

Copy link
Copy Markdown

PR Review: feat: add Tinfoil private inference provider

PR Author: Jason Lee
Size: 141 lines (+57/-1)


Summary

This PR adds support for Tinfoil, a private inference provider, to the IronClaw framework. It implements a new Tinfoil backend option for LLM operations that integrates with the Tinfoil API (https://inference.tinfoil.sh/v1) using the OpenAI-compatible client interface via the rig library. The implementation follows the existing pattern seen in other providers like Anthropic, Ollama, and OpenAI-compatible backends.

The changes are minimal and focused: adding the backend enum variant, configuration struct, environment variable parsing, provider creation logic, and a field in the setup wizard.


Pros

  • Consistent pattern: Follows the established pattern for LLM providers (Anthropic, Ollama, OpenAI-compatible)
  • Minimal footprint: Only 57 lines added, clean and focused changes
  • Proper error handling: Uses appropriate LlmError variants (AuthFailed, RequestFailed)
  • Secret handling: Correctly uses SecretString for API key storage (prevents accidental logging)
  • Good documentation: Clear doc comments (/// Configuration for Tinfoil private inference.)
  • Sensible defaults: Default model kimi-k2-5 if TINFOIL_MODEL not specified
  • Informative logging: tracing::info! log entry when Tinfoil provider is initialized
  • Environment variable hints: Helpful error message suggests setting TINFOIL_API_KEY

Concerns

  1. Missing tests: No test coverage added for:

    • LlmBackend::from_str with "tinfoil" input
    • LlmBackend::Display for Tinfoil variant
    • TinfoilConfig parsing from environment variables
    • create_tinfoil_provider function
  2. Hardcoded URL: The base URL https://inference.tinfoil.sh/v1 is a constant. Consider if this should be:

    • Configurable via environment variable (e.g., TINFOIL_BASE_URL)
    • Part of TinfoilConfig struct (like some other providers might do)
  3. Secret exposure: In create_tinfoil_provider, the API key is exposed via tf.api_key.expose_secret(). While this is passed to the OpenAI client builder, ensure:

    • The rig library doesn't log this value
    • No debug/trace logs inadvertently expose this secret
    • Consider if there's a way to pass secrets without exposing them
  4. Incomplete Display implementation: The Display impl for LlmBackend includes Tinfoil, but there's no test for this formatting consistency.

  5. No documentation update: If there's a README or CONTRIBUTING guide with LLM backend examples, it should be updated to include Tinfoil.

  6. Wizard integration: The wizard only sets tinfoil: None but doesn't appear to have interactive setup for the Tinfoil provider. Consider whether this is intentional or should be added.


Suggestions

  1. Add unit tests:

    #[cfg(test)]
    mod tests {
        use super::*;
    
        #[test]
        fn test_tinfoil_backend_from_str() {
            assert_eq!(LlmBackend::from_str("tinfoil"), Ok(LlmBackend::Tinfoil));
        }
    
        #[test]
        fn test_tinfoil_backend_display() {
            assert_eq!(LlmBackend::Tinfoil.to_string(), "tinfoil");
        }
    }
  2. Make base URL configurable:

    /// Configuration for Tinfoil private inference.
    #[derive(Debug, Clone)]
    pub struct TinfoilConfig {
        pub api_key: SecretString,
        pub model: String,
        #[serde(default = "default_tinfoil_base_url")]
        pub base_url: String,
    }
    
    fn default_tinfoil_base_url() -> String {
        "https://inference.tinfoil.sh/v1".to_string()
    }
  3. Consider integration test: Add a test that verifies the provider can be created (mocking the API call or using a test flag).

  4. Update documentation: If user-facing docs exist, add a section explaining how to configure Tinfoil:

    • Required env vars: TINFOIL_API_KEY
    • Optional env var: TINFOIL_MODEL (default: kimi-k2-5)
    • Backend selection: LLM_BACKEND=tinfoil
  5. Wizard enhancement: If interactive setup is desired, add a prompt for Tinfoil credentials in src/setup/wizard.rs.

  6. Error message consistency: Ensure the hint in the MissingRequired error is consistent with other providers' hints.

  7. Consider model validation: Add validation that the model name is non-empty or matches expected patterns.

  8. Add tracing at secret exposure point: Consider adding a tracing::debug! note when passing secrets to external libraries, to help with debugging if secrets leak.


Overall Assessment

Status: 🟢 Approve with minor suggestions

This is a clean, well-structured implementation that follows the project's existing patterns. The code is minimal, uses appropriate Rust idioms, and handles secrets correctly. The primary concern is the lack of test coverage, which should be addressed before merge. The hardcoded base URL is acceptable for an initial implementation but should be considered for future flexibility.

The PR successfully adds Tinfoil as a new LLM backend with proper configuration, error handling, and integration with the existing provider infrastructure.

Copilot AI review requested due to automatic review settings February 17, 2026 06:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/config.rs:362

  • The documentation comment for LlmConfig lists examples of backend values but does not include the newly added tinfoil backend. Consider updating the examples to include tinfoil for completeness, e.g., (e.g. 'openai', 'anthropic', 'ollama', 'openai_compatible', 'tinfoil').
/// NEAR AI remains the default backend. Users can switch to other providers
/// by setting `LLM_BACKEND` (e.g. `openai`, `anthropic`, `ollama`).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/llm/mod.rs
lwlee2608 and others added 2 commits February 17, 2026 15:56
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings February 17, 2026 07:58
@lwlee2608

Copy link
Copy Markdown
Contributor Author

Yep will merge it after CI succeeds.

Thanks @serrrfirat , CI fixed

Copilot AI review requested due to automatic review settings February 17, 2026 17:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@serrrfirat
serrrfirat self-requested a review February 17, 2026 17:56
serrrfirat
serrrfirat previously approved these changes Feb 17, 2026
Copilot AI review requested due to automatic review settings February 17, 2026 18:24
@lwlee2608

Copy link
Copy Markdown
Contributor Author

Hi @serrrfirat - just to clarify clearly: this CI failure was not caused by the Tinfoil feature changes in this PR. It was from a pre-existing rustfmt mismatch in src/llm/session.rs on the merge ref. I pushed a formatting-only fix in commit 02cce46 to unblock Code Style. Thank you!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings February 18, 2026 05:56
@ilblackdragon
ilblackdragon merged commit 96d5fc0 into nearai:main Feb 18, 2026
6 checks passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/llm/mod.rs
})?;

// Tinfoil currently only supports the Chat Completions API and not the newer Responses API,
// so we must explicitly select the completions API here (unlike other OpenAI-compatible providers).

Copilot AI Feb 18, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment phrase "unlike other OpenAI-compatible providers" is ambiguous. Consider rephrasing to "unlike the openai_compatible backend" or "unlike the default behavior for OpenAI client" to clarify that you're referring to the openai_compatible backend in this codebase, which relies on the default Responses API behavior.

Suggested change
// so we must explicitly select the completions API here (unlike other OpenAI-compatible providers).
// so we must explicitly select the completions API here, unlike the openai_compatible backend,
// which relies on the default OpenAI Responses API behavior.

Copilot uses AI. Check for mistakes.
@github-actions github-actions Bot mentioned this pull request Feb 18, 2026
jaswinder6991 pushed a commit to jaswinder6991/ironclaw that referenced this pull request Feb 26, 2026
* feat: add Tinfoil private inference provider

Add a dedicated Tinfoil LLM backend (`LLM_BACKEND=tinfoil`) for
Tinfoil's private inference service (https://tinfoil.sh).

The existing `openai_compatible` backend cannot be used with Tinfoil
because rig-core 0.30.0 defaults to the OpenAI Responses API
(`/v1/responses`), which Tinfoil does not support — it only implements
the Chat Completions API (`/v1/chat/completions`), returning 403
"shim: path not allowed" when hit on the responses endpoint.

Rather than changing `openai_compatible` to use Chat Completions (which
would break users expecting the Responses API), this adds a dedicated
provider that explicitly uses rig's `.completions_api()` client.

This also lays the groundwork for integrating Tinfoil's privacy wrapper
client (enclave attestation, TLS certificate pinning) once their Rust
SDK is available. The provider implementation can be swapped to use the
Tinfoil Rust client without changing the LlmProvider interface.

Configuration:
  LLM_BACKEND=tinfoil
  TINFOIL_API_KEY=tk_...
  TINFOIL_MODEL=kimi-k2-5   # optional, default

* style: fix rustfmt formatting in Tinfoil provider

* style: remove unnecessary tin_foil alias for Tinfoil backend

* Update src/llm/mod.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: add tinfoil field to LlmConfig test fixture

* style: fix rustfmt output in session manager

---------

Co-authored-by: firat.sertgoz <f@nuff.tech>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Illia Polosukhin <ilblackdragon@gmail.com>
bkutasi pushed a commit to bkutasi/ironclaw that referenced this pull request Mar 28, 2026
* feat: add Tinfoil private inference provider

Add a dedicated Tinfoil LLM backend (`LLM_BACKEND=tinfoil`) for
Tinfoil's private inference service (https://tinfoil.sh).

The existing `openai_compatible` backend cannot be used with Tinfoil
because rig-core 0.30.0 defaults to the OpenAI Responses API
(`/v1/responses`), which Tinfoil does not support — it only implements
the Chat Completions API (`/v1/chat/completions`), returning 403
"shim: path not allowed" when hit on the responses endpoint.

Rather than changing `openai_compatible` to use Chat Completions (which
would break users expecting the Responses API), this adds a dedicated
provider that explicitly uses rig's `.completions_api()` client.

This also lays the groundwork for integrating Tinfoil's privacy wrapper
client (enclave attestation, TLS certificate pinning) once their Rust
SDK is available. The provider implementation can be swapped to use the
Tinfoil Rust client without changing the LlmProvider interface.

Configuration:
  LLM_BACKEND=tinfoil
  TINFOIL_API_KEY=tk_...
  TINFOIL_MODEL=kimi-k2-5   # optional, default

* style: fix rustfmt formatting in Tinfoil provider

* style: remove unnecessary tin_foil alias for Tinfoil backend

* Update src/llm/mod.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: add tinfoil field to LlmConfig test fixture

* style: fix rustfmt output in session manager

---------

Co-authored-by: firat.sertgoz <f@nuff.tech>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Illia Polosukhin <ilblackdragon@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants