fix(website-server): Add tiktoken as external dependency in esbuild bundle

[yamadashy]

Summary

Fixes Cloud Run deployment error caused by missing tiktoken_bg.wasm file.

When bundling the server with esbuild, tiktoken was being inlined which caused the WASM file path resolution to fail at runtime. tiktoken loads its WASM files from __dirname, so it must be kept as an external dependency to preserve correct path resolution.

Changes

• Add --external:tiktoken to esbuild command
• Copy tiktoken module to runtime image alongside tinypool

This is a follow-up fix for #1056.

Checklist

• Run npm run test
• Run npm run lint

🤖 Generated with Claude Code

[coderabbitai]

📝 Walkthrough

Walkthrough

The Dockerfile is modified to consolidate WASM file handling. Tiktoken is treated as an external dependency in esbuild bundling, a new build step collects WASM files into a centralized directory (dist-bundled/wasm), and the runtime image configuration is updated to reference this new location instead of directly copying tree-sitter WASM files.

Changes

Cohort / File(s)	Summary
Docker build and WASM bundling website/server/Dockerfile	Treats tiktoken as external in esbuild; adds step to collect WASM files from node_modules into dist-bundled/wasm; removes direct tree-sitter WASM copying from runtime image; updates COPY commands to include tiktoken with WASM files; updates REPOMIX_WASM_DIR environment variable to point to new WASM directory location.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately and specifically describes the main change: adding tiktoken as an external dependency in esbuild configuration to fix a bundling issue.
Description check	✅ Passed	The PR description covers the required sections with clear context, changes, and checklist completion. It explains the problem, solution, and links to related work.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/website-server-dockerfile-wasm-path

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Summary of Changes

Hello @yamadashy, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical Cloud Run deployment issue where the website-server failed to start due to tiktoken's WASM files not being correctly located at runtime. The fix involves configuring esbuild to treat tiktoken as an external dependency and refining the Docker build process to ensure all required WASM assets are properly collected and made available in the runtime image. This ensures the server can deploy and function as expected.

Highlights

• tiktoken External Dependency: The tiktoken library is now explicitly marked as an external dependency in the esbuild configuration. This change prevents tiktoken's WebAssembly (WASM) files from being inlined during the bundling process, which was causing runtime path resolution failures and Cloud Run deployment errors.
• WASM File Handling Improvement: A new build step has been introduced in the Dockerfile to generically collect all necessary WASM files from node_modules and place them into a dedicated dist-bundled/wasm directory. This ensures all required WASM assets, including those for tiktoken and tree-sitter, are correctly staged for the runtime environment.
• Dockerfile Updates: The Dockerfile has been updated to reflect these changes, including copying the tiktoken module alongside tinypool and adjusting the REPOMIX_WASM_DIR environment variable to point to the new centralized WASM file location (/app/dist-bundled/wasm).

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.24%. Comparing base (a4aa798) to head (540310f).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1065   +/-   ##
=======================================
  Coverage   90.24%   90.24%           
=======================================
  Files         124      124           
  Lines        9605     9605           
  Branches     1754     1754           
=======================================
  Hits         8668     8668           
  Misses        937      937           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gemini-code-assist]

Code Review

This pull request correctly addresses the runtime error with tiktoken by marking it as an external dependency for esbuild and copying the module into the final Docker image. This ensures that tiktoken can correctly resolve the path to its WASM file at runtime. The changes also improve the handling of tree-sitter WASM files by using a more robust find command to locate them within node_modules, which is a good enhancement. I have one suggestion to improve the clarity and precision of the new command for collecting WASM files.

[gemini-code-assist]

This comment is slightly misleading as this command only collects tree-sitter WASM files, while tiktoken's WASM file is handled by copying its entire module. For clarity and robustness, the comment should be more specific. Additionally, the find command can be made more precise by using a single, more specific -path pattern that directly references the @repomix/tree-sitter-wasms package. Quoting the pattern is also important to prevent unexpected shell globbing.

# Collect tree-sitter WASM files from node_modules.
RUN mkdir -p dist-bundled/wasm && \
    find node_modules -path '*/@repomix/tree-sitter-wasms/out/*.wasm' -exec cp {} dist-bundled/wasm/ \;

[claude]

Code Review: PR #1065

Summary

This PR fixes a Cloud Run deployment error caused by tiktoken's WASM file (tiktoken_bg.wasm) not being found at runtime. The solution correctly identifies that tiktoken must be kept as an external dependency because it loads WASM files from __dirname at runtime.

✅ What looks good

1. Correct root cause analysis: The fix accurately diagnoses that esbuild was inlining tiktoken, breaking its WASM file path resolution.

2. Consistent approach: The solution follows the same pattern already used for tinypool - marking it as external and copying the entire module.

3. Good documentation: The inline comment explaining why tiktoken must be external is helpful for future maintainers.

4. WASM collection improvement: Using find with a pattern to collect tree-sitter WASM files is more robust than hardcoded paths, handling npm hoisting scenarios.

⚠️ Suggestions for improvement

1. Consider consolidating WASM file collection

Currently, tiktoken WASM files are handled implicitly by copying the entire node_modules/tiktoken directory, while tree-sitter WASMs are explicitly collected with find. Consider documenting this distinction or unifying the approach:

# Collect all WASM files from wherever they are in node_modules
RUN mkdir -p dist-bundled/wasm && \
    find node_modules -name "*.wasm" -path "*tree-sitter-wasms/out/*" -exec cp {} dist-bundled/wasm/ \; && \
    find node_modules/tiktoken -name "*.wasm" -exec cp {} dist-bundled/wasm/ \; 2>/dev/null || true

However, the current approach (copying entire tiktoken module) is likely intentional since tiktoken uses __dirname resolution internally.

2. Minor: Extra blank line at line 29

There's a double blank line after the WASM collection step (lines 28-29). Consider removing one for consistency.

🔒 Security

No security concerns identified. The changes only affect build-time file handling.

⚡ Performance

The changes should have minimal impact on Docker image size since tiktoken is a required runtime dependency. The bundled approach still provides faster cold start times.

🧪 Test coverage

This is a Dockerfile-only change that fixes a deployment issue. Manual testing during Cloud Run deployment is the appropriate validation method.

🔮 Premortem analysis

Potential Issue	Mitigation
tiktoken updates breaking WASM paths	If tiktoken changes how it resolves WASM files in future versions, this fix would need updating. Consider pinning the tiktoken version or adding a health check.
Image size increase	Copying the entire node_modules/tiktoken directory includes dev files. Could use npm prune --production before copying, but this is low risk.
Missing tiktoken dependencies	If tiktoken has peer/optional dependencies that are also externalized by esbuild, they would need similar treatment. Current testing suggests this is not an issue.

✅ Verdict

Approve - This is a well-targeted fix for a specific deployment issue. The approach is sound and follows established patterns in the codebase.

---

🤖 Review by Claude

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (2)

website/server/Dockerfile (2)

25-27: Consider verifying that WASM files are found.

The find command silently succeeds even if no files match the pattern. If tree-sitter WASM files are missing at runtime, it would be better to catch this during the build phase.

🔎 Suggested enhancement to verify files are found

 # Collect WASM files from wherever they are in node_modules
 RUN mkdir -p dist-bundled/wasm && \
-    find node_modules -name "*.wasm" -path "*tree-sitter-wasms/out/*" -exec cp {} dist-bundled/wasm/ \;
+    find node_modules -name "*.wasm" -path "*tree-sitter-wasms/out/*" -exec cp {} dist-bundled/wasm/ \; && \
+    [ "$(ls -A dist-bundled/wasm)" ] || { echo "ERROR: No WASM files found"; exit 1; }

---

28-29: Minor: Extra blank line.

There's an unnecessary consecutive blank line here.

🔎 Formatting cleanup

     find node_modules -name "*.wasm" -path "*tree-sitter-wasms/out/*" -exec cp {} dist-bundled/wasm/ \;
 
-
 # ==============================================================================

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 47a724e and 540310f.

📒 Files selected for processing (1)

• website/server/Dockerfile

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (16)

• GitHub Check: Build and run (macos-latest, 24.x)
• GitHub Check: Build and run (windows-latest, 20.x)
• GitHub Check: Build and run (macos-latest, 20.x)
• GitHub Check: Build and run (windows-latest, 22.x)
• GitHub Check: Build and run (windows-latest, 24.x)
• GitHub Check: Build and run (windows-latest, 25.x)
• GitHub Check: Build and run (ubuntu-latest, 20.x)
• GitHub Check: Build and run (ubuntu-latest, 24.x)
• GitHub Check: Test (macos-latest, 25.x)
• GitHub Check: Test (windows-latest, 25.x)
• GitHub Check: Test (windows-latest, 22.x)
• GitHub Check: Test (windows-latest, 20.x)
• GitHub Check: Build and run with Bun (windows-latest, latest)
• GitHub Check: Test with Bun (windows-latest, latest)
• GitHub Check: claude-review
• GitHub Check: Cloudflare Pages

🔇 Additional comments (4)

website/server/Dockerfile (4)

20-23: LGTM! External tiktoken configuration is correct.

The addition of --external:tiktoken properly prevents bundling, preserving the __dirname-based WASM file resolution that tiktoken requires. The explanatory comment is clear and helpful.

---

40-41: LGTM! Comment accurately reflects the bundled artifacts.

The updated comment clearly indicates that both the bundled server and WASM files are being copied.

---

51-51: LGTM! Environment variable correctly points to the new WASM directory.

The REPOMIX_WASM_DIR update aligns with the new centralized WASM file location established in the build stage.

---

43-45: The current implementation correctly preserves tiktoken's WASM files. Since the COPY directive copies the entire tiktoken module directory, all internal files—including tiktoken_bg.wasm and related tokenizer data—are preserved in the runtime image. The Dockerfile already demonstrates awareness of tiktoken's WASM loading requirements (line 20 comment), and the strategy of marking tiktoken as external in esbuild ensures the module remains intact for runtime loading via __dirname.