-
-
Notifications
You must be signed in to change notification settings - Fork 948
feat(http): add URL replacement feature for HTTP requests #6207
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
fd96bab
feat(http): add URL replacement feature for HTTP requests
ThomasSteinbach 4c3a549
[autofix.ci] apply automated fixes
autofix-ci[bot] 9af9a03
chore(http): optimized imports
ThomasSteinbach 77c07a9
Merge branch 'main' into main
ThomasSteinbach 5665a29
chore: reverted changes to .gitignore
ThomasSteinbach 11d079c
feat(http): raised log level for url replacments to 'trace'
ThomasSteinbach c1bb1d0
docs: fixed url replacement precedence
ThomasSteinbach af2c1c7
docs: use real world product names
ThomasSteinbach 4063436
docs: move bulk of url replacement doc onto a separate page
ThomasSteinbach 7a375f7
Merge branch 'main' of github.com:ThomasSteinbach/mise
ThomasSteinbach 904d4fd
fix: documentation lint fixes
ThomasSteinbach 6929311
Merge branch 'main' into main
ThomasSteinbach File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| # URL Replacements | ||
|
|
||
| mise does not include a built-in registry for downloading artifacts. | ||
| Instead, it retrieves remote registry manifests, which specify the URLs for downloading tools. | ||
|
|
||
| In some environments — such as enterprises or DMZs — these URLs may not be directly accessible and must be accessed through a proxy or internal mirror. | ||
|
|
||
| URL replacements allow you to modify or redirect any URL that mise attempts to access, making it possible to use internal proxies, mirrors, or alternative sources as needed. | ||
|
|
||
| ## Configuration Examples | ||
|
|
||
| Environment variable (JSON format): | ||
| ```bash | ||
| # Simple hostname replacement | ||
| export MISE_URL_REPLACEMENTS=' | ||
| { | ||
| "github.com": "nexus.mycompany.net", | ||
| "releases.hashicorp.com": "artifactory.xmpl.com" | ||
| }' | ||
|
|
||
| # Regex pattern (note the escaped backslashes in JSON) | ||
| export MISE_URL_REPLACEMENTS=' | ||
| { | ||
| "regex:^http://(.+)" = "https://$1", | ||
| "regex:https://github\.com/([^/]+)/([^/]+)/releases/download/(.+)": | ||
| "https://hub.corp.com/artifactory/github/$1/$2/$3" | ||
| }' | ||
| ``` | ||
|
|
||
| In mise.toml: | ||
| ```toml | ||
| [settings] | ||
| # Simple hostname replacement | ||
| url_replacements = { | ||
| "github.com" = "nexus.mycompany.net", | ||
| "releases.hashicorp.com" = "artifactory.xmpl.com" | ||
| } | ||
|
|
||
| # Regex patterns | ||
| url_replacements = { | ||
| "regex:^http://(.+)" = "https://$1", | ||
| "regex:https://github\\.com/([^/]+)/([^/]+)/releases/download/(.+)" = | ||
| "https://hub.corp.com/artifactory/github/$1/$2/$3" | ||
| } | ||
| ``` | ||
|
|
||
| ## Simple Hostname Replacement | ||
|
|
||
| For simple hostname-based mirroring, the key is the original hostname/domain to replace, | ||
| and the value is the replacement string. The replacement happens by searching and replacing | ||
| the pattern anywhere in the full URL string (including protocol, hostname, path, and query parameters). | ||
|
|
||
| Examples: | ||
| - `github.com` -> `nexus.mycompany.net` replaces GitHub hostnames | ||
| - `https://github.com` -> `https://nexus.mycompany.net` with protocol excludes e.g. 'api.github.com' | ||
| - `https://github.com` -> `https://proxy.corp.com/github-mirror` replaces GitHub with corporate proxy | ||
| - `http://host.net` -> `https://host.net` replaces protocol from HTTP to HTTPS | ||
|
|
||
| ## Advanced Regex Replacement | ||
|
|
||
| For more complex URL transformations, you can use regex patterns. When a key starts with `regex:`, | ||
| it is treated as a regular expression pattern that can match and transform any part of the URL. | ||
| The value can use capture groups from the regex pattern. | ||
|
|
||
| ### Regex Examples | ||
|
|
||
| #### 1. Protocol Conversion (HTTP to HTTPS) | ||
| ```toml | ||
| [settings] | ||
| url_replacements = { | ||
| "regex:^http://(.+)" = "https://$1" | ||
| } | ||
| ``` | ||
| This converts any HTTP URL to HTTPS by capturing everything after "http://" and replacing it with "https://". | ||
|
|
||
| #### 2. GitHub Release Mirroring with Path Restructuring | ||
| ```toml | ||
| [settings] | ||
| url_replacements = { | ||
| "regex:https://github\\.com/([^/]+)/([^/]+)/releases/download/(.+)" = | ||
| "https://hub.corp.com/artifactory/github/$1/$2/$3" | ||
| } | ||
| ``` | ||
| Transforms `https://github.com/owner/repo/releases/download/v1.0.0/file.tar.gz` | ||
| to `https://hub.corp.com/artifactory/github/owner/repo/v1.0.0/file.tar.gz` | ||
|
|
||
| #### 3. Subdomain to Path Conversion | ||
| ```toml | ||
| [settings] | ||
| url_replacements = { | ||
| "regex:https://([^.]+)\\.cdn\\.example\\.com/(.+)" = | ||
| "https://unified-cdn.com/$1/$2" | ||
| } | ||
| ``` | ||
| Converts subdomain-based URLs to path-based URLs on a unified CDN. | ||
|
|
||
| #### 4. Multiple Replacement Patterns (processed in order) | ||
| ```toml | ||
| [settings] | ||
| url_replacements = { | ||
| "regex:https://github\\.com/microsoft/(.+)" = | ||
| "https://internal-mirror.com/microsoft/$1", | ||
| "regex:https://github\\.com/(.+)" = | ||
| "https://public-mirror.com/github/$1", | ||
| "releases.hashicorp.com" = "hashicorp-mirror.internal.com" | ||
| } | ||
| ``` | ||
| First regex catches Microsoft repositories specifically, second catches all other GitHub URLs, | ||
| and the simple replacement handles HashiCorp. | ||
|
|
||
| ## Use Cases | ||
|
|
||
| 1. **Corporate Mirrors**: Replace public download URLs with internal corporate mirrors | ||
| 2. **Custom Registries**: Redirect package downloads to custom or private registries | ||
| 3. **Geographic Optimization**: Route downloads to geographically closer mirrors | ||
| 4. **Protocol Changes**: Convert HTTP URLs to HTTPS or vice versa | ||
|
|
||
| ## Regex Syntax | ||
|
|
||
| mise uses Rust regex engine which supports: | ||
| - `^` and `$` for anchors (start/end of string) | ||
| - `(.+)` for capture groups (use `$1`, `$2`, etc. in replacement) | ||
| - `[^/]+` for character classes (matches any character except `/`) | ||
| - `\\.` for escaping special characters (note: double backslash required in TOML) | ||
| - `*`, `+`, `?` for quantifiers | ||
| - `|` for alternation | ||
|
|
||
| You can check on regex101.com if your regex works (see [example](https://regex101.com/r/rmcIE1/1)). | ||
| Full regex syntax documentation: <https://docs.rs/regex/latest/regex/#syntax> | ||
|
|
||
| ## Precedence and Matching | ||
|
|
||
| - URL replacements are processed in the order they appear in the configuration (IndexMap insertion order) | ||
| - Both regex patterns (keys starting with `regex:`) and simple string replacements are processed in this same order | ||
| - The first matching pattern is used; subsequent patterns are ignored for that URL | ||
| - If no patterns match, the original URL is used unchanged | ||
|
|
||
| ## Security Considerations | ||
|
|
||
| When using regex patterns, ensure your replacement URLs point to trusted sources, | ||
| as this feature can redirect tool downloads to arbitrary locations. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.