Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
f164627
[api] Make bundled Java compilation opt-in and restrict remote URL lo…
access2rohit Aug 27, 2026
f83c5e7
[api] Fix temp-jar cleanup in UrlAccessAndCompilationTest on Windows
access2rohit Aug 31, 2026
a82030c
[integration] Enable bundled-source compilation in CustomTranslatorTest
access2rohit Aug 31, 2026
6416ca9
[api] Route metadata-declared artifact downloads through Utils.openUrl
access2rohit Sep 2, 2026
f0b9088
[api] Cover redirect resolution and repository download paths with tests
access2rohit Sep 2, 2026
dcb78a6
[api] Address review feedback on URL handling
access2rohit Sep 2, 2026
eb2f663
[api] Apply offline/opt-out semantics in the shared connection helper
access2rohit Sep 2, 2026
a672534
[api] Restore buffering, tighten scope, and harden the compile path
access2rohit Sep 2, 2026
4e5ac22
[api] Fail loudly on skipped compilation and correct redirect handling
access2rohit Sep 2, 2026
0b0ad82
[api] Leave non-redirect status codes to the caller
access2rohit Sep 2, 2026
7d915fc
[docs] Note the opt-in for compiling bundled java sources
access2rohit Sep 2, 2026
f5ec16a
[api] Correct origin comparison, header copying and compile reporting
access2rohit Sep 2, 2026
eab4095
Remove scratch file committed by mistake
access2rohit Sep 2, 2026
ac57f07
[api] Fix thread safety and null handling in the new tests
access2rohit Sep 2, 2026
54f5dec
[api] Cover all credential header names and drop a vacuous assertion
access2rohit Sep 2, 2026
9860b07
[api] Validate nested jar URLs and correct the compilation failure path
access2rohit Sep 3, 2026
3e17cae
[api] Treat null headers as empty in openHttpConnection
access2rohit Sep 3, 2026
cafc6cb
[api] Normalise null headers in the redirect loop too, and fix its test
access2rohit Sep 3, 2026
ef1f7cd
[api] Judge nested URLs by destination, not by scheme name
access2rohit Sep 3, 2026
d637802
[api] Drop the load failure on skipped compilation, and tighten redir…
access2rohit Sep 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,9 @@ protected void download(Path tmp, URI baseUri, Artifact.Item item, Progress prog
}

logger.debug("Downloading artifact: {} ...", fileUri);
try (InputStream is = new BufferedInputStream(fileUri.toURL().openStream())) {
// The artifact URI comes from the repository metadata and may be absolute, so route it
// through Utils.openUrl to apply the same URL handling as the rest of the download paths.
try (InputStream is = Utils.openUrl(fileUri.toURL())) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, and this one was my mistake — openUrl returns conn.getInputStream() for http, which is unbuffered, so routing these paths through it silently dropped the BufferedInputStream they had.

Fixed in a672534 by buffering inside openUrl rather than at each call site, so this path and SimpleUrlRepository.download both get it back and any future caller does too. openUrl already buffered the file:/jar: branch, so this also makes the two branches consistent.

save(is, tmp, item, progress);
}
}
Expand Down
11 changes: 7 additions & 4 deletions api/src/main/java/ai/djl/repository/SimpleUrlRepository.java
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
Expand Down Expand Up @@ -102,7 +101,10 @@ public List<MRL> getResources() {
protected void download(Path tmp, URI baseUri, Artifact.Item item, Progress progress)
throws IOException {
logger.debug("Downloading artifact: {} ...", uri);
try (InputStream is = new BufferedInputStream(uri.toURL().openStream())) {
// Route through Utils.openUrl so the download is subject to the same URL handling
// (scheme restriction, non-public address blocking, redirect re-validation) instead of
// calling openStream() directly, which would bypass them.
try (InputStream is = Utils.openUrl(uri.toURL())) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same root cause as the AbstractRepository comment and fixed in the same commit (a672534): openUrl now wraps the http stream in a BufferedInputStream, so this call site keeps the buffering it had before. Handling it inside openUrl rather than here keeps the two download paths from drifting apart again.

save(is, tmp, item, progress);
}
}
Expand Down Expand Up @@ -140,8 +142,9 @@ private long getContentLength() throws IOException {
HttpURLConnection conn = null;
try {
resolved = true;
conn = (HttpURLConnection) uri.toURL().openConnection();
conn.setRequestMethod("HEAD");
// Route the HEAD probe through the same handling as the download path, so
// redirects are resolved explicitly and every hop is checked.
conn = Utils.openHttpConnection(uri.toURL(), "HEAD", Collections.emptyMap());
int code = conn.getResponseCode();
Comment on lines 140 to 155

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, the description did not match the code — getContentLength() was building its own connection and only duplicating the host check. Fixed the code rather than the wording in dcb78a6: both the download and the HEAD probe now go through the shared Utils.openHttpConnection(...), so the described behavior is what actually happens.

if (code != 200) {
logger.debug("Failed detect content length, error code: {}", code);
Expand Down
23 changes: 23 additions & 0 deletions api/src/main/java/ai/djl/util/ClassLoaderUtils.java
Original file line number Diff line number Diff line change
Expand Up @@ -242,9 +242,22 @@ public static void nativeLoad(String nativeHelper, String path) {
/**
* Tries to compile java classes in the directory.
*
* <p>Compiling bundled {@code .java} sources at model-load time is disabled by default. Enable
* it with {@code DJL_COMPILE_JAVA=true} or {@code -Dai.djl.compile_java=true} when loading
* models from a trusted source. Models that ship precompiled {@code .class} or {@code .jar}
* files, or that supply a translator programmatically, are unaffected.
*
* @param dir the directory to scan java file.
*/
public static void compileJavaClass(Path dir) {
if (!isDynamicCompilationEnabled()) {
logger.warn(
"Skipping bundled Java compilation in {}: dynamic compilation is disabled by"
+ " default. Set DJL_COMPILE_JAVA=true or"
+ " -Dai.djl.compile_java=true to enable it for trusted models.",
dir);
return;
}
try {
if (!Files.isDirectory(dir)) {
logger.debug("Directory not exists: {}", dir);
Comment thread
Copilot marked this conversation as resolved.
Resolved
Expand All @@ -265,4 +278,14 @@ public static void compileJavaClass(Path dir) {
logger.warn("Failed to compile bundled java file", e);
}
}

/**
* Returns whether dynamic compilation of bundled java sources is explicitly enabled.
*
* @return true if {@code DJL_COMPILE_JAVA} / {@code ai.djl.compile_java} is set true
*/
public static boolean isDynamicCompilationEnabled() {
String mode = Utils.getenv("DJL_COMPILE_JAVA", System.getProperty("ai.djl.compile_java"));
return Boolean.parseBoolean(mode);
}
}
153 changes: 150 additions & 3 deletions api/src/main/java/ai/djl/util/Utils.java
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.InetAddress;
import java.net.URL;
import java.net.UnknownHostException;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileVisitOption;
import java.nio.file.Files;
Expand All @@ -39,6 +41,7 @@
import java.util.Map;
import java.util.Objects;
import java.util.Scanner;
import java.util.function.Predicate;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
Expand All @@ -49,6 +52,8 @@ public final class Utils {

private static final Logger logger = LoggerFactory.getLogger(Utils.class);

private static final int MAX_REDIRECTS = 5;

public static final String[] EMPTY_ARRAY = new String[0];

private Utils() {}
Expand Down Expand Up @@ -524,18 +529,160 @@ public static InputStream openUrl(URL url) throws IOException {
* @throws IOException if an I/O exception occurs
*/
public static InputStream openUrl(URL url, Map<String, String> headers) throws IOException {
// Remote fetches are limited to public http(s) destinations by default, and redirects are
// resolved explicitly so each hop is checked against the same rule. Local-resource schemes
// (file:, jar:) are unrestricted, since they do not perform a network fetch. Other schemes
// are not supported. Set DJL_ALLOW_INSECURE_URL=true (or -Dai.djl.allow_insecure_url=true)
// to restore the previous unrestricted behavior.
boolean insecureAllowed = isInsecureUrlAllowed();
String protocol = url.getProtocol();
if ("http".equalsIgnoreCase(protocol) || "https".equalsIgnoreCase(protocol)) {
if (isOfflineMode()) {
throw new IOException("Offline mode is enabled.");
}
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
if (insecureAllowed) {
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
for (Map.Entry<String, String> entry : headers.entrySet()) {
conn.addRequestProperty(entry.getKey(), entry.getValue());
}
return conn.getInputStream();
}
return openHttpConnection(url, "GET", headers).getInputStream();
}
// Local-resource schemes and full-opt-out use the legacy direct stream.
if (insecureAllowed
|| "file".equalsIgnoreCase(protocol)
|| "jar".equalsIgnoreCase(protocol)) {
return new BufferedInputStream(url.openStream());
}
throw new IOException(
"Blocked request using unsupported URL protocol: "
+ protocol
+ ". Set DJL_ALLOW_INSECURE_URL=true to override.");
}

/**
* Opens an http(s) connection using the specified request method, resolving redirects
* explicitly so that every hop is checked against the same destination rule.
*
* <p>This is the shared entry point for remote reads: {@link #openUrl(URL, Map)} uses it for
* {@code GET}, and callers that only need response metadata can use it for {@code HEAD}. The
* caller owns the returned connection and should disconnect it when finished.
*
* @param url the URL to open
* @param method the HTTP request method, for example {@code GET} or {@code HEAD}
* @param headers HTTP headers
* @return the connection for the final, validated hop
* @throws IOException if an I/O exception occurs or a hop is not allowed
*/
public static HttpURLConnection openHttpConnection(
URL url, String method, Map<String, String> headers) throws IOException {
return openHttpConnection(url, method, headers, Utils::isPublicHost);
}
Comment on lines +668 to +687

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct on both counts, and this was a regression from the previous commit — fixed in eb2f663.

When the HEAD probe moved into the shared helper, the isInsecureUrlAllowed() checks that getContentLength() previously had were dropped, so the probe stayed blocked even with DJL_ALLOW_INSECURE_URL=true, and the offline guard didn't apply to it either.

Rather than re-adding the checks at each call site, both now live in openHttpConnection(url, method, headers):

  • offline mode throws before any connection is opened;
  • with the opt-out set, it returns a plain connection for the requested method;
  • otherwise it takes the validated redirect-resolving path.

openUrl now delegates to it instead of repeating the same logic, so GET and HEAD cannot drift apart again. Added testOptOutAppliesToHeadConnection, which asserts the probe is blocked by default for a loopback host and succeeds once the flag is set.

Separately, thanks for the note about compileJavaClass logging unconditionally — also fixed in eb2f663. The gate ran before the directory and file scan, so every model load warned even with no bundled sources. It now scans first: missing directory logs at debug, a directory with no .java is silent, and the warning is emitted only when sources are actually present (including the file count).

Comment on lines +668 to +687

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 3e17cae. Normalised in the public overload, ahead of both the opt-out branch and the delegation to the validated path, so a null map is handled once regardless of which route the call takes. The package-private overload is only reachable through that call site, so I left it without a duplicate guard.

I went with treating null as empty rather than IllegalArgumentException: "no headers" is a reasonable thing for a caller to mean, and the method already has a natural representation for it.

Two things worth stating rather than leaving implicit:

  • This also widens openUrl(URL, Map), which delegates here — openUrl(url, null) previously threw NullPointerException and now behaves as an empty map. Strictly more permissive, so no existing caller can be affected, but it is a behavior change beyond the reported one.
  • Added testNullHeadersAreTreatedAsEmpty, covering both routes (rejected on the destination by default, request actually issued with the opt-out set). I confirmed it fails on the previous code with NullPointerException: Cannot invoke "java.util.Map.entrySet()" before taking the fix.

I deliberately did not add the same guard to hasSkippedJavaSources(Path), the other new public method here: it passes the path to Files.isDirectory, and every java.nio.file.Files method plus compileJavaClass itself already throw NullPointerException on a null path. Guarding only this one would be the inconsistent choice. Happy to revisit if you would rather both be lenient.


/**
* Opens an http(s) connection, resolving redirects explicitly so that every hop is checked
* against {@code hostAllowed} rather than being followed by the connection itself.
*
* <p>The host check is a parameter so the redirect handling can be exercised against a local
* test server.
*
* @param url the URL to open
* @param method the HTTP request method
* @param headers HTTP headers
* @param hostAllowed the check applied to the host of every hop
* @return the connection for the final, validated hop
* @throws IOException if an I/O exception occurs or a hop is not allowed
*/
static HttpURLConnection openHttpConnection(
URL url, String method, Map<String, String> headers, Predicate<String> hostAllowed)
throws IOException {
URL current = url;
// One initial request plus at most MAX_REDIRECTS redirect hops.
for (int i = 0; i <= MAX_REDIRECTS; ++i) {
String protocol = current.getProtocol();
if (!"http".equalsIgnoreCase(protocol) && !"https".equalsIgnoreCase(protocol)) {
throw new IOException("Blocked redirect to unsupported URL protocol: " + protocol);
}
Comment on lines +709 to +717

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is off by one — iteration i = 0 issues the initial request, not a redirect, so i <= MAX_REDIRECTS gives 1 request + at most 5 redirect hops, which matches MAX_REDIRECTS = 5 and the >5 message.

Traced against testTooManyRedirectsIsRejected (a chain where every hop redirects):

i=0  GET /hop0 -> 302  (redirect 1 resolved)
...
i=4  GET /hop4 -> 302  (redirect 5 resolved)
i=5  GET /hop5 -> 302  -> loop exits, throws "Too many redirects (>5)"

So /hop6 is never fetched: 6 requests total = 1 initial + 5 follows. Changing the bound to i < MAX_REDIRECTS would actually introduce an off-by-one (only 4 redirects followed). Since the intent wasn't obvious, I added a clarifying comment in dcb78a6: // One initial request plus at most MAX_REDIRECTS redirect hops. Happy to rename the constant if you'd still prefer that.

if (!hostAllowed.test(current.getHost())) {
throw new IOException(
"Blocked request to non-public address: " + current.getHost());
}
HttpURLConnection conn = (HttpURLConnection) current.openConnection();
conn.setInstanceFollowRedirects(false);
Comment on lines +718 to +723

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point, and it's a real limitation rather than something I can fully close here. Pinning the connection to a validated address means connecting by IP and carrying the original host in the Host header, which then has to be reconciled with TLS SNI and certificate verification — more surgery than belongs in this change, and easy to get subtly wrong.

Documented it explicitly on isPublicHost in a672534 so the gap is stated rather than implied:

Note that this resolves the host name and the subsequent connection resolves it again, so a name that resolves to an allowed address here could resolve to a different address when the connection is made. Pinning the connection to a validated address would be needed to close that gap; environments that require it should restrict egress at the network layer.

Happy to look at pinning as a follow-up if you think it's worth carrying.

conn.setRequestMethod(method);
for (Map.Entry<String, String> entry : headers.entrySet()) {
conn.addRequestProperty(entry.getKey(), entry.getValue());
}
return conn.getInputStream();
int code = conn.getResponseCode();
if (code < 300 || code >= 400) {
return conn;
}
// Redirect: re-validate the target on the next loop iteration instead of letting
// HttpURLConnection follow it blindly (which would defeat the host check above).
String location = conn.getHeaderField("Location");
conn.disconnect();
if (location == null) {
throw new IOException("Redirect (" + code + ") with no Location header");
}
// Resolve relative redirects against the current URL, then re-validate the target.
current = new URL(current, location);
}
throw new IOException("Too many redirects (>" + MAX_REDIRECTS + ") for URL: " + url);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is off by one — iteration i = 0 issues the initial request, not a redirect, so i <= MAX_REDIRECTS gives 1 request + at most 5 redirect hops, which matches MAX_REDIRECTS = 5 and the >5 message.

Traced against testTooManyRedirectsIsRejected (a chain where every hop redirects):

i=0  GET /hop0 -> 302  (redirect 1 resolved)
...
i=4  GET /hop4 -> 302  (redirect 5 resolved)
i=5  GET /hop5 -> 302  -> loop exits, throws "Too many redirects (>5)"

So /hop6 is never fetched: 6 requests total = 1 initial + 5 follows. Changing the bound to i < MAX_REDIRECTS would actually introduce an off-by-one (only 4 redirects followed). Since the intent wasn't obvious, I added a clarifying comment in dcb78a6: // One initial request plus at most MAX_REDIRECTS redirect hops. Happy to rename the constant if you'd still prefer that.

}

/**
* Returns whether insecure (non-public) URL access is explicitly allowed.
*
* @return true if {@code DJL_ALLOW_INSECURE_URL} / {@code ai.djl.allow_insecure_url} is set
* true
*/
public static boolean isInsecureUrlAllowed() {
String mode =
getenv("DJL_ALLOW_INSECURE_URL", System.getProperty("ai.djl.allow_insecure_url"));
return Boolean.parseBoolean(mode);
}

/**
* Returns whether a host resolves exclusively to public (non-internal) IP addresses.
*
* <p>A host is rejected if it is empty or if any resolved address is loopback, link-local,
* site-local (RFC 1918 private), wildcard, multicast, or an IPv6 unique local address
* (fc00::/7). The latter is checked explicitly because {@link InetAddress#isSiteLocalAddress()}
* only covers the deprecated fec0::/10 range.
*
* @param host the host name to validate
* @return true if every resolved address is a public address
*/
public static boolean isPublicHost(String host) {
if (host == null || host.isEmpty()) {
return false;
}
try {
InetAddress[] addresses = InetAddress.getAllByName(host);
if (addresses.length == 0) {
return false;
}
for (InetAddress addr : addresses) {
if (addr.isLoopbackAddress()
|| addr.isLinkLocalAddress()
|| addr.isSiteLocalAddress()
|| addr.isAnyLocalAddress()
|| addr.isMulticastAddress()
|| isIpv6UniqueLocal(addr)) {
return false;
}
}
} catch (UnknownHostException e) {
return false;
}
return new BufferedInputStream(url.openStream());
return true;
}

private static boolean isIpv6UniqueLocal(InetAddress addr) {
byte[] bytes = addr.getAddress();
// fc00::/7 - the first seven bits are 1111110
return bytes.length == 16 && (bytes[0] & 0xFE) == 0xFC;
}

/**
Expand Down
114 changes: 114 additions & 0 deletions api/src/test/java/ai/djl/repository/RepositoryDownloadTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
/*
* Copyright 2026 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
* with the License. A copy of the License is located at
*
* http://aws.amazon.com/apache2.0/
*
* or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
* OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
* and limitations under the License.
*/
package ai.djl.repository;

import org.testng.Assert;
import org.testng.annotations.Test;

import java.io.IOException;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;

/** Tests the URL handling applied when downloading artifacts declared by repository metadata. */
public class RepositoryDownloadTest {

@Test
public void testNonPublicArtifactUriIsRejected() throws IOException {
// An artifact URI declared in repository metadata may be absolute, which bypasses base-URI
// resolution. It must still be subject to the same destination rule as the other download
// paths, so a non-public destination is refused.
IOException e = downloadItemUri("http://169.254.169.254/latest/meta-data/x.tar.gz");
Assert.assertTrue(
e.getMessage().contains("non-public"), "unexpected message: " + e.getMessage());
}

@Test
public void testPrivateArtifactUriIsRejected() throws IOException {
IOException e = downloadItemUri("http://10.0.0.5/model.tar.gz");
Assert.assertTrue(
e.getMessage().contains("non-public"), "unexpected message: " + e.getMessage());
}

@Test
public void testLoopbackArtifactUriIsRejected() throws IOException {
IOException e = downloadItemUri("http://127.0.0.1:1/model.tar.gz");
Assert.assertTrue(
e.getMessage().contains("non-public"), "unexpected message: " + e.getMessage());
}

@Test
public void testUnsupportedSchemeArtifactUriIsRejected() throws IOException {
IOException e = downloadItemUri("ftp://ftp.example.com/model.tar.gz");
Assert.assertTrue(
e.getMessage().contains("URL protocol"), "unexpected message: " + e.getMessage());
}

/**
* Invokes the shared artifact-download path with an absolute item URI and returns the thrown
* exception.
*/
private IOException downloadItemUri(String itemUri) throws IOException {
Path tmp = Files.createTempDirectory("djl-download-probe");
try {
Repository repository =
new RemoteRepository(
"test", URI.create("https://resources.djl.ai/test-models/"));
Artifact.Item item = new Artifact.Item();
item.setName("data");
item.setUri(itemUri);

return Assert.expectThrows(
IOException.class,
() ->
((AbstractRepository) repository)
.download(tmp, URI.create(""), item, null));
} finally {
Files.deleteIfExists(tmp);
}
}

@Test
public void testSimpleUrlRepositoryRejectsNonPublicHost() {
// The content-length HEAD probe is a second entry point into the same download flow, so it
// applies the same destination rule.
SimpleUrlRepository repository =
new SimpleUrlRepository(
"test",
URI.create("http://169.254.169.254/latest/meta-data/x.tar.gz"),
"x.tar.gz");
// getResources() resolves metadata, which performs the HEAD probe.
Assert.assertTrue(repository.getResources().isEmpty());
}

@Test
public void testSimpleUrlRepositoryDownloadRejectsNonPublicHost() throws IOException {
SimpleUrlRepository repository =
new SimpleUrlRepository(
"test", URI.create("http://10.0.0.5/model.tar.gz"), "model.tar.gz");
Path tmp = Files.createTempDirectory("djl-simpleurl-probe");
try {
Artifact.Item item = new Artifact.Item();
item.setName("data");
item.setUri("http://10.0.0.5/model.tar.gz");
IOException e =
Assert.expectThrows(
IOException.class,
() -> repository.download(tmp, URI.create(""), item, null));
Assert.assertTrue(
e.getMessage().contains("non-public"), "unexpected: " + e.getMessage());
} finally {
Files.deleteIfExists(tmp);
}
}
}
Loading
Loading