Access to default methods in @JImplements
#1182
Conversation
|
Will be hard to get this to work as the jar file must be loaded into the base classloader. It may be doable, but will require serious effort. |
|
sorry I cannot estimate if it'd be worth the effort? How common are these "default methods"? |
|
Default methods are a newer feature of Java in which the programmers adds some functionality to an interface which will automatically be available to the implemented classes. They are common in newer code, but most of the Java library classes currently don't use them. They are difficult to access because they didn't exist when JNI was written so there is no "call the default" method and the proxy system works against them. The issue here is that the newer java security methods prevent calling code from arbitrary modules, and our jar is not located in a position of any privilege as we are not in the boot loader. About the only way I can make this happen would be to make my own Jar loader which uses the JNI special under the table class definition method which in theory would get the JPype into a lower level. However, JNI doesn't really have any support for modules or module definition so I have no idea if I can actually get to the level required to make this type of access. Making a jar side loader would require access to gzip and tar access. There may be severe technical issues if I need to load the class files in a particular order. One can expect that many libraries will have default method if they use interfaces at some point. Thus there is some value, but it is a large lift (and given I am underwater until July it won't happen soon.) |
|
Bypassing JVMs security mechanisms does not seem right. If these extended Interfaces do not posses a mechanism in JNI, we should request them upstream. It does not make any sense for them to be inacessible, right? Why are default methods not already handled by the MRO, e.g. calling the parents implementation? Actually it should, should it not? |
|
The problem is more complex than that. The issue you cited is regarding calling dynamics from JNI without the presence of a proxy.
See this posting which is more relevant…
https://stackoverflow.com/questions/37812393/how-to-explicitly-invoke-default-method-from-a-dynamic-proxy
We do see default methods in the MRO and we can try to invoke them. The issue occurs because our methods are a proxy. All methods called on a proxy, the proxy then redirects to Python or back to Java. But because of the way that defaults are implemented that just goes back to the proxy again. For normal interface method this isn’t a problem as there is nothing to call back to so we just throw UnsupportedOperation exception and the guard code prevents the user from doing this when a JProxy was declared. We can’t redirect it back to Java through the JNI because defaults don’t have an invoke method other than the one that routes to the proxy. Java can redirect to the method, but because of the new security model you must be in the same module (or be a privileged class) to access the MethodHandle. Thus the catch 22. There is no security violation to call a default method (it is public) but there is no way to access the method outside of using a secure path.
This means the only way to implement this feature is to place org.jpype in a privileged state such that it can call a public method (and be trusted to check that). This is a typical problem with the patching in of new features without considering the native interface. We faced the same issue when they started checking the module called from, without considering that JNI calls have no module. The correct solution would be to have JNI register itself to have a module presence that use that module to determine the access.
Solutions….
1. Use the JNI loadClass interface to load our jar into the bootstrap loader such that we live at the same level as core libraries. Of course this has the downside that calls through our jar may be able to access private methods introducing security flaws (through the total number of paths is small.).
2. Use the JNI loadClass interface to load only the Proxy code required to make the request. If the call checks for public access and is bullet proof, then it is better than placing our whole jar file at a privileged level.
3. Try to convert our entire infrastructure to a module and then set the module privilege flags when calling the start JVM to elevate our module. Same issue as 1.
4. Abandon old versions of the JDK…. (Please pick me! Pick me!) These problems all stem from poorly thought through patches. They eventually got reported and thus added support methods that solve the issue. The problem is those support methods don’t exist and thus I have to go through very large hoops to make us work in only versions. In any sane world JDK 1.8 would long since have hit the dust bin and only the latest LTS would need to be dealt with. As each is retired we could remove the kludges.
All of these are a lift because it means we can’t insert jar files through the typical DynamicClassLoader (which places us at a lower that typical security level) but instead must use an entirely different loading scheme.
I should note that this patch works perfectly if we simply place the org.jpype.jar on the classpath of jpype.startJVM(). The difference is that when we enter at the classpath we get loaded by the System loader rather than by the Dynamic loader. To have truly elevated privileges we have to load our classes in the bootloader.
|
|
I am still trying to come up with a solution for this. The Java9 security model is the source of the issues. Certain actions require that the code be loaded into the lowest layer of the JVM and be contained in an authorized module. There was no thought as to how the JNI layer interacts as it doesn't have a module nor is formally loaded into the ClassLoader other than by class which invoked it first. A lot of stuff we have yet to be able to support such as calling default methods, calling protected methods/accessing protected members when extending a class are all stuck on this same common element. Without appropriate privileges all calls end up as second class and are denied. I have tried a number of different ways to work around this:
I did some testing on the javaagent option. As far as I can tell noone outside of my group makes use of agents in the Looking deeper I think that agent may solve the problem though in a weird way. You can start multiple agents with the repeated arguments so it isn't like java.class.path. Second it appears that if we simply make org.jpype.jar an agent (even if it doesn't do anything) it will automatically be promoted to a level were we can access reflection freely. The problem with the JVM classpath security model is that assumes that everything must be known at start of JVM and thus the only jars that can be trusted exist at that time. Many jar types that provide services just don't work after the JVM is started. I have not found any API which allows the JNI controlling process which spawned the JVM to control the access privileges. The entire model of the JVM is "one and done" for any and all settings. I am guessing that is either laziness on the part of JVM developers (this is edge case stuff and we have better things to do), neglect (which JNI is clearly suffering from), or some lame attempt at security from back when the JVM was still running in browsers and people were sandbox breaking. Regardless of the case, it gives us the classic "sucks to be you" feeling as seeming reasonable requests just can't be met. It makes me long for that far future in which graalvm or web assembly VM simply provides a language independent platform which Python and Java can play nicely. |
|
@marscher I think this one may be working now. But we need other PR to get test matrix back on line first. |
| supportLib = os.path.join(os.path.dirname(os.path.dirname(__file__)),"org.jpype.jar") | ||
| if not os.path.exists(supportLib): | ||
| raise RuntimeError("Unable to find org.jpype.jar support library at "+supportLib) | ||
| extra_jvm_args += ('-javaagent:'+supportLib,) | ||
|
|
There was a problem hiding this comment.
This code changes the way that the jar gets loaded to place in a privileged enough position to call reflect. Prior to Java 9 this was the norm, but after 9 there was a new module based security policy and org.jpype is not organized into a module.
| static jobject toURL(JPJavaFrame &frame, const string& path) | ||
| { | ||
| // file = new File("org.jpype.jar"); | ||
| jclass fileClass = frame.FindClass("java/io/File"); | ||
| jmethodID newFile = frame.GetMethodID(fileClass, "<init>", "(Ljava/lang/String;)V"); | ||
| jvalue v[3]; | ||
| v[0].l = frame.NewStringUTF(path.c_str()); | ||
| jobject file = frame.NewObjectA(fileClass, newFile, v); | ||
|
|
||
| // url = file.toURI().toURL(); | ||
| jmethodID toURI = frame.GetMethodID(fileClass, "toURI", "()Ljava/net/URI;"); | ||
| jobject uri = frame.CallObjectMethodA(file, toURI, nullptr); | ||
| jclass uriClass = frame.GetObjectClass(uri); | ||
| jmethodID toURL = frame.GetMethodID(uriClass, "toURL", "()Ljava/net/URL;"); | ||
| return frame.CallObjectMethodA(uri, toURL, nullptr); | ||
| } | ||
|
|
There was a problem hiding this comment.
With the Jar loaded as an agent we don't need to worry about loading it late in the process so all this code can be removed.
| JP_RAISE_METHOD_NOT_FOUND(cname); | ||
| return nullptr; | ||
| } | ||
| return missing; |
There was a problem hiding this comment.
The method to fail is moved to the Java code so the corresponding JNI code to trigger has been removed.
| package org.jpype.agent; | ||
|
|
||
| import java.lang.instrument.Instrumentation; | ||
|
|
||
| public class JPypeAgent | ||
| { | ||
| public static void premain(String agentArgs, Instrumentation inst) { | ||
| // This doesn't have to do anything. | ||
| // We just need to be an agent to load elevated privileges | ||
| } | ||
| } |
There was a problem hiding this comment.
We aren't going to do anything with the agent status except use the load path. So we don't need to have a body.
We can use the agent to do things like change byte code or access low level memory statistics for the JVM which may be handy. We can also late load a jar file to the same level which is required for things like DB services.
| return MethodHandles.lookup() | ||
| .unreflectSpecial(method, method.getDeclaringClass()) | ||
| .bindTo(proxy) | ||
| .invokeWithArguments(args); |
There was a problem hiding this comment.
This is the heart of the change as we need to call these privileged methods to invoke a special method. Of course given the method we are trying to invoke is nothing more than a public method implemented in the interface there is absolutely nothing privileged about the call. The issue is simply that the API through which we are accessing a public method can also be used to invoke private methods.
| @@ -35,6 +36,7 @@ public class JPypeProxy implements InvocationHandler | |||
| public long cleanup; | |||
| Class<?>[] interfaces; | |||
| ClassLoader cl = ClassLoader.getSystemClassLoader(); | |||
| public static Object missing = new Object(); | |||
There was a problem hiding this comment.
This static object will be passed back to indicate that Python did not implement the method. There is nothing special about it other than being an object that the user is unlikely to ever return from a valid proxy. The object was left public so that I can pass it during testing to trigger the not implemented logic if needed.
| if file.endswith("manifest.txt"): | ||
| manifest = file | ||
| continue |
There was a problem hiding this comment.
This is another requirement for an agent. We must add contents to the manifest and include it in the jar file.
Default methods are a pain because the proxy to interfaces always redirects and there is no "normal" method of accessing such a method via reflection. The PR uses MethodHandles to perform the redirect. It is no clear if this works in Java 8 though as some of the comments regarding this method indicate that it may be a Java 9 feature. I added test cases to see if this is true. If it is only a Java 9 feature we can try to implement a mixed mode jar file, but that will get more complicated.
Fixes #1181