JEP 412: Pass a on-heap byte array to native code, getting UnsupportedOperationException: Not a native address

Question

All this is with the new JDK 17.

I'm attempting to turn an on-heap byte array into a MemorySegment and pass that to a native function. I created simple sample code that shows this:

        final CLinker cLinker = CLinker.getInstance();
        
        // int strlen(const char *str);
        final Optional<MemoryAddress> oSymbolAddress = CLinker.systemLookup().lookup("strlen");
        
        final MethodHandle mh = cLinker.downcallHandle(oSymbolAddress.get(),
                MethodType.methodType(int.class, MemoryAddress.class),
                FunctionDescriptor.of(C_INT, C_POINTER));        
        
        out.println("I found this method handle: " + mh);
        final byte[] ba = new byte[100];
        ba[0] = 'h';
        ba[1] = 'e';
        ba[2] = 'l';
        ba[3] = 'l';
        ba[4] = 'o';
        ba[5] = 0;
        
        final MemorySegment stringSegment = MemorySegment.ofArray(ba);
        final int result = (Integer) mh.invoke(stringSegment.address());
        
        out.println("The length of the string is: " + result);

It tries to run but it throws:

Exception in thread "main" java.lang.UnsupportedOperationException: Not a native address
    at jdk.incubator.foreign/jdk.internal.foreign.MemoryAddressImpl.toRawLongValue(MemoryAddressImpl.java:91)

If instead of using MemorySegment.ofArray(ba), I use this:

        final MemorySegment stringSegment = MemorySegment.allocateNative(100, newImplicitScope());
        stringSegment.asByteBuffer().put(ba);

it works and gives the expected answer (5).

I looked up this function in MemoryAddressImpl.java and I can see:

    @Override
    public long toRawLongValue() {
        if (segment != null) {
            if (segment.base() != null) {
                throw new UnsupportedOperationException("Not a native address");
            }
            segment.checkValidState();
        }
        return offset();
    }

Clearly segment.base() is returning null.

I don't really understand what's going on. I thought that one of the major advantages of Project Panama would be that native code could access on-heap memory, to avoid having to copy. I certainly can do an allocateNative() and then copy the byte array into that, but that's a copy that I think should be avoided.

Any ideas on this? Am I doing something wrong or misunderstanding how to use on-heap memory?

Answer 1

I thought that one of the major advantages of Project Panama would be that native code could access on-heap memory, to avoid having to copy.

Actually despite the major advance in usability coming with project Panama, this won't be possible, for multiple reasons.

• GC moves things around in the Java heap memory, hence the address of any object (including tables) may/will change over time. However the native code is given a pointer to a memory address which of course won't be updated after a GC cycle (not even mentioning accessing that memory in the middle of a cycle).
• JNI had APIs to actually prevent GC from happening while in the middle of native code through Get*Critical sections. Unfortunately preventing the GC may have significant impact on application performance.

In fact Project Panama is exactly trying to avoid blocking the GC. This is why there's a clear separation of the accessed memory and why it is necessary to copy to/from native memory.

This shouldn't be much of an issue, unless this is hot code path (ie it's called very very often), or the code deals with very big data. In such case the code may want to do most of the work off-heap. If the data is in a file, maybe access that file from native code, or use a panama memory mapped file.

var big = Path.of("path/to/big.data");
try (var scope = ResourceScope.newConfinedScope()) {
  var bigMM = MemorySegment.mapFile(big, 0, Files.size(big), FileChannel.MapMode.READ_ONLY, scope);
  return (int) mh.invoke(bigMM.address());
}