Just built j902 from source on osx, seeking confirmation that broken tests are "ok"

Question

I have been using J for a few weeks and absolutely loving the new way it makes me look at old problems. I installed the ODBC drivers for Osx (10.15.6) but I get pointer violations:

[sean]% jconsole
   load 'dd'
   ddsrc''
jconsole(77340,0x1130efdc0) malloc: *** error for object 0x7f817ddcd1c0: pointer being freed was not allocated
jconsole(77340,0x1130efdc0) malloc: *** set a breakpoint in malloc_error_break to debug
[1]    77340 abort      jconsole
[Seans-iMac:~]
[sean]%

I checked out the source from Github with the serious intent of trying to debug the problem for myself, I followed the instructions in the make.txt file, terse but mostly good; I had issues with the "./cpbin.sh" showing errors messages, and the first time I ran the tests it failed to find the .dylibs so I had to copy those manually

cp ../../bin/darwin/j64/*.dylib .

I then re-ran the tests:

sean@Seans-iMac:~/gitjsource/jsource/jlibrary/bin|master⚡ ⇒ ./jconsole ../../test/tsu.ijs

and his time the only errors were mach_vm_map errors, so a little progress but running RUN ddal gave this output which is superficially troubling!

   RUN ddall
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=281474976710656, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug

I then ran RECHO ddall and got this output:

   RECHO ddall
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g0.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g000.ijs
:
: elided; assume to have passed, then...
:
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g210.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g210a.ijs
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=281474976710656, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g211.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g212.ijs
:
: elided; assume to have passed
:
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/gxco2.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/gxinf.ijs
0

I just want to know if this is --serious-- or can be ignored, I am off to look in failed test script g210a.ijs to check it out...all I wanted to do was try to see why ODBC 'ddsrc' blows chunks!

Answer 1

It certainly looks serious, but the problem's not likely in jsource, but in the FFI code that you can find in addons/data/odbc/odbc.ijs or ddmysql. An excerpt:

select. UNAME
case. 'Linux' do. libodbc=: (0-:PREFER_IODBC){::'libiodbc.so.2';'libodbc.so.2'
case. 'Darwin' do. libodbc=: 'libiodbc.dylib'
case. 'Win' do. libodbc=: 'odbc32.dll'
case. do. libodbc=: 'libodbc.so'
end.
i.0 0
)
sqlallochandle=: (libodbc, ' SQLAllocHandle s s x *x') &cd
sqlbindcol=: (libodbc, ' SQLBindCol s x s s * x *') &cd
sqlbindparameter=: (libodbc, ' SQLBindParameter s x s s s s x s * i *') &cd

The problem could be

1. that you have a library sufficiently different from the library this was coded against (note it doesn't seem to be doing any version checking) that just trying to call into it is destroying memory and breaking j internals
2. that somewhere J is receiving what is erroneously treated as an owned pointer, and it's A) not a freeable pointer, or B) not a pointer freeable by J because the library is using its own allocator
3. that somewhere J is erroneously passing a pointer to the library that the library regards as an owned pointer that it should free, with similar problems, and the error message just deceptively sources the fault to jsource

In any case I suggest reporting this to the general@ forum

Of the g210a.ijs errors, the file itself warns about errors that seem to fit what you're seeing, and which look to be unrelated to the odbc issue.

NB. Some tests fail because memory is obtained in 65536 word increments,
NB. and malloc does not always fail gracefully.
NB. Other tests fail because they try to grab the entire result array
NB. at the outset, and fail with a limit error or an out of memory

Answer 2

The tsu reports of "malloc_error_break" are normal. They indicate a malloc request that was expected to fail so things are OK. We don't know how to turn off those warnings. They do not occur on other platforms.

The same error in ddsrc does indicate a problem. Again it indicates a malloc that does not succeed. Probably because of a bug that gives a bad arg to malloc. This is not a problem in the J engine, but is a problem in the odbc interface.