Bug: Woolz shared object and symlink should not be in git

[rleigh]

​https://github.com/openmicroscopy/bioformats/tree/develop/lib

% ls -l lib/libJWlz*
-rwxr-xr-x  1 rleigh  staff  4553328 29 Oct 13:11 libJWlz-1.4.0.so
lrwxr-xr-x  1 rleigh  staff       16 29 Oct 13:11 libJWlz.so -> libJWlz-1.4.0.so

% file lib/libJWlz*
lib/libJWlz-1.4.0.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
lib/libJWlz.so:       ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

Having the Linux/AMD64 ELF shared library in git isn't ideal. And the symlink is worse--it's not possible to check this out on platforms which don't support symlinks, and it's also not possible to put them in the release artifacts since the same limitations apply.

Testing locally, these files aren't even required for the WlzReader? to function. So long as you have a local installation of the libraries and they are on the library search path, they are loaded without trouble. So they can be provided by the system or hand-built etc., and it all just works fine. We don't need to undertake providing these libraries. Committing compiled blobs like this can never cover all the supported platforms, and is also awful practice, not least of which being a GPL violation to distribute binaries without the corresponding source (Woolz is GPL2+).

Please could these two files be deleted from git entirely?

Thanks,
Roger

[jamoore]

Was this lib a candidate for wrapping in a jar, Melissa?

[mlinkert]

I'm on the fence. On the one hand, it does allow us to pretty easily support the majority of (though not all) users. However, properly wrapping in a jar requires having a separate component for building the wrapped library, which we probably don't want to do in this case.

Any reason we can't take this to Bill and find out what he'd like to see happen?

[rleigh]

Another thing to bear in mind here is the symbol versioning:

% ldd libJWlz-1.4.0.so
./libJWlz-1.4.0.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./libJWlz-1.4.0.so)
	linux-vdso.so.1 =>  (0x00007fffa2fff000)
	libz.so.1 => /lib64/libz.so.1 (0x00007fe75fb61000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fe75f8dd000)
	libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007fe75f6cf000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe75f4b2000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fe75f120000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fe75ef17000)
	/lib64/ld-linux-x86-64.so.2 (0x0000003cefa00000)

% objdump -p libJWlz-1.4.0.so

libJWlz-1.4.0.so:     file format elf64-x86-64

Program Header:
    LOAD off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**21
         filesz 0x00000000002b3b64 memsz 0x00000000002b3b64 flags r-x
    LOAD off    0x00000000002b3c78 vaddr 0x00000000004b3c78 paddr 0x00000000004b3c78 align 2**21
         filesz 0x00000000000064d8 memsz 0x0000000000006fe8 flags rw-
 DYNAMIC off    0x00000000002b5960 vaddr 0x00000000004b5960 paddr 0x00000000004b5960 align 2**3
         filesz 0x0000000000000220 memsz 0x0000000000000220 flags rw-
    NOTE off    0x00000000000001c8 vaddr 0x00000000000001c8 paddr 0x00000000000001c8 align 2**2
         filesz 0x0000000000000024 memsz 0x0000000000000024 flags r--
EH_FRAME off    0x0000000000282df0 vaddr 0x0000000000282df0 paddr 0x0000000000282df0 align 2**2
         filesz 0x0000000000007214 memsz 0x0000000000007214 flags r--
   STACK off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**3
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
   RELRO off    0x00000000002b3c78 vaddr 0x00000000004b3c78 paddr 0x00000000004b3c78 align 2**0
         filesz 0x0000000000002388 memsz 0x0000000000002388 flags r--

Dynamic Section:
  NEEDED               libz.so.1
  NEEDED               libm.so.6
  NEEDED               libgomp.so.1
  NEEDED               libpthread.so.0
  NEEDED               libc.so.6
  SONAME               libJWlz-1.4.0.so
  INIT                 0x000000000003b5e0
  FINI                 0x000000000024ba90
  INIT_ARRAY           0x00000000004b3c78
  INIT_ARRAYSZ         0x0000000000000008
  FINI_ARRAY           0x00000000004b3c80
  FINI_ARRAYSZ         0x0000000000000008
  HASH                 0x00000000000001f0
  GNU_HASH             0x0000000000005008
  STRTAB               0x000000000001b7b0
  SYMTAB               0x000000000000a3c8
  STRSZ                0x000000000000fdec
  SYMENT               0x0000000000000018
  PLTGOT               0x00000000004b6000
  PLTRELSZ             0x000000000000a848
  PLTREL               0x0000000000000007
  JMPREL               0x0000000000030d98
  RELA                 0x000000000002cd60
  RELASZ               0x0000000000004038
  RELAENT              0x0000000000000018
  VERNEED              0x000000000002cca0
  VERNEEDNUM           0x0000000000000004
  VERSYM               0x000000000002b59c
  RELACOUNT            0x000000000000020c

Version References:
  required from libm.so.6:
    0x09691a75 0x00 06 GLIBC_2.2.5
  required from libpthread.so.0:
    0x09691a75 0x00 05 GLIBC_2.2.5
  required from libc.so.6:
    0x0d696917 0x00 09 GLIBC_2.7
    0x06969194 0x00 08 GLIBC_2.14
    0x09691a75 0x00 04 GLIBC_2.2.5
    0x0d696913 0x00 03 GLIBC_2.3
  required from libgomp.so.1:
    0x04262440 0x00 07 OMP_1.0
    0x042628d0 0x00 02 GOMP_1.0

So we have a hard dependency on glibc 2.14.

In general, committing shared libraries/DLLs into git is something I think we should draw a hard line at. It's evil, and if you want to run the WlzReader?, you need to install the supporting libraries. In this specific case, this only works if

• you're running Linux
• you're using an amd64 architecture
• you're using glibc >= 2.14
• and presumably that it's also compatible with the jvm since we use a specific <jni.h> (I guess unlikely, but not impossible)

so already you're limiting the systems it will work on dramatically, even when running "Linux". It won't work on centos6, for example (this is using /opt/hudson/workspace/BIOFORMATS-merge-develop/lib on gretzky2).

Jars are bad enough, but at least they work everywhere. These do not, and can not. Supporting even common cases is difficult, as shown above. We can't escape from the fact that native code requires compiling; but it's not our responsibility to provide the compiled libraries, and we should not try to make it so since it can't *ever* work well. Bundling them in git, without source, is awful.

(Packing them inside jars is just as bad. It's certainly possible to do these things, but I don't think it's sensible.)

[mlinkert]

PR opened to remove libJWlz*.so: ​https://github.com/openmicroscopy/bioformats/pull/786

For now at least, let's leave the compiled libraries out of JWlz-1.4.0.jar; if there's strong enough demand, we could always add it later.