Bug: tables fail on Windows during addData

[jamoore]

The following exception is thrown when connecting to bp.openmicroscopy.org.uk (10.2.1.191) which has a recent develop build deployed:

Josh-Moores-MacBook-Pro:bp moore$ ./win_fail.py 200000,10
Traceback (most recent call last):
  File "./win_fail.py", line 50, in <module>
    table.addData(data)
  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/omero_Tables_ice.py", line 715, in addData
    return _M_omero.grid.Table._op_addData.invoke(self, ((cols, ), _ctx))
InternalException: exception ::omero::InternalException
{
    serverStackTrace = Traceback (most recent call last):
  File "c:\OMERO-CURRENT\lib\python\omero\util\decorators.py", line 61, in exc_handler
    rv = func(*args, **kwargs)
  File "c:\OMERO-CURRENT\lib\python\omero\util\decorators.py", line 28, in handler
    return func(*args, **kwargs)
  File "c:\OMERO-CURRENT\lib\python\omero\tables.py", line 674, in addData
    self.storage.append(cols)
  File "c:\OMERO-CURRENT\lib\python\omero\util\decorators.py", line 81, in with_lock
    return func(*args, **kwargs)
  File "c:\OMERO-CURRENT\lib\python\omero\tables.py", line 378, in append
    self.__mea.append(records)
  File "C:\Python25\lib\site-packages\tables\table.py", line 1820, in append
    self._saveBufferedRows(wbufRA, lenrows)
  File "C:\Python25\lib\site-packages\tables\table.py", line 1826, in _saveBufferedRows
    self._append_records(lenrows)
  File "tableExtension.pyx", line 472, in tables.tableExtension.Table._append_records
HDF5ExtError: Problems appending the records.

    serverExceptionClass = 
    message = Internal exception

The attached script will connect (using the contents of $ICE_CONFIG) and, by default, will try to addData, both binned and unbinned. Once a critical point is passed, (here between 100K and 200K rows) then the failure is thrown regardless of the bins.

$ ./win_fail.py 
Passed on: bin=  1 of   1, count=100000  in   8.2344 secs. 
Passed on: bin= 10 of  10, count=100000  in   8.5244 secs. 
Failed on: bin=  1 of   1, count=200000  in  15.0579 secs. 
Failed on: bin=  7 of  10, count=200000  in  10.9766 secs. 
Failed on: bin= 66 of 100, count=200000  in  16.2557 secs.

Version info (Python 2.5.5):

2013-02-13 20:50:50,608 INFO  [                       omero.util.Server] (MainThread) Loaded dependency numpy (1.3.0)
2013-02-13 20:50:50,608 INFO  [                       omero.util.Server] (MainThread) Loaded dependency tables (2.1.2, hdf=1.6.9)

[jmoore]

The magic file size at which things fail is 1063176:

c:\> dir C:\OMERO\File
...
14/02/2013  13:08         1,063,176 82
14/02/2013  13:09         1,063,176 83
14/02/2013  13:09           815,320 84
14/02/2013  13:10           815,320 85
14/02/2013  13:11           815,320 86
14/02/2013  13:11         1,063,176 87
14/02/2013  13:11           172,152 88
14/02/2013  13:12           815,320 89
14/02/2013  13:12           815,320 90
14/02/2013  13:13         1,063,176 91
14/02/2013  13:13         1,063,176 92
14/02/2013  13:13         1,063,176 93

[jmoore]

I intended to test a newer PyTables? on the same machine, but unfortunately, the box above (bp) doesn't have VS installed. So, I'm now using 10.2.1.194. To install, I opened up the VS console as admin and did:

• python virtualenv.py venv
• pip install {numpy, numexpr, cython}
• pip install tables
• change the "python" setting in windefault.xml to venv\scripts\python

The result was:

Josh-Moores-MacBook-Pro:bp moore$ ./win_fail.py 
Passed on: bin=  1 of   1, count=100000  in   8.3297 secs. 
Passed on: bin= 10 of  10, count=100000  in   8.8379 secs. 
Passed on: bin=  1 of   1, count=200000  in  14.8944 secs. 
Passed on: bin= 10 of  10, count=200000  in  15.7285 secs. 
Passed on: bin=100 of 100, count=200000  in  23.7521 secs. 

Note: this is Python 2.6. Rolling back the change to use pytables 2.2 I get:

Josh-Moores-MacBook-Pro:bp moore$ ./win_fail.py 
Passed on: bin=  1 of   1, count=100000  in   8.8877 secs. 
Passed on: bin= 10 of  10, count=100000  in   8.7359 secs. 
Passed on: bin=  1 of   1, count=200000  in  15.4650 secs. 
Passed on: bin= 10 of  10, count=200000  in  15.8585 secs. 
Passed on: bin=100 of 100, count=200000  in  24.7168 secs. 

So, the different is either the Python version or the HDF version:

2013-02-14 14:16:28,914 INFO  [                       omero.util.Server] (MainThread) Loaded dependency numpy (1.7.0)
2013-02-14 14:16:28,914 INFO  [                       omero.util.Server] (MainThread) Loaded dependency tables (2.4.0, hdf=1.8.5-patch1)

[jamoore]

See ​https://trello.com/c/K6JDIYpt