RFE: Automated testing could rsync image data prior to run

[rleigh]

This follows testing-related discussion with josh this morning.

It would be nice if the test data could be rsynced prior to running the tests, and use the local data rather than being I/O limited by NFS (for hudson on necromancer, cls-dev-bf, etc.).

This should be able to check the available space locally, and only rsync when sufficient free space is available. It should therefore be able to fall back on using NFS if it's not possible to run using the local cached copy. It would also be useful to be able to cache the data between tests to prevent the need for repeatedly syncing the same data. Maybe just commonly used data like test_images_good etc.?

[jmoore]

Referencing ticket #8731 has changed sprint.

[mlinkert]

I just tried this with test_per_commit on necromancer against develop:

$ time rsync -avuPL /ome/data_repo/from_skyking/test_per_commit/ /homes/mlinkert-x/test_per_commit/
...
sent 11804395116 bytes  received 48822 bytes  19431183.44 bytes/sec
total size is 11802804005  speedup is 1.00

real	10m7.273s
user	0m57.592s
sys	0m48.623s

$ cd components/test-suite
$ ant -Dtestng.memory=6144m -Dtestng.directory=/homes/mlinkert-x/test_per_commit/ test-automated
...
BUILD SUCCESSFUL
Total time: 17 minutes 23 seconds

$ ant -Dtestng.memory=6144m -Dtestng.directory=/ome/data_repo/from_skyking/test_per_commit/ test-automated
...

BUILD SUCCESSFUL
Total time: 19 minutes 9 seconds

So as far as I can tell, rsyncing in advance is actually going to make the tests slower, not faster.