Implement a Redis-only cache

[cxallan]

The current webgateway caching system relies on a custom implementation which has least recently used (LRU; ​http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used), timeout and size features. Unfortunately the way these features are implemented, in particular the LRU semantics can cause real problems with larger caches and under load.

There are advantages to the existing custom implementation in that its hierarchical, directory based nature makes cache invalidation very straight forward and easily understood. If an Image's cache needs to be invalidated due to a rendering settings change one only needs to recursively remove the Image and Thumbnail cache hierarchies for that particular Image and that particular Experimenter.

An LRU is implemented but scans of that LRU become incredibly expensive as the cache grows both in number of objects (rendered JPEG image data, thumbnails or JSON) and on disk size.

As we have upgraded to Django 1.3.1 in 4.4.x and 1.6.1 in 5.x the Django cache (​https://docs.djangoproject.com/en/1.3/topics/cache/) has had significant increases in maturity. LRU backends with many features are also available such as memcached out of the box and django-redis-cache (​https://github.com/sebleier/django-redis-cache). Unfortunately the lack of accessibility to complex keys makes sane cache invalidation a real problem. Redis' hierarchical keys for instance are completely inaccessible.

Enter a Redis-only implementation.

Such an implementation has distinct advantages:

• A customisable LRU (​http://redis.io/topics/config ; "Configuration Redis as a cache")
• Hierarchical keys and more complex data types than just string (​http://redis.io/topics/data-types)
• Is easily instrumented with command line tools such as redis-cli
• Has no external dependencies and can be easily compiled on almost all UNIX varients
• A simple protocol with excellent library support in all our target languages (Java, Python and C++)

Disadvantages include:

• No Windows support (a restriction of both the existing system and memcached)
• Deployment and maintenance of another service
• Documentation burden

The current top level hash (​http://redis.io/topics/data-types ; "Hashes") key hierarchy is proposed:

• t_<client_base>_<image_id> (thumbnails)
• i_<client_base>_<image_id> (rendered JPEG image data)
• json_<client_base> (JSON)

The *entire* API of the WebGatewayCache should be completely untouched after this refactoring and integration tests should be added to cover all public methods. The omero.web.webgateway_cache settings will also need to be deprecated (it's already largely completely hidden) and the omero.web.redis.* settings documented further.

Proposed additions to omeroweb.settings.CUSTOM_SETTINGS_MAPPINGS:

• omero.web.redis.host (REDIS_HOST) default: localhost
• omero.web.redis.port (REDIS_PORT) default: 6379
• omero.web.redis.db (REDIS_DB) default: 0
• omero.web.redis.timeouts (REDIS_DEFAULT_TIMEOUTS) default: {} (no timeout; requires LRU)
• omero.web.redis.thumb_hashformat (REDIS_THUMB_HASHFORMAT default: t_{client_base}_{iid}
• omero.web.redis.image_hashformat (REDIS_IMAGE_HASHFORMAT default: i_{client_base}_{iid}

[shart-x]

Work on re-factored code can be found here:

​https://github.com/criswell/openmicroscopy/tree/cache-cleanup

Updating unit tests, will be run with:

./build.py -f components/tools/OmeroWeb/build.xml test

[shart-x]

Due to work in the github repo above, and some discussion (which, unfortunately, did not happen in this ticket), we've went with a direct Redis backend workflow as opposed to Django cache.

Do we want to update this ticket to reflect that, or make a new ticket which is more specific?

[shart-x]

The branch can be here: ​https://github.com/criswell/openmicroscopy/tree/cache-cleanup

I've split the WebGatewayTempFile? into its own file because it still seems to be needed by views.py and yet no longer is related to WebGatewayCacheRedis?. I've also renamed the cache to _Redis in order to signify the Redis requirement. If this isn't desired, it can easily be renamed.

The updated unit tests pass on the new redis-based cache:

(chroots/dev/)[sam@talyn /home/sam/openmicroscopy]
$ ./build.py -f components/tools/OmeroWeb/build.xml test
Buildfile: /home/sam/openmicroscopy/components/tools/OmeroWeb/build.xml
Entering /home/sam/openmicroscopy/components/tools/OmeroWeb...
Entering /home/sam/openmicroscopy/components/tools/OmeroWeb...

python-test:
Entering /home/sam/openmicroscopy/components/tools/OmeroWeb...
Entering /home/sam/openmicroscopy/components/tools/OmeroWeb...
Entering /home/sam/openmicroscopy/components/tools/OmeroWeb...
Entering /home/sam/openmicroscopy/components/tools/OmeroWeb...

python-test-suite:
WARNING: '' not a valid package name; please use only.-separated package names in setup.py
running test
False
running egg_info
writing target/OmeroWeb.egg-info/PKG-INFO
writing top-level names to target/OmeroWeb.egg-info/top_level.txt
writing dependency_links to target/OmeroWeb.egg-info/dependency_links.txt
reading manifest file 'target/OmeroWeb.egg-info/SOURCES.txt'
writing manifest file 'target/OmeroWeb.egg-info/SOURCES.txt'
running build_ext
============================= test session starts ==============================
platform linux2 -- Python 2.7.4 -- pytest-2.4.2 -- /usr/bin/python
collecting ... collected 12 items
test/unit/test_marshal.py:72: ShapeMarshalTest.testInsightPolyLineFloatMarshal PASSED
test/unit/test_marshal.py:63: ShapeMarshalTest.testInsightPolyLineMarshal PASSED
test/unit/test_marshal.py:81: ShapeMarshalTest.testInsightPolygonMarshal PASSED
test/unit/test_marshal.py:45: ShapeMarshalTest.testOmeXmlPolyLineFloatMarshal PASSED
test/unit/test_marshal.py:36: ShapeMarshalTest.testOmeXmlPolyLineMarshal PASSED
test/unit/test_marshal.py:54: ShapeMarshalTest.testOmeXmlPolygonMarshal PASSED
test/unit/test_marshal.py:90: ShapeMarshalTest.testShapeUnrecognisedRoiShapePointsString PASSED
test/unit/test_webgateway_cache.py:33: TestWebGatewayCache.testClear PASSED
test/unit/test_webgateway_cache.py:49: TestWebGatewayCache.testImageCache PASSED
test/unit/test_webgateway_cache.py:92: TestWebGatewayCache.testJsonCache PASSED
test/unit/test_webgateway_cache.py:30: TestWebGatewayCache.testSetThumb PASSED
test/unit/test_webgateway_cache.py:36: TestWebGatewayCache.testThumbCache PASSED
 generated xml file: /home/sam/openmicroscopy/components/tools/OmeroWeb/target/reports/junit-results.xml 
========================== 12 passed in 1.42 seconds ===========================

BUILD SUCCESSFUL
Total time: 2 seconds

[jamoore]

Closing in favor of the card ​https://trello.com/c/6wIMxr3n/54-django-cache-redis