Context Navigation

← Previous Change
Ticket History
Next Change →

Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.

Changes between Initial Version and Version 6 of Ticket #11517

Timestamp:: 01/28/14 17:52:43 (10 years ago)
Author:: cxallan
Comment:

Legend:

: Unmodified
: Added
: Removed
: Modified

Ticket #11517
- Property Status changed from new to accepted
- Property Summary changed from Move to Django cache to Implement a Redis-only cache
- Property Priority changed from major to blocker
- Property Milestone changed from Unscheduled to 5.1.0
- Property Owner changed from mhart-x to shart-x

Ticket #11517 – Description

-                      initial
+                      v6
 The current webgateway caching system relies on a custom implementation which has least recently used (LRU; http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used), timeout and size features. Unfortunately the way these features are implemented, in particular the LRU semantics can cause real problems with larger caches and under load. At a minimum a pluggable backend is required.
+The current webgateway caching system relies on a custom implementation which has least recently used (LRU; http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used), timeout and size features. Unfortunately the way these features are implemented, in particular the LRU semantics can cause real problems with larger caches and under load.
 As we have upgraded to Django 1.3.1 the Django cache (https://docs.djangoproject.com/en/1.3/topics/cache/) has had significant increases in maturity. LRU backends with many features are also available such as `memcached` out of the box and `django-redis-cache` (https://github.com/sebleier/django-redis-cache).
+There are advantages to the existing custom implementation in that its hierarchical, directory based nature makes cache invalidation very straight forward and easily understood. If an Image's cache needs to be invalidated due to a rendering settings change one only needs to recursively remove the Image and Thumbnail cache hierarchies for that particular Image  and that particular Experimenter.
+In order to keep the individual cache settings control of timeouts and LRU pool sizes for thumbnails, JSON and images the following cache names (keys of the Django `CACHES` setting dictionary) should be used:
+ * `omero.web.json`
+ * `omero.web.image`
+ * `omero.web.thumbnail`
+An LRU is implemented but scans of that LRU become incredibly expensive as the cache grows both in number of objects (rendered JPEG image data, thumbnails or JSON) and on disk size.
+The *entire* API of the `WebGatewayCache` should be completely untouched after this refactoring and integration tests should be added to cover all public methods. The `omero.web.webgateway_cache` setting will also need to be deprecated (it's already largely completely hidden) and the `omero.web.caches` setting documented further.
+As we have upgraded to Django 1.3.1 in `4.4.x` and 1.6.1 in `5.x` the Django cache (https://docs.djangoproject.com/en/1.3/topics/cache/) has had significant increases in maturity. LRU backends with many features are also available such as `memcached` out of the box and `django-redis-cache` (https://github.com/sebleier/django-redis-cache). Unfortunately the lack of accessibility to complex keys makes sane cache invalidation a real problem. Redis' hierarchical keys for instance are completely inaccessible.
+Enter a Redis-only implementation.
+Such an implementation has distinct advantages:
+* A customisable LRU (http://redis.io/topics/config ; "Configuration Redis as a cache")
+* Hierarchical keys and more complex data types than just string (http://redis.io/topics/data-types)
+* Is easily instrumented with command line tools such as `redis-cli`
+* Has no external dependencies and can be easily compiled on almost all UNIX varients
+* A simple protocol with excellent library support in all our target languages (Java, Python and C++)
+Disadvantages include:
+* No Windows support (a restriction of both the existing system and memcached)
+* Deployment and maintenance of another service
+* Documentation burden
+The current top level hash (http://redis.io/topics/data-types ; "Hashes") key hierarchy is proposed:
+* `t_<client_base>_<image_id>` (thumbnails)
+* `i_<client_base>_<image_id>` (rendered JPEG image data)
+* `json_<client_base>` (JSON)
+The *entire* API of the `WebGatewayCache` should be completely untouched after this refactoring and integration tests should be added to cover all public methods. The `omero.web.webgateway_cache` settings will also need to be deprecated (it's already largely completely hidden) and the `omero.web.redis.*` settings documented further.
+Proposed additions to `omeroweb.settings.CUSTOM_SETTINGS_MAPPINGS`:
+* `omero.web.redis.host` (`REDIS_HOST`) default: `localhost`
+* `omero.web.redis.port` (`REDIS_PORT`) default: `6379`
+* `omero.web.redis.db` (`REDIS_DB`) default: `0`
+* `omero.web.redis.timeouts` (`REDIS_DEFAULT_TIMEOUTS`) default: `{}` (no timeout; requires LRU)
+* `omero.web.redis.thumb_hashformat` (`REDIS_THUMB_HASHFORMAT` default: `t_{client_base}_{iid}`
+* `omero.web.redis.image_hashformat` (`REDIS_IMAGE_HASHFORMAT` default: `i_{client_base}_{iid}`