Performance of resetDefaults() is non-linear

[cxallan]

Over time, calls to resetDefaults() get slower and slower into the 10's of seconds. This makes browsing of a semi-large dataset of thumbnails (>500) in OMERO.insight horrendously slow (hours to fully display all thumbnails). Large numbers of calls to resetDefaults() also increase the memory usage on the server exorbitantly eventually resulting in the server VM spending almost all of its time triaging for objects to deallocate.

[cxallan]

r1581 resolves this.

[cxallan]

Josh, if you can think of a reason why we must have this debug statement I'm all ears. Perhaps it's time to reduce the logging level in some of the modules? I can't see however, how this statement can be useful if it murders the server after about 10 or 20 calls to resetDefaults() though. *shrugs*

[jmoore]

Naturally, no, it's not an issue to take out any single debug statement. However,

• Did you try just reducing the output level in log4j.xml? There should be little or no impact when turned off, since it was protected by if (log.isDebugEnabled())
• What's the actual performance hit look like with debug enabled? Do you really think this is the culprit of resetDefaults() issues?
• Let's not get into the habit of removing all debug statements, 'cause we'll just end up adding them back eventually. :)

In general, however, let's review (or just completely overhaul) the log4j.xml that we ship with the distribution bundle, and perhaps put up a page for anyone who wants to modify their own jboss. I'll add a ticket for that.

[jmoore]

See #726

[cxallan]

Replying to jmoore:

Naturally, no, it's not an issue to take out any single debug statement. However,

• Did you try just reducing the output level in log4j.xml? There should be little or no impact when turned off, since it was protected by if (log.isDebugEnabled())

Yeah, reducing the log level has the same benefit obviously.

• What's the actual performance hit look like with debug enabled? Do you really think this is the culprit of resetDefaults() issues?

It's huge. Basically the server is spending 25% or more of its time doing StringBuffer? operations and for every call the server memory allocation seems quite significant.

• Let's not get into the habit of removing all debug statements, 'cause we'll just end up adding them back eventually. :)

With out a doubt. That said, I don't think we should get into the habit of sticking whatever we feel like into the debug logging level and just assuming that the performance hit is irrelevent.

In general, however, let's review (or just completely overhaul) the log4j.xml that we ship with the distribution bundle, and perhaps put up a page for anyone who wants to modify their own jboss. I'll add a ticket for that.

Sounds perfect. Along with that I'd like to make sure that we do a couple things (which I've added to #726):

• Ensure that there are no e.printStackTrace()'s anywhere (they just make debugging using the server.log useless)
• Remove instances of System.err or System.out from the codebase

[cxallan]

I'd just like to clarify that the StringBuffer operations are just one factor here, I've now got a couple culprits in the RenderingEngine (math operations and lookup table construction) that are also some pretty heavy hitters. However, it definitely seems that the culprit of the non-linearity is the StringBuffer operations.

[cxallan]

r1582 has alleviated a lot of issues with regards to object orientation of math code inside the StatsFactory. Jean-Marie, you'll probably want to review this and see if we can improve it further.

[cxallan]

Going to have a look at this again now that OmeroSessions has been merged in.

[cxallan]

Features for Brian are now available in r2344; the importer should now hold a IRenderingSettings service and call resetDefaults() once the image has been imported successfully.

Passing to Brian.