Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #11293 (closed)

Opened 6 years ago

Closed 5 years ago

Last modified 5 years ago

Disallow unparsable timestamp

Reported by: omero-qa Owned by: ajpatterson
Priority: major Milestone: 5.1.0-m1
Component: Specification Version: 5.0.0-beta1
Keywords: n.a. Cc: m.porter@…, jamoore, rleigh
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: n.a.

Description (last modified by ajpatterson)

Create a new OME timestamp type that is a tighter restriction on the standard XSD timestamp. This will limit the object to what the code libraries we have available can actually process. Mostly this is limiting seconds to three decimal places.

The acceptable formats will follow these patterns

1111-11-11T11:11:11.111
1111-11-11T11:11:11.111Z
1111-11-11T11:11:11.111+11:11
1111-11-11T11:11:11.111-11:11
i.e.
YYYY-MM-DDThh:mm:ss.sss+HH:zz
Year
Month
Day
hour
minute 
second (with up to three decimal places)
Hour of time zone
z - minute of time zone

Was: Bug: Model supports only second resolution

https://www.openmicroscopy.org/qa2/qa2/qa/feedback/7443/

Comment: The little coloured squares at the bottom of an image reporting the acquisition time for each channel don't work for images less than one second. So in a 3 channel image I only get 2 values (for green and red) for the first 6 z-sections with the last value (blue) appearing after that. I guess this is because of the values being rounded down to the nearest second, but it does look a bit odd.

AcquisitionTime? and TimeIncrement? have only second resolution; it's not possible for higher resolution timing to be stored at present.

Change History (9)

comment:1 Changed 6 years ago by rleigh

  • Component changed from from QA to Model

comment:2 Changed 6 years ago by ajpatterson

Schema does not need changed.

xsd:dateTime already has arbitrary precision for seconds.

see: http://www.w3.org/TR/xmlschema-2/#isoformats

Functions in the OME Model API will need updated.

Display code in OMERO.insight will need updated.

comment:3 Changed 6 years ago by ajpatterson

  • Owner changed from ajpatterson to rleigh

comment:4 Changed 6 years ago by jamoore

  • Component changed from Model to Specification
  • Milestone changed from Unscheduled to OMERO-5
  • Owner changed from rleigh to ajpatterson
  • Priority changed from minor to major
  • Version changed from 4.4.8 to 5.0.0-beta1

This isn't a problem with the specification per-se, but there are no libraries for supporting all of the variants:

  • In 4.4, we could put in a documentation change, not necessarily.
  • In 5.x, we should disallow timestamps that aren't parse-able in our coding languages.

comment:5 Changed 6 years ago by ajpatterson

  • Cc jamoore rleigh added
  • Description modified (diff)
  • Status changed from new to accepted
  • Summary changed from Bug: Model supports only second resolution to Disallow unparsable timestamp

comment:6 Changed 6 years ago by rleigh

Just to document what's parseable at present, this is what Bio-Formats is currently using (from ome-xml Timestamp.java):

Input:
  /** ISO 8601 date input formatter. */
  DateTimeFormatter ISO8601_PARSER =   ISODateTimeFormat.dateTimeParser().withZone(DateTimeZone.UTC);

Output:
  /** ISO 8601 date output format with milliseconds. */
  String ISO8601_FORMAT_MS = "yyyy-MM-dd'T'HH:mm:ss.SSS";

  /** ISO 8601 date output format without milliseconds. */
  String ISO8601_FORMAT_S = "yyyy-MM-dd'T'HH:mm:ss";

  /** ISO 8601 date output formatter with milliseconds. */
  DateTimeFormatter ISO8601_FORMATTER_MS = DateTimeFormat.forPattern(ISO8601_FORMAT_MS);

  /** ISO 8601 date output formatter without milliseconds. */
  DateTimeFormatter ISO8601_FORMATTER_S = DateTimeFormat.forPattern(ISO8601_FORMAT_S);

For input, we nominally accept any ISO8601 date. However, the reality is that while this is flexible, it's only supporting a fraction of the total spec and with limited precision. It /should/ support xsd:dateTime fairly comprehensively. However, precision is limited to milliseconds internally. It should handle input with greater than ms precision (untested), but will lose the extra precision.

For output, if the date has a millisecond fraction which is not .000s we output in ISO8601 with ms precision, or else we output ISO8601 without a subsecond part. The degrading to second resolution is to keep output smaller and compatible with existing data where the millisecond component is not used.

So for java the documented input and output is the format string "yyyy-MM-dd'T'HH:mm:ss.SSS" with some tolerance for variations on the input side and degrading to second resolution on output where possible. However, other languages won't necessarily have the ability to handle such flexible input (and the java code didn't until last week--the input was the second-accuracy format string with no timezone), so using the restricted format here in the specification means that implementations get guaranteed parseable input and it gives us the option to relax it and add e.g. more precision in a future update. With this restricted format, it should be parseable by hand in any language, or using library facilities where available. NB. we aren't specifying an internal format, so this can be platform/language specific so long as it's lossless.

Regarding negative dates, this is probably acceptable, so long as the implementation can accept it. On UNIX time_t is signed, so can store dates prior to 1970. On 64-bit systems this can go back a long way (note: date(1) lacks -d on the Mac, works on Linux):

$ date +%s -d "Mon Dec 26 09:46:24 1900"
-2177936016

$ date +%s -d "Mon Dec 26 09:46:24 0000"
-62136079941

$ date -d @-2438373434543
Wed Jan  9 05:56:22 LMT -75299

However, this does /not/ mean that the year parsing can handle negative dates (this one does not directly even though I can enter the time manually and get a negative year), or that this works across different platforms and languages. In this case this would only work on a 64-bit system with library support and time database to handle it.

In short, I think we /could/ support it, but this support would never be universal, and would break some implementations. So long as the limitations are clearly documented, I don't think it's unreasonable to allow it--it's not like this is going to be seen except in very special cases. But we probably do want tests for it to make sure it's working correctly; your examples look pretty good for that.

comment:7 Changed 6 years ago by ajpatterson

Referencing ticket #10268 has changed sprint.

comment:8 Changed 5 years ago by ajpatterson

  • Resolution set to wontfix
  • Status changed from accepted to closed

Only schema changes were to add more samples, see: https://trello.com/c/P8Ob73SO/19-timestamp

comment:9 Changed 5 years ago by ajpatterson

  • Milestone changed from 5.x to 5.1.0-m1
Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.78347 sec.)

We're Hiring!