Previous Next        Current Page: NeXtMidas User's Guide / Timelines - Keywords, Updating, Known Issues
Release Information   
Getting Help   
Basic Concepts   
Result Parameters   
XML Support   
System Operation   
Java WebStart   
X-Midas Interoperability   
Third-Party Compatibility   
Installation and Customization   
Running NeXtMidas Independently   
Timelines - Keywords, Updating, Known Issues   
Installing and Using Docker   

Timelines - Keywords, Updating, Known Issues

A data file may have a TimeLine object that supports time adjustments ("time hacks") from the regular time delta specified in the header. This TimeLine can be populated from various forms of timeline keywords (extended header keywords), via PipeResource for incoming pipe data, or directly from a user's primitive.

Topics covered in this page

Enabling CoreIO.IOOptions designed for fixing issues with timeline propagation
The new TimeLine warnings
What they mean and how to address the underlying issues
How to turn the warnings off
Supported timeline keywords and how to convert between them
Modification or deletion of a DataFile's timeline Keywords
Setting or editing a TimeLine via setTimeAt(...) methods
TimeLine tolerance
Normal operation in a pipe

Enabling CoreIO.IOOptions designed for fixing issues with timeline propagation

There are two major known issues with timeline propagation in a pipe.
  1. Pipe restarts (due to things such as fft frame size changes) can cause timeline information to be lost and time to go "back in time"
  2. All keywords are naturally forwarded throughout the pipe. This is not appropriate for timeline keywords.
    • PipeResource already takes care of propagating time via setTimeAt() at each TL
    • Timeline keywords that are based on elements (rather than bytes) need to be adjusted when BPE changes in a primitive
Enabling the CoreIO.IOOptions via Java
Enabling the CoreIO.IOOptions via the macro language
env set IOOptions +FixPipeRestartTimeLines
env set IOOptions +DoNotPropagateTimelineKeywordsInPipe

Note: As of NeXtMidas 3.9.0, these fixes are not enabled by default. They do constitute a change in behavior. We are hoping our major users will vet these changes and they can become default in the near future.

The new TimeLine warnings - added in 3.7.2

What they mean and how to address the underlying issues

Previously bad TimeLine entries could be written (or read in from a file) and there was no warning to the users. A major user requested that warnings be issues, for problems like back-in-time entries.

ERROR: Invalid TimeLine entry that occurs later in the timeline, but is back in time ...
This means that a timeline entry with a later offset has a "back-in-time" time
If this is a file in a pipe and restarts are occuring, this may be caused by the problem in our PipeResource code listed above. Please try enabling the CoreIO.IOOptions.FixPipeRestartTimeLines
Especially if you get an Exception(rather than a WARN: ERROR), this may also be caused by back-in-time issues caused by user code.
ERROR: Adding in a missed TimeLine entry with an earlier offset is currently disabled
This means that there a timeline already exists and the offset specified for this new entry is not or greater than the last offset added
If this is a file in a pipe and there are timeline keywords in the original source file, this is probably caused by the problem in our code where ALL keywords are propagated (including timeline keywords) and then our pipe code tries to create the TimeLine by setting offset/time at each transfer length. Please try enabling the CoreIO.IOOptions.DoNotPropagateTimelineKeywordsInPipe.
If you are indeed trying to manual edit older entries in an existing TimeLine, please enable TimeLine editing (i.e. setEnableEditing(true)).
If you are getting this warning outside of these circumstances, the problem is likely in user code.
Globally turning warnings messages off via Java
Globally turning warnings messages off via the macro language
env set IOOptions +SuppressInvalidTimeLineEntryErrors
Controlling warnings actions via user setTimeAt(...) API

New API allows the user to specify a notification level on the TimeLine warnings

TimeLine.setTimeAt (double index, double dbpe, Time time, NotifyInvalidEntry notifyLevel)

Supported timeline keywords and how to convert between them

Supported Timeline Keywords
Original BLUE microsecond TIMELINE keyword
High Precision BLUE TC Keywords
TCSAMPLE_#, TC_WHOLE_#, TC_FRAC_#, where # is the element (1 based)
High Precision Platinum Keywords
Conversion via CONVERT command
! Read in BLUE TIMELINE file and write out matching file with BLUE TC Keywords
  convert timelinepenny KWCONV tcpenny{USEBLUETCTIMELINE=true} OPTIONS={}

! Read in BLUE TC keywords file and write out matching file with BLUE TIMELINE Keyword
  convert tcpenny KWCONV timelinefromtcpenny{USEBLUETCTIMELINE=false} OPTIONS={}

! Convert PLATINUM to BLUE: convert keywords and do not keep original Platinum keywords
  convert ^plat_file PLAT2BLUE ^blue_file options={TOA_SIGMA={VALUE=0.0},FOA_SIGMA={VALUE=0.0}}
Via Java using file qualifier
DataFile df = new DataFile(getMidasContext(), "mydatafile.tmp{USEBLUETCTIMELINE=true}");

Modification or deletion of a DataFile's timeline Keywords

DataFile API available
deleteTimeLineKeywords() - since NeXtMidas 3.8.0
deletes any timeline keywords, but preserves TimeLine format and tolerance
updateTimeLineFromKeywords() - since NeXtMidas 3.9.1
Should be used directly or indirectly after updating TimeLine Keywords
If a TimeLine exists for a DataFile, Keywords.putAll updates of TimeLine keywords will not update the timeLine and the updated keywords will not be saved in the file, unless this method is called to recreate the TimeLine with the latest keywords
updateKeywordsAndTimeLine(Map map) - since NeXtMidas 3.9.1
effectively the combination of Keywords.putAll(Map) and updateTimeLineFromKeywords ()
Timeline keywords can also be deleted via the KEYWORD TCDEL command
Example update of a DataFile's BLUE TC TimeLine Keywords
    Table kwdsForUpdate = new Table("{D:TCSAMPLE_2=2.0,D:TC_WHOLE_2=2.146032E9,D:TC_FRAC_2=0.9}");
    // note original in file is D:TCSAMPLE_2=3.0,D:TC_WHOLE_2=2.146032001E9,D:TC_FRAC_2=0.228808
    try (DataFile dfTCKeywords = new DataFile(M,tcOutFile)) {

      // Open the file with TC Keywords

      // get this fie's current Keywords object and update some keywords
      Keywords kwTCFile = dfTCKeywords.getKeywordsObject();

      // Since this file had a timeline, we need to update it based on the updated keywords
      // (otherwise, the actual timeline will not get updated and the updates will NOT appear in
      // the file when it is closed)

Setting or editing a TimeLine via setTimeAt(...) methods

Example creating a TimeLine from a Table

Note: An existing TimeLine can be converted to a table via a method like TimeLine.toTable(int decimalPlaces, double bpe), modified, and then used to create a new TimeLine. DataFile.setTimeLineObject(TimeLine tl) can then be used to swap the DataFile's TimeLine.

public static TimeLine timelineFromTable (double delta, int bpe, Table elemTimeTable) {
  TimeLine timeline = new TimeLine(TimeLine.GROW_AS_NEEDED, delta, 0);
  for (String offset : elemTimeTable) {
    int off = Integer.parseInt(offset);
    Time timelineEntry = new Time(elemTimeTable.getString(offset));
    timeline.setTimeAt(off, bpe, timelineEntry);
  return timeline;
Correcting an earlier TimeLine entry
// Note: This entry must still be within range of the current TimeLine entries arrays.
// This should not be an issue, unless the timeline has overflowed.
boolean wasInserted = timeline.setTimeAt(offset, 1,time, NotifyInvalidEntry.WARNING);

TimeLine tolerance

  • Determines whether the current time/offset pair arriving via setTimeAt has enough variance from the expected to be set as a TimeLine entry
  • Default is 0.1 seconds
  • Can be set directly using the method TimeLine.setTolerance(double)
  • Will automatically be set appropriately when high precision timeline keywords are present
  • User code that is handling the setting TimeLine for each frame (or batch of data) and not using a pipe (and indirectly PipeResouces) to handle the setting via setTimeAt methods ...
    • should make sure an appropriate tolerance is set for the TimeLine
    • should call setTimeAt each frame/batch and let TimeLine handle whether a timeline entry is added
    • should NOT do selective calling of setTimeAt based on a separate tolerance in effect in the users code, from what is in affect in TimeLine. This can cause missing or erroneous entries

Normal operation in a pipe

PipeResource handles propagation of the the current time
based on a combination of the most recent adjustment due to a timeline Extended Header Keyword (if applicable) and the current element and time delta
done at each transfer length (if the incoming data has intermediate timeline keyword entries, they will not be propagated)
handles the restart case (but probably needs CoreIO.IOOptions.FixPipeRestartTimeLines set to handle it properly)
Enabling the CoreIO.IOOptions fixes mentioned earlier is recommended

Other Timeline related CoreIO.IOptions

Prefer BLUE TC (Time Code) keywords as outlined in the JICD 4.2 ICD
Prefer Platinum timeline keywords (high precision) over the NeXtMidas TIMELINE keyword (microsecond precision)

Your feedback is appreciated

  • Experiencing issues: If you are experiencing any problems are unexpected warnings from TimeLine and timeline Keywords use, the NeXtMidas team would greatly appreciate your feedback. Please include details about your use case, details on the problem you experiences, and, if possible, instructions on how to replicate the problem.
  • CoreIO.IOOptions help (or at least don't cause any problems): If you have used the CoreIO.IOOptions recommended and they work well, we would like to know this as well. We are hoping to move these fixes to default behavior.
  • Documentation inadequate: Please let us know type of information is missing or is unclear.
  • Unusual use case or just some information you would like to relay: Feedback is always welcome. Understanding our different users' use cases can be invaluable as we try to have NextMidas meet the needs of the community. Thanks!