Maximum time limit of 4095 for annotated .vtu file

Upon loading an annotated .vtu file with TimeStep annotations such as this into Paraview 5.2.0 (under 64 bit SUSE Linux):

<DataArray type="Float64" Name="V" TimeStep="10560" format="appended" RangeMin="-87.934331219" RangeMax="-87.443353026" offset="7801035604" />

I see the following in Paraview:

such that a maximum of 4095 time steps are available, as opposed to the expected 10,560 time steps.

The number of time steps can be found in the TimeValues attribute of the UnstructuredGrid element:

<UnstructuredGrid TimeValues="0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 ...

When the non-annotated .vtu file is loaded, the expected number of frames are visible.

There is an inherent limitation within Paraview that prevents more than 4095 time steps from being loaded/visualised.

A colleague found the place in the code where the 4095 limit occurs:

int vtkXMLReader::ReadPrimaryElement(vtkXMLDataElement* ePrimary)
{
  const int tsMax = 4096;
  double timevalues[tsMax];

  int numTimeSteps =
    ePrimary->GetVectorAttribute("TimeValues", tsMax, timevalues);
  assert( numTimeSteps <= tsMax);

  this->SetNumberOfTimeSteps( numTimeSteps );
  ...

Changing 4096 to a larger value (e.g. 65536) allows more time steps to be used within Paraview, but I wondered about a solution without an upper bound on the number of time steps. I spent time looking at the code being called and what was required.

vtkXMLReader::ReadPrimaryElement calls vtkXMLDataElement::GetVectorAttribute() which calls vtkXMLDataElement::vtkXMLDataElementVectorAttributeParse().

In ReadPrimaryElement(), only the number of time steps is required. This corresponds to the number of space-delimited tokens (time step numbers) in the TimeValues attribute. The timevalues double array passed to GetVectorAttribute() is not used for anything within ReadPrimaryElement().

Accordingly, I've modified the code locally to this:

int vtkXMLReader::ReadPrimaryElement(vtkXMLDataElement* ePrimary)
{
  const char* timeValuesAttr = ePrimary->GetAttribute("TimeValues");
  int numTimeSteps =
    ePrimary->SpaceDelimitedTokenCount(timeValuesAttr);

  this->SetNumberOfTimeSteps( numTimeSteps );
  ...

and added vtkXMLDataElement::SpaceDelimitedTokenCount():

int vtkXMLDataElement::SpaceDelimitedTokenCount(const char* str)
{
    const int find_token = 1;
    const int skip_token = 2;
    const int end = 3;

    int state = find_token;
    int count = 0;

    while(str && state != end)
    {
        switch(state)
        {
            case find_token:
                // Iterate until non-space (token or EOLN: 0)
                while(*str == ' ') ++str;
                state = *str != 0 ? skip_token : end;
                break;

            case skip_token:
                ++count;
                // Iterate past token until space or EOLN
                while(*str && *str != ' ') ++str;
                state = *str != 0 ? find_token : end;
                break;

            default:
                state = end;
        }
    }

    return count;
}

This may not be the best class for it. In any case, it works well.

Initially I wrote this in terms of a variant of vtkXMLDataElement::vtkXMLDataElementVectorAttributeParse():

int vtkXMLDataElement::SpaceDelimitedTokenCount(const char* str)
{
    if(!str) { return 0; }

    std::stringstream vstr;
    vstr.imbue(std::locale::classic());
    vstr << str;

    char ch;
    int count = 0;

    while(vstr)
    {
        vstr >> ch;
        count++;
    }

    return count-1;
}

I used the following simple test cases (with spaceDelimitedTokenCount2() referring to the second implementation above passed as a non-member function to testTokenCounter()):

void testTokenCounter(int (*f)(const char*))
{
    const char* str1 = "1  2    3 4 5";
    std::cout << f(str1) << std::endl;

    const char* str2 = "1 2 3 4 5";
    std::cout << f(str2) << std::endl;

    const char* str3 = " 1 2 3 4 5 ";
    std::cout << f(str3) << std::endl;

    // Fails with spaceDelimitedTokenCount2()
    const char* str4 = " 1 2 3 4 52 ";
    std::cout << f(str4) << std::endl;

    const char* str5 = "  ";
    std::cout << f(str5) << std::endl;

    const char* str6 = " ";
    std::cout << f(str6) << std::endl;

    const char* str7 = "";
    std::cout << f(str7) << std::endl;

    const char* str8 = 0;
    std::cout << f(str8) << std::endl;

    std::cout << std::endl;
}

and found that the second approach yielded a count of 6 tokens instead of 5. I think the first implementation is less opaque in any case, at the cost of being a little more verbose.

I submit this problem and potential solution for your consideration.

From a git clone of VTK, I see:

$ git rev-parse HEAD c0a653bf5fd488f7ae7bcd24e8321b542f4b5f6b