Possibly faullty data offset counting for Base64 encoded data in AppendData-section

This issue was created automatically from an original Mantis Issue. Further discussion may take place here.

Dear All,

Paraview 13.12.0 (32bit) seems to read VTK AppendedData in a peculiar and possibly faulty way.

According to the documentation of VTK XML format, the data (later the raw data) of each

section is stored in section as follows: It starts with the total length of the data expressed as int32 (MMMM). It is followed by the data from each of the DataArray sections in a continuous manner (AAAA,BBBB,CCCCC..):

_MMMMAAAAAAAAAAAAAAAABBBBCCCCCCCCCCCCCCCCCCCC

The offset expresses the location of each section; N bytes offset from the beginning of the first data section (N=0 ~ 33th bit). Here N_A=0,N_B=16,N_C=20, M=40.

However, if the data is base64 encoded,

_Base64[MMMMAAAAAAAAAAAAAAAABBBBCCCCCCCCCCCCCCCCCCCC]

every byte (8bits) in the encoded string correspond to six bytes of the raw data. Again the offset is based again at the start of the actual data (N=0 is now the 45th bit). However, the offset N correspond to N chars of encoded array, that is 6N bit or 3N/4 bytes of raw data.

As a result the offset does not in general correspond to any byte-boundary of the raw data and never corresponds to a char-boundary of the encoded data.

In addition, to allow reference to the start of a data section, previous data has to be padded to a multiple of 3 bytes; or as float32 corresponds to 4 bytes, the data has to padded to a multiple of 12 bytes (3xfloats32).

Questions:

Is this true for VTK+base64 at large or only paraview
If not for which versions of VTK / paraview
Is this a stable arrangement - should I write my code to adhere?

I attach an example of the bit alignment and a vts-file for testing (without debugger, change the alignment of rho-data and save-data in csv). This is data is for debugging reasons stored as UInt8, but the phenomenon is independent of the data type.

Paraview 13.12.0 (32bit) Linux 3.2.0-58-generic-pae #88-Ubuntu SMP 2013 i686 i686 i386 GNU/Linux

With preciation for any comments.

EXAMPLE OF THE ALIGNMENT

|running bit |C = byte.bit encoded data |D = byte.bit orginal data |comment

---- byte 0 in base64 code ------- 0 0.8
1 0.7
2 0.6 0.8 data length, byte 0 3 0.5 0.7
4 0.4 0.6
5 0.3 0.5
6 0.2 0.4
7 0.1 0.3
---- byte 1 in base64 code ------- 8 1.8
9 1.7
10 1.6 0.2
11 1.5 0.1
12 1.4 1.8 data length, byte 1 13 1.3 1.7
14 1.2 1.6
15 1.1 1.5
---- byte 2 in base64 code ------- 16 2.8
17 2.7
18 2.6 1.4
19 2.5 1.3
20 2.4 1.2
21 2.3 1.1
22 2.2 2.8 data length, byte 2 23 2.1 2.7
---- byte 3 in base64 code ------- 24 3.8
25 3.7
26 3.6 2.6
27 3.5 2.5
28 3.4 2.4
29 3.3 2.3
30 3.2 2.2
31 3.1 2.1
---- byte 4 in base64 code ------- 32 4.8
33 4.7
34 4.6 3.8 data length, byte 3 35 4.5 3.7
36 4.4 3.6
37 4.3 3.5
38 4.2 3.4
39 4.1 3.3
---- byte 5 in base64 code ------- 40 5.8
41 5.7
42 5.6 3.2
43 5.5 3.1
44 5.4 4.8 offset=0; data byte 0; 45 5.3 4.7
46 5.2 4.6
47 5.1 4.5
---- byte 6 in base64 code ------- 48 6.8
49 6.7
50 6.6 4.4
51 6.5 4.3
52 6.4 4.2 offset=1; 53 6.3 4.1
54 6.2 5.8 data byte 1; 55 6.1 5.7
---- byte 7 in base64 code ------- 56 7.8
57 7.7
58 7.6 5.6
59 7.5 5.5
60 7.4 5.4 offset=2; 61 7.3 5.3
62 7.2 5.2
63 7.1 5.1
---- byte 8 in base64 code ------- 64 8.8
65 8.7
66 8.6 6.8 data byte 2; 67 8.5 6.7
68 8.4 6.6 offset=3; 69 8.3 6.5
70 8.2 6.4
71 8.1 6.3
---- byte 9 in base64 code ------- 72 9.8
73 9.7
74 9.6 6.2
75 9.5 6.1
76 9.4 7.8 offset=4; data byte 3; 77 9.3 7.7
78 9.2 7.6
79 9.1 7.5
---- byte 10 in base64 code ------- 80 10.8
81 10.7
82 10.6 7.4
83 10.5 7.3
84 10.4 7.2 offset=5; 85 10.3 7.1
86 10.2 8.8 data byte 4; 87 10.1 8.7
---- byte 11 in base64 code ------- 88 11.8
89 11.7
90 11.6 8.6
91 11.5 8.5
92 11.4 8.4 offset=6; 93 11.3 8.3
94 11.2 8.2
95 11.1 8.1
---- byte 12 in base64 code ------- 96 12.8
97 12.7
98 12.6 9.8 data byte 5; 99 12.5 9.7
100 12.4 9.6 offset=7; 101 12.3 9.5
102 12.2 9.4
103 12.1 9.3
---- byte 13 in base64 code ------- 104 13.8
105 13.7
106 13.6 9.2
107 13.5 9.1
108 13.4 10.8 offset=8; data byte 6; 109 13.3 10.7
110 13.2 10.6
111 13.1 10.5
112 13.0 10.4

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information