[Java] Add validation functionality

[lidavidm]

Describe the enhancement requested

Other implementations can validate that the data in memory meets all the requirements of the Arrow specification (for example: offsets are nondecreasing for strings/lists; data is UTF-8 for strings). This would also be useful in Java (especially since vectors are mutable and it's a known shortcoming that mutators do not necessarily preserve type invariants).

In C++ this is Validate/ValidateFull.

Component(s)

Java

[jduo]

take

[jduo]

@lidavidm , I see that there is ValueVectorUtility#validate() and ValueVectorUtility#Validate().
Is the ask to add instance methods to the Vector classes to validate()? Or is it to add validity checks that are implemented in C++ but not Java (the UTF-8 check and offset check mentioned above, and others)?

[lidavidm]

Honestly, I never realized that existed. Thanks for finding those. I think we should do both: add instance methods, and then generally make sure that Java validates the same things C++ does.

[jduo]

It seems the definition of validate() and validateFull() differ between C++ and Java.

In C++, validate() is an O(k) operation where k is the number of descendants and validateFull() is an O(k*n) operation where n is the length of each entry (in a string or list vector).

Whereas in Java, ValueVectorUtility#validate() is an O(1) operation that does not do any checks that iterate over the vector's fields and ValueVectorUtility#validateFull() checks offsets for string/binary/lists vectors and it drills down into container vectors. Note that Java's validateFull() currently does not do checks such as ensuring Decimal data is valid or ensuring string data is UTF-8

It seems like Java's validateFull() is somewhat similar to C++'s validate(), and Java's validate() is more like validateShallow().

[lidavidm]

If validateFull is already checking offsets, I think it's appropriate to check string data as well? That said if the only difference is validating decimal/utf8 data then perhaps we can punt on that. (Does it validate union type codes/dense union offsets?)