edit: datum struct string type added utf8 check

[dust1]

Rationale

Close #1300

Detailed Changes

Check whether it is a utf8 string when inserting data

Test Plan

pass

[dust1]

I forgot, I'll try adding a few more unit tests later

[jiacai2050]

This check may be expensive for long string, better to add an option to decide whether to do this check.

[jiacai2050]

IMO this function should return bool, not a result.

[dust1]

ok

[dust1]

I checked the rust official documentation, and for the way to build Datum objects from String in datum.rs, rust guarantees that String is a utf8 string, which I might need to modify elsewhere. 😢

[jiacai2050]

rust guarantees that String is a utf8 string, which I might need to modify elsewhere. 😢

Yes, I grep the code and find several place contains from_bytes_unchecked(bytes: Bytes).

As for debugging this issue, you can construct a GBK string using SDK, and trace why there is no error for it.

[dust1]

rust guarantees that String is a utf8 string, which I might need to modify elsewhere. 😢

Yes, I grep the code and find several place contains from_bytes_unchecked(bytes: Bytes).

As for debugging this issue, you can construct a GBK string using SDK, and trace why there is no error for it.

Ok, I'll try

[dust1]

The from_bytes_unchecked function will only be called when decoding. I think what I should be looking for is why non-UTF8 characters are saved when encoding. I'll find out later😵