HTML/CSV export corrupts UTF-8 characters outside of Basic Multilingual Pane (BMP) ie code point >10000

[Yuutakasan]

OpenRefine 2.7 rc2

After reading UTF 8 file and executing export as UTF 8 file, garbled characters occurred.

displayed characters

有限会社なべ茶屋あさ𡌛

Exported garbled characters

有限会社なべ茶屋あさ����

other garbled export charactor sample
𣘺𣳾

[thadguidry]

@Yuutakasan hmm, that's weird. What is interesting is that it is showing 4 bytes (4 question marks) to hold just 1 character. I see that the code for that last character is actually 6 bytes however (which is the maximum that UTF-8 can hold per character.

𡌛 = \x0A\xF0\xA1\x8C\x9B\x0A

Further interesting is that when I copy and paste your last character into a single OpenRefine cell, I actually get a different character...

ጛ = \xE1\x8C\x9B

instead of

𡌛 = \xF0\xA1\x8C\x9B

@jackyq2015 Can you debug this ?

[Yuutakasan]

I will attach a sample file for reference.

import file
import.txt
export file
export.txt

There is a sense that this letter is actually used in the name of the corporation registered in Japan.

有限会社なべ茶屋あさ𡌛
株式会社石𣘺組
有限会社𣳾新商事
𣳾幸１合同会社

[thadguidry]

@Yuutakasan When I export your import.txt file... I get

有限会社なべ茶屋あさ𡌛

You are probably not using a viewer like Notepad++ or similar that can show that last character as being \xED\xA1\x84\xED\xBC\x9B ?

But regardless... its a bug somewhere because somehow during export we change the bytes...

from
\xF0\xA1\x8C\x9B

to
\xED\xA1\x84\xED\xBC\x9B

[Yuutakasan]

I currently use EmEditor, I will try using Notpad ++.
EmEditor
https://www.emeditor.com/

[Yuutakasan]

I was able to reproduce the same phenomenon.

[Yuutakasan]

thank you. @thadguidry.
I tried testing with multiple character codes.
It seems that a character with a code point of 10000 or more will be garbled.

import txt
import-test-sample.txt

export txt
export-test-sample-txt.txt

[thadguidry]

@Yuutakasan Thanks, we'll have to let @jackyq2015 look into this specifically. My hunch is that we might not actually be storing it correctly in cell and so this https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/exporters/CsvExporter.java#L108 might be giving back the wrong data in the first place. Otherwise its an issue in csvwriter itself here https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/exporters/CsvExporter.java#L114

[jackyq2015]

can you try to add -Dfile.encoding=UTF-8 to java command options to enforce the encoding?

[Yuutakasan]

OK.I will try it!

[Yuutakasan]

@jackyq2015
Can I test by adding "-Dfile.encoding = UTF-8" setting to the openrefine.l4j.ini file?

[thadguidry]

@Yuutakasan Yes, but you can also test it by adding it to the refine.ini file and starting refine.bat or refine.sh if your on Linux. Just uncomment the JAVA_OPTIONS= line

[Yuutakasan]

Since there is no pervert, I was worried whether the setting really worked.
it is executed once, I will share the result later.

[jackyq2015]

@Yuutakasan Given your description, your file is not properly decoded as utf8. that's why I asked you to enforce it. Please note that system cannot 100% accurate to detect the encoding of random stream. There is some library like icu4j can help to improve the accuracy. Actually there is a PR(not merged yet) to introduce it. If you want to wet your hand, you can create your own branch and merge the PR to your own branch and have a try.