Tab character in a value in a JSON file results in a truncated data import

[ostephens]

When importing JSON to OpenRefine through 'Create Project', if there is a tab inside a value in the JSON file, the import succeeds but only imports data up to the tab character. e.g. JSON:

{ "rows": [ { "Content": "A" }, { "Content": " B" }, { "Content": "C" } ] }
(tab character is before the 'B')

Since tab is an illegal character in this position (should be escaped as \t) it makes sense for the import to fail - but failing silently is a problem.

Not sure what the best solution is - three possibilities:

1. Import should fail completely with an error
2. Import should succeed with the data truncated as currently but with feedback to the user that the truncation has occurred and why
3. Import should succeed with full data, converting illegal character to escaped version

[thadguidry]

I think 3 would be most appropriate given OpenRefine is a data cleansing tool. And we should try to cleanse a bit of that. Folks expect it not to fail for a String value and we can just escape for those cases.

[thadguidry]

Looks like we might be able to just expand on the CASE logic for VALUE_EMBEDDED_OBJECT
https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/importers/JsonImporter.java#L125
using JsonToken
http://fasterxml.github.io/jackson-core/javadoc/2.2.0/index.html?com/fasterxml/jackson/core/JsonParseException.html
or perhaps using JsonParser.Feature methods might do it
http://fasterxml.github.io/jackson-core/javadoc/2.2.0/index.html?com/fasterxml/jackson/core/JsonParseException.html

[cz2h]

Hi, I am now working with this issue and had make some progress.

Just double check if I understand the expected effect correct: file with { "rows": [ { "Content": "A" }, { "Content": " B" }, { " Content": "C" } ] } should be able to get loaded and there are two types of Content, one with a tab and one do not. And there is a tab before value B .

[wetneb]

I am not sure what you mean by "two types of Content": I would expect A, B and C to end up in the same column.

[cz2h]

I see, so unquoted control chars (like \t, \n) should be ignored during the parsing right?

[wetneb]

I would not ignore these unescaped control characters - see the 3 options mentioned by Owen above.

[tfmorris]

This currently does produce an error:

Illegal unquoted character ((CTRL-CHAR, code 9)): has to be escaped using backslash to be included in string value

although it doesn't happen until you make your record selection (ie not in initial preview).

Is there ever a time that we don't want to be permissive in parsing? ie Do we really need an import option for this? It's an invalid JSON file, but we're a data wrangling tool, not a validation tool.