Inconsistency in Byte Size of JSON Assertions Between Python and ot-node

[br41nlet]

Overview

We've come across an inconsistency in how the byte size of JSON assertions is calculated in the Python and ot-node/dkg.js. This discrepancy has led to an error in verification of the Knowledge Asset (KA) size on the node when trying to create a KA with special characters through the dkg.py.

Details

In the Python version, when encoding a specific character (e.g., \u2022) within a JSON string, I noticed that the resulting byte size of the JSON is calculated as 22. Here's an example code snippet for illustration:

import json

data = {"character": "\u2022"}
number_of_bytes = len(json.dumps(data, separators=(",", ":")).encode("utf-8"))
print(number_of_bytes)  # Outputs: 22

In contrast, using NodeJS, the calculation for the same JSON object yields a different byte size:

const data = { character: '\u2022' };
const numberOfBytes = Buffer.byteLength(JSON.stringify(data));
console.log(numberOfBytes) // Outputs: 19

Expected behaviour

The expected behavior would be a consistent byte size calculation for JSON assertions across both the Python and NodeJS implementations (and all other).

[zsculac]

We can fix this by setting ensure_ascii=False. I will check whether this is still an issue in the latest version of dkg.py

import json

data = {"character": "\u2022"}
number_of_bytes = len(json.dumps(data, separators=(",", ":"), ensure_ascii=False).encode("utf-8"))
print(number_of_bytes)  # Outputs: 19