Syntax for removing elements from JSON responses

[ameshkov]

We often face issues when we need to modify JSON strings. Usually, we use $replace for that, but it is FAR from ideal.

What we really need is a new modifier similar to $replace that will work with JSON's specifically.

Now, the question is what syntax should we use that can be understood and can be tested/debugged as easy as a regular expression.

I suggest using jq syntax for that: https://stedolan.github.io/jq/

Here's why:

1. jq is written in C under a permissive license and we can use it as a library: https://github.com/stedolan/jq/blob/master/COPYING
2. jq can be used in conjunction with curl to test the jq expressions: https://stedolan.github.io/jq/tutorial/
3. jq expressions can also be tested online: https://jqplay.org/

@sfionov @sxgunchenko any ideas on what syntax of this modifier could be?
@AdguardTeam/filters-maintainers what do you think about this idea?

edit: relevant: AdguardTeam/Scriptlets#183 (comment)

Maybe using jq is not that sane, we'd better find something that has implementations in both JS and C/C++

[ngorskikh]

@ameshkov How about this for a rule spec:

$jq modifier

$jq rules modify the response of a matching request by applying a JQ program to it. They are intended for JSON manipulation.

Syntax

• ||example.org^$jq=program – replace the response with the result of running a JQ program program with the original response as the input.

Due to the way rule parsing works, the characters $ and , must be escaped with \ inside program.
Refer to the JQ 1.6 manual for the syntax and semantics of JQ programs.

Exceptions

Basic URL exceptions shall not disable $jq rules. They can be disabled as described below:

• @@||example.org^$jq – disable all $jq rules for responses from URLs matching ||example.org^.
• @@||example.org^$jq=text – disable all $jq rules with the value of the jq modifier equal to text for responses from URLs matching ||example.org^.
• $jq rules can also be disabled by $document, $content and $urlblock exception rules.

Restrictions

• $jq rules are only allowed in trusted filters.
• $jq rules are not compatible with any other modifiers except $domain, $third-party, $app, $important, $match-case, and $xmlhttprequest.
• $jq rules do not apply if the size of the original response is more than 3 MB.

Notes

• When multiple $jq rules match the same request, they are sorted in lexicographical order,
  the first rule is applied to the original response, and each of the remaining rules is applied
  to the result of applying the previous one.
• The result of applying a $jq rule to a response is all of the outputs of the JQ program
  specified by the rule, separated by a line feed ('\n') character.
• If a JQ program fails to compile, or an error occurs while during execution, the modified
  response contains the error message.

Examples

• ||example.org^$jq=del(.foo) — remove the key "foo" from all responses from example.org.
• ||example.org^$jq={"answer": [2 as \$two|\$two * 20\,2]|add} — replace all responses from example.org with {"answer":42}. Note that , and $ need to be escaped.

[ngorskikh]

@ameshkov For the JS side of things, there are at least two possibilities:

• Use a native JavaScript implementation, e.g.: https://github.com/mwh/jqjs
• Compile JQ to WebAssembly, e.g. like they did here: https://github.com/fiatjaf/jq-web

[sfionov]

I think that we should provide a full list of supported functions, if it is possible. For example, we definitely will not support halt and halt_error :)

[ngorskikh]

@sfionov I didn't want to commit to a specific list until we have the JavaScript implementation. Also, we might remove something during further testing :)

By the way, halt and halt_error are not a problem: they halt the JQ program, not the whole AdGuard :)

[ameshkov]

@ngorskikh

Here's what bothers me about jq, there is no viable javascript implementation, the existing one is heavy as shit.

We need to find a query language that is:

1. Implemented in both C++ and Javascript
2. The implementation is REALLY lightweight (up to a couple KB of code).

Note that on the extension side most likely we'll need to run this code in the content script (i.e. hook JSON.parse) and the code size & performance is really important there.

Maybe instead of that we may agree on a limited subset of syntax and implement it ourselves in JS?

Here's what we need:

1. Selecting an element by it's path (similar to xpath, jq, jsonpath).
2. Filtering elements by the attribute value (=,<,>,contains).

[sfionov]

btw, jq-web wasm code is 860KB (286KB compressed).

Not great, not terrible.

But wasm may have problems with review process.