Incompatibility with Apache Http Client 5.6

[wilkinsona]

Java API client version

9.2.3

Java version

17

Elasticsearch Version

N/A

Problem description

Apache Http Client 5.6 has changed how it handles the Content-Encoding header. It now leaves it in place on the HTTP response. This leads to a double decoding attempt which fails:

Caused by: java.lang.RuntimeException: node: http://localhost:49627/, status: 200, [es/indices.create] Failed to decode response
	at org.springframework.data.elasticsearch.client.elc.ElasticsearchExceptionTranslator.translateException(ElasticsearchExceptionTranslator.java:69)
	... 39 common frames omitted
Caused by: co.elastic.clients.transport.TransportException: node: http://localhost:49627/, status: 200, [es/indices.create] Failed to decode response
	at co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse(ElasticsearchTransportBase.java:435)
	at co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse(ElasticsearchTransportBase.java:391)
	at co.elastic.clients.transport.ElasticsearchTransportBase.lambda$performRequestAsync$0(ElasticsearchTransportBase.java:219)
	... 30 common frames omitted
Caused by: jakarta.json.JsonException: Jackson exception
	at co.elastic.clients.json.jackson.Jackson3Utils.convertException(Jackson3Utils.java:27)
	at co.elastic.clients.json.jackson.Jackson3JsonProvider$ParserFactory.createParser(Jackson3JsonProvider.java:118)
	at co.elastic.clients.json.jackson.Jackson3JsonProvider.createParser(Jackson3JsonProvider.java:93)
	at co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse(ElasticsearchTransportBase.java:428)
	... 32 common frames omitted
Caused by: tools.jackson.core.exc.JacksonIOException: Not in GZIP format
 at [No location information]
	at tools.jackson.core.exc.JacksonIOException.construct(JacksonIOException.java:39)
	at tools.jackson.core.json.ByteSourceJsonBootstrapper._wrapIOFailure(ByteSourceJsonBootstrapper.java:460)
	at tools.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:434)
	at tools.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:143)
	at tools.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:291)
	at tools.jackson.core.json.JsonFactory._createParser(JsonFactory.java:391)
	at tools.jackson.core.base.TextualTSFactory.createParser(TextualTSFactory.java:116)
	at tools.jackson.databind.ObjectMapper.createParser(ObjectMapper.java:624)
	at co.elastic.clients.json.jackson.Jackson3JsonProvider$ParserFactory.createParser(Jackson3JsonProvider.java:116)
	... 34 common frames omitted
Caused by: java.util.zip.ZipException: Not in GZIP format
	at java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165)
	at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
	at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91)
	at org.apache.hc.client5.http.entity.GZIPInputStreamFactory.create(GZIPInputStreamFactory.java:72)
	at org.apache.hc.client5.http.entity.LazyDecompressingInputStream.initWrapper(LazyDecompressingInputStream.java:56)
	at org.apache.hc.client5.http.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:73)
	at tools.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:432)
	... 40 common frames omitted

The problem can be worked around with an HTTP client config callback that disables (de)compression support on the HTTP client but it would be nice if it worked out of the box.

Rather than looking at the headers of the response (which now match what the server sent), I believe the intent is that client-side code should look at the HTTP entity instead. In terms of Rest5Client, that would look something like this:

         HttpEntity entity = httpResponse.getEntity();
         if (entity != null) {
-            Header encoding = null;
-            try {
-                encoding = httpResponse.getHeader(CONTENT_ENCODING);
-            } catch (ProtocolException e) {
-                throw new IOException("Couldn't retrieve content encoding: " + e);
-            }
-            if (encoding != null && "gzip".equals(encoding.getValue())) {
+            if ("gzip".equals(entity.getContentEncoding())) {
                 // Decompress and cleanup response headers
                 httpResponse.setEntity(new GzipDecompressingEntity(entity));

I believe a change along these lines will allow things to work out of the box with 5.6, while also remaining compatible with earlier 5.x releases. @ok2c, if you have a minute, please correct me if I've got this wrong.

[l-trotta]

Thanks a lot for the report @wilkinsona, I'll do some tests, will update ASAP.

[ok2c]

I believe a change along these lines will allow things to work out of the box with 5.6, while also remaining compatible with earlier 5.x releases. @ok2c, if you have a minute, please correct me if I've got this wrong.

@wilkinsona Looks correct to me.

[wilkinsona]

Thanks, @ok2c.

[l-trotta]

Hello @wilkinsona, @ok2c, and thank you for the info. I've updated org.apache.httpcomponents.client5.httpclient5 from 5.4.4 to 5.6 to test this, but it's not working as expected: Content-Encoding in the entity seems to be null, while I still can see the content-encoding: gzip header, here's a snippet from my debugger

Is there any other library I should update, like org.apache.httpcomponents.core5:httpcore5? Or is there anything I should change in the way the request is created? Currently for gzip requests we set the Accept-Encoding: gzip header.

[ok2c]

@l-trotta The HttpEntity above has the correct metadata. The Content-Encoding of the entity is null which is correct because it has been automatically decompressed and its content is no longer encoded.

HttpClient no longer meddles with the headers of the original response as of version 5.6. One can still see if the original response was encoded and with what coding.

[l-trotta]

@ok2c Ah I missed the automatic decompression, thanks for clarifying. Still, I'm not understanding the purpose of the condition:

if ("gzip".equals(entity.getContentEncoding())) {

If an http client 5.6 is used then it won't matter, since decompression has already been taken care of, but if an http client <5.6 is used, the information on gzip will be in the header and not the content-encoding of the entity, so in the end the condition is always false if I'm understanding correctly.

I'd like to handle this by updating the library version to 5.6 in the next minor release of the java client and removing the decompression behavior, would it make sense?

[wilkinsona]

If an http client 5.6 is used then it won't matter, since decompression has already been taken care of

It's possible that someone will have disabled the automatic decompression. In fact, that's what I did in Spring Boot to avoid the double decompression attempt.

but if an http client <5.6 is used, the information on gzip will be in the header and not the content-encoding of the entity

It should be in both the header of the response and the content encoding of the entity with < 5.6.

By checking, as I proposed above, the entity rather than the response header, it should mean that the same code works with:

• 5.6 with automatic decompression enabled
• 5.6 with automatic decompression disabled
• < 5.6

That might then mean that you could ship a 9.2.x version of the client that works with Http Client < 5.6 and 5.6.

[ok2c]

I'd like to handle this by updating the library version to 5.6 in the next minor release of the java client and removing the decompression behavior, would it make sense?

@l-trotta This bit is not necessary but it is completely safe. And as @wilkinsona pointed out, it will also work correctly in case the automatic decompression being disabled (or someone using minimal HttpClient).

            if ("gzip".equals(entity.getContentEncoding())) {
                 // Decompress and cleanup response headers
                 httpResponse.setEntity(new GzipDecompressingEntity(entity));

What is super important that one always uses metadata of the entity, and not the message itself. In that case everything should work with all versions of HttpClient starting with 4.0 and above.