In SearchOperations.searchForStream, 'maxResults' overwrites 'size' (pageSize)

[hy2850]

Hi, I'm opening this issue to report something likely to be a bug.
When creating a SearchRequest for searchForStream method, maxResults parameter is overwriting size, while it seems like it shouldn't be.

Bug

I set both maxResult and Pageable.size (will refer to pageSize) fields for the query and passes it to SearchOperations.searchForStream.
When requestConverter.searchRequestinside the method creates a SearchRequest, maxResult overwrites pageSize.

In the below code, builder.size is called first with query.getPageable().getPageSize(), followed by another call with query.getMaxResults(), overwriting the result of the first call.

spring-data-elasticsearch/src/main/java/org/springframework/data/elasticsearch/client/elc/RequestConverter.java

Lines 1469 to 1492 in 0e5af90

	if (query.getPageable().isPaged()) {
	builder //
	.from((int) query.getPageable().getOffset()) //
	.size(query.getPageable().getPageSize());
	} else {
	builder.from(0).size(INDEX_MAX_RESULT_WINDOW);
	}
	if (!isEmpty(query.getFields())) {
	var fieldAndFormats = query.getFields().stream().map(field -> FieldAndFormat.of(b -> b.field(field))).toList();
	builder.fields(fieldAndFormats);
	}
	if (!isEmpty(query.getStoredFields())) {
	builder.storedFields(query.getStoredFields());
	}
	if (query.getIndicesOptions() != null) {
	addIndicesOptions(builder, query.getIndicesOptions());
	}
	if (query.isLimiting()) {
	builder.size(query.getMaxResults());
	}

I was stuck at the case where,

• 250,000 data available in the ES index
• Want to fetch at most 150,000 documents in total (out of 250k), using SearchOperations.searchForStream (maxResult 150000)
• Want to fetch 10,000 documents per page (per scroll search request), which fills almost all the response memory buffer (Pageable.ofSize(10000))

result

• Expected : 15 search requests in total, a single search returning a page of 10k documents
• Actual : a single search tries to fetch 150k documents, overflowing response memory buffer and causing below Exception

Caused by: org.springframework.dao.DataAccessResourceFailureException: entity content is too long [169386197] for the configured buffer limit [104857600]; nested exception is java.lang.RuntimeException: entity content is too long [169386197] for the configured buffer limit [104857600]

Possible solution

I think two parameters have totally separate roles, thus one shouldn't overwrite the other.

• pageSize parameter should decide how many documents that I want ES to fetch per page,
• while maxResult parameter decides how many documents I want to fetch in total during the whole stream operation.

👉 Therefore, the actual pageSize should be Min(maxResult, pageSize) if maxResult is set

Changes made : https://github.com/hy2850/spring-data-elasticsearch/commits/%233089-size-and-maxResults/

Further explanation with code for your understanding

SearchOperations.searchForStream returns a CloseableIterator that uses scroll API to search through a large set of data by pages.
Currently, maxResult is solely used to get maxCount for this method.

spring-data-elasticsearch/src/main/java/org/springframework/data/elasticsearch/core/AbstractElasticsearchTemplate.java

Lines 410 to 423 in 0e5af90

	@Override
	public <T> SearchHitsIterator<T> searchForStream(Query query, Class<T> clazz, IndexCoordinates index) {
	Duration scrollTime = query.getScrollTime() != null ? query.getScrollTime() : Duration.ofMinutes(1);
	long scrollTimeInMillis = scrollTime.toMillis();
	// noinspection ConstantConditions
	int maxCount = query.isLimiting() ? query.getMaxResults() : 0;
	return StreamQueries.streamResults( //
	maxCount, //
	searchScrollStart(scrollTimeInMillis, query, clazz, index), //
	scrollId -> searchScrollContinue(scrollId, scrollTimeInMillis, clazz, index), //
	this::searchScrollClear);
	}

The iterator keeps track of how many documents it has fetched so far, in currentCount instance variable.

spring-data-elasticsearch/src/main/java/org/springframework/data/elasticsearch/core/StreamQueries.java

Lines 130 to 137 in 0e5af90

	@Override
	public SearchHit<T> next() {
	if (hasNext()) {
	currentCount.incrementAndGet();
	return currentScrollHits.next();
	}
	throw new NoSuchElementException();
	}

When currentCount reachesmaxCount, the iterator is forced to return false for hasNext, even when there could be more documents available for searching.

spring-data-elasticsearch/src/main/java/org/springframework/data/elasticsearch/core/StreamQueries.java

Lines 107 to 116 in 0e5af90

	@Override
	public boolean hasNext() {
	boolean hasNext = false;
	if (!isClosed && continueScroll && (maxCount <= 0 || currentCount.get() < maxCount)) {
	if (!currentScrollHits.hasNext()) {
	SearchScrollHits<T> nextPage = continueScrollFunction.apply(scrollState.getScrollId());
	currentScrollHits = nextPage.iterator();

So, I reached a conclusion that maxResult should only decide maxCount, not the pageSize, and this is a bug.

[sothawo]

Thanks for reporting this, I will have look, but probably only towards the end of the week/

[sothawo]

This needs to be fixed; I am still checking for the best way to test this.

I don't think that using the min value in case both pagination and maxresults are set is the best way.
I'd rather go with some explicit definition like

the number of data requested from Elasticsearch in one call is:

• the pagination size if it is set
• when no pagination is set, the maxresults, if available
• otherwise max number of counted data

The only case where setting both values makes sense is the searchForStream call where the pagination determines the number of documents requested in one call, and maxresults the total number to return.

[sothawo]

you were right with the min approach, this is needed to handle the default page size and a maxresult of 0.