Merged
Conversation
vik-rant
added
the
mindtrace-datalake
vik-rant
reviewed
Nov 13, 2025
Comment on lines
332
to
352
| async def query_data( | ||
| self, query: list[dict[str, Any]] | dict[str, Any], datums_wanted: int | None = None, transpose: bool = False | ||
| ) -> list[dict[str, Any]] | dict[str, list]: | ||
| """ | ||
| Optimized version of query_data using MongoDB aggregation pipelines. | ||
|
|
||
| This method provides significant performance improvements for common query patterns | ||
| by using MongoDB's native aggregation capabilities instead of multiple round trips. | ||
|
|
||
| Args: | ||
| query: Same syntax as query_data - list of queries or single query | ||
| datums_wanted: Maximum number of results to return | ||
| transpose: Whether to return dict of lists (True) or list of dicts (False) | ||
|
|
||
| Returns: | ||
| Same format as query_data - list of dictionaries or dictionary of lists | ||
|
|
||
| Note: | ||
| This optimized version handles common cases but may fall back to the original | ||
| query_data method for complex scenarios not yet supported. | ||
| """ |
Contributor
There was a problem hiding this comment.
let's please update the docstring. it refers to query_data being the original one when it itself is query_data.
vik-rant
reviewed
Nov 13, 2025
| else: | ||
| pipeline.append({"$limit": datums_wanted}) | ||
|
|
||
| self.logger.info(f"pipeline: {pipeline}") |
Contributor
There was a problem hiding this comment.
maybe this should log as debug? might blow up log volume.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR renames the old, Python-based, presumably slow,
query_datafunction toquery_data_legacy, and replaces it with a mongoDB-native solution using aggregation pipelines.The inputs and outputs of query_data should be unchanged.