Add command to generate user summaries #12619
Open
+527
−0
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Attempting to manage process using laravel's concurrency thing ended up being miserable (process crashing for some reason and no log on exceptions) so just have separate commands instead.
Year is hardcoded for now because lazy.
Technically it can use normal job queue but adding three million jobs doesn't sound like a good idea. This one uses single redis list instead.
There are three subcommands ("task"):
Start with
queueto fill in the user ids. If the process interrupted halfway, the queue must be manually cleared withLaravelRedis::del('user_generate_summaries').Re-running the task after initial queue is done will re-add user ids which summary process interrupted halfway. There's no helping if the processing stopped after fetching the id but before creating the model but hopefully it doesn't happen too often if at all (the model creation should be pretty fast).
monitordisplays queue length and the processing speed.processprocesses the queue. This should be run in parallel accordingly. For example using supervisor: