Skip to content

Conversation

@nanaya
Copy link
Collaborator

@nanaya nanaya commented Dec 24, 2025

Attempting to manage process using laravel's concurrency thing ended up being miserable (process crashing for some reason and no log on exceptions) so just have separate commands instead.

Year is hardcoded for now because lazy.

Technically it can use normal job queue but adding three million jobs doesn't sound like a good idea. This one uses single redis list instead.

There are three subcommands ("task"):

artisan user:generate-summaries --task=queue
artisan user:generate-summaries --task=process
artisan user:generate-summaries --task=monitor

Start with queue to fill in the user ids. If the process interrupted halfway, the queue must be manually cleared with LaravelRedis::del('user_generate_summaries').
Re-running the task after initial queue is done will re-add user ids which summary process interrupted halfway. There's no helping if the processing stopped after fetching the id but before creating the model but hopefully it doesn't happen too often if at all (the model creation should be pretty fast).

monitor displays queue length and the processing speed.

process processes the queue. This should be run in parallel accordingly. For example using supervisor:

[program:user-summary-generate]
process_name = user_summary_generate_%(process_num)s
command=/bin/docker compose exec php /app/docker/development/entrypoint.sh artisan user:generate-summaries --task=process
directory=/home/php/osu-web
numprocs=20
autorestart=true
autostart=true
user=php
stdout_logfile=/var/log/supervisor/%(program_name)s-%(process_num)s.stdout.log
stderr_logfile=/var/log/supervisor/%(program_name)s-%(process_num)s.stderr.log

@nanaya nanaya force-pushed the user-summary-console branch from fcb506a to 7610ce7 Compare December 26, 2025 13:58
@notbakaneko notbakaneko mentioned this pull request Dec 27, 2025
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant