Passing a method results in rq doing some internal gymnastics to
generate this string. In our case, this function never changes so we can
get rid of those steps
* build: Bump RQ
Latest version simplifies workerpool extension a bit
* fix: Disable RQ's scheduler
It's now enabled by default with no easy way to disable it except
upstream change or overriding the run_worker method. So better to
disable it with custom worker class.
- Single worker adds some extra costs to each worker (importing scheduler)
- Workerpool can import it all once and share
- This way we need not have many different
Don't pass the stringified version - this is what goes to RQ, and the string we construct isn't always "correct"
For example, 87d121f47a/frappe/email/doctype/email_queue/email_queue.py (L735-L736) generates `frappe.email.doctype.email_queue.email_queue.QueueBuilder.send_emails` which will result in `ModuleNotFoundError: No module named 'frappe.email.doctype.email_queue.email_queue.QueueBuilder'; 'frappe.email.doctype.email_queue.email_queue' is not a package`
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
It overrides what we set with information that isn't really useful for us.
Set a more readable method name, and add in some additional job metadata.
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
- Workerpool wasn't using sentry
- Sentry was started after freezing GC which causes sentry imports to
not be shared. Freeze GC after most memory intensive work is done.
* build: add setproctitle as dependency
RQ and other tools use it to automatically set a useful proc title
* ci: print all bench logs after running tests
This can help reveal failures from background jobs etc
* fix: limit job count in RQ failed registry
* chore: remove unnecessary test
This just checks if func is called with right values, which keep
changing as things evolve.
Everything is individually tested now so need for this test.
* fix: reduce retries for rq connection
10 seconds of retries when connection isn't available is too much, just
failing might be beneficial.
- BusyLoadingError only occurs when redis is restarting
- ConnectionError mostly means redis is dead, no amount of retries will
bring it back.
* fix: dont retry if redis is down for realtime
RQ now has experimental support for workerpools.
When to use this?
Roughly when you have more than 2 workers a workerpool might make
sense, below 2 it's overhead as master "pool" process will need to run
to manager workerpool itself.
Why is it any better?
Currently we just let supervisor duplicate the worker process N number
of times. This is inefficient from shared memory POV. Forking the
original process to create workers enables sharing of more memory thus
leading upwards of 60-70% reduction in memory usage with pool size of 8
workers.
BG worker forks are not CoW friendly. Freezing right before we start
worker should lessen overall memory usage. Though this isn't useful much
because at max you're sharing with 2 processes - master and horse.
WorkerPool can improve this benefit a lot by forking each worker from
master process and horse from forked processes. TBD when WorkerPool is
out of beta.
