Commit graph

3053 commits

Author SHA1 Message Date
Ankush Menat
8436f775a5
feat(DX): PEP517 compatible build in new apps (#21704)
* chore: remove setup.py requirements.txt boilerplate

* feat(DX): PEP517 compatible builds on new apps

closes https://github.com/frappe/frappe/issues/21612
2023-07-17 12:56:41 +05:30
Ankush Menat
3f142c1dce fix: respect system setting for tracebacks
closes https://github.com/frappe/frappe/issues/19826
2023-07-16 19:18:16 +05:30
Ankush Menat
79532ea0f2 fix: incorrect cache clearing of assets
- Build version wasn't correctly computed since v14 update of build
  system. This makes client side cache useless.
- We clear cache assuming rapid reloads,but opening new tab also does
  that. This makes the cache effectively useless for most users.
2023-07-15 21:59:04 +05:30
Ankush Menat
2264c4cf47 fix: cast port to integer
closes https://github.com/frappe/frappe/issues/21687
2023-07-15 17:46:45 +05:30
David Arnold
2ccab0d625
fix: checkpoint the supported schemes for connectivity (#21576)
* fix: checkpoint the supported schemes for connectivity

This PR implements a gateway + error that clearly hints the operator at
a misconfigured system during runtime.

Particularity, against the multiple library-provided ways of configuring
redis connection strings (in python), this hard stops if an unsupported
one is chosen by accident.

* fix: remove unknown protocol

---------

Co-authored-by: Ankush Menat <ankushmenat@gmail.com>
2023-07-15 11:05:29 +05:30
Ankush Menat
adf30693a9 ci: update pyupgrade 2023-07-14 14:24:08 +05:30
David Arnold
6b2bb9a2ab
fix: add env overrides for service orchestration (#21577)
This avoids having to manipulate config files in brittle bash
entrypoints that need to react to dynamic service discovery.

This significantly improves the operability of various bench sites.
2023-07-14 11:39:01 +05:30
Ritwik Puri
cb156b6f24
Merge pull request #21460 from nabinhait/minor-fix-111
fix: Added option for setting default in Report filters
2023-07-06 23:52:30 +05:30
David Arnold
f5b93cc2a0
chore: urlparse already does the parsing for us, no need to diy (#21558)
This is a very minor cleanup. So minor, that it hopefully is a nobrainer
2023-07-02 21:46:53 +05:30
Gavin D'souza
883445aefa
perf: Defer pydantic imports until function call
Pydantic adds an additional 1-2MB in memory usage. We can defer it in
case an environment doesn't use it at all.
2023-07-02 16:00:50 +05:30
gavin
03fbdde007
Merge branch 'develop' into bump-pydantic-v2 2023-06-30 22:19:09 +05:30
Ankush Menat
441495b561
refactor!: Drop support for currentsite.txt (#21536)
* refactor!: Drop currentsite.txt

- `bench use` will continue to work.
- Instead of txt file use common_site_config to set default site using `default_site` key.
- `FRAPPE_SITE` environment variable also works

* fix(DX): warn if non-empty currentsite.txt is present
2023-06-30 17:57:40 +05:30
Suraj Shetty
793bcdb759
Merge branch 'develop' into minor-fix-111 2023-06-30 14:04:30 +05:30
Ankush Menat
d57c552e26 feat: frappe.enqueue with deduplication
use deduplicate=True and set job_id for automatic and mostly sane job deduplication.
2023-06-29 18:05:07 +05:30
Ankush Menat
1092eef7bd perf: faster pending jobs check 2023-06-29 17:31:06 +05:30
Ankush Menat
1668ba7d9e feat: Namespace all RQ jobs to site 2023-06-29 16:35:10 +05:30
Ankush Menat
95e49193c8
fix: dont retry if redis not available for realtime (#21517)
* fix: reduce retries for rq connection

10 seconds of retries when connection isn't available is too much, just
   failing might be beneficial.

- BusyLoadingError only occurs when redis is restarting
- ConnectionError mostly means redis is dead, no amount of retries will
  bring it back.

* fix: dont retry if redis is down for realtime
2023-06-28 15:21:04 +05:30
Ankush Menat
b9f000e1f9 refactor!: Log 5xx error to error log instead of error snapshot
Also move log_error function to right location
2023-06-28 10:59:19 +05:30
Ankush Menat
73bca16d77 feat: RQ WorkerPool
RQ now has experimental support for workerpools.

When to use this?

Roughly when you have more than 2 workers a workerpool might make
sense, below 2 it's overhead as master "pool" process will need to run
to manager workerpool itself.

Why is it any better?

Currently we just let supervisor duplicate the worker process N number
of times. This is inefficient from shared memory POV. Forking the
original process to create workers enables sharing of more memory thus
leading upwards of 60-70% reduction in memory usage with pool size of 8
workers.
2023-06-26 18:10:00 +05:30
Ankush Menat
7fbc6e8175 refactor: Simplify dequeue_strategy selection
Classes arent required anymore, it can just be a parm to worker class
isntead.
2023-06-26 17:42:18 +05:30
Ankush Menat
ca95b591ae refactor: Pass redis connection directly 2023-06-26 17:36:53 +05:30
Ankush Menat
e26152f0dc chore: use node18 for github workflow
[skip ci]
2023-06-26 13:29:18 +05:30
gavin
faab26ce4f
Merge branch 'develop' into bump-pydantic-v2 2023-06-26 13:12:24 +05:30
Ankush Menat
25c18e486c
Merge pull request #21461 from ruchamahabal/hooks
feat: before/after hooks for any app install/uninstall
2023-06-26 09:50:35 +05:30
Suraj Shetty
3982eeea22 chore: Fix linter warning 2023-06-26 09:15:20 +05:30
Ankush Menat
af03b76c88 perf: Preload and share common python modules 2023-06-24 21:24:02 +05:30
Ankush Menat
793f4ebba3 perf: defer loading JWT 2023-06-24 19:50:08 +05:30
Ankush Menat
150c36c74d fix: collect before freezing 2023-06-24 17:36:10 +05:30
Ankush Menat
29d28a460f perf: Freeze GC right before starting background worker
BG worker forks are not CoW friendly. Freezing right before we start
worker should lessen overall memory usage. Though this isn't useful much
because at max you're sharing with 2 processes - master and horse.

WorkerPool can improve this benefit a lot by forking each worker from
master process and horse from forked processes. TBD when WorkerPool is
out of beta.
2023-06-24 17:06:23 +05:30
Ankush Menat
32bd5d3e2c Revert "perf: defer many requests imports"
This reverts commit 71b44efcac.

This gets frequently imported from one place or another. Since with
gc.freeze we can mostly reuse the import from parent, let's just leave
it here.
2023-06-24 16:26:22 +05:30
Ankush Menat
6a9c9bd89d perf: defer pydoc import 2023-06-24 15:22:01 +05:30
Ankush Menat
45f8aff909 perf: defer LDAP import 2023-06-24 15:22:01 +05:30
Ankush Menat
71b44efcac perf: defer many requests imports 2023-06-24 15:22:01 +05:30
Ankush Menat
8a37d6d278
perf: reduce memory usage of background processes (#21467)
* perf: defer translation.py imports

This indirectly imports babel which isn't really required most of the
time.

* perf: defer gzip import

* perf: move validate_and_sanitize_search_inputs

This causes all sorts of indirect imports and increases memory usage

* perf: defer requests module imports

* perf: defer system settings import

* perf: defer LOG_DOCTYPES import

Causes many indirect imports

* perf: defer update_site_config

* perf: defer notifications import

* perf: remove unused import

* perf: defer safe exec import

* test: memory usage overhead
2023-06-23 12:51:45 +05:30
Ankush Menat
fb11b5a1b0 fix(UX): better error message for Encryption key 2023-06-23 12:40:42 +05:30
barredterra
15396ceb60 fix: add required_apps to hooks boilerplate 2023-06-22 15:48:04 +02:00
barredterra
d044b30522 fix: remove app_version from hooks boilerplate
It's not used by any frappe/erpnext code and linters complain about the
unused import
2023-06-22 15:47:34 +02:00
Nabin Hait
94ccdd628a fix: added get_quarter_ending function in safe_exec 2023-06-22 16:18:52 +05:30
Rucha Mahabal
ba8f97e789 feat: before/after hooks for any app install/uninstall 2023-06-22 16:07:06 +05:30
Ankush Menat
4f70200bd5 chore: track update actions per doctype
[skip ci]
2023-06-22 12:10:43 +05:30
Smit Vora
db6a06d204
fix: make sure number is not zero for bankers_rounding (#21431)
* fix: make sure num is not zero for bankers_rounding

* test: rounding near zero

---------

Co-authored-by: Ankush Menat <ankush@frappe.io>
2023-06-20 12:00:22 +05:30
David Arnold
23dd8e9248
refactor: normalize redis invocation and unblock unix domain socket conn (#21309)
Prior to this change, given tat a unix domain socket string was passed
ad `frappe.conf.redis_queue`, then the url split would have failed.

With this change, the upstream API for `from_url` is invoked
transparently.

passing none values for username and password is inocuous; [proof](d95d8a24ed/redis/connection.py (L604)) and
[proof](d95d8a24ed/redis/connection.py (L594))

The [entrypoint](d95d8a24ed/redis/client.py (L868)) passes these parameters on transparently.
2023-06-13 21:06:06 +05:30
Ankush Menat
3d0888a5d5 refactor: set_value usage 2023-06-13 16:00:43 +05:30
Ankush Menat
3005e66e45 refactor!: Drop previously deprecated code 2023-06-13 16:00:43 +05:30
Ankush Menat
bd60c60d4e refactor!: remove legacy password hashing
It's been 5+ years, no need to add this compatibility layer anymore.
2023-06-13 16:00:43 +05:30
Ankush Menat
cfdbad6653 refactor!: Remove custom script import 2023-06-13 13:13:23 +05:30
Ankush Menat
74cbdbf07f refactor: Deprecate importing of fixturs/custom_scripts 2023-06-13 13:13:23 +05:30
Rucha Mahabal
1bab900e6f fix: skip custom script syncing for missing doctypes 2023-06-12 22:09:08 +05:30
Rucha Mahabal
f7abf3b1cf fix: handle fixture syncing for missing doctypes 2023-06-12 22:07:00 +05:30
Ankush Menat
32f54b6734 chore: remove socketio from boilerplate workflow
[skip ci]
2023-06-12 12:14:05 +05:30