Commit graph

1532 commits

Author SHA1 Message Date
Ankush Menat
6e94cd2eb9
fix: Guess most likely exception source (#21827) 2023-07-27 17:30:04 +05:30
gavin
6cb51a536e
fix: re module cache settings - purge & cache size (#21808)
* fix: Remove re cache internals manipulation

* fix: Purge re cache after module loads

Empty cache would work better as we already got our pre-compiled
patterns at the top level of every module. This leaves the cache open
for dynamically generated patterns which are in better need of it. Over
time, workers would converge to this anyway. This change only reduces
the cache hit and eviction effort.

I'd improve this by executing `re.purge` on every module import but
complexity tradeoff lol. I'd prefer if re didn't cache patterns
generated by `re.compile` but I dont see this behaviour or any escape
hatches so this will have to do for now.
2023-07-27 10:40:50 +05:30
David Arnold
a926e64ec9
fix: procure db config from single authority (#21578)
* fix: procure db config from single authority

ensures that configuration is uniformely procured from local.conf
instead of making use of hard to audit multilevel fallback logic

Implementation Note:
- `get_db(host, port, user, password)` was stripped of any optional
  argument and therefrom all errors where fixed.
- All occurances of `grep 'frappe.db.db_'` where changed to
  `frappe.conf.db_`

* fix: revert unnecessary breaking changes
2023-07-24 10:48:29 +05:30
Ankush Menat
adf30693a9 ci: update pyupgrade 2023-07-14 14:24:08 +05:30
Ankush Menat
fefd9ac2e2
fix: connect_replica and read_only should be idempotent (#21674)
* fix: `connect_replica` should be idempotent

Calling `connect_replica` twice ends up forgetting orginal writable
replica completely.

* test: replicas

closes https://github.com/frappe/frappe/issues/21619
2023-07-14 13:56:57 +05:30
David Arnold
6b2bb9a2ab
fix: add env overrides for service orchestration (#21577)
This avoids having to manipulate config files in brittle bash
entrypoints that need to react to dynamic service discovery.

This significantly improves the operability of various bench sites.
2023-07-14 11:39:01 +05:30
Ankush Menat
a63398778e perf: tune gc by default
This is running on several prod site without any noticable problems, so
making it default behaviour.

Opt out by setting env variable `FRAPPE_TUNE_GC=False`
2023-07-02 16:29:19 +05:30
Ankush Menat
b9f000e1f9 refactor!: Log 5xx error to error log instead of error snapshot
Also move log_error function to right location
2023-06-28 10:59:19 +05:30
Ankush Menat
fbe3174914 perf: Bump alloc count to 7,000 for generation 0
This has overall 1-2% CPU usage reduction for little to no costs.
Benefits increase when doing bulk processing with lots of objects.
2023-06-24 17:02:45 +05:30
Ankush Menat
8a37d6d278
perf: reduce memory usage of background processes (#21467)
* perf: defer translation.py imports

This indirectly imports babel which isn't really required most of the
time.

* perf: defer gzip import

* perf: move validate_and_sanitize_search_inputs

This causes all sorts of indirect imports and increases memory usage

* perf: defer requests module imports

* perf: defer system settings import

* perf: defer LOG_DOCTYPES import

Causes many indirect imports

* perf: defer update_site_config

* perf: defer notifications import

* perf: remove unused import

* perf: defer safe exec import

* test: memory usage overhead
2023-06-23 12:51:45 +05:30
Ankush Menat
3005e66e45 refactor!: Drop previously deprecated code 2023-06-13 16:00:43 +05:30
Ankush Menat
fa6dc03cc8
refactor: frappe.cache() usage to frappe.cache (#21282) 2023-06-08 11:47:17 +05:30
Sagar Vora
b6669bb56e
perf: remove localproxy for frappe.cache (#21281)
* perf: remove localproxy for `frappe.cache`

* chore: fix type hints
2023-06-08 10:21:45 +05:30
Ankush Menat
40b3cd82bc
feat!: frappe.cache() -> frappe.cache (#21279)
This is more intuitive and consistent with other things like `frappe.db`.

PS: This is quite likely to break some weird usage which I can't guess right now. Normal usage inside request/job cycles will continue to work as it used to.
2023-06-07 22:47:45 +05:30
Ankush Menat
392a506a76 perf: Cache published web forms 2023-06-04 15:13:39 +05:30
Ankush Menat
7c7c11b454 perf: Cache web view routes
Each call to evaluate if route is web view makes N queries where N = #
of web view doctypes. This entire computation can be definitely cached
for short duration.

- Added cache bursting in WebsiteGenerator doctype updates.
- Added 60 minutes TTL in case cache invalidation wasn't done reliably.
2023-06-04 15:13:39 +05:30
Suraj Shetty
f0df450c6f
Merge pull request #21224 from surajshetty3416/fix-safe-eval 2023-06-04 09:24:10 +05:30
Suraj Shetty
65a2cdcffc fix(safe_eval): Normalize code passed before validating the code 2023-06-04 08:59:08 +05:30
Ankush Menat
4193a251a5 fix: Invalidate cache on rollback too
Steps:
    - Document modified
    - Document refetched from cache
    - Transaction rolled back
    - Cache now contains unmodified changes.
2023-06-03 22:53:10 +05:30
Ankush Menat
98b4693dcf perf: finer cache eviction on db.set_value
Instead of nuking everything, just clear matching prefix
2023-06-03 22:53:10 +05:30
Ankush Menat
356a2587e2 refactor: Use plain keys instead of hashes for caching
- Hashes are supposed to be used for representing complex object, not
  multiple documents of same DocType.
- Redis's auto cache clearing wont clear individual key from hashes even if they
  are rarely used.
- Keys can have expiry.
2023-06-03 22:53:10 +05:30
Ankush Menat
106ff1f1ee fix: move cache clearing away from document
Passing lambda function from inside document object would keep reference
to document alive. This means increasing memeory usage in bulk
processing.

Refer https://github.com/frappe/frappe/pull/17061 for example

This also extends it to db.set_value
2023-06-03 22:53:10 +05:30
Ankush Menat
be1da0dd00 chore: remove duplicate cache clearing 2023-06-03 22:53:10 +05:30
Ankush Menat
54ae0c4a21 refactor: move flush_realtime_log to realtime.py
This doesn't have anything to do with databases
2023-06-03 18:44:19 +05:30
Ankush Menat
680cf73cba fix: link_count
This didn't work correctly, if link_count is present in cache it would
just read and dump it back in.

This has practically never worked correctly.
2023-06-03 18:44:19 +05:30
Ankush Menat
b3d370a0b1 refactor!: remove rollback_observers
use `frappe.db.after_rollback.add` instead
2023-06-03 18:44:19 +05:30
Ankush Menat
7e9ef00bea refactor!: Remove frappe.db.add_before_commit
Not used anywhere, use `frappe.db.before_commit.add()` instead.
2023-06-03 18:44:19 +05:30
Ankush Menat
4a81d9f8e3
feat!: populate fields from kwargs in frappe.new_doc (#21190)
This makes it similar to `get_doc` API BUT still signifies intent that
it's a "NEW" document.

Minor Breaking Change: positional arguments are now forcefully keyword
arguments. Only seems to be used internally from what I can tell.

https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/frappe/.*+/frappe.new_doc%5C%28.*%3F%2C.*%3F%5C%29/+lang:python+&patternType=regexp&case=yes&sm=0&groupBy=repo
2023-05-31 12:29:31 +05:30
Raphael Krupinski
b66d8e8a40
chore(DX): add type hints to Document, BaseDocument and get_doc (#21060)
* chore: add type hints to Document, BaseDocument and get_doc

* refactor: better type hints

get_doc has multiple ways to use it, added all known ways

---------

Co-authored-by: Raphael Krupinski <10319569-mattesilver@users.noreply.gitlab.com>
Co-authored-by: Ankush Menat <ankush@frappe.io>
2023-05-29 11:01:51 +05:30
Suraj Shetty
a84da0139a
Merge pull request #20810 from gavindsouza/fix-20689 2023-05-04 18:32:16 +05:30
Sagar Vora
53c9869ce3 style: use set.isdisjoint when comparing roles (+ minor refactor) 2023-04-27 17:39:33 +05:30
Gavin D'souza
7d2f9be609 refactor: Simplify only_has_select_perm 2023-04-21 17:40:00 +05:30
Ankush Menat
1e9ede40fa
fix: stale frappe.local (#20695)
* fix: stale `frappe.local`

Co-Authored-By: Aditya Hase <aditya@adityahase.com>

* fix: force re-init in request

To ensure that any one bad request can not completely cause recurring
loop of broken requests due to bad locals, we just force-init locals on
every request.

---------

Co-authored-by: Aditya Hase <aditya@adityahase.com>
2023-04-14 14:19:01 +05:30
Daizy Modi
3db1c1aea0
fix: allowed only POST and PUT methods in rename_doc (#20504) 2023-04-03 15:04:46 +05:30
Suraj Shetty
3eff93ea6e refactor: Only send email_read_tracker_url instead of method 2023-02-28 14:16:46 +05:30
Suraj Shetty
b397ec3858 feat: Track count of views on newsletter emails 2023-02-23 13:45:53 +05:30
Ankush Menat
3f1deeba67
fix: can't sign out due to missing roles (#19905) 2023-02-02 22:53:24 +05:30
Ritwik Puri
fb9ae0823e
Merge pull request #19716 from resilient-tech/simpler-doc-cache
chore!: remove special local cache for documents (and related `cache_locally` parameter)
2023-01-25 22:21:12 +05:30
Gavin D'souza
d357af1533 refactor: Add a maxsplit limit to string splits 2023-01-24 19:22:51 +05:30
Sagar Vora
23c9d8a42d chore: remove old cache reference 2023-01-21 17:46:41 +05:30
Sagar Vora
30941c49f5 chore!: remove special local cache for documents 2023-01-21 17:28:42 +05:30
Ankush Menat
0449f851c7
fix: correct exit code on missing app failure (#19676) 2023-01-20 13:51:00 +05:30
Richard Case
e80c8ac862
fix: improved installed app fail feedback (#19668) 2023-01-20 13:02:09 +05:30
Ankush Menat
fa55793a52
Merge pull request #19405 from netchampfaris/refactor-qb-engine
refactor: qb.engine
2023-01-18 21:17:17 +05:30
Ankush Menat
4bb5b10c7f
fix(DX): better error msg for non-whitelisted methods (#19616)
[skip ci]
2023-01-17 12:06:39 +05:30
Ankush Menat
5e2bbf834f refactor: filter out apps not installed on bench 2023-01-16 13:54:40 +05:30
Sagar Vora
f5cbcec103 fix: defer local.all_apps loading 2023-01-16 13:54:27 +05:30
Sagar Vora
d8b7bc18d7 refactor!: deprecate sorting based on apps.txt in get_installed_apps 2023-01-16 13:54:27 +05:30
jiangying
0898ac2a0e
docs: fix docstring for get_list filters (#19569)
* chore: fix doc in frappe/__init__.py

should be filter as a list of dicts

* Update __init__.py

* doc: remove `filter as a list of dicts`
2023-01-11 17:16:19 +05:30
Faris Ansari
be654eaa60
Merge branch 'develop' into refactor-qb-engine 2023-01-09 15:23:20 +05:30