Commit graph

1557 commits

Author SHA1 Message Date
Ankush Menat
0a244b10fc test: basic tests v2 API 2023-10-16 18:12:53 +05:30
Ankush Menat
9983bf82e6 fix: backward compatible get_site_path
Before: `./sitename`
After:  `sitename`

This used to return `.` before, even though these are paths identical
they aren't REALLY. Tar command strips 2 subpaths so this ended up
breaking file restores.
2023-10-11 17:30:13 +05:30
Ankush Menat
a81fb2a11a
Merge pull request #22509 from netchampfaris/file-backup-path
fix: use relative path in file backup
2023-09-24 10:06:47 +05:30
Faris Ansari
fa88f1a628 fix: resolve path instead of absolute path
#22142 made the change to absolute path but it affected file backups.
Resolve path should probably be enough in most cases.
2023-09-23 11:05:49 +05:30
Ankush Menat
8cb13f922a chore(DX): Annotate all local proxies 2023-09-18 15:36:58 +05:30
Ankush Menat
970a740164
revert: #22308 (#22389)
* Revert "chore: move function to correct file"

This reverts commit ebfdfa283b.

* Revert "refactor!: merge get_site_url into get_url (#22308)"

This reverts commit 2001bc278f.
2023-09-13 14:34:52 +05:30
Ankush Menat
ebfdfa283b chore: move function to correct file 2023-09-11 19:02:34 +05:30
David Arnold
2001bc278f
refactor!: merge get_site_url into get_url (#22308)
* chore: merge get_site_url into get_url

* fix: keep alias with deprecation notice
2023-09-11 15:09:05 +05:30
niraj gautam
db61deef72
fix: Avoid doc caching if for_update is set (#22203)
* refactor: Avoid doc caching if for_update is set

* chore: simplify condition

---------

Co-authored-by: Ankush Menat <ankushmenat@gmail.com>
2023-08-25 19:35:32 +05:30
Ankush Menat
6e0b522ae3 refactor!: Disable server scripts by default
- Move the config to bench level and not site level because, server
  script "threat model" requires consent from a bench owner and not
  individual site.
- While this is a breaking change which people may not like, we believe
  it's essential to improve security model of Frappe.
2023-08-23 14:49:05 +05:30
mergify[bot]
fe820ae8c8
Merge pull request #22142 from barredterra/abs-pymodule-path
feat(DX): return absolute path
2023-08-23 05:40:38 +00:00
Ankush Menat
04a13ca97c
fix: frappe.clear_cache should drop all keys (#22156)
We are dropping some keys which are hardcoded in function.

- this over time might be out of sync with rest of the code.
- other apps cache stuff too.

Best fix: just clear everything.
2023-08-23 11:02:45 +05:30
Sagar Vora
29e5ad4fca fix: set correct document label in case doc is of type str 2023-08-22 15:24:26 +05:30
barredterra
158a6f8140 refactor: use get_app_path in get_module_list 2023-08-21 17:53:47 +02:00
barredterra
8971709655 feat: return absolute site path 2023-08-21 17:53:10 +02:00
barredterra
8cfee4dfe2 feat: return absolute pymodule path 2023-08-21 17:13:35 +02:00
Raffael Meyer
8f7a4f6697
Merge pull request #21873 from blaggacao/dont-exceed-python-module-boundary 2023-08-21 17:10:27 +02:00
Ankush Menat
3531f58b54 refactor: unify safe_exec and safe_eval code
A lot of duplication was present for this code.
2023-08-21 11:59:53 +05:30
barredterra
5944d387be Merge remote-tracking branch 'upstream/develop' into refactor-attach-print 2023-08-20 15:38:05 +02:00
David Arnold
8dabd2bae6
refactor: make the app source accessor explicit
The parent folder of a python module (inside `./site-packages`) can be
`./lib`. Separate code paths that depend on the app source into its own
getter so that patching that function becomes easier in those cases
where the python module is not served from the source checkout.
2023-08-20 01:11:05 +02:00
Ankush Menat
4d5a945861
refactor: reduce duplication in unsafe attr checks (#22033) 2023-08-13 18:23:41 +05:30
barredterra
88c8baa9ee refactor: for append to extend, merge list extend
Replace a for append loop with list extend.
Create the list with values instead of creating
an empty list and extending it with another list.
2023-08-09 13:25:39 +02:00
barredterra
8d6480df1c Merge remote-tracking branch 'upstream/develop' into refactor-attach-print 2023-08-04 00:25:09 +02:00
barredterra
b1a2c31fe8 feat: letterhead param for attach_print 2023-07-27 18:29:22 +02:00
barredterra
44a3d9d69e refactor: attach_print
- Use print_language context manager
- Move filename and ignore_print_permissions closer to usage
2023-07-27 18:07:02 +02:00
Ankush Menat
6e94cd2eb9
fix: Guess most likely exception source (#21827) 2023-07-27 17:30:04 +05:30
gavin
6cb51a536e
fix: re module cache settings - purge & cache size (#21808)
* fix: Remove re cache internals manipulation

* fix: Purge re cache after module loads

Empty cache would work better as we already got our pre-compiled
patterns at the top level of every module. This leaves the cache open
for dynamically generated patterns which are in better need of it. Over
time, workers would converge to this anyway. This change only reduces
the cache hit and eviction effort.

I'd improve this by executing `re.purge` on every module import but
complexity tradeoff lol. I'd prefer if re didn't cache patterns
generated by `re.compile` but I dont see this behaviour or any escape
hatches so this will have to do for now.
2023-07-27 10:40:50 +05:30
David Arnold
a926e64ec9
fix: procure db config from single authority (#21578)
* fix: procure db config from single authority

ensures that configuration is uniformely procured from local.conf
instead of making use of hard to audit multilevel fallback logic

Implementation Note:
- `get_db(host, port, user, password)` was stripped of any optional
  argument and therefrom all errors where fixed.
- All occurances of `grep 'frappe.db.db_'` where changed to
  `frappe.conf.db_`

* fix: revert unnecessary breaking changes
2023-07-24 10:48:29 +05:30
Ankush Menat
adf30693a9 ci: update pyupgrade 2023-07-14 14:24:08 +05:30
Ankush Menat
fefd9ac2e2
fix: connect_replica and read_only should be idempotent (#21674)
* fix: `connect_replica` should be idempotent

Calling `connect_replica` twice ends up forgetting orginal writable
replica completely.

* test: replicas

closes https://github.com/frappe/frappe/issues/21619
2023-07-14 13:56:57 +05:30
David Arnold
6b2bb9a2ab
fix: add env overrides for service orchestration (#21577)
This avoids having to manipulate config files in brittle bash
entrypoints that need to react to dynamic service discovery.

This significantly improves the operability of various bench sites.
2023-07-14 11:39:01 +05:30
Ankush Menat
a63398778e perf: tune gc by default
This is running on several prod site without any noticable problems, so
making it default behaviour.

Opt out by setting env variable `FRAPPE_TUNE_GC=False`
2023-07-02 16:29:19 +05:30
Ankush Menat
b9f000e1f9 refactor!: Log 5xx error to error log instead of error snapshot
Also move log_error function to right location
2023-06-28 10:59:19 +05:30
Ankush Menat
fbe3174914 perf: Bump alloc count to 7,000 for generation 0
This has overall 1-2% CPU usage reduction for little to no costs.
Benefits increase when doing bulk processing with lots of objects.
2023-06-24 17:02:45 +05:30
Ankush Menat
8a37d6d278
perf: reduce memory usage of background processes (#21467)
* perf: defer translation.py imports

This indirectly imports babel which isn't really required most of the
time.

* perf: defer gzip import

* perf: move validate_and_sanitize_search_inputs

This causes all sorts of indirect imports and increases memory usage

* perf: defer requests module imports

* perf: defer system settings import

* perf: defer LOG_DOCTYPES import

Causes many indirect imports

* perf: defer update_site_config

* perf: defer notifications import

* perf: remove unused import

* perf: defer safe exec import

* test: memory usage overhead
2023-06-23 12:51:45 +05:30
Ankush Menat
3005e66e45 refactor!: Drop previously deprecated code 2023-06-13 16:00:43 +05:30
Ankush Menat
fa6dc03cc8
refactor: frappe.cache() usage to frappe.cache (#21282) 2023-06-08 11:47:17 +05:30
Sagar Vora
b6669bb56e
perf: remove localproxy for frappe.cache (#21281)
* perf: remove localproxy for `frappe.cache`

* chore: fix type hints
2023-06-08 10:21:45 +05:30
Ankush Menat
40b3cd82bc
feat!: frappe.cache() -> frappe.cache (#21279)
This is more intuitive and consistent with other things like `frappe.db`.

PS: This is quite likely to break some weird usage which I can't guess right now. Normal usage inside request/job cycles will continue to work as it used to.
2023-06-07 22:47:45 +05:30
Ankush Menat
392a506a76 perf: Cache published web forms 2023-06-04 15:13:39 +05:30
Ankush Menat
7c7c11b454 perf: Cache web view routes
Each call to evaluate if route is web view makes N queries where N = #
of web view doctypes. This entire computation can be definitely cached
for short duration.

- Added cache bursting in WebsiteGenerator doctype updates.
- Added 60 minutes TTL in case cache invalidation wasn't done reliably.
2023-06-04 15:13:39 +05:30
Suraj Shetty
f0df450c6f
Merge pull request #21224 from surajshetty3416/fix-safe-eval 2023-06-04 09:24:10 +05:30
Suraj Shetty
65a2cdcffc fix(safe_eval): Normalize code passed before validating the code 2023-06-04 08:59:08 +05:30
Ankush Menat
4193a251a5 fix: Invalidate cache on rollback too
Steps:
    - Document modified
    - Document refetched from cache
    - Transaction rolled back
    - Cache now contains unmodified changes.
2023-06-03 22:53:10 +05:30
Ankush Menat
98b4693dcf perf: finer cache eviction on db.set_value
Instead of nuking everything, just clear matching prefix
2023-06-03 22:53:10 +05:30
Ankush Menat
356a2587e2 refactor: Use plain keys instead of hashes for caching
- Hashes are supposed to be used for representing complex object, not
  multiple documents of same DocType.
- Redis's auto cache clearing wont clear individual key from hashes even if they
  are rarely used.
- Keys can have expiry.
2023-06-03 22:53:10 +05:30
Ankush Menat
106ff1f1ee fix: move cache clearing away from document
Passing lambda function from inside document object would keep reference
to document alive. This means increasing memeory usage in bulk
processing.

Refer https://github.com/frappe/frappe/pull/17061 for example

This also extends it to db.set_value
2023-06-03 22:53:10 +05:30
Ankush Menat
be1da0dd00 chore: remove duplicate cache clearing 2023-06-03 22:53:10 +05:30
Ankush Menat
54ae0c4a21 refactor: move flush_realtime_log to realtime.py
This doesn't have anything to do with databases
2023-06-03 18:44:19 +05:30
Ankush Menat
680cf73cba fix: link_count
This didn't work correctly, if link_count is present in cache it would
just read and dump it back in.

This has practically never worked correctly.
2023-06-03 18:44:19 +05:30