Commit graph

48245 commits

Author SHA1 Message Date
Hussain Nagaria
673269fdcf fix: typo in test controller boilerplate 2024-12-21 13:13:22 +05:30
Ankush Menat
f243aa1942
perf: faster add_to_date (#28843)
* fix(DX): Accept None directly as input to add_to_date

It's already supported, signature is just not updated.

* perf: use efficient date parser
2024-12-21 05:23:34 +00:00
Balamurali M
773bf810af
perf(version): Make get_versions fast for autoincrement doctypes (#28847)
Since docname is varchar, indexes won't work when int as passed as
value.
2024-12-21 10:41:32 +05:30
Akhil Narang
bbf1abf3d3
Merge pull request #28845 from ruthra-kumar/log_memory_usage
feat: log peak memory usage for Prepared reports
2024-12-20 16:24:18 +05:30
ruthra kumar
63a6c8c903 refactor: log in monitor as well 2024-12-20 11:35:41 +05:30
ruthra kumar
29cf8cef8f feat: log peak memory usage for Prepared reports 2024-12-20 10:18:27 +05:30
Ankush Menat
9e9096834f
perf: pretty_date - avoid useless dt->string->dt cycle (#28842) 2024-12-19 14:53:38 +00:00
Ankush Menat
17cc356915
perf: speed up flt by 1.06x and get_system_settings by 1.32x (#28841)
* perf: resolve rounding method once

When rounding method is explcitly specified it's 1.4x faster.

* perf: reorder checks

Bankers rounding is default and most common now

* perf: speedup get_system_settings
2024-12-19 14:38:45 +00:00
Ankush Menat
1c2f8abb4e
perf: speedup get_datetime by ~9.5x (#28840) 2024-12-19 20:01:45 +05:30
Ankush Menat
9419344c76
fix: always print tracebacks (#28838)
* fix: fallback for always printing tracebacks

I don't recall ever hitting "no" to this prompt. It's of no use for me.

Also, this makes automated scripts not really automated.

* revert: prompting for exceptions

Always print full exception
2024-12-19 12:46:11 +00:00
Ankush Menat
9e8ab92371
refactor: move all optimizations and pre/post fork hooks to separate file (#28832)
Now they will truly execute before/after fork = :pinch: few bytes saved!
2024-12-19 16:46:26 +05:30
Akhil Narang
a560ba27e4
Merge pull request #28820 from rutwikhdev/fix-workflow-builder
fix: find workflow by id instead of state name
2024-12-19 16:45:10 +05:30
Akhil Narang
495d21240e
Merge pull request #28834 from akhilnarang/fix-symlink-different-filesystem
fix(build): `os.replace` -> `shutil.move`
2024-12-19 16:39:47 +05:30
Akhil Narang
6c32b79766
fix(build): os.replace -> shutil.move
In some cases, while running in docker, we end up with:

```
[Errno 18] Invalid cross-device link: 'tmp<hash>' -> './assets/frappe'
```

Using `shutil.move` fixes this as it supports different filesystems, `os.replace` doesn't

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2024-12-19 16:12:48 +05:30
Ankush Menat
ae434dabfe
Merge pull request #28825 from frappe/fix-build
fix: esbuild with cached flag
2024-12-18 19:32:11 +05:30
Ankush Menat
de4e037246
Merge pull request #28827 from ankush/default_document_cache_expiry
fix: default document cache expiry + monitor persistence
2024-12-18 19:31:02 +05:30
Ankush Menat
8df9d3acdd fix: protect monitor logs from cache eviction 2024-12-18 19:13:02 +05:30
Hussain Nagaria
e980de2788 fix(esbuild): bug that caused apps to json to not get updated when --using-cached 2024-12-18 18:16:47 +05:30
Ankush Menat
6040145109 fix: Set some expiry for cached documents
IMO 1 cache miss per document is fine. This at least ensure that a
missed-invalidation won't cause a perpetual problem.
2024-12-18 17:26:50 +05:30
Ankush Menat
004990e53e
perf: Make frappe._dict great again (#28824)
* perf: Restore dict's flat overrides

Using `super()` is unnecessary cost. This class is used A LOT. Ref: https://github.com/frappe/frappe/pull/16449/

Please consider performance while adding types, it's almost always possible to achieve good typing without this.

Also `frappe._dict` is almost always used as `dict[Any, Any]` or
`dict[str, Any]`, type annotations are useless here.

* ci: ugh wait for processes to exit
2024-12-18 16:36:31 +05:30
Ankush Menat
7dd15e3613
perf: speedup pickling of document objects (#28823)
* perf: Use latest pickle protocol

* perf: pop flags from cached documents

This is also the right thing to do, things like `doc.flags.for_update`
shouldn't be "cached".
2024-12-18 10:18:04 +00:00
Ankush Menat
9d9193800b
fix: Keep HTTP caches private by default (#28719)
Developers can easily enable `can_cache` without knowing what it
entails. Public cache means proxy can likely cache things without
talking to backend.

Obviously many endpoints which can be cached on client side should
probably not be cached in proxy.

E.g. linked PR to the PR that added this feature suggest caching
notification log for short time... we don't want to leak one user's
cached notification to another user.

I don't buy that developers should know about cache implementation to
ensure it's secure or correct to enable it on certain endpoint. In
addition to that, we have very few mechanisms to burst cache
inside proxy. End user hitting ctrl+shift+r won't do anything if proxy
wants to serve stale response.

We should figure out better way to instruct FW about final cache
control headers than hardcoding it IMO.
2024-12-18 14:57:51 +05:30
Hussain Nagaria
2f8c7c7b0d chore: remove dead code 2024-12-18 12:47:19 +05:30
Ankush Menat
23b5b0c7ae
perf: speedup QB field sanitization (#28818) 2024-12-18 05:47:02 +00:00
Sumit Bhanushali
051cedb860
chore: update readme (#28819) 2024-12-17 18:35:10 +00:00
rutwikhdev
8aa292968c fix: find workflow by id instead of state name 2024-12-17 22:08:35 +05:30
Ankush Menat
ea49c2500c
perf: speedup @redis_cache (#28813)
* perf: reduce one redis call for redis_cache

* fix: Always set value in local even if it expires
2024-12-17 18:49:07 +05:30
Ankush Menat
3297aff294 fix: Always set value in local even if it expires 2024-12-17 18:34:45 +05:30
Ankush Menat
b5218096e1 perf: reduce one redis call for redis_cache 2024-12-17 18:04:43 +05:30
Ankush Menat
30ec033747
perf: Speedup get_doc by another ~1.5x (#28807)
* perf: Reduce impact of forced cache replacement on every doc access

* fix: clear cache before processing users

Note: This is just an artifact of testing model, this won't have any real
effect on execution in real system.

Basically `enqueue_on_commit` is not respected in tests and it can't be
practically supported either.
2024-12-17 16:48:43 +05:30
Akhil Narang
b861a014ac
Merge pull request #28804 from RitvikSardana/csv-support
fix: add support for CSV files for non desk users
2024-12-17 16:47:16 +05:30
Akhil Narang
1dcf04a681
Merge pull request #28810 from akhilnarang/email-receive-imaplib-flags-string
fix(email): don't try to parse flags that seem to be invalid
2024-12-17 16:21:15 +05:30
Akhil Narang
d285dd5cbf
fix(email): don't try to parse flags that seem to be invalid
Sometimes we get invalid flags from `imap.uid()`, like `[b'System Error (Failure)']`

This leads to the flag getting parsed as 83 (`ord('S')`)

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2024-12-17 16:04:05 +05:30
Ankush Menat
d0c314090c
perf: speed up recurring redis cache accesses (#28805) 2024-12-17 14:06:01 +05:30
RitvikSardana
6e524ada8f chore: code cleanup 2024-12-17 13:11:57 +05:30
RitvikSardana
f6637d7ef0 fix: change error message 2024-12-17 13:10:27 +05:30
RitvikSardana
15f64fa7bc fix: add support for CSV files for non desk users 2024-12-17 13:06:21 +05:30
Ankush Menat
c1c4a7dd48
perf: speed up @request_cache by ~2x (#28803)
- Eagerly initialize request_cache, all requests use it, so what is the
  point of doing it lazily?
- Reduce accesses to `frappe.local` namespace, get cache once and reuse
  it in rest of the execution.

Before: 1250ns +/- 1%
After: 645ns +/- 1%

Source: Trust me bro.
(no really, for now just trust me or look at the diff)
2024-12-17 07:35:38 +00:00
Ankush Menat
98a33b4516
perf!: speedup @site_cache by ~4x (#28802)
* perf(site_cache): reduce access to frappe.local namespace

This change also allows calling @site_cache during init, as long as `site` parameter is set.

* test: frappe.init patching

* perf: use monotonic time instead of realtime for eviction

datetime is complex, slow and not really required for this use case.

* perf!: Drop support for unhashable arguments

Just like LRU cache, no need to support unhashable types in site_cache.
Current usage in codebase also shows that it's not required and json.dumps is quite slow.
2024-12-17 12:24:09 +05:30
Ankush Menat
9951f151f5 perf!: Drop support for unhashable arguments
Just like LRU cache, no need to support unhashable types in site_cache.
Current usage in codebase also shows that it's not required and json.dumps is quite slow.
2024-12-17 11:48:17 +05:30
Ankush Menat
f9ed28956d perf: use monotonic time instead of realtime for eviction
datetime is complex, slow and not really required for this use case.
2024-12-17 11:41:55 +05:30
Ankush Menat
ad0e8ed735 test: frappe.init patching 2024-12-17 11:33:21 +05:30
Ankush Menat
3675f3c797 perf(site_cache): reduce access to frappe.local namespace
This change also allows calling @site_cache during init, as long as `site` parameter is set.
2024-12-17 11:23:51 +05:30
Ankush Menat
531b9a3e3d
Merge pull request #28799 from ankush/perf/patching_bug
perf: Avoid patching QB in every request
2024-12-17 11:20:42 +05:30
Ankush Menat
5bf50b6bc2 perf: Avoid patching QB in every request 2024-12-17 10:58:31 +05:30
Ankush Menat
6ad70cb604
Merge pull request #28791 from ankush/update_mariadb
chore: update tested mariadb versions
2024-12-16 19:03:46 +05:30
Ankush Menat
54af230131 chore: update tested mariadb versions 2024-12-16 19:02:38 +05:30
Ankush Menat
5e4f23720a
perf: Use raw SQL for fetching documents (#28789)
`get_doc` is single most used commands, directly and indirectly.
Currently there's QB overhead of building queries.

These queries are SO SIMPLE, I don't believe we need QB to keep these
maintainable or compatible with databases.

Microbenchmarks (first is simple flat document, second has many child tables):

```
bench_orm_bench_get_doc: Mean +- std dev: [before] 10.7 ms +- 0.2 ms -> [after] 7.24 ms +- 0.15 ms: 1.48x faster
bench_orm_bench_get_user: Mean +- std dev: [before] 16.3 ms +- 0.2 ms -> [after] 9.50 ms +- 0.14 ms: 1.72x faster
```
2024-12-16 13:27:07 +00:00
Akhil Narang
2bbf78aa73
Merge pull request #28783 from akhilnarang/package-within-site
fix(package): validate package name, export path
2024-12-16 17:44:05 +05:30
Akhil Narang
fae07e4bbb
fix(package): validate package name, export path
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2024-12-16 17:19:54 +05:30