Commit graph

412 commits

Author SHA1 Message Date
Ankush Menat
47a47a9b5d refactor!: Change internal datastructure of db.value_cache
It's now a defaultdictionary of `[doctype][name/filters][fieldname]`

This allows us to implement granular clearing and improve usage of this cache.
2025-06-05 14:46:03 +05:30
Ankush Menat
2d14918814
fix!: Change count(cache=True) implmentation (#32779)
This makes cache implementation uniform for all methods on db API. It's
weird that this specific method was caching in redis, which defies
expectations.
2025-06-04 19:15:27 +05:30
Sagar Vora
8192a87d00 perf: prebuild types for type checking 2025-05-30 11:22:12 +05:30
Ankush Menat
e2d619504f
perf: batch bulk_insert (#32675)
* fix: reduce bulk insert batch size

Back when this feature was added it used to lazily evaluate the input.
Now the iterator is consumed upfront so large batch sizes == huge memory usage.

* perf: bring back iterator for bulk_insert

Bulk insert used to support iterator for consuming arbitrarily large
amount of data and inserting it. Since child table support was added, it
can't do it anymore because that requires collecting values.

This change now brings back iterators by batching input iterator (by
default 1000) documents.

This is almost as good as original change from design POV. Performance
is still meh for flat documents.
2025-05-26 13:36:53 +00:00
Sagar Vora
e6196672ca fix(DX): log execution time in ms 2025-04-23 22:40:23 +05:30
Akhil Narang
294013955e
chore: linting
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
ad32216040
fix: support sqlite
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
0b5e245507
feat: init sqlite
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Gavin D'souza
886adfbc43
feat: allow overriding query execution
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Sagar Vora
3139419323 perf: declare _dict as global 2025-03-16 15:53:51 +05:30
Sagar Vora
fb0e2baa5a perf: store touched_tables in local var 2025-03-15 13:47:45 +05:30
Sagar Vora
521ff071ae perf: compute query type only once 2025-03-15 13:39:57 +05:30
Sagar Vora
ab98d3716f perf: optimize query building code 2025-03-15 10:29:26 +05:30
Sagar Vora
b2cab51849
feat: re-introduce mysqlclient 🚀 (#31719)
* feat: mysqlclient

* fix: update error attrs

* fix: decode mogrified query to unicode

* fix: do some cleanup

* chore: disable cleanup for now

* fix: remove unnecessary call to as_unicode

* test: skip perf test for now

* fix: fallback to empty str

* fix: unbuffered cursor support

* fix: update converters and other changes

* fix: add cleanup back

* perf: improve timedelta converter

* fix: dont attempt to run query when explain flag is set

* test: cleanup tests

* chore: remove commented code

* perf: store conf as local var

* chore: ensure sequence

---------

Co-authored-by: Ankush Menat <ankush@frappe.io>
2025-03-15 09:48:21 +05:30
Sagar Vora
26bbb5985e
perf: use client_cache for print_sql (#31663)
* perf: use `client_cache` for `print_sql`

* chore: revert change to conn error handling code

---------

Co-authored-by: Ankush Menat <ankushmenat@gmail.com>
2025-03-12 08:43:09 +00:00
Ankush Menat
a950ba3c59
fix: bump iterator batch size (#31142)
100 is too small, 10,000 is too big... 1000 is a decent middle ground.
2025-02-05 12:31:24 +00:00
Ankush Menat
ba8cc6f9e4
perf: use client cache for tables (#31102) 2025-02-04 09:30:59 +00:00
Ankush Menat
4406116f86 feat: estimate table size 2025-02-03 09:21:56 +05:30
Ankush Menat
fdba41c682
perf: misc client cache improvements (#29070)
* perf: Reduce penalty for lack of redis connection

If redis isn't running than this client cache is slower than default
implementation because of the extra locking overhead.

* test: update perf redis counts

* perf: cache table columns in client-cache

* fix: race condition on cache-client_cache init

Rare but apparant in synthetic benchmarks.

Cache is set but client cache is still being initialized then request
will fail.

* perf: Don't run notifications when loading document

WHAT?

* fix: use cached doc to repopulate

* perf: reduce get_meta calls
2025-01-07 16:14:43 +05:30
Ankush Menat
4f628ca091
fix: Never query flag_print_sql in developer_mode=0 (#28884)
Unnecessary overhead and need to disable this everytime I want to get
realistic performance numbers out.

All the performance affecting toggles should be directly controlled by
just `developer_mode` alone.
2024-12-23 13:57:01 +00:00
Ankush Menat
eaa6e4003a fix: Don't cache anything hashable
If filters are list or dict then they aren't hashable, there was little
reason to do this IMO.

If something is indeed cacheable then where is the eviction for it?
simple k:v is only thing we can realistically cache here.
2024-12-16 13:17:09 +05:30
Smit Vora
6b02484f1c
feat: bulk update docs using case when queries (#28483) 2024-12-06 11:34:25 +00:00
Akhil Narang
84ef6ec677
refactor: fixup with ruff 0.8.1
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2024-12-04 13:18:04 +05:30
Nabin Hait
322ce6f665
fix: get multiple values from single table (#28252)
* fix: get multiple values from single table

* fix: linter issues

* fix: linter issues
2024-10-23 12:11:23 +05:30
David Arnold
86f70b9e98
chore: inline unnecessary indirection (#28250) 2024-10-22 19:33:48 +00:00
David Arnold
90e44d950e
fix: hashable is enough to serve as (composite) cache key (#28189) 2024-10-19 23:19:48 +02:00
David Arnold
2abba7b51b
fix: don't force values into the string type (#28185) 2024-10-19 19:00:25 +00:00
David Arnold
7348572af8
feat: docref identifier / proxy (#27973)
* feat: add DocRef

* feat: Add comprehensive test cases for DocRef functionality

* chore(db): add field type hints

* fix: ensure document stringer fulfills the DocRef contract
2024-10-19 09:40:26 +05:30
David Arnold
8cfeb156df
devx: add deprecation dumpster (#27887)
* feat: Add deprecation_dumpster.py file

* docs: add jovial and jocose docstring for frappe/deprecation_dumpster.py

* refactor: fill the dumpster with its own kind

* refactor: move to the deprecation dumpster

* chore: color coding class

* fix: only check import error when import errors
2024-10-08 18:56:10 +02:00
gruener
8a2aa92389
fix: Fixes mariadb orm to return list instead of tuple as the typisat… (#27179)
* fix: Fixes mariadb orm to return list instead of tuple as the typisation suggests it

* fix: inverted fix for pg: Expect tuple as data_type for _transform_result

* fix: Fixed failing upstream spec due to data_type change
2024-08-09 09:27:07 +05:30
Ankush Menat
a1bd916cfa
docs: typo in db.delete (#26464) 2024-05-17 07:32:23 +00:00
Ankush Menat
5de5e25df6
fix: datetime comparison in QB (#26364)
closes https://github.com/frappe/frappe/issues/26363
2024-05-09 10:19:27 +00:00
Kevin Shenk
cd50b025d5
fix: QB name in error message (#26345)
"Query Engine" wording made the edited error message ambiguous, since the docs all refer to this feature as Query Builder
2024-05-07 11:39:28 +05:30
Nahuel Operto
c0e779998d
fix: lstrip for query writes detection (#26180) 2024-04-30 18:09:22 +05:30
Ankush Menat
dd82ab415c
fix: add missing impl for is_column_missing (#26225)
I removed it assuming it's already implemented in actual classes, but
it's alias for is_missing_column.
2024-04-29 16:46:12 +00:00
Ankush Menat
c1bf152b89
fix: handle interface error during report timeout (#25893) 2024-04-10 06:57:36 +00:00
Akhil Narang
ae34b12ce8
Merge pull request #22548 from blaggacao/feat/uds
feat: support unix domain sockets
2024-03-29 18:11:01 +05:30
Ankush Menat
de48dc2c04
fix: db explain (#25724)
This broke from some recent changes.

It's just a wrapper, so probably no one uses it.
Anyway, fixed and added test to prevent it.
2024-03-29 10:23:12 +00:00
David
39d4318a27
feat: enable db socket connection 2024-03-29 10:29:50 +01:00
Ankush Menat
be579f9a8d
fix: sql_ddl - always ensure commit (#25704) 2024-03-28 13:13:55 +00:00
Ankush Menat
23204c0c05
Merge pull request #25256 from ankush/no_commit_doc_events
fix: disable transaction commits during doc events
2024-03-28 12:28:12 +05:30
Ankush Menat
e5a64fd50c test: transaction control 2024-03-28 11:58:38 +05:30
Ankush Menat
f66b23b96d fix: handle nested event calls
Treat disable_transaction_control as a stack incr/decr when moving in
and out of context.
2024-03-28 11:27:24 +05:30
Ankush Menat
071e269548 fix!: Switch to creation as default sort order 2024-03-27 11:18:28 +05:30
Ankush Menat
13c74ec231 fix: Use debug log to log DB queries 2024-03-10 12:49:19 +05:30
Ankush Menat
638dbb6bcd fix: disable transaction commits during doc events
- Events like doc.save and doc.submit need to be atomic
- Document hooks can make it not so atomic.

This is extending server script behaviour where server script hooks are
not allowed to commit/rollback.
2024-03-06 14:53:00 +05:30
Ankush Menat
e810fb7eca feat: nowait to skip blocking locks 2024-02-29 16:30:24 +05:30
Shankarv19bcr
437beaa735 fix: typo 2024-02-21 17:19:32 +05:30
Akhil Narang
3f1e19de85
refactor(treewide): enable RUF rules
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2024-02-21 16:20:28 +05:30
Raffael Meyer
68eb2d978d
fix: keep order in get_values_from_single (#24907)
* fix: keep order in `get_values_from_single`

* fix: add test for destructuring

---------

Co-authored-by: Ankush Menat <ankush@frappe.io>
2024-02-19 05:35:35 +00:00