Ankush Menat
b57eb60486
perf: chain db transactions ( #33004 )
...
* perf: chain transactions
Frequently used rollback/commits can be modified to chain previous
transaction.
This reduces one query to DB in most requests.
* perf: chain transactions in requests
2025-06-19 12:37:39 +05:30
Ankush Menat
3a7db9cbb7
refactor: Default to mysqlclient ( #32987 )
2025-06-18 06:04:24 +00:00
Ankush Menat
7b8eb5d1b6
feat: MariaDB 11.8 support ( #32289 )
...
* ci: Switch to MariaDB 11.8
* ci: Use mariadb's latest client libraries
Co-Authored-By: Sagar Vora <sagar@resilient.tech>
---------
Co-authored-by: Sagar Vora <sagar@resilient.tech>
2025-06-12 10:11:24 +00:00
Sagar Vora
d35c1d958f
perf(query engine): create one less copy ( #32889 )
2025-06-11 12:51:55 +05:30
Ankush Menat
271fe0e47e
fix!: Dont silently ignore bad filters ( #32871 )
2025-06-10 12:06:44 +05:30
Ankush Menat
e4bc7f361b
Revert: DocRef ( #32866 )
...
- Hardly used anywhere
- Too many hardcoded `__value__` calls without which it's not usable.
- Another type to worry about
2025-06-10 05:20:56 +00:00
Ankush Menat
292646a5a7
Revert "Revert "perf: Make get_query query mutable ( #32849 )" ( #32855 )" ( #32857 )
...
This reverts commit 85dc9e6981
2025-06-09 14:43:27 +00:00
Ankush Menat
85dc9e6981
Revert "perf: Make get_query query mutable ( #32849 )" ( #32855 )
...
This reverts commit dda62ff784
2025-06-09 13:17:16 +00:00
Ankush Menat
dda62ff784
perf: Make get_query query mutable ( #32849 )
...
pypika internally keeps copying query builder object because everything
is supposed to be immutable in pypika design, this however is terribly
slow. Often query generation takes more time than query execution.
This PR makes query builder mutable inside `get_query` function to avoid
copying while applying fields, filters, limit, order etc.
It's marked as immutable again when sending it back to users of the API.
2025-06-09 13:46:40 +05:30
Ankush Menat
f1a03200ab
fix: clear db.value_cache when clearing doctype cache
2025-06-05 14:46:03 +05:30
Ankush Menat
47a47a9b5d
refactor!: Change internal datastructure of db.value_cache
...
It's now a defaultdictionary of `[doctype][name/filters][fieldname]`
This allows us to implement granular clearing and improve usage of this cache.
2025-06-05 14:46:03 +05:30
Ankush Menat
2d14918814
fix!: Change count(cache=True) implmentation ( #32779 )
...
This makes cache implementation uniform for all methods on db API. It's
weird that this specific method was caching in redis, which defies
expectations.
2025-06-04 19:15:27 +05:30
Sagar Vora
a212ca8be5
fix: better regex for extracting query type
2025-05-30 11:23:14 +05:30
Sagar Vora
8192a87d00
perf: prebuild types for type checking
2025-05-30 11:22:12 +05:30
Ankush Menat
e2d619504f
perf: batch bulk_insert ( #32675 )
...
* fix: reduce bulk insert batch size
Back when this feature was added it used to lazily evaluate the input.
Now the iterator is consumed upfront so large batch sizes == huge memory usage.
* perf: bring back iterator for bulk_insert
Bulk insert used to support iterator for consuming arbitrarily large
amount of data and inserting it. Since child table support was added, it
can't do it anymore because that requires collecting values.
This change now brings back iterators by batching input iterator (by
default 1000) documents.
This is almost as good as original change from design POV. Performance
is still meh for flat documents.
2025-05-26 13:36:53 +00:00
Ankush Menat
c778ce370f
perf: Avoid Coalesce in QB is implementation
2025-05-02 19:09:51 +05:30
Ankush Menat
2dfb96f91c
fix: handle snapshot isolation errors better ( #32318 )
...
* fix: Avoid Snapshot violation
- Main thread created and "read" user
- Other thread modified something
- Main thread wants to delete or "write" to same row.
This violates snapshot isolation.
* fix: treat snapshot violation as deadlock for now
* test: handle snapshot violations
2025-04-28 06:18:38 +00:00
Sagar Vora
e6196672ca
fix(DX): log execution time in ms
2025-04-23 22:40:23 +05:30
Akhil Narang
ec27078266
chore(sqlite): set connection timeout to 15 seconds and set synchronous pragma
...
(wal was already enabled in the file itself, no harm setting it here for consistency.
Reference: 45622c7d54
2025-04-15 13:59:17 +05:30
Akhil Narang
dc60a65c38
fix(sqlite): don't allow starting nested transactions
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
7d62f22a3a
fix(sqlite): hack together something for read-only mode
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
2114a47256
fix(sqlite): we get frappe exception classes here sometimes
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
9fa330c075
fix(sqlite): use correct check for unique/primary key constraint violations
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
9adad5e262
fix(sqlite): get_database_size() was checking a non-existent path
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
40c07e0d9b
fix(sqlite): don't try to re-create existing indices
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
50fae37072
fix(sqlite): don't consider implicit commit for begin for now
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
5b98b4ca93
fix: convert timestamps to datetime
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
7d4634519e
fix: drop some overrides for now
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
294013955e
chore: linting
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
44e7b64aae
feat(sqlite): implement regexp
...
Python can't use the implementation that sqlite3 cli uses directly
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
8b92cca1f3
chore(sqlite): extend 573028ad3f
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
482f2cb3f5
fix(sqlite): escape some string values
...
Some strings can have ' in them
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
c2965e3dd2
feat(sqlite): implement truncate
...
(delete rows + delete from sqlite_sequence)
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
69d22a8194
fix(sqlite): implement rename and change column type
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
6e31745290
chore: handle restoring gzipped backups
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
8106e72981
refactor: use a database file directly, instead of an SQL file
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
41e4b4c76d
refactor: strpos -> instr
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
ed8191047e
fix(add_column): adjust for sqlite
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
ad32216040
fix: support sqlite
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Gavin D'souza
f8ccbfd3d7
feat(sqlite): override some methods
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
0b5e245507
feat: init sqlite
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Gavin D'souza
886adfbc43
feat: allow overriding query execution
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Ankush Menat
8c0391d085
fix: Define InterfaceError for mariadb
2025-03-19 10:46:23 +05:30
Tarun Pratap Singh
649cf2af66
fix(mysqlclient): set collation in connection
2025-03-18 12:40:25 +05:30
Akhil Narang
4a04b0a17f
Merge pull request #31523 from akhilnarang/set-mariadb-collation
...
fix(mariadb): set collation in connection
2025-03-17 11:06:27 +00:00
Sagar Vora
3139419323
perf: declare _dict as global
2025-03-16 15:53:51 +05:30
Sagar Vora
35f861fa87
perf: extract query type using re.match
2025-03-16 15:39:36 +05:30
Sagar Vora
fb0e2baa5a
perf: store touched_tables in local var
2025-03-15 13:47:45 +05:30
Sagar Vora
521ff071ae
perf: compute query type only once
2025-03-15 13:39:57 +05:30
Sagar Vora
ab98d3716f
perf: optimize query building code
2025-03-15 10:29:26 +05:30
