Commit graph

934 commits

Author SHA1 Message Date
Ankush Menat
c778ce370f perf: Avoid Coalesce in QB is implementation 2025-05-02 19:09:51 +05:30
Ankush Menat
2dfb96f91c
fix: handle snapshot isolation errors better (#32318)
* fix: Avoid Snapshot violation

- Main thread created and "read" user
- Other thread modified something
- Main thread wants to delete or "write" to same row.

This violates snapshot isolation.

* fix: treat snapshot violation as deadlock for now

* test: handle snapshot violations
2025-04-28 06:18:38 +00:00
Sagar Vora
e6196672ca fix(DX): log execution time in ms 2025-04-23 22:40:23 +05:30
Akhil Narang
ec27078266
chore(sqlite): set connection timeout to 15 seconds and set synchronous pragma
(wal was already enabled in the file itself, no harm setting it here for consistency.

Reference: 45622c7d54

Co-authored-by: 18alantom <2.alan.tom@gmail.com>
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
dc60a65c38
fix(sqlite): don't allow starting nested transactions
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
7d62f22a3a
fix(sqlite): hack together something for read-only mode
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
2114a47256
fix(sqlite): we get frappe exception classes here sometimes
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
9fa330c075
fix(sqlite): use correct check for unique/primary key constraint violations
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
9adad5e262
fix(sqlite): get_database_size() was checking a non-existent path
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
40c07e0d9b
fix(sqlite): don't try to re-create existing indices
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
50fae37072
fix(sqlite): don't consider implicit commit for begin for now
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
5b98b4ca93
fix: convert timestamps to datetime
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
7d4634519e
fix: drop some overrides for now
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
294013955e
chore: linting
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
44e7b64aae
feat(sqlite): implement regexp
Python can't use the implementation that sqlite3 cli uses directly

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
8b92cca1f3
chore(sqlite): extend 573028ad3f
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
482f2cb3f5
fix(sqlite): escape some string values
Some strings can have ' in them

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
c2965e3dd2
feat(sqlite): implement truncate
(delete rows + delete from sqlite_sequence)

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
69d22a8194
fix(sqlite): implement rename and change column type
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
6e31745290
chore: handle restoring gzipped backups
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
8106e72981
refactor: use a database file directly, instead of an SQL file
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
41e4b4c76d
refactor: strpos -> instr
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
ed8191047e
fix(add_column): adjust for sqlite
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
ad32216040
fix: support sqlite
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Gavin D'souza
f8ccbfd3d7
feat(sqlite): override some methods
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
0b5e245507
feat: init sqlite
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Gavin D'souza
886adfbc43
feat: allow overriding query execution
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Ankush Menat
8c0391d085 fix: Define InterfaceError for mariadb 2025-03-19 10:46:23 +05:30
Tarun Pratap Singh
649cf2af66 fix(mysqlclient): set collation in connection 2025-03-18 12:40:25 +05:30
Akhil Narang
4a04b0a17f
Merge pull request #31523 from akhilnarang/set-mariadb-collation
fix(mariadb): set collation in connection
2025-03-17 11:06:27 +00:00
Sagar Vora
3139419323 perf: declare _dict as global 2025-03-16 15:53:51 +05:30
Sagar Vora
35f861fa87 perf: extract query type using re.match 2025-03-16 15:39:36 +05:30
Sagar Vora
fb0e2baa5a perf: store touched_tables in local var 2025-03-15 13:47:45 +05:30
Sagar Vora
521ff071ae perf: compute query type only once 2025-03-15 13:39:57 +05:30
Sagar Vora
ab98d3716f perf: optimize query building code 2025-03-15 10:29:26 +05:30
Sagar Vora
b2cab51849
feat: re-introduce mysqlclient 🚀 (#31719)
* feat: mysqlclient

* fix: update error attrs

* fix: decode mogrified query to unicode

* fix: do some cleanup

* chore: disable cleanup for now

* fix: remove unnecessary call to as_unicode

* test: skip perf test for now

* fix: fallback to empty str

* fix: unbuffered cursor support

* fix: update converters and other changes

* fix: add cleanup back

* perf: improve timedelta converter

* fix: dont attempt to run query when explain flag is set

* test: cleanup tests

* chore: remove commented code

* perf: store conf as local var

* chore: ensure sequence

---------

Co-authored-by: Ankush Menat <ankush@frappe.io>
2025-03-15 09:48:21 +05:30
Sagar Vora
26bbb5985e
perf: use client_cache for print_sql (#31663)
* perf: use `client_cache` for `print_sql`

* chore: revert change to conn error handling code

---------

Co-authored-by: Ankush Menat <ankushmenat@gmail.com>
2025-03-12 08:43:09 +00:00
Ankush Menat
77852fd11f
fix: search instead of match (#31557)
Match isn't correct usage here
2025-03-07 06:59:23 +00:00
Ankush Menat
573028ad3f
fix: always persist all indexes added via db.add_index (#31177)
* fix: always persist all indexes added via db.add_index

* fix: Add `if not exists` clause for index creation

This allows replica to have same index and master to add it later
without causing SQL error. Just minor DX benefit.

* fix(postgres): don't cache if table doesn't exist

* chore: revert postgres changes

Hopeless to maintain this
2025-02-25 07:25:49 +00:00
Ankush Menat
f90a450bd4
feat: specify row compression for tables (#31361)
Certain tables contain A LOT of duplicate data, it makes sense to enable
compressed row format on them by default. I've seen 5-10 fold reduction
in DB size after enabling compressed format on select few tables.

This has some performance overhead:
- both compressed and uncompressed pages live in buffer pool.
- compression/decompression

Note:
- These cons don't apply much on DocTypes I am enabling this for.
- I am not enabling this on existing sites, migration can take a long
  time! Do it manually with `transform-database` command if you want to.
2025-02-21 09:44:40 +05:30
Brian Pond
00fb8e7a7a
fix: get_table_columns_description() for MariaDB database.py (#28851) 2025-02-11 17:57:52 +05:30
Ankush Menat
50a88149c1
fix: reconnect on operational error too (#31172)
MySQL seems to raise diff error based on which state the connection was
in when remote connection closed. Anyway, this should guard against both
kinds of failures.
2025-02-07 09:10:07 +00:00
gruener
02d7c7c913
style: fix errormsg to refer postgres instead of mariadb 2025-02-05 14:48:11 +01:00
Ankush Menat
a950ba3c59
fix: bump iterator batch size (#31142)
100 is too small, 10,000 is too big... 1000 is a decent middle ground.
2025-02-05 12:31:24 +00:00
Ankush Menat
ba8cc6f9e4
perf: use client cache for tables (#31102) 2025-02-04 09:30:59 +00:00
Ankush Menat
4406116f86 feat: estimate table size 2025-02-03 09:21:56 +05:30
Ankush Menat
fdba41c682
perf: misc client cache improvements (#29070)
* perf: Reduce penalty for lack of redis connection

If redis isn't running than this client cache is slower than default
implementation because of the extra locking overhead.

* test: update perf redis counts

* perf: cache table columns in client-cache

* fix: race condition on cache-client_cache init

Rare but apparant in synthetic benchmarks.

Cache is set but client cache is still being initialized then request
will fail.

* perf: Don't run notifications when loading document

WHAT?

* fix: use cached doc to repopulate

* perf: reduce get_meta calls
2025-01-07 16:14:43 +05:30
Ankush Menat
4f628ca091
fix: Never query flag_print_sql in developer_mode=0 (#28884)
Unnecessary overhead and need to disable this everytime I want to get
realistic performance numbers out.

All the performance affecting toggles should be directly controlled by
just `developer_mode` alone.
2024-12-23 13:57:01 +00:00
Akhil Narang
098d4896e3
fix(restore): remove MariaDB view security definers
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2024-12-23 17:23:17 +05:30
Brian Pond
61a16f399e
fix: MariaDBDatabase.get_tables() should not query the entire database schema (#28846) 2024-12-22 13:45:23 +05:30