Commit graph

931 commits

Author SHA1 Message Date
Akhil Narang
ec27078266
chore(sqlite): set connection timeout to 15 seconds and set synchronous pragma
(wal was already enabled in the file itself, no harm setting it here for consistency.

Reference: 45622c7d54

Co-authored-by: 18alantom <2.alan.tom@gmail.com>
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
dc60a65c38
fix(sqlite): don't allow starting nested transactions
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
7d62f22a3a
fix(sqlite): hack together something for read-only mode
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
2114a47256
fix(sqlite): we get frappe exception classes here sometimes
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
9fa330c075
fix(sqlite): use correct check for unique/primary key constraint violations
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
9adad5e262
fix(sqlite): get_database_size() was checking a non-existent path
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
40c07e0d9b
fix(sqlite): don't try to re-create existing indices
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
50fae37072
fix(sqlite): don't consider implicit commit for begin for now
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
5b98b4ca93
fix: convert timestamps to datetime
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
7d4634519e
fix: drop some overrides for now
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
294013955e
chore: linting
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
44e7b64aae
feat(sqlite): implement regexp
Python can't use the implementation that sqlite3 cli uses directly

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
8b92cca1f3
chore(sqlite): extend 573028ad3f
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
482f2cb3f5
fix(sqlite): escape some string values
Some strings can have ' in them

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
c2965e3dd2
feat(sqlite): implement truncate
(delete rows + delete from sqlite_sequence)

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
69d22a8194
fix(sqlite): implement rename and change column type
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
6e31745290
chore: handle restoring gzipped backups
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
8106e72981
refactor: use a database file directly, instead of an SQL file
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
41e4b4c76d
refactor: strpos -> instr
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
ed8191047e
fix(add_column): adjust for sqlite
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
ad32216040
fix: support sqlite
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Gavin D'souza
f8ccbfd3d7
feat(sqlite): override some methods
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
0b5e245507
feat: init sqlite
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Gavin D'souza
886adfbc43
feat: allow overriding query execution
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Ankush Menat
8c0391d085 fix: Define InterfaceError for mariadb 2025-03-19 10:46:23 +05:30
Tarun Pratap Singh
649cf2af66 fix(mysqlclient): set collation in connection 2025-03-18 12:40:25 +05:30
Akhil Narang
4a04b0a17f
Merge pull request #31523 from akhilnarang/set-mariadb-collation
fix(mariadb): set collation in connection
2025-03-17 11:06:27 +00:00
Sagar Vora
3139419323 perf: declare _dict as global 2025-03-16 15:53:51 +05:30
Sagar Vora
35f861fa87 perf: extract query type using re.match 2025-03-16 15:39:36 +05:30
Sagar Vora
fb0e2baa5a perf: store touched_tables in local var 2025-03-15 13:47:45 +05:30
Sagar Vora
521ff071ae perf: compute query type only once 2025-03-15 13:39:57 +05:30
Sagar Vora
ab98d3716f perf: optimize query building code 2025-03-15 10:29:26 +05:30
Sagar Vora
b2cab51849
feat: re-introduce mysqlclient 🚀 (#31719)
* feat: mysqlclient

* fix: update error attrs

* fix: decode mogrified query to unicode

* fix: do some cleanup

* chore: disable cleanup for now

* fix: remove unnecessary call to as_unicode

* test: skip perf test for now

* fix: fallback to empty str

* fix: unbuffered cursor support

* fix: update converters and other changes

* fix: add cleanup back

* perf: improve timedelta converter

* fix: dont attempt to run query when explain flag is set

* test: cleanup tests

* chore: remove commented code

* perf: store conf as local var

* chore: ensure sequence

---------

Co-authored-by: Ankush Menat <ankush@frappe.io>
2025-03-15 09:48:21 +05:30
Sagar Vora
26bbb5985e
perf: use client_cache for print_sql (#31663)
* perf: use `client_cache` for `print_sql`

* chore: revert change to conn error handling code

---------

Co-authored-by: Ankush Menat <ankushmenat@gmail.com>
2025-03-12 08:43:09 +00:00
Ankush Menat
77852fd11f
fix: search instead of match (#31557)
Match isn't correct usage here
2025-03-07 06:59:23 +00:00
Ankush Menat
573028ad3f
fix: always persist all indexes added via db.add_index (#31177)
* fix: always persist all indexes added via db.add_index

* fix: Add `if not exists` clause for index creation

This allows replica to have same index and master to add it later
without causing SQL error. Just minor DX benefit.

* fix(postgres): don't cache if table doesn't exist

* chore: revert postgres changes

Hopeless to maintain this
2025-02-25 07:25:49 +00:00
Ankush Menat
f90a450bd4
feat: specify row compression for tables (#31361)
Certain tables contain A LOT of duplicate data, it makes sense to enable
compressed row format on them by default. I've seen 5-10 fold reduction
in DB size after enabling compressed format on select few tables.

This has some performance overhead:
- both compressed and uncompressed pages live in buffer pool.
- compression/decompression

Note:
- These cons don't apply much on DocTypes I am enabling this for.
- I am not enabling this on existing sites, migration can take a long
  time! Do it manually with `transform-database` command if you want to.
2025-02-21 09:44:40 +05:30
Brian Pond
00fb8e7a7a
fix: get_table_columns_description() for MariaDB database.py (#28851) 2025-02-11 17:57:52 +05:30
Ankush Menat
50a88149c1
fix: reconnect on operational error too (#31172)
MySQL seems to raise diff error based on which state the connection was
in when remote connection closed. Anyway, this should guard against both
kinds of failures.
2025-02-07 09:10:07 +00:00
gruener
02d7c7c913
style: fix errormsg to refer postgres instead of mariadb 2025-02-05 14:48:11 +01:00
Ankush Menat
a950ba3c59
fix: bump iterator batch size (#31142)
100 is too small, 10,000 is too big... 1000 is a decent middle ground.
2025-02-05 12:31:24 +00:00
Ankush Menat
ba8cc6f9e4
perf: use client cache for tables (#31102) 2025-02-04 09:30:59 +00:00
Ankush Menat
4406116f86 feat: estimate table size 2025-02-03 09:21:56 +05:30
Ankush Menat
fdba41c682
perf: misc client cache improvements (#29070)
* perf: Reduce penalty for lack of redis connection

If redis isn't running than this client cache is slower than default
implementation because of the extra locking overhead.

* test: update perf redis counts

* perf: cache table columns in client-cache

* fix: race condition on cache-client_cache init

Rare but apparant in synthetic benchmarks.

Cache is set but client cache is still being initialized then request
will fail.

* perf: Don't run notifications when loading document

WHAT?

* fix: use cached doc to repopulate

* perf: reduce get_meta calls
2025-01-07 16:14:43 +05:30
Ankush Menat
4f628ca091
fix: Never query flag_print_sql in developer_mode=0 (#28884)
Unnecessary overhead and need to disable this everytime I want to get
realistic performance numbers out.

All the performance affecting toggles should be directly controlled by
just `developer_mode` alone.
2024-12-23 13:57:01 +00:00
Akhil Narang
098d4896e3
fix(restore): remove MariaDB view security definers
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2024-12-23 17:23:17 +05:30
Brian Pond
61a16f399e
fix: MariaDBDatabase.get_tables() should not query the entire database schema (#28846) 2024-12-22 13:45:23 +05:30
Ankush Menat
23b5b0c7ae
perf: speedup QB field sanitization (#28818) 2024-12-18 05:47:02 +00:00
Ankush Menat
54af230131 chore: update tested mariadb versions 2024-12-16 19:02:38 +05:30
Ankush Menat
eaa6e4003a fix: Don't cache anything hashable
If filters are list or dict then they aren't hashable, there was little
reason to do this IMO.

If something is indeed cacheable then where is the eviction for it?
simple k:v is only thing we can realistically cache here.
2024-12-16 13:17:09 +05:30