Faris Ansari
63afc0601b
fix: restrict child table access if user has only "select" on parent
2025-06-18 16:00:22 +05:30
Faris Ansari
87664ad604
refactor: Enhance field and function parsing in query engine
...
- Introduce `SqlFunctionParser` for robust parsing of supported SQL functions (e.g., `COUNT(*)`, `SUM(amount) as total`, `AVG(price - cost)`), replacing get_function_object and has_function.
- Refactor `DynamicTableField.parse` for improved handling of:
- Aliases (case-insensitive `as`, quoted/unquoted).
- `tabDocType.fieldname` notation (distinguishing child vs. main doctype refs).
- Add validation and better error handling during parsing.
- Rewrite filter field validation (`_validate_and_prepare_filter_field`):
- Disallow backticks (`) in filter field names.
- Enforce specific patterns for dot notation (link/child fields only, reject `tabDoc.field`).
- Validate character sets for simple field names.
- Update standard field parsing (`parse_string_field`, `ALLOWED_FIELD_PATTERN`, `FIELD_PARSE_REGEX`):
- Support quoted table names potentially containing spaces (e.g., `tabTable Name`.`field`).
- Improve `parse_fields` and `_parse_single_field_item` logic:
- Handle direct pypika `Field`/`AggregateFunction` inputs.
- Reliably split comma-separated field strings.
```
2025-06-18 16:00:22 +05:30
Faris Ansari
ddca77429c
fix: secure query building
...
Add strict validation using regex for fields in SELECT, filters, GROUP BY, and ORDER BY clauses to avoid potential SQL injection risks.
Refactor field parsing and validation logic into dedicated functions.
2025-06-18 16:00:22 +05:30
Faris Ansari
8aa4c1030f
fix: add support for AND, OR, NOT in RawCriterion
2025-06-18 15:56:11 +05:30
Faris Ansari
a94c143314
fix: add support for permission query conditions
2025-06-18 15:56:11 +05:30
Faris Ansari
f707cf5722
fix: raise PermissionError instead of ValidationError
2025-06-18 15:56:11 +05:30
Faris Ansari
f580cb3dad
fix: add child query to allowed fields
2025-06-18 15:56:11 +05:30
Faris Ansari
471e001ebb
feat: apply permissions in get_query
2025-06-18 15:56:11 +05:30
Ankush Menat
3a7db9cbb7
refactor: Default to mysqlclient ( #32987 )
2025-06-18 06:04:24 +00:00
Ankush Menat
7b8eb5d1b6
feat: MariaDB 11.8 support ( #32289 )
...
* ci: Switch to MariaDB 11.8
* ci: Use mariadb's latest client libraries
Co-Authored-By: Sagar Vora <sagar@resilient.tech>
---------
Co-authored-by: Sagar Vora <sagar@resilient.tech>
2025-06-12 10:11:24 +00:00
Sagar Vora
d35c1d958f
perf(query engine): create one less copy ( #32889 )
2025-06-11 12:51:55 +05:30
Ankush Menat
271fe0e47e
fix!: Dont silently ignore bad filters ( #32871 )
2025-06-10 12:06:44 +05:30
Ankush Menat
e4bc7f361b
Revert: DocRef ( #32866 )
...
- Hardly used anywhere
- Too many hardcoded `__value__` calls without which it's not usable.
- Another type to worry about
2025-06-10 05:20:56 +00:00
Ankush Menat
292646a5a7
Revert "Revert "perf: Make get_query query mutable ( #32849 )" ( #32855 )" ( #32857 )
...
This reverts commit 85dc9e6981
2025-06-09 14:43:27 +00:00
Ankush Menat
85dc9e6981
Revert "perf: Make get_query query mutable ( #32849 )" ( #32855 )
...
This reverts commit dda62ff784
2025-06-09 13:17:16 +00:00
Ankush Menat
dda62ff784
perf: Make get_query query mutable ( #32849 )
...
pypika internally keeps copying query builder object because everything
is supposed to be immutable in pypika design, this however is terribly
slow. Often query generation takes more time than query execution.
This PR makes query builder mutable inside `get_query` function to avoid
copying while applying fields, filters, limit, order etc.
It's marked as immutable again when sending it back to users of the API.
2025-06-09 13:46:40 +05:30
Ankush Menat
f1a03200ab
fix: clear db.value_cache when clearing doctype cache
2025-06-05 14:46:03 +05:30
Ankush Menat
47a47a9b5d
refactor!: Change internal datastructure of db.value_cache
...
It's now a defaultdictionary of `[doctype][name/filters][fieldname]`
This allows us to implement granular clearing and improve usage of this cache.
2025-06-05 14:46:03 +05:30
Ankush Menat
2d14918814
fix!: Change count(cache=True) implmentation ( #32779 )
...
This makes cache implementation uniform for all methods on db API. It's
weird that this specific method was caching in redis, which defies
expectations.
2025-06-04 19:15:27 +05:30
Sagar Vora
a212ca8be5
fix: better regex for extracting query type
2025-05-30 11:23:14 +05:30
Sagar Vora
8192a87d00
perf: prebuild types for type checking
2025-05-30 11:22:12 +05:30
Ankush Menat
e2d619504f
perf: batch bulk_insert ( #32675 )
...
* fix: reduce bulk insert batch size
Back when this feature was added it used to lazily evaluate the input.
Now the iterator is consumed upfront so large batch sizes == huge memory usage.
* perf: bring back iterator for bulk_insert
Bulk insert used to support iterator for consuming arbitrarily large
amount of data and inserting it. Since child table support was added, it
can't do it anymore because that requires collecting values.
This change now brings back iterators by batching input iterator (by
default 1000) documents.
This is almost as good as original change from design POV. Performance
is still meh for flat documents.
2025-05-26 13:36:53 +00:00
Ankush Menat
c778ce370f
perf: Avoid Coalesce in QB is implementation
2025-05-02 19:09:51 +05:30
Ankush Menat
2dfb96f91c
fix: handle snapshot isolation errors better ( #32318 )
...
* fix: Avoid Snapshot violation
- Main thread created and "read" user
- Other thread modified something
- Main thread wants to delete or "write" to same row.
This violates snapshot isolation.
* fix: treat snapshot violation as deadlock for now
* test: handle snapshot violations
2025-04-28 06:18:38 +00:00
Sagar Vora
e6196672ca
fix(DX): log execution time in ms
2025-04-23 22:40:23 +05:30
Akhil Narang
ec27078266
chore(sqlite): set connection timeout to 15 seconds and set synchronous pragma
...
(wal was already enabled in the file itself, no harm setting it here for consistency.
Reference: 45622c7d54
2025-04-15 13:59:17 +05:30
Akhil Narang
dc60a65c38
fix(sqlite): don't allow starting nested transactions
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
7d62f22a3a
fix(sqlite): hack together something for read-only mode
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
2114a47256
fix(sqlite): we get frappe exception classes here sometimes
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
9fa330c075
fix(sqlite): use correct check for unique/primary key constraint violations
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
9adad5e262
fix(sqlite): get_database_size() was checking a non-existent path
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
40c07e0d9b
fix(sqlite): don't try to re-create existing indices
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
50fae37072
fix(sqlite): don't consider implicit commit for begin for now
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
5b98b4ca93
fix: convert timestamps to datetime
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
7d4634519e
fix: drop some overrides for now
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
294013955e
chore: linting
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
44e7b64aae
feat(sqlite): implement regexp
...
Python can't use the implementation that sqlite3 cli uses directly
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
8b92cca1f3
chore(sqlite): extend 573028ad3f
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
482f2cb3f5
fix(sqlite): escape some string values
...
Some strings can have ' in them
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
c2965e3dd2
feat(sqlite): implement truncate
...
(delete rows + delete from sqlite_sequence)
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:17 +05:30
Akhil Narang
69d22a8194
fix(sqlite): implement rename and change column type
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
6e31745290
chore: handle restoring gzipped backups
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
8106e72981
refactor: use a database file directly, instead of an SQL file
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
41e4b4c76d
refactor: strpos -> instr
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
ed8191047e
fix(add_column): adjust for sqlite
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
ad32216040
fix: support sqlite
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Gavin D'souza
f8ccbfd3d7
feat(sqlite): override some methods
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Akhil Narang
0b5e245507
feat: init sqlite
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Gavin D'souza
886adfbc43
feat: allow overriding query execution
...
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2025-04-15 13:59:16 +05:30
Ankush Menat
8c0391d085
fix: Define InterfaceError for mariadb
2025-03-19 10:46:23 +05:30
