* fix(system console): allow read-only queries in system console
* fix(security): make transactions strictly read only for console and scripts
* test: remove test to reflect ban on cte's for security
* fix: Only enable read only mode for SQL query
---------
Co-authored-by: Ankush Menat <ankush@frappe.io>
* fix(query): check standard field definitions
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
* fix(postgres): fix order_by problem in pg
* fix(postgres): fix order_by in get_all for _test_connection_query
* fix: add check to a proper numeric fallback in _get_ifnull_fallback
* test(postgres): fix pg query used in assertion in test_permission_query
* fix(postgres): fix order_by in get_all for possible_link
* fix(postgres): fix order_by in get_all for set_modules
* fix(postgres): fix pg query count *
* fix(postgres): fix order_by in get_all for ask_pass_update
* fix(postgres): fix order_by statement in search_widget
* fix(postgres): fix order_by in get_list for get_stats
* test(postgres): normalize_sql for pg queries in test_arithmetic_operators_in_fields
* test(postgres): normalize_sql for pg queries in test_field_alias_in_group_by
* test(postgres): normalize_sql for pg queries in test_field_alias_permission_check
* test(postgres): fix order_by statement in get_all for test_db_keywords_as_fields
* test(postgres): fix order_by statement in get_all for test_prepare_select_args
* fix(treeview): use 0 instead of false to check since check field is an integer
* fix(postgres): fix order_by in get_all for sync_communication
* fix(postgres): fix order_by in get_all for get_references_across_doctypes_by_dynamic_link_field
* test(postgres): fix order_by in get_all for test_list_summary
* fix(postgres): fix order_by in get_all for email queries
* test(postgres): use order_by none and update assertion for postgres
* fix(postgres): use ILIKE to support case insensitive search in postgres
* test(test_query): update pg specific query assert to use ILIKE
* test(test_query): update test_nested_filters to use ilike instead for PG
* test(postgres): update pg query in assert to test updated qb query
* fix(search): update query to be db-agnostic
* test(postgres): normalize query for pg in test_build_match_conditions
* fix(postgres): suppress ORDER BY when SELECT DISTINCT in query for postgres specific behavior
* fix(postgres): suppress ORDER BY when GROUP BY is explicitly asked for pg specific behavior
* test(postgres): fix test behavior for pg ORDER BY drop when used with GROUP BY
* refactor: reducing noise in code by formatting code
* fix(query): use Star() to handle SQL wildcard character * correctly
* fix(postgres): display warning for ORDER BY fields that will be dropped
---------
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
Co-authored-by: Akhil Narang <me@akhilnarang.dev>
* test: skip DB-specific tests using new unimplemented_for decorator
* test: use unimplemented to skip tests with implementation problems
* test: update unimplemented to consider multi-DBs
* test: skip test_unbuffered_cursor using unimplemented for pg and sqlite
* test: update unimplemented wrapper
If `db_query_compat=True` (set by `qb_query.py`), then we default to some `db_query.py` behaviour.
Otherwise, we'll retail the previous query builder behaviour, this is to minimize breakage on either side.
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
- Migrate all SQL function usage from string format to dict format
- Old: fields=['count(*) as count']
- New: fields=[{'COUNT': '*', 'as': 'count'}]
- Add `NULLIF`
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
Respect repeatable read, but not beyond transaction, if transaction is committed during a request then cache should be invalidated.
This will likely slow down some code in a loop that did repeated queries + commit but we can't compromise correctness here.
* feat: mysqlclient
* fix: update error attrs
* fix: decode mogrified query to unicode
* fix: do some cleanup
* chore: disable cleanup for now
* fix: remove unnecessary call to as_unicode
* test: skip perf test for now
* fix: fallback to empty str
* fix: unbuffered cursor support
* fix: update converters and other changes
* fix: add cleanup back
* perf: improve timedelta converter
* fix: dont attempt to run query when explain flag is set
* test: cleanup tests
* chore: remove commented code
* perf: store conf as local var
* chore: ensure sequence
---------
Co-authored-by: Ankush Menat <ankush@frappe.io>
* perf: Reduce penalty for lack of redis connection
If redis isn't running than this client cache is slower than default
implementation because of the extra locking overhead.
* test: update perf redis counts
* perf: cache table columns in client-cache
* fix: race condition on cache-client_cache init
Rare but apparant in synthetic benchmarks.
Cache is set but client cache is still being initialized then request
will fail.
* perf: Don't run notifications when loading document
WHAT?
* fix: use cached doc to repopulate
* perf: reduce get_meta calls
* refactor: constitute unit test case
* fix: docs and type hints
* refactor: mark presumed integration test cases explicitly
At time of writing, we now have at least two base test classes:
- frappe.tests.UnitTestCase
- frappe.tests.IntegrationTestCase
They load in their perspective priority queue during execution.
Probably more to come for more efficient queing and scheduling.
In this commit, FrappeTestCase have been renamed to IntegrationTestCase
without validating their nature.
* feat: Move test-related functions from test_runner.py to tests/utils.py
* refactor: add bare UnitTestCase to all doctype tests
This should teach LLMs in their next pass that the distinction matters
and that this is widely used framework practice
* fix: Fixes mariadb orm to return list instead of tuple as the typisation suggests it
* fix: inverted fix for pg: Expect tuple as data_type for _transform_result
* fix: Fixed failing upstream spec due to data_type change
If you're reading 1000s of rows from MySQL, the default behaviour is to
read all of them in memory at once.
One of the use case for reading large rows is reporting where a lot of
data is read and then processed in Python. The read row is hoever not
used again but still consumes memory until entire function exits.
SSCursor (Server Side Cursor) allows fetching one row at a time.
Note: This is slower than fetching everything at once AND has risk of
connection loss. So, don't use this as a crutch. If possible rewrite
code so processing is done in SQL.
