Commit graph

850 commits

Author SHA1 Message Date
Gavin D'souza
cf699fe40b Merge branch 'develop' of github.com:frappe/frappe into mariadb-client-refactor 2022-06-29 13:49:18 +05:30
Ankush Menat
181833848f
Merge pull request #17337 from ankush/lint_cleanup
fix: remove bare exception catching
2022-06-29 11:39:05 +05:30
gavin
91e76ff874
Merge branch 'develop' into get-all-mod 2022-06-28 19:47:20 +05:30
Aradhya
303d94494d refactor: atomic functions & removed complicated checks
Co-authored-by: gavin <gavin18d@gmail.com>
2022-06-28 19:44:03 +05:30
Aradhya
6db6be1f3c refactor: frappe.qb.engine
* Small fixes in set_fields and clean code
* Optimize casefolds
* Fixed functions passed in List
* get_sql => get_query - more expressive, less confusion
* Updated tests
2022-06-28 19:43:57 +05:30
Aradhya
7732accded feat: Attached Engine object to qb & added dynamic type hints 2022-06-28 19:43:23 +05:30
Aradhya
960952cfc3 feat(qb-engine): Added Aggregation function support
* Added Min, Max
* Added tests
2022-06-28 19:41:39 +05:30
Aradhya
fca026927e refactor: moved all query logic to query class 2022-06-28 19:41:32 +05:30
Aradhya
d0680941ad refactor: frappe.qb.engine
* feat: supporting empty iterables for Contains objects
* fix: explicitly setting empty iterables as tuples to support more operators
* feat: Added locate to frappe.qb Functions
* feat: Added support for functions passed as strings in fields
* feat: Included Criterion objects as fields
* fix: picking up only function intended fields to pass to get_function_objects
* feat: Added iterable for available functions, added support for Field objects
* fix: fixed * passed in fields in lists
2022-06-28 19:41:23 +05:30
Aradhya
d1f5c49b02 refactor(qb-engine): frappe.db.query
* feat: Added support for True as filter and pluck in the query engine

* feat: Added join support

* fix: return if filters are None

* feat: Added support for sets as filters and additional conditions

* fix: fixed IS operator in query builder

* feat: Added support for 'as' in query engine

* fix: fixed 'as' for criterion objects passed directly

* fix: fixed frappe.db.count

* fix: fixed functions in fieldname

* feat: Added support for multiple fields passed as a single string :) fixed None fields in a list

* feat: Added support for "as" in single string fields

* fix: fixed queries with invalid syntax
2022-06-28 19:40:34 +05:30
Ankush Menat
d35d7ffbe2 fix: remove bare exception catching
A bare except catches lots of things (like generator iteration end) and should never be used.
2022-06-28 18:05:00 +05:30
Gavin D'souza
b8d2c195a6 fix: Disable connection pooling via bench commands unless specified 2022-06-23 19:42:12 +05:30
Gavin D'souza
ef078a4ab5 refactor(db-read_only): Track conn type in Database instance 2022-06-23 19:40:17 +05:30
gavin
574f160d3c
Merge branch 'develop' into mariadb-client-refactor 2022-06-23 16:13:03 +05:30
Gavin D'souza
14003e5ac9 refactor: DISABLE_DATABASE_CONNECTION_POOLING conf + var name 2022-06-20 15:31:15 +05:30
Gavin D'souza
a58a5bb848 fix: Improve _transform_query unique key identification 2022-06-20 13:10:13 +05:30
Gavin D'souza
1f1d91a056 fix: Make postgres' last_query lazy decodable 2022-06-18 14:04:38 +05:30
phot0n
f6c1eb10dd refactor(minor): modify_values
* don't truncate float(s)
2022-06-18 10:11:58 +05:30
phot0n
2ce6597a69 fix: cast list to tuple when sending parameterized query for postgres 2022-06-17 18:47:06 +05:30
Gavin D'souza
9b6a048bcd refactor(minor): Use db.db_type instead of conf.db_type 2022-06-17 12:49:12 +05:30
Gavin D'souza
3af8d5caea fix: Add fallbacks for values
Psycopg seems to like None over () and MariaDB - PyMySQL can't seem to
agree on anything - so this seems to keep everyone happy...a very
delicate balance :crie:
2022-06-17 12:47:44 +05:30
Gavin D'souza
90c716bce0 fix(sequence): Setup & use SequenceGeneratorLimitExceeded error 2022-06-17 12:10:43 +05:30
Gavin D'souza
a03bf6b0bb fix(db): Transform values only if not None 2022-06-17 11:31:55 +05:30
Gavin D'souza
1a772e304c fix(db): Store result of last executed query under frappe.db.last_result 2022-06-16 16:01:38 +05:30
Gavin D'souza
25b87a9d49 chore: NoneType alias for < PY310 2022-06-16 15:04:35 +05:30
Gavin D'souza
bc3780560e perf: Use lazy mogrified query for logging
For parameterized queries, there's an improvement of ~30% in query
execution via frappe.db.sql - from 58.6 µs ± 2.37 µs to 44.6 µs ± 1.56 µs
2022-06-16 14:26:34 +05:30
Gavin D'souza
effa942f4c refactor: frappe.db.log_query
* Mogrify queries and set them as frappe.db.last_query instead of
  directly interfacing with the clients
* This is required for now as the MariaDB client uses binary protocol to
  talk to the server and doesn't build the queries itself
* Add typing hints
* Imported Query object as FilterEngine - Query is too ambiguous lol

ref: https://jira.mariadb.org/browse/CONPY-208?focusedCommentId=226873&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-226873
2022-06-16 13:28:54 +05:30
Gavin D'souza
889ced3035 refactor: frappe.db.sql
* Move everything except _cursor.execute outside try-except block - This
  caused multiple traceback printing (by recursion of db.sql)
* Include values mogrifying & executing via client alone in execution
  time for query
* Reduce indentations lol
2022-06-16 12:39:30 +05:30
Gavin D'souza
0adf5e127a fix(db): Track query engine under db._filter_engine 2022-06-16 12:31:14 +05:30
Gavin D'souza
ded55fd98e fix(db): Skip transformation of query if no named params found 2022-06-15 17:43:44 +05:30
Gavin D'souza
5a29177e6b fix(db): Log queried tables through generated query 2022-06-15 17:42:52 +05:30
Gavin D'souza
b4578dc632 Merge branch 'develop' of github.com:frappe/frappe into mariadb-client-refactor 2022-06-15 16:40:29 +05:30
Gavin D'souza
57bd1b6031 fix(db): Import cast as cast_fieldtype to manevour ambiguity 2022-06-15 10:54:48 +05:30
Gavin D'souza
bbc90e6578 refactor!: frappe.db.get_singles_dict
* Don't cast values by default - only if cast kwarg is set
* Reverts breaking behaviour added via f74dc5023d
2022-06-15 10:33:06 +05:30
Gavin D'souza
f74dc5023d refactor!: frappe.db.get_singles_dict
* Cast single's values as their fieldtypes before returning
* Support previously dead debug parameter
* Consider single with no meta as non-existent; skip query

Decided to go ahead with the breaking change given the nature of the
existing usages of get_singles_dict :crie:
2022-06-13 18:39:56 +05:30
Gavin D'souza
13cf4964a6 perf: Check query type via is_query_type 2022-06-13 18:39:56 +05:30
Gavin D'souza
b696fa6da5 perf: Pre-compile and re-use regexp pattern
Converted all possible usages of re.* that weren't compiling the regex
separately and re-using it. Separated out the compiled patterns as
global variables. Repetitive patterns could be made DRY-er.

Would be nicer to have all regexes in a single module so that we could
re-use better, keep track of outdated, and keep checks for possible
reDos' etc
2022-06-08 14:07:38 +05:30
gavin
44dba28159
Merge branch 'develop' into refactor-file 2022-06-08 12:43:35 +05:30
Ankush Menat
94920ab4ae
Merge pull request #16727 from ankush/pg_transaction_management
fix: explicitly start a new transaction
2022-06-07 12:46:23 +05:30
Gavin D'souza
50b0eb8cca perf: Initialize Query engine only if used 2022-06-06 15:42:59 +05:30
Ankush Menat
8861752675 fix: restart transaction after commit 2022-06-05 00:26:12 +05:30
Ankush Menat
654ded1a05 chore: print full stack on postgres DB errors 2022-06-04 22:42:03 +05:30
Gavin D'souza
5ec2690160 Merge branch 'develop' into refactor-file 2022-06-01 15:47:55 +05:30
Gavin D'souza
cf76b944b1 feat(db.query): Add support for List[Dict] filters 2022-05-30 18:46:24 +05:30
gavin
8ffdc2d465 refactor(minor): PostgresDatabase's exceptions
Focus on Non-blocking Postgres exception checking by:

* Use safe getattr to fetch pgcode instead
* Use psycopg errorcodes module to use named variables instead of direct
  codes...for way superior readability xD

Also, moved exceptions out of the main class - just code separation, no
namespace change.
2022-05-25 14:09:31 +05:30
gavin
ff9c89450e feat(db): _transform_query to convert args passed to db.cursor
Transform query & parameters based on client requirements. Eg: MariaDB
client doesn't behave similar to PyMySQL or Psycopg2.

ref: https://jira.mariadb.org/projects/CONPY/issues/CONPY-205
2022-05-25 13:58:23 +05:30
gavin
958fc2b0b2 refactor: Database
* Change query notations - QB > raw
* Update logic of DB APIs - simplify & perf improvements
2022-05-25 12:26:24 +05:30
gavin
4f72eb9eac refactor: Base Database class
* DRY, explicit > implicit usages
* Don't re-compute and do multiple calls for errprint, log, mogrify, etc
* Use consistent logging methods
* Simplify logic / Use newer APIs where applicable
2022-05-25 11:44:08 +05:30
gavin
f7c5c27d81 fix: Use sql_ddl instead of sql
Previous attempts to run this would lead to stalls...long long stalls.
And nothing blocking the query, it would stay in a "Waiting for table
metadata lock" state.
2022-05-24 19:40:27 +05:30
gavin
d87197f042 fix: Correct use of is_connection_pooling_enabled check 2022-05-24 19:19:53 +05:30