vassili/seitime-frappe
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 4
38627 commits 1 branch 0 tags 587 MiB
Commit graph

488 commits

Author SHA1 Message Date
Gavin D'souza
ef078a4ab5 refactor(db-read_only): Track conn type in Database instance 2022-06-23 19:40:17 +05:30
gavin
574f160d3c
Merge branch 'develop' into mariadb-client-refactor 2022-06-23 16:13:03 +05:30
Gavin D'souza
14003e5ac9 refactor: DISABLE_DATABASE_CONNECTION_POOLING conf + var name 2022-06-20 15:31:15 +05:30
Gavin D'souza
a58a5bb848 fix: Improve _transform_query unique key identification 2022-06-20 13:10:13 +05:30
Gavin D'souza
1f1d91a056 fix: Make postgres' last_query lazy decodable 2022-06-18 14:04:38 +05:30
phot0n
f6c1eb10dd refactor(minor): modify_values 
* don't truncate float(s)
 2022-06-18 10:11:58 +05:30
phot0n
2ce6597a69 fix: cast list to tuple when sending parameterized query for postgres 2022-06-17 18:47:06 +05:30
Gavin D'souza
9b6a048bcd refactor(minor): Use db.db_type instead of conf.db_type 2022-06-17 12:49:12 +05:30
Gavin D'souza
3af8d5caea fix: Add fallbacks for values 
Psycopg seems to like None over () and MariaDB - PyMySQL can't seem to
agree on anything - so this seems to keep everyone happy...a very
delicate balance :crie:
 2022-06-17 12:47:44 +05:30
Gavin D'souza
90c716bce0 fix(sequence): Setup & use SequenceGeneratorLimitExceeded error 2022-06-17 12:10:43 +05:30
Gavin D'souza
a03bf6b0bb fix(db): Transform values only if not None 2022-06-17 11:31:55 +05:30
Gavin D'souza
1a772e304c fix(db): Store result of last executed query under frappe.db.last_result 2022-06-16 16:01:38 +05:30
Gavin D'souza
25b87a9d49 chore: NoneType alias for < PY310 2022-06-16 15:04:35 +05:30
Gavin D'souza
bc3780560e perf: Use lazy mogrified query for logging 
For parameterized queries, there's an improvement of ~30% in query
execution via frappe.db.sql - from 58.6 µs ± 2.37 µs to 44.6 µs ± 1.56 µs
 2022-06-16 14:26:34 +05:30
Gavin D'souza
effa942f4c refactor: frappe.db.log_query 
* Mogrify queries and set them as frappe.db.last_query instead of
  directly interfacing with the clients
* This is required for now as the MariaDB client uses binary protocol to
  talk to the server and doesn't build the queries itself
* Add typing hints
* Imported Query object as FilterEngine - Query is too ambiguous lol

ref: https://jira.mariadb.org/browse/CONPY-208?focusedCommentId=226873&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-226873
 2022-06-16 13:28:54 +05:30
Gavin D'souza
889ced3035 refactor: frappe.db.sql 
* Move everything except _cursor.execute outside try-except block - This
  caused multiple traceback printing (by recursion of db.sql)
* Include values mogrifying & executing via client alone in execution
  time for query
* Reduce indentations lol
 2022-06-16 12:39:30 +05:30
Gavin D'souza
0adf5e127a fix(db): Track query engine under db._filter_engine 2022-06-16 12:31:14 +05:30
Gavin D'souza
ded55fd98e fix(db): Skip transformation of query if no named params found 2022-06-15 17:43:44 +05:30
Gavin D'souza
5a29177e6b fix(db): Log queried tables through generated query 2022-06-15 17:42:52 +05:30
Gavin D'souza
b4578dc632 Merge branch 'develop' of github.com:frappe/frappe into mariadb-client-refactor 2022-06-15 16:40:29 +05:30
Gavin D'souza
57bd1b6031 fix(db): Import cast as cast_fieldtype to manevour ambiguity 2022-06-15 10:54:48 +05:30
Gavin D'souza
bbc90e6578 refactor!: frappe.db.get_singles_dict 
* Don't cast values by default - only if cast kwarg is set
* Reverts breaking behaviour added via f74dc5023d
 2022-06-15 10:33:06 +05:30
Gavin D'souza
f74dc5023d refactor!: frappe.db.get_singles_dict 
* Cast single's values as their fieldtypes before returning
* Support previously dead debug parameter
* Consider single with no meta as non-existent; skip query

Decided to go ahead with the breaking change given the nature of the
existing usages of get_singles_dict :crie:
 2022-06-13 18:39:56 +05:30
Gavin D'souza
13cf4964a6 perf: Check query type via is_query_type 2022-06-13 18:39:56 +05:30
Gavin D'souza
b696fa6da5 perf: Pre-compile and re-use regexp pattern 
Converted all possible usages of re.* that weren't compiling the regex
separately and re-using it. Separated out the compiled patterns as
global variables. Repetitive patterns could be made DRY-er.

Would be nicer to have all regexes in a single module so that we could
re-use better, keep track of outdated, and keep checks for possible
reDos' etc
 2022-06-08 14:07:38 +05:30
gavin
44dba28159
Merge branch 'develop' into refactor-file 2022-06-08 12:43:35 +05:30
Ankush Menat
94920ab4ae
Merge pull request #16727 from ankush/pg_transaction_management 
fix: explicitly start a new transaction
 2022-06-07 12:46:23 +05:30
Gavin D'souza
50b0eb8cca perf: Initialize Query engine only if used 2022-06-06 15:42:59 +05:30
Ankush Menat
8861752675 fix: restart transaction after commit 2022-06-05 00:26:12 +05:30
Ankush Menat
654ded1a05 chore: print full stack on postgres DB errors 2022-06-04 22:42:03 +05:30
Gavin D'souza
5ec2690160 Merge branch 'develop' into refactor-file 2022-06-01 15:47:55 +05:30
Gavin D'souza
cf76b944b1 feat(db.query): Add support for List[Dict] filters 2022-05-30 18:46:24 +05:30
gavin
8ffdc2d465 refactor(minor): PostgresDatabase's exceptions 
Focus on Non-blocking Postgres exception checking by:

* Use safe getattr to fetch pgcode instead
* Use psycopg errorcodes module to use named variables instead of direct
  codes...for way superior readability xD

Also, moved exceptions out of the main class - just code separation, no
namespace change.
 2022-05-25 14:09:31 +05:30
gavin
ff9c89450e feat(db): _transform_query to convert args passed to db.cursor 
Transform query & parameters based on client requirements. Eg: MariaDB
client doesn't behave similar to PyMySQL or Psycopg2.

ref: https://jira.mariadb.org/projects/CONPY/issues/CONPY-205
 2022-05-25 13:58:23 +05:30
gavin
958fc2b0b2 refactor: Database 
* Change query notations - QB > raw
* Update logic of DB APIs - simplify & perf improvements
 2022-05-25 12:26:24 +05:30
gavin
4f72eb9eac refactor: Base Database class 
* DRY, explicit > implicit usages
* Don't re-compute and do multiple calls for errprint, log, mogrify, etc
* Use consistent logging methods
* Simplify logic / Use newer APIs where applicable
 2022-05-25 11:44:08 +05:30
gavin
f7c5c27d81 fix: Use sql_ddl instead of sql 
Previous attempts to run this would lead to stalls...long long stalls.
And nothing blocking the query, it would stay in a "Waiting for table
metadata lock" state.
 2022-05-24 19:40:27 +05:30
gavin
d87197f042 fix: Correct use of is_connection_pooling_enabled check 2022-05-24 19:19:53 +05:30
gavin
3ed808aec7 fix: Connection Pooling 
* Reduce _POOL_SIZE from 4 to 1. New pools will have just one
  connection. They can scale up as per requirement there after.
* Set auto_connect flag in MariaDB connection - https://mariadb.com/docs/connect/programming-languages/python/connect/
 2022-05-24 19:19:53 +05:30
gavin
a7838ccca4 fix: Remove half baked support for filters_config in db.query 2022-05-24 19:19:53 +05:30
gavin
32a30ce933 refactor: DBManager 
* Simplify logic: DRY, lesser indentation & all DAT
* Utilize newer APIs, f-strings & more
* Cleaner namespace
* Conform inconsistent behaviours
 2022-05-24 19:19:53 +05:30
gavin
639fa62138 fix: frappe.DISABLE_DATABASE_POOLING to override frappe.conf.disable_database_connection_pooling 
Also, don't pool root connections
 2022-05-24 19:19:53 +05:30
gavin
6637a4ebbe fix: Allow non-blocking checks via MariaDBExceptionUtils 2022-05-24 19:19:53 +05:30
gavin
0fe764c9b8 fix: Depend on replica details being there for Replica ConnectionPool 
This logic mirror how replica connections are handled
 2022-05-24 19:19:53 +05:30
gavin
aaef732581 fix!: Remove frappe.db.create_help_table 
Help table has been deprecated for a while. It's safe to get rid of the
API too.
 2022-05-24 19:19:53 +05:30
gavin
f021250308 refactor: Move connection & pool management out, inherit instead 2022-05-24 19:19:53 +05:30
gavin
993b935097 fix: Separate pool for replica setup 
_SITE_POOLS[frappe.local.site].read_only will hold ConnectionPool for
replica database connections. _SITE_POOLS[frappe.local.site].default
will hold pool for connections that allow read+writes
 2022-05-24 19:19:53 +05:30
gavin
363708d7f2 fix: frappe.conf.disable_database_connection_pooling 
For the times you don't want to use pooling ;) Calling
get_connection/connect will close all pooled connections
for the current process / worker!
 2022-05-24 19:19:53 +05:30
gavin
dae57d0de2 fix(db)!: Base methods should throw NotImplementedError 2022-05-24 19:19:53 +05:30
gavin
0d80f6ac52 feat: mariadb connection pooling 
Start initial pool of 4 (_POOL_SIZE) connections for a given site. When
the pool is exhausted, generate new connections upon request and add
them to the pool. Max allowed size for each pool is 64 (_MAX_POOL_SIZE)
connections. However, you may have more than 64 active connections, just
that they won't be pooled but destroyed upon job completion/end of
request/similar cycle.
 2022-05-24 19:19:53 +05:30