Commit graph

223 commits

Author SHA1 Message Date
Gavin D'souza
1a772e304c fix(db): Store result of last executed query under frappe.db.last_result 2022-06-16 16:01:38 +05:30
Gavin D'souza
25b87a9d49 chore: NoneType alias for < PY310 2022-06-16 15:04:35 +05:30
Gavin D'souza
bc3780560e perf: Use lazy mogrified query for logging
For parameterized queries, there's an improvement of ~30% in query
execution via frappe.db.sql - from 58.6 µs ± 2.37 µs to 44.6 µs ± 1.56 µs
2022-06-16 14:26:34 +05:30
Gavin D'souza
effa942f4c refactor: frappe.db.log_query
* Mogrify queries and set them as frappe.db.last_query instead of
  directly interfacing with the clients
* This is required for now as the MariaDB client uses binary protocol to
  talk to the server and doesn't build the queries itself
* Add typing hints
* Imported Query object as FilterEngine - Query is too ambiguous lol

ref: https://jira.mariadb.org/browse/CONPY-208?focusedCommentId=226873&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-226873
2022-06-16 13:28:54 +05:30
Gavin D'souza
889ced3035 refactor: frappe.db.sql
* Move everything except _cursor.execute outside try-except block - This
  caused multiple traceback printing (by recursion of db.sql)
* Include values mogrifying & executing via client alone in execution
  time for query
* Reduce indentations lol
2022-06-16 12:39:30 +05:30
Gavin D'souza
0adf5e127a fix(db): Track query engine under db._filter_engine 2022-06-16 12:31:14 +05:30
Gavin D'souza
5a29177e6b fix(db): Log queried tables through generated query 2022-06-15 17:42:52 +05:30
Gavin D'souza
b4578dc632 Merge branch 'develop' of github.com:frappe/frappe into mariadb-client-refactor 2022-06-15 16:40:29 +05:30
Gavin D'souza
57bd1b6031 fix(db): Import cast as cast_fieldtype to manevour ambiguity 2022-06-15 10:54:48 +05:30
Gavin D'souza
bbc90e6578 refactor!: frappe.db.get_singles_dict
* Don't cast values by default - only if cast kwarg is set
* Reverts breaking behaviour added via f74dc5023d
2022-06-15 10:33:06 +05:30
Gavin D'souza
f74dc5023d refactor!: frappe.db.get_singles_dict
* Cast single's values as their fieldtypes before returning
* Support previously dead debug parameter
* Consider single with no meta as non-existent; skip query

Decided to go ahead with the breaking change given the nature of the
existing usages of get_singles_dict :crie:
2022-06-13 18:39:56 +05:30
Gavin D'souza
13cf4964a6 perf: Check query type via is_query_type 2022-06-13 18:39:56 +05:30
Gavin D'souza
b696fa6da5 perf: Pre-compile and re-use regexp pattern
Converted all possible usages of re.* that weren't compiling the regex
separately and re-using it. Separated out the compiled patterns as
global variables. Repetitive patterns could be made DRY-er.

Would be nicer to have all regexes in a single module so that we could
re-use better, keep track of outdated, and keep checks for possible
reDos' etc
2022-06-08 14:07:38 +05:30
gavin
44dba28159
Merge branch 'develop' into refactor-file 2022-06-08 12:43:35 +05:30
Ankush Menat
94920ab4ae
Merge pull request #16727 from ankush/pg_transaction_management
fix: explicitly start a new transaction
2022-06-07 12:46:23 +05:30
Gavin D'souza
50b0eb8cca perf: Initialize Query engine only if used 2022-06-06 15:42:59 +05:30
Ankush Menat
8861752675 fix: restart transaction after commit 2022-06-05 00:26:12 +05:30
Ankush Menat
654ded1a05 chore: print full stack on postgres DB errors 2022-06-04 22:42:03 +05:30
Gavin D'souza
5ec2690160 Merge branch 'develop' into refactor-file 2022-06-01 15:47:55 +05:30
gavin
ff9c89450e feat(db): _transform_query to convert args passed to db.cursor
Transform query & parameters based on client requirements. Eg: MariaDB
client doesn't behave similar to PyMySQL or Psycopg2.

ref: https://jira.mariadb.org/projects/CONPY/issues/CONPY-205
2022-05-25 13:58:23 +05:30
gavin
958fc2b0b2 refactor: Database
* Change query notations - QB > raw
* Update logic of DB APIs - simplify & perf improvements
2022-05-25 12:26:24 +05:30
gavin
4f72eb9eac refactor: Base Database class
* DRY, explicit > implicit usages
* Don't re-compute and do multiple calls for errprint, log, mogrify, etc
* Use consistent logging methods
* Simplify logic / Use newer APIs where applicable
2022-05-25 11:44:08 +05:30
gavin
dae57d0de2 fix(db)!: Base methods should throw NotImplementedError 2022-05-24 19:19:53 +05:30
gavin
d3900485a6 refactor: Use mariadb client for creating connections
Move to use MariaDB's official Python client written in C instead of
the PyMySQL library. This change doesn't rid Frappe of the PyMySQL
library. Instead, it continues to utilize it for the ER module and
converter methods until the MariaDB library adds support for the same.

Ticket: https://jira.mariadb.org/projects/CONPY/issues/CONPY-203
2022-05-24 19:19:53 +05:30
gavin
7c77bedbf2 refactor: Simplify logic + Add typing hints 2022-05-18 16:57:54 +05:30
Suraj Shetty
1469339335 revert: "refactor: new query engine in get_count()"
Co-Authored-By: Gavin Dsouza <gavin18d@gmail.com>
2022-05-17 13:30:36 +05:30
gavin
d29641f734
Merge branch 'develop' into aks-feat-dbquery_update 2022-05-09 12:02:36 +05:30
Ritwik Puri
b41379c78b
fix: misc fixes (integer primary keys) (#16307)
* fix: misc fixes

local.x gets resetted on every request so switched to a simple dict

simplified is_val_used in set_next_val function for sequences

* chore: use multisql for sequence methods

* fix: fields not updating on form

* minor(base_input): removed unnecessary branching in update_input

* chore: remove prints and rename autoincremented_status_map

* chore: added proper type hint + comment + formatting

* fix: added searching in cast_name rather than handling it manually

* fix: share condition query + test_build_match_conditions

* fix: add cast_name to more places

* test: test for sequence

* fix: sequence functions

* fix: inherit frappetestcase

* minor: attach sequence methods to db context local

* chore: update sequence function names in Database

use frappe.db for sequences in naming.py

* fix: convert filename to str (for autoincremented doctypes)

* chore: better regex for modifying values for postgres

* minor: allow changing name column type (if no data is present in the doctype)

* refactor: validate_autoname

converted it to a simple function
enabled changing autoincrement autoname from customize form

* fix: use sql_ddl for change_column_type in postgres

* fix: use not null constraint in postgres when changing name type

* fix(test): updated test_autoincremented_doctype_transition with transitioning when no data is present

* fix(test): updated test_cast_name

probably messed up during rebase

* fix(test): used rollback upon error in transaction for postgres

* chore: use frappe.db.x methods for sequences

* minor: use temporary sequences in test

* minor: use generate_hash for sequence naming in sequence tests

* chore: replace sequence imports with frappe.db.x

* chore: move out casting name fields to a separate method

* refactor: cast_name

more explicit cases for casts and added docstring

* fix: added space in test_cast_name

* chore: fix linter

* chore: better naming for can_change_name_column_type

* chore: add comment for autoincremented_site_status_map

* chore: update/add docstrings
2022-04-29 15:06:03 +05:30
saxenabhishek
b1a0a3816b style: Applied Frappe linting 2022-04-28 17:10:31 +05:30
saxenabhishek
d39e6a7890 refactor: new query engine in get_count() 2022-04-28 17:10:31 +05:30
saxenabhishek
c9df86f9ef feat: multi table queries 2022-04-28 17:10:31 +05:30
gavin
2da85fa8da
Merge pull request #16698 from gavindsouza/traceback-with-context
feat: Traceback with context
2022-04-22 13:04:07 +05:30
Ankush Menat
dfef7192da
refactor: remove duplicate code from db.get_descendants (#16699) 2022-04-21 15:27:59 +05:30
Gavin D'souza
c691537e61 chore: Add typing for ease of development 2022-04-21 13:32:13 +05:30
Gavin D'souza
1d84483289 Merge branch 'develop' of github.com:frappe/frappe into refactor-file 2022-04-21 11:45:50 +05:30
Gavin D'souza
35c606707e fix(bulk_insert): Cast values as list before subscripting 2022-04-20 16:43:19 +05:30
Pruthvi Patel
8b010e1732
refactor: make frappe.db.bulk_insert work as expected (#16527)
## Issue

`frappe.db.bulk_insert` is not working as expected:
 - It will not insert any row if there are less than 3 values
 - It will not add 1st row at all. e.g if I'm adding 5 values, it will only add 4.
 - It will add values one by one after 2 values, instead it should have inserted items (in db) in chunk of 10000 (as per the code written before).

## Changes Made
 - Solved above issues 
 - use better way to chunk list
 - Added Postgres support for bulk_insert API
 
And now `bulk_insert` will only do **1 db call for each 10000** values.

Note: For testing purpose I made `Test Bulk Insert` doctype and keep chunk size of 100.

## Before
![image](https://user-images.githubusercontent.com/43115036/161979365-c1100745-7602-47d2-a9b8-62d797d2039f.png)

## After
![image](https://user-images.githubusercontent.com/43115036/161978344-3c17d56b-2195-40f4-b00c-e9478d4083f1.png)

nodocs
2022-04-19 14:47:22 +00:00
Gavin D'souza
b014ff8ce6 refactor: Simplify DB/QB API definitions 2022-04-19 19:09:10 +05:30
Gavin D'souza
08402a76f1 fix: Use query.run to execute query instead of db.sql 2022-04-19 19:06:43 +05:30
Gavin D'souza
97e911e069 Merge branch 'develop' of github.com:frappe/frappe into refactor-file 2022-04-13 10:49:26 +05:30
Suraj Shetty
c0c5b2ebdd
style: format all python files using black (#16453)
Co-authored-by: Frappe Bot <developers@frappe.io>
2022-04-12 10:59:25 +05:30
Gavin D'souza
ce5ac24b61 feat(db): Set frappe.db.debug to mass debug queries 2022-04-04 15:15:51 +05:30
gavin
3f6bbd6349
Merge pull request #16474 from resilient-tech/perf-get-doc-filters
perf: reduce query when calling `get_doc` with filters
2022-04-01 15:53:55 +05:30
Sagar Vora
e73c552632 fix: implement for_update for Single documents 2022-04-01 14:15:35 +05:30
Sagar Vora
493c7025f5 fix(get_value): make as_dict work for list-style filters 2022-04-01 00:33:15 +05:30
Ankush Menat
c4ca58fedf refactor: query stripping 2022-03-28 10:56:55 +05:30
Ankush Menat
456cba71cd refactor: expand unreadable one-liner in get_value 2022-03-28 10:56:55 +05:30
Ankush Menat
3e1df641da fix: force keyword arguments in new ORM args 2022-03-28 10:56:55 +05:30
Gavin D'souza
47cf46cd49 Merge branch 'develop' of github.com:frappe/frappe into refactor-file 2022-03-24 21:38:21 +05:30
gavin
a7f1639e65
Merge branch 'develop' into log-settings 2022-03-23 13:20:33 +05:30