Commit graph

866 commits

Author SHA1 Message Date
Gavin D'souza
f74dc5023d refactor!: frappe.db.get_singles_dict
* Cast single's values as their fieldtypes before returning
* Support previously dead debug parameter
* Consider single with no meta as non-existent; skip query

Decided to go ahead with the breaking change given the nature of the
existing usages of get_singles_dict :crie:
2022-06-13 18:39:56 +05:30
Gavin D'souza
13cf4964a6 perf: Check query type via is_query_type 2022-06-13 18:39:56 +05:30
Gavin D'souza
b696fa6da5 perf: Pre-compile and re-use regexp pattern
Converted all possible usages of re.* that weren't compiling the regex
separately and re-using it. Separated out the compiled patterns as
global variables. Repetitive patterns could be made DRY-er.

Would be nicer to have all regexes in a single module so that we could
re-use better, keep track of outdated, and keep checks for possible
reDos' etc
2022-06-08 14:07:38 +05:30
gavin
44dba28159
Merge branch 'develop' into refactor-file 2022-06-08 12:43:35 +05:30
Ankush Menat
94920ab4ae
Merge pull request #16727 from ankush/pg_transaction_management
fix: explicitly start a new transaction
2022-06-07 12:46:23 +05:30
Gavin D'souza
50b0eb8cca perf: Initialize Query engine only if used 2022-06-06 15:42:59 +05:30
Ankush Menat
8861752675 fix: restart transaction after commit 2022-06-05 00:26:12 +05:30
Ankush Menat
654ded1a05 chore: print full stack on postgres DB errors 2022-06-04 22:42:03 +05:30
Gavin D'souza
5ec2690160 Merge branch 'develop' into refactor-file 2022-06-01 15:47:55 +05:30
Gavin D'souza
cf76b944b1 feat(db.query): Add support for List[Dict] filters 2022-05-30 18:46:24 +05:30
gavin
8ffdc2d465 refactor(minor): PostgresDatabase's exceptions
Focus on Non-blocking Postgres exception checking by:

* Use safe getattr to fetch pgcode instead
* Use psycopg errorcodes module to use named variables instead of direct
  codes...for way superior readability xD

Also, moved exceptions out of the main class - just code separation, no
namespace change.
2022-05-25 14:09:31 +05:30
gavin
ff9c89450e feat(db): _transform_query to convert args passed to db.cursor
Transform query & parameters based on client requirements. Eg: MariaDB
client doesn't behave similar to PyMySQL or Psycopg2.

ref: https://jira.mariadb.org/projects/CONPY/issues/CONPY-205
2022-05-25 13:58:23 +05:30
gavin
958fc2b0b2 refactor: Database
* Change query notations - QB > raw
* Update logic of DB APIs - simplify & perf improvements
2022-05-25 12:26:24 +05:30
gavin
4f72eb9eac refactor: Base Database class
* DRY, explicit > implicit usages
* Don't re-compute and do multiple calls for errprint, log, mogrify, etc
* Use consistent logging methods
* Simplify logic / Use newer APIs where applicable
2022-05-25 11:44:08 +05:30
gavin
f7c5c27d81 fix: Use sql_ddl instead of sql
Previous attempts to run this would lead to stalls...long long stalls.
And nothing blocking the query, it would stay in a "Waiting for table
metadata lock" state.
2022-05-24 19:40:27 +05:30
gavin
d87197f042 fix: Correct use of is_connection_pooling_enabled check 2022-05-24 19:19:53 +05:30
gavin
3ed808aec7 fix: Connection Pooling
* Reduce _POOL_SIZE from 4 to 1. New pools will have just one
  connection. They can scale up as per requirement there after.
* Set auto_connect flag in MariaDB connection - https://mariadb.com/docs/connect/programming-languages/python/connect/
2022-05-24 19:19:53 +05:30
gavin
a7838ccca4 fix: Remove half baked support for filters_config in db.query 2022-05-24 19:19:53 +05:30
gavin
32a30ce933 refactor: DBManager
* Simplify logic: DRY, lesser indentation & all DAT
* Utilize newer APIs, f-strings & more
* Cleaner namespace
* Conform inconsistent behaviours
2022-05-24 19:19:53 +05:30
gavin
639fa62138 fix: frappe.DISABLE_DATABASE_POOLING to override frappe.conf.disable_database_connection_pooling
Also, don't pool root connections
2022-05-24 19:19:53 +05:30
gavin
6637a4ebbe fix: Allow non-blocking checks via MariaDBExceptionUtils 2022-05-24 19:19:53 +05:30
gavin
0fe764c9b8 fix: Depend on replica details being there for Replica ConnectionPool
This logic mirror how replica connections are handled
2022-05-24 19:19:53 +05:30
gavin
aaef732581 fix!: Remove frappe.db.create_help_table
Help table has been deprecated for a while. It's safe to get rid of the
API too.
2022-05-24 19:19:53 +05:30
gavin
f021250308 refactor: Move connection & pool management out, inherit instead 2022-05-24 19:19:53 +05:30
gavin
993b935097 fix: Separate pool for replica setup
_SITE_POOLS[frappe.local.site].read_only will hold ConnectionPool for
replica database connections. _SITE_POOLS[frappe.local.site].default
will hold pool for connections that allow read+writes
2022-05-24 19:19:53 +05:30
gavin
363708d7f2 fix: frappe.conf.disable_database_connection_pooling
For the times you don't want to use pooling ;) Calling
get_connection/connect will close all pooled connections
for the current process / worker!
2022-05-24 19:19:53 +05:30
gavin
dae57d0de2 fix(db)!: Base methods should throw NotImplementedError 2022-05-24 19:19:53 +05:30
gavin
0d80f6ac52 feat: mariadb connection pooling
Start initial pool of 4 (_POOL_SIZE) connections for a given site. When
the pool is exhausted, generate new connections upon request and add
them to the pool. Max allowed size for each pool is 64 (_MAX_POOL_SIZE)
connections. However, you may have more than 64 active connections, just
that they won't be pooled but destroyed upon job completion/end of
request/similar cycle.
2022-05-24 19:19:53 +05:30
gavin
a17c978065 refactor!: frappe.db.get_database_list
* Drop mandatory unused arg in method call
* Use pluck instead of list comprehension + subscripting
2022-05-24 19:19:53 +05:30
gavin
756e385362 refactor: Move exception detection & abstraction in separate class
This change has only been done to separate and club only like methods /
utils together
2022-05-24 19:19:53 +05:30
gavin
1fe3624c43 fix: DB error detection API usage
* Make all methods static
* Add typing hints
* Don't safe fetch attribute value for errno - MariaDB exceptions if
  raised will have errno attr in them. If the class doesn't have one,
  it's not meant to be passed in these methods.
2022-05-24 19:19:53 +05:30
gavin
d3900485a6 refactor: Use mariadb client for creating connections
Move to use MariaDB's official Python client written in C instead of
the PyMySQL library. This change doesn't rid Frappe of the PyMySQL
library. Instead, it continues to utilize it for the ER module and
converter methods until the MariaDB library adds support for the same.

Ticket: https://jira.mariadb.org/projects/CONPY/issues/CONPY-203
2022-05-24 19:19:53 +05:30
gavin
15af545acf chore: Update typing + noqa - flake8 failures 2022-05-23 18:55:55 +05:30
gavin
6dedf899ff
Merge pull request #16899 from ChillarAnand/imports
refactor: Removed unused imports & added flake8 check in CI
2022-05-23 15:53:18 +05:30
gavin
a20800ee2b fix: mport module from line N shadowed by loop variable
Flake8 F402 reported by sider
2022-05-19 17:45:22 +05:30
gavin
db11af2a5c feat(wip): Custom filters in db.query engine
* Added provision for semi-implemneted version
    * Hard to fix it completely given it's broken on develop / desk
* Added TODO for adding nestedset related filters
2022-05-19 17:31:35 +05:30
chillaranand
1d763a6659 refactor: Fix flake8 issues 2022-05-19 15:34:35 +05:30
gavin
72ad582b8a fix(db.query): Casefold key to match operators 2022-05-18 16:59:53 +05:30
gavin
5ebaf42468 feat(db.query): Add support for timespan operator 2022-05-18 16:59:10 +05:30
gavin
7c77bedbf2 refactor: Simplify logic + Add typing hints 2022-05-18 16:57:54 +05:30
Aradhya Tripathi
7a0dc40693
feat!: Added support for IS in frappe qb (#16877)
Fixes #16869
Using 53148c3f90
2022-05-18 09:09:22 +00:00
Suraj Shetty
84122c6eec fix: Pass field object instead of string 2022-05-17 18:27:19 +05:30
Suraj Shetty
1469339335 revert: "refactor: new query engine in get_count()"
Co-Authored-By: Gavin Dsouza <gavin18d@gmail.com>
2022-05-17 13:30:36 +05:30
Himanshu
70409a3c7b
fix: set translated text to link field (#15451)
- Fixes [Issue/15324](https://github.com/frappe/frappe/issues/15324)
- When selecting a value for Link Field, the English text was set as link field value.
- This PR aims to fix it by setting the translated text as link field value on selection.

Todo:

- [x] Show translated text in the select dropdown
- [x] Set translated value to Link field on select
- [x] Show original value when link field is in focus
- [x] Add option to toggle this behaviour

### Behaviour

- Link field loses focus: show the translated name.
- Link field is focused
    - If old value is present
        - If options **are** one of [Role, DocType]: show the translated name
        - Else: show the name (untranslated) to enable search in untranslated values
    - Else: show what the user typed (untranslated) to enable search in untranslated values
- Value is selected: link field loses focus

## Demo

### Link to UOM in a custom Item DocType

The UOM names are in english, so the search needs to happen in english. When possible, the translation is displayed.

#### Before

https://user-images.githubusercontent.com/14891507/156415248-e5e80d05-53dc-4ca8-89c7-998986ff6e99.mov

#### After

https://user-images.githubusercontent.com/14891507/156410386-a874430c-f340-43ed-9c3a-92e8d4d50fc9.mov

### Link to DocType in Customize Form

The DocType names get translated before being searched. This is a preexisting hack in the framework for DocType and Role. In this case, we can search in the translations.

#### Before

https://user-images.githubusercontent.com/14891507/156414648-8e505f8c-9dee-4358-8182-3b358c28bb62.mov

#### After

https://user-images.githubusercontent.com/14891507/156411881-c4ca22e1-1397-4e13-9768-5e16b72f8d6d.mov

https://docs.erpnext.com/docs/v13/user/manual/en/customize-erpnext/customize-form/edit?wiki_page_patch=fdafee2715
2022-05-16 13:21:42 +00:00
Ritwik Puri
968343b302
Merge branch 'develop' into autoinc-customize-form-fix 2022-05-13 00:41:58 +05:30
phot0n
f6dac70033 refactor(minor): convert change_name_type_and_make_sequence function to setup_autoincrement_and_sequence method
* chore(Customize Form): remove autoincrement naming option from autoname description

* chore: move sequence cache comment from schema.py to database.py

* chore: added docstrings to some functions
2022-05-13 00:03:19 +05:30
gavin
d29641f734
Merge branch 'develop' into aks-feat-dbquery_update 2022-05-09 12:02:36 +05:30
phot0n
cfc905b567 fix: use cast when chaninging name type from varchar to bigint in postgres 2022-05-05 21:31:13 +05:30
phot0n
9f0c40dbba fix: create sequence onchange from any other autoname whcih is not autoincrement
* chore: use class variable for determining sequence cache

* chore: use sql_ddl when creating sequence
2022-05-05 21:31:13 +05:30
Ritwik Puri
b41379c78b
fix: misc fixes (integer primary keys) (#16307)
* fix: misc fixes

local.x gets resetted on every request so switched to a simple dict

simplified is_val_used in set_next_val function for sequences

* chore: use multisql for sequence methods

* fix: fields not updating on form

* minor(base_input): removed unnecessary branching in update_input

* chore: remove prints and rename autoincremented_status_map

* chore: added proper type hint + comment + formatting

* fix: added searching in cast_name rather than handling it manually

* fix: share condition query + test_build_match_conditions

* fix: add cast_name to more places

* test: test for sequence

* fix: sequence functions

* fix: inherit frappetestcase

* minor: attach sequence methods to db context local

* chore: update sequence function names in Database

use frappe.db for sequences in naming.py

* fix: convert filename to str (for autoincremented doctypes)

* chore: better regex for modifying values for postgres

* minor: allow changing name column type (if no data is present in the doctype)

* refactor: validate_autoname

converted it to a simple function
enabled changing autoincrement autoname from customize form

* fix: use sql_ddl for change_column_type in postgres

* fix: use not null constraint in postgres when changing name type

* fix(test): updated test_autoincremented_doctype_transition with transitioning when no data is present

* fix(test): updated test_cast_name

probably messed up during rebase

* fix(test): used rollback upon error in transaction for postgres

* chore: use frappe.db.x methods for sequences

* minor: use temporary sequences in test

* minor: use generate_hash for sequence naming in sequence tests

* chore: replace sequence imports with frappe.db.x

* chore: move out casting name fields to a separate method

* refactor: cast_name

more explicit cases for casts and added docstring

* fix: added space in test_cast_name

* chore: fix linter

* chore: better naming for can_change_name_column_type

* chore: add comment for autoincremented_site_status_map

* chore: update/add docstrings
2022-04-29 15:06:03 +05:30