Commit graph

91 commits

Author SHA1 Message Date
Ankush Menat
263055e247
fix: site creation using non-root users (#19014) 2022-11-26 12:45:57 +05:30
Ankush Menat
ce360b6fce feat: Set default SQL statement timeouts 2022-11-08 14:51:43 +05:30
Ankush Menat
da561c237d fix: index column should be first 2022-10-06 11:11:03 +05:30
Ankush Menat
cbe4b59100 fix: correct index re-syncing
The implementation of syncing unique and non-unique index depended on
index names which used to be different before because of that there's
tendency to incorrectly identify index.

This PR adds a separate util for checking if a column has index without
relying on naming convention. It just goes and checks if there's any
index with that column in it, hence far more reliable.
2022-10-06 11:11:03 +05:30
Ankush Menat
f96505fae0 feat: wrap read only mode SQL errors 2022-09-09 17:34:44 +05:30
Gavin D'souza
b13298d69c fix(db): Log mogrified queries
Prior to this, queries passed to `frappe.db.sql` with values looked like:

"SELECT `defkey`,`defvalue` FROM `tabDefaultValue` WHERE `parent`=%(param1)s ORDER BY `creation`"

Now, they'll look "normal" or built like:

"SELECT `defkey`,`defvalue` FROM `tabDefaultValue` WHERE `parent`='__global' ORDER BY `creation`"
2022-08-10 18:31:44 +05:30
Ritwik Puri
cb6438158b
fix: don't use cache for sequence in mariadb (#17640)
* fix: don't use cache for sequence in mariadb

* chore: update sequence related comments
2022-07-27 22:46:56 +05:30
Gavin D'souza
2bef29bb46 chore: Deprecate backup script + minor refactors 2022-07-22 13:21:29 +05:30
Gavin D'souza
bcfa8c276e fix: Set default port attribute for Database classes
db.default_port wil be available as a class attribute to hold defaults
for DB types.

Usage: frappe.conf.db_port or frappe.db.default_port
Why: I couldn't run the mariadb command because the defaults aren't set
for my system. server is remote / containerized. Setting port in
equivalent mysql command fixes this.
2022-07-22 13:16:26 +05:30
Gavin D'souza
006ebcbede refactor: Use pymysql over mariadb client
This is supposed to be a temporary switch to make the parent PR easier
to digest. MariaDB client has some issues with release, and system
dependencies.

This commit may be reverted to enable mariadb client again.
2022-07-22 12:10:54 +05:30
Gavin D'souza
8ccc0d039b fix: Database._transform_query
* Don't try to convert None to sequence
* Skip transforming values that mariadb~=1.1 can handle - only worry
  about sequences for now
2022-07-07 15:59:48 +05:30
Gavin D'souza
576fa32af4 fix(mariadb): SequenceGeneratorLimitExceeded is an OperationalError now
:D
2022-07-07 12:09:36 +05:30
Gavin D'souza
41b93392b1 fix: Database._transform_result
Transform data not already casted by mariadb client
2022-07-06 15:58:45 +05:30
Gavin D'souza
4dc2ecefba fix: Use mariadb's constants instead of pymysql 2022-07-05 17:47:54 +05:30
Gavin D'souza
71b5c77e6e Merge branch 'develop' of github.com:frappe/frappe into mariadb-client-refactor 2022-07-05 14:55:02 +05:30
Ankush Menat
261fbfcd11
Revert "fix(doc)!: Always cast datetime, date and time fields"
Revert "fix(doc)!: Always cast datetime, date and time fields (#15891)"

This reverts commit d7789ab6ff.
2022-07-05 13:43:32 +05:30
Himanshu
d7789ab6ff
fix(doc)!: Always cast datetime, date and time fields (#15891)
### BREAKING CHANGE
#### Datetime, Date and Time fields will always be cast to respective objects in `setattr`, this will ensure uniformity while accessing the values, no more `getdate`, `get_datetime`, `to_timedelta` wrapper.
- While importing data, the framework does check for `set_only_once`.
- In normal case scenarios, this will work flawlessly since most date fields might not be set_only_once.
- But in Subscription, the date field is set to `set_only_once` and in `after_insert`, `document.save` is called, and while doing so, `set_only_once` is checked [here](1944a547f9/frappe/model/document.py (L566)).
-This works fine if the data imported is in the correct format.
- If the date's data is not in the correct format, the framework throws an error.
- for eg `06-02-2022 00:00:00 != 06-02-2022`
- fixes [Issue/#15370](https://github.com/frappe/frappe/issues/15370)

> no-docs
2022-07-05 07:07:16 +00:00
Ankush Menat
81b37cb7d2
refactor: clean up code to py310 supported features (#17367)
refactor: clean up code to py39+ supported syntax

- f-strings instead of format
- latest typing support instead of pre 3.9 TitleCase
- remove UTF-8 declarations.
- many more changes

Powered by https://github.com/asottile/pyupgrade/ + manual cleanups
2022-07-01 11:51:05 +05:30
Gavin D'souza
39d30ffeaa fix: Database._transform_query
* Handle dict substitutions in transformations too
* Allow list / tuple values
* Check for values inconsistencies before flattening step
2022-06-30 19:12:10 +05:30
Gavin D'souza
daf3f05fea fix(db): Hanlde sequences in db.sql values 2022-06-29 16:32:00 +05:30
Gavin D'souza
ef078a4ab5 refactor(db-read_only): Track conn type in Database instance 2022-06-23 19:40:17 +05:30
Gavin D'souza
14003e5ac9 refactor: DISABLE_DATABASE_CONNECTION_POOLING conf + var name 2022-06-20 15:31:15 +05:30
Gavin D'souza
a58a5bb848 fix: Improve _transform_query unique key identification 2022-06-20 13:10:13 +05:30
Gavin D'souza
90c716bce0 fix(sequence): Setup & use SequenceGeneratorLimitExceeded error 2022-06-17 12:10:43 +05:30
Gavin D'souza
effa942f4c refactor: frappe.db.log_query
* Mogrify queries and set them as frappe.db.last_query instead of
  directly interfacing with the clients
* This is required for now as the MariaDB client uses binary protocol to
  talk to the server and doesn't build the queries itself
* Add typing hints
* Imported Query object as FilterEngine - Query is too ambiguous lol

ref: https://jira.mariadb.org/browse/CONPY-208?focusedCommentId=226873&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-226873
2022-06-16 13:28:54 +05:30
Gavin D'souza
ded55fd98e fix(db): Skip transformation of query if no named params found 2022-06-15 17:43:44 +05:30
gavin
ff9c89450e feat(db): _transform_query to convert args passed to db.cursor
Transform query & parameters based on client requirements. Eg: MariaDB
client doesn't behave similar to PyMySQL or Psycopg2.

ref: https://jira.mariadb.org/projects/CONPY/issues/CONPY-205
2022-05-25 13:58:23 +05:30
gavin
958fc2b0b2 refactor: Database
* Change query notations - QB > raw
* Update logic of DB APIs - simplify & perf improvements
2022-05-25 12:26:24 +05:30
gavin
d87197f042 fix: Correct use of is_connection_pooling_enabled check 2022-05-24 19:19:53 +05:30
gavin
3ed808aec7 fix: Connection Pooling
* Reduce _POOL_SIZE from 4 to 1. New pools will have just one
  connection. They can scale up as per requirement there after.
* Set auto_connect flag in MariaDB connection - https://mariadb.com/docs/connect/programming-languages/python/connect/
2022-05-24 19:19:53 +05:30
gavin
639fa62138 fix: frappe.DISABLE_DATABASE_POOLING to override frappe.conf.disable_database_connection_pooling
Also, don't pool root connections
2022-05-24 19:19:53 +05:30
gavin
6637a4ebbe fix: Allow non-blocking checks via MariaDBExceptionUtils 2022-05-24 19:19:53 +05:30
gavin
0fe764c9b8 fix: Depend on replica details being there for Replica ConnectionPool
This logic mirror how replica connections are handled
2022-05-24 19:19:53 +05:30
gavin
aaef732581 fix!: Remove frappe.db.create_help_table
Help table has been deprecated for a while. It's safe to get rid of the
API too.
2022-05-24 19:19:53 +05:30
gavin
f021250308 refactor: Move connection & pool management out, inherit instead 2022-05-24 19:19:53 +05:30
gavin
993b935097 fix: Separate pool for replica setup
_SITE_POOLS[frappe.local.site].read_only will hold ConnectionPool for
replica database connections. _SITE_POOLS[frappe.local.site].default
will hold pool for connections that allow read+writes
2022-05-24 19:19:53 +05:30
gavin
363708d7f2 fix: frappe.conf.disable_database_connection_pooling
For the times you don't want to use pooling ;) Calling
get_connection/connect will close all pooled connections
for the current process / worker!
2022-05-24 19:19:53 +05:30
gavin
0d80f6ac52 feat: mariadb connection pooling
Start initial pool of 4 (_POOL_SIZE) connections for a given site. When
the pool is exhausted, generate new connections upon request and add
them to the pool. Max allowed size for each pool is 64 (_MAX_POOL_SIZE)
connections. However, you may have more than 64 active connections, just
that they won't be pooled but destroyed upon job completion/end of
request/similar cycle.
2022-05-24 19:19:53 +05:30
gavin
a17c978065 refactor!: frappe.db.get_database_list
* Drop mandatory unused arg in method call
* Use pluck instead of list comprehension + subscripting
2022-05-24 19:19:53 +05:30
gavin
756e385362 refactor: Move exception detection & abstraction in separate class
This change has only been done to separate and club only like methods /
utils together
2022-05-24 19:19:53 +05:30
gavin
1fe3624c43 fix: DB error detection API usage
* Make all methods static
* Add typing hints
* Don't safe fetch attribute value for errno - MariaDB exceptions if
  raised will have errno attr in them. If the class doesn't have one,
  it's not meant to be passed in these methods.
2022-05-24 19:19:53 +05:30
gavin
d3900485a6 refactor: Use mariadb client for creating connections
Move to use MariaDB's official Python client written in C instead of
the PyMySQL library. This change doesn't rid Frappe of the PyMySQL
library. Instead, it continues to utilize it for the ER module and
converter methods until the MariaDB library adds support for the same.

Ticket: https://jira.mariadb.org/projects/CONPY/issues/CONPY-203
2022-05-24 19:19:53 +05:30
phot0n
f6dac70033 refactor(minor): convert change_name_type_and_make_sequence function to setup_autoincrement_and_sequence method
* chore(Customize Form): remove autoincrement naming option from autoname description

* chore: move sequence cache comment from schema.py to database.py

* chore: added docstrings to some functions
2022-05-13 00:03:19 +05:30
phot0n
9f0c40dbba fix: create sequence onchange from any other autoname whcih is not autoincrement
* chore: use class variable for determining sequence cache

* chore: use sql_ddl when creating sequence
2022-05-05 21:31:13 +05:30
Ritwik Puri
b41379c78b
fix: misc fixes (integer primary keys) (#16307)
* fix: misc fixes

local.x gets resetted on every request so switched to a simple dict

simplified is_val_used in set_next_val function for sequences

* chore: use multisql for sequence methods

* fix: fields not updating on form

* minor(base_input): removed unnecessary branching in update_input

* chore: remove prints and rename autoincremented_status_map

* chore: added proper type hint + comment + formatting

* fix: added searching in cast_name rather than handling it manually

* fix: share condition query + test_build_match_conditions

* fix: add cast_name to more places

* test: test for sequence

* fix: sequence functions

* fix: inherit frappetestcase

* minor: attach sequence methods to db context local

* chore: update sequence function names in Database

use frappe.db for sequences in naming.py

* fix: convert filename to str (for autoincremented doctypes)

* chore: better regex for modifying values for postgres

* minor: allow changing name column type (if no data is present in the doctype)

* refactor: validate_autoname

converted it to a simple function
enabled changing autoincrement autoname from customize form

* fix: use sql_ddl for change_column_type in postgres

* fix: use not null constraint in postgres when changing name type

* fix(test): updated test_autoincremented_doctype_transition with transitioning when no data is present

* fix(test): updated test_cast_name

probably messed up during rebase

* fix(test): used rollback upon error in transaction for postgres

* chore: use frappe.db.x methods for sequences

* minor: use temporary sequences in test

* minor: use generate_hash for sequence naming in sequence tests

* chore: replace sequence imports with frappe.db.x

* chore: move out casting name fields to a separate method

* refactor: cast_name

more explicit cases for casts and added docstring

* fix: added space in test_cast_name

* chore: fix linter

* chore: better naming for can_change_name_column_type

* chore: add comment for autoincremented_site_status_map

* chore: update/add docstrings
2022-04-29 15:06:03 +05:30
Suraj Shetty
6b7fda495b Merge branch 'develop' of https://github.com/frappe/frappe into phone_field_control 2022-04-14 14:19:35 +05:30
Shridhar Patil
39f8267a15
feat: added support for data type json (#16187)
> Please provide enough information so that others can review your pull request:

Added json support for postgres and mariadb



> Explain the **details** for making this change. What existing problem does the pull request solve?
https://github.com/frappe/frappe/projects/4#card-50160428


> Screenshots/GIFs
![json](https://user-images.githubusercontent.com/11792643/156367383-8f8492c2-3817-449d-a2dd-c983eeadbb48.gif)



---

**Previous attempts:** 

https://github.com/frappe/frappe/pull/8128
https://github.com/frappe/frappe/pull/7096


Docs: https://frappeframework.com/docs/v13/user/en/basics/doctypes/fieldtypes#json
2022-04-14 04:21:41 +00:00
Noah Jacob
c2f2fc10e7 Merge branch develop into phone_field_control 2022-04-12 20:40:29 +05:30
Suraj Shetty
c0c5b2ebdd
style: format all python files using black (#16453)
Co-authored-by: Frappe Bot <developers@frappe.io>
2022-04-12 10:59:25 +05:30
Noah Jacob
579d0643c4 Merge branch 'develop' into phone_field_control 2022-03-22 15:05:45 +05:30