Commit graph

158 commits

Author SHA1 Message Date
Sagar Vora
4ea39d1a68
chore!: drop device support from session (#18729)
* chore!: drop device support from session

* chore: remove `session_expiry_mobile` field

* fix: remove extra `%s`
2022-12-05 16:37:32 +05:30
Ankush Menat
186b88a8cb
build: Bump minimum required MariaDB version (#19116) 2022-12-05 16:24:35 +05:30
Ankush Menat
9c104a8e5e fix: Warn about incompatible mariadb versions 2022-12-03 12:20:35 +05:30
Ankush Menat
11cdd21b88 fix: drop mariadb 10.2 checks 2022-12-03 12:20:35 +05:30
Ankush Menat
263055e247
fix: site creation using non-root users (#19014) 2022-11-26 12:45:57 +05:30
Ankush Menat
ce360b6fce feat: Set default SQL statement timeouts 2022-11-08 14:51:43 +05:30
Ankush Menat
8cca50bf26
refactor: remove naming case feature (#18672)
This has never worked since 2016... which can mean two things:
- No one really uses this.
- If I fix this now suddenly people will find different behaviour in
  naming because `name_case` is selected in some doctypes (but never
  tested)
2022-10-31 11:23:49 +05:30
Ritwik Puri
dad080842a
fix: don't print multiple print statements when database settings dont match with expected settings (#18492)
Also, fetch mariadb variables only once when checking mariadb variables
2022-10-22 21:55:12 +05:30
Ankush Menat
da561c237d fix: index column should be first 2022-10-06 11:11:03 +05:30
Ankush Menat
cbe4b59100 fix: correct index re-syncing
The implementation of syncing unique and non-unique index depended on
index names which used to be different before because of that there's
tendency to incorrectly identify index.

This PR adds a separate util for checking if a column has index without
relying on naming convention. It just goes and checks if there's any
index with that column in it, hence far more reliable.
2022-10-06 11:11:03 +05:30
Ankush Menat
6de41a78e9 refactor: guard clause 2022-10-06 11:11:03 +05:30
Ankush Menat
82cd238f54
perf!: Only add modified index on parent doctypes (#18119)
When are child tables sorted by timestamp by default?
2022-09-15 19:16:12 +05:30
Ankush Menat
f96505fae0 feat: wrap read only mode SQL errors 2022-09-09 17:34:44 +05:30
Himanshu
3faaf5f361
Merge branch 'develop' into merge_translated_doctypes 2022-08-12 13:10:49 +01:00
Gavin D'souza
91a47c2602 fix: Deprecate setup help database APIs
APIs have been unused since ~v4/6? Don't see any need to keep them in.
2022-08-12 12:52:19 +05:30
Himanshu
e68aac866f
Merge branch 'develop' into merge_translated_doctypes 2022-08-10 18:49:04 +01:00
Gavin D'souza
b13298d69c fix(db): Log mogrified queries
Prior to this, queries passed to `frappe.db.sql` with values looked like:

"SELECT `defkey`,`defvalue` FROM `tabDefaultValue` WHERE `parent`=%(param1)s ORDER BY `creation`"

Now, they'll look "normal" or built like:

"SELECT `defkey`,`defvalue` FROM `tabDefaultValue` WHERE `parent`='__global' ORDER BY `creation`"
2022-08-10 18:31:44 +05:30
hrwx
b01929405d refactor: translatable doctypes 2022-08-05 21:38:10 +01:00
Ritwik Puri
cb6438158b
fix: don't use cache for sequence in mariadb (#17640)
* fix: don't use cache for sequence in mariadb

* chore: update sequence related comments
2022-07-27 22:46:56 +05:30
Gavin D'souza
2bef29bb46 chore: Deprecate backup script + minor refactors 2022-07-22 13:21:29 +05:30
Gavin D'souza
bcfa8c276e fix: Set default port attribute for Database classes
db.default_port wil be available as a class attribute to hold defaults
for DB types.

Usage: frappe.conf.db_port or frappe.db.default_port
Why: I couldn't run the mariadb command because the defaults aren't set
for my system. server is remote / containerized. Setting port in
equivalent mysql command fixes this.
2022-07-22 13:16:26 +05:30
Gavin D'souza
006ebcbede refactor: Use pymysql over mariadb client
This is supposed to be a temporary switch to make the parent PR easier
to digest. MariaDB client has some issues with release, and system
dependencies.

This commit may be reverted to enable mariadb client again.
2022-07-22 12:10:54 +05:30
Gavin D'souza
8ccc0d039b fix: Database._transform_query
* Don't try to convert None to sequence
* Skip transforming values that mariadb~=1.1 can handle - only worry
  about sequences for now
2022-07-07 15:59:48 +05:30
Gavin D'souza
576fa32af4 fix(mariadb): SequenceGeneratorLimitExceeded is an OperationalError now
:D
2022-07-07 12:09:36 +05:30
Gavin D'souza
41b93392b1 fix: Database._transform_result
Transform data not already casted by mariadb client
2022-07-06 15:58:45 +05:30
Gavin D'souza
4dc2ecefba fix: Use mariadb's constants instead of pymysql 2022-07-05 17:47:54 +05:30
Gavin D'souza
71b5c77e6e Merge branch 'develop' of github.com:frappe/frappe into mariadb-client-refactor 2022-07-05 14:55:02 +05:30
Ankush Menat
261fbfcd11
Revert "fix(doc)!: Always cast datetime, date and time fields"
Revert "fix(doc)!: Always cast datetime, date and time fields (#15891)"

This reverts commit d7789ab6ff.
2022-07-05 13:43:32 +05:30
Himanshu
d7789ab6ff
fix(doc)!: Always cast datetime, date and time fields (#15891)
### BREAKING CHANGE
#### Datetime, Date and Time fields will always be cast to respective objects in `setattr`, this will ensure uniformity while accessing the values, no more `getdate`, `get_datetime`, `to_timedelta` wrapper.
- While importing data, the framework does check for `set_only_once`.
- In normal case scenarios, this will work flawlessly since most date fields might not be set_only_once.
- But in Subscription, the date field is set to `set_only_once` and in `after_insert`, `document.save` is called, and while doing so, `set_only_once` is checked [here](1944a547f9/frappe/model/document.py (L566)).
-This works fine if the data imported is in the correct format.
- If the date's data is not in the correct format, the framework throws an error.
- for eg `06-02-2022 00:00:00 != 06-02-2022`
- fixes [Issue/#15370](https://github.com/frappe/frappe/issues/15370)

> no-docs
2022-07-05 07:07:16 +00:00
Ankush Menat
81b37cb7d2
refactor: clean up code to py310 supported features (#17367)
refactor: clean up code to py39+ supported syntax

- f-strings instead of format
- latest typing support instead of pre 3.9 TitleCase
- remove UTF-8 declarations.
- many more changes

Powered by https://github.com/asottile/pyupgrade/ + manual cleanups
2022-07-01 11:51:05 +05:30
Gavin D'souza
39d30ffeaa fix: Database._transform_query
* Handle dict substitutions in transformations too
* Allow list / tuple values
* Check for values inconsistencies before flattening step
2022-06-30 19:12:10 +05:30
Gavin D'souza
daf3f05fea fix(db): Hanlde sequences in db.sql values 2022-06-29 16:32:00 +05:30
Gavin D'souza
ef078a4ab5 refactor(db-read_only): Track conn type in Database instance 2022-06-23 19:40:17 +05:30
Gavin D'souza
14003e5ac9 refactor: DISABLE_DATABASE_CONNECTION_POOLING conf + var name 2022-06-20 15:31:15 +05:30
Gavin D'souza
a58a5bb848 fix: Improve _transform_query unique key identification 2022-06-20 13:10:13 +05:30
Gavin D'souza
90c716bce0 fix(sequence): Setup & use SequenceGeneratorLimitExceeded error 2022-06-17 12:10:43 +05:30
Gavin D'souza
effa942f4c refactor: frappe.db.log_query
* Mogrify queries and set them as frappe.db.last_query instead of
  directly interfacing with the clients
* This is required for now as the MariaDB client uses binary protocol to
  talk to the server and doesn't build the queries itself
* Add typing hints
* Imported Query object as FilterEngine - Query is too ambiguous lol

ref: https://jira.mariadb.org/browse/CONPY-208?focusedCommentId=226873&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-226873
2022-06-16 13:28:54 +05:30
Gavin D'souza
ded55fd98e fix(db): Skip transformation of query if no named params found 2022-06-15 17:43:44 +05:30
gavin
ff9c89450e feat(db): _transform_query to convert args passed to db.cursor
Transform query & parameters based on client requirements. Eg: MariaDB
client doesn't behave similar to PyMySQL or Psycopg2.

ref: https://jira.mariadb.org/projects/CONPY/issues/CONPY-205
2022-05-25 13:58:23 +05:30
gavin
958fc2b0b2 refactor: Database
* Change query notations - QB > raw
* Update logic of DB APIs - simplify & perf improvements
2022-05-25 12:26:24 +05:30
gavin
f7c5c27d81 fix: Use sql_ddl instead of sql
Previous attempts to run this would lead to stalls...long long stalls.
And nothing blocking the query, it would stay in a "Waiting for table
metadata lock" state.
2022-05-24 19:40:27 +05:30
gavin
d87197f042 fix: Correct use of is_connection_pooling_enabled check 2022-05-24 19:19:53 +05:30
gavin
3ed808aec7 fix: Connection Pooling
* Reduce _POOL_SIZE from 4 to 1. New pools will have just one
  connection. They can scale up as per requirement there after.
* Set auto_connect flag in MariaDB connection - https://mariadb.com/docs/connect/programming-languages/python/connect/
2022-05-24 19:19:53 +05:30
gavin
639fa62138 fix: frappe.DISABLE_DATABASE_POOLING to override frappe.conf.disable_database_connection_pooling
Also, don't pool root connections
2022-05-24 19:19:53 +05:30
gavin
6637a4ebbe fix: Allow non-blocking checks via MariaDBExceptionUtils 2022-05-24 19:19:53 +05:30
gavin
0fe764c9b8 fix: Depend on replica details being there for Replica ConnectionPool
This logic mirror how replica connections are handled
2022-05-24 19:19:53 +05:30
gavin
aaef732581 fix!: Remove frappe.db.create_help_table
Help table has been deprecated for a while. It's safe to get rid of the
API too.
2022-05-24 19:19:53 +05:30
gavin
f021250308 refactor: Move connection & pool management out, inherit instead 2022-05-24 19:19:53 +05:30
gavin
993b935097 fix: Separate pool for replica setup
_SITE_POOLS[frappe.local.site].read_only will hold ConnectionPool for
replica database connections. _SITE_POOLS[frappe.local.site].default
will hold pool for connections that allow read+writes
2022-05-24 19:19:53 +05:30
gavin
363708d7f2 fix: frappe.conf.disable_database_connection_pooling
For the times you don't want to use pooling ;) Calling
get_connection/connect will close all pooled connections
for the current process / worker!
2022-05-24 19:19:53 +05:30