Commit graph

98 commits

Author SHA1 Message Date
Ankush Menat
0d5f28d569 feat: rename custom field 2023-09-25 12:25:28 +05:30
Ankush Menat
fa6dc03cc8
refactor: frappe.cache() usage to frappe.cache (#21282) 2023-06-08 11:47:17 +05:30
Ankush Menat
4104e7d733
feat: show db table utilization on doctype (#21193)
* feat: show db table utilization on doctype

* fix: nicer error message with docs
2023-05-31 16:28:44 +05:30
Ankush Menat
48539dc0a3
feat: log all DDL queries (#21107) 2023-05-31 16:00:59 +05:30
Ritwik Puri
d099c9376b
feat(minor): db.get_column_type for postgres (#21125) 2023-05-26 16:03:28 +05:30
Ankush Menat
024faff025
build: bump pymysql (#20478)
Actual fix for this bandaid fix: https://github.com/frappe/frappe/pull/20475

Keeping pymysql hard pinned until we have better way to get last full
query.
2023-03-28 13:04:27 +05:30
Ankush Menat
95039a1265
fix: db ssl connection (#19308)
ref: https://pymysql.readthedocs.io/en/latest/modules/connections.html

[skip ci]
2022-12-19 11:21:15 +05:30
Ankush Menat
263055e247
fix: site creation using non-root users (#19014) 2022-11-26 12:45:57 +05:30
Ankush Menat
ce360b6fce feat: Set default SQL statement timeouts 2022-11-08 14:51:43 +05:30
Ankush Menat
da561c237d fix: index column should be first 2022-10-06 11:11:03 +05:30
Ankush Menat
cbe4b59100 fix: correct index re-syncing
The implementation of syncing unique and non-unique index depended on
index names which used to be different before because of that there's
tendency to incorrectly identify index.

This PR adds a separate util for checking if a column has index without
relying on naming convention. It just goes and checks if there's any
index with that column in it, hence far more reliable.
2022-10-06 11:11:03 +05:30
Ankush Menat
f96505fae0 feat: wrap read only mode SQL errors 2022-09-09 17:34:44 +05:30
Gavin D'souza
b13298d69c fix(db): Log mogrified queries
Prior to this, queries passed to `frappe.db.sql` with values looked like:

"SELECT `defkey`,`defvalue` FROM `tabDefaultValue` WHERE `parent`=%(param1)s ORDER BY `creation`"

Now, they'll look "normal" or built like:

"SELECT `defkey`,`defvalue` FROM `tabDefaultValue` WHERE `parent`='__global' ORDER BY `creation`"
2022-08-10 18:31:44 +05:30
Ritwik Puri
cb6438158b
fix: don't use cache for sequence in mariadb (#17640)
* fix: don't use cache for sequence in mariadb

* chore: update sequence related comments
2022-07-27 22:46:56 +05:30
Gavin D'souza
2bef29bb46 chore: Deprecate backup script + minor refactors 2022-07-22 13:21:29 +05:30
Gavin D'souza
bcfa8c276e fix: Set default port attribute for Database classes
db.default_port wil be available as a class attribute to hold defaults
for DB types.

Usage: frappe.conf.db_port or frappe.db.default_port
Why: I couldn't run the mariadb command because the defaults aren't set
for my system. server is remote / containerized. Setting port in
equivalent mysql command fixes this.
2022-07-22 13:16:26 +05:30
Gavin D'souza
006ebcbede refactor: Use pymysql over mariadb client
This is supposed to be a temporary switch to make the parent PR easier
to digest. MariaDB client has some issues with release, and system
dependencies.

This commit may be reverted to enable mariadb client again.
2022-07-22 12:10:54 +05:30
Gavin D'souza
8ccc0d039b fix: Database._transform_query
* Don't try to convert None to sequence
* Skip transforming values that mariadb~=1.1 can handle - only worry
  about sequences for now
2022-07-07 15:59:48 +05:30
Gavin D'souza
576fa32af4 fix(mariadb): SequenceGeneratorLimitExceeded is an OperationalError now
:D
2022-07-07 12:09:36 +05:30
Gavin D'souza
41b93392b1 fix: Database._transform_result
Transform data not already casted by mariadb client
2022-07-06 15:58:45 +05:30
Gavin D'souza
4dc2ecefba fix: Use mariadb's constants instead of pymysql 2022-07-05 17:47:54 +05:30
Gavin D'souza
71b5c77e6e Merge branch 'develop' of github.com:frappe/frappe into mariadb-client-refactor 2022-07-05 14:55:02 +05:30
Ankush Menat
261fbfcd11
Revert "fix(doc)!: Always cast datetime, date and time fields"
Revert "fix(doc)!: Always cast datetime, date and time fields (#15891)"

This reverts commit d7789ab6ff.
2022-07-05 13:43:32 +05:30
Himanshu
d7789ab6ff
fix(doc)!: Always cast datetime, date and time fields (#15891)
### BREAKING CHANGE
#### Datetime, Date and Time fields will always be cast to respective objects in `setattr`, this will ensure uniformity while accessing the values, no more `getdate`, `get_datetime`, `to_timedelta` wrapper.
- While importing data, the framework does check for `set_only_once`.
- In normal case scenarios, this will work flawlessly since most date fields might not be set_only_once.
- But in Subscription, the date field is set to `set_only_once` and in `after_insert`, `document.save` is called, and while doing so, `set_only_once` is checked [here](1944a547f9/frappe/model/document.py (L566)).
-This works fine if the data imported is in the correct format.
- If the date's data is not in the correct format, the framework throws an error.
- for eg `06-02-2022 00:00:00 != 06-02-2022`
- fixes [Issue/#15370](https://github.com/frappe/frappe/issues/15370)

> no-docs
2022-07-05 07:07:16 +00:00
Ankush Menat
81b37cb7d2
refactor: clean up code to py310 supported features (#17367)
refactor: clean up code to py39+ supported syntax

- f-strings instead of format
- latest typing support instead of pre 3.9 TitleCase
- remove UTF-8 declarations.
- many more changes

Powered by https://github.com/asottile/pyupgrade/ + manual cleanups
2022-07-01 11:51:05 +05:30
Gavin D'souza
39d30ffeaa fix: Database._transform_query
* Handle dict substitutions in transformations too
* Allow list / tuple values
* Check for values inconsistencies before flattening step
2022-06-30 19:12:10 +05:30
Gavin D'souza
daf3f05fea fix(db): Hanlde sequences in db.sql values 2022-06-29 16:32:00 +05:30
Gavin D'souza
ef078a4ab5 refactor(db-read_only): Track conn type in Database instance 2022-06-23 19:40:17 +05:30
Gavin D'souza
14003e5ac9 refactor: DISABLE_DATABASE_CONNECTION_POOLING conf + var name 2022-06-20 15:31:15 +05:30
Gavin D'souza
a58a5bb848 fix: Improve _transform_query unique key identification 2022-06-20 13:10:13 +05:30
Gavin D'souza
90c716bce0 fix(sequence): Setup & use SequenceGeneratorLimitExceeded error 2022-06-17 12:10:43 +05:30
Gavin D'souza
effa942f4c refactor: frappe.db.log_query
* Mogrify queries and set them as frappe.db.last_query instead of
  directly interfacing with the clients
* This is required for now as the MariaDB client uses binary protocol to
  talk to the server and doesn't build the queries itself
* Add typing hints
* Imported Query object as FilterEngine - Query is too ambiguous lol

ref: https://jira.mariadb.org/browse/CONPY-208?focusedCommentId=226873&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-226873
2022-06-16 13:28:54 +05:30
Gavin D'souza
ded55fd98e fix(db): Skip transformation of query if no named params found 2022-06-15 17:43:44 +05:30
gavin
ff9c89450e feat(db): _transform_query to convert args passed to db.cursor
Transform query & parameters based on client requirements. Eg: MariaDB
client doesn't behave similar to PyMySQL or Psycopg2.

ref: https://jira.mariadb.org/projects/CONPY/issues/CONPY-205
2022-05-25 13:58:23 +05:30
gavin
958fc2b0b2 refactor: Database
* Change query notations - QB > raw
* Update logic of DB APIs - simplify & perf improvements
2022-05-25 12:26:24 +05:30
gavin
d87197f042 fix: Correct use of is_connection_pooling_enabled check 2022-05-24 19:19:53 +05:30
gavin
3ed808aec7 fix: Connection Pooling
* Reduce _POOL_SIZE from 4 to 1. New pools will have just one
  connection. They can scale up as per requirement there after.
* Set auto_connect flag in MariaDB connection - https://mariadb.com/docs/connect/programming-languages/python/connect/
2022-05-24 19:19:53 +05:30
gavin
639fa62138 fix: frappe.DISABLE_DATABASE_POOLING to override frappe.conf.disable_database_connection_pooling
Also, don't pool root connections
2022-05-24 19:19:53 +05:30
gavin
6637a4ebbe fix: Allow non-blocking checks via MariaDBExceptionUtils 2022-05-24 19:19:53 +05:30
gavin
0fe764c9b8 fix: Depend on replica details being there for Replica ConnectionPool
This logic mirror how replica connections are handled
2022-05-24 19:19:53 +05:30
gavin
aaef732581 fix!: Remove frappe.db.create_help_table
Help table has been deprecated for a while. It's safe to get rid of the
API too.
2022-05-24 19:19:53 +05:30
gavin
f021250308 refactor: Move connection & pool management out, inherit instead 2022-05-24 19:19:53 +05:30
gavin
993b935097 fix: Separate pool for replica setup
_SITE_POOLS[frappe.local.site].read_only will hold ConnectionPool for
replica database connections. _SITE_POOLS[frappe.local.site].default
will hold pool for connections that allow read+writes
2022-05-24 19:19:53 +05:30
gavin
363708d7f2 fix: frappe.conf.disable_database_connection_pooling
For the times you don't want to use pooling ;) Calling
get_connection/connect will close all pooled connections
for the current process / worker!
2022-05-24 19:19:53 +05:30
gavin
0d80f6ac52 feat: mariadb connection pooling
Start initial pool of 4 (_POOL_SIZE) connections for a given site. When
the pool is exhausted, generate new connections upon request and add
them to the pool. Max allowed size for each pool is 64 (_MAX_POOL_SIZE)
connections. However, you may have more than 64 active connections, just
that they won't be pooled but destroyed upon job completion/end of
request/similar cycle.
2022-05-24 19:19:53 +05:30
gavin
a17c978065 refactor!: frappe.db.get_database_list
* Drop mandatory unused arg in method call
* Use pluck instead of list comprehension + subscripting
2022-05-24 19:19:53 +05:30
gavin
756e385362 refactor: Move exception detection & abstraction in separate class
This change has only been done to separate and club only like methods /
utils together
2022-05-24 19:19:53 +05:30
gavin
1fe3624c43 fix: DB error detection API usage
* Make all methods static
* Add typing hints
* Don't safe fetch attribute value for errno - MariaDB exceptions if
  raised will have errno attr in them. If the class doesn't have one,
  it's not meant to be passed in these methods.
2022-05-24 19:19:53 +05:30
gavin
d3900485a6 refactor: Use mariadb client for creating connections
Move to use MariaDB's official Python client written in C instead of
the PyMySQL library. This change doesn't rid Frappe of the PyMySQL
library. Instead, it continues to utilize it for the ER module and
converter methods until the MariaDB library adds support for the same.

Ticket: https://jira.mariadb.org/projects/CONPY/issues/CONPY-203
2022-05-24 19:19:53 +05:30
phot0n
f6dac70033 refactor(minor): convert change_name_type_and_make_sequence function to setup_autoincrement_and_sequence method
* chore(Customize Form): remove autoincrement naming option from autoname description

* chore: move sequence cache comment from schema.py to database.py

* chore: added docstrings to some functions
2022-05-13 00:03:19 +05:30