Commit graph

119 commits

Author SHA1 Message Date
Rutwik Hiwalkar
ba2715582b
fix: init db conn for unbuffered cursor if not set (#26220)
* fix: init db conn for unbuffered cursor if not set

* chore: check conn and not cursor

---------

Co-authored-by: Ankush Menat <ankushmenat@gmail.com>
2024-04-30 12:54:12 +00:00
Ankush Menat
c1bf152b89
fix: handle interface error during report timeout (#25893) 2024-04-10 06:57:36 +00:00
Ankush Menat
eef9c2c8cc refactor!: Better Integer handling
- Remove length: it makes no difference, it's for ZEROFILL only.
- Switch to tinyint for checkboxes in mysql and smallint on postgres.
- Use `int` instead of `bigint` by default.
2024-04-06 11:11:29 +05:30
Ankush Menat
a6732a0e12
fix: set db name in filter (#25732) 2024-03-29 16:06:02 +00:00
David
39d4318a27
feat: enable db socket connection 2024-03-29 10:29:50 +01:00
Ankush Menat
b60c64cacc
fix: rollback invalid customize form changes (#25198)
* fix: rollback invalid customize form changes

* fix: commit before doing DDL
2024-03-03 17:17:12 +05:30
Shankarv19bcr
437beaa735 fix: typo 2024-02-21 17:19:32 +05:30
Akhil Narang
3f1e19de85
refactor(treewide): enable RUF rules
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2024-02-21 16:20:28 +05:30
Akhil Narang
26ae0f3460
fix: ruff fixes
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2024-02-07 17:04:31 +05:30
Ankush Menat
de9ac89748 style: re-format with ruff 2024-02-05 18:53:33 +05:30
Ankush Menat
c02f5d5876
fix: Skip fulltext indexes during sync (#24728)
These are not managed by framework's migration system so we shouldn't
delete them.
2024-02-05 05:30:44 +00:00
David Arnold
f889d09d0b
fix: set defaults at the highest level interface 2024-01-17 18:59:34 +01:00
Ankush Menat
a2525e545a
perf: Unbuffered cursors for large result sets (#24365)
If you're reading 1000s of rows from MySQL, the default behaviour is to
read all of them in memory at once.

One of the use case for reading large rows is reporting where a lot of
data is read and then processed in Python. The read row is hoever not
used again but still consumes memory until entire function exits.

SSCursor (Server Side Cursor) allows fetching one row at a time.

Note: This is slower than fetching everything at once AND has risk of
connection loss. So, don't use this as a crutch. If possible rewrite
code so processing is done in SQL.
2024-01-16 11:00:12 +05:30
Ankush Menat
d5b27061ad perf: avoid duplicate copies of result set
When as_list, as_dict is done we hold on to original result set until
next query is performed. This can be HUGE for large queries.
2024-01-15 12:32:07 +05:30
David Arnold
3fa76b827c
chore: name current db handle properly 2024-01-14 17:14:21 +01:00
Xiaoguang Sun
68d4a5ad82
Do not call begin in updatedb after syncing (#23934)
Do not call begin in updatedb after syncing as commit does it already.

Signed-off-by: Xiaoguang Sun <sunxiaoguang@gmail.com>
2023-12-26 12:12:19 +05:30
Hussain Nagaria
8d2137c265 docs: consistent doc strings 2023-12-18 18:27:39 +05:30
barredterra
c35476256f refactor: simplify conditional logic
Command: `sourcery review --fix --enable de-morgan .`
2023-12-05 11:14:41 +01:00
Akhil Narang
ccc2bdad52
fix: go back to original transaction handling mechanism
This reverts the following:
07acfeed47 chore: move check_implicit_commit() call back to its original location
fc38a0b503 chore: simplify transaction handling
1e29e81543 chore: rework implicit commit checks
ae0a3fd202 feat: allow ignoring implicit commit warning

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2023-11-16 14:51:57 +05:30
Akhil Narang
806748b063
chore: simplify transaction handling
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2023-11-16 14:51:57 +05:30
Akhil Narang
e41fa2dfc7
feat: migrate columns to be non-nullable if required
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
2023-11-16 14:51:57 +05:30
Ankush Menat
0d5f28d569 feat: rename custom field 2023-09-25 12:25:28 +05:30
Ankush Menat
fa6dc03cc8
refactor: frappe.cache() usage to frappe.cache (#21282) 2023-06-08 11:47:17 +05:30
Ankush Menat
4104e7d733
feat: show db table utilization on doctype (#21193)
* feat: show db table utilization on doctype

* fix: nicer error message with docs
2023-05-31 16:28:44 +05:30
Ankush Menat
48539dc0a3
feat: log all DDL queries (#21107) 2023-05-31 16:00:59 +05:30
Ritwik Puri
d099c9376b
feat(minor): db.get_column_type for postgres (#21125) 2023-05-26 16:03:28 +05:30
Ankush Menat
024faff025
build: bump pymysql (#20478)
Actual fix for this bandaid fix: https://github.com/frappe/frappe/pull/20475

Keeping pymysql hard pinned until we have better way to get last full
query.
2023-03-28 13:04:27 +05:30
Ankush Menat
95039a1265
fix: db ssl connection (#19308)
ref: https://pymysql.readthedocs.io/en/latest/modules/connections.html

[skip ci]
2022-12-19 11:21:15 +05:30
Ankush Menat
263055e247
fix: site creation using non-root users (#19014) 2022-11-26 12:45:57 +05:30
Ankush Menat
ce360b6fce feat: Set default SQL statement timeouts 2022-11-08 14:51:43 +05:30
Ankush Menat
da561c237d fix: index column should be first 2022-10-06 11:11:03 +05:30
Ankush Menat
cbe4b59100 fix: correct index re-syncing
The implementation of syncing unique and non-unique index depended on
index names which used to be different before because of that there's
tendency to incorrectly identify index.

This PR adds a separate util for checking if a column has index without
relying on naming convention. It just goes and checks if there's any
index with that column in it, hence far more reliable.
2022-10-06 11:11:03 +05:30
Ankush Menat
f96505fae0 feat: wrap read only mode SQL errors 2022-09-09 17:34:44 +05:30
Gavin D'souza
b13298d69c fix(db): Log mogrified queries
Prior to this, queries passed to `frappe.db.sql` with values looked like:

"SELECT `defkey`,`defvalue` FROM `tabDefaultValue` WHERE `parent`=%(param1)s ORDER BY `creation`"

Now, they'll look "normal" or built like:

"SELECT `defkey`,`defvalue` FROM `tabDefaultValue` WHERE `parent`='__global' ORDER BY `creation`"
2022-08-10 18:31:44 +05:30
Ritwik Puri
cb6438158b
fix: don't use cache for sequence in mariadb (#17640)
* fix: don't use cache for sequence in mariadb

* chore: update sequence related comments
2022-07-27 22:46:56 +05:30
Gavin D'souza
2bef29bb46 chore: Deprecate backup script + minor refactors 2022-07-22 13:21:29 +05:30
Gavin D'souza
bcfa8c276e fix: Set default port attribute for Database classes
db.default_port wil be available as a class attribute to hold defaults
for DB types.

Usage: frappe.conf.db_port or frappe.db.default_port
Why: I couldn't run the mariadb command because the defaults aren't set
for my system. server is remote / containerized. Setting port in
equivalent mysql command fixes this.
2022-07-22 13:16:26 +05:30
Gavin D'souza
006ebcbede refactor: Use pymysql over mariadb client
This is supposed to be a temporary switch to make the parent PR easier
to digest. MariaDB client has some issues with release, and system
dependencies.

This commit may be reverted to enable mariadb client again.
2022-07-22 12:10:54 +05:30
Gavin D'souza
8ccc0d039b fix: Database._transform_query
* Don't try to convert None to sequence
* Skip transforming values that mariadb~=1.1 can handle - only worry
  about sequences for now
2022-07-07 15:59:48 +05:30
Gavin D'souza
576fa32af4 fix(mariadb): SequenceGeneratorLimitExceeded is an OperationalError now
:D
2022-07-07 12:09:36 +05:30
Gavin D'souza
41b93392b1 fix: Database._transform_result
Transform data not already casted by mariadb client
2022-07-06 15:58:45 +05:30
Gavin D'souza
4dc2ecefba fix: Use mariadb's constants instead of pymysql 2022-07-05 17:47:54 +05:30
Gavin D'souza
71b5c77e6e Merge branch 'develop' of github.com:frappe/frappe into mariadb-client-refactor 2022-07-05 14:55:02 +05:30
Ankush Menat
261fbfcd11
Revert "fix(doc)!: Always cast datetime, date and time fields"
Revert "fix(doc)!: Always cast datetime, date and time fields (#15891)"

This reverts commit d7789ab6ff.
2022-07-05 13:43:32 +05:30
Himanshu
d7789ab6ff
fix(doc)!: Always cast datetime, date and time fields (#15891)
### BREAKING CHANGE
#### Datetime, Date and Time fields will always be cast to respective objects in `setattr`, this will ensure uniformity while accessing the values, no more `getdate`, `get_datetime`, `to_timedelta` wrapper.
- While importing data, the framework does check for `set_only_once`.
- In normal case scenarios, this will work flawlessly since most date fields might not be set_only_once.
- But in Subscription, the date field is set to `set_only_once` and in `after_insert`, `document.save` is called, and while doing so, `set_only_once` is checked [here](1944a547f9/frappe/model/document.py (L566)).
-This works fine if the data imported is in the correct format.
- If the date's data is not in the correct format, the framework throws an error.
- for eg `06-02-2022 00:00:00 != 06-02-2022`
- fixes [Issue/#15370](https://github.com/frappe/frappe/issues/15370)

> no-docs
2022-07-05 07:07:16 +00:00
Ankush Menat
81b37cb7d2
refactor: clean up code to py310 supported features (#17367)
refactor: clean up code to py39+ supported syntax

- f-strings instead of format
- latest typing support instead of pre 3.9 TitleCase
- remove UTF-8 declarations.
- many more changes

Powered by https://github.com/asottile/pyupgrade/ + manual cleanups
2022-07-01 11:51:05 +05:30
Gavin D'souza
39d30ffeaa fix: Database._transform_query
* Handle dict substitutions in transformations too
* Allow list / tuple values
* Check for values inconsistencies before flattening step
2022-06-30 19:12:10 +05:30
Gavin D'souza
daf3f05fea fix(db): Hanlde sequences in db.sql values 2022-06-29 16:32:00 +05:30
Gavin D'souza
ef078a4ab5 refactor(db-read_only): Track conn type in Database instance 2022-06-23 19:40:17 +05:30
Gavin D'souza
14003e5ac9 refactor: DISABLE_DATABASE_CONNECTION_POOLING conf + var name 2022-06-20 15:31:15 +05:30