seitime-frappe/frappe/model/delete_doc.py
Rushabh Mehta 2e6a202652 Postgres support for Frappe (#5919)
* [start] postgres

* [wip] started refactoring db_schema

* Add psycopg2 to requirements.txt

* Add support for Postgres SQL

- Separate frameworkSQL, database, schema, setup_db file for
mariaDB and postgres
- WIP

* Remove quotes from sql to make it compatible with postgres as well

* Moved some code from db_schema to database.py

* Move code from db_schema to schema.py

Add other required refactoring

* Add schema chages

* Remove redundant code in file

* Add invalid column name exception class to exceptions.py

* Add back tick in query wherever needed and replace ifnull with coalesce

* Update get_column_description code in database.py file

* Remove a print statement

* Add keys to get on_duplicate query

* Add bactick wherever necessary

- Remove db_schema.py file

* Remove DATE_SUB as it is incompatible with postgres

- Fix prepare_filter_condition

* Add backtick and quotes wherever necessary
- Move get_database_size to frappe.db namespace
- fix some left out bugs and errors

* Add code to create key and unique index
- added mysql and posgres in their respective database.py

* Add more bacticks in queries and fix some errors
- Pass keys to on_duplicate_update method
- Replace MONTH with EXTRACT function
- Remove DATEDIFF and CURDATE usage

* Cast state value to int in toggle_two_factor_auth
- since two_factor_auth has the datatype of Int

* Refactor
- Replace Timediff with normal arithmetic operator
- Add MAX_COLUMN_LENGTH
- Remove Redundant code
- Add regexp character constant
- Move create_help_table to database.py
- Add get_full_text_search_condition method
- Inherit MariaDBTable from DBTable

* Replace Database instance with get_db method

* Move db_manager to separate file

* Refactor
- Remove some unwanted code
- Separate alter table code for postgres and mysql
- Replace data_type with column_type in database.py

* Make fulltext search changes in global_search.py

* Add empty string check

* Add root_password to site config

* Create cli command for postgres console

* Move setup of help database to setup_db.py

* Add get_database_list method

* Fix exception handling
- Replace bad_field handler with missing_column handler

* Fix tests and sql queries

* Fix import error

* Fix typo db -> database

* Fix error with make_table in help.py

* Try test for postgres

* Remove pyhton 2.7 version to try postgres travis test

* Add test fixes

* Add db_type to the config of test_site_postgres

* Enable query debug to check the reason for travis fail

* Add backticks to check if the test passes

* Update travis.yml
- Add postgres addon

* Try appending 'd_' to hash for db_name
- since postgres does not support dbname starting with a number

* Try adding db_type for global help to make travis work

* Add print statements to debug travis failure

* Enable transaction and remove debug flag

* Fix help table creation query (postgres)

* Fix import issue

* Add some checks to prevent errors
- Some doctypes used to get called even before they are created

* Try fixes

* Update travis config

* Fix create index for help table

* Remove unused code

* Fix queries and update travis config

* Fix ifnull replace logic (regex)

* Add query fixes and code cleanup

* Fix typo
- get_column_description -> get_table_columns_description

* Fix tests
- Replace double quotes in query with single quote

* Replace psycopg2 with psycopg2-binary to avoid warnings
- http://initd.org/psycopg/docs/install.html#binary-install-from-pypi

* Add multisql api

* Add few multisql queries

* Remove print statements

* Remove get_fulltext_search_condition method and replace with multi query

* Remove text slicing in create user

* Set default for 'values' argument in multisql

* Fix incorrect queries and remove few debug flags
- Fix multisql bug

* Force delete user to fix test
- Fix Import error
- Fix incorrect query

* Fix query builder bug

* Fix bad query

* Fix query (minor)

* Convert boolean text to int since is_private has datatype of int
- Some query changes like removed double quotes
and replace with interpolated string to pass multiple
value pass in one of the query

* Extend database class from an object to support python 2

* Fix query
- Add quotes around value passed to the query for variable comparision

* Try setting host_name for each test site
- To avoid "RemoteDisconnected" error while testing data migration test
- Update travis.yml to add hosts
- Remove unwanted commit in setup_help_database

* Set site hostname to data migration connector (in test file)
- To connect the same site host

* Fix duplicate entry issue
- the problem is in naming series file.
In previous commits I unknowingly changed a part of a series query
due to which series were not getting reset

* Replace few sql queries with orm methods

* Fix codacy

* Fix 'Doctype Sessions not found' issue

* Fix bugs induced during codacy fixes

* Fix Notification Test

- Use ORM instead of raw sql

* Set Date fallback value to 0001-01-01

- 0000-00-00 is invalid date in Postgres
- 0001-01-01 works in both

* Fix date filter method

* Replace double quotes with single quote for literal value

* Remove print statement

* Replace double quotes with single

* Fix tests

- Replace few raw sql with ORM

* Separate query for postgres

- update_fields_to_fetch_query

* Fix tests

- replace locate with strpos for postgres

* Fix tests

- Skip test for datediff
- convert bytes to str in escape method

* Remove TestBot

* Skip fieldname extraction

* Replace docshare raw sql with ORM

* Fix typo

* Fix ancestor query test

* Fix test data migration

* Remove hardcoded hostname

* Add default option and option list for db_type

* Remove frappe.async module

* Remove a debug flag from test

* Fix codacy

* fix import issue

* Convert classmethod to static method

* Convert few instance methods to static methods

* Remove some unused imports

* Fix codacy

- Add exception type
- Replace few instance methods with static methods
- Remove unsued import

* Fix codacy

* Remove unused code

* Remove some unused codes

- Convert some instance methods to static function

* Fix a issue with query modification

* Fix add_index query

* Fix query

* Fix update_auth patch

* Fix a issue with exception handling

* Add try catch to a reload_doc

* Add try-catch to file_manager_hook patch

* import update_gravatar to set_user_gravatar patch

* Undo all the wrong patch fixes

* Fix db_setup code 😪
- previously it was not restoring db from source SQL
which is why few old patched were breaking
(because they were getting different schema structure)

* Fix typo !

* Fix exception(is_missing_column) handling

* Add deleted code
- This code is only used in a erpnext patch.
Can be moved to that patch file

* Fix codacy

* Replace a mariadb specific function in a query used in validate_series

* Remove a debug flag

* Revert changes (rename_parent_and_child)

* Fix validate_one_root method

* Fix date format issue

* Fix codacy
- Disable a pylint for variable argument warning
- Convert an instance method to static method

* Add bandit.yml

The Codacy seems to use Bandit which generates
warning for every subprocess import and its usage during pytest
Since we have carefully used subprocess (avoided user input),
warnings needs to be avoided.
This can be removed if we have any alternative for subprocess usage.

* Skip start_process_with_partial_path check

* Fix typo

* Add python 2.7 test

* Move python versions in travis.yml

* Add python versions to jobs

* Overwrite python version inheritance for postgres in travis.yml

* Add quotes around python version in .travis.yml

* Add quotes around the name of the job

* Try a travis fix

* Try .travis.yml fix

* Import missing subprocess

* Refactor travis.yml

* Refactor travis.yml
- move install and tests commands to separate files
- Use matrix to build combination of python version and db type

* Make install.sh and run-tests.sh executable

* Add sudo required to travis.yml to allow sudo cmmands in shell files

* Load nvm

* Remove verbose flag from scripts

* Remove command-trace-print flag

* Change to build dir in before script

* Add absolute path for scripts

* Fix tests

* Fix typo

* Fix codacy
- fixes - "echo won't expand escape sequences." warning

* Append (_) underscore instead of 'd' for db_name

* Remove printf and use mysql execute flag
2018-09-21 10:20:48 +05:30

326 lines
11 KiB
Python

# Copyright (c) 2015, Frappe Technologies Pvt. Ltd. and Contributors
# MIT License. See license.txt
from __future__ import unicode_literals
import frappe
import frappe.model.meta
from frappe.model.dynamic_links import get_dynamic_link_map
import frappe.defaults
from frappe.utils.file_manager import remove_all
from frappe.utils.password import delete_all_passwords_for
from frappe import _
from frappe.model.naming import revert_series_if_last
from frappe.utils.global_search import delete_for_document
from six import string_types, integer_types
def delete_doc(doctype=None, name=None, force=0, ignore_doctypes=None, for_reload=False,
ignore_permissions=False, flags=None, ignore_on_trash=False, ignore_missing=True):
"""
Deletes a doc(dt, dn) and validates if it is not submitted and not linked in a live record
"""
if not ignore_doctypes: ignore_doctypes = []
# get from form
if not doctype:
doctype = frappe.form_dict.get('dt')
name = frappe.form_dict.get('dn')
names = name
if isinstance(name, string_types) or isinstance(name, integer_types):
names = [name]
for name in names or []:
# already deleted..?
if not frappe.db.exists(doctype, name):
if not ignore_missing:
raise frappe.DoesNotExistError
else:
return False
# delete passwords
delete_all_passwords_for(doctype, name)
doc = None
if doctype=="DocType":
if for_reload:
try:
doc = frappe.get_doc(doctype, name)
except frappe.DoesNotExistError:
pass
else:
doc.run_method("before_reload")
else:
doc = frappe.get_doc(doctype, name)
update_flags(doc, flags, ignore_permissions)
check_permission_and_not_submitted(doc)
frappe.db.sql("delete from `tabCustom Field` where dt = %s", name)
frappe.db.sql("delete from `tabCustom Script` where dt = %s", name)
frappe.db.sql("delete from `tabProperty Setter` where doc_type = %s", name)
frappe.db.sql("delete from `tabReport` where ref_doctype=%s", name)
frappe.db.sql("delete from `tabCustom DocPerm` where parent=%s", name)
delete_from_table(doctype, name, ignore_doctypes, None)
else:
doc = frappe.get_doc(doctype, name)
if not for_reload:
update_flags(doc, flags, ignore_permissions)
check_permission_and_not_submitted(doc)
if not ignore_on_trash:
doc.run_method("on_trash")
doc.flags.in_delete = True
doc.run_method('on_change')
frappe.enqueue('frappe.model.delete_doc.delete_dynamic_links', doctype=doc.doctype, name=doc.name,
is_async=False if frappe.flags.in_test else True)
# check if links exist
if not force:
check_if_doc_is_linked(doc)
check_if_doc_is_dynamically_linked(doc)
update_naming_series(doc)
delete_from_table(doctype, name, ignore_doctypes, doc)
doc.run_method("after_delete")
# delete attachments
remove_all(doctype, name, from_delete=True)
# delete global search entry
delete_for_document(doc)
if doc and not for_reload:
add_to_deleted_document(doc)
if not frappe.flags.in_patch:
try:
doc.notify_update()
insert_feed(doc)
except ImportError:
pass
# delete user_permissions
frappe.defaults.clear_default(parenttype="User Permission", key=doctype, value=name)
def add_to_deleted_document(doc):
'''Add this document to Deleted Document table. Called after delete'''
if doc.doctype != 'Deleted Document' and frappe.flags.in_install != 'frappe':
frappe.get_doc(dict(
doctype='Deleted Document',
deleted_doctype=doc.doctype,
deleted_name=doc.name,
data=doc.as_json(),
owner=frappe.session.user
)).db_insert()
def update_naming_series(doc):
if doc.meta.autoname:
if doc.meta.autoname.startswith("naming_series:") \
and getattr(doc, "naming_series", None):
revert_series_if_last(doc.naming_series, doc.name)
elif doc.meta.autoname.split(":")[0] not in ("Prompt", "field", "hash"):
revert_series_if_last(doc.meta.autoname, doc.name)
def delete_from_table(doctype, name, ignore_doctypes, doc):
if doctype!="DocType" and doctype==name:
frappe.db.sql("delete from `tabSingles` where `doctype`=%s", name)
else:
frappe.db.sql("delete from `tab{0}` where `name`=%s".format(doctype), name)
# get child tables
if doc:
tables = [d.options for d in doc.meta.get_table_fields()]
else:
def get_table_fields(field_doctype):
return frappe.db.sql_list("""select options from `tab{}` where fieldtype='Table'
and parent=%s""".format(field_doctype), doctype)
tables = get_table_fields("DocField")
if not frappe.flags.in_install=="frappe":
tables += get_table_fields("Custom Field")
# delete from child tables
for t in list(set(tables)):
if t not in ignore_doctypes:
frappe.db.sql("delete from `tab%s` where parenttype=%s and parent = %s" % (t, '%s', '%s'), (doctype, name))
def update_flags(doc, flags=None, ignore_permissions=False):
if ignore_permissions:
if not flags: flags = {}
flags["ignore_permissions"] = ignore_permissions
if flags:
doc.flags.update(flags)
def check_permission_and_not_submitted(doc):
# permission
if (not doc.flags.ignore_permissions
and frappe.session.user!="Administrator"
and (
not doc.has_permission("delete")
or (doc.doctype=="DocType" and not doc.custom))):
frappe.msgprint(_("User not allowed to delete {0}: {1}")
.format(doc.doctype, doc.name), raise_exception=frappe.PermissionError)
# check if submitted
if doc.docstatus == 1:
frappe.msgprint(_("{0} {1}: Submitted Record cannot be deleted.").format(_(doc.doctype), doc.name),
raise_exception=True)
def check_if_doc_is_linked(doc, method="Delete"):
"""
Raises excption if the given doc(dt, dn) is linked in another record.
"""
from frappe.model.rename_doc import get_link_fields
link_fields = get_link_fields(doc.doctype)
link_fields = [[lf['parent'], lf['fieldname'], lf['issingle']] for lf in link_fields]
for link_dt, link_field, issingle in link_fields:
if not issingle:
for item in frappe.db.get_values(link_dt, {link_field:doc.name},
["name", "parent", "parenttype", "docstatus"], as_dict=True):
linked_doctype = item.parenttype if item.parent else link_dt
if linked_doctype in ("Communication", "ToDo", "DocShare", "Email Unsubscribe", 'File', 'Version', "Activity Log"):
# don't check for communication and todo!
continue
if not item:
continue
elif (method != "Delete" or item.docstatus == 2) and (method != "Cancel" or item.docstatus != 1):
# don't raise exception if not
# linked to a non-cancelled doc when deleting or to a submitted doc when cancelling
continue
elif link_dt == doc.doctype and (item.parent or item.name) == doc.name:
# don't raise exception if not
# linked to same item or doc having same name as the item
continue
else:
reference_docname = item.parent or item.name
raise_link_exists_exception(doc, linked_doctype, reference_docname)
else:
if frappe.db.get_value(link_dt, None, link_field) == doc.name:
raise_link_exists_exception(doc, link_dt, link_dt)
def check_if_doc_is_dynamically_linked(doc, method="Delete"):
'''Raise `frappe.LinkExistsError` if the document is dynamically linked'''
for df in get_dynamic_link_map().get(doc.doctype, []):
if df.parent in ("Communication", "ToDo", "DocShare", "Email Unsubscribe", "Activity Log", 'File', 'Version'):
# don't check for communication and todo!
continue
meta = frappe.get_meta(df.parent)
if meta.issingle:
# dynamic link in single doc
refdoc = frappe.db.get_singles_dict(df.parent)
if (refdoc.get(df.options)==doc.doctype
and refdoc.get(df.fieldname)==doc.name
and ((method=="Delete" and refdoc.docstatus < 2)
or (method=="Cancel" and refdoc.docstatus==1))
):
# raise exception only if
# linked to an non-cancelled doc when deleting
# or linked to a submitted doc when cancelling
raise_link_exists_exception(doc, df.parent, df.parent)
else:
# dynamic link in table
df["table"] = ", `parent`, `parenttype`, `idx`" if meta.istable else ""
for refdoc in frappe.db.sql("""select `name`, `docstatus` {table} from `tab{parent}` where
{options}=%s and {fieldname}=%s""".format(**df), (doc.doctype, doc.name), as_dict=True):
if ((method=="Delete" and refdoc.docstatus < 2) or (method=="Cancel" and refdoc.docstatus==1)):
# raise exception only if
# linked to an non-cancelled doc when deleting
# or linked to a submitted doc when cancelling
reference_doctype = refdoc.parenttype if meta.istable else df.parent
reference_docname = refdoc.parent if meta.istable else refdoc.name
at_position = "at Row: {0}".format(refdoc.idx) if meta.istable else ""
raise_link_exists_exception(doc, reference_doctype, reference_docname, at_position)
def raise_link_exists_exception(doc, reference_doctype, reference_docname, row=''):
doc_link = '<a href="#Form/{0}/{1}">{1}</a>'.format(doc.doctype, doc.name)
reference_link = '<a href="#Form/{0}/{1}">{1}</a>'.format(reference_doctype, reference_docname)
#hack to display Single doctype only once in message
if reference_doctype == reference_docname:
reference_doctype = ''
frappe.throw(_('Cannot delete or cancel because {0} {1} is linked with {2} {3} {4}')
.format(doc.doctype, doc_link, reference_doctype, reference_link, row), frappe.LinkExistsError)
def delete_dynamic_links(doctype, name):
delete_doc("ToDo", frappe.db.sql_list("""select name from `tabToDo`
where reference_type=%s and reference_name=%s""", (doctype, name)),
ignore_permissions=True, force=True)
frappe.db.sql('''delete from `tabEmail Unsubscribe`
where reference_doctype=%s and reference_name=%s''', (doctype, name))
# delete shares
frappe.db.sql("""delete from `tabDocShare`
where share_doctype=%s and share_name=%s""", (doctype, name))
# delete versions
frappe.db.sql('delete from tabVersion where ref_doctype=%s and docname=%s', (doctype, name))
# delete comments
frappe.db.sql("""delete from `tabCommunication`
where
communication_type = 'Comment'
and reference_doctype=%s and reference_name=%s""", (doctype, name))
# unlink communications
frappe.db.sql("""update `tabCommunication`
set reference_doctype=null, reference_name=null
where
communication_type = 'Communication'
and reference_doctype=%s
and reference_name=%s""", (doctype, name))
# unlink secondary references
frappe.db.sql("""update `tabCommunication`
set link_doctype=null, link_name=null
where link_doctype=%s and link_name=%s""", (doctype, name))
# unlink feed
frappe.db.sql("""update `tabCommunication`
set timeline_doctype=null, timeline_name=null
where timeline_doctype=%s and timeline_name=%s""", (doctype, name))
# unlink activity_log reference_doctype
frappe.db.sql("""update `tabActivity Log`
set reference_doctype=null, reference_name=null
where
reference_doctype=%s
and reference_name=%s""", (doctype, name))
# unlink activity_log timeline_doctype
frappe.db.sql("""update `tabActivity Log`
set timeline_doctype=null, timeline_name=null
where timeline_doctype=%s and timeline_name=%s""", (doctype, name))
def insert_feed(doc):
from frappe.utils import get_fullname
if frappe.flags.in_install or frappe.flags.in_import or getattr(doc, "no_feed_on_delete", False):
return
frappe.get_doc({
"doctype": "Communication",
"communication_type": "Comment",
"comment_type": "Deleted",
"reference_doctype": doc.doctype,
"subject": "{0} {1}".format(_(doc.doctype), doc.name),
"full_name": get_fullname(doc.owner)
}).insert(ignore_permissions=True)