Commit graph

185 commits

Author SHA1 Message Date
Ameya Shenoy
c5d276b813
Merge branch 'staging' into develop 2018-11-28 08:24:00 +00:00
Ameya Shenoy
f513739d1a
Merge branch 'master' into staging-fixes 2018-11-28 07:53:35 +00:00
Saurabh
4738a9711a fix: tighten protection against sql injection (#6546) 2018-11-28 10:22:30 +05:30
Ameya Shenoy
13524a1ad5
Merge branch 'staging' into develop 2018-11-23 08:14:29 +00:00
Ameya Shenoy
d1d0edc636
Merge branch 'master' into staging-fixes 2018-11-23 08:07:36 +00:00
rohitwaghchaure
c247ec17ea fix: child table access using get_list function (#6511)
User was able to fetch child table data without having access from client side using frappe.db.get_list function
2018-11-22 12:44:32 +05:30
Ameya Shenoy
d0e7690ae2
Merge branch 'staging' into develop 2018-11-05 06:48:46 +00:00
Ameya Shenoy
5a791cd099
Merge branch 'master' into staging-fixes 2018-11-05 06:37:49 +00:00
Saurabh
856a721073 [fix] sql injection fix (#6390) 2018-11-05 10:51:57 +05:30
Suraj Shetty
15d3f4660e fix(security): pop ignore_permissions arg from whitelisted method 2018-11-05 10:29:50 +05:30
Ameya Shenoy
62736ba219
Merge branch 'staging' into develop 2018-10-31 10:58:47 +00:00
Ameya Shenoy
55fbf7ad99
Merge branch 'master' into staging-fixes 2018-10-31 10:40:55 +00:00
Saurabh
a1e68b6fd5 [fix] potential sql injection by sanitizing fields parameter 2018-10-31 14:26:22 +05:30
Rushabh Mehta
2e6a202652 Postgres support for Frappe (#5919)
* [start] postgres

* [wip] started refactoring db_schema

* Add psycopg2 to requirements.txt

* Add support for Postgres SQL

- Separate frameworkSQL, database, schema, setup_db file for
mariaDB and postgres
- WIP

* Remove quotes from sql to make it compatible with postgres as well

* Moved some code from db_schema to database.py

* Move code from db_schema to schema.py

Add other required refactoring

* Add schema chages

* Remove redundant code in file

* Add invalid column name exception class to exceptions.py

* Add back tick in query wherever needed and replace ifnull with coalesce

* Update get_column_description code in database.py file

* Remove a print statement

* Add keys to get on_duplicate query

* Add bactick wherever necessary

- Remove db_schema.py file

* Remove DATE_SUB as it is incompatible with postgres

- Fix prepare_filter_condition

* Add backtick and quotes wherever necessary
- Move get_database_size to frappe.db namespace
- fix some left out bugs and errors

* Add code to create key and unique index
- added mysql and posgres in their respective database.py

* Add more bacticks in queries and fix some errors
- Pass keys to on_duplicate_update method
- Replace MONTH with EXTRACT function
- Remove DATEDIFF and CURDATE usage

* Cast state value to int in toggle_two_factor_auth
- since two_factor_auth has the datatype of Int

* Refactor
- Replace Timediff with normal arithmetic operator
- Add MAX_COLUMN_LENGTH
- Remove Redundant code
- Add regexp character constant
- Move create_help_table to database.py
- Add get_full_text_search_condition method
- Inherit MariaDBTable from DBTable

* Replace Database instance with get_db method

* Move db_manager to separate file

* Refactor
- Remove some unwanted code
- Separate alter table code for postgres and mysql
- Replace data_type with column_type in database.py

* Make fulltext search changes in global_search.py

* Add empty string check

* Add root_password to site config

* Create cli command for postgres console

* Move setup of help database to setup_db.py

* Add get_database_list method

* Fix exception handling
- Replace bad_field handler with missing_column handler

* Fix tests and sql queries

* Fix import error

* Fix typo db -> database

* Fix error with make_table in help.py

* Try test for postgres

* Remove pyhton 2.7 version to try postgres travis test

* Add test fixes

* Add db_type to the config of test_site_postgres

* Enable query debug to check the reason for travis fail

* Add backticks to check if the test passes

* Update travis.yml
- Add postgres addon

* Try appending 'd_' to hash for db_name
- since postgres does not support dbname starting with a number

* Try adding db_type for global help to make travis work

* Add print statements to debug travis failure

* Enable transaction and remove debug flag

* Fix help table creation query (postgres)

* Fix import issue

* Add some checks to prevent errors
- Some doctypes used to get called even before they are created

* Try fixes

* Update travis config

* Fix create index for help table

* Remove unused code

* Fix queries and update travis config

* Fix ifnull replace logic (regex)

* Add query fixes and code cleanup

* Fix typo
- get_column_description -> get_table_columns_description

* Fix tests
- Replace double quotes in query with single quote

* Replace psycopg2 with psycopg2-binary to avoid warnings
- http://initd.org/psycopg/docs/install.html#binary-install-from-pypi

* Add multisql api

* Add few multisql queries

* Remove print statements

* Remove get_fulltext_search_condition method and replace with multi query

* Remove text slicing in create user

* Set default for 'values' argument in multisql

* Fix incorrect queries and remove few debug flags
- Fix multisql bug

* Force delete user to fix test
- Fix Import error
- Fix incorrect query

* Fix query builder bug

* Fix bad query

* Fix query (minor)

* Convert boolean text to int since is_private has datatype of int
- Some query changes like removed double quotes
and replace with interpolated string to pass multiple
value pass in one of the query

* Extend database class from an object to support python 2

* Fix query
- Add quotes around value passed to the query for variable comparision

* Try setting host_name for each test site
- To avoid "RemoteDisconnected" error while testing data migration test
- Update travis.yml to add hosts
- Remove unwanted commit in setup_help_database

* Set site hostname to data migration connector (in test file)
- To connect the same site host

* Fix duplicate entry issue
- the problem is in naming series file.
In previous commits I unknowingly changed a part of a series query
due to which series were not getting reset

* Replace few sql queries with orm methods

* Fix codacy

* Fix 'Doctype Sessions not found' issue

* Fix bugs induced during codacy fixes

* Fix Notification Test

- Use ORM instead of raw sql

* Set Date fallback value to 0001-01-01

- 0000-00-00 is invalid date in Postgres
- 0001-01-01 works in both

* Fix date filter method

* Replace double quotes with single quote for literal value

* Remove print statement

* Replace double quotes with single

* Fix tests

- Replace few raw sql with ORM

* Separate query for postgres

- update_fields_to_fetch_query

* Fix tests

- replace locate with strpos for postgres

* Fix tests

- Skip test for datediff
- convert bytes to str in escape method

* Remove TestBot

* Skip fieldname extraction

* Replace docshare raw sql with ORM

* Fix typo

* Fix ancestor query test

* Fix test data migration

* Remove hardcoded hostname

* Add default option and option list for db_type

* Remove frappe.async module

* Remove a debug flag from test

* Fix codacy

* fix import issue

* Convert classmethod to static method

* Convert few instance methods to static methods

* Remove some unused imports

* Fix codacy

- Add exception type
- Replace few instance methods with static methods
- Remove unsued import

* Fix codacy

* Remove unused code

* Remove some unused codes

- Convert some instance methods to static function

* Fix a issue with query modification

* Fix add_index query

* Fix query

* Fix update_auth patch

* Fix a issue with exception handling

* Add try catch to a reload_doc

* Add try-catch to file_manager_hook patch

* import update_gravatar to set_user_gravatar patch

* Undo all the wrong patch fixes

* Fix db_setup code 😪
- previously it was not restoring db from source SQL
which is why few old patched were breaking
(because they were getting different schema structure)

* Fix typo !

* Fix exception(is_missing_column) handling

* Add deleted code
- This code is only used in a erpnext patch.
Can be moved to that patch file

* Fix codacy

* Replace a mariadb specific function in a query used in validate_series

* Remove a debug flag

* Revert changes (rename_parent_and_child)

* Fix validate_one_root method

* Fix date format issue

* Fix codacy
- Disable a pylint for variable argument warning
- Convert an instance method to static method

* Add bandit.yml

The Codacy seems to use Bandit which generates
warning for every subprocess import and its usage during pytest
Since we have carefully used subprocess (avoided user input),
warnings needs to be avoided.
This can be removed if we have any alternative for subprocess usage.

* Skip start_process_with_partial_path check

* Fix typo

* Add python 2.7 test

* Move python versions in travis.yml

* Add python versions to jobs

* Overwrite python version inheritance for postgres in travis.yml

* Add quotes around python version in .travis.yml

* Add quotes around the name of the job

* Try a travis fix

* Try .travis.yml fix

* Import missing subprocess

* Refactor travis.yml

* Refactor travis.yml
- move install and tests commands to separate files
- Use matrix to build combination of python version and db type

* Make install.sh and run-tests.sh executable

* Add sudo required to travis.yml to allow sudo cmmands in shell files

* Load nvm

* Remove verbose flag from scripts

* Remove command-trace-print flag

* Change to build dir in before script

* Add absolute path for scripts

* Fix tests

* Fix typo

* Fix codacy
- fixes - "echo won't expand escape sequences." warning

* Append (_) underscore instead of 'd' for db_name

* Remove printf and use mysql execute flag
2018-09-21 10:20:48 +05:30
sivankar
a48b76b940 refactored code to add support in filters to get descendant or ancest… (#5847)
* refactored code to add support in filters to get descendant or ancestor for tree type documents

* added semicolon and comments to imporve code quality

* refactored code to add not ancestor and not descendants of filters

* added test cases for ancestors of and descendants of

* Translate labels
2018-08-14 12:19:18 +05:30
Saurabh
a135c6c431 resolved merge conflicts 2018-07-06 13:37:51 +05:30
Saurabh
e2b1ebe84c [Security][fix] To avoid possible sql injection via filters and or_filters parameters and tighten the field level checks (#5721)
* [fix] sanitize filters and or_filters to avoid sql injection

* add test cases for filter sanitizer

* codacy fix

* added test cases to test valid scenarios
2018-06-27 09:15:42 +05:30
Saurabh
55eaba90f5 [fix] Sanitize blacklisted functions 2018-05-30 15:25:40 +05:30
Nabin Hait
418d654331
Fixed sanitizing fields for tabDeleted Documents (#5329) 2018-05-30 15:01:12 +05:30
Suraj Shetty
3109123249 Permission improvements (#5494)
* Overwrite "if owner" restriction*
- Overwrite "if owner" restriction if user has other valid roles
(with access rights) without if_owner enabled for it in doc perm
eg:
If a doctype has 2 docperm for Role A and Role B and the docperm with
Role A has  if owner enabled.

Suppose User 1 has Role A, then User 1 should see only docs
which was created by him.

Now if User 2 has Role A and Role B, User 2 should be able to see all the
records because Role B has unrestricted access on doctype.
------------------------------------------------------

Previously in case of User 2, the restriction(of if owner) from Role A
used to reflect and User 2 was only able to view only records made by
him.

This commit will overwrite the if_owner restriction in such case.
- Refactor get_role_permission method

* Add advanced control for user permission
- Add ability to skip user permission check for specific doctype

* Refactor "linked with" method
- Add option to skip linked with doctype which has
ignore_user_permission enabled against them.

* Clear "linked with doctype" cache on doctype changes

* [Minor] Fix formatting

* Fix error and bugs in linked_with method

* Format code

* Add fix to get allowed docs from user_permission

* Fix has_user_permission bug

* Add before migrate patch for user permission

* Fix typo
2018-05-04 13:13:29 +05:30
Suraj Shetty
fb25fdb609 Auto email report fixes (#5410)
* fixes https://github.com/frappe/erpnext/issues/13319

* fixes https://github.com/frappe/erpnext/issues/13422

* change auto email report template

* change auto email report template

* Update data.py

* Update data.py
2018-04-10 10:53:05 +05:30
Suraj Shetty
0be4342649 User permissions redux (#5314)
* [start] removing simpler user permission structure

* permission changes
removed "apply user permission" from has_permission

* test fixes and other supporting fixes

* Removed Apply user permission code

* additional check to see if user has some user permissions related to a doctype

* test fixes

* removed a test

* Permission changes
- Code to restrict access if module is blocked for user.
- [Fix] ignore user perm if applicable

* List view restrictions ui changes

* removed unwanted apply user perm code

* permission (block module fix)

* permission fix

* change user permission
user permission will not be applied if user doesn't have role permission

* removed left "apply_user_permissions" property

* add user permission help template

* perm js refactor

* ui / ux changes
restrict list view before entering view if user is not permitterd
add description for if owner check in role permission manager

* codacy fixes

* permission code fix
add controller permission check
add blocked module check in build_permission

* optimized get blocked module method
2018-04-03 10:42:04 +05:30
Saurabh
676fdc0098 Merge branch 'master' into develop 2018-02-14 11:43:00 +05:30
Saurabh
40d818af90 [hotfix] sanitize fields to avoid mysql injection (#4994)
* [fix] sanitize fields to avoid mysql injection

* sanitize sql statements to avoid subqueries

* Added test cases

* Raise exception if mysql injection found in fields and related test-cases

* [fix] riase exception if blacklistes function or keyworkds found in fields
2018-02-13 14:45:51 +05:30
Rohit Waghchaure
87a0b6213b Merge branch 'master' into develop 2018-01-19 13:06:50 +05:30
rohitwaghchaure
97e18c809e [Fix] Date between filter not working properly on report and listview (#4810) 2018-01-18 15:39:12 +05:30
Faris Ansari
35811df7ef Remove get_count method, use the same db_query method for count (#4736)
* Remove get_count method, use the same db_query method for count

* Revert changes, db_query supported `count(name)` already
2018-01-02 17:55:51 +05:30
Faris Ansari
909234ab9d Listview get_count fix (#4740) 2018-01-02 11:55:49 +05:30
Nabin Hait
3e7a19e3bb
Update db_query.py 2017-11-15 14:05:01 +05:30
Manas Solanki
7031473915 fixed between filters (#4497) 2017-11-15 13:57:53 +05:30
Zlash65
539328f591 between date filter fix 2017-11-01 11:34:18 +05:30
Zlash65
fec63a32f1 test case added for count 2017-10-31 16:14:36 +05:30
Zlash65
2b78da673a better table join and fetch from multiple tables 2017-10-31 16:14:36 +05:30
Zlash65
7d2e30d529 count issue if child table exists in filter fix 2017-10-31 16:14:36 +05:30
Faris Ansari
bc45beb868 Show record count in list header 2017-10-23 20:10:15 +05:30
Faris Ansari
91861c7769 [minor]
- watch js files
- add frappe.db.get_list in js
2017-10-23 17:48:35 +05:30
Rushabh Mehta
5a31e583a2 Merge pull request #3912 from adityahase/fix-raise
Replace Py2 raise syntax with Py 3
2017-08-11 13:01:40 +05:30
Aditya Hase
2da880fc42 Replace Py2 raise syntax with Py 3 2017-08-11 00:38:58 +05:30
Aditya Hase
769338f136 Replaced all instances of basestring with six.string_types 2017-08-11 00:10:11 +05:30
mbauskar
53167626ab resolved merge conflicts 2017-08-02 18:32:30 +05:30
Makarand Bauskar
fc715efdbb [hotfix] fixes for AttributeError: 'NoneType' object has no attribute 'split' (#3840)
* [hotfix] fixes for AttributeError: 'NoneType' object has no attribute 'split'

* [tests] added test cases for not in and in filters
2017-08-02 12:47:10 +05:30
Rushabh Mehta
f609a478ae [refactor] user permissions (#3713)
* [refactor] user permissions

* [fix] tests

* [ux] user-permissions

* [minor] cleanup system settings

* [minor] end progress
2017-07-28 14:33:31 +05:30
Manas Solanki
d0857f8013 Apply User Permissions is being ignored for blank field values (#3496)
* issue where Apply User Permissions is being ignored for blank field values

* configurable settings and added test case

* Update system_settings.json

* save and use locally stored value for strict user permission
2017-07-06 10:48:12 +05:30
tundebabzy
bbde2ec4cf rewrite use of dict.iteritems (#3498)
* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility
2017-06-19 09:10:21 +05:30
Rushabh Mehta
0a5df4fb4d [debug] show where permission is failing (#3264)
* [debug] show where permission is failing

* [fix] document.py
2017-05-11 11:46:47 +05:30
Rushabh Mehta
262f361021 [minor] ignore permissions option for filters in db_query (#3234) 2017-05-06 12:43:47 +05:30
Makarand Bauskar
8eeda18692 [fixes] minor fix for between operator filter (#3196)
* [fixes] minor fix for between operator filter

* [minor] added a test case for between operator filter
2017-05-01 14:14:04 +05:30
Faris Ansari
95edd73b5b Fix Date between query 2017-04-28 14:43:32 +05:30
Rushabh Mehta
4832c3b50f [fix] missing first column in HTML for auto_email_report 2017-04-26 15:33:59 +05:30
Rushabh Mehta
346bbb008e [fix] ux for auto email report 2017-04-25 12:58:10 +05:30