Commit graph

267 commits

Author SHA1 Message Date
Aditya Hase
4740e4f7e6 refactor: Monitor
Do not collect request headers
Collect job wait time and whether it was scheduled
2020-03-05 15:06:15 +05:30
Aditya Hase
af3c4feb64 feat: Monitor
Collect HTTP Request and Background Job logs
2020-03-05 15:06:15 +05:30
Chinmay D. Pai
f239684740
chore: update werkzeug to 1.0.0
* move all werkzeug.wsgi and werkzeug.contrib imports to
  werkzeug.middleware to make things work.
* update pinned requirement to 1.0.0

Signed-off-by: Chinmay D. Pai <chinmaydpai@gmail.com>
2020-02-07 12:16:42 +05:30
Saurabh
264a3a2a5d fix: if maintenance mode on, connect database explicitly before… (#7731) 2019-06-28 19:39:25 +05:30
Faris Ansari
f63ad574e5 fix: Dont append request data to form_dict
This breaks many whitelisted methods which do not expect a data arg,
and also the methods which expect an optional data arg but the value is
not passed.

Examples:
frappe.desk.reportview.get
frappe.desk.doctype.bulk_update.bulk_update.submit_cancel_or_update_docs
2019-03-24 12:12:53 +05:30
Aditya Hase
b0faf64386 style: Linting fixes 2019-02-19 21:58:57 +05:30
Aditya Hase
9dea898bae refactor(recorder): Remove --record flag from bench serve 2019-02-19 21:58:55 +05:30
Aditya Hase
9a83ae322c refactor(recorder): Remove RecorderMiddleware 2019-02-19 21:58:55 +05:30
Aditya Hase
c54b8a65ba refactor(recorder): Major refactor 2019-02-19 21:58:55 +05:30
Aditya Hase
4594836731 refactor(recorder): Use recorder_start and recorder_stop hooks 2019-02-19 21:58:55 +05:30
Aditya Hase
be6f1a638b refactor(recorder): Remove unused import 2019-02-19 21:58:55 +05:30
Aditya Hase
f4d770739d refactor(recorder): Move recorder related functions to recorder.py 2019-02-19 21:58:55 +05:30
Aditya Hase
7a348d9dbb fix(recorder): Sometimes serializing cache args and kwargs throws Exception 2019-02-19 21:58:55 +05:30
Aditya Hase
f7c5e149db feat(recorder): Record cache activity by monkeypatching RedisWrapper instance methods 2019-02-19 21:58:55 +05:30
Aditya Hase
25ae5757fa refactor(recorder): Store calls list as one object, instead of a list of objects. 2019-02-19 21:58:55 +05:30
Aditya Hase
320aacd14c feat(recorder): Record time before and after function call 2019-02-19 21:58:55 +05:30
Aditya Hase
23727c9302 feat(recorder): Record each executed query as-it-is 2019-02-19 21:58:55 +05:30
Aditya Hase
3cdbb0b40f feat(recorder): Record stack for each frappe.db.sql call 2019-02-19 21:58:55 +05:30
Aditya Hase
dcd9a1053d feat(recorder): Create uuid in RecorderMiddleware 2019-02-19 21:58:55 +05:30
Aditya Hase
8a62b135e2 fix(recorder): Serialize Datetime objects with str() 2019-02-19 21:58:55 +05:30
Aditya Hase
3bc9ccd433 feat(recorder): Record hit count for every path 2019-02-19 21:58:54 +05:30
Aditya Hase
7431a6b682 refactor(recorder): Add explanation for patching frappe.db.sql 2019-02-19 21:58:54 +05:30
Aditya Hase
dc6ea19c2a refactor(recorder): Store path, request, calls at the end of request in cache 2019-02-19 21:58:54 +05:30
Aditya Hase
a0dd5ceb56 refactor(recorder): Don't push data to cache on every call 2019-02-19 21:58:54 +05:30
Aditya Hase
ef3d952379 feat(recorder): Group and lines by frappe.request.path and show on page 2019-02-19 21:58:54 +05:30
Aditya Hase
5ad3d3ce59 feat(recorder): Store arguments and results for all calls to frappe.db.sql in cache 2019-02-19 21:58:54 +05:30
Aditya Hase
07f9d6cefc feat(recorder): Add --record flag to use RecorderMiddleware 2019-02-19 21:58:54 +05:30
Rushabh Mehta
41d90fa6d1 BREAKING CHANGE: Comments are now handled separately from Communication and Version 2019-02-07 16:35:50 +05:30
Himanshu Mishra
dcf701087c fix: extract data as string before passing to json.loads (#6552) 2018-11-29 12:02:12 +05:30
Nabin Hait
d37d111f61 Fixed merge conflict 2018-10-15 15:53:26 +05:30
Aditya Hase
dc9b377115 feat(debugger): Allow VS Code python debugger to work (#6216) 2018-10-12 15:12:25 +05:30
Andrew McLeod
0031ca5d39 Pass Werkzeug/WSGI middleware bytestrings, not unicode. (#6180)
* Pass Werkzeug/WSGI middleware bytestrings, not unicode.

This fixes a bug that prevents website routing to non-ascii routes, such as
will be automatically created by items with non-ascii characters in their
names.

Werkzeug/WSGI actually does careful decoding of URLs passed to it which
ensure they are in the correct format internally. However, using a non-ascii
URL was causing a unicode error and exceptions.

In app.py, unicode_literals is in effect. SharedDataMiddleware and
StaticDataMiddleware instances are created to deal with files in 'assets'
and 'files', respectively. The paths passed need to be bytestrings; however,
due to the use of unicode, these were passed as unicode. This was causing a
later exception when using a (unicode) string method on the decoded URL
string. By explicitly passing bytestrings, this is avoided and everything
works as expected.

* Instead of calling str, call .encode('ascii') method for compatibility with Python3.

* Unicode URL WSGI fix: After testing, it is actually the key not the value of
the export that matters, however both can be wrapped in str() to get 'bytes'
in Python 2 and unicode in Python 3 (which WSGI seems to expect).
2018-10-02 15:32:59 +05:30
Rushabh Mehta
2e6a202652 Postgres support for Frappe (#5919)
* [start] postgres

* [wip] started refactoring db_schema

* Add psycopg2 to requirements.txt

* Add support for Postgres SQL

- Separate frameworkSQL, database, schema, setup_db file for
mariaDB and postgres
- WIP

* Remove quotes from sql to make it compatible with postgres as well

* Moved some code from db_schema to database.py

* Move code from db_schema to schema.py

Add other required refactoring

* Add schema chages

* Remove redundant code in file

* Add invalid column name exception class to exceptions.py

* Add back tick in query wherever needed and replace ifnull with coalesce

* Update get_column_description code in database.py file

* Remove a print statement

* Add keys to get on_duplicate query

* Add bactick wherever necessary

- Remove db_schema.py file

* Remove DATE_SUB as it is incompatible with postgres

- Fix prepare_filter_condition

* Add backtick and quotes wherever necessary
- Move get_database_size to frappe.db namespace
- fix some left out bugs and errors

* Add code to create key and unique index
- added mysql and posgres in their respective database.py

* Add more bacticks in queries and fix some errors
- Pass keys to on_duplicate_update method
- Replace MONTH with EXTRACT function
- Remove DATEDIFF and CURDATE usage

* Cast state value to int in toggle_two_factor_auth
- since two_factor_auth has the datatype of Int

* Refactor
- Replace Timediff with normal arithmetic operator
- Add MAX_COLUMN_LENGTH
- Remove Redundant code
- Add regexp character constant
- Move create_help_table to database.py
- Add get_full_text_search_condition method
- Inherit MariaDBTable from DBTable

* Replace Database instance with get_db method

* Move db_manager to separate file

* Refactor
- Remove some unwanted code
- Separate alter table code for postgres and mysql
- Replace data_type with column_type in database.py

* Make fulltext search changes in global_search.py

* Add empty string check

* Add root_password to site config

* Create cli command for postgres console

* Move setup of help database to setup_db.py

* Add get_database_list method

* Fix exception handling
- Replace bad_field handler with missing_column handler

* Fix tests and sql queries

* Fix import error

* Fix typo db -> database

* Fix error with make_table in help.py

* Try test for postgres

* Remove pyhton 2.7 version to try postgres travis test

* Add test fixes

* Add db_type to the config of test_site_postgres

* Enable query debug to check the reason for travis fail

* Add backticks to check if the test passes

* Update travis.yml
- Add postgres addon

* Try appending 'd_' to hash for db_name
- since postgres does not support dbname starting with a number

* Try adding db_type for global help to make travis work

* Add print statements to debug travis failure

* Enable transaction and remove debug flag

* Fix help table creation query (postgres)

* Fix import issue

* Add some checks to prevent errors
- Some doctypes used to get called even before they are created

* Try fixes

* Update travis config

* Fix create index for help table

* Remove unused code

* Fix queries and update travis config

* Fix ifnull replace logic (regex)

* Add query fixes and code cleanup

* Fix typo
- get_column_description -> get_table_columns_description

* Fix tests
- Replace double quotes in query with single quote

* Replace psycopg2 with psycopg2-binary to avoid warnings
- http://initd.org/psycopg/docs/install.html#binary-install-from-pypi

* Add multisql api

* Add few multisql queries

* Remove print statements

* Remove get_fulltext_search_condition method and replace with multi query

* Remove text slicing in create user

* Set default for 'values' argument in multisql

* Fix incorrect queries and remove few debug flags
- Fix multisql bug

* Force delete user to fix test
- Fix Import error
- Fix incorrect query

* Fix query builder bug

* Fix bad query

* Fix query (minor)

* Convert boolean text to int since is_private has datatype of int
- Some query changes like removed double quotes
and replace with interpolated string to pass multiple
value pass in one of the query

* Extend database class from an object to support python 2

* Fix query
- Add quotes around value passed to the query for variable comparision

* Try setting host_name for each test site
- To avoid "RemoteDisconnected" error while testing data migration test
- Update travis.yml to add hosts
- Remove unwanted commit in setup_help_database

* Set site hostname to data migration connector (in test file)
- To connect the same site host

* Fix duplicate entry issue
- the problem is in naming series file.
In previous commits I unknowingly changed a part of a series query
due to which series were not getting reset

* Replace few sql queries with orm methods

* Fix codacy

* Fix 'Doctype Sessions not found' issue

* Fix bugs induced during codacy fixes

* Fix Notification Test

- Use ORM instead of raw sql

* Set Date fallback value to 0001-01-01

- 0000-00-00 is invalid date in Postgres
- 0001-01-01 works in both

* Fix date filter method

* Replace double quotes with single quote for literal value

* Remove print statement

* Replace double quotes with single

* Fix tests

- Replace few raw sql with ORM

* Separate query for postgres

- update_fields_to_fetch_query

* Fix tests

- replace locate with strpos for postgres

* Fix tests

- Skip test for datediff
- convert bytes to str in escape method

* Remove TestBot

* Skip fieldname extraction

* Replace docshare raw sql with ORM

* Fix typo

* Fix ancestor query test

* Fix test data migration

* Remove hardcoded hostname

* Add default option and option list for db_type

* Remove frappe.async module

* Remove a debug flag from test

* Fix codacy

* fix import issue

* Convert classmethod to static method

* Convert few instance methods to static methods

* Remove some unused imports

* Fix codacy

- Add exception type
- Replace few instance methods with static methods
- Remove unsued import

* Fix codacy

* Remove unused code

* Remove some unused codes

- Convert some instance methods to static function

* Fix a issue with query modification

* Fix add_index query

* Fix query

* Fix update_auth patch

* Fix a issue with exception handling

* Add try catch to a reload_doc

* Add try-catch to file_manager_hook patch

* import update_gravatar to set_user_gravatar patch

* Undo all the wrong patch fixes

* Fix db_setup code 😪
- previously it was not restoring db from source SQL
which is why few old patched were breaking
(because they were getting different schema structure)

* Fix typo !

* Fix exception(is_missing_column) handling

* Add deleted code
- This code is only used in a erpnext patch.
Can be moved to that patch file

* Fix codacy

* Replace a mariadb specific function in a query used in validate_series

* Remove a debug flag

* Revert changes (rename_parent_and_child)

* Fix validate_one_root method

* Fix date format issue

* Fix codacy
- Disable a pylint for variable argument warning
- Convert an instance method to static method

* Add bandit.yml

The Codacy seems to use Bandit which generates
warning for every subprocess import and its usage during pytest
Since we have carefully used subprocess (avoided user input),
warnings needs to be avoided.
This can be removed if we have any alternative for subprocess usage.

* Skip start_process_with_partial_path check

* Fix typo

* Add python 2.7 test

* Move python versions in travis.yml

* Add python versions to jobs

* Overwrite python version inheritance for postgres in travis.yml

* Add quotes around python version in .travis.yml

* Add quotes around the name of the job

* Try a travis fix

* Try .travis.yml fix

* Import missing subprocess

* Refactor travis.yml

* Refactor travis.yml
- move install and tests commands to separate files
- Use matrix to build combination of python version and db type

* Make install.sh and run-tests.sh executable

* Add sudo required to travis.yml to allow sudo cmmands in shell files

* Load nvm

* Remove verbose flag from scripts

* Remove command-trace-print flag

* Change to build dir in before script

* Add absolute path for scripts

* Fix tests

* Fix typo

* Fix codacy
- fixes - "echo won't expand escape sequences." warning

* Append (_) underscore instead of 'd' for db_name

* Remove printf and use mysql execute flag
2018-09-21 10:20:48 +05:30
Aditya Hase
f3b959b1fe frappe: refactor async to realtime (#6015)
* frappe: refactor async to realtime

* frappe: refactor async to is_async
2018-08-28 15:31:20 +05:30
Achilles Rasquinha
0f3b717447 conditional checks 2018-03-12 22:59:23 +05:30
Achilles Rasquinha
4d160052b4 Check if server is alive and let socket.io then request 2018-03-12 22:58:16 +05:30
crossxcell99
9a51d76665 Accomodate header variations (#5064) 2018-02-23 11:17:51 +05:30
Revant Nandgaonkar
9f0c0aecf6 handle IndexError for form_dict 2018-02-15 13:24:19 +05:30
Sachin Mane
8d8358644c Fix JSON parse failure incase POST with no content is called. (#4743) 2018-01-03 17:48:10 +05:30
Faris Ansari
2724205955 Handle content_type application/json in request 2017-12-17 00:31:11 +05:30
Achilles Rasquinha
b8825ca116 Pure Python MySQL instead mysqlclient (with Python C API) (#4280)
* added PyMYSQL instead mysqlclient-python

* added pymysql import

* fixed db module import

* foxed self._conn wrapper

* updated cursor migration

* removed existent converters

* some more fixes towards API port

* modified travis

* updated doctype and revereted yml

* modified travis

* refreshed logging

* raw fix

* moved from hard coded constants to declarable instances

* moved from hard coded constants to declarable instances

* refactoring and logging

* moved to global import declaration

* fixed codacy

* unfixed codacy

* minor fix

* binary_type has a single

* deprecated alternative

* merged with latest

* fixed merge conflicts

* using deprecated alternative

* raw fix

* stupid fix

* using StringTypes instead

* brutal hack

* log

* tundebazy to the rescue

* fixed content_hash error

* frappe/database.py

* frappe/database.py

* updated database.py

* updated requirements

* updated requirements

* fixed codacy

* fixed codacy

* moved from DatabaseOperationalError to pymysql.InternalError

* moved from DatabaseOperationalError to pymysql.InternalError

* fixed codacy

* empty commit

* fixed codacy

* fixed codacy
2017-11-06 16:42:01 +05:30
Faris Ansari
d20f9e2895 Data Migration Tool (for hub) (#4144)
* migration tool

* custom field for primary key added

* foreign key and multiple linking F_key issue resolved

* refined code

* many-to-one mapping temp fix

* added support for pre-process + cleaned up code

* [various] fixes to setup wizard for developer mode, frappe.enqueue_doc, share with assign

* Refactor data migration module

* added migration for hub

* Add "Skip errors" in data import tool

* move db_set to document.py

* Add Data Migration Run

* Dynamic Migration ID

* move run() from Mapping to Run

* Push Deleted Documents

* fixes

* [migration] doc operation counts

* insert and update instead of push in connection

* fix count and total_pages, skip sync if total_pages is 0

* [migration] child tables

* fix complete()

* [page] remove required libs

* Add sidebar.js, rename old sidebar.js to form_sidebar.js

* [minor] get_empty_state fixes

* svg in icon

* remove image check

* fix codacy

* fix is_child_table check

* [connector] add get_list()

* Add test for Data Migration Run

* fix test

* truncate tabNote

* fix test

* sync todo with event to fix test

* fix db count

* [mapping] export Mapping to json

* Add docs for Data Migration Tool

* [migration] pull data as list, test case

* [hub] remove mapping export to files

* Pull refactor

* [test]

* Add comments

* [mapping] exec in mapping formula

* fix codacy

* fix codacy

* Remove exec for pre-process and post-process

* Add pre and post process for Push

* Remove formula

* fixes

* [refactor] add failed_log to pull, handle error in pull

* [test] Push, pull, update

* Fix codacy, fix insert_doc for pull

* Set migration id on successful insert

* fix update_doc

* fix update_doc

* method is a function

* child table mapping

* Refactor logging

* fix update_doc again

* fix hostname, password

* update docs, minors

* Remove assign_if_none

* Remove error handling from connection methods

* [refactor] Data migration run

* Break push stages into methods

* Migration run refactor

- fix test
- add separate fields for logging

* fix codacy

* fix hostname password

* fix test
2017-10-05 11:15:35 +05:30
Rushabh Mehta
faee27f773 [fix] [translation] switch to desk, #3328 2017-09-04 22:26:12 +05:30
Aditya Hase
24f2669795 Use str instead of bytes for keys in exports param 2017-09-04 16:05:57 +05:30
Rushabh Mehta
7dedc69461 [print-style] sync, fix tests and remove selection 2017-08-18 14:53:38 +05:30
Rushabh Mehta
19e9834850 [added] print style 2017-08-17 18:34:14 +05:30
Aditya Hase
0a6ecbf2c5 Don't unnecessarily encode os.path.join() 2017-08-11 18:09:32 +05:30
Rushabh Mehta
aa370aef36 [tests] less verbose logs 2017-08-10 11:02:36 +05:30
tundebabzy
bbde2ec4cf rewrite use of dict.iteritems (#3498)
* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility

* dict.iteritems to six.iteritems(dict) for python3 compatibility
2017-06-19 09:10:21 +05:30
Rushabh Mehta
687af5ccc4 [minor] set response in app.py 2017-05-23 11:17:09 +05:30