Commit graph

54 commits

Author SHA1 Message Date
Faris Ansari
5e2c2d8337
fix: Index rendered page content (#7405)
* fix: Index rendered page content

- Order results based on words matched in title

* fix: Reset session user after indexing page

* chore: comments
2019-05-12 19:21:07 +05:30
Faris Ansari
766e26ac5f
feat: Index all web pages for search (#7322)
* feat: Index Web Pages for search

- Walks and indexes all webpages in global_search

* test: Web Page Indexing
2019-04-26 12:41:17 +05:30
Faris Ansari
5d1ecb2e47 fix: Global Search fixes
- Update title, route and published in sync_value
- Update search_template for BS4
2019-03-03 00:27:13 +05:30
Sagar Vora
89efa968d8 Merge branch 'master' into develop 2019-02-05 18:39:43 +05:30
Chinmay Pai
9873c04231 fix(sync_global_search): decode json object as str (#6867) 2019-01-31 15:16:47 +05:30
Saurabh
76bd4e4f19 Merge branch 'master' into develop 2019-01-30 12:36:43 +05:30
Rushabh Mehta
0704928fd8 fix(global search): fallback if redis is not yet started 2019-01-30 11:26:07 +05:30
Rushabh Mehta
d09792c606 fix: refactor global search to not insert after every commit (#6845)
* fix: refactor global search to not insert after every commit

* feat: configuration to disable_global_search
2019-01-30 11:13:51 +05:30
Rushabh Mehta
6c7be62f24 fix: file upload via url 2019-01-29 12:42:17 +05:30
Faris Ansari
3b86f16b73 New Control: Table MultiSelect (#6675)
* feat(control): Add Table MultiSelect control

* fix: Use btn-group instead of span

* fix: Remove functionality

* fix: Add 'Table MultiSelect' to Field doctypes

* fix: Replace usage of string 'Table' with array `table_fields`

* fix: Use internal array to store values instead of building from HTML elements

* fix(style): Add semicolon

* fix: Read only mode and click to navigate to form

* style: indent

* fix: fallback to empty array

* fix: Add formatters in js and py

* style: missing semicolon

* fix: Add docfield validation
2018-12-28 15:36:59 +05:30
Chinmay Pai
5f0dbecfed data-import: fix UnicodeEncodeError on data import in python2 (#6448)
why is this not built into re by default?

Signed-off-by: Chinmay Pai <chinmaydpai@gmail.com>
2018-11-13 10:35:27 +05:30
Rushabh Mehta
2e6a202652 Postgres support for Frappe (#5919)
* [start] postgres

* [wip] started refactoring db_schema

* Add psycopg2 to requirements.txt

* Add support for Postgres SQL

- Separate frameworkSQL, database, schema, setup_db file for
mariaDB and postgres
- WIP

* Remove quotes from sql to make it compatible with postgres as well

* Moved some code from db_schema to database.py

* Move code from db_schema to schema.py

Add other required refactoring

* Add schema chages

* Remove redundant code in file

* Add invalid column name exception class to exceptions.py

* Add back tick in query wherever needed and replace ifnull with coalesce

* Update get_column_description code in database.py file

* Remove a print statement

* Add keys to get on_duplicate query

* Add bactick wherever necessary

- Remove db_schema.py file

* Remove DATE_SUB as it is incompatible with postgres

- Fix prepare_filter_condition

* Add backtick and quotes wherever necessary
- Move get_database_size to frappe.db namespace
- fix some left out bugs and errors

* Add code to create key and unique index
- added mysql and posgres in their respective database.py

* Add more bacticks in queries and fix some errors
- Pass keys to on_duplicate_update method
- Replace MONTH with EXTRACT function
- Remove DATEDIFF and CURDATE usage

* Cast state value to int in toggle_two_factor_auth
- since two_factor_auth has the datatype of Int

* Refactor
- Replace Timediff with normal arithmetic operator
- Add MAX_COLUMN_LENGTH
- Remove Redundant code
- Add regexp character constant
- Move create_help_table to database.py
- Add get_full_text_search_condition method
- Inherit MariaDBTable from DBTable

* Replace Database instance with get_db method

* Move db_manager to separate file

* Refactor
- Remove some unwanted code
- Separate alter table code for postgres and mysql
- Replace data_type with column_type in database.py

* Make fulltext search changes in global_search.py

* Add empty string check

* Add root_password to site config

* Create cli command for postgres console

* Move setup of help database to setup_db.py

* Add get_database_list method

* Fix exception handling
- Replace bad_field handler with missing_column handler

* Fix tests and sql queries

* Fix import error

* Fix typo db -> database

* Fix error with make_table in help.py

* Try test for postgres

* Remove pyhton 2.7 version to try postgres travis test

* Add test fixes

* Add db_type to the config of test_site_postgres

* Enable query debug to check the reason for travis fail

* Add backticks to check if the test passes

* Update travis.yml
- Add postgres addon

* Try appending 'd_' to hash for db_name
- since postgres does not support dbname starting with a number

* Try adding db_type for global help to make travis work

* Add print statements to debug travis failure

* Enable transaction and remove debug flag

* Fix help table creation query (postgres)

* Fix import issue

* Add some checks to prevent errors
- Some doctypes used to get called even before they are created

* Try fixes

* Update travis config

* Fix create index for help table

* Remove unused code

* Fix queries and update travis config

* Fix ifnull replace logic (regex)

* Add query fixes and code cleanup

* Fix typo
- get_column_description -> get_table_columns_description

* Fix tests
- Replace double quotes in query with single quote

* Replace psycopg2 with psycopg2-binary to avoid warnings
- http://initd.org/psycopg/docs/install.html#binary-install-from-pypi

* Add multisql api

* Add few multisql queries

* Remove print statements

* Remove get_fulltext_search_condition method and replace with multi query

* Remove text slicing in create user

* Set default for 'values' argument in multisql

* Fix incorrect queries and remove few debug flags
- Fix multisql bug

* Force delete user to fix test
- Fix Import error
- Fix incorrect query

* Fix query builder bug

* Fix bad query

* Fix query (minor)

* Convert boolean text to int since is_private has datatype of int
- Some query changes like removed double quotes
and replace with interpolated string to pass multiple
value pass in one of the query

* Extend database class from an object to support python 2

* Fix query
- Add quotes around value passed to the query for variable comparision

* Try setting host_name for each test site
- To avoid "RemoteDisconnected" error while testing data migration test
- Update travis.yml to add hosts
- Remove unwanted commit in setup_help_database

* Set site hostname to data migration connector (in test file)
- To connect the same site host

* Fix duplicate entry issue
- the problem is in naming series file.
In previous commits I unknowingly changed a part of a series query
due to which series were not getting reset

* Replace few sql queries with orm methods

* Fix codacy

* Fix 'Doctype Sessions not found' issue

* Fix bugs induced during codacy fixes

* Fix Notification Test

- Use ORM instead of raw sql

* Set Date fallback value to 0001-01-01

- 0000-00-00 is invalid date in Postgres
- 0001-01-01 works in both

* Fix date filter method

* Replace double quotes with single quote for literal value

* Remove print statement

* Replace double quotes with single

* Fix tests

- Replace few raw sql with ORM

* Separate query for postgres

- update_fields_to_fetch_query

* Fix tests

- replace locate with strpos for postgres

* Fix tests

- Skip test for datediff
- convert bytes to str in escape method

* Remove TestBot

* Skip fieldname extraction

* Replace docshare raw sql with ORM

* Fix typo

* Fix ancestor query test

* Fix test data migration

* Remove hardcoded hostname

* Add default option and option list for db_type

* Remove frappe.async module

* Remove a debug flag from test

* Fix codacy

* fix import issue

* Convert classmethod to static method

* Convert few instance methods to static methods

* Remove some unused imports

* Fix codacy

- Add exception type
- Replace few instance methods with static methods
- Remove unsued import

* Fix codacy

* Remove unused code

* Remove some unused codes

- Convert some instance methods to static function

* Fix a issue with query modification

* Fix add_index query

* Fix query

* Fix update_auth patch

* Fix a issue with exception handling

* Add try catch to a reload_doc

* Add try-catch to file_manager_hook patch

* import update_gravatar to set_user_gravatar patch

* Undo all the wrong patch fixes

* Fix db_setup code 😪
- previously it was not restoring db from source SQL
which is why few old patched were breaking
(because they were getting different schema structure)

* Fix typo !

* Fix exception(is_missing_column) handling

* Add deleted code
- This code is only used in a erpnext patch.
Can be moved to that patch file

* Fix codacy

* Replace a mariadb specific function in a query used in validate_series

* Remove a debug flag

* Revert changes (rename_parent_and_child)

* Fix validate_one_root method

* Fix date format issue

* Fix codacy
- Disable a pylint for variable argument warning
- Convert an instance method to static method

* Add bandit.yml

The Codacy seems to use Bandit which generates
warning for every subprocess import and its usage during pytest
Since we have carefully used subprocess (avoided user input),
warnings needs to be avoided.
This can be removed if we have any alternative for subprocess usage.

* Skip start_process_with_partial_path check

* Fix typo

* Add python 2.7 test

* Move python versions in travis.yml

* Add python versions to jobs

* Overwrite python version inheritance for postgres in travis.yml

* Add quotes around python version in .travis.yml

* Add quotes around the name of the job

* Try a travis fix

* Try .travis.yml fix

* Import missing subprocess

* Refactor travis.yml

* Refactor travis.yml
- move install and tests commands to separate files
- Use matrix to build combination of python version and db type

* Make install.sh and run-tests.sh executable

* Add sudo required to travis.yml to allow sudo cmmands in shell files

* Load nvm

* Remove verbose flag from scripts

* Remove command-trace-print flag

* Change to build dir in before script

* Add absolute path for scripts

* Fix tests

* Fix typo

* Fix codacy
- fixes - "echo won't expand escape sequences." warning

* Append (_) underscore instead of 'd' for db_name

* Remove printf and use mysql execute flag
2018-09-21 10:20:48 +05:30
Prateeksha Singh
213279ec87
Merge pull request #5783 from szuyxy/global_search_support_and
[Enhance] Global search multi key words to pin point search result
2018-08-14 12:09:29 +05:30
Saurabh
2a4b8f1ec3 [fix] merge-conflicts 2018-07-09 15:56:43 +05:30
Prateeksha Singh
479f51e23a [global-search] add doc check for route 2018-07-09 12:49:26 +05:30
Saurabh
a135c6c431 resolved merge conflicts 2018-07-06 13:37:51 +05:30
Fisher Yu
7b89d3ee46
Update global_search.py
remove trailing space
2018-07-06 08:45:07 +08:00
Fisher Yu
f0aed45c76
Update global_search.py
support the user to use and operator (&) in the search text, to search target document by combined multi search term(key words separated by &, thus we can search purchase order by vendor name, the items purchased all together!
2018-07-05 20:25:14 +08:00
Fisher Yu
b912b40f62 [New Feature] Global Search By User Tag for every doc (#5711)
* Update global_search.py

add user tag

* Update tags.py

trigger update global search table

* Update tags.py

remove trailing space

* Update global_search.py

remove trailing space

* Update global_search.py

remove title user tag

* Update global_search.py

* Update global_search.py
2018-07-05 10:49:43 +05:30
Nabin Hait
58b7d766dd minor fix in global search 2018-06-28 13:25:56 +05:30
Prateeksha Singh
fdde12b85a [global-search] trim to varchar length 2018-06-25 17:41:27 +05:30
Rushabh Mehta
9e2e65f305 [fix] varchar_len in global_search.py 2018-06-25 16:00:01 +05:30
Prateeksha Singh
cf1338bbca [global-search] route check 2018-06-25 09:45:12 +05:30
Prateeksha Singh
76be772e4c [global-search] trim strings directly 2018-06-22 17:21:00 +05:30
Prateeksha Singh
f3b2ba50f0 [global-search] trim varchar columns 2018-06-22 04:10:15 +05:30
tundebabzy
d5dcc0b98f Issue 4616 (#4617)
* fail silently when key not found

* PEP 8: spacing, docstrings

* codacy
2017-12-14 14:57:05 +05:30
Revant Nandgaonkar
4df6135c3c enqueue_after_commit option added to frappe.enqueue (#4167)
* enqueue_after_commit option added to frappe.enqueue

* Fixes

Codacy https://www.codacy.com/app/frappe/frappe/pullRequest?prid=934733
clear frappe.flags.enqueue_after_commit after executing enqueued jobs

* sync_global_search() uses enqueue_after_commit=True

* Update database.py

* webhooks enqueue after commit

* Update __init__.py
2017-10-18 11:43:02 +05:30
pratu16x7
9cee53c33f [fix] specify exception 2017-09-08 16:03:51 +05:30
Rushabh Mehta
f66cfcece5 Update global_search.py 2017-09-08 15:47:32 +05:30
pratu16x7
1a3d0bb459 [fix] clear messages on exception 2017-09-08 15:46:03 +05:30
pratu16x7
0aaaca9b1a [fix] ignore results with non-existent doctypes 2017-09-08 15:45:36 +05:30
Aditya Hase
e6bd1e4995 Replaced all instances of 'unicode' with 'six.text_type' (#3825) 2017-08-02 16:57:39 +05:30
Aditya Hase
928ebb66a6 Replaced HTMLParser imports with six.moves.html_parser (#3812) 2017-07-31 10:58:22 +05:30
Aditya Hase
bab692ff59 Replaced mixed usage of tabs and spaces for indents with tabs only (#3813)
Python 3 (PEP 8) strictly disallows mixed use of tabs and spaces
for indentation
frappe innstalltion only throws error on frappe/utils/jinja.py as
of now but the rest are almost equally likely to fail, so changed
all of them in one commit
2017-07-31 10:50:26 +05:30
Prateeksha Singh
d8fb2c2c0d [fix] Enqueue syncing global search (#3455)
* [fix] Enqueue syncing global search

* make passing flags optional

* Create global_search.py
2017-06-16 11:02:22 +05:30
Nabin Hait
72ba2b41d5 Global search patch: Ignore checking is_website_published, if controller does not exists (#3379) 2017-05-24 17:52:01 +05:30
Nabin Hait
44619bcf38 Insert records into global search table in small batches, added progress bar in patch (#3313) 2017-05-17 19:51:44 +05:30
Nabin Hait
d9c559777e Ignore single doctypes for global search 2017-05-16 16:47:11 +05:30
Nabin Hait
6cb0d6da77 Re-run global search patch (#3271) 2017-05-12 15:25:31 +05:30
Nabin Hait
901e8eaf92 minor fix global search 2017-05-04 12:01:57 +05:30
Saurabh
0b2acfc4c8 [hot-fix] reset length to 140 2017-04-29 18:39:51 +05:30
Rushabh Mehta
592ac64fc1 [fix] ignore bulk update for global search 2017-04-29 17:50:05 +05:30
Rushabh Mehta
7e625f9219 [fix] escape doctype for patch 2017-04-28 15:11:47 +05:30
Nabin Hait
5882df1ca1 Optimised rebuilding global search 2017-04-22 15:42:56 +05:30
Prateeksha Singh
ea9d777238 Still more search fixes (#3054)
* [fix] awesome bar translation frappe/erpnext#8279 frappe/erpnext#8306

* [fix] frappe/erpnext#8348

* unescape and remove html entities

* ellipsify long field values

* [fix] test

* Add global search in custom field, Email Inbox searchable

* remove beautiful soup, make_field test case

* [fix] test

* Patch to update existing record in global search

* restore update_published patch

* more specific test cases

* Code descriptions for complex result ellipsifying
2017-04-19 15:47:21 +05:30
Prateeksha Singh
f2df24803c Search refactor (#2983)
* [start] basic sidebar function

* remove flicker on fetch

* use promises for global and help results

* add summary view

* single modal, sidebar keyboard selection

* Placeholder and no_results state, result render prep

* Specific empty states, results template no longer special: thus remove reset(), forum results trial

* Add loading state

* more results working

* commonify more results, added images

* All nav results in modal, with recents

* use common utils in awesome bar

* UI fixes

* Remove forum results

* frappe/erpnext#8222 frappe/erpnext#8264
2017-04-03 13:28:37 +05:30
pratu16x7
09bf08ca9b sanitize and trim search result description 2017-03-29 17:10:48 +05:30
pratu16x7
39e4952ccf Title case, use gs title column, exclude docstatus 2 2017-03-13 10:12:25 +05:30
pratu16x7
5a62375ee5 generic doctype check 2017-03-09 15:55:28 +05:30
pratu16x7
82c122ecea child table (__) exclusion 2017-03-09 15:54:18 +05:30