pypika internally keeps copying query builder object because everything
is supposed to be immutable in pypika design, this however is terribly
slow. Often query generation takes more time than query execution.
This PR makes query builder mutable inside `get_query` function to avoid
copying while applying fields, filters, limit, order etc.
It's marked as immutable again when sending it back to users of the API.
This makes cache implementation uniform for all methods on db API. It's
weird that this specific method was caching in redis, which defies
expectations.
* fix: reduce bulk insert batch size
Back when this feature was added it used to lazily evaluate the input.
Now the iterator is consumed upfront so large batch sizes == huge memory usage.
* perf: bring back iterator for bulk_insert
Bulk insert used to support iterator for consuming arbitrarily large
amount of data and inserting it. Since child table support was added, it
can't do it anymore because that requires collecting values.
This change now brings back iterators by batching input iterator (by
default 1000) documents.
This is almost as good as original change from design POV. Performance
is still meh for flat documents.
* fix: Avoid Snapshot violation
- Main thread created and "read" user
- Other thread modified something
- Main thread wants to delete or "write" to same row.
This violates snapshot isolation.
* fix: treat snapshot violation as deadlock for now
* test: handle snapshot violations
(wal was already enabled in the file itself, no harm setting it here for consistency.
Reference: 45622c7d54
Co-authored-by: 18alantom <2.alan.tom@gmail.com>
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
* feat: mysqlclient
* fix: update error attrs
* fix: decode mogrified query to unicode
* fix: do some cleanup
* chore: disable cleanup for now
* fix: remove unnecessary call to as_unicode
* test: skip perf test for now
* fix: fallback to empty str
* fix: unbuffered cursor support
* fix: update converters and other changes
* fix: add cleanup back
* perf: improve timedelta converter
* fix: dont attempt to run query when explain flag is set
* test: cleanup tests
* chore: remove commented code
* perf: store conf as local var
* chore: ensure sequence
---------
Co-authored-by: Ankush Menat <ankush@frappe.io>
