vassili/seitime-frappe
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 4
48464 commits 1 branch 0 tags 587 MiB
Commit graph

117 commits

Author SHA1 Message Date
Ankush Menat
fdba41c682
perf: misc client cache improvements (#29070) 
* perf: Reduce penalty for lack of redis connection

If redis isn't running than this client cache is slower than default
implementation because of the extra locking overhead.

* test: update perf redis counts

* perf: cache table columns in client-cache

* fix: race condition on cache-client_cache init

Rare but apparant in synthetic benchmarks.

Cache is set but client cache is still being initialized then request
will fail.

* perf: Don't run notifications when loading document

WHAT?

* fix: use cached doc to repopulate

* perf: reduce get_meta calls
 2025-01-07 16:14:43 +05:30
Ankush Menat
b9c0662de6 fix: Make client cache work without redis too 2025-01-07 13:46:47 +05:30
Ankush Menat
06816f2817 perf: client-cache for defaults 2025-01-07 12:51:10 +05:30
Ankush Menat
33bd1bc09b fix: Flushall shouldn't attempt reading keys 2025-01-07 11:56:22 +05:30
Ankush Menat
b5e2569c0c feat: statistics for cache performance 
These can be used for logging or debugging the behaviour of the cache.
 2025-01-07 11:47:52 +05:30
Ankush Menat
8dd4ad53e2 fix: data race between GET and INVALIDATE 
Fix for the documented example given here: https://redis.io/docs/latest/develop/reference/client-side-caching/#avoiding-race-conditions
 2025-01-07 11:09:42 +05:30
Ankush Menat
c14d416d19 docs: add usage / notes in code 2025-01-06 19:28:37 +05:30
Ankush Menat
15f5adc25a refactor: use namedtuple for readability 2025-01-06 19:28:33 +05:30
Ankush Menat
b51cfc1705 perf: minimize penalty for broken client cache 2025-01-06 18:57:57 +05:30
Ankush Menat
119af71ae3 refactor: variable names, force RESP2 2025-01-06 18:57:57 +05:30
Ankush Menat
98b1df7dac fix: guard all writes with an RLock 2025-01-06 18:57:57 +05:30
Ankush Menat
53f085e0f4 test: ttl and maxsize 2025-01-06 18:57:57 +05:30
Ankush Menat
55ae5615d0 fix: clear all meta cache only when doctype is not specified 2025-01-06 18:57:57 +05:30
Ankush Menat
f332852415 fix: start tracking keys set by us immediately 2025-01-06 18:57:57 +05:30
Ankush Menat
f74ada155e fix: Don't serve local cache values when it becomes unhealthy 2025-01-06 18:57:57 +05:30
Ankush Menat
45d414fe29 refactor: per-key TTL 2025-01-06 18:57:57 +05:30
Ankush Menat
21f0cda732 refactor: Reuse original cache connection 
Drops redis connection per worker from 3 to 2!

Nothing wrong with doing this.
 2025-01-06 18:57:57 +05:30
Ankush Menat
01cfa647a2 fix: set/del locally too 
This is required if we do NOLOOP
 2025-01-06 18:57:57 +05:30
Ankush Menat
e7139a1395 perf: store meta in client cache 2025-01-06 18:57:57 +05:30
Ankush Menat
cd47bee65e fix: Avoid data races by accessing request specific cache 
Imagine:
| client 1 | client 2 |
| ---      | ---      |
| rd(x)    |          |
|          |  wr(x)   |
| inv(x)   |          |
| rd(x)    |          |  <- This will end up using request specific cache!
 2025-01-06 18:57:57 +05:30
Ankush Menat
764c3134cb fix: expire client cache every 10 minutes 
We can be more aggressive later, this is a good starting parting.
 2025-01-06 18:57:57 +05:30
Ankush Menat
796e51df62 fix: limit # of keys by maxsize 
Picking 1024 keys assuming 4kb avg size. In practice most things are
smaller so 1024 should be good enough!

This will likely only affect multitenant deploy with many sites.
 2025-01-06 18:57:57 +05:30
Ankush Menat
590e7ff185 fix: Ensure that RESP3 is not in use 
One can set it in conf and it won't work as expected because pubsub
format is different
 2025-01-06 18:57:57 +05:30
Ankush Menat
67848aa920 fix: handle temporary disconnects 2025-01-06 18:57:57 +05:30
Ankush Menat
e21a7112e9 fix: flushall handler 
Also no need to check channel names?
 2025-01-06 18:57:57 +05:30
Ankush Menat
b96b8c815d feat: Initate client-side caching 2025-01-06 18:57:57 +05:30
Ankush Menat
416919e880
Revert "feat: Use RESP3 for Redis cache connections (#28929)" (#28938) 
This reverts commit 6f7b4c412e.
 2024-12-27 13:19:35 +00:00
Ankush Menat
6f7b4c412e
feat: Use RESP3 for Redis cache connections (#28929) 
RESP3 has PUSH support which is useful for implementing client side
caching. Enabling this before I work on that to test if anything breaks
with this.

No need to do this for background jobs instance just yet, infrequent
accesses and performance doesn't matter as much.
 2024-12-27 06:15:59 +00:00
Ankush Menat
7dd15e3613
perf: speedup pickling of document objects (#28823) 
* perf: Use latest pickle protocol

* perf: pop flags from cached documents

This is also the right thing to do, things like `doc.flags.for_update`
shouldn't be "cached".
 2024-12-18 10:18:04 +00:00
Ankush Menat
3297aff294 fix: Always set value in local even if it expires 2024-12-17 18:34:45 +05:30
Ankush Menat
d0c314090c
perf: speed up recurring redis cache accesses (#28805) 2024-12-17 14:06:01 +05:30
Gavin D'souza
aae0cd896e
refactor: frappe.cache.set_value 
redis.set accepts ex kwarg, etc
 2024-07-12 19:55:38 +02:00
Ankush Menat
724d886f88
perf: Reduce 1 redis call while dumping monitor logs (#26337) 2024-05-06 12:53:36 +00:00
Ankush Menat
b7a5884344 perf: import sentinal locally 2024-03-15 11:31:04 +05:30
Revant Nandgaonkar
77618cde1f
feat: connect to redis sentinel for redis cache (#25398) 2024-03-14 13:54:46 +05:30
Akhil Narang
648828c3b5
fix(redis_wrapper): handle redis not being connected during cache clearing 
Ignore the redis connection errors, we can still proceed here

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
 2024-02-21 11:26:57 +05:30
Akhil Narang
5343bcb756
fix(redis_wrapper/hdel): don't pipeline if we can avoid it 
For a single deletion it doesn't make much sense, so just directly delete and return

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
 2024-02-19 14:28:35 +05:30
Akhil Narang
7c82115f5b
refactor(redis): DEL -> UNLINK 
This is beneficial since unlink is non blocking, it will run in a different thread if the data passed is large
https://redis.io/commands/unlink/

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
 2024-02-19 14:28:35 +05:30
Akhil Narang
aa38d9e2f4
refactor: pipeline wherever possible, optimize calls 
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
 2024-02-19 14:28:35 +05:30
Akhil Narang
26ae0f3460
fix: ruff fixes 
Signed-off-by: Akhil Narang <me@akhilnarang.dev>
 2024-02-07 17:04:31 +05:30
Hussain Nagaria
8d2137c265 docs: consistent doc strings 2023-12-18 18:27:39 +05:30
Ankush Menat
fa6dc03cc8
refactor: frappe.cache() usage to frappe.cache (#21282) 2023-06-08 11:47:17 +05:30
Ankush Menat
40b3cd82bc
feat!: frappe.cache() -> frappe.cache (#21279) 
This is more intuitive and consistent with other things like `frappe.db`.

PS: This is quite likely to break some weird usage which I can't guess right now. Normal usage inside request/job cycles will continue to work as it used to.
 2023-06-07 22:47:45 +05:30
Ankush Menat
26722b1a1c fix: ignore ConnectionError in frappe.cache().exists() 2023-06-03 22:53:10 +05:30
Ankush Menat
7d50ef19d3 perf: Delete multiple keys in O(1) redis calls 
Currently we call redis for each key, redis already supports deleting
multiple keys in one go.
 2023-06-03 22:53:10 +05:30
Saqib Ansari
32dbbb47bf
feat: redis cache decorator (#20452) 
* feat: redis cache decorator

* fix: review changes

* fix: remove unintentional changes

* fix: remove unintentional changes

* refactor: cleanup and simplify code for redis

AIs suck

* fix: bug

* test: redis cache

* fix: remove unused import

* feat: make redis cache user specific

redis cache utils already support this, extending so everyone can use it

* feat: support @redis_cache without params

* test: flake in request site cache test

---------

Co-authored-by: Ankush Menat <ankush@frappe.io>
 2023-03-27 17:03:20 +05:30
Sagar Vora
05c03a9345 chore!: remove cache_locally parameter 2023-01-21 17:32:24 +05:30
Daizy Modi
fce9ccedaa
fix: only execute generator if value is not found in redis cache (#18472) 
* fix: use of generator in

* fix: improve docstring

* fix: improve docstring

* fix: directly assign value to flags

Co-authored-by: Daizy <DaizyModi>
 2022-10-20 16:48:03 +05:30
Ankush Menat
1f089f44f6 feat: RedisearchWrapper 
Like rediswrapper but for redisearch to allow multitenancy.
 2022-08-31 17:04:31 +05:30
Ankush Menat
81b37cb7d2
refactor: clean up code to py310 supported features (#17367) 
refactor: clean up code to py39+ supported syntax

- f-strings instead of format
- latest typing support instead of pre 3.9 TitleCase
- remove UTF-8 declarations.
- many more changes

Powered by https://github.com/asottile/pyupgrade/ + manual cleanups
 2022-07-01 11:51:05 +05:30