This feels overengineered and it kinda is, but other efforts to
inroduce sequential naming/UUID naming haven't been that fruitful
either.
10 character random "hash" i now changed to.
1. first character - last character in UUID4 ID of request/job
2. three characters - derived from current timestamp.
4. 6 characters - random data.
This satisfies all three requirements:
1. Readers - temporal locality should result in spatial locality on disk. (fewer pages accessed)
2. Single writer - temporal locality should result in spatial locality. (fewer dirty pages)
3. Multiple writers - temporal locality should NOT result in spatial locality. (less lock contention)
Mostly concludes https://github.com/frappe/frappe/pull/25309 and https://github.com/frappe/frappe/pull/28349
Rough probabiliy numbers
Assumptions:
- Unique per worker prefix - 16 (uuid's base16 version)
- Rough time spent generating names - 10% of request (very very conservative estimate)
Probability(collision) = P(at least one prefix collision) * P(time collision)
Probability(collision) = (1 - p(all different)) * 10%
Probability(collision) = (1 - (16! / 16-N! )/ 16^N ) * 10%
| N (concurrency) | Probability(collision) |
| 1 | 0.0% |
| 2 | 0.6% |
| 3 | 1.8% |
| 4 | 3.3% |
| 5 | 5.0% |
| 6 | 6.6% |
| 7 | 7.9% |
| 8 | 8.8% |
* perf: faster `LocalProxy`
* refactor: use callable style local
* test: add some tests for local proxy override
---------
Co-authored-by: Ankush Menat <ankush@frappe.io>
1. It's severly outdated and free version isn't that accurate to begin
with.
2. I replaced country detection using timezone a long time ago: c8ec528aa7/frappe/desk/page/setup_wizard/setup_wizard.js (L664-L676)
This is good enough for setup wizard for most popular countries. Use
IP-API if you need better guarantees (we have a license).
Even though we never use tz aware object in frappe this function
previously used to retain them, so keep TZinfo for sake of compat.
Nothing should be affected ideally.
* perf: Reduce penalty for lack of redis connection
If redis isn't running than this client cache is slower than default
implementation because of the extra locking overhead.
* test: update perf redis counts
* perf: cache table columns in client-cache
* fix: race condition on cache-client_cache init
Rare but apparant in synthetic benchmarks.
Cache is set but client cache is still being initialized then request
will fail.
* perf: Don't run notifications when loading document
WHAT?
* fix: use cached doc to repopulate
* perf: reduce get_meta calls
If I have to hazard a guess, 99% API calls are not server scripts, then
why check it first and pay the costs?
This PR first checks if method is a real method in python code and if
it's not found then only attempts to fetch it from server script map.
I'll revert this if I can bring the costs in acceptable limits with
client-side caching.
Currently one test runner takes significantly longer than another. This
is entirely due to test_commands.py which needs to create new site and
do backup/restore tests etc. All of which are far far slower than other
tests.
* perf: No need to set expiry for key everytime
* fix: Set expiry on first request and never again
This prevents problem of rate limiter keys growing constantly.
Call it a new year's resolution.
`frappe/__init__.py` has grown crazy big over time and left unattended it will continue to grow.
This new ~test~ tax will require reducing 3 line per day (so ~1000 in a year) from 1st Jan 2025 onwards. I am offering a headstart of 50 days in this PR by moving ~150 lines: #28869
This is middle ground between caching it completely and requiring a
restart/signal to reload vs always reloading it.
I don't know any use cases that can break from this, nowhere in code
configs should be expected to reload instantly.
This change is only applied to requests for now
* perf: reuse current time
now_datetime is site-tz-aware, we don't need it here.
* perf: dont need redis transactions
* perf: use `time.time()` instead of datetime
Using `datetime.timestamp()` is a round-about way to use `time.time()`
with extra cost of dealing with datetime and timezones.
* perf: define slots for rate_limiter
* fix!: Remove used rate limit header
This just shares how much was consumed in current request, people can
just time requests to get an approximation for this, not sure why is this
useful.
* chore: remove verbose output from test runner
This is same output that's shared by test runner in different format?
This makes it annoying to scroll through when just running single test
locally.
* fix: Remove clutter from test output
Test records don't change after first run.
Tests are executed many many times locally
* test: retry flaky postgres backup tests
