d466578348
Python's multithreaded model is _inefficient_ because of Global
Interpreter Lock (GIL). Any one thread of process can run at any given
time. Thus only valid use case for threads in Python are:
1. Hiding I/O latency by switching to a different thread.
2. Using compiled extensions that yield GIL for long enough time to do
meaningful work in other threads.
Both of these are not as frequent as you'd imagine and gthread worker
with multiple threads often just end up contending on lock and waste
useful CPU cycles doing nothing. Pinning worker process to a core nearly
eliminates this contention wastage. This waste can be 5-10% and goes up
sharply with more threads.
E.g. FC typically has maxed out config of 24 workers which allows
"accepting" and working on 24 requests at a time. But that doesn't mean
24 requests are on CPU at any given time, that would require 24 physical
cores.
Why do this?
1. Context switching in threads is faster than switching process - fewer
cache misses, fewer TLB misses etc.
2. The model is simple
True parallelism = count(cores) = count(processes).
Expected concurrency = count(processes) * count(threads).
3. This is far simpler to reason about than something like async
executor model.
4. Ability to queue more requests than what can be handled is already
implemented by `bind(2)` and `accept(2)` in kernel. There is no real
benefit of accepting 1000 requests if you can only work on 20 of them
at a time. This is because we do a lot of "work" in requests, it's
not just issuing an external request and waiting for it.
5. We can achieve practically same concurrency as 24 workers with 4
process x 6 threads. That's a lot of memory saved to run other useful
things.
Caveats:
- This kind of pinning can potentially make Linux scheduler inefficient.
I don't quite think it's going to be a big problem because there are
plenty of other things to run which a core can steal from other core
if it doesn't have enough work.
- Load balancing in single-server multi-bench setup. I *think* by nature
of how `accept(2)` works, load balancing will still happen pretty much
automatically. If certain core is overloaded, naturally other cores
will reach `accept(2)` more frequently and take the load off of that
core. This is something worth validating in practice by creating
skewed affinities.
- This code is not NUMA-aware. None of our machines have NUMA nodes so,
I am ignoring it. Don't use it if you have a NUMA setup.
- If new CPUs are hotplugged or existing ones are disabled then it can
be inefficient (worse than current) until that worker auto-restarts (which
happens after N requests in FC setup).
Ideal solution: We write userspace scheduler to implement
"soft-affinity" using Linux's new eBPF based sched_ext feature. That's
too much extra work but I'll consider this too at some point.
closes https://github.com/frappe/caffeine/issues/13
|
||
|---|---|---|
| .github | ||
| billing | ||
| cypress | ||
| esbuild | ||
| frappe | ||
| realtime | ||
| tests | ||
| .coveragerc | ||
| .editorconfig | ||
| .eslintignore | ||
| .eslintrc | ||
| .git-blame-ignore-revs | ||
| .gitignore | ||
| .mergify.yml | ||
| .pre-commit-config.yaml | ||
| .releaserc | ||
| .semgrepignore | ||
| attributions.md | ||
| babel_extractors.csv | ||
| CODE_OF_CONDUCT.md | ||
| codecov.yml | ||
| CODEOWNERS | ||
| commitlint.config.js | ||
| crowdin.yml | ||
| cypress.config.js | ||
| generate_bootstrap_theme.js | ||
| hooks.md | ||
| LICENSE | ||
| node_utils.js | ||
| package.json | ||
| pyproject.toml | ||
| README.md | ||
| SECURITY.md | ||
| sider.yml | ||
| socketio.js | ||
| yarn.lock | ||
Frappe Framework
Low Code Web Framework For Real World Applications, In Python And JavaScript
Frappe Framework
Full-stack web application framework that uses Python and MariaDB on the server side and a tightly integrated client side library. Built for ERPNext.
Motivation
Started in 2005, Frappe Framework was inspired by the Semantic Web. The "big idea" behind semantic web was of a framework that not only described how information is shown (like headings, body etc), but also what it means, like name, address etc.
By creating a web framework that allowed for easy definition of metadata, it made building complex applications easy. Applications usually designed around how users interact with a system, but not based on semantics of the underlying system. Applications built on semantics end up being much more consistent and extensible. The first application built on Framework was ERPNext, a beast with more than 700 object types. Framework is not for the light hearted - it is not the first thing you might want to learn if you are beginning to learn web programming, but if you are ready to do real work, then Framework is the right tool for the job.
Key Features
-
Full-Stack Framework: Frappe covers both front-end and back-end development, allowing developers to build complete applications using a single framework.
-
Built-in Admin Interface: Provides a pre-built, customizable admin dashboard for managing application data, reducing development time and effort.
-
Role-Based Permissions: Comprehensive user and role management system to control access and permissions within the application.
-
REST API: Automatically generated RESTful API for all models, enabling easy integration with other systems and services.
-
Customizable Forms and Views: Flexible form and view customization using server-side scripting and client-side JavaScript.
-
Report Builder: Powerful reporting tool that allows users to create custom reports without writing any code.
Screenshots
Production Setup
Managed Hosting
You can try Frappe Cloud, a simple, user-friendly and sophisticated open-source platform to host Frappe applications with peace of mind.
It takes care of installation, setup, upgrades, monitoring, maintenance and support of your Frappe deployments. It is a fully featured developer platform with an ability to manage and control multiple Frappe deployments.
Self Hosting
Docker
Prerequisites: docker, docker-compose, git. Refer Docker Documentation for more details on Docker setup.
Run following commands:
git clone https://github.com/frappe/frappe_docker
cd frappe_docker
docker compose -f pwd.yml up -d
After a couple of minutes, site should be accessible on your localhost port: 8080. Use below default login credentials to access the site.
- Username: Administrator
- Password: admin
See Frappe Docker for ARM based docker setup.
Development Setup
Manual Install
The Easy Way: our install script for bench will install all dependencies (e.g. MariaDB). See https://github.com/frappe/bench for more details.
New passwords will be created for the Frappe "Administrator" user, the MariaDB root user, and the frappe user (the script displays the passwords and saves them to ~/frappe_passwords.txt).
Local
To setup the repository locally follow the steps mentioned below:
-
Setup bench by following the Installation Steps and start the server
bench start -
In a separate terminal window, run the following commands:
# Create a new site bench new-site frappe.dev # Map your site to localhost bench --site frappe.dev add-to-hosts -
Open the URL
http://frappe.dev:8000/appin your browser, you should see the app running
Learning and community
- Frappe School - Learn Frappe Framework and ERPNext from the various courses by the maintainers or from the community.
- Official documentation - Extensive documentation for Frappe Framework.
- Discussion Forum - Engage with community of Frappe Framework users and service providers.
- buildwithhussain.dev - Watch Frappe Framework being used in the wild to build world-class web apps.
Contributing
