User cgroups: cpu/memory/io per-User resource control (SKINS)(LANG)

Version 1.62.0

Feature
Finished

BETA This feature is part of the DirectAdmin Pro Pack: https://docs.directadmin.com/getting-started/pro-pack/overview This new feature relies on "cgroups v2", which is a kernel option. Not, to set a limit to "unlimited", pass a limit with blank value Setting a limit to be blank tells the system it should be unlimited.. ====================== ENABLE To use this feature, you must have the "Pro Pack" enabled in your license. See the full install guide here: https://docs.directadmin.com/other-hosting-services/pro-resource-throttling/general/#installation-instructions ---- Debian 11 will have it enabled in the kernel by default. CentOS 8, Debian 10, and older, will need to have it turned on in the kernel (requires reboot) cd /usr/local/directadmin/custombuild ./build update ./build grub_conf ./build apache ./build php n Anything older (and FreeBSD) do not support cgroups. DirectAdmin will hide or ignore values if the system has not created: /sys/fs/cgroup/user.slice ====================== JSON ---------------------- PACKAGES ----- Show package: CMD_SHOW_USER_PACKAGE?package=gold&json=yes CMD_SHOW_RESELLER_PACKAGE?package=gold&json=yes New array, eg: "cgroup": { "options": { "CPUQuota": { "cgroup_type": "percent", "default": "", "desc": "Percent of CPU Core. >100% for more cores.", "name": "CPUQuota", "placeholder": "400%", "string": "CPU Quota", "type": "text" }, "IOReadBandwidthMax": { "cgroup_type": "io", "default": "", "desc": "Max rate data can be read from a disk.", "minimum": "256K", "name": "IOReadBandwidthMax", "placeholder": "5M", "string": "IO Read Bandwidth Max", "type": "text" }, "IOReadIOPSMax": { "cgroup_type": "iops", "default": "", "desc": "Max disk read operations per second.", "minimum": "256", "name": "IOReadIOPSMax", "placeholder": "1K", "string": "IOPS Read Max", "type": "text" }, "IOWriteBandwidthMax": { "cgroup_type": "io", "default": "", "desc": "Max rate data can be written to a disk.", "minimum": "256K", "name": "IOWriteBandwidthMax", "placeholder": "5M", "string": "IO Write Bandwidth Max", "type": "text" }, "IOWriteIOPSMax": { "cgroup_type": "iops", "default": "", "desc": "Max disk write operations per second.", "minimum": "256", "name": "IOWriteIOPSMax", "placeholder": "1K", "string": "IOPS Write Max", "type": "text" }, "MemoryHigh": { "cgroup_type": "bytes", "default": "", "desc": "Throttle Memory above this limit.", "name": "MemoryHigh", "placeholder": "1G", "string": "Memory High", "type": "text" }, "TasksMax": { "cgroup_type": "int", "default": "", "desc": "Max Tasks that may be created in the unit", "minimum": "5", "name": "TasksMax", "placeholder": "4915", "string": "Tasks Max", "type": "text" } }, "saved": { "CPUQuota": { "cgroup_type": "percent", "default": "", "desc": "Percent of CPU Core. >100% for more cores.", "name": "CPUQuota", "placeholder": "400%", "string": "CPU Quota", "type": "text", "value": "400%" } } }, Where "options" are what can be used, with info about the item.. and "saved" is which of those items actually exist in the package, including the "value". Items that have a "minimum" are enforced at the back-end and blocks the save entirely (Creation/Modification) ---- OPTION VARIABLES Each group option will have items: cgroup_type: see below. The specific format for a given option. default: the default value, usually blank (unlimited) desc: longer description, translated in the internal gettext name: the form name to pass, matches the system's cgroup option name. placeholder: for empty inputs, an example/background text string: the human readable name (shorter), translated in the internal gettext type:text form field type (eg: future might have checkbox, for example, no others for now) value: Only for saved values in the package itself. The "options" vs "saved" does have mostly redundant info. Essentially, just the "value" will exist for the "saved" array. ----- Package Defaults: CMD_SHOW_USER_PACKAGE?package=unlimited CMD_SHOW_RESELLER_PACKAGE?package=unlimited Same as listing a specific package, but the "saved" array will not be passed, since it's just the default list. ----- Save Package CMD_MANAGE_USER_PACKAGES CMD_MANAGE_RESELLER_PACKAGES method: POST plus all new values. To set an option to be "unlimited", it must be set, but with a blank value. Eg: CPUQuota= will removing any cpu limits. As before, all values must be sent when saving a package. The requirement to save these options is not required, but saving them once, then not passing them later, will not delete them from User settings. cgroup settings are not "deleted" from the system, the must be saved to unlimited (blank value) to remove the limits. ------ CGROUP TYPES DA uses a variable called: "cgroup_type", as defined: percent: 1% - 10000% int: 1 - 1000000 bytes: 1, 2K, 5M, 2G, 3T io: same as "bytes", but internally DA adds the list of devices to the value. Packages/Users only set the number, not the device, eg: IOReadBandwidthMax=5M iops: same as "io" ====================== USERS Create/Modify Users/Resellers use the same formatting as packages. Pass the values you'd like to set during: - Custom User Creation (CMD_ACCOUNT_USER, without setting package) - User Modification (CMD_MODIFY_USER/CMD_MODIFY_RESELLER) - Show User: CMD_SHOW_USER?user=fred&json=yes ====================== RESELLER When setting CGroup values for a Reseller, they're saved into that Resller's reseller.conf file. The values set here are optional. If they are blank, the Reseller is free to set (or not set) values for their Users as desired. If they are set, then the value set is the maximum any of the Reseller's Users can have. For example, if the reseller.conf has: CPUQuota=400% This implies that any User under this Reseller *must* have a CPUQuota set (cannot be blank) and it *must* not exceed 400%. All Users under the Reseller can have at most CPUQuota=400%. This is different than the other cumulative values, like bandwidth or disk usage, where the Users are added together. This is not like that.. the values are the max for each User. ====================== STATS ==== Admin Level ---- Show Live Slice info: CMD_ADMIN_LIMITS?json=yes CMD_ADMIN_LIMITS?json=yes&user=fred Include live "rates" (values over time) in the above output, eg: CMD_ADMIN_LIMITS?json=yes&live_rates=yes CMD_ADMIN_LIMITS?json=yes&live_rates=yes&user=fred ---- Show Logged Events Cache: This will show the total count of each area, and the date this area was later triggered: CMD_ADMIN_LIMITS?json=yes&show=events_cache CMD_ADMIN_LIMITS?json=yes&show=events_cache&user=fred where the "date" (timestamp) will be the last time an event was caught, and the value is the current highest value. If User-Slice resets to zero, this count will continue to rise. (TODO: sort out when to reset this to zero, likely monthly reset) ---- Show Events: CMD_ADMIN_LIMITS?json=yes&show=events&user=fred Default values of nothing passed: range_start=1d range_end=0s where these values can be passed to control which logs are returned. The range_*= values are: - if value ends in a time unit: s,m,h,d,M,y, will be that amount of time before now. - If there are no units, the value must be a timestamp for the desired time. The output contains an array "events", with a list of sub-arrays. Each sub-arrays' have an index of the timestamp the event was logged. The contents of the sub-array will be which category and event variable was increased, only showing the change since the last check. eg; { "events": { "1612253281": { "memory.events": { "high": "20" }, "pids.events": { "recent": "19" }, "processes": { "cmd": "php-fpm74", "cpu": "0.0", "memory": "4884", "pid": "157784", "started": "21:13:43", "time": "00:00:00" } } } } where there were 20 new 'high" memory events, and 19 new process limit events since the last time the dataskq checked the count. The file that stores the "last check" will be at: /usr/local/directadmin/data/admin/cgroup_cache/fred/user.slice.json where the diff from user.slice.json vs the /sys/fs/cgroup/user.slice/user-1007.slice* are used for the triggers. Other events[12345] values: "cpu.stat": { "nr_periods": "55", "nr_throttled": "42", "throttled_percentage": "76.363640" }, and: "io.stat": { "dbytes": "55", "253:0": { "dbytes" : "55" } }, where - dbytes could also be dios for discarded IOPs. - io.stats["dbytes"] will be a sum of all devices that trigged this check (not likely to have multiple, but supported - "253:0" is the name of the device (the io.max is used as the list to check) - io.stats["253:0"]['dbytes"] will be the discarded bytes for this device. Currently a "rate" does not trigger an event, only dropped bytes/ios, as the rate can be viewed at any time. Will consider a "input_bps_percentage" or "output_bps_percentage" if needed, eg: if (((last input_bps / max input_bps) + (this input_bps / max input_bps)) / 2) > threshold percentage) trigger event. but this is likely to trigger events too often if the website is I/O intensive, hence not implemented. ====================== RESELLER / USER LEVELS The calls to the information by Resellers is the same as for Admins, with the difference being the call name, and that instead of "All Users", it's only the list from their users.list file. CMD_RESELLER_LIMITS?json=yes CMD_RESELLER_LIMITS?json=yes&show=events&user=fred CMD_RESELLER_LIMITS?json=yes&show=events_cache with &user=fred being optional for all, except show=events, which must always have a user passed. Users, again, can make the same calls, except all cases are as if user=fred was passed (this is done internally) CMD_USER_LIMITS?json=yes CMD_USER_LIMITS?json=yes&show=events CMD_USER_LIMITS?json=yes&show=events_cache ====================== EVENTS CACHE For all Levels, the show=events_cache will show the count for the listed Users from the file: /usr/local/directadmin/data/admin/cgroup_cache/fred/events.cache which holds the count for all type events for this Users. It's deleted with the monthly reset. By default the internal "newer_than" variable is set to 1 day ago, such that only file mtime timestamps newer than 24 hours are loaded. To include older (or only newer) event.cache files, you can specify either: newer_than=1612736565 #exact timestamp to be the limit/threshold or a unit value to be subtracted from now: newer_than=2d #all newer than 2 days ago, allowing all units. Absence of a unit is not seconds, it's the timestamp. Sample: { "users": { "fred": { "cpu.stat": { "nr_periods": { "new": "594", "timestamp": "1613085301", "timestamp_diff": "59", "total": "4412" }, "nr_throttled": { "new": "594", "timestamp": "1613085301", "timestamp_diff": "59", "total": "4403" }, "throttled_percentage": "100.000000" }, "memory.events": { "high": { "new": "16", "timestamp": "1613085361", "timestamp_diff": "119", "total": "370" } } } } } } ====================== SKINS # modified: data/skins/enhanced/files_reseller.conf HTM_CGROUP=reseller/cgroup.html # new file: data/skins/enhanced/reseller/cgroup.html See file for table data/tokens # modified: data/skins/enhanced/reseller/create_customized_user.html # modified: data/skins/enhanced/reseller/modify_user.html # modified: data/skins/enhanced/reseller/show_user_package.html # modified: data/skins/enhanced/admin/create_customized_reseller.html # modified: data/skins/enhanced/admin/modify_reseller.html # modified: data/skins/enhanced/admin/show_reseller_package.html 6 files add: |HTM_CGROUP| Before CUSTOM_ITEM_1 tokens line. ====================== GLOBAL TOKENS "HAVE_CGROUP": "yes", "CGROUP_CPUQuota": "25%", "CGROUP_IOReadBandwidthMax": "1M", "CGROUP_IOReadIOPSMax": "256", "CGROUP_IOWriteBandwidthMax": "1M", "CGROUP_IOWriteIOPSMax": "", "CGROUP_MemoryHigh": "128M", "CGROUP_TasksMax": "5", If any unset or blank value, they'll always be set here, but blank value of "" ====================== LANG # modified: data/skins/enhanced/lang/en/admin/create_reseller.html LANG_CGROUPS_RESOURCE_LIMITS=Linux Control Groups - Resource Limits LANG_BLANK_FOR_UNLIMITED=Blank value for unlimited LANG_CUSTOM_PACKAGE_ITEMS=Custom Package Items ----- EVO2038

Interested to try DirectAdmin? Get a 30-day Free Trial!