TechReset

介绍docker的的过程中，提到lxc利用cgroup来提供资源的限额和控制，本文主要介绍cgroup的用法和操作命令，主要内容来自

[1]https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch01.html

[2]https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt

##cgroup

cgroup的功能在于将一台计算机上的资源(CPU,memory, network)进行分片，来防止进程间不利的资源抢占。

Terminology

由此可见，cgroup对资源的管理是一个树形结构，类似进程。

相同点 - 分层结构，子进程/cgroup继承父进程/cgroup

不同点 - 进程是一个单根树状结构(pid=0为根)，而cgroup整体来看是一个多树的森林结构(hierarchy为根)。

一个典型的hierarchy挂载目录如下

/cgroup/
├── blkio                           <--------------- hierarchy/root cgroup
│   ├── blkio.io_merged             <--------------- subsystem parameter
... ...
│   ├── blkio.weight
│   ├── blkio.weight_device
│   ├── cgroup.event_control
│   ├── cgroup.procs
│   ├── lxc                         <--------------- cgroup
│   │   ├── blkio.io_merged         <--------------- subsystem parameter
│   │   ├── blkio.io_queued
... ... ...
│   │   └── tasks                   <--------------- task list
│   ├── notify_on_release
│   ├── release_agent
│   └── tasks
...

subsystem列表

RHEL/centos支持的subsystem如下

##cgroup操作准则与方法

操作准则

1.一个hierarchy可以有多个 subsystem (mount 的时候hierarchy可以attach多个subsystem)

A single hierarchy can have one or more subsystems attached to it.

eg.

  mount -t cgroup -o cpu,cpuset,memory cpu_and_mem /cgroup/cpu_and_mem

cgroup-rule1

2.一个已经被挂载的 subsystem 只能被再次挂载在一个空的 hierarchy 上 (已经mount一个subsystem的hierarchy不能挂载一个已经被其它hierarchy挂载的subsystem)

Any single subsystem (such as cpu) cannot be attached to more than one hierarchy if one of those hierarchies has a different subsystem attached to it already.

cgroup-rule2

3.每个task只能在一同个hierarchy的唯一一个cgroup里(不能在同一个hierarchy下有超过一个cgroup的tasks里同时有这个进程的pid)

Each time a new hierarchy is created on the systems, all tasks on the system are initially members of the default cgroup of that hierarchy, which is known as the root cgroup. For any single hierarchy you create, each task on the system can be a member of exactly onecgroup in that hierarchy. A single task may be in multiple cgroups, as long as each of those cgroups is in a different hierarchy. As soon as a task becomes a member of a second cgroup in the same hierarchy, it is removed from the first cgroup in that hierarchy. At no time is a task ever in two different cgroups in the same hierarchy.

cgroup-rule3

4.子进程在被fork出时自动继承父进程所在cgroup，但是fork之后就可以按需调整到其他cgroup

Any process (task) on the system which forks itself creates a child task. A child task automatically inherits the cgroup membership of its parent but can be moved to different cgroups as needed. Once forked, the parent and child processes are completely independent.

cgroup-rule4

5.其它

操作方法

1.挂载subsystem

eg. 挂载 cpuset, cpu, cpuacct, memory 4个subsystem到/cgroup/cpu_and_mem 目录(hierarchy)

  mount {
      cpuset  = /cgroup/cpu_and_mem;
      cpu    = /cgroup/cpu_and_mem;
      cpuacct = /cgroup/cpu_and_mem;
      memory  = /cgroup/cpu_and_mem;
  }

or

  mount -t cgroup -o remount,cpu,cpuset,memory cpu_and_mem /cgroup/cpu_and_mem

2. 新建/删除 cgroup

3. 权限管理

eg.

  group daemons {
      cpuset {
          cpuset.mems = 0;
          cpuset.cpus = 0;
      }
  }
  group daemons/sql {
      perm {
          task {
              uid = root;
              gid = sqladmin;
          } admin {
              uid = root;
              gid = root;
          }
      }
      cpuset {
          cpuset.mems = 0;
          cpuset.cpus = 0;
      }
  }

or

  ~]$ mkdir -p /cgroup/red/daemons/sql
  ~]$ chown root:root /cgroup/red/daemons/sql/*
  ~]$ chown root:sqladmin /cgroup/red/daemons/sql/tasks
  ~]$ echo 0 > /cgroup/red/daemons/cpuset.mems
  ~]$ echo 0 > /cgroup/red/daemons/cpuset.cpus
  ~]$ echo 0 > /cgroup/red/daemons/sql/cpuset.mems
  ~]$ echo 0 > /cgroup/red/daemons/sql/cpuset.cpus

4. cgroup参数设定

eg.

  cgset -r cpuset.cpus=0-1 group1
  cgset --copy-from group1/ group2/
  echo 0-1 > /cgroup/cpuset/group1/cpuset.cpus

5. 添加task

eg.

  cgclassify -g cpu,memory:group1 1701 1138
  echo -e "1701\n1138" |tee -a /cgroup/cpu/group1/tasks /cgroup/memory/group1/tasks
  cgexec -g cpu:group1 lynx http://www.redhat.com
  sh -c "echo \$$ > /cgroup/lab1/group1/tasks && lynx http://www.redhat.com"

通过/etc/cgrules.conf 对特定服务限制

  maria          devices        /usergroup/staff
  maria:ftp      devices        /usergroup/staff/ftp
  @student       cpu,memory     /usergroup/student/
  %              memory         /test2/

6. 其他

##subsystem配置

###1. blkio - BLOCK IO限额

###2. cpu - CPU使用时间限额

###3. cpuacct - CPU资源报告

###4. cpuset - CPU绑定

###5. device - cgoup的device限制

###6. freezer - 暂停/恢复 cgroup的限制

###7. memory - 内存限制

###8. net_cls

###9.net_prio 指定task网络设备优先级

###10.其他

##总结

  1. 本文总结了cgroup的操作方法和详细的可配置项，为对更好的控制系统中的资源分配打下基础
  2. 对于限制资源分配的两个场景，在针对特殊APP的场景中可进行非常细致的调优，而在通用的资源隔离的角度上看，可能更关注的是CPU和内存相关的主要属性