DRA: device taints and tolerations

Initial documentation of this new feature in Kubernetes 1.33.

Author: pohly

content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md

@@ -71,9 +71,13 @@ DeviceClass
   in a ResourceClaim must reference exactly one DeviceClass.
 
 ResourceSlice
-: Used by DRA drivers to publish information about resources
+: Used by DRA drivers to publish information about resources (typically devices)
   that are available in the cluster.
 
+DeviceTaintRule
+: Used by admins or control plane components to add device taints
+  to the devices described in ResourceSlices.
+
 All parameters that select devices are defined in the ResourceClaim and
 DeviceClass with in-tree types. Configuration parameters can be embedded there.
 Which configuration parameters are valid depends on the DRA driver -- Kubernetes
@@ -357,6 +361,95 @@ spec:
           value: 6Gi
 ```
 
+## Device taints and tolerations
+
+{{< feature-state feature_gate_name="DRADeviceTaints" >}}
+
+Device taints are similar to node taints: a taint has a string key, a string
+value, and an effect. The effect is applied to the ResourceClaim which is
+using a tainted device and to all Pods referencing that ResourceClaim.
+The "NoSchedule" effect prevents scheduling those Pods.
+Tainted devices are ignored when trying to allocate a ResourceClaim
+because using them would prevent scheduling of Pods.
+
+The "NoExecute" effect implies "NoSchedule" and in addition causes eviction
+of all Pods which have been scheduled already. This eviction is implemented
+in the device taint eviction controller in kube-controller-manager by
+deleting affected Pods.
+
+ResourceClaims can tolerate taints. If a taint is tolerated, its effect does
+not apply. An empty toleration matches all taints. A toleration can be limited to
+certain effects and/or match certain key/value pairs. A toleration can check
+that a certain key exists, regardless which value it has, or it can check
+for specific values of a key.
+For more information on this matching see the
+[node taint concepts](/docs/concepts/scheduling-eviction/taint-and-toleration#concepts).
+
+Eviction can be delayed by tolerating a taint for a certain duration.
+That delay starts at the time when a taint gets added to a device, which is recorded in a field
+of the taint.
+
+Taints apply as described above also to ResourceClaims allocating "all" devices on a node.
+All devices must be untainted or all of their taints must be tolerated.
+Allocating a device with admin access (described [above](#admin-access))
+is not exempt either. An admin using that mode must explicitly tolerate all taints
+to access tainted devices.
+
+Taints can be added to devices in two different ways:
+
+### Taints set by the driver
+
+A DRA driver can add taints to the device information that it publishes in ResourceSlices.
+Consult the documentation of a DRA driver to learn whether the driver uses taints and what
+their keys and values are.
+
+### Taints set by an admin
+
+An admin or a control plane component can taint devices without having to tell
+the DRA driver to include taints in its device information in ResourceSlices. They do that by
+creating DeviceTaintRules. Each DeviceTaintRule adds one taint to devices which
+match the device selector. Without such a selector, no devices are tainted. This
+makes it harder to accidentally evict all pods using ResourceClaims when leaving out
+the selector by mistake.
+
+Devices can be selected by giving the name of a DeviceClass, driver, pool,
+and/or device. The DeviceClass selects all devices that are selected by the
+selectors in that DeviceClass. With just the driver name, an admin can taint
+all devices managed by that driver, for example while doing some kind of
+maintenance of that driver across the entire cluster. Adding a pool name can
+limit the taint to a single node, if the driver manages node-local devices.
+
+Finally, adding the device name can select one specific device. The device name
+and pool name can also be used alone, if desired. For example, drivers for node-local
+devices are encouraged to use the node name as their pool name. Then tainting with
+that pool name automatically taints all devices on a node.
+
+Drivers might use stable names like "gpu-0" that hide which specific device is
+currently assigned to that name. To support tainting a specific hardware
+instance, CEL selectors can be used in a DeviceTaintRule to match a vendor-specific
+unique ID attribute, if the driver supports one for its hardware.
+
+The taint applies as long as the DeviceTaintRule exists. It can be modified and
+and removed at any time. Here is one example of a DeviceTaintRule for a fictional
+DRA driver:
+
+```yaml
+apiVersion: resource.k8s.io/v1alpha3
+kind: DeviceTaintRule
+metadata:
+  name: example
+spec:
+  # The entire hardware installation for this
+  # particular driver is broken.
+  # Evict all pods and don't schedule new ones.
+  deviceSelector:
+    driver: dra.example.com
+  taint:
+    key: dra.example.com/unhealthy
+    value: Broken
+    effect: NoExecute
+```
+
 ## Enabling dynamic resource allocation
 
 Dynamic resource allocation is a *beta feature* which is off by default and only enabled when the
@@ -426,6 +519,13 @@ and only enabled when the `DRAPartitionableDevices`
 [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
 is enabled in the kube-apiserver and kube-scheduler.
 
+### Enabling device taints and tolerations
+
+[Device taints and tolerations](#device-taints-and-tolerations) is an *alpha feature* and only enabled when the
+`DRADeviceTaints` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+is enabled in the kube-apiserver, kube-controller-manager and kube-scheduler. To use DeviceTaintRules, the
+`resource.k8s.io/v1alpha3` API version must be enabled.
+
 ## {{% heading "whatsnext" %}}
 
 - For more information on the design, see the

content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md

@@ -322,9 +322,18 @@ tolerations to all daemons, to prevent DaemonSets from breaking.
 Adding these tolerations ensures backward compatibility. You can also add
 arbitrary tolerations to DaemonSets.
 
+## Device taints and tolerations
+
+Instead of tainting entire nodes, administrators can also [taint individual devices](/docs/concepts/scheduling-eviction/dynamic-resource-allocation#device-taints-and-tolerations)
+when the cluster uses [dynamic resource allocation](/docs/concepts/scheduling-eviction/dynamic-resource-allocation)
+to manage special hardware. The advantage is that tainting can be targeted towards exactly the hardware that
+is faulty or needs maintenance. Tolerations are also supported and can be specified when requesting
+devices. Like taints they apply to all pods which share the same allocated device.
+
 ## {{% heading "whatsnext" %}}
 
 * Read about [Node-pressure Eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/)
   and how you can configure it
 * Read about [Pod Priority](/docs/concepts/scheduling-eviction/pod-priority-preemption/)
+* Read about [device taints and tolerations](/docs/concepts/scheduling-eviction/dynamic-resource-allocation#device-taints-and-tolerations)
 

content/en/docs/reference/command-line-tools-reference/feature-gates/DRADeviceTaints.md

@@ -0,0 +1,17 @@
+---
+title: DRADeviceTaints
+content_type: feature_gate
+_build:
+  list: never
+  render: false
+
+stages:
+  - stage: alpha
+    defaultValue: false
+    fromVersion: "1.33"
+---
+Enables support for
+[tainting devices and selectively tolerating those taints](/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#device-taints-and-tolerations)
+when using dynamic resource allocation to manage devices.
+
+This feature gate has no effect unless you also enable the `DynamicResourceAllocation` feature gate.