Skip to main content
Version: Preview

Simulate pod faults

Pod faults support pod failure, pod kill, and container kill.

  • Pod failure: injects fault into a specified Pod to make the Pod unavailable for a while.
  • Pod kill: kills a specified Pod. To ensure that the Pod can be successfully restarted, you need to configure ReplicaSet or similar mechanisms.
  • Container kill: kills the specified container in the target Pod.

Usage restrictions

Chaos Mesh can inject PodChaos into any Pod, no matter whether the Pod is bound to Deployment, StatefulSet, DaemonSet, or other controllers. However, when you inject PodChaos into an independent Pod, some different situations might occur. For example, when you inject pod-kill chaos into an independent Pod, Chaos Mesh cannot guarantee that the application recovers from its failure.

Before you start

  • Make sure there is no Control Manager of Chaos Mesh running on the target Pod.
  • If the fault type is pod-kill, ReplicaSet or a similar mechanism is configured to ensure that Pod can restart automatically.

Simulate fault injections by kbcli

Common flags for all types of Pod faults.

📎 Table 1. Pod faults flags description

OptionDescriptionDefault valueRequired
pod nameSpecify the name of the Pod to inject the fault. For example, add the Pod name mysql-cluster-mysql-0 to the command, and the complete command would be kubectl fault pod kill mysql-cluster-mysql-0.DefaultNo
--namespaceIt specifies the namespace where the Chaos is created.Current namespaceNo
--ns-faultIt specifies a namespace to make all Pods in this namespace unavailable. For example,
kbcli fault pod kill --ns-fault=kb-system
DefaultNo
--nodeIt specifies a node to make all Pods on this node unavailable. For example,
kbcli fault pod kill --node=minikube-m02
NoneNo
--labelIt specifies a label to make the Pod with this label in the default namespace unavailable. For example,
kbcli fault pod kill --label=app.kubernetes.io/component=mysql
NoneNo
--node-labelIt specifies a node label to make all Pods on the node with this node label unavailable. For example,
kbcli fault pod kill --node-label=kubernetes.io/arch=arm64
NoneNo
--modeIt specifies the mode of the experiment. The mode options include one (selecting a random Pod), all (selecting all eligible Pods), fixed (selecting a specified number of eligible Pods), fixed-percent (selecting a specified percentage of Pods from the eligible Pods), and random-max-percent (selecting the maximum percentage of Pods from the eligible Pods).allNo
--valueIt provides parameters for the mode configuration, depending on mode. For example, when mode is set to fixed-percent, --value specifies the percentage of Pods. For example,
kbcli fault pod kill --mode=fixed-percent --value=50
NoneNo
--durationIt specifies how long the experiment lasts.10 secondsNo

Pod kill

Run the command below to inject pod-kill into all Pods in the default namespace and make the Pods unavailable for 30 seconds.

kbcli fault pod kill

Pod failure

Run the command below to inject pod-failure into all Pods in the default namespace and make the Pods unavailable for 10 seconds.

kbcli fault pod failure --duration=10s

Container kill

Run the command below to inject container-kill into the containers of all Pods in the default namespace once and make the containers unavailable for 10 seconds. --container is required.

kbcli fault pod kill-container --container=mysql

You can also add multiple containers. For example, run the command below to kill the mysql and config-manager containers in the default namespace.

kbcli fault pod kill-container --container=mysql --container=config-manager

Simulate fault injections by YAML file

This section introduces the YAML configuration file examples. You can view the YAML file by adding --dry-run at the end of the above kbcli commands. Meanwhile, you can also refer to the Chaos Mesh official docs for details.

Pod-kill example

  1. Write the experiment configuration to the pod-kill.yaml file.

    In the following example, Chaos Mesh injects pod-kill into the specified Pod and kills the Pod once.

    apiVersion: chaos-mesh.org/v1alpha1
    kind: PodChaos
    metadata:
    creationTimestamp: null
    generateName: pod-chaos-
    namespace: default
    spec:
    action: pod-kill
    duration: 10s
    mode: fixed-percent
    selector:
    namespaces:
    - default
    labelSelectors:
    'app.kubernetes.io/component': 'mysql'
    value: "50"
  2. Run kubectl to start an experiment.

    kubectl apply -f ./pod-kill.yaml

Pod-failure example

  1. Write the experiment configuration to the pod-failure.yaml file.

    In the following example, Chaos Mesh injects pod-failure into the specified Pod and kills the Pod once.

    apiVersion: chaos-mesh.org/v1alpha1
    kind: PodChaos
    metadata:
    creationTimestamp: null
    generateName: pod-chaos-
    namespace: default
    spec:
    action: pod-failure
    duration: 30s
    mode: fixed-percent
    selector:
    namespaces:
    - default
    labelSelectors:
    'app.kubernetes.io/component': 'mysql'
    value: "50"
  2. Run kubectl to start an experiment.

    kubectl apply -f ./pod-failure.yaml

Container-kill example

  1. Write the experiment configuration to the container-kill.yaml file.

    In the following example, Chaos Mesh injects container-kill into the specified Pod and kills the Pod once.

    apiVersion: chaos-mesh.org/v1alpha1
    kind: PodChaos
    metadata:
    creationTimestamp: null
    generateName: pod-chaos-
    namespace: default
    spec:
    action: container-kill
    duration: 10s
    mode: fixed-percent
    selector:
    namespaces:
    - default
    labelSelectors:
    'app.kubernetes.io/component': 'mysql'
    value: "50"
  2. Run kubectl to start an experiment.

    kubectl apply -f ./container-kill.yaml

Field description

This table describes the fields in the YAML file.

ParameterTypeDescriptionDefault valueRequiredExample
actionstringIt specifies the fault type to inject. The supported types include pod-failure, pod-kill, and container-kill.NoneYespod-kill
durationstringIt specifies the duration of the experiment.NoneYes10s
modestringIt specifies the mode of the experiment. The mode options include one (selecting a random Pod), all (selecting all eligible Pods), fixed (selecting a specified number of eligible Pods), fixed-percent (selecting a specified percentage of Pods from the eligible Pods), and random-max-percent (selecting the maximum percentage of Pods from the eligible Pods).NoneYesfixed-percent
valuestringIt provides parameters for the mode configuration, depending on mode. For example, when mode is set to fixed-percent, value specifies the percentage of Pods.NoneNo50
selectorstructIt specifies the target Pod by defining node and labels.NoneYes.
If not specified, the system kills all pods under the default namespece.
containerNamesstringWhen you configure action to container-kill, this configuration is mandatory to specify the target container name for injecting faults.NoneNomysql
gracePeriodint64When you configure action to pod-kill, this configuration is mandatory to specify the duration before deleting Pod.0No0
durationstringIt specifies the duration of the experiment.NoneYes30s