Corrective repair happens (immediately) when one of the components has failed. Repair takes 1 time unit, afterwards the component(s) that is/are repaired is in state 1. If 1 component is being repaired, the other one does not deteriorate

c) Allow for preventive repair of 1 component if the other has failed. Determine using value iteration the optimal policy. Give its value and plot the optimal policy. Interpret your results.

d) Now allow for preventive repair of 1 or 2 components in any state. Determine using value iteration the optimal policy. Give its value and plot the optimal policy. Interpret your results.

In this case, we only consider actions A = ${a_0, a_4, a_5, a_6}$. The optimal long-term average reward 0.5857. When both components are functioning, the optimal action is always to do nothing because that is the only available action (Fig.C). When one component is failing but the other one is still in state 1, the optimal action is corrective repair ($a_4$). Once that component enters state 2 and the other one fails, the optimal action is to preventively repair the other component ($a_5$).

In this case, the available actions are the full action set $A = {a_0, a_1, a_2, a_3, a_4, a_5, a_6}$. The optimal long-term reward is 0.8542. In this setting, it is significantly higher, because it is possible to take preventive action and reset components to state 1 more often.
Whenever a component reaches state 8-9, the optimal action is to repair preventively, which cuts costs $(r=-5 > r=-25)$ (Fig.1(b)). When both components inch closer to the fail state $((8,7),(7,8), (8,8))$, the optimal action is to preventively repair both. In the cases where one component fails, the distribution of optimal actions for the other component is different from that suggested by the optimal policy in c): current policy favours corrective repair ($a_4$) up to state 6 of the functioning component.

Assignment 2 (c & d), Dynamic Programming and Reinforcement Learning (November 2025)