Here you can find the exercise for Lecture 5: Markov Decision Processes. After the live session, also the solution can be found here.