(in response to message #15)
My understanding of the role of potential function in Q-function is lacking in some parts. However, for the sake of class participation and also of letting others know of my concerns I would like to state what I think I know.

I also think that using function of the form
F(s,a,s')=phi(s')-phi(s) to make one circular transition from state s1,s2,..sn, and finally to itself will result in a total F of zero. However when F() is used in the optimal Q-function, presence of the gamma factor makes value of F() time dependent.