I liked the main idea of this paper -- that the virtual reinforcement is not helpful is somewhat counterintuitive, yet very simple to see once the asymmetry of the reinforcement is explained. I am wondering how well this applies to other problems, though. The asymmetry in the information agent A receives stems from the asymetry of the problem -- one action gains A some amount while another action gains A nothing. Of course, the asymmetry would still be present if A gains a larger amount from one action and a smaller amount from another, if the same sort of reinforcement is used. However, I think that even in a problem as simple as this, there is more reasoning that could be performed than that assumed. For example, if one is assuming an adaptive agent B, then agent A could analyze how his actions would train B. The asymmetry stems ultimately from having just one measure of the goodness of an action, and perhaps this can be avoided in many problems.

I think this is a good example of what Professor Elkan was mentioning on Monday, about how humans actually make poorer decisions than machines in some cases.