Could you clarify the log score explanation? Wouldn't sum_{over all data} log P(Class|Data) give a negative number since all the summands are logs of probabilities? How does this translate into the small positive numbers in the paper?

Thanks.

After some experimentation : The log-score is (sum_{samples} abs(log P(Class|Data)))/#samples.

Yes, the log score is the average negative log probability of the true class:

(1/n) SUM_{i=1 to i=n} -log p(true class|example i)

The minus sign is there just to make the log score positive. Confusingly, we then need to minimize the log score, in order to be equivalent to maximizing the log likelihood.

With regard to the log score, when l2-norm regularization is added, there is an extra term in our objective function that we are minimizing (i.e. -log probability + lambda * sum weights.^2).

When we have l2 regularization should we report the actual objective function (including regularization terms) or the 'pseudo score', pretending we are not including a regularization term?

On thetraining set, I don't think it matters much whether or not you include the extra term when you compute log-score. On the test set, you should not include the extra term, so that you can compare results directly between different methods.


On Mon, 23 Apr 2007, QT - Nick wrote:

>
< replied-to message removed by QT >

regarding training, it seems to help the optimization functions to have a function value that actually reflects the gradients.

Can you explain more what you mean? Thanks, Charles



On Mon, 23 Apr 2007,
QT - Nick wrote:

>
< replied-to message removed by QT >

When using lbfgs, if the gradients (including the weight regularization term) don't match the objective function, lbfgs seems to get confused (iterations start taking much longer than normal) and eventually it quits early even though the objective function seems to still be going down. This is fixed when the objective function is commensurate with the gradients. Each iteration takes about the same amount of time, the process runs for many more iterations, and the objective function gets to the point where it goes down by tiny increments.

I'm sorry, I don't understand what you mean by "don't match" and "commensurate." The behavior you describe sounds reasonable if the gradient function is incorrect. In this case the BFGS software will be confused and behave in an undefined way. It is unfortunately very easy to write an incorrect gradient function, if you make a mistake in calculating the partial derivatives symbolically by hand, for example.

On Mon, 23 Apr 2007, QT - Nick wrote:

>
< replied-to message removed by QT >

When adding l2 regularization, the cost function is changed to include a cost in the parameters. The derivatives change as well.

For optimization packages you must supply a cost function and a derivative function.

Assuming you had no l2 cost and you wanted to add it to an existing function, if you only change the derivatives, the cost function will no longer have the gradients that you assert it does. The optimization algorithm may become confused.

Therefore you must also add the appropriate term to the cost function.

Thus, for an assignment such as this, you need two cost functions - one which includes l2 normalization and one which doesn't. The first is the thing you are actually trying to optimize. The second is the thing you are using as a point of comparison.

Hopefully this is clear and brings the discussion back to the initial question.

Nick, thanks for clarifying. Everything you wrote is correct. In other words, if you use regularization, you have to change the objective function and the gradient function. Changing just one of these functions would be a mistake.

Yes also: Generally the CLL (with or without regularization) is not exactly what we really want to optimize from an end-user perspective. For example, the "end-user" objective might be 0/1 accuracy on test data.