I am not clear on 'the most valuable contribution' of this paper (i.e. using state-observation transition functions rather than the separate transition and observation functions in HMMs). To put this in a question, why did authors use maximum entropy framework? Can HMMs not implement similar state-observation transition functions as in MEMM?

Greg:
I wasn't clear on what you mean by 'vocabulary-based, free-structure environment.' I am going to assume that a news article satisfies these two requirements since it certainly consists of vocabularies and no structures such as 'question-answer' pairs exist. Consider the problem of extracting the purchasing price from each dodument in a collection of articles describing corporate acquisitions. As was done using HMM in one of the references mentioned in the paper, I think MEMM can be trained with features such as 'set of words indicating an action of buying/acquisition is present' and 'words suggesting company names are present' to solve the problem. Relevent states might be S1 = state that produced tokens such as purchase values of the acquisitions with company names and S2 = state that produced irrelevant words such as 'that, is,...'. I am saying these with no experience of actually applying HMMs and MEMMs to real problems so I welcome anyone spotting holes in my reasonings.