ITs, a structure sensitive information theory

Abstract

Broadly speaking Information theory (IT) assumes no structure of the underlying states. But what about contexts where states do have a clear structure - how should IT cope with such situations? And if such coping is at all possible then - how should structure be expressed so that it can be coped with? A possible answer to these questions is presented here. Noting that IT can cope well with a structure expressed as an accurate clustering (by shifting to the implied reduced alphabet), a generalization is suggested in which structure is expressed as a measure on reduced alphabets. Given such structure an extension of IT is presented where the reduced alphabets are treated simultaneously. This structure-sensitive IT, called ITs, extends traditional IT in the sense that: a)there are structure-sensitive analogs to the notions of traditional IT and b)translating a theorem in IT by replacing its notions with their structure-sensitive counterparts, yields a (provable) theorem of ITs. Seemingly paradoxically, ITs extends IT but it's completely within the framework of IT. The richness of the suggested structures is demonstrated by two disparate families studied in more detail: the family of hierarchical structures and the family of linear structures. The formal findings extend the scope of cases to which a rigorous application of IT can be applied (with implications on quantization, for example). The implications on the foundations of IT are that the assumption regarding no underlying structure of states is not mandatory and that there is a framework for expressing such underlying structure.