Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · cond-mat/0512017

Combinatorial Information Theory: I. Philosophical Basis of Cross-Entropy and Entropy

Robert K. Niven
View original source ↗Read source document ↗Export metadata ↓

Abstract

This study critically analyses the information-theoretic, axiomatic and combinatorial philosophical bases of the entropy and cross-entropy concepts. The combinatorial basis is shown to be the most fundamental (most primitive) of these three bases, since it gives (i) a derivation for the Kullback-Leibler cross-entropy and Shannon entropy functions, as simplified forms of the multinomial distribution subject to the Stirling approximation; (ii) an explanation for the need to maximize entropy (or minimize cross-entropy) to find the most probable realization; and (iii) new, generalized definitions of entropy and cross-entropy - supersets of the Boltzmann principle - applicable to non-multinomial systems. The combinatorial basis is therefore of much broader scope, with far greater power of application, than the information-theoretic and axiomatic bases. The generalized definitions underpin a new discipline of ``{\it combinatorial information theory}'', for the analysis of probabilistic systems of any type. Jaynes' generic formulation of statistical mechanics for multinomial systems is re-examined in light of the combinatorial approach. (abbreviated abstract)

Explore related subjects

cond-mat.stat-mechcs.ITmath-phmath.ITmath.MPphysics.data-an

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Robert K. Niven. 2007-04-20. Combinatorial Information Theory: I. Philosophical Basis of Cross-Entropy and Entropy. https://arxiv.org/abs/cond-mat/0512017

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

The meaning of entropy (demonstration of a much needed theorem)

The association of information with entropy has been argued on plausibility arguments involving the operation of imaginary engines and beings, and it is not a universal theorem. In this paper, a theorem by Charles Bennett on reversible computation that associates entropy with erasure of information is recognized as this much needed theorem. It is proposed a real, non thermal engine, operated by humans. It is proved: (1) The engine obeys two laws, identical {\it mutatis mutandis} to the two laws of thermodynamics; therefore, the entropy that arises in the operation of the engine has the same meaning of the entropy that arises in the operation of thermal engines. (2) The engine operates in stages similar to the stages in Bennett's three tapes reversible computer; therefore the entropy in the engine has the same meaning of the entropy in computation. The conclusion is that also the thermal entropy is a measure of erased or missing information. As a side result, information is measured in physical units, which complies with Landauer's principle. A prototype at work is shown in video.

cond-mat.stat-mech↗
arXiv2026

Information geometry of perturbed gradient flow systems on hypergraphs: A perspective towards nonequilibrium physics

This article serves to concisely review the link between gradient flow systems on hypergraphs and information geometry which has been established within the last five years. Gradient flow systems describe a wealth of physical phenomena and provide powerful analytical technquies which are based on the variational energy-dissipation principle. Modern nonequilbrium physics has complemented this classical principle with thermodynamic uncertaintly relations, speed limits, entropy production rate decompositions, and many more. In this article, we formulate these modern principles within the framework of perturbed gradient flow systems on hypergraphs. In particular, we discuss the geometry induced by the Bregman divergence, the physical implications of dual foliations, as well as the corresponding infinitesimal Riemannian geometry for gradient flow systems. Through the geometrical perspective, we are naturally led to new concepts such as moduli spaces for perturbed gradient flow systems and thermodynamical area which is crucial for understanding speed limits. We hope to encourage the readers working in either of the two fields to further expand on and foster the interaction between the two fields.

cond-mat.stat-mech↗
arXiv2026

Anomalous diffusion and singular transport from hydrodynamic recoupling

In charge neutral fluids, such as the Dirac fluid in graphene at the Dirac point, charge transport remains diffusive despite the presence of ballistically propagating sound waves: sound waves ``hydrodynamically decouple'' from the slower charge fluctuations. For quasi-one-dimensional charge neutral fluids, we show that this convective charge diffusion is not smoothly connected to the normal diffusion that arises when momentum conservation is broken by noise (or static impurities). Instead, the charge diffusion constant is a discontinuous function of noise, which (in the weak-noise limit) depends only on the ratio of momentum and energy relaxation rates. In the special limit of momentum-conserving noise (e.g., spatially uniform fluctuations of the Hamiltonian), the diffusion constant diverges in the presence of noise. We describe the resulting superdiffusion in terms of coupled Burgers equations. We present a general mechanism---hydrodynamic recoupling---by which weak noise can induce singular changes in transport coefficients. Our results highlight the limits of zero-noise extrapolation for predicting dynamical quantities like diffusion constants.

cond-mat.stat-mech↗