The Double-Sided Information Bottleneck Function

A double-sided variant of the information bottleneck method is considered. Let $(\mathsf{X},\mathsf{Y})$ be a bivariate source characterized by a joint pmf ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$. The problem is to find two independent channels ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ (setting the Markovian structure $\mathsf{U}\to \mathsf{X}\to \mathsf{Y}\to \mathsf{V}$), that maximize $I(\mathsf{U};\mathsf{V})$ subject to constraints on the relevant mutual information expressions: $I(\mathsf{U};\mathsf{X})$ and $I(\mathsf{V};\mathsf{Y})$. For jointly Gaussian $\mathsf{X}$ and $\mathsf{Y}$, we show that Gaussian channels are optimal in the low-SNR regime but not for general SNR. Similarly, it is shown that for a doubly symmetric binary source, binary symmetric channels are optimal when the correlation is low and are suboptimal for high correlations. We conjecture that Z and S channels are optimal when the correlation is 1 (i.e., $\mathsf{X}=\mathsf{Y}$) and provide supporting numerical evidence. Furthermore, we present a Blahut–Arimoto type alternating maximization algorithm and demonstrate its performance for a representative setting. This problem is closely related to the domain of biclustering.