As regular conformal blocks describe the N=2 superconformal gauge theories in four dimensions, irregular conformal blocks are expected to reproduce the instanton partition functions of the Argyres-Douglas theories. In this paper, we construct matrix models which reproduce the irregular conformal conformal blocks of the Liouville theory on sphere, by taking a colliding limit of the Penner-type matrix models. The resulting matrix models have not only logarithmic terms but also rational terms in the potential. We also discuss their relation to the Argyres-Douglas type theories.