Abstract

I will argue that it is highly desirable to frame scene understanding in terms of hierarchical probabilistic models. These allow top-down and bottom-up flows of information to take place, in order to provide a scene interpretation. Encoded within such a model would be knowledge at various levels, e.g. lower-level models of regions and boundaries, and at a higher level the shape and appearance of object classes, and their contextual relationships. I will further argue that such models should be learned in a largely unsupervised fashion from image data. Hinton’s “deep learning” agenda is attractive here in that it provides an upgrade path from lower-level to higher-level regularities.

We assess the generative power of the mPoT-model of Ranzato et al (2010) with tiled-convolutional weight sharing as a model for visual textures by specifically training on this task, evaluating model performance on texture synthesis and inpainting tasks using quantitative metrics. We also analyze the relative importance of the mean and covariance parts of the mPoT model by comparing its performance to those of its subcomponents. Our results suggest that while state-of-the-art or better performance can be achieved using the mPoT, similar performance can be achieved with the mean-only model. We then extend this model to handle multiple textures, using a shared set of weights but texture-specific hidden unit biases. We show comparable performance of the multiple texture model to individually trained texture models, providing state-of-the-art results in the texture synthesis and inpainting tasks.

Joint work with Jyri Kivinen, Ali Eslami, Nicolas Heess.