User feedback in the form of movie-watching history, item ratings, or product consumption is very helpful in training recommender systems. However, relatively few interactions between items and users can be observed. Instances of missing user-item entries are caused by the user not seeing the item (although the actual preference to the item could still be positive) or the user seeing the item but not liking it. Separating these two cases enables missing interactions to be modeled with finer granularity, and thus reflects user preferences more accurately. However, most previous studies on the modeling of missing instances have not fully considered the case where the user has not seen the item. Social connections are known to be helpful for modeling users' potential preferences more extensively, although a similar visibility problem exists in accurately identifying social relationships. That is, when two users are unaware of each other's existence, they have no opportunity to connect. In this paper, we propose a novel user preference model for recommender systems that considers the visibility of both items and social relationships. Furthermore, the two kinds of information are coordinated in a unified model inspired by the idea of transfer learning. Extensive experiments have been conducted on three real-world datasets in comparison with five state-of-the-art approaches. The encouraging performance of the proposed system verifies the effectiveness of social knowledge transfer and the modeling of both item and social visibilities.