The ability to decode social and emotional information from interaction partners is fundamental to the initiation and maintenance of social interactions and interpersonal relationships. A vital source of such socio-emotional information is the face. Nevertheless, accurately interpreting facial expressions during social exchanges is a deceptively complex task. In particular, a variety of additional contextual cues—including the expresser’s body language—are also present within these interactions, each of which can alter how facial expressions are evaluated. In addition, facial expressions may simultaneously shape perceptions of these contextual cues as part of a dynamic bidirectional relationship. The aim of the present chapter is to review the empirical work which has attempted to assess the role of various contextual factors on social and emotional judgements from faces. Furthermore, we recommend that the fields of affective and social psychology take a more holistic and contextualised approach to face processing in future.