Similarity has been a central concern for millennia, drawing the attention of philosophers, scientists, and artists. The issue of similarity is especially intriguing in the domain of the senses (and of sensory perception), where considerable research effort has been devoted to describing the affinities and divergences between different sensory modalities. In this work, we critically review the multiple different ways in which 'sensory similarity' has been theorized, highlighting how the similarities and analogies that researchers have proposed between the senses tend to constrain the theorization about how the senses operate. Similarities between different pairs of senses have been identified at multiple levels including in terms of stimuli/energy, transduction mechanisms, information-processing, synthetic/analytic analysis, perceptual organization, similarity of structure of the sensory space, and/or of the temporal evolution of different kinds of sensations. Based on the reviewed evidence, we propose an integrated framework, suggesting that sensory similarity can be understood as arising from (i) shared phenomenal qualities, (ii) analogous structural or organizational patterns, or (iii) convergent affective or semantic meanings, each grounded in increasingly cognitive and culturally-mediated processes.