Making flexible decisions based on prior knowledge about causal environmental structures is a hallmark of goal-directed cognition in mammalian brains. Although several association brain regions, including the orbitofrontal cortex (OFC), have been implicated, the precise neuronal circuit mechanisms underlying knowledge-based decision-making remain elusive. Here, we established an inference-based auditory categorization task where mice performed within-session flexible stimulus re-categorization by inferring the changing task rules. We constructed a reinforcement learning model to recapitulate the inference-based flexible behavior and quantify the hidden variables associated with task structural knowledge. Combining two-photon population imaging and projection-specific optogenetics, we found that auditory cortex (ACx) neurons encoded the hidden task rule variable, which requires feedback input from the OFC. Silencing OFC-ACx input specifically disrupted re-categorization behavior. Direct imaging from OFC axons in the ACx revealed task state-related feedback signals, supporting the knowledge-based updating mechanism. Our data reveal a cortical circuit mechanism underlying structural knowledge-based flexible decision-making.