Plants contain rapidly evolving specialized metabolic enzymes to support the synthesis of a myriad of functionally diverse natural products. In the case of coumarin biosynthesis, a BAHD acyltransferase-family enzyme COSY was recently discovered in Arabidopsis that catalyzes coumarin formation from o-hydroxylated trans-hydroxycinnamoyl-CoA substrates. COSY is the first and only BAHD enzyme known to date that catalyzes an intramolecular acyl transfer reaction. Here we combine structural, biochemical, and computational approaches to investigate the mechanistic basis for the unique coumarin synthase activity of COSY. Comparative analyses of crystal structures of Arabidopsis thaliana COSY relative to other BAHD proteins reveal that COSY possesses an unconventional active-site configuration adapted to its specialized activity. Through deuterium exchange experiments, we discover a unique proton exchange mechanism at the β-carbon of the o-hydroxylated trans-hydroxycinnamoyl-CoA substrates during the catalytic cycle of COSY. Mutagenesis studies and quantum mechanical cluster modeling further support that this mechanism is key to COSY’s ability to lower the activation energy of the trans-to-cis isomerization of the hydroxycinnamoyl-CoA substrates, a critical rate-limiting step leading to courmarin production. This study unveils the emergence of an unconventional catalytic mechanism mediated by a BAHD-family enzyme, and sheds light on the potential evolutionary origin of COSY and its recruitment to the evolutionarily new coumarin biosynthetic pathway in eudicots.