A host of new observations regarding BPS states in supersymmetric string compactifications point towards tantalizing new mathematical structures underlying string theory. The mathematical subjects of finite group theory, number theory (especially modular and mock modular forms), vertex operator algebras, generalized Kac-Moody algebras, enumerative geometry, and more have come to the fore in string theory of late, prompted in part by the ``Mathieu moonshine" observation in 2010 by Eguchi, Ooguri, and Tachikawa. This thesis describes progress in understanding mathematical aspects of string compactifications, with a focus on Mathieu and Umbral moonshines, automorphic forms, manifolds of special holonomy, and the algebraic and symmetric properties enjoyed by BPS subspaces of string theory. We make progress towards using string theory as an explanatory framework that ties together some of these mathematical disciplines. We also employ new mathematical ideas to constrain and describe highly symmetric string vacua.