Just thought I'd throw this out there and mark this as resolved. For anyone looking to do what I did which is fully configure your RetroPie, then remove your controller configuration for a clean backup solution (Where you can then restore and configure whatever controller you want as it came Out of the Box) all you have to do is this.
Configure your Raspberry Pi fully (with RetroPie)
Make sure all settings are exactly as you'd like with splash screen, themes, core configs, scrapes, roms
Delete any save files for the cleanest backups
Go to RetroPie > RetroPie Setup > Manage Configurations > EmulationStation > Clear input Configuration
Exit to ES, press start on controller and Shutdown System
Remove MicroSD card, and insert in computer
Create your backup with your favorite backup software, then "Pi shrink" it