Okay, solved and this is one I recommend making more clear in the documentation. My issue was that I ALWAYS keep a wireless keyboard connected to my Retropie. It's just a tiny dongle taking one port so why not? It's so much easier for certain tasks.
However, I kept wondering why the test rom for Intellivision was reporting that I was on controller 2 and every other comment pointed to controller one. Then I noticed posts where people said that Daphne and Intellivion (jzintv anyway) assign the first controller they see, which in this case was the keyboard -- and that explained why the keyboard worked. Ugh.