Thank you @petrockblog , I found out the switch I used was a normally on momentary switch, and I expected it to be normally off. I've had those switches forever in my components to reuse bin. I never bothered to measure with multimeter based on the fact that pressing it did shut down Retropie as I expected. This one's totally on me and sorry to bug you, I appreciate you helping out.