My power settings are the stock settings.
I think this is what happens: When the boxart image size is large, the Pi needs a bit of time to load the image. This causes a slight delay when scrolling from one game to the next. Other controllers ignore this, but the DS4 seems to remember the directional button input during the delay and keeps it going, without me having to press the button anymore.
But yes I am very glad I found the solution. Thanks for your input, as you asking me if specific games were causing this, got me on the right track.