I did some more tests and I guess I found out what was causing this problem. After cloning the sd card everything works fine until I copy my rom's to the ssd. I just attached the ssd with the usb adapter to my Ubuntu machine and copied everything there to get faster speeds. But it seems that this method changes something in the filesystem that then causes all sorts of problems when booting from the ssd.
I now simply copied the sd card to the ssd again to get a fresh start and then copied everything with SFTP and now everything seems to be running fine.