I've read about Wayland fixing things, and it seems like it fixed the issue in my case. Installed KDE, enabled Wayland and it seems to work great now :)