Hi,
If you have overclocked your Raspberry Pi at all, the first thing I'd suggest it removing all overclock settings and seeing how the system runs.
Secondly, I'd check that your power supply is up to the job: Looking after your Raspberry Pi – Part 1: The importance of a quality power supply (PSU)
Whether or not you have overclocking enabled, the following guides will allow you to stress/stability test the cpu, ram, and sd storage.
Part 1: Overclocking the Raspberry Pi
Part 2: Stress testing the CPU with mprime
Part 3: Stress testing the RAM with Memtester
Part 4: Stress testing the SD card storage with the Stability Test Script
Regarding the appearance of the yellow square, overheating is common on the Pi 3, however the single-core nature of the emulators in RetroPie generally means this doesn't become a problem - for more information, please see:
Overclocking the Raspberry Pi 3: Thermal Limits and Optimising for Single vs Multicore Performance
Please note that soak testing is important - I found some stability issues that were uncovered only when running these tests for up to 24 hours (that said, for any serious instability these tests will reveal problems far more quickly)