Improve gamelist parsing for large ROM collection in EmulationStation for Windows

[uzi.vu]

Re: Emulation slow startup when roms are in network folder
I faced the same problem when starting up ES with my ROM collections stay on my home NAS. Even when the "Parse gamelist only" option was checked, the ES loading screens till take more than a minutes to finish with about 1000 ROMs.

After a few days searching thru the web without much success, I decided to download the ES source code and tried looking.

The problem lies in the parseGamelist() function (EmulationStation\es-app\src\Gamelist.cpp), which call a few functions, where them all call isDirectory() function (EmulationStation\es-core\src\utils\FileSystemUtil.cpp). This boolean function in turn, calls system's stat() function to check if the given path is a directory or not. THIS is where the speed slow down.

Since most of the time, this check is uncessary, if you got your configuration done right, just like mine, so I went ahead and modified the code to bypass this check, recompiled, and finally, my ES now load fine in just a few seconds, regardless of how many ROMs it gets.

So, the answer is:

1. You can't simply get much better loading speed in ES by just turning on "Parse gamelist only", ofcorse this option does improve loading speed a lot.
2. You must alter the source code and compile ES your, per instruction in https://retropie.org.uk/forum/topic/5202/step-by-step-how-to-build-emulationstation-on-windows, by fieldofcows. I used the latest fork from Retropie, which added RapidJSON to the source, but the instruction stay true for most part.

You must skip isDirectory checks in 2 functions: resolveRelativePath() and removeCommonPath(), both in EmulationStation\es-core\src\utils\FileSystemUtil.cpp. Or in my case, I create 2 new clones of them, called removeCommonPath2 and removeCommonPath2, in which I skip isDirectory check, then alter the following functions to call them instead of the original ones:

1. findOrCreateFile() and parseGamelist(), both on EmulationStation\es-app\src\Gamelist.cpp.
2. createFromXML(), in EmulationStation\es-app\src\MetaData.cpp.

My method compiled and ran well on ES (RetroPie fork) v2.10.0.0. The altered code is shown below:

In FileSystemUtil.h

std::string resolveRelativePath2(const std::string& _path, const std::string& _relativeTo, const bool _allowHome);
		std::string removeCommonPath2(const std::string& _path, const std::string& _common, bool& _contains);

In FileSystemUtil.cpp

std::string resolveRelativePath2(const std::string& _path, const std::string& _relativeTo, const bool _allowHome)
		{
			std::string path = getGenericPath(_path);
			std::string relativeTo = getGenericPath(_relativeTo);

			// nothing to resolve
			if (!path.length())
				return path;

			// replace '.' with relativeTo
			if ((path[0] == '.') && (path[1] == '/'))
				return (relativeTo + &(path[1]));

			// replace '~' with homePath
			if (_allowHome && (path[0] == '~') && (path[1] == '/'))
				return (getHomePath() + &(path[1]));

			// nothing to resolve
			return path;

		} // resolveRelativePath2

std::string removeCommonPath2(const std::string& _path, const std::string& _common, bool& _contains)
		{
			std::string path = getGenericPath(_path);
			std::string common = getGenericPath(_common);

			// check if path contains common
			if (path.find(common) == 0)
			{
				_contains = true;
				return path.substr(common.length() + 1);
			}

			// it didn't
			_contains = false;
			return path;

		} // removeCommonPath2

Or if you dont' have the time, download the compiled ES executable here (RetroPie fork, 2.10.0.0), tested on Windows 10 64bit: https://drive.google.com/file/d/1IE8noWCWyFVbsY8oHc9Q5EmtjMUNG5YV/view?usp=sharing

[mitu]

It's probably because stat is slow when using your NAS - have you tried comparing the speed when the ROM folder is on the local disc ?
This doesn't sound like an improvement, but a hack, which will probably break some functionality for systems where a folder can be used as a game/rom (i.e. Daphne).

[uzi.vu]

@mitu Yes this is a quick hack for people running ROMs from NAS. Specifically, for accessing via SMB shares. stat() has no speed problem with local ROM collection at all. If we look at it further, may be there's a better solution.