a gamelist.xml cleaner tool

greenie

@meleu I've now tried this on several PCs, (including the Windows Subsystem for Linux!) and it's the same result on all of them.

To reproduce...

Download one of the gamelist.xml and downloaded_images files/folders from the this master repository: https://github.com/recalbox/recalbox-gamelists/tree/master/gamelists .

Drop them in your system's ROMS folder. This is based on the no-intro romset, which my collection is based on.

You should now have the gamelist.xml file, downloaded_images folder and ROMs under your ROMS folder (in this example I'm using the NES system).

Run the gamelist-cleaner.sh script after editing the settings to point at your ROMS folder.

The script appears to run perfectly. I can watch the output in Terminal with glee!

But, after inspecting the new xml file, you'll see that entries such as Advanced Dungeons & Dragons - DragonStrike (USA).zip are still present even though, in my case, I don't have the ROM for that game.

Is this a formatting issue? Has anyone else experienced this?

Any advice would be hugely appreciated as this is the last step I need to nail before my system is finished. I'll be editing out the remaining entries in the mean time.

Thanks.

meleu

@greenie
Regarding the & char, it's a known issue and I posted about it here. In the linked post you can see a workaround to fix this issue (I agree that it isn't the ideal solution, though).

And thanks for this report. ;-)

greenie

@meleu Thanks for the response. I don't think this is exactly the same issue, I'm not sure what workaround you refer to on your post either.

The problem I was getting is that anything in the gamelist.xml file with "& amp ;" in the name was not getting removed. It could see the entry, and list it as removed in the terminal, but the entry remained.

I have managed to find my own workaround:

Find/Replace on gamelist.xml, changing all "& amp ;" to "ampersand".
Use A Better Finder Rename on the Mac (I'm sure there are similar tools out there, but I have this handy) to replace all "&" in ROM file names to "ampersand".
Run your script, then reverse the above steps.

Bish bash bosh. Done.

Tim

I'm new to RetroPie, but wanted to say thank you for this tool. My OCD thanks you as well. Sweet, sweet, clean gamelist!

AndersHP

@meleu could my issues here relate to the symlink issues I have in the other thread regarding launching images?

chipsnblip

i think i might have found the culprit regarding games that have an ampersand in the filename (& in the gamelist.xml)

line 221 in the script reads:

xmlstarlet ed -L -d "/gameList/game[path=\"$path\"]" "$clean_gamelist"

but if i change it to:

xmlstarlet ed -L -d "/gameList/game[path=\"$full_path\"]" "$clean_gamelist"

they are handled properly. i'm not a programmer by any means (just a computer enthusiast), so hopefully someone with more knowledge than myself can confirm that it's working.

duiz

I suffer from some rather messy gamelists, but I am a newbie when it comes to these commandline things... Is it possible to request a small step by step guide to get this up and running for a beginner with little experience in github and etc?

hansolo77

Wow, @meleu releases another script I didn't even know about! :) Can't wait to try it out tomorrow when I'm more awake.

Degsy

@chipsnblip Nearly right.. and that led me on to the fix

If you instead add in this above the xmlstarlet line, it will handle it correctly.

path="$(echo "$path" | sed 's@&amp;@\&@g')"

If you want to go further and want to see feedback onscreen when its triggered, then INSTEAD add in this block above the xmlstarlet line:

	#&amp; fix
	if [[ $path == *"&amp;"* ]]; then
		path="$(echo "$path" | sed 's@&amp;@\&@g')" 
		echo "&amp; fix applied"
	fi

The reason why it was failing was that either bash or xmlstarlet was translating it incorrectly.
The $full_path that you were using already has the & translated back to &, but that variable also includes the full path of the rom (/home/pi/RetroPie/roms/… etc) which does not match what is in the XML (didn't match mine anyway).

paffley

@meleu I like the look of this so thank you.
I have it installed on my pi.
I have my gamelist.xml in my roms folder along with all my artwork (boxart/cartart/snap etc).

Please could you let me know what I type in the command line to run the script?

I've tried all but failed miserably haha :P

Update: I believe I have sorted it, its running the script now as I type :)

Update 2: This script is amazing! Thank you so much, works flawlessly with @Degsy fix in the above post.

fefitohh

Hi

Is it possible to use this tool to remove certain tags contained in every available gamelist.xml?

What I would like to do is to remove the scraped "raiting" from all my gamlists.

mitu

@fefitohh Not with this script, but someone posted something similar here for the lastplayed and playcount tags. You can adapt it to remove the rating instead.

hansolo77

@meleu said in a gamelist.xml cleaner tool:

NOTE: the script doesn't work if the gamelist.xml has invalid xml characters. Such as & in Track & Field. I will try to take a look at it soon.

Fortunately I implemented the --update feature, then the users can always try to run the script with that argument to get the latest version :-)

EDIT: HA! Looks like EmulationStation auto fix the gamelist.xml files with invalid xml characters as soon as it needs to update a gamelist.xml file. Then I'm not worried about this issue for a while! :D

This "bug" is bugging me. I have a huge Nintendo DS collection I've been going through. I have a lot of duplicate games as a result of different regions (USA, Europe, Japan). So after scraping, I'm going through my gamelist and removing all the extra non-USA versions where possible. I'm up to the letter P now, after about 4 days lol. I was editing the gamelist.xml manually until letter G before I remembered this script! Anyway, when I'm running the script, it's always missing some roms. I tried re-running it multiple times with the same result. I couldn't find a repo in your github to report it as an error, so I looked here. I discovered that any games with the & symbol in the filename (designated as & it would report as removed but wouldn't be). Sure enough, you already recognized it. So I'm just confirming that the issue still remains. I've even tried running the -u flag to update but it doesn't look like you've done anything with yet (cough 2 years). :)

I can still edit these out manually, but a fix would be nice for other people too. Hope you're doing well. Have a good one!

Example output of games NOT getting removed:

The game with <path> = "/home/pigaming/RetroPie/roms/nds/Gakken Chuugokugo Zanmai DS - Kiki &amp; Kaki Training (Japan) [b].zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Gakken Hangul Zanmai DS - Kiki &amp; Kaki Training (Japan) [b].zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Happy Party with Hello Kitty &amp; Friends! (Europe) (En,Fr,De,Es,It,Nl).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/My Baby 2 - Boy &amp; Girl (Europe) (En,Fr,De,Es,It,Nl).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/My Best Friends - Cats &amp; Dogs (Europe) (En,Fr,De,Es,It).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Nintendogs - Labrador &amp; Friends (Europe) (En,Fr,De,Es,It).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Nintendogs - Miniature Dachshund &amp; Friends (Japan).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Nintendogs - Shiba &amp; Friends (Japan).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Percy Jackson &amp; the Lightning Thief (Europe) (En,Fr,De,Es,It).zip" has been removed from xml.

shantigilbert

@hansolo77 I was having the same isue but I think I found a solution that works for me, I just need to confirm that it works on other setups, could you do a ls -la in your NDS rom folder and post the result to pastebin ? also paste your gamelist.xml

EDIT: I just opened a PR to deal with this issue

https://github.com/meleu/share/pull/12

hansolo77

Since YOu posted a pull request, do you still need me to give you the folder list and gamelist?

shantigilbert

@hansolo77 It would be helpful to test, or you can test it yourself see if that fixes your issue (in case you still have it)

meleu

@hansolo77 to test @shantigilbert's script you must have to download it:

wget https://raw.githubusercontent.com/shantigilbert/share/master/gamelist-cleaner.sh

hansolo77

Grrr... I was all excited and ready to test this, but it looks like I already manually cleaned my gamelist. I'll have to wait and see if it shows up on any new systems I add.

pjft

@meleu just saying that I'm late to the party, but this is spectacular. Thank you for putting this together! Hope you're doing great :)

Edit: actually, some feedback: this totally fails for games whose ROM is the folder - such as Daphne. It doesn't detect the folder and removes it from the gamelist. Other than that, all seems to be working well!

meleu

@pjft oh! I seriously need to update that script... I'll try it soon...