a gamelist.xml cleaner tool

chipsnblip

i think i might have found the culprit regarding games that have an ampersand in the filename (& in the gamelist.xml)

line 221 in the script reads:

xmlstarlet ed -L -d "/gameList/game[path=\"$path\"]" "$clean_gamelist"

but if i change it to:

xmlstarlet ed -L -d "/gameList/game[path=\"$full_path\"]" "$clean_gamelist"

they are handled properly. i'm not a programmer by any means (just a computer enthusiast), so hopefully someone with more knowledge than myself can confirm that it's working.

duiz

I suffer from some rather messy gamelists, but I am a newbie when it comes to these commandline things... Is it possible to request a small step by step guide to get this up and running for a beginner with little experience in github and etc?

hansolo77

Wow, @meleu releases another script I didn't even know about! :) Can't wait to try it out tomorrow when I'm more awake.

Degsy

@chipsnblip Nearly right.. and that led me on to the fix

If you instead add in this above the xmlstarlet line, it will handle it correctly.

path="$(echo "$path" | sed 's@&amp;@\&@g')"

If you want to go further and want to see feedback onscreen when its triggered, then INSTEAD add in this block above the xmlstarlet line:

	#&amp; fix
	if [[ $path == *"&amp;"* ]]; then
		path="$(echo "$path" | sed 's@&amp;@\&@g')" 
		echo "&amp; fix applied"
	fi

The reason why it was failing was that either bash or xmlstarlet was translating it incorrectly.
The $full_path that you were using already has the & translated back to &, but that variable also includes the full path of the rom (/home/pi/RetroPie/roms/… etc) which does not match what is in the XML (didn't match mine anyway).

paffley

@meleu I like the look of this so thank you.
I have it installed on my pi.
I have my gamelist.xml in my roms folder along with all my artwork (boxart/cartart/snap etc).

Please could you let me know what I type in the command line to run the script?

I've tried all but failed miserably haha :P

Update: I believe I have sorted it, its running the script now as I type :)

Update 2: This script is amazing! Thank you so much, works flawlessly with @Degsy fix in the above post.

fefitohh

Hi

Is it possible to use this tool to remove certain tags contained in every available gamelist.xml?

What I would like to do is to remove the scraped "raiting" from all my gamlists.

mitu

@fefitohh Not with this script, but someone posted something similar here for the lastplayed and playcount tags. You can adapt it to remove the rating instead.

hansolo77

@meleu said in a gamelist.xml cleaner tool:

NOTE: the script doesn't work if the gamelist.xml has invalid xml characters. Such as & in Track & Field. I will try to take a look at it soon.

Fortunately I implemented the --update feature, then the users can always try to run the script with that argument to get the latest version :-)

EDIT: HA! Looks like EmulationStation auto fix the gamelist.xml files with invalid xml characters as soon as it needs to update a gamelist.xml file. Then I'm not worried about this issue for a while! :D

This "bug" is bugging me. I have a huge Nintendo DS collection I've been going through. I have a lot of duplicate games as a result of different regions (USA, Europe, Japan). So after scraping, I'm going through my gamelist and removing all the extra non-USA versions where possible. I'm up to the letter P now, after about 4 days lol. I was editing the gamelist.xml manually until letter G before I remembered this script! Anyway, when I'm running the script, it's always missing some roms. I tried re-running it multiple times with the same result. I couldn't find a repo in your github to report it as an error, so I looked here. I discovered that any games with the & symbol in the filename (designated as & it would report as removed but wouldn't be). Sure enough, you already recognized it. So I'm just confirming that the issue still remains. I've even tried running the -u flag to update but it doesn't look like you've done anything with yet (cough 2 years). :)

I can still edit these out manually, but a fix would be nice for other people too. Hope you're doing well. Have a good one!

Example output of games NOT getting removed:

The game with <path> = "/home/pigaming/RetroPie/roms/nds/Gakken Chuugokugo Zanmai DS - Kiki &amp; Kaki Training (Japan) [b].zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Gakken Hangul Zanmai DS - Kiki &amp; Kaki Training (Japan) [b].zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Happy Party with Hello Kitty &amp; Friends! (Europe) (En,Fr,De,Es,It,Nl).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/My Baby 2 - Boy &amp; Girl (Europe) (En,Fr,De,Es,It,Nl).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/My Best Friends - Cats &amp; Dogs (Europe) (En,Fr,De,Es,It).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Nintendogs - Labrador &amp; Friends (Europe) (En,Fr,De,Es,It).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Nintendogs - Miniature Dachshund &amp; Friends (Japan).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Nintendogs - Shiba &amp; Friends (Japan).zip" has been removed from xml.
The game with <path> = "/home/pigaming/RetroPie/roms/nds/Percy Jackson &amp; the Lightning Thief (Europe) (En,Fr,De,Es,It).zip" has been removed from xml.

shantigilbert

@hansolo77 I was having the same isue but I think I found a solution that works for me, I just need to confirm that it works on other setups, could you do a ls -la in your NDS rom folder and post the result to pastebin ? also paste your gamelist.xml

EDIT: I just opened a PR to deal with this issue

https://github.com/meleu/share/pull/12

hansolo77

Since YOu posted a pull request, do you still need me to give you the folder list and gamelist?

shantigilbert

@hansolo77 It would be helpful to test, or you can test it yourself see if that fixes your issue (in case you still have it)

meleu

@hansolo77 to test @shantigilbert's script you must have to download it:

wget https://raw.githubusercontent.com/shantigilbert/share/master/gamelist-cleaner.sh

hansolo77

Grrr... I was all excited and ready to test this, but it looks like I already manually cleaned my gamelist. I'll have to wait and see if it shows up on any new systems I add.

pjft

@meleu just saying that I'm late to the party, but this is spectacular. Thank you for putting this together! Hope you're doing great :)

Edit: actually, some feedback: this totally fails for games whose ROM is the folder - such as Daphne. It doesn't detect the folder and removes it from the gamelist. Other than that, all seems to be working well!

meleu

@pjft oh! I seriously need to update that script... I'll try it soon...

pjft

@meleu no prob, it is a great tool nonetheless :) It's just a matter of not replacing the Daphne gamelist :D

How are things? Good to hear from you!

bmulligan

Great tool. You've saved me hours of eye, back and neck strain. Thank you for this!