a gamelist.xml cleaner tool

meleu

Looks like I'm getting addicted to data manipulation through shell scripts. Or, as @lilbud likes to say, tripping down a rabbit hole.

The fact is: I'm having fun coding stuff to manipulate xml data with xmlstarlet.

Leaving the metaphors and these nerd stuff aside, I would like to say that I've made a tool to clean up your gamelist.xml files.

Maybe you got somewhere else a really big gamelist.xml file with many entries for games you don't have. This tool is for you. If you have no idea what I'm talking about, forget this post and move on. :-)

For pragmatic reasons I am obliged to say "use at your own risk!", but I can assure that none of your gamelist.xml files will be modified, the script just creates another gamelist named gamelist.xml-clean. If you want to use the cleaner file, you have to manage the files accordingly (if you don't know what to do, probably this tool isn't for you).

The script's --help describes what it does:

[PROMPT]$ ./gamelist-cleaner.sh --help

This script gets a gamelist.xml as input and check if the path for the games
leads to an existing file. If the file doesn't exist, the <game> entry will
be deleted and a cleaner gamelist.xml file will be generated.

The resulting file will be named "gamelist.xml-clean" and will be in the
same folder as the original file. Nothing changes in the original gamelist.xml.

Usage:
./gamelist-cleaner.sh [OPTIONS] [gamelist.xml]...

The OPTIONS are:

-h|--help           print this message and exit.

-u|--update         update the script and exit.

-s|--system SYSTEM  specifies to which system the gamelist.xml file belongs,
                    e.g.: nes, megadrive. Default: name of the directory where
                    the file is located.

-d|--directory DIR  specifies the ROMs directory. Default:
                    /home/pi/RetroPie/roms

You can see the script here: https://github.com/meleu/share/blob/master/gamelist-cleaner.sh

And donwload it from command line with this command:

wget https://raw.githubusercontent.com/meleu/share/master/gamelist-cleaner.sh

Add permission to execute:

chmod a+x gamelist-cleaner.sh

At the end of processing you can see the difference between file sizes. Example:

[PROMPT]$ ./gamelist-cleaner.sh ~/RetroPie/roms/nes/gamelist.xml
The game with <path> = "./Weird ROM.nes" has been removed from xml.
The game with <path> = "./NonExistentFile.nes.nes" has been removed from xml.
The game with <path> = "./BlaBlaBla.nes" has been removed from xml.
...

The "/home/pi/RetroPie/roms/nes/gamelist.xml-clean" is ready!

See the difference between file sizes:
2.1M     /home/pi/RetroPie/roms/nes/gamelist.xml
1.3M     /home/pi/RetroPie/roms/nes/gamelist.xml-clean

TMNTturtlguy

@meleu Awesome!

Too bad i just got done cleaning all mine up manually! I have to add, my gamelists are terrible as I have used a few different scraping sources as well as created some on my own.

So my question to you @meleu is - what happens when i have a gamelist where the rom name and path match the rom name and path they are located, but the scraped meta data names don't match the rom name? Example:

NES GAME 1:

<game id="1514" source="ScreenScraper">
	<path>/home/pi/RetroPie/roms/nes/Blades of Steel.zip</path>
	<name>Blades of Steel</name>
	<desc>shortened for this post!</desc>
	<image>./downloaded_images/Blades of Steel-image.png</image>
	<thumbnail />
	<marquee>./downloaded_images/wheel/Blades of Steel (USA).png</marquee>
	<video>./downloaded_images/Blades of Steel-video.mp4</video>
	<rating>0.75</rating>
	<releasedate>19881202T000000</releasedate>
	<developer>Konami</developer>
	<publisher>Konami</publisher>
	<genre>Sports</genre>
	<players>1-2</players>
</game>

NES Game 2

<game>
	<path>./Chip 'n Dale Rescue Rangers 2 (USA).zip</path>
	<name>Chip 'n Dale Rescue Rangers 2</name>
	<image>./downloaded_images/boxart/Chip 'n Dale - Rescue Rangers 2 (USA).png</image>
	<video>./downloaded_images/snap/Chip 'n Dale - Rescue Rangers 2 (USA).mp4</video>
	<marquee>./downloaded_images/wheel/Chip 'n Dale - Rescue Rangers 2 (USA).png</marquee>
	<rating>0</rating>
	<developer />
	<publisher />
	<genre />
	<playcount>1</playcount>
	<lastplayed>20170525T020840</lastplayed>
</game>

As you can see, both of these work fine, and both have different folder locations and naming conventions. The ROM name does not match up exactly with the scraped art names in either scraping method. This disgusts me, however at the moment they all work fine and I don't have the time to rename every file and repath the scraped data.

Any idea how your script would react to this?

meleu

@TMNTturtlguy the script does exactly what it is saying it does: check if the rom exists (the file in <path> field) and generate a cleaner gamelist only with <game> entries with valid <path>s. (By valid I mean existing files)

The script wouldn't change any of the examples you gave.

Give the script a try. It doesn't change your gamelist, but creates a cleaner one. ;-)

meleu

NOTE: the script doesn't work if the gamelist.xml has invalid xml characters. Such as & in Track & Field. I will try to take a look at it soon.

Fortunately I implemented the --update feature, then the users can always try to run the script with that argument to get the latest version :-)

EDIT: HA! Looks like EmulationStation auto fix the gamelist.xml files with invalid xml characters as soon as it needs to update a gamelist.xml file. Then I'm not worried about this issue for a while! :D

theWinterDojer

@meleu Awesome! I was looking for this and didn't know it existed. I'm going to give it a try tonight, thank you.

How possible do you think it would be to do the same thing but for downloaded_images? I have duplicates of images from multiple scrapes taking up space but there is no way I can go through them all. I'd be happy to test it :D

meleu

@thewinterdojer it's a fair request. Will implement it as time permits. :-)

Sano

A (very) quick and (extremely) dirty script I just made to clean up images after your script, @meleu :

for sys in .emulationstation/downloaded_images/*
        do
        sys=`echo $sys|awk -F "/" '{print $NF}'`
        for file in .emulationstation/downloaded_images/$sys/*
                do
                cfile=`echo $file | sed s/\&/\&amp\;/g`
                if ! grep -q "$cfile" .emulationstation/gamelists/$sys/gamelist.xml
                        then
                        echo deleting $file
                        # rm "$file"
                fi
        done
done

theWinterDojer

@sano Awesome, so what exactly does this do? Will it just delete the images from the downloaded_images folder, or create a new clean folder like Meleu's script?

Sano

@thewinterdojer It's really quick and dirty, so if you don't understand it, you should wait for @meleu to add this feature in his script in a cleaner manner ;)
For this reason I didn't put the bash header, and commented the rm line.
It was just a proof of concept to help @meleu (but I think he will come with a cleverer solution anyway).

Basically, after @meleu script, all games listed in gamelist.xml files are existing games.

This script just browse the downloaded_images folder for each system, and verify that the images are used in the corresponding gamelist.xml file.
If an image is not used, it prints "deleting image" (and delete it only if the rm line is uncommented).

theWinterDojer

@sano Gotcha, I understand the concept of how it works, but I don't understand how to tell Linux that. Good stuff.

Omnija

Awww sweet, I've always hated the ES logs telling me how games are missing... and NOW "THEY SHALL BE GONE". :D

Considering im always looking for means of saving space, that other scripts to clean up images sounds great.

meleu

@sano thanks for the inspiration. It really helps! ;-)
I'll implement that when I have a chance.

Sano

@meleu said in a gamelist.xml cleaner tool:

@sano thanks for the inspiration. It really helps! ;-)

According to you bash skills, I don't think it's the truth ;)
Nevertheless, I had fun forgetting work for 10min, and doing a bash script for pleasure.

AndersHP

This is really cool! Good work, guys!

Kaltinril

Suggestion:

Script should clean up all related items to the file (As @Sano has suggested with his images deletion).

Change default behavior to:

Copy gamelist.xml to gameslist<DATE>.xml
Modify the gamelist.xml file directly
Move all images/video's into a sub-folder "Unused"

New Option: -r (remove)
Removes any files in the Unused folder (If supplied during the initial scan, does not move files to Unused, just deletes them)

The default behavior would mean if you have 15 systems, you have no work to do, but you still have the "backup" file.
The new option means if you are confident or don't care, after running the cleanup script a user would have nothing to do.

Thoughts:

Do people cross link images/movies for 1 system to the same game in another system or folder? If so, the script would need to check for ANY/ALL occurrences or references to the image/movie.
If a user provides this new -r option, should a backup of gameslist.xml be created at all?
Perhaps create multiple other shell scripts for the separate functions, and a master script that calls them.
A) gamelist-cleanup-master.sh - orchestrates the cleanup process by calling all sub-scripts and passing in parameters
B) gamelist-cleaner.sh - Does what is has always done.
C) downloaded-images-cleaner.sh - Would remove un-linked images and movies if no reference is found in any of the XML files

meleu

@kaltinril oh, I forgot this tool on my TODO list. :-)

Will try to improve it next week. I'm currently pretty busy.

theWinterDojer

@meleu I ended up manually going through my SNES folder and deleting the unused images. Took a few days, I'm gonna go ahead and patiently wait for you to do the other ones haha. It's not a space issue or anything, I'm just a perfectionist and don't want any unnecessary files on my machine.

Thanks for all the other tools, and really no rush. Real life takes precedent to our Pi's :)

Kaltinril

Going to clone and edit it and then send a pull request.

You can accept whatever you want, or reject it all :)

Kaltinril

I've got the -r working. I also added -a for "all" so that I can just type this:
./gamelist-cleaner.sh -a -r

and it cleaned up all folders and replaces the gamelist.xml instead of creating the -clean version.

https://github.com/kaltinril/share

I want to do some more testing before I submit a pull request.

meleu

@kaltinril you can submit a PR and then we can discuss about the code on github. ;-)