Making premade gamelist.xml with xtra media collections

findrbot_admin

You could easily hack a script to do all that in ShellScript, Python, or even PHP... (EDIT: My bad, you already did! Hacking code is fun, especially shellscript!)

I am working on a shellscript right now that will copy all the ROM files from the subdirectories and deletes all the garbage... This way I can download a butt-tonne of roms, put them all in the console folders, and automatically move them to the top level dir...

So basically it moves this:
/roms/n64/Super Mario 64/US/ROM/Folder1/Folder2/Folder3/Folder4/Folder5/Folder6/Folder7/Folder8/Super Mario 64 (U).n64
To this:
/roms/n64/Super Mario 64 (U).n64

And it removes any file that is not a rom - all the images, txt files, exe files, etc. because their is no point in adding those files into a ROM pack as they just weigh it down and that's what the scraper is for... I had over 100GB of ROM's downloaded from the internet, and after running my script to get rid of all the garbage it was almost 50 GB... I will post accurate stats after its fully coded and tested...

The script runs right on the RasPi or you can run it on your computer to clean up your ROM download folder. I will post a link to my script/website (http://wiki.findrbot.com/) when it is complete, I am still adding features... There are too many ideas of things to add! I coded the entire script over a day or two, just putting final touches in... I am also releasing mega ROM packs on my website, but those are shunned here so I won't link to them ;)

I was going to make a scraper, but there is no point... Just use SSelph's scraper (google it) right on the RasPi to get all the game information and boxart. It only makes sense to offload most of the downloading of non-essentials to the person doing the ROM downloading and the gamesdb websites... Less load on my server!

There really is also no point in making a gameslist.xml generator either, because you can just run SSelph's scraper, which will not only generate the gameslist.xml, it will also download the boxart and descriptions :)

You could make one anyways, for the 'function' of it (lol)

Cheers :)

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu Final results of second run (before running again with folder find code):
Start Time: 3:42, End Time: 5:56
Total Time: 2 hours and 14 minutes
Previous Errors: 108 out of 745
Current Errors: 5 errors out of 745

meleu

@Used2BeRX hey man, --update the script and run it again. I've fixed those 5 errors and the folder synopsis is working (at least on my quick tests).

Note: the changes I've made in the code ignores (doesn't update) those 5 old entries with problems. It creates new fine and working entries.

Used2BeRX

@meleu lol... nice pic.

I updated it and it's running now. Looks like it takes about 30-60 seconds per entry now, so I'm guessing it's going to take about 8 hours to make the file this time, but I checked the file so far and all of the folders are being entered right now. :)

I put the "time" operator so I'll have an exact time for you when it's done. Thanks again.

meleu

@Used2BeRX

Looks like it takes about 30-60 seconds per entry now

Wow! :( It's a bad news...

Well at least I have a good news: I've just added the --only-new option, so if you just wanna add new entries and ignore the existing ones, use this option (check the --help message I've updated accordingly).
(edit: I've uploaded a few minutes ago, probably you don't have this version)

Example of use case:

You're starting a new gamelist.xml and the script is running.
For some reason you have to turn off your raspi and the script didn't process all files yet.
Hit ctrl+c and turn off your raspi.
When you have a chance to run the script again use the --only-new option and it will ignore the synopsis.txt of all the existing entries in the gamelist.xml.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

Looks like it takes about 30-60 seconds per entry now

Please, do the following:

cancel the execution (ctrl+c),
remove the generated gamelist.xml,
--update the script,
run it again with the same parameters you used (including the time operator).
let me know if it takes more than 30 seconds per entry.

IMO 30 seconds per entry is unacceptable. OK, it's faster than human, but too slow for a computer (even a raspiZero). I have to find the bottleneck!

EDIT:
By the way, the script took almost 4 minutes to process 718 synopsis for atari2600 games on my Core i7 computer. OK, it has much more CPU power than a raspiZero, but 8 hours?! Try it again, please.

Used2BeRX

@meleu Sorry. I was away from the computer and didn't get your messages until now. I will upgrade and try it again.

FYI, I think I was right about 8 hours. As of writing this, it had only done 425 entries out of 745. That was in 274 minutes, real-time. (4:34)

Part of this was my fault for running the wrong command. I was running it with the --full option. I tried that again with --full in your new script and timed it and it was taking 40-41 seconds per entry.

I stopped it, deleted the output, and re-ran it with --no-desc and timed it at 10-11 seconds per entry, so I'm letting it run now. This is probably going to end up being just a little longer than the 2 hours and 12 minutes before you added the code to find the folders.

I'd still like to see if there are ways of tweaking this. I won't be using the --full option for my raspi zero setup, but I will be using it for the Pi 3 setup. Granted that is a faster processor itself, but this just seems to be taking a lot longer than it should to get the job done.

Just curious, are you doing the following things?

Searching for roms only in the following folder and sub-folders: /opt/pi/RetroPie/roms/{system_name}/ (system_name from the matching one used in the command running the script, ie: /opt/pi/RetroPie/Media/atari2600/Synopsis/*.*)
Searching for folders and sub-folders only in the same folder and sub-folders as number one above. And ONLY looking for folder names, completely disregarding any other files.
Searching for Media only in the following folder and sub-folders: /opt/pi/RetroPie/Media/{system_name}/ (You could get the system_name from the command running the script like above).
You could further narrow this down by pointing particular media fields to their exact folders, like <gamefaqs/> to /Media/{system_name}/GameFAQs/ and <threedbox> to /Media/{system_name}/Artwork/3D Boxart/

When I get a chance after this is done I'll re-check everything in EmulationStation and see if there are any more individual problems with files/folders displaying. When we can speed up the code I will start running a few --full and default runs and see if all of the media is being done right.

meleu

@Used2BeRX

re-ran it with --no-desc and timed it at 10-11 seconds per entry (...). This is probably going to end up being just a little longer than the 2 hours and 12 minutes before you added the code to find the folders.

It's an unfair comparison. I noticed that creating a new gamelist.xml from scratch takes more time than just updating an existing one. Then I'm assuming the time didn't changed.

Just curious, are you doing the following things?

Searching for roms only in the following folder and sub-folders: /opt/pi/RetroPie/roms/{system_name}/ (system_name from the matching one used in the command running the script, ie: /opt/pi/RetroPie/Media/atari2600/Synopsis/*.*)

Yes. But the system_name is based on the Platform: field.

Example: If Platform: Atari 2600, then system_name=atari2600.

Searching for folders and sub-folders only in the same folder and sub-folders as number one above. And ONLY looking for folder names, completely disregarding any other files.

Yes.

Searching for Media only in the following folder and sub-folders: /opt/pi/RetroPie/Media/{system_name}/ (You could get the system_name from the command running the script like above).

Yes. But system_name is obtained as I described above.

You could further narrow this down by pointing particular media fields to their exact folders, like <gamefaqs/> to /Media/{system_name}/GameFAQs/ and <threedbox> to /Media/{system_name}/Artwork/3D Boxart/

Yes, I'm doing this.

I think I've made some progress in optimization...

Suggestion: to measure the optimizations, there's no need to wait all those 2+ hours, try to parse only the synopsis.txt files starting with A. The command should be like this:

time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/atari2600/Synoipsis/A*.*

Got it?

For a fair comparison do the following:

do NOT --update your script for now.
remove the atari2600_gamelist.xml file.
run the command above (only parse synopsis.txt files starting with A).
take note of the times (real, user and sys).
remove the atari2600_gamelist.xml file.
--update the script.
run the command above (only parse synopsis.txt files starting with A).
take note of the times (real, user and sys).
post the times here.

Looking forward for the results...

Used2BeRX

@meleu Alright. I'm going to do what you asked. I just looked at the time for the last run at it was 136m/102m/28m.

I'll let you know more when I know more.

Used2BeRX

@meleu Wow.... what did you just do?
Old times: 6m4s/4m34s/1m13s
New times: 1m5s/0m33s/0m19s

You just cut the time down to about 20% of what it was before!

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu Wow.... what did you just do?

Tweaks on find command.

Now let's see the time for the complete atari2600 set! :-)

EDIT
And check if the gamelist.xml is fine, please.

Used2BeRX

@meleu Just finished running the full --no-desc
30m12s, 17m58s, 8m10s

I'm going to check to see if there were any problems with it now.

Used2BeRX

@meleu 100% hits for boxart so that means that it worked perfect for path and image :)

Had to do a long overdue reboot so it took a while for me to get back up but I did a few others while I was waiting.

Atari 7800: 3m44s, 2m11s, 0m57s
Atari Lynx: 3m25s, 1m56s, 0m54s
Atari 5200: 19m17s, 12m43s, 4m51s

CURRENT ISSUES:

None of the `<path/> were written for Atari 5200/800. This might be because it's a special case. Of all of the systems I have for the RetroPie, this is the only one that you can't ZIP the files. All 5200/800 images I have are in BIN format.
Folders for other systems are being added to "atari2600_gamelist.xml". (I know why this is happening. All of my folders.txt files still say the information for the Atari 2600).
With the Atari 5200/800, I'd like for them to all write to atari5200_gamelist.xml. I don't include a lot of 800 games in my collection and rather than have two entries for it in the system I have all of the roms and media in the 5200 file structure, so I was wondering if we could maybe do somthing about putting those two scripts together. If that's too much of a hassle, no worries. I can just manually combine them when I'm done.

I went through both Atari 7800 and Atari LYNX and we have 100% boxart showing up for both of them as well!!!

As soon as I get the folder synopsis text files updated the folders should work perfect the next time I run the script.

Looking great though.

Now that we got the speed issue handled, I was wondering if we could talk about a few more options to add to the script?

I had a few ideas, such as creating all of the gamelist.xmls for all existing systems one after another without having to be there to put in the next command.(or from a list I could create in a file).

I was also thinking that we could do a --no-desc but with all of the Artwork folders, and then there could be a script that would easily change all of the entries to point to a different piece of artwork (ie: displaying action shots for all of the games instead of Boxart).

meleu

@Used2BeRX

CURRENT ISSUES:

None of the `<path/> were written for Atari 5200/800. This might be because it's a special case. Of all of the systems I have for the RetroPie, this is the only one that you can't ZIP the files. All 5200/800 images I have are in BIN format.

Yep. The script was only trying to find .zip. I've made it search for .bin too. Do you have other extensions?

Folders for other systems are being added to "atari2600_gamelist.xml". (I know why this is happening. All of my folders.txt files still say the information for the Atari 2600).

Exactly. The synopsis.txt files must be accurate. In this case, the problem is in the Platform: field.

With the Atari 5200/800, I'd like for them to all write to atari5200_gamelist.xml. I don't include a lot of 800 games in my collection and rather than have two entries for it in the system I have all of the roms and media in the 5200 file structure, so I was wondering if we could maybe do somthing about putting those two scripts together. If that's too much of a hassle, no worries. I can just manually combine them when I'm done.

The script puts every entry on its own platform gamelist.xml. Do they use the same emulator in RetroPie? Anyway, if you want to see atari 800 games in the atari 5200 you'll have to combine manually, as you said.

I had a few ideas, such as creating all of the gamelist.xmls for all existing systems one after another without having to be there to put in the next command.(or from a list I could create in a file).

You can give everything in only one command line. Like this:

./Used2BeTXT /path/to/atari2600/synopsis/*.* /path/to/nes/synopsis/*.* /path/to/megadrive/synopsis/*.*

I was also thinking that we could do a --no-desc but with all of the Artwork folders, and then there could be a script that would easily change all of the entries to point to a different piece of artwork (ie: displaying action shots for all of the games instead of Boxart).

I've added the --image option. Then you can do:

./Used2BeTXT --no-desc --image 3dbox /path/to/atari2600/synopsis/*.*

And the generated gamelist.xml will have the <image> pointing to the 3D Box art.

Edit: Run --help to see the --image details.

It's not fast if you just wanna change this field on an existing gamelist.xml. But if you just wanna do that, you don't need a script. You just need a basic knowledge about sed command. I'll leave this task with you. ;-)

Used2BeRX

@meleu

The script was only trying to find .zip. I've made it search for .bin too. Do you have other extensions?

This won't increase search time, will it? All consoles have .zip except for .bin in 5200. I haven't tried any CD consoles yet, but .bin will probably be an issue there. I'm pretty sure that with SegaCD and TurboGrafxCD it will need to look for .cue and actually NOT look for .bin

The script puts every entry on its own platform gamelist.xml. Do they use the same emulator in RetroPie? Anyway, if you want to see atari 800 games in the atari 5200 you'll have to combine manually, as you said.

Yeah. They both use the same emulator.

You can give everything in only one command line. Like this:

./Used2BeTXT /path/to/atari2600/synopsis/*.* /path/to/nes/synopsis/*.* /path/to/megadrive/synopsis/*.*

Is it possible for me to write the command once and save it so I can just type a quick command to run it?

I've added the --image option. Then you can do:

./Used2BeTXT --no-desc --image 3dbox /path/to/atari2600/synopsis/*.*

What does --image alone do? Does it add all of the image file fields, but default to Box Front? And then it would change it to 3D Box art or other types if I put the second command after it?

It's not fast if you just wanna change this field on an existing gamelist.xml. But if you just wanna do that, you don't need a script. You just need a basic knowledge about sed command. I'll leave this task with you. ;-)

The reason I wanted the --image option to add all 5 types of artwork is because before I had any help writing scripts I did this switch between artwork types manually myself with a few search and replace commands in Notepad++.

I was hoping that if the --image option wrote all 5 fields for all 5 pieces of artwork, that a quick script could be written that would automatically do this. For instance, if you wanted to show Title shots, but it was currently showing Box Front, the script would first replace all <image></image> lines to be <boxfront></boxfront> and then replace all <title></title> lines to be <image></image>.

But... It would have to do this ONLY in <game> entries. The script would break the <folder> if it changed the <image> fields for the folders.

Used2BeRX

@meleu BTW dude... I've been learning some cool stuff in a great photo manipulation program called paint.net. Never did any graphic arts before and I always thought Photoshop was too complicated to bother learning...

Check these folder images out. :)

ATARI 5200 PROTOTYPES / HOMEBREW
alt text

US LICENSED GAME GEAR:
alt text

EUROPE LICENSED GAME GEAR:
alt text

JAPAN LICENSED GAME GEAR:
alt text

Used2BeRX

@meleu Yes... I definitely think we need a way to simplify a command that makes a gamelist.xml for all of the systems. I just started the script for all included systems so far and this is what it looked like:

time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/atari2600/Synopsis/*.* /home/pi/RetroPie/Media/atari5200/Synopsis/*.* /home/pi/RetroPie/Media/atari7800/Synopsis/*.* /home/pi/RetroPie/Media/atarilynx/Synopsis/*.* /home/pi/RetroPie/Media/gamegear/Synopsis/*.* /home/pi/RetroPie/Media/gb/Synopsis/*.* /home/pi/RetroPie/Media/gba/Synopsis/*.* /home/pi/RetroPie/Media/gbc/Synopsis/*.* /home/pi/RetroPie/Media/mastersystem/Synopsis/*.* /home/pi/RetroPie/Media/megadrive/Synopsis/*.* /home/pi/RetroPie/Media/nes/Synopsis/*.* /home/pi/RetroPie/Media/ngp/Synopsis/*.* /home/pi/RetroPie/Media/pcengine/Synopsis/*.* /home/pi/RetroPie/Media/sega32x/Synopsis/*.* /home/pi/RetroPie/Media/sg-1000/Synopsis/*.* /home/pi/RetroPie/Media/snes/Synopsis/*.* /home/pi/RetroPie/Media/videopac/Synopsis/*.* /home/pi/RetroPie/Media/virtualboy/Synopsis/*.* /home/pi/RetroPie/Media/wonderswan/Synopsis/*.*

Not fun.... :(

Anyway....

I completed editing EVERY folder synopsis file, so once this finishes running I should be able to compile a full list of any problems with the script for every system except for the CD based systems.

I'll let you know more as soon as I know.

Thanks again.

meleu

@Used2BeRX

I've just noticed that there's a smallbug in the code when some metadata has & and you re-run the script with the same synopsis. I'm trying to fix it now. In the first run (creating a new entry in the gamelist.xml) this bug doesn't happen.

This won't increase search time, will it?

Maybe some miliseconds. But the script must be able to find those files, doesn't it? :-)

I haven't tried any CD consoles yet, but .bin will probably be an issue there. I'm pretty sure that with SegaCD and TurboGrafxCD it will need to look for .cue and actually NOT look for .bin

OK, I've made the script search for .cue if Media: CD, no matter what platform it is.

Is it possible for me to write the command once and save it so I can just type a quick command to run it?

Yes, you can create an alias. But if you prefer to create a list in a file, I can show you some tricks on how to use this file (nothing to be changed in the script).

What does --image alone do?

gives an error message and exit.

I'm planning add more options for <image> soon.

meleu

@Used2BeRX

@meleu Yes... I definitely think we need a way to simplify a command that makes a gamelist.xml for all of the systems.

I definitely think you need to invest some time to learn the basics of Linux shell. :-)

That huge command can be simplified to:

time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/*/Synopsis/*.*

Also, don't know if you are aware, but you can auto-complete some commands using TAB key.

I've updated the script a few minutes ago. Small bugfixes and added the --only-image option (see --help to get details).

It was fun, I learned some find and xmlstarlet tricks, but I think I'm done with it. Time to move on...

Used2BeRX

@meleu

I've just noticed that there's a smallbug in the code when some metadata has & and you re-run the script with the same synopsis. I'm trying to fix it now. In the first run (creating a new entry in the gamelist.xml) this bug doesn't happen.

No worries. I'm creating all new gamelist.xml files right now.

Maybe some miliseconds. But the script must be able to find those files, doesn't it? :-)

Milliseconds on your PC probably adds a lot on the Pi Zero. I don't have a time for just Atari 2600 since I was doing all of them, but I think it's taking 3-4 seconds each entry compared to the 2 seconds it was taking before. Maybe a separate script for the Atari 5200 is in order?

OK, I've made the script search for .cue if Media: CD, no matter what platform it is.

Nice... I won't be able to test this until I have a Pi 3 though, unfortunately :(

Yes, you can create an alias. But if you prefer to create a list in a file, I can show you some tricks on how to use this file (nothing to be changed in the script).

I'd be interesting in learning more about this. I gave you the full command above... what would I do with that to run it without typing 1,000 characters next time?

I'm planning add more options for <image> soon.

Sweet.

The script is still running. I started it over 5 hours ago and right now it's on the megadrive. Still got quite a lot to go through with some smaller systems as well as the TurboGrafx and SNES.