Making premade gamelist.xml with xtra media collections

meleu

Looks like it takes about 30-60 seconds per entry now

Wow! :( It's a bad news...

Well at least I have a good news: I've just added the --only-new option, so if you just wanna add new entries and ignore the existing ones, use this option (check the --help message I've updated accordingly).
(edit: I've uploaded a few minutes ago, probably you don't have this version)

Example of use case:

You're starting a new gamelist.xml and the script is running.
For some reason you have to turn off your raspi and the script didn't process all files yet.
Hit ctrl+c and turn off your raspi.
When you have a chance to run the script again use the --only-new option and it will ignore the synopsis.txt of all the existing entries in the gamelist.xml.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

Looks like it takes about 30-60 seconds per entry now

Please, do the following:

cancel the execution (ctrl+c),
remove the generated gamelist.xml,
--update the script,
run it again with the same parameters you used (including the time operator).
let me know if it takes more than 30 seconds per entry.

IMO 30 seconds per entry is unacceptable. OK, it's faster than human, but too slow for a computer (even a raspiZero). I have to find the bottleneck!

EDIT:
By the way, the script took almost 4 minutes to process 718 synopsis for atari2600 games on my Core i7 computer. OK, it has much more CPU power than a raspiZero, but 8 hours?! Try it again, please.

Used2BeRX

@meleu Sorry. I was away from the computer and didn't get your messages until now. I will upgrade and try it again.

FYI, I think I was right about 8 hours. As of writing this, it had only done 425 entries out of 745. That was in 274 minutes, real-time. (4:34)

Part of this was my fault for running the wrong command. I was running it with the --full option. I tried that again with --full in your new script and timed it and it was taking 40-41 seconds per entry.

I stopped it, deleted the output, and re-ran it with --no-desc and timed it at 10-11 seconds per entry, so I'm letting it run now. This is probably going to end up being just a little longer than the 2 hours and 12 minutes before you added the code to find the folders.

I'd still like to see if there are ways of tweaking this. I won't be using the --full option for my raspi zero setup, but I will be using it for the Pi 3 setup. Granted that is a faster processor itself, but this just seems to be taking a lot longer than it should to get the job done.

Just curious, are you doing the following things?

Searching for roms only in the following folder and sub-folders: /opt/pi/RetroPie/roms/{system_name}/ (system_name from the matching one used in the command running the script, ie: /opt/pi/RetroPie/Media/atari2600/Synopsis/*.*)
Searching for folders and sub-folders only in the same folder and sub-folders as number one above. And ONLY looking for folder names, completely disregarding any other files.
Searching for Media only in the following folder and sub-folders: /opt/pi/RetroPie/Media/{system_name}/ (You could get the system_name from the command running the script like above).
You could further narrow this down by pointing particular media fields to their exact folders, like <gamefaqs/> to /Media/{system_name}/GameFAQs/ and <threedbox> to /Media/{system_name}/Artwork/3D Boxart/

When I get a chance after this is done I'll re-check everything in EmulationStation and see if there are any more individual problems with files/folders displaying. When we can speed up the code I will start running a few --full and default runs and see if all of the media is being done right.

meleu

@Used2BeRX

re-ran it with --no-desc and timed it at 10-11 seconds per entry (...). This is probably going to end up being just a little longer than the 2 hours and 12 minutes before you added the code to find the folders.

It's an unfair comparison. I noticed that creating a new gamelist.xml from scratch takes more time than just updating an existing one. Then I'm assuming the time didn't changed.

Just curious, are you doing the following things?

Searching for roms only in the following folder and sub-folders: /opt/pi/RetroPie/roms/{system_name}/ (system_name from the matching one used in the command running the script, ie: /opt/pi/RetroPie/Media/atari2600/Synopsis/*.*)

Yes. But the system_name is based on the Platform: field.

Example: If Platform: Atari 2600, then system_name=atari2600.

Searching for folders and sub-folders only in the same folder and sub-folders as number one above. And ONLY looking for folder names, completely disregarding any other files.

Yes.

Searching for Media only in the following folder and sub-folders: /opt/pi/RetroPie/Media/{system_name}/ (You could get the system_name from the command running the script like above).

Yes. But system_name is obtained as I described above.

You could further narrow this down by pointing particular media fields to their exact folders, like <gamefaqs/> to /Media/{system_name}/GameFAQs/ and <threedbox> to /Media/{system_name}/Artwork/3D Boxart/

Yes, I'm doing this.

I think I've made some progress in optimization...

Suggestion: to measure the optimizations, there's no need to wait all those 2+ hours, try to parse only the synopsis.txt files starting with A. The command should be like this:

time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/atari2600/Synoipsis/A*.*

Got it?

For a fair comparison do the following:

do NOT --update your script for now.
remove the atari2600_gamelist.xml file.
run the command above (only parse synopsis.txt files starting with A).
take note of the times (real, user and sys).
remove the atari2600_gamelist.xml file.
--update the script.
run the command above (only parse synopsis.txt files starting with A).
take note of the times (real, user and sys).
post the times here.

Looking forward for the results...

Used2BeRX

@meleu Alright. I'm going to do what you asked. I just looked at the time for the last run at it was 136m/102m/28m.

I'll let you know more when I know more.

Used2BeRX

@meleu Wow.... what did you just do?
Old times: 6m4s/4m34s/1m13s
New times: 1m5s/0m33s/0m19s

You just cut the time down to about 20% of what it was before!

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu Wow.... what did you just do?

Tweaks on find command.

Now let's see the time for the complete atari2600 set! :-)

EDIT
And check if the gamelist.xml is fine, please.

Used2BeRX

@meleu Just finished running the full --no-desc
30m12s, 17m58s, 8m10s

I'm going to check to see if there were any problems with it now.

Used2BeRX

@meleu 100% hits for boxart so that means that it worked perfect for path and image :)

Had to do a long overdue reboot so it took a while for me to get back up but I did a few others while I was waiting.

Atari 7800: 3m44s, 2m11s, 0m57s
Atari Lynx: 3m25s, 1m56s, 0m54s
Atari 5200: 19m17s, 12m43s, 4m51s

CURRENT ISSUES:

None of the `<path/> were written for Atari 5200/800. This might be because it's a special case. Of all of the systems I have for the RetroPie, this is the only one that you can't ZIP the files. All 5200/800 images I have are in BIN format.
Folders for other systems are being added to "atari2600_gamelist.xml". (I know why this is happening. All of my folders.txt files still say the information for the Atari 2600).
With the Atari 5200/800, I'd like for them to all write to atari5200_gamelist.xml. I don't include a lot of 800 games in my collection and rather than have two entries for it in the system I have all of the roms and media in the 5200 file structure, so I was wondering if we could maybe do somthing about putting those two scripts together. If that's too much of a hassle, no worries. I can just manually combine them when I'm done.

I went through both Atari 7800 and Atari LYNX and we have 100% boxart showing up for both of them as well!!!

As soon as I get the folder synopsis text files updated the folders should work perfect the next time I run the script.

Looking great though.

Now that we got the speed issue handled, I was wondering if we could talk about a few more options to add to the script?

I had a few ideas, such as creating all of the gamelist.xmls for all existing systems one after another without having to be there to put in the next command.(or from a list I could create in a file).

I was also thinking that we could do a --no-desc but with all of the Artwork folders, and then there could be a script that would easily change all of the entries to point to a different piece of artwork (ie: displaying action shots for all of the games instead of Boxart).

meleu

@Used2BeRX

CURRENT ISSUES:

None of the `<path/> were written for Atari 5200/800. This might be because it's a special case. Of all of the systems I have for the RetroPie, this is the only one that you can't ZIP the files. All 5200/800 images I have are in BIN format.

Yep. The script was only trying to find .zip. I've made it search for .bin too. Do you have other extensions?

Folders for other systems are being added to "atari2600_gamelist.xml". (I know why this is happening. All of my folders.txt files still say the information for the Atari 2600).

Exactly. The synopsis.txt files must be accurate. In this case, the problem is in the Platform: field.

With the Atari 5200/800, I'd like for them to all write to atari5200_gamelist.xml. I don't include a lot of 800 games in my collection and rather than have two entries for it in the system I have all of the roms and media in the 5200 file structure, so I was wondering if we could maybe do somthing about putting those two scripts together. If that's too much of a hassle, no worries. I can just manually combine them when I'm done.

The script puts every entry on its own platform gamelist.xml. Do they use the same emulator in RetroPie? Anyway, if you want to see atari 800 games in the atari 5200 you'll have to combine manually, as you said.

I had a few ideas, such as creating all of the gamelist.xmls for all existing systems one after another without having to be there to put in the next command.(or from a list I could create in a file).

You can give everything in only one command line. Like this:

./Used2BeTXT /path/to/atari2600/synopsis/*.* /path/to/nes/synopsis/*.* /path/to/megadrive/synopsis/*.*

I was also thinking that we could do a --no-desc but with all of the Artwork folders, and then there could be a script that would easily change all of the entries to point to a different piece of artwork (ie: displaying action shots for all of the games instead of Boxart).

I've added the --image option. Then you can do:

./Used2BeTXT --no-desc --image 3dbox /path/to/atari2600/synopsis/*.*

And the generated gamelist.xml will have the <image> pointing to the 3D Box art.

Edit: Run --help to see the --image details.

It's not fast if you just wanna change this field on an existing gamelist.xml. But if you just wanna do that, you don't need a script. You just need a basic knowledge about sed command. I'll leave this task with you. ;-)

Used2BeRX

@meleu

The script was only trying to find .zip. I've made it search for .bin too. Do you have other extensions?

This won't increase search time, will it? All consoles have .zip except for .bin in 5200. I haven't tried any CD consoles yet, but .bin will probably be an issue there. I'm pretty sure that with SegaCD and TurboGrafxCD it will need to look for .cue and actually NOT look for .bin

The script puts every entry on its own platform gamelist.xml. Do they use the same emulator in RetroPie? Anyway, if you want to see atari 800 games in the atari 5200 you'll have to combine manually, as you said.

Yeah. They both use the same emulator.

You can give everything in only one command line. Like this:

./Used2BeTXT /path/to/atari2600/synopsis/*.* /path/to/nes/synopsis/*.* /path/to/megadrive/synopsis/*.*

Is it possible for me to write the command once and save it so I can just type a quick command to run it?

I've added the --image option. Then you can do:

./Used2BeTXT --no-desc --image 3dbox /path/to/atari2600/synopsis/*.*

What does --image alone do? Does it add all of the image file fields, but default to Box Front? And then it would change it to 3D Box art or other types if I put the second command after it?

It's not fast if you just wanna change this field on an existing gamelist.xml. But if you just wanna do that, you don't need a script. You just need a basic knowledge about sed command. I'll leave this task with you. ;-)

The reason I wanted the --image option to add all 5 types of artwork is because before I had any help writing scripts I did this switch between artwork types manually myself with a few search and replace commands in Notepad++.

I was hoping that if the --image option wrote all 5 fields for all 5 pieces of artwork, that a quick script could be written that would automatically do this. For instance, if you wanted to show Title shots, but it was currently showing Box Front, the script would first replace all <image></image> lines to be <boxfront></boxfront> and then replace all <title></title> lines to be <image></image>.

But... It would have to do this ONLY in <game> entries. The script would break the <folder> if it changed the <image> fields for the folders.

Used2BeRX

@meleu BTW dude... I've been learning some cool stuff in a great photo manipulation program called paint.net. Never did any graphic arts before and I always thought Photoshop was too complicated to bother learning...

Check these folder images out. :)

ATARI 5200 PROTOTYPES / HOMEBREW
alt text

US LICENSED GAME GEAR:
alt text

EUROPE LICENSED GAME GEAR:
alt text

JAPAN LICENSED GAME GEAR:
alt text

Used2BeRX

@meleu Yes... I definitely think we need a way to simplify a command that makes a gamelist.xml for all of the systems. I just started the script for all included systems so far and this is what it looked like:

time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/atari2600/Synopsis/*.* /home/pi/RetroPie/Media/atari5200/Synopsis/*.* /home/pi/RetroPie/Media/atari7800/Synopsis/*.* /home/pi/RetroPie/Media/atarilynx/Synopsis/*.* /home/pi/RetroPie/Media/gamegear/Synopsis/*.* /home/pi/RetroPie/Media/gb/Synopsis/*.* /home/pi/RetroPie/Media/gba/Synopsis/*.* /home/pi/RetroPie/Media/gbc/Synopsis/*.* /home/pi/RetroPie/Media/mastersystem/Synopsis/*.* /home/pi/RetroPie/Media/megadrive/Synopsis/*.* /home/pi/RetroPie/Media/nes/Synopsis/*.* /home/pi/RetroPie/Media/ngp/Synopsis/*.* /home/pi/RetroPie/Media/pcengine/Synopsis/*.* /home/pi/RetroPie/Media/sega32x/Synopsis/*.* /home/pi/RetroPie/Media/sg-1000/Synopsis/*.* /home/pi/RetroPie/Media/snes/Synopsis/*.* /home/pi/RetroPie/Media/videopac/Synopsis/*.* /home/pi/RetroPie/Media/virtualboy/Synopsis/*.* /home/pi/RetroPie/Media/wonderswan/Synopsis/*.*

Not fun.... :(

Anyway....

I completed editing EVERY folder synopsis file, so once this finishes running I should be able to compile a full list of any problems with the script for every system except for the CD based systems.

I'll let you know more as soon as I know.

Thanks again.

meleu

@Used2BeRX

I've just noticed that there's a smallbug in the code when some metadata has & and you re-run the script with the same synopsis. I'm trying to fix it now. In the first run (creating a new entry in the gamelist.xml) this bug doesn't happen.

This won't increase search time, will it?

Maybe some miliseconds. But the script must be able to find those files, doesn't it? :-)

I haven't tried any CD consoles yet, but .bin will probably be an issue there. I'm pretty sure that with SegaCD and TurboGrafxCD it will need to look for .cue and actually NOT look for .bin

OK, I've made the script search for .cue if Media: CD, no matter what platform it is.

Is it possible for me to write the command once and save it so I can just type a quick command to run it?

Yes, you can create an alias. But if you prefer to create a list in a file, I can show you some tricks on how to use this file (nothing to be changed in the script).

What does --image alone do?

gives an error message and exit.

I'm planning add more options for <image> soon.

meleu

@Used2BeRX

@meleu Yes... I definitely think we need a way to simplify a command that makes a gamelist.xml for all of the systems.

I definitely think you need to invest some time to learn the basics of Linux shell. :-)

That huge command can be simplified to:

time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/*/Synopsis/*.*

Also, don't know if you are aware, but you can auto-complete some commands using TAB key.

I've updated the script a few minutes ago. Small bugfixes and added the --only-image option (see --help to get details).

It was fun, I learned some find and xmlstarlet tricks, but I think I'm done with it. Time to move on...

Used2BeRX

@meleu

I've just noticed that there's a smallbug in the code when some metadata has & and you re-run the script with the same synopsis. I'm trying to fix it now. In the first run (creating a new entry in the gamelist.xml) this bug doesn't happen.

No worries. I'm creating all new gamelist.xml files right now.

Maybe some miliseconds. But the script must be able to find those files, doesn't it? :-)

Milliseconds on your PC probably adds a lot on the Pi Zero. I don't have a time for just Atari 2600 since I was doing all of them, but I think it's taking 3-4 seconds each entry compared to the 2 seconds it was taking before. Maybe a separate script for the Atari 5200 is in order?

OK, I've made the script search for .cue if Media: CD, no matter what platform it is.

Nice... I won't be able to test this until I have a Pi 3 though, unfortunately :(

Yes, you can create an alias. But if you prefer to create a list in a file, I can show you some tricks on how to use this file (nothing to be changed in the script).

I'd be interesting in learning more about this. I gave you the full command above... what would I do with that to run it without typing 1,000 characters next time?

I'm planning add more options for <image> soon.

Sweet.

The script is still running. I started it over 5 hours ago and right now it's on the megadrive. Still got quite a lot to go through with some smaller systems as well as the TurboGrafx and SNES.

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

@Used2BeRX

@meleu Yes... I definitely think we need a way to simplify a command that makes a gamelist.xml for all of the systems.

I definitely think you need to invest some time to learn the basics of Linux shell. :-)

That huge command can be simplified to:
time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/*/Synopsis/*.*
Also, don't know if you are aware, but you can auto-complete some commands using TAB key.

I've updated the script a few minutes ago. Small bugfixes and added the --only-image option (see --help to get details).

It was fun, I learned some find and xmlstarlet tricks, but I think I'm done with it. Time to move on...

Ah... didn't notice you posted this before I replied.

Still have some possible issues that I mentioned though. I think we effectively doubled the search time by adding the BIN files to the search for the 5200.

Used2BeRX

@meleu I'm still making pics for the folders on about 3 more systems before I test this out, but the script finally finished!!! Just wanted to give you the time.

673m44s, 420m10s, 197m50s.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I think we effectively doubled the search time by adding the BIN files to the search for the 5200.

The searching for bin occurs only when platform is atari5200 or atari800.

Used2BeRX

@meleu 1st run through is done and the results are great overall.

100% HITS FOR THE FOLLOWING SYSTEMS!!!:
Atari 2600
Atari 7800
Atari LYNX
Game Gear
Gameboy Advance
Gameboy Color
Neo Geo Pocket/Neo Geo Pocket Color
Sega 32x
Sega SG-1000
Wonderswan/Wonderswan Color

MAJOR FULL-SYSTEM PROBLEMS:

SNES did not work. 338kb gamelist.xml file was created, but not a single rom or image was entered in the fields. Instead of writing to "snes_gamelist.xml" it wrote to "-supernintendoentertainmentsystem_gamelist.xml".
Odyssey 2 did not work. It makes magnavoxodyssey2_gamelist.xml, but RetroPie is expecting videopac for the folder, so it didn't find any roms or media.
Virtual Boy didn't work. It makes gamelist.xml with no other identifiers. RetroPie is expecting virtualboy for the folder.

FIXES TO SYNOPIS I MADE:
Changed "Platform: " on the following.
All "Atari 800" to "Atari 5200"
All "Sega Net" to "Sega Genesis / Mega Drive"
All "The Famicom Disk System" to "Nintendo Entertainment System"

Big thing to note is that you cannot have two 1st line synopsis.txt descriptions in the same gamelist.xml. It confuses the script. So I made the following changes by system to get the rest of the working systems up to 100%

TurboGrafx:
(2) Translated Hu-Card Games
(2) Translated CD Games
Bonk 3: Bonk's Big Adventure (CD)
Altered Beast (CD)

NES:
Metroid (Famicom Disk System)
Clu Clu Land (Famicom Disk System)
Gyruss (Famicom Disk System)
Kid Icarus (Famicom Disk System)
Super Mario Bros (Famicom Disk System)
Super Mario Bros 2 (Famicom Disk System)
Vs Excite Bike (Famicom Disk System)
AD&D Dragons of the Flame (Translated)
Battleship (Unlicensed)
Contra (Translated)
Maniac Mansion [Uncensored]
Ms. Pac-Man (Unlicensed)
Rainbow Islands (Europe)
Fantasy Zone I (Japan)
Super Mario Bros. 2 (Japan)
Othello (Australia)
Airwolf (Translated)
Contra (Translated)
Squoon (Translated)
Super Mario Bros. 2 (Prototype)
"Adult Games" 2/3rds of them were "Famicom Disk System" and showed up after changing this to "Nintendo Entertainment System"

GAME BOY:
Bomberman GB (Japan)

GENESIS:
Dynamite Headdy (Japan)
Rings of Power: The Hands of Fate (Hack)

MASTER SYSTEM:
Pit Pot..... accidentially had it listed as Game Gear.
Assault City (Light Phaser Version)

ATARI 5200:
Centipede (Atari 800)
Choplifter! (Atari 800)
Crystal Castles (Atari 800)
Mario Bros. (Atari 800)
Qix (Atari 800)
Rally Speedway (Atari 800)