Making premade gamelist.xml with xtra media collections

Used2BeRX

@meleu Got it. Will re-run it now. BTW.... I checked the entry for "Adventure (New Graphics)" which previously didn't work and it works now! :)

by the way, if you want to interrupt the script in execution, hit ctrl+c.

I accidentally figured that out when I stopped the script 10 minutes ago for your new update. Sometimes I get lucky and Linux behaves like DOS did. ;)

EDIT: Start time was 3:42 by me. I'll try to gauge when it will finish so I can get an exact time to see how long it takes to do 745 entries with "--no-desc"

meleu

@Used2BeRX

EDIT: Start time was 3:42 by me. I'll try to gauge when it will finish so I can get an exact time to see how long it takes to do 745 entries with "--no-desc"

Here's a tip to measure this kind of thing: Use time before a command.

Examples:

with --no-desc

meleu@mlenovo ~/src/Used2BeTXT $ time ./Used2BeTXT.sh --no-desc ../XBox\ Synopsis\ Files/GENESIS/synopsis/MEGA\ MAN\ -\ THE\ WILY\ WARS.TXT 
"../XBox Synopsis Files/GENESIS/synopsis/MEGA MAN - THE WILY WARS.TXT" data has been added to "megadrive_gamelist.xml".

real    0m0.242s
user    0m0.124s
sys     0m0.084s

default use (no options)

meleu@mlenovo ~/src/Used2BeTXT $ time ./Used2BeTXT.sh ../XBox\ Synopsis\ Files/GENESIS/synopsis/MEGA\ MAN\ -\ THE\ WILY\ WARS.TXT
"../XBox Synopsis Files/GENESIS/synopsis/MEGA MAN - THE WILY WARS.TXT" data has been added to "megadrive_gamelist.xml".

real    0m0.257s
user    0m0.148s
sys     0m0.072s

with --full

meleu@mlenovo ~/src/Used2BeTXT $ time ./Used2BeTXT.sh --full ../XBox\ Synopsis\ Files/GENESIS/synopsis/MEGA\ MAN\ -\ THE\ WILY\ WARS.TXT 
"../XBox Synopsis Files/GENESIS/synopsis/MEGA MAN - THE WILY WARS.TXT" data has been added to "megadrive_FULL_gamelist.xml".

real    0m1.200s
user    0m0.664s
sys     0m0.312s

If you are curious about the meaning of real, user and sys, this is a good reading: https://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1

Used2BeRX

@meleu Good to know. Thanks. I will use that going forward.

Right now it's 4:49, so we're at 1 hour and 7 minutes and we're only at "Miner 2049er", which is number 359 out of 745.

I didn't time it last time, but I think it took just over 3 hours to make the list. The way this is going now, I think it will be about 2 1/2 hours. Looks like changing the way it searches for files like you did today may have cut down the time about 15-20%, but I can't be sure. Some of this time savings might also be because it's not doing the <desc>

I'm thinking that if you can write the code so it looks for specific media in specific folders instead of all of the folders that we could put a serious dent in the time it takes to create these lists.

I didn't check all of them so far, but a cursory "find" on the document shows that every entry that I looked at that wasn't working before is working now. I will let you know what our percentage of good files was when the list completes and I can scroll through the games in the romlist in EmulationStation.

Also, the file is extremely smaller than it was with the description like I wanted it to be. I'm hoping this helps with a Pi Zero. The file I ran yesterday was over 1.2MB when it was finished. With the file nearly halfway done right now, the file is only 168kb. I think that "--no-desc" option is going to work out great for the Pi Zero when coupled with a skin that has no metadata like "Carbon No-Meta"

meleu

@Used2BeRX I was working on the folder path finding and then the kids demanded my atention. I uploaded what I've done so far but didn't make enough tests. If you wanna test, give it a try. There's no need to parse every synopsis file again, only the folders ones. The script updates the file accordingly.

Now I have to play with kids! :-)

Used2BeRX

@meleu So you're saying that even though the folder entries are already in the gamelist.xml it will automatically make the updates to them? If that's the case, that's sweet. :)

When this is done I'm going to verify which games if any still have problems and let you know how close to 100% we came, then I'll re-run the script to see if we get the folders working too.

Have fun with the kids. :)

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

So you're saying that even though the folder entries are already in the gamelist.xml it will automatically make the updates to them?

The script updates an existing .xml file instead of creating a new one since the first version. I did this way for a case you get a new synopsis.txt for a game, there's no need to parse every single file again. Just run the script with the new synopsis file and the metadata will be added to the existing gamelist.xml.

Used2BeRX

@meleu Nice.

Does it alter the existing entry when you do this, or does it create a second entry?

If I run the same command with the *.* will it take a few hours to complete again, or will it skip all of the entries that are complete with the given parameter of --no-desc?

Used2BeRX

@meleu Final results of second run (before running again with folder find code):
Start Time: 3:42, End Time: 5:56
Total Time: 2 hours and 14 minutes
Previous Errors: 108 out of 745
Current Errors: 5 errors out of 745

List of errors:

(2_2) Prototypes
DUKES OF HAZZARD [STUNT CYCLE VERSION] / Dukes of Hazzard (Stunt Cycle Version)
ROCKY & BULLWINKLE / Rocky &amp; Bullwinkle
(2_3) Hacks
TARG (ALT) / NO ENTRY IN GAMELIST.XML (both synopsis have same description of "Targ"
(3) Adult Games
TERRANCE & PHILLIP / Terrance &amp; Phillip: Asses of Fire
TERRANCE & PHILLIP 2 / Terrance &amp; Phillip 2: Not Without My Anus

TARG (ALT): This is an easy fix. I will just have to make sure that every synopsis has a different first line entry. Both "TARG.TXT" AND "TARG (ALT).TXT" had just "Targ" for the first line. No changes necessary for these on the script needed.

ROCKY & BULLWINKLE, TERRANCE & PHILLIP, TERRANCE & PHILLIP 2:

"Phantom II & Pirate" and "Track & Field" worked fine because there was no "&" in the file names ("PHANTOM II" and "TRACK AND FIELD").

The other three failed to find the files because there was an & in the rom/media/synopsis file name. Is it possible to have the program search for just "&" instead of the full &amp text that is required in the XML? I know that over other systems the use of the "&" is quite prevalent and changing those would require a LOT of file renaming otherwise.

DUKES OF HAZZARD [STUNT CYCLE VERSION]: Not sure about this one. Maybe there is a special character used for the "[]" like the "&" that is giving us problems?

I'm going to run the command with the update for the folders now and see what happens. This is exciting. :)

EDIT: 3rd run with folder code added

Folder code is not working. This may be because it's trying to use the first line name in the TXT file which is different than the actual folder name. Examples:

"(1_1) US.txt" / <name>(1) US: Licensed Atari 2600 Games</name>
"(1) Licensed.txt" / <name>(1) Licensed Atari 2600 Games</name>
"(1_5) Cassettes.txt" / <name>(5) Atari 2600 Cassettes</name>

(Unlike the roms/media names, I did not make the folders or the folder.txt files in all caps. They are exactly as you see them above. I do not know if this is possibly the problem. They aren't uppercase, but every txt file matches case perfectly with the folders).

I answered my first two questions as well. It looks like running *.* on the same file will take exactly as long as it took the first time. It does not look like it adds multiple entries. It simply overwrites any data onto the current file. If I were to manually run the command on each individual folder.txt after you fix it, I would be able to update this file much quicker than running the *.* command again.

meleu

@Used2BeRX

Is it possible to have the program search for just "&" instead of the full &amp text that is required in the XML?

Yes. Please don't rename anything from now on. The script must be smart to deal wit those problems. I'll take a look when I have a chance.

Good to know you've found the answers for your own questions. :-)

Used2BeRX

@meleu Understood. I won't edit anything that shouldn't be edited. In the case of Targ, I shouldn't have had them named the same anyhow since one was a hack that changed the game.

Right now I'm working on updating all of the information for the many, many, many sub folders and getting good images to display for them. I think my favorite image so far is the one I'm going to use for "(4) Requires Unsupported Controller"...

alt text

EDIT:

I did all the folder entries by hand just to see what this bad boy would look like, and it looks amazing now in the Carbon No-Meta skin!

I'm going to wait for the program to do everything right before I do any of the other systems, but I couldn't wait to see what the final product would look like for a system so I spent about 45 minutes doing what your program is going to do in about 45 seconds lol...

I'm going to put here every single folder entry that I have that works so you can get a good idea of what they should look like when you're altering the script to do the folders.

  <folder>
    <name>(1) US: Licensed Atari 2600 Games</name>
    <path>/home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_1) US/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(1_1) US.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(2) Europe: NTSC Conversions</name>
    <path>/home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_2) Europe (NTSC Conversions)/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(1_2) Europe (NTSC Conversions).png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(3) Europe: Non-NTSC Conversions</name>
    <path>/home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_3) Europe (Non-NTSC Conversions)/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(1_3) Europe (Non-NTSC Conversions).png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(4) Brazil: Licensed Atari 2600 Games</name>
    <path>/home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_4) Brazil/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(1_4) Brazil.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(5) Atari 2600 Cassettes</name>
    <path>/home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_5) Cassettes</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(1_5) Cassettes.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(6) Atari 2600 Light Gun Games</name>
    <path>/home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_6) Light Gun</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(1_6) Light Gun.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(7) Atari 2600 Special Releases</name>
    <path>/home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_7) Special Releases</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(1_7) Special Releases.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(1) Licensed Atari 2600 Games</name>
    <path>/home/pi/RetroPie/roms/atari2600/(1) Licensed/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(1) Licensed.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(1) Atari 2600 Homebrew Games</name>
    <path>/home/pi/RetroPie/roms/atari2600/(2) Homebrew - Prototypes/(2_1) Homebrew</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(2_1) Homebrew.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(2) Atari 2600 Prototypes</name>
    <path>/home/pi/RetroPie/roms/atari2600/(2) Homebrew - Prototypes/(2_2) Prototypes</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(2_2) Prototypes.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(3) Atari 2600 Game Hacks</name>
    <path>/home/pi/RetroPie/roms/atari2600/(2) Homebrew - Prototypes/(2_3) Hacks</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(2_3) Hacks.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(2) Atari 2600 Homebrew &amp; Prototypes</name>
    <path>/home/pi/RetroPie/roms/atari2600/(2) Homebrew - Prototypes</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(2) Homebrew - Prototypes.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(3) Adult Games</name>
    <path>/home/pi/RetroPie/roms/atari2600/(3) Adult Games</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(3) Adult Games.jpg</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(1) Androman Controller</name>
    <path>/home/pi/RetroPie/roms/atari2600/(4) Requires Unsupported Controller/(4_1) Androman Controller/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(4_1) Androman Controller.jpg</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(2) Atari Keypad</name>
    <path>/home/pi/RetroPie/roms/atari2600/(4) Requires Unsupported Controller/(4_2) Atari Keypad/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(4_2) Atari Keypad.jpg</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(3) Foot Craz</name>
    <path>/home/pi/RetroPie/roms/atari2600/(4) Requires Unsupported Controller/(4_3) Foot Craz/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(4_3) Foot Craz.jpg</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(4) Kid Vid</name>
    <path>/home/pi/RetroPie/roms/atari2600/(4) Requires Unsupported Controller/(4_4) Kid Vid/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(4_4) Kid Vid.jpg</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(5) Kid's Controller</name>
    <path>/home/pi/RetroPie/roms/atari2600/(4) Requires Unsupported Controller/(4_5) Kid's Controller/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(4_5) Kid's Controller.jpg</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(6) Mindlink Controller</name>
    <path>/home/pi/RetroPie/roms/atari2600/(4) Requires Unsupported Controller/(4_6) Mindlink Controller/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(4_6) Mindlink Controller.jpg</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(4) Requires Unsupported Controller</name>
    <path>/home/pi/RetroPie/roms/atari2600/(4) Requires Unsupported Controller/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(4) Requires Unsupported Controller.jpg</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>
  <folder>
    <name>(5) Non-Working Games</name>
    <path>/home/pi/RetroPie/roms/atari2600/(5) Non-Working/</path>
    <image>/home/pi/RetroPie/Media/atari2600/Artwork/Folders/(5) Non-Working.png</image>
    <video/>
    <marquee/>
    <desc/>
    <releasedate>19770101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>

Hope that helps!!!

EDIT 2:
I just learned some art stuff with paint.net. Took some transparent images and combined them for the (2) Homebrew - Prototypes folder.

alt text

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I'm going to put here every single folder entry that I have that works so you can get a good idea of what they should look like

Hmm... Then for folders things are different.

The synopsis.txt file name with no extension is the <name>.

The Folder: field is the name of the folder that will have the full path in <path>.

Am I right?

Used2BeRX

@meleu

The <name> is the first line of the folder.txt file.

<name>(1) US: Licensed Atari 2600 Games</name>

The <path> is the Folder: field in the folder.txt file.

<path>/home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_1) US/</path>

/home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_1) US/

The highlighted part in the line above is the Folder: field as well as the name of the folder.txt file without the .txt extension.

Here's the full entry for "(1_1) US.txt"

(1) US: Licensed Atari 2600 Games
Folder: (1_1) US
Platform: Atari 2600
Release Year: 1977
Developer: Atari Inc.
Publisher: Atari Inc.
_________________________

LICENSED ATARI 2600 GAMES
=====================================

The Atari 2600 (or Atari VCS before 1982) is a home video game console by Atari, Inc. Released on September 11, 1977, it is credited with popularizing the use of microprocessor-based hardware and ROM cartridges containing game code, a format first used with the Fairchild Channel F video game console in 1976. This format contrasts with the older model of having non-microprocessor dedicated hardware, which could only play the games that were physically built into the unit.

The console was originally sold as the Atari VCS, an abbreviation for Video Computer System. Following the release of the Atari 5200 in 1982, the VCS was renamed to the "Atari 2600", after the unit's Atari part number, CX2600. The 2600 was typically bundled with two joystick controllers, a conjoined pair of paddle controllers, and a game cartridge: initially Combat, and later Pac-Man.

So the two important lines are going to be <path> and <image>

Synopsis Location: /home/pi/RetroPie/Media/atari2600/Synopsis/(1_1) US.txt

Image Location: /home/pi/RetroPie/Media/atari2600/Artwork/Folders/(1_1) US.png

Folder Location: /home/pi/RetroPie/Roms/atari2600/(1) Licensed/(1_1) US

findrbot_admin

You could easily hack a script to do all that in ShellScript, Python, or even PHP... (EDIT: My bad, you already did! Hacking code is fun, especially shellscript!)

I am working on a shellscript right now that will copy all the ROM files from the subdirectories and deletes all the garbage... This way I can download a butt-tonne of roms, put them all in the console folders, and automatically move them to the top level dir...

So basically it moves this:
/roms/n64/Super Mario 64/US/ROM/Folder1/Folder2/Folder3/Folder4/Folder5/Folder6/Folder7/Folder8/Super Mario 64 (U).n64
To this:
/roms/n64/Super Mario 64 (U).n64

And it removes any file that is not a rom - all the images, txt files, exe files, etc. because their is no point in adding those files into a ROM pack as they just weigh it down and that's what the scraper is for... I had over 100GB of ROM's downloaded from the internet, and after running my script to get rid of all the garbage it was almost 50 GB... I will post accurate stats after its fully coded and tested...

The script runs right on the RasPi or you can run it on your computer to clean up your ROM download folder. I will post a link to my script/website (http://wiki.findrbot.com/) when it is complete, I am still adding features... There are too many ideas of things to add! I coded the entire script over a day or two, just putting final touches in... I am also releasing mega ROM packs on my website, but those are shunned here so I won't link to them ;)

I was going to make a scraper, but there is no point... Just use SSelph's scraper (google it) right on the RasPi to get all the game information and boxart. It only makes sense to offload most of the downloading of non-essentials to the person doing the ROM downloading and the gamesdb websites... Less load on my server!

There really is also no point in making a gameslist.xml generator either, because you can just run SSelph's scraper, which will not only generate the gameslist.xml, it will also download the boxart and descriptions :)

You could make one anyways, for the 'function' of it (lol)

Cheers :)

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu Final results of second run (before running again with folder find code):
Start Time: 3:42, End Time: 5:56
Total Time: 2 hours and 14 minutes
Previous Errors: 108 out of 745
Current Errors: 5 errors out of 745

meleu

@Used2BeRX hey man, --update the script and run it again. I've fixed those 5 errors and the folder synopsis is working (at least on my quick tests).

Note: the changes I've made in the code ignores (doesn't update) those 5 old entries with problems. It creates new fine and working entries.

Used2BeRX

@meleu lol... nice pic.

I updated it and it's running now. Looks like it takes about 30-60 seconds per entry now, so I'm guessing it's going to take about 8 hours to make the file this time, but I checked the file so far and all of the folders are being entered right now. :)

I put the "time" operator so I'll have an exact time for you when it's done. Thanks again.

meleu

@Used2BeRX

Looks like it takes about 30-60 seconds per entry now

Wow! :( It's a bad news...

Well at least I have a good news: I've just added the --only-new option, so if you just wanna add new entries and ignore the existing ones, use this option (check the --help message I've updated accordingly).
(edit: I've uploaded a few minutes ago, probably you don't have this version)

Example of use case:

You're starting a new gamelist.xml and the script is running.
For some reason you have to turn off your raspi and the script didn't process all files yet.
Hit ctrl+c and turn off your raspi.
When you have a chance to run the script again use the --only-new option and it will ignore the synopsis.txt of all the existing entries in the gamelist.xml.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

Looks like it takes about 30-60 seconds per entry now

Please, do the following:

cancel the execution (ctrl+c),
remove the generated gamelist.xml,
--update the script,
run it again with the same parameters you used (including the time operator).
let me know if it takes more than 30 seconds per entry.

IMO 30 seconds per entry is unacceptable. OK, it's faster than human, but too slow for a computer (even a raspiZero). I have to find the bottleneck!

EDIT:
By the way, the script took almost 4 minutes to process 718 synopsis for atari2600 games on my Core i7 computer. OK, it has much more CPU power than a raspiZero, but 8 hours?! Try it again, please.

Used2BeRX

@meleu Sorry. I was away from the computer and didn't get your messages until now. I will upgrade and try it again.

FYI, I think I was right about 8 hours. As of writing this, it had only done 425 entries out of 745. That was in 274 minutes, real-time. (4:34)

Part of this was my fault for running the wrong command. I was running it with the --full option. I tried that again with --full in your new script and timed it and it was taking 40-41 seconds per entry.

I stopped it, deleted the output, and re-ran it with --no-desc and timed it at 10-11 seconds per entry, so I'm letting it run now. This is probably going to end up being just a little longer than the 2 hours and 12 minutes before you added the code to find the folders.

I'd still like to see if there are ways of tweaking this. I won't be using the --full option for my raspi zero setup, but I will be using it for the Pi 3 setup. Granted that is a faster processor itself, but this just seems to be taking a lot longer than it should to get the job done.

Just curious, are you doing the following things?

Searching for roms only in the following folder and sub-folders: /opt/pi/RetroPie/roms/{system_name}/ (system_name from the matching one used in the command running the script, ie: /opt/pi/RetroPie/Media/atari2600/Synopsis/*.*)
Searching for folders and sub-folders only in the same folder and sub-folders as number one above. And ONLY looking for folder names, completely disregarding any other files.
Searching for Media only in the following folder and sub-folders: /opt/pi/RetroPie/Media/{system_name}/ (You could get the system_name from the command running the script like above).
You could further narrow this down by pointing particular media fields to their exact folders, like <gamefaqs/> to /Media/{system_name}/GameFAQs/ and <threedbox> to /Media/{system_name}/Artwork/3D Boxart/

When I get a chance after this is done I'll re-check everything in EmulationStation and see if there are any more individual problems with files/folders displaying. When we can speed up the code I will start running a few --full and default runs and see if all of the media is being done right.

meleu

@Used2BeRX

re-ran it with --no-desc and timed it at 10-11 seconds per entry (...). This is probably going to end up being just a little longer than the 2 hours and 12 minutes before you added the code to find the folders.

It's an unfair comparison. I noticed that creating a new gamelist.xml from scratch takes more time than just updating an existing one. Then I'm assuming the time didn't changed.

Just curious, are you doing the following things?

Searching for roms only in the following folder and sub-folders: /opt/pi/RetroPie/roms/{system_name}/ (system_name from the matching one used in the command running the script, ie: /opt/pi/RetroPie/Media/atari2600/Synopsis/*.*)

Yes. But the system_name is based on the Platform: field.

Example: If Platform: Atari 2600, then system_name=atari2600.

Searching for folders and sub-folders only in the same folder and sub-folders as number one above. And ONLY looking for folder names, completely disregarding any other files.

Yes.

Searching for Media only in the following folder and sub-folders: /opt/pi/RetroPie/Media/{system_name}/ (You could get the system_name from the command running the script like above).

Yes. But system_name is obtained as I described above.

You could further narrow this down by pointing particular media fields to their exact folders, like <gamefaqs/> to /Media/{system_name}/GameFAQs/ and <threedbox> to /Media/{system_name}/Artwork/3D Boxart/

Yes, I'm doing this.

I think I've made some progress in optimization...

Suggestion: to measure the optimizations, there's no need to wait all those 2+ hours, try to parse only the synopsis.txt files starting with A. The command should be like this:

time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/atari2600/Synoipsis/A*.*

Got it?

For a fair comparison do the following:

do NOT --update your script for now.
remove the atari2600_gamelist.xml file.
run the command above (only parse synopsis.txt files starting with A).
take note of the times (real, user and sys).
remove the atari2600_gamelist.xml file.
--update the script.
run the command above (only parse synopsis.txt files starting with A).
take note of the times (real, user and sys).
post the times here.

Looking forward for the results...