RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Making premade gamelist.xml with xtra media collections

    Scheduled Pinned Locked Moved Ideas and Development
    gamelist.xmlpremade scrapeeasy config
    448 Posts 16 Posters 153.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • meleuM
      meleu @Used2BeRX
      last edited by meleu

      @Used2BeRX said in Making premade gamelist.xml with xtra media collections:

      I'm going to put here every single folder entry that I have that works so you can get a good idea of what they should look like

      Hmm... Then for folders things are different.

      The synopsis.txt file name with no extension is the <name>.

      The Folder: field is the name of the folder that will have the full path in <path>.

      Am I right?

      • Useful topics
      • joystick-selection tool
      • rpie-art tool
      • achievements I made
      Used2BeRXU 1 Reply Last reply Reply Quote 0
      • Used2BeRXU
        Used2BeRX @meleu
        last edited by Used2BeRX

        @meleu

        The <name> is the first line of the folder.txt file.

        <name>(1) US: Licensed Atari 2600 Games</name>
        

        The <path> is the Folder: field in the folder.txt file.

        <path>/home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_1) US/</path>
        

        /home/pi/RetroPie/roms/atari2600/(1) Licensed/(1_1) US/

        The highlighted part in the line above is the Folder: field as well as the name of the folder.txt file without the .txt extension.

        Here's the full entry for "(1_1) US.txt"

        (1) US: Licensed Atari 2600 Games
        Folder: (1_1) US
        Platform: Atari 2600
        Release Year: 1977
        Developer: Atari Inc.
        Publisher: Atari Inc.
        _________________________
        
        LICENSED ATARI 2600 GAMES
        =====================================
        
        The Atari 2600 (or Atari VCS before 1982) is a home video game console by Atari, Inc. Released on September 11, 1977, it is credited with popularizing the use of microprocessor-based hardware and ROM cartridges containing game code, a format first used with the Fairchild Channel F video game console in 1976. This format contrasts with the older model of having non-microprocessor dedicated hardware, which could only play the games that were physically built into the unit.
        
        The console was originally sold as the Atari VCS, an abbreviation for Video Computer System. Following the release of the Atari 5200 in 1982, the VCS was renamed to the "Atari 2600", after the unit's Atari part number, CX2600. The 2600 was typically bundled with two joystick controllers, a conjoined pair of paddle controllers, and a game cartridge: initially Combat, and later Pac-Man.
        

        So the two important lines are going to be <path> and <image>

        Synopsis Location: /home/pi/RetroPie/Media/atari2600/Synopsis/(1_1) US.txt

        Image Location: /home/pi/RetroPie/Media/atari2600/Artwork/Folders/(1_1) US.png

        Folder Location: /home/pi/RetroPie/Roms/atari2600/(1) Licensed/(1_1) US

        meleuM 1 Reply Last reply Reply Quote 0
        • F
          findrbot_admin
          last edited by findrbot_admin

          You could easily hack a script to do all that in ShellScript, Python, or even PHP... (EDIT: My bad, you already did! Hacking code is fun, especially shellscript!)

          I am working on a shellscript right now that will copy all the ROM files from the subdirectories and deletes all the garbage... This way I can download a butt-tonne of roms, put them all in the console folders, and automatically move them to the top level dir...

          So basically it moves this:
          /roms/n64/Super Mario 64/US/ROM/Folder1/Folder2/Folder3/Folder4/Folder5/Folder6/Folder7/Folder8/Super Mario 64 (U).n64
          To this:
          /roms/n64/Super Mario 64 (U).n64

          And it removes any file that is not a rom - all the images, txt files, exe files, etc. because their is no point in adding those files into a ROM pack as they just weigh it down and that's what the scraper is for... I had over 100GB of ROM's downloaded from the internet, and after running my script to get rid of all the garbage it was almost 50 GB... I will post accurate stats after its fully coded and tested...

          The script runs right on the RasPi or you can run it on your computer to clean up your ROM download folder. I will post a link to my script/website (http://wiki.findrbot.com/) when it is complete, I am still adding features... There are too many ideas of things to add! I coded the entire script over a day or two, just putting final touches in... I am also releasing mega ROM packs on my website, but those are shunned here so I won't link to them ;)

          I was going to make a scraper, but there is no point... Just use SSelph's scraper (google it) right on the RasPi to get all the game information and boxart. It only makes sense to offload most of the downloading of non-essentials to the person doing the ROM downloading and the gamesdb websites... Less load on my server!

          There really is also no point in making a gameslist.xml generator either, because you can just run SSelph's scraper, which will not only generate the gameslist.xml, it will also download the boxart and descriptions :)

          You could make one anyways, for the 'function' of it (lol)

          Cheers :)

          1 Reply Last reply Reply Quote 0
          • meleuM
            meleu @Used2BeRX
            last edited by

            @Used2BeRX said in Making premade gamelist.xml with xtra media collections:

            @meleu Final results of second run (before running again with folder find code):
            Start Time: 3:42, End Time: 5:56
            Total Time: 2 hours and 14 minutes
            Previous Errors: 108 out of 745
            Current Errors: 5 errors out of 745

            0_1498133175559_used2berx.jpg

            • Useful topics
            • joystick-selection tool
            • rpie-art tool
            • achievements I made
            1 Reply Last reply Reply Quote 1
            • meleuM
              meleu @Used2BeRX
              last edited by

              @Used2BeRX hey man, --update the script and run it again. I've fixed those 5 errors and the folder synopsis is working (at least on my quick tests).

              Note: the changes I've made in the code ignores (doesn't update) those 5 old entries with problems. It creates new fine and working entries.

              • Useful topics
              • joystick-selection tool
              • rpie-art tool
              • achievements I made
              Used2BeRXU 1 Reply Last reply Reply Quote 0
              • Used2BeRXU
                Used2BeRX @meleu
                last edited by

                @meleu lol... nice pic.

                I updated it and it's running now. Looks like it takes about 30-60 seconds per entry now, so I'm guessing it's going to take about 8 hours to make the file this time, but I checked the file so far and all of the folders are being entered right now. :)

                I put the "time" operator so I'll have an exact time for you when it's done. Thanks again.

                meleuM 2 Replies Last reply Reply Quote 0
                • meleuM
                  meleu @Used2BeRX
                  last edited by meleu

                  @Used2BeRX

                  Looks like it takes about 30-60 seconds per entry now

                  Wow! :( It's a bad news...

                  Well at least I have a good news: I've just added the --only-new option, so if you just wanna add new entries and ignore the existing ones, use this option (check the --help message I've updated accordingly).
                  (edit: I've uploaded a few minutes ago, probably you don't have this version)

                  Example of use case:

                  • You're starting a new gamelist.xml and the script is running.
                  • For some reason you have to turn off your raspi and the script didn't process all files yet.
                  • Hit ctrl+c and turn off your raspi.
                  • When you have a chance to run the script again use the --only-new option and it will ignore the synopsis.txt of all the existing entries in the gamelist.xml.
                  • Useful topics
                  • joystick-selection tool
                  • rpie-art tool
                  • achievements I made
                  1 Reply Last reply Reply Quote 0
                  • meleuM
                    meleu @Used2BeRX
                    last edited by meleu

                    @Used2BeRX said in Making premade gamelist.xml with xtra media collections:

                    Looks like it takes about 30-60 seconds per entry now

                    Please, do the following:

                    1. cancel the execution (ctrl+c),
                    2. remove the generated gamelist.xml,
                    3. --update the script,
                    4. run it again with the same parameters you used (including the time operator).
                    5. let me know if it takes more than 30 seconds per entry.

                    IMO 30 seconds per entry is unacceptable. OK, it's faster than human, but too slow for a computer (even a raspiZero). I have to find the bottleneck!

                    EDIT:
                    By the way, the script took almost 4 minutes to process 718 synopsis for atari2600 games on my Core i7 computer. OK, it has much more CPU power than a raspiZero, but 8 hours?! Try it again, please.

                    • Useful topics
                    • joystick-selection tool
                    • rpie-art tool
                    • achievements I made
                    Used2BeRXU 1 Reply Last reply Reply Quote 0
                    • Used2BeRXU
                      Used2BeRX @meleu
                      last edited by Used2BeRX

                      @meleu Sorry. I was away from the computer and didn't get your messages until now. I will upgrade and try it again.

                      FYI, I think I was right about 8 hours. As of writing this, it had only done 425 entries out of 745. That was in 274 minutes, real-time. (4:34)

                      Part of this was my fault for running the wrong command. I was running it with the --full option. I tried that again with --full in your new script and timed it and it was taking 40-41 seconds per entry.

                      I stopped it, deleted the output, and re-ran it with --no-desc and timed it at 10-11 seconds per entry, so I'm letting it run now. This is probably going to end up being just a little longer than the 2 hours and 12 minutes before you added the code to find the folders.

                      I'd still like to see if there are ways of tweaking this. I won't be using the --full option for my raspi zero setup, but I will be using it for the Pi 3 setup. Granted that is a faster processor itself, but this just seems to be taking a lot longer than it should to get the job done.

                      Just curious, are you doing the following things?

                      1. Searching for roms only in the following folder and sub-folders: /opt/pi/RetroPie/roms/{system_name}/ (system_name from the matching one used in the command running the script, ie: /opt/pi/RetroPie/Media/atari2600/Synopsis/*.*)

                      2. Searching for folders and sub-folders only in the same folder and sub-folders as number one above. And ONLY looking for folder names, completely disregarding any other files.

                      3. Searching for Media only in the following folder and sub-folders: /opt/pi/RetroPie/Media/{system_name}/ (You could get the system_name from the command running the script like above).

                      4. You could further narrow this down by pointing particular media fields to their exact folders, like <gamefaqs/> to /Media/{system_name}/GameFAQs/ and <threedbox> to /Media/{system_name}/Artwork/3D Boxart/

                      When I get a chance after this is done I'll re-check everything in EmulationStation and see if there are any more individual problems with files/folders displaying. When we can speed up the code I will start running a few --full and default runs and see if all of the media is being done right.

                      meleuM 1 Reply Last reply Reply Quote 0
                      • meleuM
                        meleu @Used2BeRX
                        last edited by meleu

                        @Used2BeRX

                        re-ran it with --no-desc and timed it at 10-11 seconds per entry (...). This is probably going to end up being just a little longer than the 2 hours and 12 minutes before you added the code to find the folders.

                        It's an unfair comparison. I noticed that creating a new gamelist.xml from scratch takes more time than just updating an existing one. Then I'm assuming the time didn't changed.

                        Just curious, are you doing the following things?

                        1. Searching for roms only in the following folder and sub-folders: /opt/pi/RetroPie/roms/{system_name}/ (system_name from the matching one used in the command running the script, ie: /opt/pi/RetroPie/Media/atari2600/Synopsis/*.*)

                        Yes. But the system_name is based on the Platform: field.

                        Example: If Platform: Atari 2600, then system_name=atari2600.

                        1. Searching for folders and sub-folders only in the same folder and sub-folders as number one above. And ONLY looking for folder names, completely disregarding any other files.

                        Yes.

                        1. Searching for Media only in the following folder and sub-folders: /opt/pi/RetroPie/Media/{system_name}/ (You could get the system_name from the command running the script like above).

                        Yes. But system_name is obtained as I described above.

                        1. You could further narrow this down by pointing particular media fields to their exact folders, like <gamefaqs/> to /Media/{system_name}/GameFAQs/ and <threedbox> to /Media/{system_name}/Artwork/3D Boxart/

                        Yes, I'm doing this.


                        I think I've made some progress in optimization...

                        Suggestion: to measure the optimizations, there's no need to wait all those 2+ hours, try to parse only the synopsis.txt files starting with A. The command should be like this:

                        time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/atari2600/Synoipsis/A*.*
                        

                        Got it?

                        For a fair comparison do the following:

                        1. do NOT --update your script for now.
                        2. remove the atari2600_gamelist.xml file.
                        3. run the command above (only parse synopsis.txt files starting with A).
                        4. take note of the times (real, user and sys).
                        5. remove the atari2600_gamelist.xml file.
                        6. --update the script.
                        7. run the command above (only parse synopsis.txt files starting with A).
                        8. take note of the times (real, user and sys).
                        9. post the times here.

                        Looking forward for the results...

                        • Useful topics
                        • joystick-selection tool
                        • rpie-art tool
                        • achievements I made
                        Used2BeRXU 2 Replies Last reply Reply Quote 1
                        • Used2BeRXU
                          Used2BeRX @meleu
                          last edited by

                          @meleu Alright. I'm going to do what you asked. I just looked at the time for the last run at it was 136m/102m/28m.

                          I'll let you know more when I know more.

                          1 Reply Last reply Reply Quote 1
                          • Used2BeRXU
                            Used2BeRX @meleu
                            last edited by

                            @meleu Wow.... what did you just do?
                            Old times: 6m4s/4m34s/1m13s
                            New times: 1m5s/0m33s/0m19s

                            You just cut the time down to about 20% of what it was before!

                            meleuM 1 Reply Last reply Reply Quote 1
                            • meleuM
                              meleu @Used2BeRX
                              last edited by meleu

                              @Used2BeRX said in Making premade gamelist.xml with xtra media collections:

                              @meleu Wow.... what did you just do?

                              Tweaks on find command.

                              Now let's see the time for the complete atari2600 set! :-)

                              EDIT
                              And check if the gamelist.xml is fine, please.

                              • Useful topics
                              • joystick-selection tool
                              • rpie-art tool
                              • achievements I made
                              Used2BeRXU 2 Replies Last reply Reply Quote 1
                              • Used2BeRXU
                                Used2BeRX @meleu
                                last edited by

                                @meleu Just finished running the full --no-desc
                                30m12s, 17m58s, 8m10s

                                I'm going to check to see if there were any problems with it now.

                                1 Reply Last reply Reply Quote 1
                                • Used2BeRXU
                                  Used2BeRX @meleu
                                  last edited by Used2BeRX

                                  @meleu 100% hits for boxart so that means that it worked perfect for path and image :)

                                  Had to do a long overdue reboot so it took a while for me to get back up but I did a few others while I was waiting.

                                  Atari 7800: 3m44s, 2m11s, 0m57s
                                  Atari Lynx: 3m25s, 1m56s, 0m54s
                                  Atari 5200: 19m17s, 12m43s, 4m51s

                                  CURRENT ISSUES:

                                  1. None of the `<path/> were written for Atari 5200/800. This might be because it's a special case. Of all of the systems I have for the RetroPie, this is the only one that you can't ZIP the files. All 5200/800 images I have are in BIN format.

                                  2. Folders for other systems are being added to "atari2600_gamelist.xml". (I know why this is happening. All of my folders.txt files still say the information for the Atari 2600).

                                  3. With the Atari 5200/800, I'd like for them to all write to atari5200_gamelist.xml. I don't include a lot of 800 games in my collection and rather than have two entries for it in the system I have all of the roms and media in the 5200 file structure, so I was wondering if we could maybe do somthing about putting those two scripts together. If that's too much of a hassle, no worries. I can just manually combine them when I'm done.

                                  I went through both Atari 7800 and Atari LYNX and we have 100% boxart showing up for both of them as well!!!

                                  As soon as I get the folder synopsis text files updated the folders should work perfect the next time I run the script.

                                  Looking great though.

                                  Now that we got the speed issue handled, I was wondering if we could talk about a few more options to add to the script?

                                  I had a few ideas, such as creating all of the gamelist.xmls for all existing systems one after another without having to be there to put in the next command.(or from a list I could create in a file).

                                  I was also thinking that we could do a --no-desc but with all of the Artwork folders, and then there could be a script that would easily change all of the entries to point to a different piece of artwork (ie: displaying action shots for all of the games instead of Boxart).

                                  meleuM 1 Reply Last reply Reply Quote 0
                                  • meleuM
                                    meleu @Used2BeRX
                                    last edited by meleu

                                    @Used2BeRX

                                    CURRENT ISSUES:

                                    1. None of the `<path/> were written for Atari 5200/800. This might be because it's a special case. Of all of the systems I have for the RetroPie, this is the only one that you can't ZIP the files. All 5200/800 images I have are in BIN format.

                                    Yep. The script was only trying to find .zip. I've made it search for .bin too. Do you have other extensions?

                                    1. Folders for other systems are being added to "atari2600_gamelist.xml". (I know why this is happening. All of my folders.txt files still say the information for the Atari 2600).

                                    Exactly. The synopsis.txt files must be accurate. In this case, the problem is in the Platform: field.

                                    1. With the Atari 5200/800, I'd like for them to all write to atari5200_gamelist.xml. I don't include a lot of 800 games in my collection and rather than have two entries for it in the system I have all of the roms and media in the 5200 file structure, so I was wondering if we could maybe do somthing about putting those two scripts together. If that's too much of a hassle, no worries. I can just manually combine them when I'm done.

                                    The script puts every entry on its own platform gamelist.xml. Do they use the same emulator in RetroPie? Anyway, if you want to see atari 800 games in the atari 5200 you'll have to combine manually, as you said.

                                    I had a few ideas, such as creating all of the gamelist.xmls for all existing systems one after another without having to be there to put in the next command.(or from a list I could create in a file).

                                    You can give everything in only one command line. Like this:

                                    ./Used2BeTXT /path/to/atari2600/synopsis/*.* /path/to/nes/synopsis/*.* /path/to/megadrive/synopsis/*.*
                                    

                                    I was also thinking that we could do a --no-desc but with all of the Artwork folders, and then there could be a script that would easily change all of the entries to point to a different piece of artwork (ie: displaying action shots for all of the games instead of Boxart).

                                    I've added the --image option. Then you can do:

                                    ./Used2BeTXT --no-desc --image 3dbox /path/to/atari2600/synopsis/*.*
                                    

                                    And the generated gamelist.xml will have the <image> pointing to the 3D Box art.

                                    Edit: Run --help to see the --image details.

                                    It's not fast if you just wanna change this field on an existing gamelist.xml. But if you just wanna do that, you don't need a script. You just need a basic knowledge about sed command. I'll leave this task with you. ;-)

                                    • Useful topics
                                    • joystick-selection tool
                                    • rpie-art tool
                                    • achievements I made
                                    Used2BeRXU 3 Replies Last reply Reply Quote 1
                                    • Used2BeRXU
                                      Used2BeRX @meleu
                                      last edited by Used2BeRX

                                      @meleu

                                      The script was only trying to find .zip. I've made it search for .bin too. Do you have other extensions?

                                      This won't increase search time, will it? All consoles have .zip except for .bin in 5200. I haven't tried any CD consoles yet, but .bin will probably be an issue there. I'm pretty sure that with SegaCD and TurboGrafxCD it will need to look for .cue and actually NOT look for .bin

                                      The script puts every entry on its own platform gamelist.xml. Do they use the same emulator in RetroPie? Anyway, if you want to see atari 800 games in the atari 5200 you'll have to combine manually, as you said.

                                      Yeah. They both use the same emulator.

                                      You can give everything in only one command line. Like this:

                                      ./Used2BeTXT /path/to/atari2600/synopsis/*.* /path/to/nes/synopsis/*.* /path/to/megadrive/synopsis/*.*
                                      

                                      Is it possible for me to write the command once and save it so I can just type a quick command to run it?

                                      I've added the --image option. Then you can do:

                                      ./Used2BeTXT --no-desc --image 3dbox /path/to/atari2600/synopsis/*.*
                                      

                                      What does --image alone do? Does it add all of the image file fields, but default to Box Front? And then it would change it to 3D Box art or other types if I put the second command after it?

                                      It's not fast if you just wanna change this field on an existing gamelist.xml. But if you just wanna do that, you don't need a script. You just need a basic knowledge about sed command. I'll leave this task with you. ;-)

                                      The reason I wanted the --image option to add all 5 types of artwork is because before I had any help writing scripts I did this switch between artwork types manually myself with a few search and replace commands in Notepad++.

                                      I was hoping that if the --image option wrote all 5 fields for all 5 pieces of artwork, that a quick script could be written that would automatically do this. For instance, if you wanted to show Title shots, but it was currently showing Box Front, the script would first replace all <image></image> lines to be <boxfront></boxfront> and then replace all <title></title> lines to be <image></image>.

                                      But... It would have to do this ONLY in <game> entries. The script would break the <folder> if it changed the <image> fields for the folders.

                                      meleuM 1 Reply Last reply Reply Quote 0
                                      • Used2BeRXU
                                        Used2BeRX @meleu
                                        last edited by

                                        @meleu BTW dude... I've been learning some cool stuff in a great photo manipulation program called paint.net. Never did any graphic arts before and I always thought Photoshop was too complicated to bother learning...

                                        Check these folder images out. :)

                                        ATARI 5200 PROTOTYPES / HOMEBREW
                                        alt text

                                        US LICENSED GAME GEAR:
                                        alt text

                                        EUROPE LICENSED GAME GEAR:
                                        alt text

                                        JAPAN LICENSED GAME GEAR:
                                        alt text

                                        1 Reply Last reply Reply Quote 2
                                        • Used2BeRXU
                                          Used2BeRX @meleu
                                          last edited by

                                          @meleu Yes... I definitely think we need a way to simplify a command that makes a gamelist.xml for all of the systems. I just started the script for all included systems so far and this is what it looked like:

                                          time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/atari2600/Synopsis/*.* /home/pi/RetroPie/Media/atari5200/Synopsis/*.* /home/pi/RetroPie/Media/atari7800/Synopsis/*.* /home/pi/RetroPie/Media/atarilynx/Synopsis/*.* /home/pi/RetroPie/Media/gamegear/Synopsis/*.* /home/pi/RetroPie/Media/gb/Synopsis/*.* /home/pi/RetroPie/Media/gba/Synopsis/*.* /home/pi/RetroPie/Media/gbc/Synopsis/*.* /home/pi/RetroPie/Media/mastersystem/Synopsis/*.* /home/pi/RetroPie/Media/megadrive/Synopsis/*.* /home/pi/RetroPie/Media/nes/Synopsis/*.* /home/pi/RetroPie/Media/ngp/Synopsis/*.* /home/pi/RetroPie/Media/pcengine/Synopsis/*.* /home/pi/RetroPie/Media/sega32x/Synopsis/*.* /home/pi/RetroPie/Media/sg-1000/Synopsis/*.* /home/pi/RetroPie/Media/snes/Synopsis/*.* /home/pi/RetroPie/Media/videopac/Synopsis/*.* /home/pi/RetroPie/Media/virtualboy/Synopsis/*.* /home/pi/RetroPie/Media/wonderswan/Synopsis/*.*
                                          

                                          Not fun.... :(

                                          Anyway....

                                          I completed editing EVERY folder synopsis file, so once this finishes running I should be able to compile a full list of any problems with the script for every system except for the CD based systems.

                                          I'll let you know more as soon as I know.

                                          Thanks again.

                                          meleuM 1 Reply Last reply Reply Quote 0
                                          • meleuM
                                            meleu @Used2BeRX
                                            last edited by

                                            @Used2BeRX

                                            I've just noticed that there's a smallbug in the code when some metadata has & and you re-run the script with the same synopsis. I'm trying to fix it now. In the first run (creating a new entry in the gamelist.xml) this bug doesn't happen.

                                            This won't increase search time, will it?

                                            Maybe some miliseconds. But the script must be able to find those files, doesn't it? :-)

                                            I haven't tried any CD consoles yet, but .bin will probably be an issue there. I'm pretty sure that with SegaCD and TurboGrafxCD it will need to look for .cue and actually NOT look for .bin

                                            OK, I've made the script search for .cue if Media: CD, no matter what platform it is.

                                            Is it possible for me to write the command once and save it so I can just type a quick command to run it?

                                            Yes, you can create an alias. But if you prefer to create a list in a file, I can show you some tricks on how to use this file (nothing to be changed in the script).

                                            What does --image alone do?

                                            gives an error message and exit.

                                            I'm planning add more options for <image> soon.

                                            • Useful topics
                                            • joystick-selection tool
                                            • rpie-art tool
                                            • achievements I made
                                            Used2BeRXU 1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.