RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Gamelist Excel export script

    Scheduled Pinned Locked Moved Ideas and Development
    emulationstatioexportgamelist
    126 Posts 19 Posters 31.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • mituM
      mitu Global Moderator @Used2BeRX
      last edited by

      @used2berx What I meant for example was an actual Excel file, don't you want to convert from Excel to a gamelist.xml file ? My script does not convert - right now - all the fields you have in this sample gamelist.xml file, it only reads the standard ES metadata tags.
      You want me to modify the export script to produce an Excel from this sample gamelist XML file ?

      Used2BeRXU 2 Replies Last reply Reply Quote 1
      • Used2BeRXU
        Used2BeRX @mitu
        last edited by Used2BeRX

        @mitu Oh... yeah. I don't have any excel or Google Spreadsheet with any of this information now. I haven't run your script at all yet.

        I was actually asking if you could add new columns for your spreadsheet conversion program that would use all of the tags listed in the gamelist.xml file above. That way, after I do the editing to fields like <desc/>, <genre/>, <publisher/>, etc., then I could convert the spreadsheet back to a gamelist.xml file and none of the other fields would be missing when converted back.

        Some of these fields will actually never be used in ES, but I'm hoping once I start releasing sets that some of them are, such as manuals, gamefaqs and more image types. The fields that will never be used in ES are actually things that will be carried all the way back to the synopsis.txt files, which are individual files with the game rom name that are used to display all of the game information in the XBox Madmab Edition emulators. (These files are currently how meleu's script gets all of the individual game information into a gamelist.xml file, and with some other wizardry he's managed to also add all of the file location tags so long as media with the exact same name as the rom/synopsis.txt name is in the corresponding location).

        This is why I had suggested a "Regular Spreadsheet" and a separate "Rx Spreadsheet" option so people who just want the ES tags wouldn't have all of my tags cluttering up their spreadsheet... or you wouldn't have to go out of your way to make a completely separate script for me that would have to be updated separately if you added other cool stuff to the one you have now.

        Sorry for the confusion. :|

        1 Reply Last reply Reply Quote 0
        • Used2BeRXU
          Used2BeRX @mitu
          last edited by

          @mitu BTW... figured I should show you an example of what the synopsis.txt files look like. This is for Mario in Some Usual Day.txt, same game as the example above:

          Mario in Some Usual Day
          Original Title: Super Mario Bros. 3
          Platform: Nintendo Entertainment System
          Region: USA
          Media: Cartridge
          Controller: NES Gamepad
          Genre: Action / Platformer
          Gametype: Hack
          Release Year: 1990
          Hack Release Year: 2005
          Developer: Nintendo R&D2
          Publisher: Nintendo of America Inc.
          Hacked by: JaSp
          Version: 1.1
          Players: 1
          _________________________
          
          Some usual day isn't so usual after all. This game features enemies from Zelda, Metroid, and Kirby, new power-ups, and a brand-new secret coin system. And yes, that is a fire-ball-chucking Raccoon Mario below. This is a complete hack of Super Mario Bros 3, featuring 4 new worlds and 1 secret world! All worlds are filled with new levels, new graphics, new asm mods and more!
          
          https://www.romhacking.net/hacks/53/
          http://www.mobygames.com/game/nes/super-mario-bros-3
          

          Mario in Some Usual Day.txt is the same name as the <xtrasname/> tag in the gamelist.xml. All media for this game such as the rom, synopsis, artwork, etc. will have this exact file name.

          1 Reply Last reply Reply Quote 0
          • mituM
            mitu Global Moderator @Used2BeRX
            last edited by mitu

            @used2berx Here it is

            • https://gist.github.com/cmitu/2f1a0dab4e5086b8fca99b081677b6ec

            Installation instructions are the same as my initial script, needs the python module mentioned there in order to work.
            You can run it with -h to see the usage, by default reads a gamelist.xml from the running dir and exports to gamelist.xlsx there, but you can change the paths via parameters.
            Note that the gamelist fragment you posted has some structure errors - mismatched tags or missing closing tags. You need to validate the XML structure before running the export, since it will error out otherwise.

            You can add more tags (see line 120) to be parsed and added to the export. I think I covered most of the ones in your example, but you can freely add more.

            Used2BeRXU 1 Reply Last reply Reply Quote 1
            • Used2BeRXU
              Used2BeRX @mitu
              last edited by

              @mitu Thank you so much man. :)

              I don't think I'll have a chance to really dive into this and test it for about two more weeks. Because of the nature of the cart work I'm doing and the way that I have to have my current compatibility spreadsheet that I use to do all of my work off of, I absolutely have to finish them before I can move on to anything else. Works's been getting in the way of it this week, but I will have some free time next week to hopefully finish that up and move on to this so I can give you feedback, and ask you any questions I might have about being able to add any missing tags myself in the future.

              Once this is all in place I hope @meleu has some free time to put the other tags in his script that will make converting from synopsis.txt to gamelist.xml to spreadsheet and all the way back again without losing any data a snap.

              Please bear with me until I can get the time to properly devote to this. :)

              Oh... and yeah. It looks like I forgot to add the </desc> tag when I was doing some manual edits to the file above. I wanted to make sure I added all the tags I could currently think that we'd need... even the ones that meleu hasn't put in the script yet. I didn't see any others that were missing the closing tags. We shouldn't have any problems with this once the script is taking care of this and I don't have to manually edit anything in the gamelist.xml.

              1 Reply Last reply Reply Quote 0
              • papinistP
                papinist
                last edited by

                I was just thinking about how to have an excel game list.. and voilĂ ! :)

                However since I'm really a noob with linux commands, I'm having an error and no xlsx file at the end.

                '''
                pi@retropie:~ $ python export.py /home/pi/RetroPie/roms/export_list.xlsx
                2018-09-09 12:03:13,223 INFO Emulationstation systems file used: /home/pi/.emulationstation/es_systems.cfg
                Traceback (most recent call last):
                File "export.py", line 584, in <module>
                systems = parse_systems()
                File "export.py", line 261, in parse_systems
                gamelist = et.parse(gamelist_path)
                File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1182, in parse
                tree.parse(source, parser)
                File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 657, in parse
                self._root = parser.close()
                File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1654, in close
                self._raiseerror(v)
                File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
                raise err
                xml.etree.ElementTree.ParseError: no element found: line 1, column 0
                '''
                (sorry I can't remember the 'code' character everytime..)

                mituM 1 Reply Last reply Reply Quote 0
                • mituM
                  mitu Global Moderator @papinist
                  last edited by

                  @papinist Can you run the script with the -d flag (debug) to find out which gamelist file has this error, then upload the gamelist.xml somewhere so I can take a look ?

                  1 Reply Last reply Reply Quote 0
                  • papinistP
                    papinist
                    last edited by papinist

                    sure, here it is:
                    https://pastebin.com/QsT8qVjA

                    thanks for the help!

                    mituM 1 Reply Last reply Reply Quote 0
                    • mituM
                      mitu Global Moderator @papinist
                      last edited by

                      @papinist OK, so the problem seems to be the gamelist for the nds system (/home/pi/.emulationstation/gamelists/nds/gamelist.xml), can you add it to pastebin.com also so I can take a look ?

                      1 Reply Last reply Reply Quote 0
                      • papinistP
                        papinist
                        last edited by papinist

                        Actually, every time I exit from emulationstation to terminal, I got two errors regarding nds (see attached pic).
                        I am not sure what the problem is - this is a pre-made image that I edited to suit my needs - it has background music that I removed (this is probably the third error regarding autostart.sh), and many systems that I simply deleted from roms folder (and also image\video previews), but I only have this error regarding nds.

                        I don't have nds folder in 'roms' so I haven't a gamelist.xml to get!

                        alt text

                        EDIT: The gamelist.xml at the path reported in that error is a blank file.

                        mituM 1 Reply Last reply Reply Quote 0
                        • mituM
                          mitu Global Moderator @papinist
                          last edited by

                          @papinist said in Gamelist Excel export script:

                          EDIT: The gamelist.xml at the path reported in that error is a blank file.

                          Remove the file, this should get rid of the error in the export and also the error in Emulationstation. Btw, your disk is really low on free space, you should do something to free up space so you don't get into other errors.

                          1 Reply Last reply Reply Quote 0
                          • papinistP
                            papinist
                            last edited by

                            You are always the man, @mitu : it worked! Thanks so much!
                            Now I'm looking at the numbers and they are a bit off - total games 7850 in xlsx, but 7917 in emulationstation.
                            I see that Daphne is missing completely (12 games), and every sistem is missing something.
                            Also I have to admit that I didn't read the last page, maybe you have already explained this.

                            Re: space. Do you think it's low? This is a closed system, I have no plans to do updates in future (it is meant for a little boy) so I thought that 1.4 gb was enough!

                            mituM 1 Reply Last reply Reply Quote 0
                            • mituM
                              mitu Global Moderator @papinist
                              last edited by mitu

                              @papinist said in Gamelist Excel export script:

                              Now I'm looking at the numbers and they are a bit off - total games 7850 in xlsx, but 7917 in emulationstation.
                              I see that Daphne is missing completely (12 games), and every sistem is missing something.

                              I don't think the script will correctly count the systems where the game is an actual folder. If you could add the daphne system gamelist.xml to pastebin.com, I could take a look to see why it's not read correctly. A log file from running with the -d switch can also help to determine where's the error.

                              Note that the export will check if the ROM is present, so if you have the Parse metadata only option checked, then you'll see more entries than you might actually have on disk.

                              Attach the gamelist.xml for a system that's not exported and tell me which entry/entries are not appearing in the export, I'll take a look.

                              1 Reply Last reply Reply Quote 0
                              • papinistP
                                papinist
                                last edited by

                                This is daphne's gamelist.txt https://pastebin.com/uRFWiKnC

                                As example, ES reports 63 games for Atari 7800; Checking roms folder it has 63 roms; xls reports 57 games.

                                (how can I attach a file here? This message editor is really difficult to use)

                                mituM 1 Reply Last reply Reply Quote 0
                                • mituM
                                  mitu Global Moderator @papinist
                                  last edited by

                                  @papinist said in Gamelist Excel export script:

                                  how can I attach a file here

                                  I meant upload it to pastebin.com. You can only attach images and the gamelist is too large to paste it inline.

                                  1 Reply Last reply Reply Quote 0
                                  • papinistP
                                    papinist
                                    last edited by

                                    Understood!
                                    Atari 7800 gamelist.txt https://pastebin.com/wusbd0DJ

                                    mituM 1 Reply Last reply Reply Quote 0
                                    • mituM
                                      mitu Global Moderator @papinist
                                      last edited by mitu

                                      @papinist OK, so the Daphne export was missing because folders where not counted as valid ROMs. This should be fixed now, if you re-download the script it should work.

                                      For the Atari part, the gamelist you uploaded has 58 game entries, of which 1 doesn't have a name tag, so the export is correct. Do you have a gamelist.xml file in the ROMs folder (atari7800) ? If you run the script with the -d parameter, it should print the path of the gamelist used for export.

                                      [..]
                                      2018-09-09 DEBUG  Analyzing system: Atari 7800 ProSystem (atari7800)
                                      2018-09-09 DEBUG  Gamelist for atari7800 is read from /home/pi/.emulationstation/gamelists/atari7800/gamelist.xml
                                      2018-09-09 DEBUG  Found 57 game(s) for Atari 7800 ProSystem atari7800
                                      [..]
                                      
                                      1 Reply Last reply Reply Quote 0
                                      • papinistP
                                        papinist
                                        last edited by

                                        the gamelist.xml I uploaded is the one from roms folder - I recall some setting somewhere where you choose to have them in roms folder on under emulationstation.. Maybe I have some mess in this setup :-|

                                        However gamelist.txt in that path (/home/pi/.emulationstation/gamelists/atari7800/) is almost blank:

                                        <?xml version="1.0"?>
                                        <gameList />

                                        mituM 1 Reply Last reply Reply Quote 0
                                        • mituM
                                          mitu Global Moderator @papinist
                                          last edited by mitu

                                          @papinist said in Gamelist Excel export script:

                                          the gamelist.xml I uploaded is the one from roms folder - I recall some setting somewhere where you choose to have them in roms folder on under emulationstation.. Maybe I have some mess in this setup :-|

                                          The export script will gladly accept both paths and is really reading the gamelist from the ROMs folder (as it's the 1st path tried). However, as I said, your gamelist.xml file only has 57 valid entries, so the export is ok as far as the gamelist.xml content is concerned. Do you have turned off saving of metadata on exit in Emulationstation ? That would explain why you have 63 ROMs in that folder, but the gamelist.xml has less.

                                          1 Reply Last reply Reply Quote 0
                                          • papinistP
                                            papinist
                                            last edited by papinist

                                            No, that option is ON, and "parse gamelists only" is OFF.

                                            Your fix is working, now it correctly reports 12 daphne games.

                                            I don't know why games are mess in gamelist than in folder, since I never added roms myself.. :-O
                                            (I added games reported by ES on the right)

                                            0_1536527278217_edd57c52-7570-4d06-90f5-514b62654bde-image.png

                                            mituM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.