Gamelist Excel export script

mitu · 3 Jul 2019, 19:14

@mostym It looks like there's no gamelist.xml files found by the script. The script tries to read it from the default Emulationstation location (~/.emulationstation/gamelists/<system>) or from the the ROMs folder, but it can't find any.
4.4.1 is actually the RetroPie version, not the Emulationstation version - you can find it by pressing Start while in Emulationstation.
Can you check where your gamelist.xml files are located ? Did you disable the Save metadata on exit option in Emulationstation ?

mostym · 3 Jul 2019, 19:23

@mitu Oh, ok, emulationstation version is 2.7.5RP. Save metadata on exit is enabled. If I do a locate gamelist.xml i can see the following:

root@retropie:/# locate gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/amstradcpc/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/arcade/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/atari2600/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/atari7800/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/atarilynx/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/c64/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/daphne/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/dreamcast/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/fba/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/fds/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/gamegear/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/gb/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/gba/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/gbc/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/mame-advmame/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/mame-libretro/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/mame-mame4all/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/mastersystem/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/megadrive/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/msx/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/n64/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/neogeo/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/nes/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/ngp/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/ngpc/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/pc/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/pcengine/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/ports/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/psp/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/psx/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/retropie/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/sega32x/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/segacd/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/sg-1000/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/snes/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/vectrex/gamelist.xml
/home/pi/v3/opt/retropie/configs/configs/all/emulationstation/gamelists/zxspectrum/gamelist.xml
/opt/retropie/configs/all/emulationstation/gamelists/mame-libretro/gamelist.xml
/opt/retropie/configs/all/emulationstation/gamelists/pcengine/gamelist.xml
/opt/retropie/configs/all/emulationstation/gamelists/ports/gamelist.xml
/opt/retropie/configs/all/emulationstation/gamelists/retropie/gamelist.xml
/opt/retropie/configs/all/emulationstation/gamelists/scummvm/gamelist.xml

If I look at the contents of each of the gamelist.xml files, there is the following in each:

<?xml version="1.0"?>
<gameList />

mitu · 7 Mar 2019, 19:25

@mostym Your gamelists are empty, so that's why the script cannot find any games - they're read from those files. I suspect the permissions on your ROM folders (and probably the gamelists folder) are incorrect and ES cannot save the files.
Can you post the ~/.emulationstation/es_log.txt file ? There might be an error logged if the gamelist.xml files are not saved.

mostym · 3 Jul 2019, 19:31

@mitu Are the gamelist.xml files supposed to be manually created? Here is the es_log.txt contents(I couldn't post here because it's too long) https://pastebin.com/eJgJqk8W

mitu · 3 Jul 2019, 19:36

@mostym The gamelist.xml files are created automatically by Emulationstation when you restart/shutdown, they're not created manually. Either way, you don't have them - check the permissions on the folders where they are created and restart ES at least once for them to be created.

mostym · 7 Mar 2019, 19:38

@mitu What are the permissions supposed to be? Are these set correctly?

drwxr-xr-x   4 pi   pi    69632 May 24  2018 mame-mame4all
drwxr-xr-x   2 pi   pi     4096 May 24  2018 markiii
drwxr-xr-x   2 pi   pi    20480 May 24  2018 mastersystem
drwxr-xr-x   4 pi   pi     4096 May 24  2018 megadrive
drwxr-xr-x   2 pi   pi     4096 May 24  2018 megadrive-japan
drwxr-xr-x   2 pi   pi     4096 May 24  2018 msk2+
drwxr-xr-x   2 pi   pi    36864 May 24  2018 msx
drwxr-xr-x   2 pi   pi    20480 May 24  2018 msx2
drwxr-xr-x   2 pi   pi    36864 May 24  2018 n64
drwxr-xr-x   2 pi   pi     4096 May 24  2018 nds
drwxr-xr-x   3 pi   pi    12288 May 31  2018 neogeo
drwxr-xr-x   8 pi   pi     4096 May 24  2018 nes
drwxr-xr-x   2 pi   pi     4096 May 24  2018 nesclassic
drwxr-xr-x   2 pi   pi     4096 May 24  2018 nesh
drwxr-xr-x   2 pi   pi     4096 May 24  2018 ngp
drwxr-xr-x   2 pi   pi    16384 May 24  2018 ngpc
drwxr-xr-x   2 pi   pi    12288 May 24  2018 oddyse
drwxr-xr-x   5 pi   pi     4096 May 24  2018 Oddysey2
drwxr-xr-x  45 pi   pi     4096 May 24  2018 pc
drwxr-xr-x   4 pi   pi     4096 May 31  2018 pcengine
drwxr-xr-x   2 pi   pi     4096 May 24  2018 pcenginecd
drwxr-xr-x   3 pi   pi     4096 May 24  2018 playchoice
drwxr-xr-x   8 pi   pi     4096 May 22  2018 ports
drwxr-xr-x   2 pi   pi     4096 May 22  2018 psp
drwxr-xr-x   2 pi   pi     4096 May 24  2018 pspminis
drwxr-xr-x 112 pi   pi     4096 May 24  2018 psx
drwxr-xr-x   2 pi   pi     4096 May 24  2018 psx (only use this if you have Scott's PBP set)
drwxr-xr-x   2 pi   pi     4096 May 24  2018 sc-3000
drwxr-xr-x  17 pi   pi     4096 May 24  2018 scummvm
drwxr-xr-x   2 pi   pi     4096 May 24  2018 sega32x
drwxr-xr-x   2 pi   pi     4096 May 24  2018 segacd
drwxr-xr-x   2 pi   pi     4096 May 24  2018 sfc
drwxr-xr-x   5 pi   pi     4096 May 24  2018 sg-1000
drwxr-xr-x   2 pi   pi     4096 May 24  2018 sgfx
drwxr-xr-x  10 pi   pi     4096 May 24  2018 snes
drwxr-xr-x   2 pi   pi     4096 May 24  2018 snesclassic
drwxr-xr-x   2 pi   pi     4096 May 24  2018 snesh
drwxr-xr-x   2 pi   pi    94208 May 24  2018 superfamicom
drwxr-xr-x   2 pi   pi     4096 May 24  2018 tg16
drwxr-xr-x   2 pi   pi     4096 May 24  2018 tg16cd
drwxr-xr-x   7 pi   pi     4096 May 24  2018 vectrex
drwxr-xr-x   2 pi   pi     4096 May 24  2018 vic20
drwxr-xr-x   2 pi   pi     4096 May 24  2018 videopac
drwxr-xr-x   2 pi   pi     4096 May 24  2018 videopac2
drwxr-xr-x   2 pi   pi     4096 May 24  2018 virtualboy
drwxr-xr-x   3 pi   pi     4096 May 24  2018 vs
drwxr-xr-x   2 pi   pi    12288 May 24  2018 wonderswan
drwxr-xr-x   2 pi   pi    12288 May 24  2018 wonderswancolor
drwxr-xr-x   2 pi   pi     4096 May 24  2018 zmachine
drwxr-xr-x  60 pi   pi     4096 May 24  2018 zxspectrum

mitu · 7 Mar 2019, 19:40

@mostym The permissions seem to be find on your ROM folders. How did you install your RetroPie system ?

mostym · 7 Mar 2019, 19:42

@mitu I installed it using rufus and the retropie image.

mitu · 7 Mar 2019, 19:46

@mostym Well, then beats me - the script will not work if your gamelist.xml are empty, so if you solve that, you'll make it work.

mostym · 7 Mar 2019, 19:47

@mitu I also just noticed that the PARSE GAMELISTS ONLY option is off. I don't know if that matters or not.

mitu · 7 Mar 2019, 19:51

@mostym That's ok, the option tells Emulationstation to ignore the ROMs folders contents and show only what you have registered in gamelist.xml. In your case, that would be nothing, since your files are empty.

mostym · 3 Jul 2019, 19:53

@mitu I see. What about

 -rwxr-xr-x  1 pi pi   35 May 22  2018 gamelist.xml

is that permission correct?

mitu · 7 Mar 2019, 20:02

@mostym The permission seems fine. Did you restart at least once Emulationstation since you installed it ? I see in the log the configuration of the controller, which usually happens on the 1st install.

mostym · 7 Mar 2019, 20:12

@mitu I just restarted it, and tried running the script and the same error occurred.

mitu · 7 Mar 2019, 20:15

@mostym Sorry, but the problem is somewhere else. Did you use the RetroPie supplied image from retropie.org.uk/download or another image ?

mostym · 7 Mar 2019, 20:19

@mitu I downloaded it from the retropie download page, yes.

ppuspfc · 21 Apr 2019, 18:03

I am interested in this work. Congratulations

FetusSandwich · 18 Apr 2020, 14:39

@mitu Hopefully you don't mind me resurrecting this thread, as this was an amazing tool when it was working for me and I think more people in the RetroPie need visibility into your awesome work.

Problem for me now is that: it was working for me, but now it has stopped running, and I'm completely unsure as to why.

Here is the error at the end of the script. It seems to get through all gamelists with no errors, then provides me with:

Traceback (most recent call last):
  File "export.py", line 583, in <module>
    systems = parse_systems()
  File "export.py", line 261, in parse_systems
    gamelist = et.parse(gamelist_path)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1182, in parse
    tree.parse(source, parser)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse
    parser.feed(data)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 564, column 32

Any troubleshooting or insight you could provide here would be greatly appreciated. Thanks!

mitu · 18 Apr 2020, 16:14

@FetusSandwich Run it with --debug and see where it stops. Looks like an issue with a malformed gamelist.xml file.

FetusSandwich · 12 Jun 2020, 13:47

@mitu Thanks for your help on this, mitu.

I was indeed able to get this script running eventually and found my way eventually with error logs to the problem child gamelist. What a lifesaver this thing is...

Thanks for the help.