Has Selph's Scraper been updated and now broken?

hansolo77

I've added a bunch of new systems to my setup, including Amstrad CPC, Atari 5200, Atari 800, Atari St, and Sega SG-1000. The last couple of days I've been trying to scrape with UXS but there is a lot of stuff missing. So I was about to go to step #2 and scrape with Selph's. Then I'll last-ditch try the internal EmulationStation scraper. When I tried to scrape Amstrad CPC in Selph's though, it ran through the ROMs really quick and came back saying it was done. I did it again and watched and it was actually saying a bunch of stuff like this:

2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blackbeard (1988)(Topo Soft)(es).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blackbeard (1988)(Topo Soft)(es)[a].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blackbeard (1988)(Topo Soft)[t +2].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blade Runner (1986)(CRL Group).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blade Warrior (1989)(Codemasters).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blade Warrior (1989)(Codemasters)[t +2].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blagger (1985)(Alligata Software - Amsoft).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blasted Squares (19xx)(-)(de).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blasteroids (1987)(Image Works)[cr Mc Spe].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blasteroids (1987)(Image Works)[t].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blazing Thunder (1990)(Hi-Tec Software)[cr GPA][t GPA].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blip! (1989)(Silverbird Software).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Block Invaders (2002)(Richard Wilson).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Block Invaders (2002)(Richard Wilson)[a].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blockbusters (1984)(Macsen Software).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blockhaus (1986)(P. Balhade)(fr).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blokker (1985)(CT).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blood Brothers (1988)(Gremlin Graphics Software).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blood Brothers (1988)(Gremlin Graphics Software)[a][cpm version].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blood Brothers (1988)(Gremlin Graphics Software)[cpm version].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blood Valley (1987)(Gremlin Graphics Software)(M3).dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Bloodwych (1990)(Image Works)[cr CBS].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blue Crystal, The (1985)(Rainbow Arts)(Disk 1 of 2)[cr NPS].dsk
2016/12/14 22:25:58 Can't find original path: /home/pi/RetroPie/roms/amstradcpc/Blue Crystal, The (1985)(Rainbow Arts)(Disk 2 of 2)[cr NPS].dsk

I checked the path, and it IS correct... I then tried to run it against Atari 5200, and initially had the same type of results:

2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/QIX (USA).zip
2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/Quest for Quintana Roo (USA).zip
2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/RealSports Baseball (USA).zip
2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/RealSports Basketball (USA) (82-11-05) (Proto).zip
2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/RealSports Basketball (USA) (83-10-31) (Proto).zip
2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/RealSports Basketball (USA).zip
2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/RealSports Football (USA).zip
2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/RealSports Soccer (USA).zip
2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/RealSports Tennis (USA).zip
2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/Rescue on Fractalus! (USA).zip
2016/12/14 22:26:52 Can't find original path: /home/pi/RetroPie/roms/atari5200/Road Runner (USA) (Proto).zip

Then it started a 2nd pass and was actually finding stuff. So I'm confused. Before starting this pass, I ran the RetroPie-Setup updates and also updated Selph's Scraper script. So I'm using the latest version (as of the writing of this message). Any ideas what might have happened?

hansolo77

I think I might be on to something when figuring this out on my own.... So I restarted the system and exited EmulationStation. What happened next was shocking. My screen went black like it should, then started reading off a whole bunch of

lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/amstradcpc"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atari2600"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atari5200"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atari7280"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atari800"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atarilynx"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atarist"

It did that for every system on my Pi. This comes right after scraping with UXS. So maybe UXS is to blame?

hansolo77

I'm still having problems with this. I erased all instances of gamelist.xml I have on my SD card. I then installed the latest version of UXS (I like it's MIX artwork). After spending 1 Day and 20 Hours scraping my entire system (yeah, I have a lot of systems lol), I then loaded up Selph's to do an append on anything UXS missed. Once I started the scan on the first system (Amstrad CPC), it started doing that whole big "Can't find original path: /home/pi/RetroPie/roms/amstradcpc/" stuff listing every single file. If I let it run through the whole thing, it starts over and then does a bunch of "skipping already in gamelist". The path it shows is the exact same. So I'm not sure what's going on here. Can anybody help?

For the record, I am storing gamelist.xml files in the /opt/retropie/configs/all/emulationstation/gamelists path, with each system in it's own folder. This is actually symlinked from /home/pi/.emulationstation/gamelists So like:

/opt/retropie/configs/all/emulationstation/gamelists/amstradcpc/gamelist.xml
/opt/retropie/configs/all/emulationstation/gamelists/atari2600/gamelist.xml
/opt/retropie/configs/all/emulationstation/gamelists/nes/gamelist.xml
etc

I checked, and there are no other gamelist.xml files hidden elsewhere (like in the roms path).

vbs

According to the scraper's website here:
https://github.com/sselph/scraper

It does only support this systems:

An auto-scraper for EmulationStation written in Go using hashes. This currently works with NES, SNES, N64, GB, GBC, GBA, MD, SMS, 32X, GG, PCE, A2600, LNX, MAME/FBA(see below), Dreamcast(bin/gdi), PSX(bin/cue), ScummVM, SegaCD ROMs.

hansolo77

Ah, so you think all the problems are because it doesn't actually have the ability to scan those systems?

vbs

@hansolo77
Hm, I don't know, probably only some of your problems are related to systems not supported by sselph's.

vbs

@hansolo77
Have you tried to shutdown ES, then delete all files named "gamelist.xml" (everywhere) and then scrape a system using UXS?

hansolo77

Yup. It's after I scrape with UXS that I start having the problems. I think I'm on to something though. I've clearly narrowed it down to UXS, as using Selph's alone works fine. Using UXS though causes really slow boot ups, and when ES closes I get those errors about the "<gameList> node". I did a comparison of the way Selph creates the xml vs UXS, and it looks like UXS is making them differently. So I'm experimenting now.

vbs

@hansolo77
I had that error once when ES was complaining about the gameList-node. In my case I had a 0 byte sized gamelist.xml that ES tried to read.
I can't believe that your ES cannot read the gamelist.xml generated by UXS, I mean it does for everyone else so I doubt it is a general error related to the format of the gamelist.xml that UXS generates.

hansolo77

I don't get it for just 1 file though, it's all of them:

@hansolo77 said in Has Selph's Scraper been updated and now broken?:

lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/amstradcpc"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atari2600"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atari5200"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atari7280"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atari800"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atarilynx"
lvl10:   Could not find <gameList> node in gamelist "/home/pi/RetroPie/roms/atarist"

I agree though, it is strange that I'm the only one who notices this. Unless I'm the only one who's being OCD about metadata and trying to scrape from multiple utilities.

vbs

@hansolo77
I would bet money that you have somewhere a broken (empty?) gamelist.xml.
Try to delete all instances of gamelist.xml on your system.
Or you could try to ES manually using parameter "--debug", then you should see the full path of the gamelist.xml ES is trying to read.