Scraper not saving information

johnnypanic

@Rion Thanks, that's exactly what I am doing, but once emulationstation boots back up, there's nothing... no info or anything, just the games list as it was prior to scraping.

(EDIT: I've noticed upon exiting ES, that it states there are no XML gameslists for the rom folders I have, could that be contributing to the problem?)

mediamogul

@johnnypanic

Can you think of anything special you might have done during your setup process that could possibly cause a problem? For example, have you relocated your roms folder or added/removed anything using an online tutorial in any way? Was it ever necessary to manually change any file permissions? Anything that might have been outside the boundaries of a normal install might give a clue.

Also, while we're at it:

What type of pi are you using?
What are the specs of your power supply?
What size sd card are you using?

dunnster4u

Hi

I posted the same couple days ago, I am running a PI3, scraped using stephens scraper (after exiting Emustation f4) it has scraped for the snes and I can see the files in the download folder however it does not display aginst the games when I go back in and run Emustation!! some have on diff systems but some not, same a Rion scrape a system completes then nothing there????

Pi3
64gb card
5v 2a power

johnnypanic

@mediamogul No, that's the strange thing. Once it was set up, I simply transferred the roms into the relevant folders. I did attempt to download the Scraper setup as - at the time - I wasn't aware it was already on the pie. However the download file could not be located, and I then discovered the Scraper was already on, and just needed setting up. The only other thing I've done was configure the controllers. And that's it.

As I said above, the only thing that rings a bell with me is that it says there is no .xml file in the rom folders when ES is shut down.

I'm running pi 3.7
64gb card
5v power

mediamogul

@johnnypanic said in Scraper not saving information:

(EDIT: I've noticed upon exiting ES, that it states there are no XML gameslists for the rom folders I have, could that be contributing to the problem?)

Check this link to see if it might be the same issue...

https://retropie.org.uk/forum/topic/968/missing-gamelist-file-error

johnnypanic

@mediamogul Ok, using WinSCP and looking in home/pi/RetroPie/BIOS/Databases (Is that the correct place to look?), I only have gameslists .xml files for some systems, none of which are the ones I have installed. However I have a brand new version of RP/ES and haven't moved anything. I don't have a backup - is there anyway to generate these, or am I looking in the wrong place before I begin?

mediamogul

@johnnypanic said in Scraper not saving information:

However I have a brand new version of RP/ES and haven't moved anything.

Seeing as how it's practically a fresh install now, you really have nothing to lose by starting over to solve this problem. However, you want to try to make sure it doesn't happen again. When you first launched your setup, did you go into the "Retro Pie" settings, update your setup script and then run a full update? If you didn't, try doing it next time. If you did, try not doing it.

Riverstorm

@johnnypanic said in Scraper not saving information:

@mediamogul Ok, using WinSCP and looking in home/pi/RetroPie/BIOS/Databases (Is that the correct place to look?)

I don't think that is correct. Try looking under /opt/retropie/configs/all/emulationstation/gamelists/

Try exiting ES to a prompt and you should see the path of the missing files if that's not correct. Check one of those files. If it is a zero byte file then you have a problem that Media pointed out above. All the files should have something in them. You can also check your RetroPie emulator menu and verify it has long/full names vs the short command line names.

It's completely a preference but I store my gamelist.xml (once it's created) right with my ROMs in the same folder and also locate the images in the same directory named ./images/. ES seems to check the ROMs folder for the presence of a gamelist.xml file first. The image path is really optional in the gamelist.xml but it sure makes it easy to backup one folder that has all my ROMs, gamelist.xml and images all in one directory. In fact I backup the whole roms folder with the emulators I am using and restore ALL emulator ROMs, images and gamelist.xml's with one drag and drop with WinSCP.

johnnypanic

@Riverstorm

Ok, looking in the path you described: /opt/retropie/configs/all/emulationstation/gamelists/ there are gameslist.xml files in each of the folders I have systems for, all of which seem to be populated correctly, save for the Atari 7800 folder which simply reads:

<?xml version="1.0"?>
<gamelist />

Although I'm not too bothered about that one as there are only a few titles in there. If the files are there, how come they're classed as missing when the ES is shut down? (it gives the pathways on the error message as not being in the roms folder itself - should I copy them over to the roms folders also?).

And if the data on these files are there, why is it not storing information when I scrape, is it because they're not in that file?

johnnypanic

@mediamogul If I can't get it sorted, would you not recommend a full update to attempt to solve the issue? If a new install isn't helping, would an update remove my roms, but still give the chance that it would not save information?

mediamogul

@johnnypanic

Yeah, as a first step, I would update the setup script & do a full system update to what you have now, but if that doesn't do it, I would start over fresh. I normally like to work through a problem rather than nuking from orbit in most cases, but since you have little to no time invested in this installation, I would consider starting over if all else fails.

Riverstorm

@johnnypanic

I agree with Media if you don't have a lot of time invested maybe start over as something still seems wrong with the continued errors. I don't scrape from the menu so I don't have much experience with it.

I usually exit ES (this is important even if using Putty, etc. or it will fail to work properly) and work from the prompt. It creates gamelist.xml right in with the ROMs and it also creates an image folder for images. I usually use WinSCP to backup my gamelist.xml right away as once you run ES it reorganizes the file and changes things around and also removes some items like the mamedb tags, etc. I keep the original to add or modify ROMs as needed. Here's the steps I usually do:

Check https://github.com/sselph/scraper/releases for the latest release version/name
wget https://github.com/sselph/scraper/releases/download/<release_name>/scraper_rpi2.zip
sudo unzip scraper_rpi2.zip scraper -d /usr/local/bin/ to unpack the files
sudo rm scraper_rpi2.zip optionally remove the zip
cd ~/RetroPie/roms/nes or whatever emulator you want to scrape
scraper with any optional parameters you prefer

johnnypanic

@Riverstorm Ok, I seem to have cracked it for the most part on a new setup. The only one I'm having issues with now is the fba folder. Any idea what parameters you use for that one, as 'scraper -fba' doesn't seem to work, like it did when scraping mame roms.

Riverstorm

@johnnypanic said in Scraper not saving information:

@Riverstorm Ok, I seem to have cracked it for the most part on a new setup. The only one I'm having issues with now is the fba folder. Any idea what parameters you use for that one, as 'scraper -fba' doesn't seem to work, like it did when scraping mame roms.

Take a look at this thread all the parameters for scraper are there about half way down.

https://retropie.org.uk/forum/topic/217/scraping-scummvm-games/6

EDIT: Try the -mame switch as FBA would fall under that category I think.

johnnypanic

@Riverstorm @mediamogul Awesome, it's now sorted. Every rom on there now has the info - things are a lot clearer. No more errors for missing .xml files on shutdown either. Thanks a lot for the help guys!

Riverstorm

@johnnypanic said in Scraper not saving information:

@Riverstorm @mediamogul Awesome, it's now sorted. Every rom on there now has the info - things are a lot clearer. No more errors for missing .xml files on shutdown either. Thanks a lot for the help guys!

That's great. It's a nice feeling once you get all up and running that way you like. :)

lphillimore

@johnnypanic Hey, any specifics on the build or fresh install steps that worked for you? Im going mad here trying to get this scraper to work. Originally tired on my original build but no joy. Just re-imaged to the latest release, updated the startup script, setup the scraper via f4 command line (and updated it) yet when i go back into emulisation, no info or images on any of the roms.