It worked! thanks Clyde and a Merry Christmas.

@retrocake It happens from time to time, because thegamesdb.net is too busy and no response is received by the scraper. You can try scraping at another time or use the alternate scraper - https://retropie.org.uk/docs/Scraper/

@retroprogrammer XD thanks. It is actually pretty fast. So ill keep trying things. Thanks tho!

@aardvark Yeah that really depends on the number of games you have. So that is hard to nail down until you actually have everything scraped. Each video is a different size. For example an Atari 2600 games snap that is 30 seconds long will be much smaller than say a PSX games of 30 seconds. The videos that are scraped and converted for the pi are 30fps @ 320x240 4:3 aspect. Great for the pi and they are smaller size too! (the vidsnaps can be 640x480 @ 60fps 4:3 the pi can handle them..but the converted ones work much better!)

Honestly a one terabyte hard drive will get you most of everything the pi can handle. However a 128gb micro sd card is "perfect" and a 256gb should be more than enough..unless you want to run a LOT of psx games. (This does not include PC-ish games.) My system with games, videos and snaps fit on a 64gb card happily (and that includes daphne). Note though that i pulled out alot of arcade games (clones/mature/mahjong).

@matchaman There is a command line option for that:

-missing file The file where information about ROMs that weren't scraped is added.

The script that runs the scraper from the setup menu is located at /home/pi/RetroPie-Setup/scriptmodules/supplementary/scraper.sh. It doesn't contain any -missing parameter, so I assume this option isn't used by it.

Nevertheless, adding the line params+=(-missing "/home/pi/missing.txt") right after line 76 (local params=()) should add a parameter that writes all missed roms into the file missing.txt in RetroPie's home directory.

I just tested this with my Atari 2600 collection and all roms with missing hashes were written into the missing.txt. Thanks for inspiring me to do this, I've been planning it for weeks. :)

@frankenbenz Good to hear. If there was a general issue with the scraper install we definitely would have heard more about it by now.
Sometimes things just get screwy for no obvious reason. Computers, eh?

@andrewh said in unable to install Scraper due to ~ key:

Alternately, you can do it in two steps.

the original command should be as follows (capitalization of folder names is important) ;

sudo ~/RetroPie-Setup/retropie_setup.sh

It tells the system to run the retropie_setup.sh script found in the RetroPie-Setup subdirectory of the user's home directory (the user being the 'pi' account you logged in with). And this command can be run successfully no matter where in the filesystem you happen to have navigated to.

So, instead, you can do it in two steps as follows;

cd sudo ./RetroPie-Setup/retropie_setup.sh

The first line will take you to the home directory for the user you're logged in as.
The second line runs the script found in the subdirectory of the current directory (which will be the home directory, since that's where you moved to with the first command).

Just gave it a try and it’s worked perfectly, ive set the game list for videos and the majority have them but some are just still images (may of been where I done manual scraping)
Thanks for the help guys I would of never sorted that other wise 😀😀😀

@nestor1924 If you want to have a better scrapping for your games/systems i would suggest using Universal XML Scraper V2 since it looks better than the default scraper on the Pi.
Also Universal XML Scraper V2 you have to use on your PC but it is very simple to use.

@buzz I've just checked and Scraper got installed successfully.

Deleting /opt/retropie/supplementary/golang/ and running sudo ./retropie_packages.sh scraper was the solution.

Thank you so much

On the https://screenscraper.fr site, there's a list titled "Server" in the upper right of the page. There, you can see the current CPU workload in percent. If the load gets too high, the green checkmarks will go red to show that these users will be blocked from scraping until the load goes down. Non-registered users will be blocked first, then registered, then (maybe) contributors.

So, I guess registering will get you blocked later. But I'm not registered there so I can't tell you for sure.

@gordonfreemanjr Batman Returns (1992).conf worked for me on skyscraper with screenscraper set as the source. On the second link you posted you can see there is a "filename" tab, if you click on it you can see other filename aliases.

So I ended up downloading the Hyperspin 2D cover set and it worked incredibly for this purpose.

I do have one question though: We used to be able to scrape folders and add metadata to the folder itself. For example, for multi-disc PSX games, I'd put both discs in a folder and scrape the folder to have the metadata displayed when highlighted in the gamelist. Every time I try and do that now though, ES crashes and I get the following error...

"lv10: TheGamesDBRequest - Error parsing XML.
Start-end tags mismatch
terminate called after throwing an insyance of 'std::out_of_range'
what(): map::at
Aborted"

Does anyone know if attaching metadata to a folder was removed recently? I am currently running 4.3.8.

@mydriaze Oh, the games still go in the megadrive folder, the platform.cfg file just changes the display and scrapping settings.

Oh, this is the txt you'd need in the platform.cfg file if you create it:

megadrive_theme="genesis"
megadrive_platform="genesis"
pcengine_theme="tg16"
pcengine_platform="tg16"

Then after creating that file and putting it in /opt/retropie/configs/all/ you need to update the TG and genesis cores, and you're good to go.

I went through this just last night. I "remade" all my cue files (made blank zero byte files and just named them "gamename.cue"). However the scraper scraped the same game over and over again and the game was for the sega megadrive not the psx sooooo shrugs

You can add the videos manually yes. Place them into your /home/pi/RetroPie/roms/psx/images and then re-scrape them. Two things though..

your videos need to be renamed so they have -video.mp4 on the end (i.e. Tekken (USA)-video.mp4) I would strongly suggest you use a batch re-namer to do so! (for windows use Bulk Renemer or for linux use Pyrenamer it is in the repos))

The BIG problem is the two scrapers don't play nice with each other. The "built in scraper for ES will delete the video tags and Steven's scraper may not pick up the correct info due ti the database (I guess).

So what happens is you don't get boxart or info and the wrong vid but everything is tagged...then you go trough and redo them and you get boxart and info but no tags. I have not tried to scrape the videos locally from the folder after using the built ES scraper yet. It might not erase anything and add the video tags.

So you want to do the built in ES scraper first and then use Steven's scraper to scrape from your folder after you have renamed all your videos. I can try that later and see it it works..but again you will need files that at least end in the correct extension. I had I batch/script that would do just that..but that was a lifetime ago when i was using MameWah..I'll see if I can dig it up...but don't hold your breath ;)

Edit: Ok so here is what i did..and it worked..okay...

I scraped all my pbp files with the built in scraper then I used Steven's scraper scraping just the cue files and append that to my list and it scraped some of my videos. So I still had to hand edit my list..so might be the only why do to it is by hand..

Here is an Idea..scrape just the cue files (making sure thy have the correct name and all that) with the built in first...then scrape with Steven's scraper appending the information to the list (that way your video tags are added) last go and do a find and replace on the gamelist.xml and search for .cue< replace with .pbp<
(yes add the less-than sign..just to make it harder to replace something else.)
The only catch with that is your pbp will have to match your .cue file names exactly it's an idea anyway.

EDIT: You can batch rename your .pbp to .cue then scrape..then rename them back from .cue to .pbp

screen scraper mix is pretty good lets me scrap my custom folders with ease

Anyone?

Sselph's graphical menu from the RetroPie-Setup options is fairly limited - it's the command line where it really comes into its own.

In essence, quit ES by pressing F4 (scraping whilst ES is running usually doesn't work very well). cd to the roms directory and then into your console of choice and then run the following:

/opt/retropie/supplementary/scraper/scraper -add_not_found -download_images -download_marquees -download_videos -console_src ss,gdb -console_img mix4,3b,b,s,f,l -image_dir=boxart -image_path=boxart -img_format png -video_dir=snap -video_path=snap -marquee_dir=wheel -marquee_path=wheel -marquee_format png -image_suffix="" -video_suffix="" -marquee_suffix="" -missing "games_missing.csv"

Please note, I've been separating out video snaps and marquees in prep for an Attract Mode build so you may want to adjust accordingly. To break the command down, it's scraping a mixed mode image from screenscraper and putting into a 'boxart' folder within the console's rom folder (-console_img mix4,3b,b,s,f,l means it looks for a mixed image first, then 3d box art, then boxart, then screenshot etc). It's then doing the same for video snaps and marquee images and all images will be output as png. I've also set it to output any missing titles into a csv for reference. If you didn't want videos, you could just remove the -download_videos flag.

If you are a Windows user you can run this directly on your roms folders via DOS if you download the DOS version of Sselph scraper.

Hope that helps.

thegamesdb,net API is offline. Use a different scraper or just wait. How many times is this issue going to be posted? Oh, just checked their site, it's all new and fancy.

@elshiftos spectrum scraping works just fine with screenscraper. I've uploaded a ton of box scans but nowhere near a complete set. It is picky with the filenames though so maybe check on the website that you're romnames are correctly named?