Scraper by Sselph, can´t scrape chosen systems?!
-
sometimes the scraper doesn´t work so i checked if normal "all system" scraping works. It did. but "chosen system" scraping still does not.
-
Sorry no offense man, I actually managed to mess up what I described when I once tried to scrape a single System...
-
Something is going on for weeks now with these scrapers, like ALL the scrapers...I can't scrape for my life, even with a hardwired 150mb/s internet connection. (ok, I can, but it takes hours, and I have to literally space it out over days to find a time where it actually works).
SSelph's uses thegamedb.com by default. If you go to thegamedb.com and search for something you'll see the website is at a crawl....no good. The other option (forget the scraper website name, "open_something" maybe...?), was taking 2 minutes per ROM...and is failing at the most simple names and giving me "XML syntax error on line 22: element <meta> closed by </head>" errors.
Go and try the built-in scraper (which the Reddit folks say is MUCH faster in Retropie 4.1), I choose "THEARCHIVE" instead of thegamesdb, and tried to scrape my 26 SNES games...it didn't find a SINGLE one. Running it with thegamesdb.com right now and it's averaging 17-90 seconds per rom...(yes, I'm bored, and aggravated, I used the stopwatch on my phone ;) ...only found half of them (all No-Intro ROMs, with simple non-confusing names).
-
I noticed this crawl issue today as well. I thought maybe it was just me, as I typically start a scraping session before leaving for work (8 hours) and it's usually done by the time I got home. But today, being a holiday, I've been working on my system some more and have been running a scrape session for 4 hours now, and I'm only up to my 3rd system out of 10. When I look at the SSH log, it is showing a lot of failing hash checks. Don't know what's wrong. I always start by updating the script to make sure I have the latest version.. maybe something was updated that is causing issues?
-
I'd recommend you all to take a look at the UXS:
https://retropie.org.uk/forum/topic/5291/soft-universal-xml-scraper-v2-easy-scrape-with-high-quality-pictureIt uses ScreenScraper.fr as a source (which is available for sselph's scraper as well).
-
@paradadf said in Scraper by Sselph, can´t scrape chosen systems?!:
I'd recommend you all to take a look at the UXS:
https://retropie.org.uk/forum/topic/5291/soft-universal-xml-scraper-v2-easy-scrape-with-high-quality-pictureIt uses ScreenScraper.fr as a source (which is available for sselph's scraper as well).
I just posted in that topic, but at first glance it looks like this is only for PC users (I'm on a Mac) who have their Pi setup as a network drive (or similar setup). It would be a big shame if that's the case and this scraper can't be used on the Pi itself...which leads me back to being mad that NO scraper works well on the Pi itself...
-
@Dochartaigh have you tried to use ScreenScraper as a source for sselph's scraper? Set TGDB to false, otherwise you'll get its media first. It works pretty good.
One other option you have is to create a bash script and make the whole process work from your MAC (like the one I created for Windows and you can find here and use for inspiration). I don't find appropriate saying that MAC users are lef out. Simply (for some unknown reason), no MAC user has tried to help the community in this regard. Do you want to do it?
-
@Dochartaigh luckily i own both, pc and mac :9 i´ll give it a shot and answer back
-
@paradadf said in Scraper by Sselph, can´t scrape chosen systems?!:
Yes, I've tried all the database options for Selph's scraper (and the built-in scraper). I'm also on a couple different Raspberry Pi/RetroPi forums and people are reporting left and right their numerous problems with scraping for the last week or so - I just think something is broken/malfunctioning down the line.
As far as programming for the Mac, I don't even know what a bash script is! So I'm definitely not the person to embark on a project like that (if you need a hand with graphics, let me know!, but programming....not so much unless it's basic web stuff).
-
@Dochartaigh tried the universal XML Scraper, i did set it up right it says 3 roms found 3 scraped, but nothing happens :D
-
I found out what the problem was.
The Scrapers just dont work as reliable as they should. I used the XML Scraper in this thread, it works but it doesn´t save anything in the pi after its done, even though i checked multiple times if i set it up correctly.
Secondly i tried the Terminal Scraper again, and tried to scrape just ONE system, now it worked.
Weird stuff.
So if anyone has problems, wait a day try again that sould solve all your problems. -
@KingVarus
I had the same problem and I think is a workaround:
https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/issues/51#issuecomment-263076654 -
I tried to scrape my library yesterday and got an error 429 - "Too Many Request Responses".
I assume there's something wrong with the servers so it's not working as it should. I tried now and got the same results. Will wait until tomorrow and give it another go.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.