Versatile C++ game scraper: Skyscraper

quicksilver · 3 Jun 2019, 21:10

@muldjord sorry to be a nuisance again so soon, but I'm having issues generating Gameslist for "ports". Using the GUI I was unable to scrape any games because the platform is set as "ports" which the sources don't recognize as a valid platform. So I scraped them all from the commandline using pc as the platform. However, when I go to generate the Gameslist it just says the games cannot be found.

Edit: I see scummvm set to look for PC and Amiga matches. Perhaps ports can be set to look at pc?

muldjord · 3 Jun 2019, 21:22

@quicksilver said in Versatile C++ game scraper: Skyscraper:

Using the GUI I was unable to scrape any games because the platform is set as "ports" which the sources don't recognize as a valid platform.

The ports platform already has several platform aliases. Among them is pc, so it should work just fine with the file name based modules provided your scripts have a good naming convention (Such as Game Name.sh). It will scrape .sh file when using ports. screenscraper might give you issues though, as I don't send a proper system id with the query since I don't know what platform the games will be for from the get-go as I do for other platforms. So there you will have to work around it as you already do.

EDIT: I actually just noticed that I can make this work with screenscraper aswell. The PC Dos platform they have is the parent ID also for the Windows platform. So I can set that and it will work.

If you want to test it out you can simply set line 571 in screenscraper.cpp to read return "135";. Then compile Skyscraper with make and then sudo make install. But this will only work if you've compiled it yourself using the approach from the github documentation.

quicksilver · 6 Mar 2019, 22:09

@muldjord the strange thing is thegamesdb will find the games if I search manually from the commandline with the platform set as pc. But if I search using the GUI it will return all games as not found.

I installed skyscraper through the retropie setup script. In order to test your fix would I have to uninstall and build it manually?

mitu · 7 Mar 2019, 04:44

@quicksilver Hm, I'll see if I can reproduce this. Can you give me the details - name of the file, folder (I guess it's ports ?) ?

quicksilver · 3 Jul 2019, 05:03

@mitu correct the game folders+the .sh scripts to run the games are in my /roms/ports/ directory.

If I attempt to scrape the .sh scripts on the command line using "pc" as the platform I will get successful scrapes for doom, doom 2, quake, hexen etc. But it won't work if the platform is set as "ports". Screenscraper shows a message that it's not a supported platform and the gamesdb can't find anything (unless I change the platform to PC like I said). Oddly enough, if I trick it by scraping my ports using PC as the platform the Gameslist will not generate anything.

Edit: and speaking of PC, it looks like skyscraper doesn't recognize .conf files as a valid extension in my PC folder.

mitu · 7 Mar 2019, 05:25

@quicksilver said in Versatile C++ game scraper: Skyscraper:

if I trick it by scraping my ports using PC as the platform the Gameslist will not generate anything.

How exactly are you 'tricking it' ? Are you copying the .sh files to the pc folder and scraping it from there ?

quicksilver · 3 Jul 2019, 05:41

@mitu no, the .sh files are for my ports games (doom, quake etc.) and they are in my ports folder. What I'm doing is changing the platform on the command line to "pc"

So for example this is the default and it returns no matching games:

Skyscraper -p ports -s thegamesdb ./RetroPie/roms/ports/doom.sh

But if I change to:

Skyscraper -p pc -s thegamesdb ./RetroPie/roms/ports/doom.sh

It will find the game no problem. I believe this even works for screenscraper as well if the name is a 100% match.

mitu · 3 Jul 2019, 05:51

@quicksilver Ah, ok, then it's normal that generating the list won't work, since the cache entry is associated with the pc platform.

quicksilver · 7 Mar 2019, 06:06

@mitu I figured it had something to do with that. But according to @muldjord, ports should already be looking for PC matches but unless I tell it to look for them by setting pc as the platform, it can't find them.

muldjord · 7 Mar 2019, 07:50

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@mitu I figured it had something to do with that. But according to @muldjord, ports should already be looking for PC matches but unless I tell it to look for them by setting pc as the platform, it can't find them.

Quick note: You should never give it a file name on command line that is not related to the current input folder (which in this case is roms/pc). If you do, it won't be included when you generate the game list afterwards since it is no longer in the input directory. You are basically caching data for a file that effectively doesn't exist when you generate the platform later on (since it will look for files in roms/ports at that point). :)

Ports currently looks for both Windows, Dos and PC as platforms when using the file name based modules (all modules except screenscraper). So that should work. If you use the fix I gave you earlier it might also work with screenscraper.

quicksilver · 3 Jul 2019, 12:45

@muldjord said in Versatile C++ game scraper: Skyscraper:

Ports currently looks for both Windows, Dos and PC as platforms when using the file name based modules (all modules except screenscraper). So that should work. If you use the fix I gave you earlier it might also work with screenscraper.

Except that it isn't working properly. The gamesdb will give a "file not found" for every one of my ports games. If I manually switch the platform to pc it will find the games no problem (I am aware this will not work properly for generating the Gameslist). This says to me that the platform "ports" is not properly searching "pc" like it's supposed to.

Also do you have a PayPal account? Or only patreon? I would like to tip you for all your hard work on this but would like to give a one-time payment. Wasn't sure if cancelling a patreon after a one month paemt was considered bad form.

A Former User · 7 Mar 2019, 13:42

I really wish I could use this on my Windows machine. :(

Maybe I'll set up a Linux partition just to run this.

muldjord · 3 Jul 2019, 16:42

@quicksilver said in Versatile C++ game scraper: Skyscraper:

Except that it isn't working properly. The gamesdb will give a "file not found" for every one of my ports games. If I manually switch the platform to pc it will find the games no problem (I am aware this will not work properly for generating the Gameslist). This says to me that the platform "ports" is not properly searching "pc" like it's supposed to.

Also do you have a PayPal account? Or only patreon? I would like to tip you for all your hard work on this but would like to give a one-time payment. Wasn't sure if cancelling a patreon after a one month paemt was considered bad form.

pc and ports have the exact same aliases, so this is caused by something else. The only difference between the platforms is that pc also looks for *.com *.bat and *.exe in addition to *.sh where ports only look for *.sh. So that might be the issue instead. The aliases are the same. But once again, you need to do the code change I mentioned earlier if you want it to work for screenscraper as well (I know you are using thegamesdb). Just to be clear. :)

You can donate directly to paypal by using muldjordlars@gmail.com, but I'd actually suggest you donate to the RetroPie project instead. :) That's up to you of course.

muldjord · 7 Mar 2019, 17:22

@LiveFastCyYoung said in Versatile C++ game scraper: Skyscraper:

I really wish I could use this on my Windows machine. :(

Maybe I'll set up a Linux partition just to run this.

I'm not sure how much work it would take to get it on Windows (probably not that much). But if I do port it to Windows, it will be command line only. :)

A Former User · 7 Mar 2019, 17:33

@muldjord I'd be fine with command line only. :D

quicksilver · 3 Jul 2019, 18:53

@muldjord thanks for the reply. All of my ports games are launched via .sh files so it shouldn't be a problem. But obviously it's not working so I'm not entirely sure what to do then.

I do donate already to RetroPie from time to time but I do also like to reward others as well!

Edit: looks like it's just thegamesdb that has an issue with ports. I scraped using openretro and mobygames and it picked up a lot of the games without an issue.

mitu · 3 Jul 2019, 18:31

@LiveFastCyYoung Skyscraper is command line only. If you feel adventurous, install Cygwin and try to compile it on Windows.
Have you tried Skraper ?

muldjord · 3 Nov 2019, 22:13

@LiveFastCyYoung said in Versatile C++ game scraper: Skyscraper:

@muldjord I'd be fine with command line only. :D

I've compiled a Windows version that seemingly works. Please follow the instructions in the INSTALL_README.txt after unzipping it. This is important, otherwise it won't run!

Also note that this version is 100% unsupported, so you are on your own in using it. :) The only thing I am interested in is to know whether it runs with the dll's I've included. If not I need to include them. So please let me know if it complains about missing dll's when running it. Other than that, it should work just fine. I did a really quick test with it and it scraped nes roms just fine and also generated a gamelist without problems.

Here ya go: http://www.muldjord.com/downloads/Skyscraper_3.1.2_unsupported_win_version.zip

EDIT: Another quick note. Windows probably has the worst command prompt of any OS and so it does not support the color codings I use. So the output will look pretty weird.

EDIT2: I just tried running it from the Powershell. This won't work. You apparently need to use the older Windows command prompt that you can run with "cmd".

EDIT3: Updated it to 3.1.1

EDIT4: Updated it to 3.1.2

A Former User · 3 Jul 2019, 19:48

@mitu I currently use Skraper. Since Skyscraper can use more sources than just ScreenScraper, I'd like to give this a shot.

@muldjord Excellent! I will run it from cmd prompt when I get a chance tonight and let you know how it goes. Thanks! :)

muldjord · 3 Jul 2019, 23:58

Skyscraper 3.1.1 released: https://github.com/muldjord/skyscraper

Worked around a bug in the 'screenscraper' xml output improving stability
Added limiter to media getter functions for 'screenscraper' which should improve stability

I've been working to improve the stability of the screenscrapermodule. The fixes included in this release seem to have made quite a difference, so please update.