Versatile C++ game scraper: Skyscraper

muldjord · 29 Apr 2020, 16:22

@mitu I've forked and updated the skyscraper.sh script to use the --flags FLAG1,FLAG2 format. I've also added version checks for at least 3.5 to the gathering and generating functions that will display a message in case the user needs to update. Lastly I've updated all the places where the old flags were mentioned in the help texts so they now show the new --flags versions.

Should I just create a pull request for the RetroPie-Setup project?

mitu · 29 Apr 2020, 16:51

@muldjord said in Versatile C++ game scraper: Skyscraper:

Should I just create a pull request for the RetroPie-Setup project?

Why not ? I'll take a look - just be careful with those tabs :).

muldjord · 29 Apr 2020, 17:30

@mitu Yeah, I know. I think I got it right... Not 100% sure though. :D I'll make a pull request.

EDIT: Here it is: https://github.com/RetroPie/RetroPie-Setup/pull/3119

muldjord · 5 Nov 2020, 17:55

Skyscraper 3.5.3 released: https://github.com/muldjord/skyscraper

MAJOR: Added video conversion support through config.ini option 'videoConvertCommand="COMMAND %i %o"' option. See docs for more details.
Removed deprecated '<cover>' node and replaced it with '<thumbnail>' for ES gamelist generation (Thank you to 'yisraeldov' for letting me know)
Added simple error handling for media network requests
'--flags help' and '--cache help' no longer requires '-p PLATFORM' to be set
Changed 'worldofspectrum' scraping module to use https

Most prominent new feature is the addition of video conversion options. The videos delivered by the scraping sources can vary quite a lot in codec and color format. The new videoConvertCommand="COMMAND %i %o" config.ini option allows you to funnel the videos through an external command, before saving them to the cache. This all happens on the fly.
I have added an example command in the config.ini.example file. Obviously, if you use ffmpeg as the example uses, you need to install it first. For RetroPie and other Debian-derived distros you can install it with sudo apt install ffmpeg. Read more about the possibilities here. The video conversion makes good use of the --verbosity N option. So be sure to use --verbosity 3 when testing your own commands to ensure everything works as expected.

Let me know if you test this out.

ohmycommodore · 11 May 2020, 19:55

Using it a week or so ago, it completely missed on some major systems like Sega and SNES and more. I'll have to try it again. Odd that I could see it scraping for those systems but in the end presented me with nothing.

ohmycommodore · 11 May 2020, 19:57

I recall I was going to try ScreenScraper as a source this next time.

muldjord · 5 Nov 2020, 21:15

@ohmycommodore So it showed the data while scraping? Then you probably forgot to generate the gamelists afterwards, or you ran it while EmulationStation was running. Please check the F.A.Q.

muldjord · 20 May 2020, 20:25

Interesting... While I was at work today, Github has decided to rewind my repository by several commits, without me touching it. Anyone tried this before? My code is literally just gone!

How do I know this is a Github error? My 3.5.5 release is still there. The code is not. The repository seems to have reverted to 3.5.4. The 3.5.5 commits are nowhere to be found, except for the release, which can't exist without the commits.

If I download the 3.5.5 release, all of the missing code is in the release. Even the version number is 3.5.5 in the release files. But not in the repository.

I can't come up with a way I could have made this happen myself. All I ever do is git commit -m "MESSAGE" FILES, git tag TAG and git push origin TAG for releases. I never use branches or rebasing or any of that stuff. I never move the head manually either.

This is pretty frustrating to be honest. Code gone missing is the last thing a developer wants.

Clyde · 20 May 2020, 20:37

@muldjord My sincerest sympathies! I don't know if I understand you correctly, though. Is the code really gone or do you have a backup somewhere?

mitu · 20 May 2020, 20:39

@muldjord said in Versatile C++ game scraper: Skyscraper:

If I download the 3.5.5 release, all of the missing code is in the release. Even the version number is 3.5.5 in the release files. But not in the repository.

The commits still show up on the 3.5.5 tag, but don't seem associated with the master branch. Do they still show locally on you master branch ?

muldjord · 20 May 2020, 20:44

@Clyde Luckily I have the code in the release so I didn't loose anything. But it makes my eyes twitch to think about this even being possible. How can I trust Github to not do this again? If it is in fact their fault of course. Maybe I did something in my sleep on one of my other developer machines that would result in this... I just can't think of how.

@mitu Yeah, good catch, I hadn't seen that. I have no idea how it would get detached like that. I don't have the code on my current developer machine and git pull doesn't give it to me either. I'll have to check the other developer machines in a couple of days when I can get to them.

mitu · 20 May 2020, 20:53

@muldjord I cloned the repo and the 'missing' commits show up (screenshot from gitk):

My guess you checked out 4.3.4 and then added the subsequent commits without a branch ? My git-fu is not so strong.

muldjord · 20 May 2020, 20:57

@mitu I'll have to check the bash history on those other machines. It might shed some light on this. I really hope this is my own fault.

muldjord · 21 May 2020, 07:08

@mitu , @Clyde Just a quick update on my little problem. I think I know what has happened. I probably made the 3.5.5 release on a different machine and forgot to push the local commits to master before tagging and pushing the tag to origin. So the code change from 3.5.4 to 3.5.5 was never pushed to master, but the tag was. I had no idea you could even do that, and I guess it's a testament to how well I actually remember to commit the changes since this has never happened to me before. I must've been tired.

I am really happy this was an error on my part. I was worried this was some weird mirror-caching Github error. It is not. Phew. :)

Guess who'll never forget to push local commits to master before tagging again. Ah, who am I kidding, I'll probably forget it again next week. :D

Clyde · 21 May 2020, 09:18

@muldjord Maybe there's a way to automate the push the commits before pushing the tags, or any other method to keep both machines on-par? (I don't know enough bash-fu for more than this suggestion.)

muldjord · 21 May 2020, 09:35

I'll get to the other machine tomorrow and push the commits to master. I actually think that will fix it as the tag seems to be bound to a commit, which just happens to not be glued to master yet. So if that's the case it's not a big deal if I forget in the future.

raspnoobie · 22 May 2020, 18:43

Hi,

Thanks for making this scraper; it is really nice to use it and looks better than the one that is in emulationstation.

I have a question regarding the script used in retropie: I don't see a parameter to use the scraper only for the roms that hasn't been parsed yet. This result in downloading the same games over and over , even if a different source scraped that game.
Is this a limitation of the existing script for retropie or it is not even an option, and require you to run the scraper by hand via terminal?

I wish there was an aggregate function that pars all sources in sequence, only for roms that were not identified; so you can leave it churning overninght :) Thanks !

muldjord · 22 May 2020, 20:38

@raspnoobie There is, but it's not implemented in the script yet. It's a command-line flag --flags onlymissing. It will make sure only games that have no data cached for them (from any source) are scraped with any source you choose. Be aware that if a game has even a single resource cached by any source, it will then be skipped. If instead you want to scrape only games that are missing a certain resource, you can use the --cache reportoption as documented here

Clyde · 22 May 2020, 20:39

Hello @muldjord, I recently ran into a problem on my new Pi 4 while starting to include Playstation 1 games into my Retropie setup.

While Skyscraper does scrape covers, e.g. from Screenscraper & TheGamesDB, and they are saved in the covers cache directory, they won't be copied to ../psx/media/covers and thus, they're not used as Runcommand loading screens.

Do I miss something? Any help is appreciated.

muldjord · 22 May 2020, 20:40

@Clyde You need to edit your artwork.xml file and include an <output>node that outputs the cover. See the artwork docs here.