• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

Versatile C++ game scraper: Skyscraper

Scheduled Pinned Locked Moved Ideas and Development
skyscraperscrapergamelist.xmlscrapinggithub
1.6k Posts 113 Posters 1.6m Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • D
    Deltax5
    last edited by 9 Oct 2017, 02:07

    I thank everyone who helped me. It was acting weird at first for some reason. Wasn't tring to be rude or anything. im still a noob and really don't know much about coding other then following the instructions.

    1 Reply Last reply Reply Quote 0
    • D
      Deltax5
      last edited by 9 Oct 2017, 16:58

      Alright idk whats going on. The images are fine just the video's don't show up. nosing around in Winscp they actually downloaded. The thing is that it's not playing. It's in the /home/pi/skyscraper/dbs/gamegear/videos/screenscraper. The video's are labled a bunch of gibberish.
      Do I have to edit the gamelist.xml or something.
      Once again im sorry if I bothered anyone. This is getting annoying for me also.

      M 1 Reply Last reply 9 Oct 2017, 17:31 Reply Quote 0
      • M
        muldjord @Deltax5
        last edited by muldjord 10 Sept 2017, 18:49 9 Oct 2017, 17:31

        @deltax5 Run: "Skyscraper -p gamegear -s localdb --videos --unattend" and it will work.

        Lastly, a couple lessons to take away from this:

        1. When following a howto and a command fails, double check that you wrote the command correctly before asking for help.
        2. If a command still fails, then don't expect the rest of the howto to work. Instead, ask for help from the community, but please remember point 1!
        3. Read the README's thoroughly before asking questions. It's kind of the whole point of a readme, and I spend quite a lot of time writing them. Don't expect me to answer questions explained in the readme's, it's annoying and makes me think that you basically just want me to do the work for you, not putting in the required time to try and understand it yourself.

        I don't mind people asking questions, but it is common courtesy to adhere to these simple points.

        Ok, so with that said, I am glad you got it to work.

        1 Reply Last reply Reply Quote 1
        • B
          BladeHunter @Deltax5
          last edited by 10 Oct 2017, 23:13

          @deltax5 said in Versatile C++ game scraper: Skyscraper:

          you know what my problem is. I notice the 1 file it found was .zip the others where .7zip. for some reason any auto scraper doesn't recognize .7zip.

          Because most emulators don't use 7zip so why would anyone need to scrape them? The scraping is done by MD5, SHA1, CRC and name, if you rename a file, the MD5, SHA1 and CRC don't change, this is why we use these bits of information.

          On the database side where these are scraped from, someone needs to physically link the bits of information to the game, no one links 7zip files because as I said before, how many emulators actually read a 7zipped file?

          D 1 Reply Last reply 11 Oct 2017, 02:18 Reply Quote 0
          • D
            Deltax5 @BladeHunter
            last edited by 11 Oct 2017, 02:18

            @bladehunter it played fine on retro pi. I got my roms in 7z for some reason. Atleast I noticed it and pass it on if someone else has trouble in the future.

            1 Reply Last reply Reply Quote 0
            • K
              kb1ujs
              last edited by 14 Oct 2017, 04:19

              Is there a way to limit what types of resources are scraped? For example, I'm after text resources (descriptions, ratings, etc) but not images.

              M 1 Reply Last reply 14 Oct 2017, 08:23 Reply Quote 0
              • M
                muldjord @kb1ujs
                last edited by 14 Oct 2017, 08:23

                @kb1ujs Videos are off by default (turn on with '--videos'), but images are always scraped and can't be disabled.

                1 Reply Last reply Reply Quote 0
                • A
                  AnalogHero
                  last edited by AnalogHero 31 Oct 2017, 06:46

                  @muldjord Just a small bugreport from what ive learned about videoscreensaver: Skyscrapers gamelist.xml files contain empty video tags looking like this <video />

                  This is a problem at the moment when using videoscreensavers. Could you make those tags optional somehow?

                  M 1 Reply Last reply 5 Nov 2017, 13:38 Reply Quote 1
                  • M
                    muldjord @AnalogHero
                    last edited by 5 Nov 2017, 13:38

                    @analoghero Yes, this has been fixed in 2.0.4 which as just been released: https://github.com/muldjord/skyscraper

                    1 Reply Last reply Reply Quote 2
                    • A
                      AnalogHero
                      last edited by 5 Nov 2017, 15:32

                      @muldjord Thanks alot.

                      1 Reply Last reply Reply Quote 1
                      • M
                        muldjord
                        last edited by 14 Nov 2017, 22:32

                        Skyscraper 2.0.5 released: https://github.com/muldjord/skyscraper

                        • Added support for 'scummvm' platform in scraping module 'thegamesdb' or 'openretro'.
                        • Now only removes 'the' from searchName if longer than 10 chars.
                        • Now always converts underscores to spaces in search- and compare names.
                        • Added edit distance optimization ('the sequel' will now match 'some game: the sequel' 100%).

                        Happy scraping! :)

                        1 Reply Last reply Reply Quote 2
                        • M
                          muldjord
                          last edited by 14 Nov 2017, 23:22

                          A little note on what is going on behind the scenes. I've been looking into creating some sort of GUI for Skyscraper to make it easier to first-time users to get good results. I've had a look at a bunch of sfml and sdl2 based gui frameworks, but not a single one of them are suitable. Believe me, I've been scouring the web for one that fits my requirements, and it's just not gonna happen. They are either too complex, or have too many dependencies to be feasible without making everything unnecessarily complex to install.

                          So, what I've basically decided to do, is to include a 'script-maker' mode when running Skyscraper. It will present the user with a bunch of questions, such as "What platform do you wish to scrape?" and other relevant stuff. And when all questions have been answered, it will create a script or possibly config file (or both) and ask if you want to run it.

                          Comments on this approach will be appreciated.

                          1 Reply Last reply Reply Quote 2
                          • A
                            AnalogHero
                            last edited by 15 Nov 2017, 16:57

                            @muldjord Maybe you can go with a shell script gui like the retropie-setup. Dont know if there is any framework there?

                            M 1 Reply Last reply 15 Nov 2017, 18:48 Reply Quote 1
                            • M
                              muldjord @AnalogHero
                              last edited by 15 Nov 2017, 18:48

                              @analoghero Yes, I am also considering that. Although I am partial to the "questionnaire" method since it can then be implemented directly in the software as I see fit. Thanks for suggesting it.

                              B 1 Reply Last reply 16 Nov 2017, 11:21 Reply Quote 0
                              • B
                                bleuge @muldjord
                                last edited by 16 Nov 2017, 11:21

                                @muldjord I compiled this in Windows using Cygwin some months ago, but got many problems running it. Do you think it is too much effort to make it run ok in Windows, scrapping is lot of faster and easier that connectin to Pi, etc. Thanks!

                                About commandline, I'm happy with it, but I know many users will be too complicated, an interactive shell as you suggest it will ok, while a graphical gui is thought and implemented. If we could run this in Windows i'll do a gui myself if it's needed. Regards!

                                M 1 Reply Last reply 16 Nov 2017, 16:32 Reply Quote 0
                                • M
                                  muldjord @bleuge
                                  last edited by muldjord 16 Nov 2017, 16:32

                                  @bleuge You can easily cross-compile it for Windows. Just download and install Qt 5 from the official site (open source version). When installing, put a checkmark in the latest mingw32 version, I don't support MSVC++ although it might work, not sure. When it's installed you'll find a Qt specific command prompt in your start menu. From there navigate to the Skyscraper source folder and run:
                                  $ qmake
                                  $ mingw32-make release

                                  That will give you a compiled version in a subfolder called "release". Then you need to copy the needed dll's from the qt5 install folder into the release folder and start an ordinary command prompt there. Then you should be able to run it.

                                  I thiiiink it will create the .skyscraper data folder inside "C:\Users[username]... somewhere". Just keep in mind that there are some minor Linux specific stuff in Skyscraper's "make install" command that will copy relevant files to the .skyscraper folder. So you'd be missing those. But I believe I auto-create empty ones...

                                  So yeah, shouldn't be too hard, but it's not really something I am going to look into myself at this time. Happy hacking! :)

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    bleuge
                                    last edited by 17 Nov 2017, 08:53

                                    Thanks for tips, I already compiled ok in Cygwin (after some work with qt5). Yesterday i updated it and recompiled it. Copied the needed DLLs and ...


                                    Running Skyscraper v2.0.5a by Lars Muldjord

                                    Please set a valid platform with '-p [platform]'
                                    Check '--help' for a list of supported platforms, now exiting...

                                    What i meant about windows, was the folder/linux structure, making it portable or something like that. I'll try with some roms to see if i can make it working normally.
                                    Thanks!

                                    M 1 Reply Last reply 17 Nov 2017, 09:37 Reply Quote 0
                                    • M
                                      muldjord @bleuge
                                      last edited by 17 Nov 2017, 09:37

                                      @bleuge I am not sure what you mean. It IS cross-platform compatible (and clearly runs just fine looking at your quoted text). It uses the home folder variable to specify where to save settings, which depends on you OS. So if you are running Windows it will save stuff in the Users folder as expected. Please elaborate.

                                      As I mentioned the only thing that isn't cross-platform is the "make install" part which I've made more convenient and automated on Linux. But it shouldn't really be necessary for it to run properly on Windows.

                                      B 1 Reply Last reply 27 Nov 2017, 15:33 Reply Quote 0
                                      • B
                                        bleuge @muldjord
                                        last edited by bleuge 27 Nov 2017, 15:33

                                        @muldjord thanks for your answers, i'll elaborate.
                                        I have subfolders with roms in c:\roms

                                        Skyscraper.exe -p atari2600 -i c:\roms\2600 -t 8 -d c:\roms\00skdb --nobrackets --verbose

                                        I got

                                        Folder '/home/myuser/.skyscraper/c:\roms\2600/images' doesn't exist, trying to create it... Failed! Please check path and permissions

                                        I've looked for .skyscraper file and found that Skyscraper created folders in

                                        c:\home\myuser.skyscraper\dbs
                                        c:\home\myuser.skyscraper\import...various subs here

                                        I don't have any problem Skyscraper creating files and folders whenever it needs it. But well, i can't make it work under Windows.

                                        Maybe i can create a config.ini file in the folder of the executable?

                                        Thanks for your time, this scraper is very very good! Your idea of the localdb is wonderful.

                                        M 1 Reply Last reply 3 Dec 2017, 19:22 Reply Quote 0
                                        • M
                                          muldjord @bleuge
                                          last edited by 3 Dec 2017, 19:22

                                          @bleuge I see a potential problem. Please try writing your paths with forward slashes instead of backward slashes.

                                          I do find it odd that it looks for /home/myuser though. I use the builtin user home from Qt, which should adjust for Windows and use the Users folder.

                                          B 1 Reply Last reply 5 Dec 2017, 08:44 Reply Quote 0
                                          • First post
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            This community forum collects and processes your personal information.
                                            consent.not_received