RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • muldjordM
      muldjord @A Former User
      last edited by

      @livefastcyyoung There's not a feature to just scrape one title per se. But you can always just copy just 1 rom to a different folder, set the input folder to that folder with '-i'. Then scrape that rom using the different modules 1 by 1, then prioritize the resource type in '~/.skyscraper/dbs/[platform]/priorities.xml' and finish by scraping using '-s localdb'. That would give you exactly the result you are asking for, except it would have created a new gamelist.xml file with just that entry in it, so you'd have to copy the files over manually afterwards.

      If you are in doubt about the features I mention, please check the readme's on github. It's all very well explained in there (although I am aware that it can seem confusing since Skyscraper can do so many things! Hence why I am working on a gui).

      I do not have plans to enable scraping just 1 rom from a folder in a simple way. It doesn't work well with the current design of the code and how it creates the gamelist.xml, sorry.

      ? 1 Reply Last reply Reply Quote 0
      • ?
        A Former User @muldjord
        last edited by

        @muldjord I think I was misunderstood. I wasn’t referring to a single rom. I was referring to scraping a set of roms as we can now but setting a flag to just scrape the genre, release date, etc.

        I’ve found no one database has all elements in great shape so putting together a gamelist by scraping the elements that database does best seems like the next best alternative.

        Hopefully that made sense now.

        muldjordM 1 Reply Last reply Reply Quote 0
        • AnalogHeroA
          AnalogHero
          last edited by AnalogHero

          @LiveFastCyYoung Im not sure that this is what youre looking for but you can set priorities in the priorites.xml, but its only for a whole system. You cant say use info A on rom A and info B on rom B afaik.

          A gui will really help here imo.

          muldjordM 1 Reply Last reply Reply Quote 1
          • muldjordM
            muldjord @A Former User
            last edited by

            @livefastcyyoung Makes perfect sense now. That is not currently supported by Skyscraper but it can do just what you want in a similar way. Have you read up on the '-s localdb' scraping module contained in Skyscraper? You can basically do what you are referring to simply prioritizing the resources correctly. It's all in the readme's. :)

            1 Reply Last reply Reply Quote 0
            • muldjordM
              muldjord @AnalogHero
              last edited by

              @AnalogHero Exactly, beat me to the punch :)

              1 Reply Last reply Reply Quote 0
              • AnalogHeroA
                AnalogHero
                last edited by

                @muldjord Tested c64 with screenscraper module. Doesnt produce any good output, infact none (aborts after 30 fails) although there are c64 games in the screenscraper database online. Any idea?

                muldjordM 1 Reply Last reply Reply Quote 0
                • muldjordM
                  muldjord @AnalogHero
                  last edited by

                  @AnalogHero Basically c64 has a shitstorm of different versions of every single game available (I've literally seen 1 game with 130 different d64 images available, and that's just for the d64 format!). It is not very useful to scrape it with screescraper in my experience. But I'll look into it later and see if something is in fact buggy. :)

                  1 Reply Last reply Reply Quote 0
                  • AnalogHeroA
                    AnalogHero
                    last edited by

                    @muldjord Yeah, true. Scraping with crc doesnt make sense here. Each crack would have a different checksum. Maybe going by name is better. Is this possible with screenscraper?

                    muldjordM 1 Reply Last reply Reply Quote 0
                    • muldjordM
                      muldjord @AnalogHero
                      last edited by muldjord

                      @analoghero Unfortunately not. They only support checksum search and EXACT filename match (which is also one of the parses I currently do). So you'd have to use thegamesdb.

                      1 Reply Last reply Reply Quote 0
                      • muldjordM
                        muldjord
                        last edited by

                        Ok, so I am looking into creating a gui for Skyscraper, and I've been looking for a lightweight gui widget framework that works on the RetroPie distro without too much fuzz. I am currently using the Qt framework for a lot of the stuff in Skyscraper, which is really useful. BUT, I can't for the life of me get Qt to work without X11, which it can, but it just freezes my Pi.

                        So, any suggestions for lightweight gui frameworks? I thought about doing some with ncurses or similar, but I reeeeeally would like to have a graphical one and not a text-based one.

                        mituM 1 Reply Last reply Reply Quote 1
                        • mituM
                          mitu Global Moderator @muldjord
                          last edited by mitu

                          @muldjord Try running some of the samples from AGAR, it seems it can run of SDL and seems to support both framebuffer and GL based contexts.

                          EDIT: Scratch that, It seem the project is not active anymore and the SDL version supported is 1.2.

                          1 Reply Last reply Reply Quote 0
                          • D
                            Deltax5
                            last edited by Deltax5

                            how do I install it. WHere do I put the files? I put the commend and says it can't find gt5-defalt packege. You need to explain where to put the files.

                            How to install Skyscraper

                            Install prerequisites

                            Install this package:

                            • $ sudo apt-get install qt5-default
                            • [enter your 'pi' user password, default is 'raspberry']
                              I skiped this since it doesn't work.

                            Download and compile

                            Create a folder for the Skyscraper source, download the latest release, compile it and install it:

                            • $ cd /home/pi
                            • $ mkdir sources
                            • $ cd sources
                            • $ wget https://github.com/muldjord/skyscraper/archive/2.0.3.tar.gz
                            • $ tar xvzf 2.0.3.tar.gz
                            • $ cd skyscraper-2.0.3

                            Doesn't find these

                            • $ qmake
                            • $ make
                            • $ sudo make install
                            • [enter your 'pi' user password, default is 'raspberry']
                            muldjordM 1 Reply Last reply Reply Quote 0
                            • muldjordM
                              muldjord @Deltax5
                              last edited by

                              You don't put the files anywhere. Just follow the guide. Type in the commands exactly as they are shown but without the '$' and it will work if you are using a RetroPie image. If not, you are doing something wrong.

                              D 1 Reply Last reply Reply Quote 0
                              • D
                                Deltax5 @muldjord
                                last edited by

                                @muldjord i followed it.
                                $ sudo apt-get install qt5-default can't find file
                                i did the rest got to the point where i have to put qmake and make and those can't be found. IDK whats going on. I am typing the commands in puddy with retropi is in f4.

                                muldjordM 1 Reply Last reply Reply Quote 0
                                • muldjordM
                                  muldjord @Deltax5
                                  last edited by muldjord

                                  @deltax5 I can't help you then. Either your system is broken, or you are not typing in the command properly. "sudo apt-get install qt5-default" WILL work on a RetroPie image.

                                  Note: If the prerequisites part won't work for you the rest won't either. So that's why you are having problems with the remaining commands.

                                  D 1 Reply Last reply Reply Quote 0
                                  • fluffypillowF
                                    fluffypillow
                                    last edited by

                                    @muldjord for the gui, have you tried Qt Quick/QML?

                                    D muldjordM 2 Replies Last reply Reply Quote 0
                                    • D
                                      Deltax5 @fluffypillow
                                      last edited by

                                      @fluffypillow whats that sorry im new to this is that a command?

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        Deltax5 @muldjord
                                        last edited by

                                        @muldjord that command works now thanks going to try the rest now

                                        1 Reply Last reply Reply Quote 0
                                        • fluffypillowF
                                          fluffypillow
                                          last edited by

                                          @Deltax5 not, that's a GUI framework. It works without X11 and since Skyscraper is already based on Qt, it should be easier to integrate it with a GUI written in Qt Quick.

                                          1 Reply Last reply Reply Quote 0
                                          • muldjordM
                                            muldjord @fluffypillow
                                            last edited by muldjord

                                            @fluffypillow Qt Quick is just a different way of writing Qt programs. The rendering abstraction is still the same. Qt doesn't require X11, but the eglfs and linufb pipelines don't work for me, so unfortunately it isn't an option. And I don't want X11 as a requirement. But thank you for the suggestion. If I could in fact just write the UI using Qt that would be the best option, but I've spend hours trying to get it to work without X11. So far no luck.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.