RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      SammyBoy
      last edited by

      @muldjord i send pictures and the command I use is Skyscraper -p gba -s import

      1 Reply Last reply Reply Quote 0
      • S
        SammyBoy
        last edited by

        Visit my Twitter Take a look at Sammy boy (@samsaju04_boy): https://twitter.com/samsaju04_boy?s=09

        1 Reply Last reply Reply Quote 0
        • S
          SammyBoy
          last edited by

          @muldjord the pictures are on Twitter - the first thing you see

          1 Reply Last reply Reply Quote 0
          • muldjordM
            muldjord
            last edited by muldjord

            Please see next post.

            muldjordM 1 Reply Last reply Reply Quote 0
            • muldjordM
              muldjord @muldjord
              last edited by muldjord

              Please try rerunning it with 'Skyscraper -p gba -s import --updatedb'. That might be the problem.

              1 Reply Last reply Reply Quote 0
              • muldjordM
                muldjord
                last edited by

                Skyscraper 2.2.6a released: https://github.com/muldjord/skyscraper

                • Now always sets '--updatedb' when using 'import' scraping module

                Get this release, then it'll work without '--updatedb'

                S 1 Reply Last reply Reply Quote 0
                • S
                  SammyBoy @muldjord
                  last edited by

                  @muldjord Thank you so much!

                  1 Reply Last reply Reply Quote 0
                  • juraJ
                    jura
                    last edited by

                    Hi @muldjord , I used your software for the scraping the images for roms and I loved it.

                    Seems to work almost OK, sometimes when game names are closely the same e.g. gameName 1 and gameName 2 it messes the description for these games. There's might be a chance that these descriptions are messed up in the location where it get it from at the first place, dunno.

                    When I checked the Lakka which is another Retro gaming fontend seems that they get images and thimbnails from the libretro. Dunno is it only based on the name of the game.

                    Any chance to use also this location which is used in Lakka for the images to the games? :)

                    https://github.com/libretro/libretro-thumbnails/tree/master/

                    There's nothing unclear about IT

                    muldjordM 1 Reply Last reply Reply Quote 0
                    • muldjordM
                      muldjord @jura
                      last edited by muldjord

                      @jura It doesn't mess them up, rather it sometimes gets faulty results from the sources, which is a side-effect of the highly automated way Skyscraper works. For instance, if you have a relatively unknown game called "Star Blob" and a game called "Star Fox", it will quite likely get a correct hit for "Star Fox" since it's a well-known game. BUT, it will probably also return the data for "Star Fox" when it searches for "Star Blob" since that's how the source search engines work. So it doesn't mess them up, it just, sometimes, gets faulty results where it seems like the previous game has "spilled over" into the next entry, when in fact it only does so because the names are closely related. In this example by the "Star" in the name.

                      You can force it to not accept these by changing the minimum match percentage with the '-m' flag on command line. :) It defaults to 50%, which means that "Star Blob" and "Star Fox" gives a positive result.

                      I can't use a github repository for scraping, simply because I do not have the permission from github to do so. Github is not meant to be used for game media scraping and I highly doubt they would allow it if they knew people used it for that purpose.

                      EDIT: On a sidenote, I plan on completely rewriting the way matches are achieved, so matches will improve in a future version. No ETA though.

                      1 Reply Last reply Reply Quote 2
                      • muldjordM
                        muldjord
                        last edited by muldjord

                        A huge thank you to @chipsnblip for spreading the word about Skyscraper on reddit. I notice these things, and I appreciate it a lot! :)

                        chipsnblipC 1 Reply Last reply Reply Quote 3
                        • chipsnblipC
                          chipsnblip @muldjord
                          last edited by

                          @muldjord it's my pleasure to help! i've been lurking in this thread since day one, and to tell the truth haven't even used skyscraper yet (but plan on it very soon-- life keeps getting in the way). i was just so impressed watching one person's hard work and dedication to a project of this nature. seeing people pour their heart into the retro gaming community is truly inspiring.

                          1 Reply Last reply Reply Quote 4
                          • muldjordM
                            muldjord
                            last edited by muldjord

                            On a related note: If any of you good people feel Skyscraper is a project you want to support, then a good way to do so is simply to spread the word about it. :) I'm not exactly a PR guru myself, so I'll concentrate on the coding part of it. When people do lend a hand it means quite a lot, and is quite motivating.

                            As you know Skyscraper is completely open source and free to use (I'm a huge advocate of openness in general). I do this work entirely in my spare time simply because I enjoy working on it. Part of that is watching the community actually make use of the project. So a thank you to all of you for simply using it is in order aswell. :)

                            ClydeC 1 Reply Last reply Reply Quote 3
                            • ClydeC
                              Clyde @muldjord
                              last edited by

                              @muldjord said in Versatile C++ game scraper: Skyscraper:

                              As you know Skyscraper is completely open source and free to use (I'm a huge advocate of openness in general).

                              Let's drink to that. o/ᵁ

                              muldjordM 1 Reply Last reply Reply Quote 2
                              • muldjordM
                                muldjord @Clyde
                                last edited by

                                @clyde *clink* ᵁ\o

                                1 Reply Last reply Reply Quote 2
                                • AnalogHeroA
                                  AnalogHero
                                  last edited by

                                  I really think skyscraper should included in Retropie-Setup. Im sure i mentioned this in a different topic.

                                  1 Reply Last reply Reply Quote 3
                                  • AnalogHeroA
                                    AnalogHero
                                    last edited by

                                    @muldjord Could it be that version 2.2.5 is a lot slower than previous versions? Just added a few whdload games and rescraped my amiga folder. I thought it crashed because the first output took sol long. Scraping with localdb takes the same amount of time. Over 10 minutes for 61 .uae files...

                                    muldjordM 1 Reply Last reply Reply Quote 0
                                    • muldjordM
                                      muldjord @AnalogHero
                                      last edited by muldjord

                                      @analoghero It is slower yes, which is caused by the new compositor rendering the shadow. I've optimized it a bit, but it could probably be optimized even further in future releases. I've basically implemented my own custom shadow renderer based on gauss kernel, which takes a bit of time to render... If you want to speed it up, try disabling the drop shadow in artwork.xml (chech the documentation for that).

                                      The first results sometimes (not always) take a long time, that's normal (also for pre-2.2.5 versions), there's nothing I can do to change that since it is caused by the Qt frameworks network manager.

                                      EDIT: 10 minutes for 61 uae files means something is wrong on your end. Can't tell you what exactly. It's a lot faster on my Pi 3 (not overclocked).

                                      1 Reply Last reply Reply Quote 1
                                      • AnalogHeroA
                                        AnalogHero
                                        last edited by AnalogHero

                                        @muldjord Cant say whats wrong... just tested it with psx as platform (18 games here). Took 4:06 :( This was faster in the past for sure.

                                        (ES is closed, nothing is overclocked or running too hot)

                                        Shame that i dont have a older version, would test it.

                                        Edit: When --pretend ist set, its fast. When it dont find any matches (using wrong scraper) its also fast.

                                        1 Reply Last reply Reply Quote 0
                                        • AnalogHeroA
                                          AnalogHero
                                          last edited by

                                          @muldjord Ok tested it. Compiled old version 2.2.1 and scraped my psx system with 18 games in it. Result: Took 12 seconds

                                          ---- Scraping run completed! YAY! ----
                                          Writing 366 resources to local database, please wait... Success!
                                          Now writing '/home/pi/RetroPie/roms/psx/gamelist.xml'... Success!!!
                                          
                                          ---- And here are some neat stats :) ----
                                          Total completion time: 00:00:12
                                          
                                          Average search match: 93%
                                          Average entry completeness: 83%
                                          
                                          Total number of games: 18
                                          Successfully scraped games: 17
                                          Skipped games: 1 (Filenames saved to 'skipped-localdb.txt')
                                          

                                          With 2.2.5:

                                          ---- Scraping run completed! YAY! ----
                                          Writing 366 resources to local database, please wait... Success!
                                          Now writing '/home/pi/RetroPie/roms/psx/gamelist.xml'... Success!!!
                                          
                                          ---- And here are some neat stats :) ----
                                          Total completion time: 00:03:58
                                          
                                          Average search match: 93%
                                          Average entry completeness: 83%
                                          
                                          Total number of games: 18
                                          Successfully scraped games: 17
                                          Skipped games: 1 (Filenames saved to 'skipped-localdb.txt')
                                          
                                          1 Reply Last reply Reply Quote 0
                                          • muldjordM
                                            muldjord
                                            last edited by

                                            I will do some testing on this when I get the time. In the meantime, please post your artwork.xml file.

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.