• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

Skyscraper now officially part of RetroPie, please test

Scheduled Pinned Locked Moved Ideas and Development
skyscraperscraperretropie-setupoptionalexperimental
275 Posts 28 Posters 109.7k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    mitu Global Moderator @iainjh
    last edited by 19 Mar 2019, 05:18

    @iainjh Use the command line parameters to purge/vacuum - https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md.

    1 Reply Last reply Reply Quote 0
    • M
      mitu Global Moderator @2fst4u
      last edited by 19 Mar 2019, 16:51

      @2fst4u @iainjh The fix has been added to the module in the project's repository - update your RetroPie-Setup script and then try again the purge/vacuum functions to see if you still get the error.

      2 1 Reply Last reply 19 Mar 2019, 17:15 Reply Quote 1
      • 2
        2fst4u @mitu
        last edited by 19 Mar 2019, 17:15

        @mitu awesome that works fine now. Thank you very much for your help.

        M 1 Reply Last reply 19 Mar 2019, 17:18 Reply Quote 1
        • M
          mitu Global Moderator @2fst4u
          last edited by 19 Mar 2019, 17:18

          @2fst4u Alright then - thank you again for reporting the bug.
          If you find any bugs/problems/questions with the menu/scriptmodule, please use this topic, I'm always notified when someone posts.

          1 Reply Last reply Reply Quote 1
          • I
            iainjh
            last edited by 20 Mar 2019, 23:46

            Lovely job, working beautifully now. thank you!

            1 Reply Last reply Reply Quote 1
            • 2
              2fst4u
              last edited by 24 Mar 2019, 07:14

              Hi, just a question out of curiosity:

              What happens when you scrape with a second source after having already scraped everything once? Does Skyscraper only scrape for information/files that were missing, or does it re-scrape everything? I see in the docs it mentions a couple of times about using a second source to fill in gaps, but I can't find anywhere what it actually does when you use a second source.

              I also understand there is a way to set priorities for scraping sources. If the answer to the above question is "it only scrapes missing information", then what does this priority do? Does it overwrite data from a lower priority source and leave higher priority data untouched?

              M 1 Reply Last reply 24 Mar 2019, 08:17 Reply Quote 0
              • M
                muldjord @2fst4u
                last edited by muldjord 24 Mar 2019, 08:17

                @2fst4u said in Skyscraper now officially part of RetroPie, please test:

                What happens when you scrape with a second source after having already scraped everything once? Does Skyscraper only scrape for information/files that were missing, or does it re-scrape everything? I see in the docs it mentions a couple of times about using a second source to fill in gaps, but I can't find anywhere what it actually does when you use a second source.

                Please read this: https://github.com/muldjord/skyscraper/blob/master/docs/USECASE.md

                If you gather resources from the same source twice and it didn't finish all of your roms the first time around, it will load all of the ones you already gathered for with that source from the cache until it reaches one it doesn't have cached data for. Then it will start fetching it from the source directly. So in a way, it "skips" the ones you already gathered for using said source. You can tell from the From cache: YES/NO in the output. But it's much easier to use the --startat and --endat options so you don't have to wait for it to load each entry from cache again.

                Concerning gathering using a /second/ source, it will regather for /every/ file. Skyscraper can cache each resource once per source. So when gathering from a second source it will cache all data once more.

                The clever part of this is that you can at /any/ time change anything in your source priorities or your artwork config, and even decide to change your frontend to attractmode. And all you have to do is run a game list generation run again, and you will have a new game list for that frontend using the new configs you've applied. No need to gather new data at all because you already have all of it cached.

                I also understand there is a way to set priorities for scraping sources. If the answer to the above question is "it only scrapes missing information", then what does this priority do? Does it overwrite data from a lower priority source and leave higher priority data untouched?

                There is no overwriting going on anywhere. The priorities are used /only/ when generating the game lists. If you prefer the screenshots from a particular source, you'd prioritize that higher than other sources for the screenshots and so on. You can do this will all resource types for each platform.

                2 1 Reply Last reply 24 Mar 2019, 08:20 Reply Quote 0
                • 2
                  2fst4u @muldjord
                  last edited by 24 Mar 2019, 08:20

                  @muldjord thanks, good to know. One more question about priorities then, what happens to your gamelist.xml if you don't set priorities but you have data from multiple sources scraped?

                  M 1 Reply Last reply 24 Mar 2019, 08:28 Reply Quote 0
                  • M
                    muldjord @2fst4u
                    last edited by muldjord 24 Mar 2019, 08:28

                    @2fst4u said in Skyscraper now officially part of RetroPie, please test:

                    @muldjord thanks, good to know. One more question about priorities then, what happens to your gamelist.xml if you don't set priorities but you have data from multiple sources scraped?

                    Then it uses the default priorities.xml. If the source isn't mentioned it uses timestamps, so the latest gathered resource for any type has priority over the others.

                    2 1 Reply Last reply 24 Mar 2019, 23:04 Reply Quote 0
                    • 2
                      2fst4u @muldjord
                      last edited by 24 Mar 2019, 23:04

                      @muldjord oh cool that's awesome.

                      1 Reply Last reply Reply Quote 0
                      • F
                        frezeen
                        last edited by frezeen 28 Mar 2019, 01:43

                        the windows version used with cmder look perfect
                        https://cmder.net/

                        can u update the windows version to last one?
                        or can u write a guide for to compile it for windows. i dont ask everytime for updated version~~

                        edit: forgot all , i success to compile it with qt 5.

                        Starting scraping run on 2825 files using 1 threads.
                        Sit back, relax and let me do the work! :)

                        ScreenScraper APIv2 returned invalid XML
                        First 256 characters of answer was:

                        #1/2825 (T1) Pass 1 ---- Game '005' not found :( ----

                        0/1/1
                        Elapsed time : 00:00:04
                        Est. time left : 03:11:25

                        User wants to quit, trying to exit nicely. This can take a few seconds depending on how many threads you have running...
                        Request timed out, aborting request...

                        arcadedb work great, screenscraper got error.
                        is this a windows problem or what?
                        version 3.1.5
                        very very thanks for your work

                        M 1 Reply Last reply 28 Mar 2019, 09:04 Reply Quote 0
                        • M
                          muldjord @frezeen
                          last edited by muldjord 28 Mar 2019, 09:04

                          @frezeen I don't know but I recon it's because of screenscraper limits. The network requests work exactly the same under Windows, so I don't think that's the problem. Might also be a firewall problem, might be that you compiled it wrong, I have no idea. You're on your own here basically. Good luck! :)

                          F 1 Reply Last reply 28 Mar 2019, 11:16 Reply Quote 0
                          • F
                            frezeen @muldjord
                            last edited by frezeen 28 Mar 2019, 11:16

                            @muldjord got same error with 3.1.2 from github, its not a compile problem. arcadedb work good and its very fast vs rpi. ill try to found the problem.. anyways i know , no support for windows but thank for the reply , i appreciate it. ill update if i found the problem ;).
                            skyscraper has a debug function? i try with verbose 3 but cant log what happened when got error

                            @all asking for someone in forum has a 3.1.2 windows from github working with screenscraper... this help me to understand if its my problem or not.

                            1 Reply Last reply Reply Quote 0
                            • 2
                              2fst4u
                              last edited by 29 Mar 2019, 20:35

                              I think I've found another issue. I have set my cache folder to be on my USB drive. Here's what works properly:

                              1. Scraping caches the data in the correct place on USB.
                              2. Compiling the gamelist correctly uses the cache for all the data.
                              3. Scraping again will use the cache to skip over things that don't need to be downloaded.

                              But what doesn't work is:

                              1. Vacuum
                              2. Purge

                              It seems the things that are working are using the correct cache location but the vacuum and purge aren't looking at my cache folder set in config.ini, it looks like it's using the default location? Whenever I go into the cache menu it says the size is "56K" which looks to be a baseline default, and it doesn't show all my systems that I want to purge, it only shows the ones that I previously scraped on my SD card.

                              The cache purge works fine in the default location, but now that I want it in a different place I'm still not able to purge.

                              M 1 Reply Last reply 29 Mar 2019, 20:39 Reply Quote 0
                              • M
                                mitu Global Moderator @2fst4u
                                last edited by 29 Mar 2019, 20:39

                                @2fst4u The cache size is calculated from the default location, the script doesn't support custom cache locations.
                                However, the purge and vacuum actions are using Skyscraper and they don't look there - what happens if you run the purge and vacuum from the CLI ?

                                M 1 Reply Last reply 29 Mar 2019, 20:48 Reply Quote 1
                                • M
                                  muldjord @mitu
                                  last edited by muldjord 29 Mar 2019, 20:48

                                  @mitu, @2fst4u If cacheFolder is set in config.ini it will cache data there and purge and vacuum data from there aswell unless a separate -d is set on commandline by the script. There is no difference in the cache location between those operations, they all adhere to cacheFolder in config.ini.

                                  EDIT: Just moved my own cache to a non-default location, changed the cacheFolder to point there and ran both vacuum and purge:all. It works as expected.

                                  1 Reply Last reply Reply Quote 0
                                  • 2
                                    2fst4u
                                    last edited by 2fst4u 29 Mar 2019, 23:26

                                    I'm not able to run the purge command from the script because it only shows one system - GBC, when I have many systems on the USB drive that have been scraped.

                                    Edit: here is my list when I go to "gather resources" or "generate gamelist":

                                    Screenshot_20190330-125152.jpg

                                    But here is my list when I try to "vacuum" or "purge":

                                    Screenshot_20190330-125239.jpg
                                    Screenshot_20190330-125250.jpg

                                    (Excuse the keyboard, sorry)

                                    M 1 Reply Last reply 30 Mar 2019, 06:02 Reply Quote 0
                                    • M
                                      mitu Global Moderator @2fst4u
                                      last edited by mitu 30 Mar 2019, 06:02

                                      @2fst4u Man, those are some big pictures. I realised later that the purge/vacuum actions are only shown in the context of a platform (system) and if you moved the cache before starting scraping, the newly cached platforms will not be available in the selection screen.
                                      You'll have to use the CLI and run the cleanup actions in this case.

                                      I might be able to improve the detection of cache folder, but my reasoning is that if a user is knowledgeable enough to edit the .ini file and move the cache, then they are already familiar with the command line and don't need a GUI.

                                      @muldjord is the cache folder a global options, or can it be also set in a platform section in the configuration file ? I might be able to extract the cache location, but if the .ini can contain more than one location (i.e. default location is X, but for platform amiga it's Y, etc.) , then this would not work.

                                      2 M 2 Replies Last reply 30 Mar 2019, 06:05 Reply Quote 0
                                      • 2
                                        2fst4u @mitu
                                        last edited by 30 Mar 2019, 06:05

                                        @mitu unfortunately when I try a manual purge now I get that artwork.xml error I was having earlier. Would that be related to moving the cache too?

                                        M 1 Reply Last reply 30 Mar 2019, 06:09 Reply Quote 0
                                        • M
                                          mitu Global Moderator @2fst4u
                                          last edited by 30 Mar 2019, 06:09

                                          @2fst4u What is the command line you use for purge/vacuum ?

                                          2 1 Reply Last reply 30 Mar 2019, 06:24 Reply Quote 0
                                          247 out of 275
                                          • First post
                                            247/275
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            This community forum collects and processes your personal information.
                                            consent.not_received