• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

Test a new data source for sselph/scraper

Scheduled Pinned Locked Moved Ideas and Development
scrapertestingscreenscraper
43 Posts 16 Posters 15.7k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    sselph
    last edited by sselph 20 Sept 2016, 04:32

    Hi,

    I started some more development on sselph/scraper. I was asked to add some new data sources and the first one I've added is screenscraper.fr. It is a multilingual and multiregional source so will hopefully be nice for international folks. If anyone wants to test it out and see if the new code is working give it a shot.

    I haven't updated the released version but the test code is at https://drive.google.com/drive/folders/0B35kk05UWTo0b2ptT0ZfNGliTWs?usp=sharing

    To use the new source you can do
    scraper -use_gdb=false -use_ss=true

    The new options are -region and -lang. The options for region are us,eu,jp,fr,xx where xx stands for other. You can order them in your preferred order. This will apply to things that have multiple regions and it isn't clear from the ROM what the region is (ie World roms) The options for lang are en,fr,es,de,pt you can provide more that one like -lang="de,en" the data source is mainly french and english at this time.

    Known Issues:

    • I've noticed that a couple systems were missing sha1 data. If you have roms that worked normally but nothing from that system scrape, let me know and I can relay that to the developer. There was a small issue in the import script and it takes a couple minutes to fix.
    • Scumm and name based systems probably won't work
    • PSX and bin/cue systems probably won't work

    Auto-scraper: https://github.com/sselph/scraper
    Donate to Extra-Life 2018 and help save lives: https://goo.gl/diu5oU

    1 Reply Last reply Reply Quote 6
    • T
      tien_huu_1408
      last edited by 21 Sept 2016, 02:40

      @sselph Hi, I am very interested in using your scraper script. Many thanks for your hard work. If possible, could you integrate 3D box arts and other features( MIX Template function (shadow, layer mask, reflections,...) into Retropie setup script? Something is like the way Universal XML scrapper of @screech with Retropie profile (but it works through windows pc).

      Read here: https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/releases/tag/1.5.0.11-Rp

      Thanks a lot and wait for your good news!

      1 Reply Last reply Reply Quote 0
      • S
        sselph
        last edited by 22 Sept 2016, 01:12

        The option to download the 3D box art is there. For the cool looking composite image, technically it would be possible but I haven't looked in to it at all.

        Auto-scraper: https://github.com/sselph/scraper
        Donate to Extra-Life 2018 and help save lives: https://goo.gl/diu5oU

        1 Reply Last reply Reply Quote 0
        • S
          sselph
          last edited by 22 Sept 2016, 05:05

          I went ahead and released the new code. This isn't the default source and disabled by default.

          PS I took a minute to play around with image processing in Go and it is pretty neat. I was able to merge the images together similar to what @screech was doing so definitely possible but I have no clue how I would easily add some template functionality and all that. I am considering refactoring the image portion of the code so will keep this in mind. So if I do add something, it will be easier.

          0_1474520314987_comp_test.png

          Auto-scraper: https://github.com/sselph/scraper
          Donate to Extra-Life 2018 and help save lives: https://goo.gl/diu5oU

          T 1 Reply Last reply 24 Sept 2016, 05:55 Reply Quote 1
          • D
            DrMaxwell
            last edited by DrMaxwell 22 Sept 2016, 07:50

            Nice to see you back at it and screenzone.fr are a great source for the databases. They have some excellent tools for metadata and media on their website.

            Download Scraper for your respective OS

            • Roms Move or copy the scraper to the roms subdirectory for the system you want to scrape. For example: //roms/snes/

            • ImagesPlace matched images into system subfolder named images. For example: //roms//snes/images/

            • Use the command below to scrape metadata and match images without downloading images and keeping them in png format that Raspberry Pi plays nicely with.

            scraper -add_not_found=true -append=true -download_images=false -image_dir="images" -image_path="images" -image_suffix="" -img_format="png" -no_thumb=true -thumb_only=false -thumb_suffix="" -use_gdb=true -use_ss=true -use_ovgdb=true -workers=4

            1 Reply Last reply Reply Quote 0
            • L
              lilbud
              last edited by lilbud 22 Sept 2016, 10:39

              Will you add the libretro thumbnails as a scraper source?

              Creator of the Radiocade: https://retropie.org.uk/forum/topic/6077/radiocade

              Backlog: http://backloggery.com/lilbud

              1 Reply Last reply Reply Quote 0
              • P
                Pyjamarama
                last edited by 22 Sept 2016, 11:27

                Does SScraper (running in Windows) need a special parameter for FBA? (for MAME the -mame parameter is used)

                1 Reply Last reply Reply Quote 0
                • S
                  sselph
                  last edited by 22 Sept 2016, 11:38

                  @DrMaxwell Nice to see you again.

                  @lilbud https://github.com/sselph/scraper/issues/102 The issue was that I assumed data sources would handle both metadata and images so having a source just for images was something that requires more work.

                  @Pyjamarama With my limited knowledge of MAME and arcade I called the flag -mame but it goes for all arcade systems like fba, neogeo, etc.

                  Auto-scraper: https://github.com/sselph/scraper
                  Donate to Extra-Life 2018 and help save lives: https://goo.gl/diu5oU

                  1 Reply Last reply Reply Quote 1
                  • D
                    DrMaxwell
                    last edited by 22 Sept 2016, 12:07

                    @sselph I assume you have documented any and all new options in the help section of the program:

                    scraper --help

                    1. How can people access 3D versus 2D versions of art from the screenzone database?
                    2. I've noticed if all three databases are set to true, the metadata is more complete; is there an order or hierarchical preference that the scraper pulls this metadata from?

                    Thanks for the further development of this tool. Your scraper is the fastest in the business!

                    1 Reply Last reply Reply Quote 0
                    • S
                      sselph
                      last edited by 22 Sept 2016, 12:35

                      Yes -help should have it. -console_img="b" is the default which is 2D boxart -console_img="3b"would get the 3D boxart. Right now the order is gdb, ss, ovgdb but I need to rework some of this to handle specifying the order.

                      Auto-scraper: https://github.com/sselph/scraper
                      Donate to Extra-Life 2018 and help save lives: https://goo.gl/diu5oU

                      D 1 Reply Last reply 22 Sept 2016, 12:48 Reply Quote 1
                      • D
                        DrMaxwell @sselph
                        last edited by 22 Sept 2016, 12:48

                        @sselph I think having a preference would be good. Silly question, but does the scraper check against all three sequentially; for example, if the first doesn't provide the metadata, then it goes to the next database until both image and metadata are found?

                        S 1 Reply Last reply 23 Sept 2016, 01:33 Reply Quote 0
                        • S
                          sselph @DrMaxwell
                          last edited by 23 Sept 2016, 01:33

                          @DrMaxwell Thats the idea but it might stop without getting an image and only metadata. I can't remember.

                          Auto-scraper: https://github.com/sselph/scraper
                          Donate to Extra-Life 2018 and help save lives: https://goo.gl/diu5oU

                          P 1 Reply Last reply 3 Oct 2016, 11:42 Reply Quote 0
                          • T
                            tien_huu_1408 @sselph
                            last edited by 24 Sept 2016, 05:55

                            @sselph I like this mix template for scraping. How could you do this? Could you integrate into your script option? Thanks a lot!

                            1 Reply Last reply Reply Quote 0
                            • T
                              tien_huu_1408
                              last edited by 1 Oct 2016, 09:12

                              @sselph Any update for the script?

                              1 Reply Last reply Reply Quote 0
                              • P
                                paradadf @sselph
                                last edited by paradadf 10 Mar 2016, 15:27 3 Oct 2016, 11:42

                                @sselph it is amazing how fast SS as a source is!!!

                                I notice two things while using it though...

                                • I can't scrape any arcade games (mame, fba, neogeo)
                                • With one specific gba rom I get always the following error message and the scraper stops:
                                2016/10/03 13:36:15 INFO: Checking for new hash.csv.
                                2016/10/03 13:36:15 INFO: hash.csv "e2315a44c3d4c5d200477a109e079f29" up to date.
                                2016/10/03 13:36:15 INFO: Starting: gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip
                                2016/10/03 13:36:16 ERR: error processing gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip: image: unknown format
                                2016/10/03 13:36:16 INFO: Starting: gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip
                                2016/10/03 13:36:16 ERR: error processing gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip: image: unknown format
                                2016/10/03 13:36:16 INFO: Starting: gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip
                                2016/10/03 13:36:16 ERR: error processing gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip: image: unknown format
                                panic: runtime error: invalid memory address or nil pointer dereference
                                [signal 0xc0000005 code=0x0 addr=0x20 pc=0x407bc7]
                                
                                goroutine 28 [running]:
                                panic(0x81cd20, 0xc0420040b0)
                                        /usr/local/go/src/runtime/panic.go:500 +0x1af
                                main.crawlROMs.func2(0xc04340c010, 0xc043408000, 0x0, 0xc042a8dfe0, 0xc0425e9980, 0xc042a971c0)
                                        /home/sselph/go/src/github.com/sselph/scraper/scraper.go:281 +0x247
                                created by main.crawlROMs
                                        /home/sselph/go/src/github.com/sselph/scraper/scraper.go:314 +0x8c7
                                
                                1 Reply Last reply Reply Quote 0
                                • L
                                  lilbud
                                  last edited by 3 Oct 2016, 14:01

                                  Can you enable scraping games that are on an external drive.

                                  Creator of the Radiocade: https://retropie.org.uk/forum/topic/6077/radiocade

                                  Backlog: http://backloggery.com/lilbud

                                  D 1 Reply Last reply 3 Oct 2016, 21:18 Reply Quote 0
                                  • D
                                    DrMaxwell @lilbud
                                    last edited by 3 Oct 2016, 21:18

                                    @lilbud Just put the scraper executable in the same folder as your roms; whether or not the drive is external is irrelevant.

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      sselph
                                      last edited by 4 Oct 2016, 00:30

                                      @paradadf Thanks. I found the issue and have it fixed on github. I wasn't passing the error returned when the bad image was found and caused it to crash when using -missing. I can't reproduce the image error. Maybe that game was fixed? or it was temporary. I also only set this up to scrape console games. I didn't do arcade since it uses names instead of hashes and hadn't done any testing on that.

                                      Auto-scraper: https://github.com/sselph/scraper
                                      Donate to Extra-Life 2018 and help save lives: https://goo.gl/diu5oU

                                      P 1 Reply Last reply 4 Oct 2016, 13:29 Reply Quote 0
                                      • P
                                        paradadf @sselph
                                        last edited by 4 Oct 2016, 13:29

                                        @sselph the rom works perfectly, so I don't understand what's the problem.
                                        Anyway, I just noticed that the letters with accents á, é, ó, etc... aren't being shown on the xml file (those letters are just erased).

                                        1 Reply Last reply Reply Quote 0
                                        • N
                                          Nightfire
                                          last edited by 4 Oct 2016, 18:17

                                          Hi @sselph, happy to hear you're working on the scraper again, I've used it every time I've worked with RetroPie. Not sure if you had seen recent development news of RetroPie, but they are working on emulators for PSP (ppsspp), Nintendo DS (desmune) and GameCube (dolphin). So, I was wondering, will you be adding support for scraping PSP/NDS/GameCube games? Or is there a way to scrape for those systems that I'm just not aware of?

                                          R 1 Reply Last reply 5 Oct 2016, 11:24 Reply Quote 0
                                          • First post
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            This community forum collects and processes your personal information.
                                            consent.not_received