[SOFT] New Scraper in the works
-
Here are a few minor things.
Typo :
Deleing ImagesEcho'ed twice :
Writing Gamelist
Writing GamelistI did 2 tests.
python3 /opt/retropie/supplementary/retroscraper/retroscraper.py --mediadir media/emulationstation/ --systems upright --cleanmedia --relativepaths
Nice to see that all media is now in 1 directory.
Did not expect that but I think it's nice for editing (all in one place)
--mediadir media/emulationstation/ works as expected ;-)Also tried the --mediadir with absolutepath.
This was my test :python3 /opt/retropie/supplementary/retroscraper/retroscraper.py --mediadir /home/pi/RetroPie/roms/upright --systems upright --cleanmedia
I had a look at your explanation, it looks like it doesn't actually do the same as when adding a relative path.
This could be confusing.
Though I see what you want to accomplish.
Putting the gamelist media outside the roms directory with adding the systemname.
I did the above test and used a path with a roms directory.
According to the gamelist the media files would be in :
/home/pi/retropie/roms/upright/upright/
Though the directory isn't created and there is no media.
So there is probably an issue with creating the correct directory.I think you need to remove adding the system name to the absolute path so the absolute path option can be used both ways for placing media outside the roms directory and inside the roms directory.
Or only add the system name if the path is outside the roms directory.I hope I gave you some good feedback.
-
@Folly Hi, thanks for the typo :-)
The problem with having everything in a single directory comes when you're scanning multiple systems at the same type. Imagine you scan snes and md, and ask absolute path for images such as '/home/pi/media' (just as an example)... and you have one file 'xenon 2.zip' in snes, and the exact same file 'xenon 2.zip' in md... now, if I didn't use subfolders (system name) one image would be 'xenon 2-image.zip' and the other would be 'xenon 2-image.zip' ... see where I'm getting?
Another option would be to create 'xenon 2-snes-image.zip' and 'xenon 2-md-image.zip', but I'd be willing to say that from a management persepctive (imagine you had 120k roms in your whole rpie setup) it would be messier having everything in just 1 folder. Thoughts?
I mean, in the end, is just a matter of taste, I could add a '--nosysfolder' flag or something quite easily to avoid creating a folder per system if using relative paths.
Remember that the end objective is to allow you to scrap ALL your roms in a quick way.
-
Good info.
Yes I am only seeing it from perspective scanning only one system.
I now see why you did it this way.If you add a good description it shouldn't be a problem.
Though the creation of the folder did not work with my 2nd test.
What's your opinion.
Edit :
My fault I looked into the wrong folderRetroPie
in stead ofretropie
.
So it does work ! , sorry. -
@Folly maybe have a look at the logs? I know this could happen due to permission issues, sometimes... it should ouptut something like :
CANNOT CREATE DIRECTORY [Your/directory] - ERROR [Explanation] - PLS VERIFY AND TRY AGAIN
And then exit ... so strange is not exiting...have you got he latest version?
-
@kiro said in [SOFT] New Scraper in the works:
so strange is not exiting...have you got he latest version?
It does exit now always, so no problem.
I checked the --mediadir option again with absolute path.
The issue is that when using a path with the folder, for example,RetroPie
(R and P are uppercase) RetroPie isn't used/created butretropie
is. (everything with lowercase)I see you have fixed the other issues ;-)
-
@Folly About the folders, didn't quite understand, so if you put /RetroPie/ will it create /retropie/ or you put /retropie/ and it will not be recognized as /RetroPie/ ? The latter being totally ok as directories under unix/linux are case sensitive..
-
@kiro said in [SOFT] New Scraper in the works:
if you put /RetroPie/ will it create /retropie/
This.
-
@Folly Ok, then probably I'm putting everything lowercase where I should not, going to check right now..
Should be fixed and uploaded to git...
-
I get this :
python3 /opt/retropie/supplementary/retroscraper/retroscraper.py --mediadir /home/pi/RetroPie/roms/upright --systems upright --cleanmedia Starting scraping Scanning Files Scanning Directory ERROR SYSTEM upright: NO ROMS IN DIRECTORY /home/pi/RetroPie/roms/upright/ Scan Done! SCRAPPING ENDED --- Thank you for using retroscraper!!
-
@Folly said in [SOFT] New Scraper in the works:
NO ROMS IN DIRECTORY
Ok, should be fixed, interesting. To avoid scanning through images, I remove anything whose path is the same as the 'media' path, of course, if the mediapath is the same as the system path, well guess what.... I just added the system name to the media path so it should be ok now.
-
@kiro
Did a second update !
I works ok. -
@Folly Thanks for that, would've never thought of doing that path combination!
-
So you like it ?
-
@Folly I like the testing being done, usually when developing there are so many things you give by 'understood' that you tend to forget some of the possibilities a user may input...
-
It's a good process,
It's more fun/productive when there is some interaction, don't you agree ?Btw.
How do we add media that is not recognized right now for our project ? -
@Folly Well, here's the difficult question... how do we do that.. ok there are 2 things to consider:
1- I'll have to manually add the images to the DB, that is not really a concern as long as I understand to which game each image corresponds, which should be fairly easy (thanks to filenames).
2-What do we do if there's already a screenshot available for the game in question, for example, Archimedes. (although I see that most of your images correspond rather to 'systems' than games? And some images seem to be fairly similar (game&watch for instance)
So, if you want to make a small test, pack all the all_in_1 images, which I see are missing from my backend, into a file and I'll try to put them in the right place, once done, we can try to see if scraping gets these images. makes sense?
-
@kiro
I see your point.
Perhaps we can start with an easy one, tigerrz (Tiger R-Zone) which has just 3 games.
I scraped it earlier with your scraper and, as far as I know there are no marquees, images and videos in your DB. -
@Folly Let's try it :-)
-
Here is the googledrive of DTEAM with predefined gamelists, which is used by our project :
https://drive.google.com/drive/folders/1f_jXMG0XMBdyOOBpz8CHM6AFj9vC1R6mYou can just download the file tigerrz.
I just tested my files with your scraper and these have no media and no description and full name etc :
rzbatfor.zip rzindy500.zip rztoshden.zipDon't know if .7z files have media, etc in your DB.
-
@Folly thx!
But, when I try to download (I guess right click then download on folder), I get
Download failed...do I need permissions or am I not downloading what I'm supposed to be downloading?
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.