Take and Scrape Your Own Screenshots
-
@herb_fargus said in Take and Scrape Your Own Screenshots:
Also have you tested your code changes ;)
hahaha... after one acquires a bad reputation it's hard to miss it...
yes! I tested my code! :D
I'm glad it works for you too. -
@herb_fargus I'm currently working on a simpler way to achieve what we want here (take and scrape your own screenshots). No need for those
sselph scraper options complexity, just take your screenshots as you like and then execute a script to put the most recent screenshot as the scrape image. Hope to show something useful soon...runcommand-on{start,end}.sh
-
@herb_fargus
Well...
After reading this series of brainstorming posts, reading the gamelist.xml doc, and taking a look at somegamelist.xml
files, I found a simpler solution for what we want.Let me talk about another method...
user point of view
Take screenshots of a game and set the most recent screenshot as the emulationstation image for this game.
prerequisites
auto_screenshot_filename = false
in globalretroarch.cfg
. It will be the flag to turn on/off the "scrape your own screenshots" functionality (it can be confusing, because "false" means "on", we can think about another flag later.).screenshot_directory
must be set to some directory inretroarch.cfg
(system specific or global, system specific takes precedence).- it was made to work after a scraping (the
$HOME/.emulationstation/gamelists/$system/gamelist.xml
or/etc/emulationstation/gamelists/$system/gamelist.xml
files must exist).
how it works
Summing up: put the game screenshot full path file name in the respective
<image>
entry, replacing the old content. If the game isn't present ingamelist.xml
, create an entry for it.Let's take a look at the
runcommand-onend.sh
:
[EDIT: the script below is just a "proof of concept". A robust and updated version is in https://raw.githubusercontent.com/meleu/share/master/screeper.sh]#!/bin/bash echo "--- start of $(basename $0) ---" >&2 readonly system="$1" readonly full_path_rom="$3" readonly retroarch_cfg="/opt/retropie/configs/all/retroarch.cfg" readonly system_ra_cfg="/opt/retropie/configs/$system/retroarch.cfg" readonly gamelist="$HOME/RetroPie/roms/$system/gamelist.xml" readonly gamelist1="$HOME/.emulationstation/gamelists/$system/gamelist.xml" readonly gamelist2="/etc/emulationstation/gamelists/$system/gamelist.xml" rom="${full_path_rom##*/}" rom="${rom%.*}" scrap_img="$rom.png" source "/opt/retropie/lib/inifuncs.sh" iniConfig ' = ' '"' # only go on if the auto_screenshot_filename is false iniGet "auto_screenshot_filename" "$retroarch_cfg" if ! [[ "$ini_value" =~ ^(false|0)$ ]]; then exit 0 fi # getting the screenshots directory # try system specific retroarch.cfg, if not found try the global one iniGet "screenshot_directory" "$system_ra_cfg" screenshot_dir="$ini_value" if [[ -z "$screenshot_dir" ]]; then iniGet "screenshot_directory" "$retroarch_cfg" screenshot_dir="$ini_value" if [[ -z "$screenshot_dir" ]]; then echo "You must set a path for 'screenshot_directory' in \"retroarch.cfg\"." >&2 echo "Aborting..." >&2 exit 1 fi fi # if there is no screenshot named "ROM Name.png", we have nothing to do here if ! [[ -f "$screenshot_dir/$scrap_img" ]]; then echo "There is no screenshot for \"$rom\". Exiting..." >&2 exit 0 fi # if there is no "customized gamelist.xml", try the user specific, # if it fails, get the global one if ! [[ -f "$gamelist" ]]; then echo "Copying \"$gamelist1\" to \"$gamelist\"." >&2 if ! cp "$gamelist1" "$gamelist" 2>/dev/null; then echo "Failed to copy \"$gamelist1\"." >&2 echo "Copying \"$gamelist2\" to \"$gamelist\"." >&2 if ! cp "$gamelist2" "$gamelist" 2>/dev/null; then echo "Failed to copy \"$gamelist2\"." >&2 echo "Aborting..." >&2 exit 1 fi fi fi # the <image> entry MUST be on a single line and match the pattern: # anything followed by rom name followed or not by "-image" followed by dot followed by 3 chars old_img_regex="<image>.*$rom\(-image\)\?\....</image>" new_img_regex="<image>$screenshot_dir/$scrap_img</image>" if grep -q "$old_img_regex" "$gamelist"; then sed -i "s|$old_img_regex|$new_img_regex|" "$gamelist" else # oh! there is no entry for this game yet! gamelist_entry="\\ <game id=\"\" source=\"\">\\ <path>$full_path_rom</path> \\ <name>$rom</name> \\ <desc></desc> \\ $new_img_regex \\ <releasedate></releasedate> \\ <developer></developer> \\ <publisher></publisher> \\ <genre></genre> \\ </game>" sed -i "/<\/gameList>/ s|.*|${gamelist_entry}\n&|" "$gamelist" fi echo "--- end of $(basename $0) ---" >&2
limitations
Currently this is a kind of "proof of concept". The limitations below can be overcome if we feel that this is the way to achieve what we want.
[The updated and more robust version of the script is available here: https://raw.githubusercontent.com/meleu/share/master/screeper.sh]- This method
onlychanges the<image>
entry of a particular game.So, if this game is NOT present in the gamelist.xml, nothing happens.[EDIT: if the game is not present in gamelist.xml, the script creates an entry for it] - The
<image></image>
entry in thegamelist.xml
must be in a single line (it seems to be the default, so probably we don't have to worry about it). - The original image filename in the
<image>
entry must be named asROM Name.ext
orROM Name-image.ext
(ext
can be any 3 characters, eg: png, bmp, jpg, etc), otherwise thesed
command won't replace it. [EDIT: it wasn't a problem in my first tests here] - After a succeeded image changing, the respective
<image>
entry will have a full path to the image. It can be an inconvenience if the user wants to copy the gamelist.xml between computers (IMHO it's not so important. Besides that it probably won't be a problem to those who use thepi
user).
"I didn't like how it looks! I want my old scrapes back!"
Change the
auto_screenshot_filename
to true in/opt/retropie/configs/all/retroarch.cfg
and then delete the system specificgamelist.xml
that is at the system roms directory (example for SNES:~/RetroPie/roms/snes/gamelist.xml
). Now emulationstation will get the gamelist from$HOME/.emulationstation/gamelists/$system/gamelist.xml
.Restart emulationstation and you'll get back your old scrapes.
where to go from here?
After learn how to use the scraper with-append=true
option (no success on my first tests) and if it proves to be good enough, we can evolve it to what I think would be a cool feature: scrape with screenshots only the games that aren't scraped. In other words: if the user is playing a non-scraped game and takes screenshots, then scrape this game with the most recent taken screenshot.I think its done! :D
Maybe some minor bug fixes (if we found some) and minor improvements. -
Yes, I tested the code. ;-)
If you want to avoid the copy'n'paste process you can get it with this command:
wget "https://raw.githubusercontent.com/meleu/share/master/screeper.sh"
And then rename it to
runcommand-onend.sh
. -
@herb_fargus
Man! I've updated the script to create an entry for the games that aren't present ingamelist.xml
. I updated my big post above including the changes (yes! I tested my code!).You can get the updated code at github:
wget "https://raw.githubusercontent.com/meleu/src/master/screeper.sh"
Dude, it was really fun to accomplish this task. Thanks for this challenge!
When you have time to test it (and agree with my method) maybe we can edit the wiki to use this method. It's up to you. (If you agree I can edit it next night).
Cheers!
-
@meleu so where does metadata fit in with this ;)
-
@herb_fargus
one of the prerequisites is:- it was made to work after a scraping (the
$HOME/.emulationstation/gamelists/$system/gamelist.xml
or/etc/emulationstation/gamelists/$system/gamelist.xml
files must exist).
I think it's better to leave the scraping up to the user, because it is very time consuming to be in
runcommand-onend.sh
. - it was made to work after a scraping (the
-
@meleu that is true. and the download-images-false flag with sselphs scraper can pull the metadata without the images- or I guess they may have to scrape metadata first unless its appended.
-
I think the code as is may have a bug. I haven't run it but just reading it over and following the logic.
ROMs named:
Advanced Dungeons & Dragons - Eye of the Beholder (USA).smc
Zool - Ninja of the 'Nth' Dimension (USA).smc
Will scrape to something like:
<image>./images/Zool - Ninja of the 'Nth' Dimension (USA).png</image>
so the regex used to find and replace the old entry won't work and a new entry will be appended.That brings up a second bug with the appending code. If a rom is named with a quote in it will probably parse okay but an ampersand would be considered invalid and skipped:
<path>./Advanced Dungeons & Dragons - Eye of the Beholder (USA).smc</path>
instead of
<path>./Advanced Dungeons & Dragons - Eye of the Beholder (USA).smc</path>
If the file named happened to have a < or > things would be very bad
test <US>.bin
would convert to:
<path>test <US>.bin</path>
and the extra<US>
would throw off the parser.
it should be something like
<path>test <US>.bin</path>
Using something like passing the file name through
sed "s/\&/\&/g;s/>/\>/g;s/</\</g;s/'/\'/g"
might work to convert things to a similar the escaped XML. -
Something else that may be an issue, but I'm not 100% certain, is having files with names like
test [!].bin
this seems like it would create a grep statement likegrep "test [!].bin"
which means the [ and ] wouldn't be taken literally. Also the ! probably causes issues. -
@sselph good points! I'll fix it!
Thanks! -
@sselph
Thanks a lot for your comments! I googled a bit about what you said and I think I've fixed it.Sharing some info...
From what I understand we have three levels of interpretation to take care of:
- shell
- XML
- sed/grep
shell
As we are getting the file names from
runcommand.sh
, they are already valid. I know that file names with strange characters (eg.: percent sign) don't work in emulationstation... So, there's a lot of filtering before the filename gets in theruncommand-onend.sh
.XML
As far as I know (from this source), we have five characters to take care:
" " ' ' < < > > & &
I did it this way:
function echo_xml_safe() { output="$(sed 's#\&#\&#g' <<< "$@")" output="$( sed " s#\"#\"#g s#'#\'#g s#<#\<#g s#>#\>#g" <<< "$output" )" echo "$output" }
[EDIT:
echo
has a limitation: it won't print these exact strings:-e
or-n
. And the use of theprintf
alternative would add another series of strings to be escaped. Well... the "scrape your own screenshot" trick won't work for roms named-e.ext
or-n.ext
. We can survive with it... :-) ]sed/grep
For a safe sed/grep used this function:
function echo_regex_safe() { echo "$(sed 's#[][&\^]#\\&#g' <<< "$@")" }
-
@herb_fargus
I've created another wiki page describing my method.https://github.com/RetroPie/RetroPie-Setup/wiki/Take-and-Scrape-Your-Own-Screenshots-(method-2)
I took the freedom to take several parts of your text. I hope you don't mind. :-)
A huge thanks to @sselph for predict the bugs!
Cheers!
-
Now I would like to share some info I obtained during the time I was working on this script for future references.
I made intensive tests with several
really% $trange&char*cter's [in] ^ROM\ <file> "names".ext
and I've got some conclusions:-
Linux file systems (and most of the UNIX-like file systems) are very permissive with characters used in filenames. Actually, the only two reserved chars for filenames in POSIX systems are
/
and null char (source). -
Many of the files with strange chars don't even launch the emulator.
-
Some failed launches occur due to ES limitations: the ES try to launch a game and I can't see the runcommand dialog, then it goes back to ES.
-
Aloshi says in the gamelists documentation: "One thing to be aware of: the EmulationStation text rendering code doesn't currently support Unicode. If I fix this in the future, it will probably use UTF-8. For now, you'll just have to convert names and descriptions to ASCII. Sorry!"
-
Some other failed launches occur due to runcommand limitations: I can see the runcommand dialog, then it goes back to ES.
-
if the rom name has two or more consecutive spaces, the game launches OK, but the RetroArch isn't able to create a screenshot for this game. Everything seems to be OK, I play the game, I take screenshots with no failed messages, but when I look at the screenshots directory the image isn't there. It seems to be a RetroArch bug.
All the games I managed to launch AND take screenshot, my scrape method worked fine.
Example of strange rom name that worked fine:
mega$ man[vs]dr& willy\< USA >[!].nes
-
-
@meleu I don't like the idea of two wiki pages explaining the same thing. It would make more sense to just have a heading in the original wiki page for each method. The less pages we can have in the wiki the better as its getting quite large
-
@herb_fargus
I'm gonna try to edit and put both methods in one page. -
@herb_fargus
When trying to merge the common info I realized a few things that I would like to know your opinion:retroarch.cfg
method 1
The
runcommand-onstart.sh
always force the configsauto_screenshot_filename = "false" screenshot_directory = "$HOME/RetroPie/roms/$system/images"
Pros:
- no need to manually edit the
retroarch.cfg
. - simplifies the
scraper
command.
Cons:
- if the user want to configure these options with different values in
retroarch.cfg
the script will always overwrite the changes with those from the script. It can bring confusion. - if the user wants to disable the script, he must to delete the
runcommand-onstart.sh
AND manually edit every single systemretroarch.cfg
(therefore, it invalidates the first pro).
method 2
The user must to edit
retroarch.cfg
manually.Pros:
- Respect what is in
retroarch.cfg
. - Only user comfortable with file editing will feel confident to do it. (ok, ok... putting this as a pro is a matter of opinion)
Cons:
- Inexperienced users will avoid it. (the very same thing of the 2nd pro, but put in other point of view).
My question
Do you want to keep that
runcommand-onstart.sh
in the wiki?[EDIT]
We can instruct the users to manually set the configs in the system'sretroarch.cfg
as expected by the method 1 instead of using the script. - no need to manually edit the
-
@meleu yes I think we should keep the runcommand option, I didn't think about the whole configs staying thing but I think we could technically add a function to remove the configs with onend or really tbh if people are going to be using this method I would expect they have a basic competency in editing files and if they dont I would expect they have a basic enough competency to learn.
I think we've coddled people too much with RetroPie . The pi is for learning and doing cool things and that requires taking the time to learn it and make mistakes.
I think both approaches are valid and as you mentioned both have their pros and cons. We should allow the user to make a decision on what will work best for them, and if they become confused on it they should take the time to troubleshoot and learn why so they can actually learn.
-
@herb_fargus
Merged the two methods in the wiki. I hope you like it.
[EDIT: if you have you have time, could you check it for grammar mistakes? English is not my native language.] -
@meleu looks great, thanks for doing that :)
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.