Take and Scrape Your Own Screenshots

herb_fargus

@meleu those are good tips to keep in mind. I suppose that we could set the screenshot directory to just one screenshot directory (perhaps even just leave the default) and then use sselphs scraper to point to that, eliminates another config step but I like the idea of having all the images and gamelists in each rom folder because you only have to scrape once and then you can just backup/copy the roms folder complete with images etc with no need to scrape again.

also if you use the -skip_check=true and -use_gdb=false would that not eliminate the metadata for the detailed view so you only end up having an image?

I guess we'd have to get feedback to see what people prefer

herb_fargus

@hiulit I've created a wiki page for it with the latest working code. See first post for link

hiulit

@herb_fargus nice! Thanks!

meleu

@herb_fargus
[EDIT: I've edited the wiki including those two options for a faster scraper execution. I intentionally didn't edit the runcommand-onstart.sh to let it as you prefer.]

Some points on the wiki runcommand-onstart.sh:

You can set the iniConfig just once and then pass the file to the iniSet in the 3rd argument.
the function make_screenshot_dir can be replaced by mkdir -p "$imgdir" (the -p option means "no error if existing, make parent directories as needed").
There is no need to explicitly call retroarchIncludeToEnd, since it is automatically executed when editing any file with a name that ends with "retroarch.cfg" (inifuncs feature).

The runcommand-onstart.sh can be simplified to this:

#!/usr/bin/env bash

system="$1"
imgdir="$HOME/RetroPie/roms/$system/images"
configdir="/opt/retropie/configs" 

mainretroarch="$configdir/all/retroarch.cfg"
systemretroarch="$configdir/$system/retroarch.cfg"

source "/opt/retropie/lib/inifuncs.sh"

iniConfig " = " '"'

# If there is no auto screenshot setting in the main retroarch.cfg add it
if ! grep -q "auto_screenshot_filename" "$mainretroarch"; then
    iniSet "auto_screenshot_filename" "false" "$mainretroarch"
fi

mkdir -p "$imgdir"

# If there is no system based screenshot directory defined then define it in the system based retroarch.cfg
if ! grep -q "screenshot_directory" $systemretroarch; then
    iniSet "screenshot_directory" "$imgdir" "$systemretroarch"
fi

herb_fargus

@meleu you still didn't address:

also if you use the -skip_check=true and -use_gdb=false would that not eliminate the metadata for the detailed view so you only end up having an image?

meleu

@herb_fargus I was so focused on the images thing and didn't give a proper attention to this important detail. Sorry.

Re-edited the wiki excluding these options.

herb_fargus

@meleu I personally don't care about metadata but I know lots of others do.

Once gridview is incorporated I think those extra flags will be more useful (though I presume that it may be possible to theme metadata in with that at some point) might be a cool design to have a box on the side that changes with each game. I'll have to play with that in the future.

Also have you tested your code changes ;)

herb_fargus

@meleu I tested your code changes and they work :) thanks for the refactor. I've updated the wiki page accordingly.

meleu

@herb_fargus said in Take and Scrape Your Own Screenshots:

Also have you tested your code changes ;)

hahaha... after one acquires a bad reputation it's hard to miss it...

yes! I tested my code! :D
I'm glad it works for you too.

meleu

@herb_fargus I'm currently working on a simpler way to achieve what we want here (take and scrape your own screenshots). No need for those ~~runcommand-on{start,end}.sh~~ sselph scraper options complexity, just take your screenshots as you like and then execute a script to put the most recent screenshot as the scrape image. Hope to show something useful soon...

meleu

@herb_fargus
Well...
After reading this series of brainstorming posts, reading the gamelist.xml doc, and taking a look at some gamelist.xml files, I found a simpler solution for what we want.

Let me talk about another method...

user point of view

Take screenshots of a game and set the most recent screenshot as the emulationstation image for this game.

prerequisites

auto_screenshot_filename = false in global retroarch.cfg. It will be the flag to turn on/off the "scrape your own screenshots" functionality (it can be confusing, because "false" means "on", we can think about another flag later.).
screenshot_directory must be set to some directory in retroarch.cfg (system specific or global, system specific takes precedence).
it was made to work after a scraping (the $HOME/.emulationstation/gamelists/$system/gamelist.xml or /etc/emulationstation/gamelists/$system/gamelist.xml files must exist).

how it works

Summing up: put the game screenshot full path file name in the respective <image> entry, replacing the old content. If the game isn't present in gamelist.xml, create an entry for it.

Let's take a look at the runcommand-onend.sh:
[EDIT: the script below is just a "proof of concept". A robust and updated version is in https://raw.githubusercontent.com/meleu/share/master/screeper.sh]

#!/bin/bash
echo "--- start of $(basename $0) ---" >&2

readonly system="$1"
readonly full_path_rom="$3"
readonly retroarch_cfg="/opt/retropie/configs/all/retroarch.cfg"
readonly system_ra_cfg="/opt/retropie/configs/$system/retroarch.cfg"
readonly gamelist="$HOME/RetroPie/roms/$system/gamelist.xml"
readonly gamelist1="$HOME/.emulationstation/gamelists/$system/gamelist.xml"
readonly gamelist2="/etc/emulationstation/gamelists/$system/gamelist.xml"

rom="${full_path_rom##*/}"
rom="${rom%.*}"
scrap_img="$rom.png"

source "/opt/retropie/lib/inifuncs.sh"

iniConfig ' = ' '"'

# only go on if the auto_screenshot_filename is false
iniGet "auto_screenshot_filename" "$retroarch_cfg"
if ! [[ "$ini_value" =~ ^(false|0)$ ]]; then
    exit 0
fi

# getting the screenshots directory
# try system specific retroarch.cfg, if not found try the global one
iniGet "screenshot_directory" "$system_ra_cfg"
screenshot_dir="$ini_value"
if [[ -z "$screenshot_dir" ]]; then
    iniGet "screenshot_directory" "$retroarch_cfg"
    screenshot_dir="$ini_value"
    if [[ -z "$screenshot_dir" ]]; then
        echo "You must set a path for 'screenshot_directory' in \"retroarch.cfg\"." >&2
        echo "Aborting..." >&2
        exit 1
    fi
fi

# if there is no screenshot named "ROM Name.png", we have nothing to do here
if ! [[ -f "$screenshot_dir/$scrap_img" ]]; then
    echo "There is no screenshot for \"$rom\". Exiting..." >&2
    exit 0
fi

# if there is no "customized gamelist.xml", try the user specific,
# if it fails, get the global one
if ! [[ -f "$gamelist" ]]; then
    echo "Copying \"$gamelist1\" to \"$gamelist\"." >&2

    if ! cp "$gamelist1" "$gamelist" 2>/dev/null; then
        echo "Failed to copy \"$gamelist1\"." >&2
        echo "Copying \"$gamelist2\" to \"$gamelist\"." >&2

        if ! cp "$gamelist2" "$gamelist" 2>/dev/null; then
            echo "Failed to copy \"$gamelist2\"." >&2
            echo "Aborting..." >&2
            exit 1
        fi
    fi
fi

# the <image> entry MUST be on a single line and match the pattern:
# anything followed by rom name followed or not by "-image" followed by dot followed by 3 chars
old_img_regex="<image>.*$rom\(-image\)\?\....</image>"
new_img_regex="<image>$screenshot_dir/$scrap_img</image>"

if grep -q "$old_img_regex" "$gamelist"; then
    sed -i "s|$old_img_regex|$new_img_regex|" "$gamelist"

else # oh! there is no entry for this game yet!
    gamelist_entry="\\
    <game id=\"\" source=\"\">\\
        <path>$full_path_rom</path> \\
        <name>$rom</name> \\
        <desc></desc> \\
        $new_img_regex \\
        <releasedate></releasedate> \\
        <developer></developer> \\
        <publisher></publisher> \\
        <genre></genre> \\
    </game>"

    sed -i "/<\/gameList>/ s|.*|${gamelist_entry}\n&|" "$gamelist"
fi

echo "--- end of $(basename $0) ---" >&2

limitations

Currently this is a kind of "proof of concept". The limitations below can be overcome if we feel that this is the way to achieve what we want.
[The updated and more robust version of the script is available here: https://raw.githubusercontent.com/meleu/share/master/screeper.sh]

This method ~~only~~ changes the <image> entry of a particular game. ~~So, if this game is NOT present in the gamelist.xml, nothing happens.~~ [EDIT: if the game is not present in gamelist.xml, the script creates an entry for it]
The <image></image> entry in the gamelist.xml must be in a single line (it seems to be the default, so probably we don't have to worry about it).
The original image filename in the <image> entry must be named as ROM Name.ext or ROM Name-image.ext (ext can be any 3 characters, eg: png, bmp, jpg, etc), otherwise the sed command won't replace it. [EDIT: it wasn't a problem in my first tests here]
After a succeeded image changing, the respective <image> entry will have a full path to the image. It can be an inconvenience if the user wants to copy the gamelist.xml between computers (IMHO it's not so important. Besides that it probably won't be a problem to those who use the pi user).

"I didn't like how it looks! I want my old scrapes back!"

Change the auto_screenshot_filename to true in /opt/retropie/configs/all/retroarch.cfg and then delete the system specific gamelist.xml that is at the system roms directory (example for SNES: ~/RetroPie/roms/snes/gamelist.xml). Now emulationstation will get the gamelist from $HOME/.emulationstation/gamelists/$system/gamelist.xml.

Restart emulationstation and you'll get back your old scrapes.

where to go from here?

After learn how to use the scraper with -append=true option (no success on my first tests) and if it proves to be good enough, we can evolve it to what I think would be a cool feature: scrape with screenshots only the games that aren't scraped. In other words: if the user is playing a non-scraped game and takes screenshots, then scrape this game with the most recent taken screenshot.

I think its done! :D
Maybe some minor bug fixes (if we found some) and minor improvements.

meleu

Yes, I tested the code. ;-)

If you want to avoid the copy'n'paste process you can get it with this command:

wget "https://raw.githubusercontent.com/meleu/share/master/screeper.sh"

And then rename it to runcommand-onend.sh.

meleu

@herb_fargus
Man! I've updated the script to create an entry for the games that aren't present in gamelist.xml. I updated my big post above including the changes (yes! I tested my code!).

You can get the updated code at github:

wget "https://raw.githubusercontent.com/meleu/src/master/screeper.sh"

Dude, it was really fun to accomplish this task. Thanks for this challenge!

When you have time to test it (and agree with my method) maybe we can edit the wiki to use this method. It's up to you. (If you agree I can edit it next night).

Cheers!

herb_fargus

@meleu so where does metadata fit in with this ;)

meleu

@herb_fargus
one of the prerequisites is:

it was made to work after a scraping (the $HOME/.emulationstation/gamelists/$system/gamelist.xml or /etc/emulationstation/gamelists/$system/gamelist.xml files must exist).

I think it's better to leave the scraping up to the user, because it is very time consuming to be in runcommand-onend.sh.

herb_fargus

@meleu that is true. and the download-images-false flag with sselphs scraper can pull the metadata without the images- or I guess they may have to scrape metadata first unless its appended.

sselph

I think the code as is may have a bug. I haven't run it but just reading it over and following the logic.
ROMs named:
Advanced Dungeons & Dragons - Eye of the Beholder (USA).smc
Zool - Ninja of the 'Nth' Dimension (USA).smc
Will scrape to something like:
<image>./images/Zool - Ninja of the 'Nth' Dimension (USA).png</image>
so the regex used to find and replace the old entry won't work and a new entry will be appended.

That brings up a second bug with the appending code. If a rom is named with a quote in it will probably parse okay but an ampersand would be considered invalid and skipped:
<path>./Advanced Dungeons & Dragons - Eye of the Beholder (USA).smc</path>
instead of
<path>./Advanced Dungeons & Dragons - Eye of the Beholder (USA).smc</path>

If the file named happened to have a < or > things would be very bad
test <US>.bin
would convert to:
<path>test <US>.bin</path>
and the extra <US> would throw off the parser.
it should be something like
<path>test <US>.bin</path>

Using something like passing the file name through
sed "s/\&/\&/g;s/>/\>/g;s/</\</g;s/'/\'/g"
might work to convert things to a similar the escaped XML.

sselph

Something else that may be an issue, but I'm not 100% certain, is having files with names like test [!].bin this seems like it would create a grep statement like grep "test [!].bin" which means the [ and ] wouldn't be taken literally. Also the ! probably causes issues.

meleu

@sselph good points! I'll fix it!
Thanks!

meleu

@sselph
Thanks a lot for your comments! I googled a bit about what you said and I think I've fixed it.

Sharing some info...

From what I understand we have three levels of interpretation to take care of:

shell
XML
sed/grep

shell

As we are getting the file names from runcommand.sh, they are already valid. I know that file names with strange characters (eg.: percent sign) don't work in emulationstation... So, there's a lot of filtering before the filename gets in the runcommand-onend.sh.

XML

As far as I know (from this source), we have five characters to take care:

"   &quot;
'   &apos;
<   &lt;
>   &gt;
&   &amp;

I did it this way:

function echo_xml_safe() {
    output="$(sed 's#\&#\&amp;#g' <<< "$@")"
    output="$(
        sed "
            s#\"#\&quot;#g
            s#'#\&apos;#g
            s#<#\&lt;#g
            s#>#\&gt;#g" <<< "$output"
    )"
    echo "$output"
}

[EDIT: echo has a limitation: it won't print these exact strings: -e or -n. And the use of the printf alternative would add another series of strings to be escaped. Well... the "scrape your own screenshot" trick won't work for roms named -e.ext or -n.ext. We can survive with it... :-) ]

sed/grep

For a safe sed/grep used this function:

function echo_regex_safe() {
    echo "$(sed 's#[][&\^]#\\&#g' <<< "$@")"
}