how to use the sselph scraper "-append" option

meleu

What I want to do:
Add a new game entry in an existent gamelist.xml for a game that isn't in the file yet, scraping it with a local image file.

What I tried:
Copied an existing gamelist.xml to my specific system roms folder:

pi@retropie:~ $ ls -lh ~/.emulationstation/gamelists/nes/gamelist.xml 
-rw-r--r-- 1 pi pi 533K Aug 20 19:49 /home/pi/.emulationstation/gamelists/nes/gamelist.xml
pi@retropie:~ $ cp ~/.emulationstation/gamelists/nes/gamelist.xml ~/RetroPie/roms/nes/gamelist.xml
pi@retropie:~ $

I have a Mega Man ROM and I'm pretty sure that there is no entry for it in my gamelist.xml:

pi@retropie:~/RetroPie/roms/nes $ ls -l M/Mega\ Man\ \(USA\).nes 
-rw-r--r-- 1 pi pi 131088 Apr 15 00:03 M/Mega Man (USA).nes
pi@retropie:~/RetroPie/roms/nes $ grep 'Mega Man (USA)' gamelist.xml
pi@retropie:~/RetroPie/roms/nes $

I'm pretty sure that there is an image file with the same name as the ROM (except the extension):

$ ls -l /home/pi/screenshots/Mega\ Man\ \(USA\).png 
-rw-r--r-- 1 pi pi 6774 Aug 21 10:33 /home/pi/screenshots/Mega Man (USA).png

Then I tried this command (the multiple lines are just for clarity):

pi@retropie:~/RetroPie/roms/nes $ /opt/retropie/supplementary/scraper/scraper \
  -append=true \
  -image_path="/home/pi/screenshots/" \
  -image_dir="/home/pi/screenshots/" \
  -download_images=false \
  -output_file="/home/pi/RetroPie/roms/nes/gamelist.xml" \
  -rom_dir="/home/pi/RetroPie/roms/nes" \
  -skip_check=true \
  -use_gdb=false \
  -image_suffix=

According to sselph scraper documentation, the -append option should do this trick. But the command above doesn't work. The copied file is equal to the original one:

pi@retropie:~ $ diff ~/RetroPie/roms/nes/gamelist.xml ~/.emulationstation/gamelists/nes/gamelist.xml 
pi@retropie:~ $

My questions:

The -append doc says:

If the gamelist file already exists, skip files that are already listed and only append new files.

As I showed above, the "Mega Man (USA).nes" aren't listed in the gamelist.xml (I thought it was enough to characterize it as a new file). And it has an image with the same name in the directory passed to the -image_dir/-image_path (I thought it was enough to make the scraper create an entry).

Finally, here are my questions:

What exactly characterizes a "new file"?
What the scraper needs to decide "hey! let's an entry for this ROM in gamelist.xml!"??

( @sselph would you mind to give me some hint here?)

sselph

@meleu I haven't looked in depth just have a sec to give a quick response. It doesn't look like there is anything to do in your command. There are no data sources so nothing to scrape. If you want it to add files it can't find you can add -add_not_found and it will make an empty entry with just the file name and image.

meleu

Why -add_not_found isn't what I want:
I've read the scraping your own images in the wiki, but it's not what I want. Because the -add_not_found option will add several game entries with no <image> entry. And I need it to what I want to do.

meleu

@sselph
BTW: I think I can achieve what I want (create an individual game entry in gamelist.xml) with some sed commands. I was only wondering if it is easier to do with your tool.
Thanks!

sselph

@meleu I don't think there is a combination of flags that does exactly what you want. Be careful directly editing xml files and make sure you escape special characters. I'm sure there are some scripts out there to help with that.

https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references#Predefined_entities_in_XML

meleu

@sselph
I did it with sed and it worked like a charm.

If you are curious, here is the code snippet:

    gamelist_entry="\\
    <game id=\"\" source=\"\">\\
        <path>$full_path_rom</path> \\
        <name>$rom</name> \\
        <desc></desc> \\
        $new_img_regex \\
        <releasedate></releasedate> \\
        <developer></developer> \\
        <publisher></publisher> \\
        <genre></genre> \\
    </game>"

    # add the entry right before the </gameList>
    sed -i "/<\/gameList>/ s|.*|${gamelist_entry}\n&|" "$gamelist"

Thanks anyway for your quick response! :-)

sselph

@meleu Glad you got something working. If you intend to make this available to others it would be good to make sure you escape $full_path_rom, $rom, and $new_img_regex The characters < > ' " & should be escaped in xml. < and > would wreak havoc for sure. I think the others may be okay in some situations but might be best to escape those as well.

meleu

@sselph
I used it for the task I explained here:
https://retropie.org.uk/forum/topic/3353/take-and-scrape-your-own-screenshots/47
And it's working fine.

It's a scheme to let the user take screenshots of a game and set the most recent screenshot as the emulationstation image for this game.

Here is the complete code:
https://github.com/meleu/src/blob/master/screep.sh

OBS.: It's intended to be the runcommand-onend.sh script.

Cheers!