Fast Alternate scraping technique

[iCLint]

EDit: For complete noobs see the easy instructions posted further down however I suggest reading the thread in full before begining

I wasn't happy with either the built in ES-Scraper or the Sselph-Scraper due to being very slow, grossly inaccurate and ultimately more work to go back and correct things and fill in the blanks, so I set about finding a better way to do it.

My needs were simple - Game name and graphic. and then a way to generate the gamelist.xml.

I came across one solution that used a DOS Batch file run from windows on the remote RPi which worked but was very slow for what should be a near instant scan and generation of the xml. the solution, convert the Batch file to a shell script to run on the host RPI.

I posted a thread asking for help as I knew nothing about scripting other than a very limited understanding of general coding. Didn't get any help, whether that was because no-one knew how or there is literally no-one looking for a similar solution? however there seems to be a lot of questions and people looking to do the same thing as I wanted i.e. fast simple scraping.

Okay so this is the solution and what you will need.

create a folder called "images" in each of your rom folders.

e.g.
./roms/arcade/images/
./roms/snes/images/
./roms/megadrive/images/

you could do this from a GUI e.g.. windows, OS X, Raspbian Pixel etc or you could do it from the command line e.g. while inside the coms folder

mkdir images

now with the folder structure in place you need to put your image for each rom in the images folder.

e.g.
./roms/arcade/pacman.zip

needs an image by the same name in png format in the nested "images" folder

./roms/arcade/images/pacman.png

now you could go and google all these images, you could capture in game screenshots and set up retro arch to place them in the correct folder with the correct name or do that manually also.... or there is the easy way.

The emumovies project has a tool for scraping roms and downloading all kinds of media the scraping is super fast and accurate. unfortunately this tool is windows only so if you run an OS other than windows like me you will need a windows machine, bootcamp or VM to run it.

to summarise:

1. register for an account at emumovies
2. download/install emumovies sync tool
3. open emumovies sync tool and input your emumovies account details and you are ready to go

Now just point emumovies sync tool at your rom folder e.g. ./arcade tell hypersync what kind of roms they are e.g. MAME tell hypersync what media you want to download and click sync.

emumovies sync tool will download all the chosen artwork for your rom and name it the same as your rom name. provided your rom name is in the ball park of the official rom name it will find what you are looking for and fast.

you then only need to copy the dowloaded art work to the images folder e.g. ./roms/arcade/images/

Final step run my script.

how to setup the script:

run these commands from the ./roms folder on the RPi

touch makexml.sh -creates an empty text file

EDIT: alternatively you could create the script file in a text editor on you PC if that is easier then just copy to Pi Roms folder and run the chmod command on the file.

chmod 755 makexml.sh -sets permissions

chmod +x makexml.sh -makes the file executable

nano makexml.sh -opens makexml.sh in nano text editor

with the makexml.sh file open in nano now paste this script into the editor my client ssh terminal allows for copy and paste YMMV other wise type the script in yourself.

#!/bin/bash

# Script use $ sudo ./makexml.sh [RomDirectoryName] [-n optional]
# -n does not right name to <name> tag useful for MAME roms as ES will autofill correct name

# Store arguments
arg1=$1
arg2=$2
output="gamelist.xml"

# Change deliminator for $IFS 
# Allows script to work with names with white space
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")

# print scripts current status 
printf "Scanning dir: "$arg1"...\n"

# Creates output XML file (gamelist.xml)
printf "<?xml version=\"1.0\"?>\n\n" > ./$arg1/$output  
printf "<gameList>\n" >> ./$arg1/$output

# Scans directory given by first argument
for file in ./$arg1/*
do
        # Just some manipulatiuon of data returned by $file 
        filename=$(echo $file | cut -f 3 -d '/')
        name=$(echo $filename | cut -f 1 -d '.')

        # Checking for existance of correctly named image in ./images dir.
        [[ -f ./$arg1/images/$name.png ]] && theImage=./images/$name.png || theImage=""

        # Start writing to gamelist.xml
        printf "\n" >> ./$arg1/$output
        printf "\t<game>\n" >> ./$arg1/$output
        printf "\t\t<path>./"$filename"</path>\n" >> ./$arg1/$output

        # Checking whether the -n argument was used
        if [ "$arg2" = "-n" ]; then
        name=""
        fi

        # Continuing writing to gamelist.xml
        printf "\t\t<name>"$name"</name>\n" >> ./$arg1/$output
        printf "\t\t<desc></desc>\n" >> ./$arg1/$output
        printf "\t\t<image>"$theImage"</image>\n" >> ./$arg1/$output
        printf "\t\t<rating></rating>\n" >> ./$arg1/$output
        printf "\t\t<releasedate></releasedate>\n" >> ./$arg1/$output
        printf "\t\t<developer></developer>\n" >> ./$arg1/$output
        printf "\t\t<publisher></publisher>\n" >> ./$arg1/$output
        printf "\t\t<genre></genre>\n" >> ./$arg1/$output
        printf "\t\t<players></players>\n" >> ./$arg1/$output
        printf "\t</game>\n" >> ./$arg1/$output

        # Print scripts current status
        printf $filename
        clear
        printf "Scanning dir: "$arg1"...\n"
        done

# Closing gamelist.xml file
printf "</gameList>" >> ./$arg1/$output

# print scripts current status
clear
printf "Finished\n"

I think the script is fairly self explanatory and documented where needed.

Now save the script control+o and enter. then exit nano control+x

to run the script use the following command.

sudo ./makexml.sh [nameOfRomFolder] [-n]

where [nameOfRomFolder] is for example "arcade" and [-n] is an optional switch which used to turn off naming of the rom in the <name></name> tags it is only needed for MAME roms as ES is able to pull the proper name from the rom. but for all your console roms do not use the switch.

example:
sudo ./makexml.sh arcade -n -For your name coms

sudo ./makexml.sh gb -for your gameboy coms

sudo ./makexml.sh mastersystem -etc. etc. etc.

[UnkleMonty]

Excellent stuff - thanks! Like you, I've been struggling with scraping recently. Going to try this later today.

[Rookervik]

Wow that's in-depth. I updated the DOS batch file to be a windows program and it is instant, now. Some people have said it doesn't always catch all the files. But I've done all my systems with it without any problems. https://retropie.org.uk/forum/topic/4471/updated-gamelist-xml-creator

[iCLint]

@Rookervik said in Fast Alternate scraping technique:

Wow that's in-depth. I updated the DOS batch file to be a windows program and it is instant, now. Some people have said it doesn't always catch all the files. But I've done all my systems with it without any problems. https://retropie.org.uk/forum/topic/4471/updated-gamelist-xml-creator

Hi, yes I posted in your thread I tried multiple times to get the windows app to work, it would just throw up errors snd and garbage xml's for me. It was your Batch file though which was the basis for this script.

[iCLint]

The easy version

Step 1: everything you need

1. Create an account at Emumovies.com
2. login to emumovies and download their sync tool
3. Download my scripts ready for use
4. Optional - download putty or have a keyboard connected to your Pi

Step 2: install needed software

1. Extract the scripts from the zip file theScripts.zip and copy them to your roms folder
   
2. install the emumovies sync tool
3. Optional install Putty if you plan to SSH into the Pi to run scripts

Step 3: Setup needed folders on the Pi
I have tried to make this easy by supplying a script to do it
from the command line on the pi while in the ~/RetroPie/roms/

type: sudo ./makeimgfldrs.sh

if all went well you now hove an "images folder in everyone of your system folders

step 4: setup and run the emumovies sync tool

1. open emumovies sync tool and enter your login details

For this example I will scrape the snes system and create the needed xml file.

2. Setup the system to be scraped, what artwork you want to download and where you want the artwork saved

3. Click "Go" to begin scraping.

Step 4: copy your scraped artwork to the images folder for the system you scraped e.g. /roms/snes/images

Step 5: create your gamelist.xml file

1. exit Emulation station. From the keyboard F4 from the controller Start button > Quit > Quit Emulationstation. this is very important.

2. from the command line type: sudo ./makexml.sh snes


3. start emulation station by entering the command: emulationstation

4. if everything went well you should be able to navigate to the snes system and see all your games with scraped images

To scrap further systems you only need to complete steps 3,4 & 5

[zmeta]

When I run this command.....pi@retropie:~/RetroPie/roms $ sudo ./makeimgfldr.sh
I get this....... sudo: ./makeimgfldr.sh: command not found

I have confirmed that the makeimgfldr.sh file is located in the directory

[buckoo]

i have the same problem

[pjft]

@zmeta @buckoo you probably haven't granted it the right permissions.

chmod +x makeimgfldr.sh

[A Former User]

Help! The files you posted are not available.

[g00n3r]

@itsnitro Like a fool I clicked the link.....is that a virus?