[SOFT] New Scraper in the works

Folly

Hi,

I recently scraped my arcade shooters.
Somehow there were many videos missing that I think were there before when I scraped them a few moths ago.

For example "1942" and "aburner2", etc.

Any idea what could be wrong ?

kiro

@Folly Hi, very difficult to know like that, nothing has changed so it should still be working....can I maybe have the SHA1 of a couple of your roms that are failing so I can check in the DB? Thanks!

Folly

@kiro

Here are just a few that all had videos before when the there was an issue in your script saving videos as .png files :
(V = has still video / X=no video anymore )

V 42a03233d270efc668abfd06f3c8cebdbf7a6cac  005.7z
X ea7268365be6a23c9e1f4a03344c6c2359c3b65e  1941.7z
X 437e4be153d03f8cacbdfd00f4c3ccc7a141c2fe  1942.7z
X 495898ab638658cb6720c958cd392a14f4eb6af8  1943.7z
X cdb758e00e1eb3cb7d4d9d799c495c1ac3b3c736  1943kai.7z
X e7da3b3920d10015a36f02965dc528ca11560e39  1943mii.7z
X baba62550857d0c1f0280f4cfc49f4b620b6006e  1944.7z
X 567c08e38c0c9604553d0982425f20df853d92e0  1945kiii.7z
X 2dfe192c9c237bc00173396d683d785931092dad  19xx.7z
V 02388bef2d3a1ea53208dd20f9560c4196e04436  3stooges.7z
X 2ef88584052aecfaa536427232df02d0ff46778f  4dwarrio.7z
X e609337aa53ded59b9d9d77ac1446ce043be3351  aburner2.7z

... many more

I believe the current .zip variants have the same issue.
(will check that again later)

kiro

@Folly hi! I've been testing and I think I've found an issue, not sure why it is appearing now, but so be it. I've think I've fixed it, I'm testing some more and will update the tool ASAP. Will keep you informed. Thanks a lot for the report!!

kiro

@Folly Fix is the repository. Let me know if this solves your problems or not.

Thanks one more time!

Folly

@kiro

No change but I am doing the scraping from my script which uses the -rpie version.
So you probably have to change some in there too.

kiro

@Folly I've updated the -rpie version, are you sure you have the latest version from the repository? Thanks

Folly

@kiro

I see you have just made a commit.
I just tried again, same result.
My script runs the retroscraper-rpie version "online" so it should be the latest.

Did the dependencies change ?

kiro

@Folly no, I did not change any dependencies...do you have somehow a cache enabled in your setup that would not get the latest version? Not sure...I'm scanning my whole collection again and it seems to be working fine....

Folly

@kiro said in [SOFT] New Scraper in the works:

do you have somehow a cache enabled in your setup that would not get the latest version?

No I haven't, I also did a re-run of the RetroPie-Setup script.

kiro

@Folly if you try to get retroscraper and run it locally, do you have the same problem?

Folly

@kiro

I just did, with this command :

python3 retroscraper.py --systems arcade --relativepaths

That works, I have videos now.

My script has a different command, will do that test also locally.

I tested this command, with similar options with the one in my script :

python3 retroscraper.py --systems arcade --recursive --relativepaths --mediadir media/emulationstation --nobackup

That works locally too now.

kiro

@Folly cool, let me know how it goes.

Folly

@kiro

Running this locally also works :

curl https://raw.githubusercontent.com/zayamatias/retroscraper-remote/main/retroscraper.py|python3 - --systems arcade --recursive --relativepaths --mediadir media/emulationstation --nobackup

From my script however it still has the same issue.
I will reboot my system to see if this will help.

kiro

@Folly yes, I totally forgot about the retroscraper -remote version, just uploaded the fix there too

Folly

@kiro

Yea, I see it too now.
I was thinking also that it was the -rpie version but it's the remote indeed.
So It didn't have the fix yet.

I rebooted and now it works again, great.
But the reboot would not have been necessary, I believe now.

Very strange indeed that it somehow didn't work anymore but it seems it has been fixed ;-)

Thanks for fixing.
Good team work !

kiro

@Folly there seems to be an issue with the database and some video naming, I fixed the video naming conventions in the front-end, but still need some work in the backend. Thanks again! And yes, great teamwork!

Enjoy your weekend!

Folly

@kiro said in [SOFT] New Scraper in the works:

Enjoy your weekend!

You too.

Folly

@kiro

Today I changed how the dependencies for retroscraper-remote are installed in my WIP mamedev.sh module-script as I wasn't happy about how it was.

It took me quite some time to figure things out but I am quite happy with the result as it looks like it's a bit faster.

This is the commit I made.

This is how the function looks now :

function retroscraper_remote_depends_mamedev () {
	#install pip if the pip module is not installed
    if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi

    #update pip if there is a new release
	[[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip"

    #install python modules when not detected as installed
    #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online"
    #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix
	local retroscraper_remote_module
	local retroscraper_remote_modules=()
	retroscraper_remote_modules=(
	wheel==0.40.0
	setuptools==67.7.2
	googletrans==4.0.0rc1
	Pillow==9.2.0
	requests==2.21.0
	httpimport==1.1.0
	)
	for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do 
	[[ $(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'") != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module"
	done
}

I think you will find it interesting too.

What is your opinion about also using a fixed version for wheel and setuptools ?

If you have a question regarding this, then let me know.

Edit :
After testing it more it still seems that it's a bit slow on the raspberry pi.
Will have a second look if I can improve it.

Folly

@kiro

By changing the function slightly I doubled the speed.
See this commit

This is the function now :

function retroscraper_remote_depends_mamedev () {
	#install pip if the pip module is not installed
    if [[ $(su $user -c "python3 -m pip list" 2>&1) == *"No module named pip"* ]]; then
        su $user -c "wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py"
        su $user -c "python3 /tmp/get-pip.py"
    fi

    #update pip if there is a new release (2>&1 is used to redirect stderr to stdout so we can use the if function on the stderr info)
	[[ $(su $user -c "python3 -m pip list" 2>&1) == *"new release of pip"* ]] && su $user -c "python3 -m pip install --user --upgrade pip"

    #install python modules when not detected as installed
    #retroscraper-remote needs httpimport as extra library so some dependancies can be used "online"
    #https://stackoverflow.com/questions/23106621/replace-multiple-consecutive-white-spaces-with-one-comma-in-unix
	local pip_list_output
	local retroscraper_remote_module
	local retroscraper_remote_modules=()
	retroscraper_remote_modules=(
	wheel==0.40.0
	setuptools==67.7.2
	googletrans==4.0.0rc1
	Pillow==9.2.0
	requests==2.21.0
	httpimport==1.1.0
	)
	#this command was the first test and filters out the required modules but has no speed advantage vs the used command
	#pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'|awk \"/${retroscraper_remote_modules[0]}/ || /${retroscraper_remote_modules[1]}/ || /${retroscraper_remote_modules[2]}/ || /${retroscraper_remote_modules[3]}/ ||  /${retroscraper_remote_modules[4]}/ || /${retroscraper_remote_modules[5]}/\"")
	pip_list_output=$(su $user -c "python3 -m pip list|sed 's/ \{1,\}/==/g'")
	
	for retroscraper_remote_module in ${retroscraper_remote_modules[@]};do 
	[[ $pip_list_output != *$retroscraper_remote_module* ]] && su $user -c "python3 -m pip install --user $retroscraper_remote_module"
	done
}