[SOFT] New Scraper in the works

Folly

A pain ?
I can imagine.
Just be patient there is no hurry.

Hopefully we can manage that the outputs are be becoming a good alternative for the predefined gamelists,
because the scraping speed is much higher than downloading speed of the predefined ones.
So all in all that would really be great.

kiro

Hi @Folly! Finally all roms from mame-sl are in the backend and I've added the pictures found in your gamelists where missing, phew... hope this helps!

Supported systems with figures here: http://77.68.23.83/

Folly

@kiro

Very cool that you managed all of this.
I can only imagine how much work it must be.
Thumbs up ^^ :-)

I will do some scraping when I can ;-)

Folly

@kiro said in [SOFT] New Scraper in the works:

I've added the pictures found in your gamelists where missing

Which pictures are missing ?

kiro

@Folly I meant to say, when there were no pictures available on the backend, I imported those on your gamelists :-) Sorry for the confusion!

Folly

@kiro

Ok , I understand.

Looked at your page, I must say the list is very impressive !!!

Just a question :
How are the new systems detected and what is the accompanied rom-directory for detection ?
Or does the directory name make no difference in detecting ?

Edit :
Oh yea,
When it doesn't detect it by default it assumes it is "arcade/non-arcade(mame)", Right ?

kiro

@Folly So, systems are detected on base of common names, which for mame-sl these are the 'software list' names... for example "pv1000" for the casio-pv1000. This name is expected to be in the 'system' tag of the es_systems.xml, for example:

  <system>
    <name>gameandwatch</name>
    <fullname>Arcade</fullname>
    <path>/home/pi/RetroPie/roms/gameandwatch</path>
    <extension>.7z .cue .fba .iso .zip .7Z .CUE .FBA .ISO .ZIP</extension>
    <command>/opt/retropie/supplementary/runcommand/runcommand.sh 0 _SYS_ arcade %ROM%</command>
    <platform>arcade</platform>
    <theme>arcade</theme>
  </system>

Now, this being said, this is only necessary if you have roms which are not in the DB and want to match by maybe name or simlarities to another one in that system. Retroscraper will always use the checksum from your file and find a match accordingly, regardless of which system they actually belong to (you cannot have 2 file swith same checksums).

Hope this helos.

Folly

@kiro

Ok, good explanation.

Will let you know what my experiences are when scraping those mame-sl roms / systems.

Folly

@kiro

I just tried to scrape apogee, dragon64 and gamecom.
The systems seem to be recognized.
Sadly the files aren't recognized.

kiro

@Folly Hmm.. Interesting, for some reason the new additions have not been 'apified' if I may say..investigating...will get back to you asap...

ARGH! I updated with the test DB :-) fixing this, will be back soon...

kiro

@Folly Things should be better now, although I've seen some inconsistencies in certain of the imported games (which I'll be fixing) it should be working much better now. Let me know!

Folly

@kiro

Just did gamecom.
Looks very good ;-)
Will try some others later.

Great work !

kiro

@Folly great! Happy to hear that!!

Folly

@kiro

I just tried dragon32 and apogee again.
There still seems to be an issue.

I would expect that the filenames are recognised and that the "long names" are matched with the "short names"
dragon32 does seem to do that partly.
However many files aren't detected and are still ".zip" after scraping.

For apogee I see that the hash table doesn't have the "long names" so that is, partly, the reason why, for apogee, no "long names" are implemented.
However for apogee only a few files are checked/recognised from 103 files.
This is my output :

Folly

@kiro

Just noticed that the issue could be that my wifi connection is bad.
I will try again.

edit:

for apogee the bad wifi connection was the issue
for dragon32 the issue remains

A part of the dragon32 scrape output :

System: dragon32 | Game : crzyfoota.zip
System: dragon32 | Game : crzyfoota2.zip
System: dragon32 | Game : Cruising On Broadway
System: dragon32 | Game : crzyfoota3.zip
System: dragon32 | Game : Crusader - An Arcade Adventure
System: dragon32 | Game : crzypaint.zip
System: dragon32 | Game : cstldoom.zip
System: dragon32 | Game : cstldrac.zip
System: dragon32 | Game : cuthchalic.zip
System: dragon32 | Game : cuthcoolr.zip

Searched in the hash table for cstldoom ( dragon_cass ):
( Castle of Doom )

<software name="cstldoom">
		<description>Castle of Doom</description>
		<year>1983</year>
		<publisher>Paramount Software</publisher>
		<part name="cass1" interface="dragon_cass">
			<dataarea name="cass" size="22699">
				<rom name="Castle Of Doom (1983)(Paramount Software).cas" size="22699" crc="d32875d3" sha1="ce3996a595a4e4d310dead4b910ab270b3ef0d0b"/>
			</dataarea>
		</part>
	</software>

kiro

@Folly said in [SOFT] New Scraper in the works:

ce3996a595a4e4d310dead4b910ab270b3ef0d0b

That's weird, the sha1 is in the backend DB and also in the API... would it be possible to try again?

Thx!

Folly

@kiro

Succeeded, it works now for dragon 32 !

kiro

@Folly Brilliant! Thanks for all the testing!

Folly

@kiro

Is there a possibility that I can see whats in your database ?

kiro

@Folly What would you like to see?