[SOFT] New Scraper in the works

kiro

@Folly Things should be better now, although I've seen some inconsistencies in certain of the imported games (which I'll be fixing) it should be working much better now. Let me know!

Folly

@kiro

Just did gamecom.
Looks very good ;-)
Will try some others later.

Great work !

kiro

@Folly great! Happy to hear that!!

Folly

@kiro

I just tried dragon32 and apogee again.
There still seems to be an issue.

I would expect that the filenames are recognised and that the "long names" are matched with the "short names"
dragon32 does seem to do that partly.
However many files aren't detected and are still ".zip" after scraping.

For apogee I see that the hash table doesn't have the "long names" so that is, partly, the reason why, for apogee, no "long names" are implemented.
However for apogee only a few files are checked/recognised from 103 files.
This is my output :

Folly

@kiro

Just noticed that the issue could be that my wifi connection is bad.
I will try again.

edit:

for apogee the bad wifi connection was the issue
for dragon32 the issue remains

A part of the dragon32 scrape output :

System: dragon32 | Game : crzyfoota.zip
System: dragon32 | Game : crzyfoota2.zip
System: dragon32 | Game : Cruising On Broadway
System: dragon32 | Game : crzyfoota3.zip
System: dragon32 | Game : Crusader - An Arcade Adventure
System: dragon32 | Game : crzypaint.zip
System: dragon32 | Game : cstldoom.zip
System: dragon32 | Game : cstldrac.zip
System: dragon32 | Game : cuthchalic.zip
System: dragon32 | Game : cuthcoolr.zip

Searched in the hash table for cstldoom ( dragon_cass ):
( Castle of Doom )

<software name="cstldoom">
		<description>Castle of Doom</description>
		<year>1983</year>
		<publisher>Paramount Software</publisher>
		<part name="cass1" interface="dragon_cass">
			<dataarea name="cass" size="22699">
				<rom name="Castle Of Doom (1983)(Paramount Software).cas" size="22699" crc="d32875d3" sha1="ce3996a595a4e4d310dead4b910ab270b3ef0d0b"/>
			</dataarea>
		</part>
	</software>

kiro

@Folly said in [SOFT] New Scraper in the works:

ce3996a595a4e4d310dead4b910ab270b3ef0d0b

That's weird, the sha1 is in the backend DB and also in the API... would it be possible to try again?

Thx!

Folly

@kiro

Succeeded, it works now for dragon 32 !

kiro

@Folly Brilliant! Thanks for all the testing!

Folly

@kiro

Is there a possibility that I can see whats in your database ?

kiro

@Folly What would you like to see?

Folly

@kiro

Well, if some files are recognised and others not then it would be practical if I can search in the database to see where the problem lies or what info is missing or not.

Also would like to see how the data is structured so I have a better understanding of how it is working.
For now one example could do, take for instance cstldoom of dragon32.

This also brings me to another question.
How can we add stuff in the future, for instance adding :

descriptions
video files
pictures
etc

Edit :
A good example is now the "All In One" category.
When I scrape all names are detected but all pictures are 0kb and there are no videos.
So I recon you have not added the media yet.
Looking it up in the database would reveal why there are 0kb image files and why there are no videos.

kiro

@Folly Check your mail...

The all_in_one is a tricky one, because this is a composition of several systems, and the way how retroscraper works, it needs to understand which system the 'rom' belongs to in order to import new information. I need to fiugre out a way to figure out the system for each rom in the 'allinone' gamelist :-)

Folly

@kiro

Hi,

I have a request for channelf and it would be nice if you could implement it.

Can you change the cartridge names of the range from Videocart-1 to Videocart-9 to :
Videocart-01 - Videocart-09
Adding the zero makes sure that the games are properly sorted in the list of emulationstation.

Good idea ?

kiro

@Folly yeah should be easy to do, will do the changes in the DB today and they should be available tomorrow morning.

kiro

Hi @Folly, changes should already be available. Have a nice day!

Folly

@kiro

Thanks

Folly

@kiro

Hi,

I recently scraped my arcade shooters.
Somehow there were many videos missing that I think were there before when I scraped them a few moths ago.

For example "1942" and "aburner2", etc.

Any idea what could be wrong ?

kiro

@Folly Hi, very difficult to know like that, nothing has changed so it should still be working....can I maybe have the SHA1 of a couple of your roms that are failing so I can check in the DB? Thanks!

Folly

@kiro

Here are just a few that all had videos before when the there was an issue in your script saving videos as .png files :
(V = has still video / X=no video anymore )

V 42a03233d270efc668abfd06f3c8cebdbf7a6cac  005.7z
X ea7268365be6a23c9e1f4a03344c6c2359c3b65e  1941.7z
X 437e4be153d03f8cacbdfd00f4c3ccc7a141c2fe  1942.7z
X 495898ab638658cb6720c958cd392a14f4eb6af8  1943.7z
X cdb758e00e1eb3cb7d4d9d799c495c1ac3b3c736  1943kai.7z
X e7da3b3920d10015a36f02965dc528ca11560e39  1943mii.7z
X baba62550857d0c1f0280f4cfc49f4b620b6006e  1944.7z
X 567c08e38c0c9604553d0982425f20df853d92e0  1945kiii.7z
X 2dfe192c9c237bc00173396d683d785931092dad  19xx.7z
V 02388bef2d3a1ea53208dd20f9560c4196e04436  3stooges.7z
X 2ef88584052aecfaa536427232df02d0ff46778f  4dwarrio.7z
X e609337aa53ded59b9d9d77ac1446ce043be3351  aburner2.7z

... many more

I believe the current .zip variants have the same issue.
(will check that again later)

kiro

@Folly hi! I've been testing and I think I've found an issue, not sure why it is appearing now, but so be it. I've think I've fixed it, I'm testing some more and will update the tool ASAP. Will keep you informed. Thanks a lot for the report!!