Possible bug?

rkoster

This is easy enough to solve manually, but I wanted to report it just in case it's a legit bug.

Install your MAME roms in /roms/mame-libretro
go ahead and mark favorites from within emulationstation
exit emulationstation and launch Selph's Scraper and scrape the mame-libretro roms

What happened to me, at least, is that each game marked as favorite failed to scrape, with the message "already in gamelist."

Order of operations thing? Not sure, but it would take me a bit of time to replicate because I'd have to wipe everything and start over...

gaavoid

@rkoster I think when a game is added to favourites, an entry in the gamelist.xml file is created for that game, so regardless if it has any other info (video/metadata/box art etc.), sselph's scraper will detect that it's already there and the gamelist.xml won't be updated unless you set it to overwrite rather than append. But if you overwrite it, then I would assume it would be removed from the favourites list.

Hex

@pjft Might wanna take a look.

pjft

Correct. Adding a favorite effectively creates an entry in the gamelist. As such, if the scraper does not support updating existing entries, it will claim it's already in the gamelist (which it is).

I do wonder if that's the case, though - might be worth asking @sselph if the scraper supports metadata merging for previously existing entries. If it doesn't, my concern is that re-scraping will overwrite the previous metadata (which means losing the favorite status).

It's the same if you edit a game's metadata and then decide to scrape it afterwards :/

sselph

@pjft Today with regards to the favorites it is either overwriting the entry(removing any favorite/last played/etc) or appending (completely ignoring it).

rkoster

I took the plunge and went ahead and scraped again with Selph's Scraper set to overwrite. Favorites for that system, as expected, disappeared. I'll just recreate the faves for that system, but it does seem like a note should be made in the docs for favorites that one should scrape BEFORE marking faves.

pjft

@rkoster thanks. I suppose when I document the whole collections bit I'll make sure to add that there.

Thank you!

pjft

Hi all.

I was thinking about this yesterday, per chance, and I think I have a fix for it in ES.

https://github.com/pjft/EmulationStation/tree/Fix-Scraping-Data

I haven't compiled it and tested it yet, as I don't have access to a build and runtime environment for the moment, alas. That being said, it seems reasonably straightforward.

Now, the concern with using @sselph 's scraper is that, while for me it is the best, it does not run within ES, so this fix will not help with that.

@sselph : a question/though - would you think it'd be manageable, in your scraper, to implement this in a non-destructive manner? Meaning, any existing non-scrapeable data for a particular entry should be kept, rather than destroyed?

There are a few that come to mind, at least: favorites, last played and times played, but in the future potentially kid-friendly, etc.

I don't know the inner workings of your scraper - do you check if there's an entry before scraping at all? - but I wanted to bring this up here as well, and also open to thoughts.

Thanks.

sselph

@pjft So as mentioned before the default is to overwrite the entire xml. I have another mode that will append to the xml which doesn't touch anything that already exists. I think what you are asking for is something in the middle. If the entry already exists and I get a result from scraping, preserve the non-scraped data. If the entry can't be scraped, keep what exists.

This could be possible but I have to know all the fields that need to be preserved. The way the xml library works, I have to declare every field.

pjft

@sselph thanks and sorry for the delay. I replied in more detail on GitHub, but let me know if it's not clear.

Thank you.

sselph

@pjft I know you don't want to dig through my code but these are the fields I recognize already:
https://github.com/sselph/scraper/blob/master/rom/rom.go#L505
You said favorites, lastplayed and playcount
I have lastplayed and playcount but have favorite not favorites which is it?

pjft

@sselph I don't have a problem digging through your code :)

It is "favorite", that's correct. Sorry for that - I've been mostly answering these on my phone for the moment, mistakes will happen.

Thanks!

Zigurana

@sselph , could you please also add hidden and kidgame to the list of tags to ignore and retain?

sselph

@zigurana said in Possible bug?:

hidden and kidgame

done