Scraper optimization

meleu

Another great feature is an optimization of the EmulationStation Scraper (I think it's an upstream issue, but I'll talk about this here anyway).

The option "user decides on conflicts" could be optimized, making the user decide at the end of the process. Or better: the scraping process keeps running on background while the user decides what to do with the conflict.

Think about this scenario: The user have a romset with 2000+ games. He puts the system to scrap and go drink some coffee. The second rom has a conflict, and the user is drinking coffee away from the screen. The entire scraping process is stopped!

Another simpler way to deal with this issue: an "ignore the conflicts" option. This way, I can scrap my entire romset and don't care about the conflicts. Then after I can do a second scraping scan, and give attention to the conflicts.

herb_fargus

@meleu emulationstation dev is dead which is why we have sselphs scraper which automates much of what you're talking about and does a much better and faster job anyways: https://github.com/retropie/retropie-setup/wiki/scraper

meleu

@herb_fargus I'll take a look! Thanks.

Lodan-Zark

@herb_fargus O,o He is dead?!? literally or metaphorically?

meleu

@Lodan-Zark I think @herb_fargus wanted to say dev with the meaning of development, not developer. I hope that!

Rion

The development have stopped but a fork is in development
https://github.com/Herdinger/EmulationStation

schmulkfuster

@Rion Thats good to see... I wonder if Retropie should start offering the option to switch and/or default to the new port version instead of the old dead version?

herb_fargus

@schmulkfuster it needs work before we'd do that.

Zigurana

There are many forks of ES, herdinger is not the first to try and continue the work.

BuZz

@schmulkfuster there is nothing significant in the new fork yet. I am keeping an eye on things. We currently run from our own fork, as we have our own customisations. No plans to rebase over a new upstream yet

Rion

@BuZz said:

@schmulkfuster there is nothing significant in the new fork yet. I am keeping an eye on things. We currently run from our own fork, as we have our own customisations. No plans to rebase over a new upstream yet

Any plans to have a look at what the other forks are doing for example Recalbox and incorporate that into your fork?

@Zigurana said:

There are many forks of ES, herdinger is not the first to try and continue the work.

That's news to me. Do you have any links to the other forks? Kind of interested to see what they are doing with Emulationstation.

BuZz

@Rion I already do this, however I don't currently want to spend my time working on Emulation Station.

meleu

You guys have to agree that the emulationstation is an important part of RetroPie.

I don't want to be that boring dude saying what the developers have to do, but I would like to see some little improvements on emulationstation.

What do you developers think about a donating system where the donors could say where they would like to see improvements? (Just as a suggestion!)

I think it will increase the number of donations ;-)

meleu

Another idea: the developer that is used to play with the emulationstation code say here "if I get X bucks I will make some effort to implement the Y feature".
Something like the kickstarter.
I don't know the details of how to do this. It's just a brainstorm, OK? :D

BuZz

I wouldn't be interested - I do this for fun, and not as a job.

Nothing is stopping you from contributing back improvements yourself of course - but bear in mind that I got involved in RetroPie because I wanted to see some improvements as a RetroPie user. I do listen to and implement many requests from users, but I don't want this to turn into something where I get told what I should be working on - that would no longer be something I would like to do.

Zigurana

@Rion See here for a list of people who have created a fork of ES. Apparently its very difficult on Github to get a full list of forks, although you can seen them all when you actually want to do comparisons for pull-requests.

I guess the point right now is not that there are no people willing to work on it, but more who will stand up to draw up a comprehensive action plan with a long term vision for ES, and is capable of pulling that particular cart for the foreseeable future?
It will need to show these things before RetroPie can even consider switching upstream branches.

The good thing about Herdingers project is that at least he/she is willing to publictly discuss the long-term goals. See here.

Zigurana

@BuZz @meleu
I fully agree with that sentiment, if you want something different, its primarily up to you to educate yourself to do it.

Rion

@Zigurana said:

@Rion See here for a list of people who have created a fork of ES. Apparently its very difficult on Github to get a full list of forks, although you can seen them all when you actually want to do comparisons for pull-requests.

I guess the point right now is not that there are no people willing to work on it, but more who will stand up to draw up a comprehensive action plan with a long term vision for ES, and is capable of pulling that particular cart for the foreseeable future?
It will need to show these things before RetroPie can even consider switching upstream branches.

The good thing about Herdingers project is that at least he/she is willing to publictly discuss the long-term goals. See here.

Wow that's more then i could imagine.
But i see your point. To many forks and not many working together.

It's a shame really.

Twitch0815

Even other forks for example Recalbox scraping is more of a pain in the but then it is on Retropie.
They do the same thing .fullscrape

Until someone puts a massive amount of time into developing a gui wrapper for a scraper service, the better option is always going to be a command line scraper. And that is assuming thegaemesdb.net stays up. There are just some things cli clients do better.

We need to keep ask's in perspective, we as a community put a huge ask on thegamesdb but their donations are not reflective of the demand we as a community put on them and the demand for their service is literally crushing their servers and we get vocal about our dislike of that feature being unavailable regardless of whether we have ever donated or making vocal and somewhat alarmist replace thegamesdb develop an alternative switch or code something better. Additionally asking the limited developers we have to focus on the one demand we have a working solution for its just a few more manual steps to get it working takes them away from kernel improvements new ports, features etc etc.

Honestly as long as gamesdb.net scraping artwork with sspeh is a breeze and there is only one other frontend period I have seen do a better job easier with artwork and it is a mac OSX frontend. Mess with Hyperspin and emumovies accounts paid subscriptios download rate limits having to edit xml by hand renaming romsets its a pain in the but.

And honestly its not going to be very long before Attract Mode or Meh replace Emulation Station as the default front-end for Retropie or alternatives at least and the artwork structure for those is going to be manual and manual for a good while. Luckily the Hyperspin and EmuMovies community have already done that heavy lifting though were going to have to deal with getting emumovie accounts and all the hoops that comes with that but the end product will look great. I would much rather the devs look forward or we put the ask to the meh and attract mode developers to make the art scraping and organizing process as user friendly as possible and let our devs keep focusing on what they want to work on as well as the good ideas we come forward with but we as a community need to be fair and not ask for major development on a platform that will be irrelevant in coming months.

Zigurana

@Twitch0815 said:

...
And honestly its not going to be very long before Attract Mode or Meh replace Emulation Station as the default front-end for Retropie or alternatives at least and the artwork structure for those is going to be manual and manual for a good while.
...
I would much rather the devs look forward or we put the ask to the meh and attract mode developers to make the art scraping and organizing process as user friendly as possible and let our devs keep focusing on what they want to work on as well as the good ideas we come forward with but we as a community need to be fair and not ask for major development on a platform that will be irrelevant in coming months.

Do you know these things, or are you speculating? I think there has been a lot of investments into ES, and there is a certain amount of momentum that needs to build up before any switch will be worth the hassle. I agree that we need to keep looking outside our bubble, and also not over-work the few developers we have, but switching to a completely new front-end is a large step.