Skyscraper: problem with re-naming rom names without bracket info

timmay

Pi 3 Model B+
RetroPie 4.5.2
SanDisk 64GB SD card

I used Skyscraper to get video previews and create the gamelists which worked fine but I need to re-name roms of a couple of systems because they include all the infos (year, region etc.) that I don't want.

So I generated a new gamelist through Skyscraper with no bracket info enabled so that the roms are listed with the game name only. It worked but now video previews are not shown anymore because in the new gamelist the path info for screenshots and videos are removed.

I can't get it to work - is it possible to list the games nice & clean without bracket info without deleting the paths to videos in the gamelist.xml?

I know I could adjust the game names in gamelist.xml manually but with multiple systems some of which containing 100+ roms it would take forever...

(I searched forums and read all related threads I could find but did not find solutions to this specific issue)

mitu

Normally the re-generation of the gamelist should not have this problem - the videos and images should be added to the gamelist from the cache and the paths adjusted.
Did you leave the video and image options enabled when you generated the gamelist with the no bracked info option ?

muldjord

I think the main culprit here is the renaming. If you have zip-files (and several script files) I do the sha1 cache id checksum on the filename instead of the file contents due to zip-files being so unstable (several different ways to zip files and people might decompress/recompress). It's a decision I've since halfway regretted since I've learned a lot about how people use zips since I started Skyscraper, and it's quite clear that zip is almost the only filetype used. I used to decompress my roms, always. I don't anymore. So basically we're stuck with this decision I made 3 years ago since it would break the caches for everyone if I made it checksummed the the contents of the zips instead. So, when using zip-files, you need to keep the filenames as they were, or the files will no longer match up with the cache id.

Let me emphasize that I would love to change this, but can't really see a good way of doing it without risking thousands of Skyscraper users cracking down on me when I've missed a bug that would F up their caches when migrating. I try to leave the rom id system alone as much as possible. I get cold sweats every time I touch that code even a tiny bit.

EDIT: I might do this for Skyscraper 4.x though and introduce a "backup first and do it at your own risk" migration.

EDIT2: This is the cache id generation code, should anyone be interested:

QString NameTools::getCacheId(const QFileInfo &info)
{
  QCryptographicHash cacheId(QCryptographicHash::Sha1);

  // Use checksum of filename if file is a script or an "unstable" compressed filetype
  bool cacheIdFromData = true;
  if(info.suffix() == "uae" || info.suffix() == "cue" ||
     info.suffix() == "sh" || info.suffix() == "svm" ||
     info.suffix() == "scummvm" || info.suffix() == "mds" ||
     info.suffix() == "zip" || info.suffix() == "7z" ||
     info.suffix() == "gdi" || info.suffix() == "ml" ||
     info.suffix() == "bat") {
    cacheIdFromData = false;
  }
  // If file is larger than 50 MBs, use filename checksum for cache id for optimization reasons
  if(info.size() > 52428800) {
    cacheIdFromData = false;
  }
  // If file is empty always do checksum on filename
  if(info.size() == 0) {
    cacheIdFromData = false;
  }
  if(cacheIdFromData) {
    QFile romFile(info.absoluteFilePath());
    if(romFile.open(QIODevice::ReadOnly)) {
      while(!romFile.atEnd()) {
	cacheId.addData(romFile.read(1024));
      }
      romFile.close();
    } else {
      printf("Couldn't calculate cache id of rom file '%s', please check permissions and try again, now exiting...\n", info.fileName().toStdString().c_str());
      exit(1);
    }
  } else {
    cacheId.addData(info.fileName().toUtf8());
  }

  return cacheId.result().toHex();
}

mitu

@muldjord cache invalidation is one of the hardest problems in CS :). But

are you sure it's the same problem here ?
I thought Skyscraper was already at version 3.x ?

muldjord

@mitu said in Skyscraper: problem with re-naming rom names without bracket info:

@muldjord cache invalidation is one of the hardest problems in CS :). But

are you sure it's the same problem here ?

CS?

I thought Skyscraper was already at version 3.x ?

Haha, yes, it is. I'm currently working on a different project that's on 2.x, so I got it mixed up. :D

mitu

@muldjord said in Skyscraper: problem with re-naming rom names without bracket info:

CS?

Computer science. Or you might just say programming/software engineering.

muldjord

@mitu said in Skyscraper: problem with re-naming rom names without bracket info:

Computer science. Or you might just say programming/software engineering.

Ah yes, the joys of supporting legacy formats. And after I just re-read OP, you might be right. He does not seem to be talking about renaming the files themselves. I'll need to think a bit more about this.

muldjord

@timmay Can you clear this up for me: Did you rename the files as well? I'm guessing you actually didn't.

Also, are you sure screenshots are gone from the gamelist.xml as well (please post an entry example)? Or is it just video? Then you might have forgotten to use the --videos option, otherwise they'll be ignored when regenerating (or in case of the RetroPie script, make sure videos are enabled). Remember that you can always set the videos=true in ~/.skyscraper/config.ini in the [main] section to always enable them.

If not, then there might be an issue here that I am not aware of and that I would like to look into.

timmay

Sorry, maybe I was a bit fuzzy - correct, I never renamed any rom files I just want to rename the game names shown in ES. To be precise in gamelist.xml it's the line

<name>GameName</name>

that has to be adjusted. I always used Skyscraper through Retropie-script and I'm sure videos are enabled since I never changed that option but I will double check on that and also post an entry example. I am not at home right now, will do that asap!

timmay

I did another run renaming the games for atari5200 and sega32x. The current settings in Skyscraper are:
gather source: Screenscraper
cache enabled for: screenshots, covers, marquees
rom names: source name
remove bracket info: enabled
use rom folders for gamelist+media: disabled
download videos: enabled

Somehow it worked for sega32x but did not for atari5200. The games of both systems have been renamed successfully but only for sega32x the paths for videos etc. have been kept in the gamelist.xml. Seems it works correctly for some systems but not all of them. So I have something wrong going on somewhere??
Here are the entry examples before and after renaming:

atari5200 before:

<game>
<path>/home/pi/RetroPie/roms/atari5200/Beamrider (USA).a52</path>
<name>Beamrider (USA)</name>
<cover />
<image>/home/pi/.emulationstation/downloaded_media/atari5200/screenshots/Beamrider (USA).png</image>
<marquee>/home/pi/.emulationstation/downloaded_media/atari5200/marquees/Beamrider (USA).png</marquee>
<video>/home/pi/.emulationstation/downloaded_media/atari5200/videos/Beamrider (USA).mp4</video>
<rating>0.8</rating>
<desc>Beamrider is a ...</desc>
<releasedate>19830101T000000</releasedate>
<developer>Activision</developer>
<publisher>Activision</publisher>
<genre>Action, Shoot'em Up</genre>
<players>4</players>
<lastplayed>20191030T190656</lastplayed>
<playcount>13</playcount>
</game>

atari5200 afterwards:

<game>
<path>/home/pi/RetroPie/roms/atari5200/Beamrider (USA).a52</path>
<name>Beamrider</name>
<cover />
<image />
<marquee />
<video />
<rating>0.8</rating>
<desc>Beamrider is a ...</desc>
<releasedate>19830101T000000T000000</releasedate>
<developer>Activision</developer>
<publisher>Activision</publisher>
<genre>Action, Shoot'em Up</genre>
<players>4</players>
<lastplayed>20191030T190656</lastplayed>
<playcount>13</playcount>
</game>

sega32x before:
(I post the first lines only, until '<video>' and leave out rating, description, publisher etc.)*

<game>
<path>/home/pi/RetroPie/roms/sega32x/After Burner Complete (Europe).zip</path>
<name>After Burner Complete (Europe)</name>
<cover />
<image>/home/pi/.emulationstation/downloaded_media/sega32x/screenshots/After Burner Complete (Europe).png</image>
<marquee>/home/pi/.emulationstation/downloaded_media/sega32x/marquees/After Burner Complete (Europe).png</marquee>
<video>/home/pi/.emulationstation/downloaded_media/sega32x/videos/After Burner Complete (Europe).mp4</video>
...
</game>

sega32x afterwards:

<game>
<path>/home/pi/RetroPie/roms/sega32x/After Burner Complete (Europe).zip</path>
<name>After Burner Complete</name>
<cover />
<image>/home/pi/.emulationstation/downloaded_media/sega32x/screenshots/After Burner Complete (Europe).png</image>
<marquee>/home/pi/.emulationstation/downloaded_media/sega32x/marquees/After Burner Complete (Europe).png</marquee>
<video>/home/pi/.emulationstation/downloaded_media/sega32x/videos/After Burner Complete (Europe).mp4</video>
...
</game>

Clyde

@muldjord said in Skyscraper: problem with re-naming rom names without bracket info:

EDIT: I might do this for Skyscraper 3.x though and introduce a "backup first and do it at your own risk" migration.

I would decidedly support this decision. :)

[end of offtopic, please carry on]

mitu

@timmay
The paths are the same before and after, but I don't understand why the video/image are skipped for the atari2600.

Can you try and re-generate the gamelist again, then post the last log file from /home/pi/RetroPie-Setup/logs to pastebin.com ? It should have the output from Skyscraper's operations.

timmay

Sure, here's the log after re-generating the atari5200 gamelist.
https://pastebin.com/adUsQ4Az

I also checked the previous log (renaming atari5200 and sega32x games) and noticed that with sega32x all paths for videos, screenshots, publisher etc. are logged and therefore included in the new gamelist.xml. However the log for atari5200 just lists a bunch of lines with

Resource is missing 'sha1' attribute, skipping...

I assume that is the issue, I'm just not sure what that means or how to correct it.

timmay

I also tried re-generating gamelists for fds, gb, gba, gbc - the log does not have the 'resource is missing' line as in the atari5200 log but it's the same result: all of those 4 are missing the paths for videos etc.
log is here:
https://pastebin.com/Kk2n4E2u

mitu

@timmay said in Skyscraper: problem with re-naming rom names without bracket info:

I assume that is the issue, I'm just not sure what that means or how to correct it.

Probably yes, try to re-scrape the atari5200 system, it looks like Skycraper cannot find anything in the cache.

timmay

Done. That seems to solve it. After re-scraping those systems re-generating the gamelists worked as it should.

I wonder why that issue occurred in the first place, because of course I checked the folders for videos, screenshots etc. and in both Skyscraper/cache and emulationstation/downloaded media all media files were present, same number of videos as roms in each systems rom folder, too. I never changed anything in there and used the same settings +source in Skyscraper when initially scraping, so that's odd?

Anyways, thanks a lot for helping out!
Gonna try it now for the remaining systems that need re-naming game names.

muldjord

@timmay said in Skyscraper: problem with re-naming rom names without bracket info:

Resource is missing 'sha1' attribute, skipping...

I assume that is the issue, I'm just not sure what that means or how to correct it.

That message only occurs while parsing the resource cache db.xml file for the system. And it indicates that the obsoleted sha1 attribute is missing. I think I know what is going on. Do you perhaps sometimes run Skyscraper from command-line and sometimes from the RetroPie script? And do you then maybe have two different versions? This would make sense, as the old version would look for the sha1 attribute. The newer version's give the message Resource is missing unique id, skipping...instead.

timmay

I had indeed run Skyscraper from Terminal once because from RP-script it wouldn't scrape the CPS1,2,3 systems. How is it possible to have two Skyscraper versions? If that's the case how to repair it- update Skyscraper? (Last update was about a week ago)

I now have successfully re-scraped and re-generated gamelists for a couple of systems like atari7800, gb, gba, mastersystem, megadrive, n64, nds, nes, psx, segacd, vectrex, zxspectrum.

Only Atari800 won't work, re-scraping skips all games with the line game not found and therefor the gamelist entries for video paths etc. are blank. Before all the media was downloaded and in the correct locations. In the beginning when setting up the systems scraping Atari800 caused no trouble at all. Since then I did not alter anything in the rom folder, so maybe something else got corrupted within this system.
Maybe completely set up atari800 again or if it makes sense is it possible to delete or replace the db.xml file?

timmay

@muldjord said in Skyscraper: problem with re-naming rom names without bracket info:

Do you perhaps sometimes run Skyscraper from command-line and sometimes from the RetroPie script? And do you then maybe have two different versions?

Maybe I misunderstood and by two versions you meant two different scrapes for a system instead two versions of Skyscraper?
In this case I'm pretty sure I only used Skyscraper from terminal for CPS systems as those were the only ones that originally didn't scrape from RP-script.