Skyscraper two ways: games beginning with "The", and config.ini settings not working

[sleve_mcdichael]

@muldjord, or anyone else who can help. I'm trying to figure out why some of my manual edits to esgamelist are not being retained, and I think I have identified a couple of issues.

1: Game titles beginning with "The," e.g. The Simpsons. No matter what I do, including manually setting the title with --cache edit, this gets rendered as Simpsons, The when building a gamelist. I'll do:

Skyscraper -p arcade --cache edit ~/RetroPie/roms/arcade/simpsons2p.zip

I can see:

Title:      'The Simpsons' (user)

I do:

Skyscraper -p arcade --flags unattend

The resulting gamelist says Simpsons, The.

Is this a setting that can be disabled somewhere, or just something I'll have to cope with? (I already have "sort name" set manually as Simpsons, The in ES so it sorts correctly either way.

(Incidentally, this sort name setting seems to persist through gamelist rebuilds, even though it doesn't seem to be one of the resources cached by Skyscraper. Does anyone know where in the system this info is stored, then?) (EDIT: I was wrong, "sort name" does not persist through a full gamelist rebuild. I must have only been doing --flags unattendskip when I was seeing this. How trivial would it be to incorporate this "sort name" field in Skyscraper? I know the scraping sources won't have it but it could be imported from esgamelist, not just for "the" games but also for things like Donkey Kong Jr. being properly sorted between 1 and 3, or sequels like DoDonPachi and Ms. Pac-Man.)

2: The config.ini settings unattend="true" and cacheRefresh="true" don't seem to be working, in the [esgamelist] section, at least. Did I do something wrong? Complete contents of /home/pi/.skyscraper/config.ini:

[main]

# define paths
gameListFolder="/home/pi/.emulationstation/gamelists"
mediaFolder="/home/pi/.emulationstation/downloaded_media"
#inputFolder="/home/pi/RetroPie/roms"
#cacheFolder="/home/pi/.skyscraper/cache"
#importFolder="/home/pi/.skyscraper/import"

# video settings
videos="true"
videoSizeLimit="42"
videoConvertCommand="videoconvert.sh %i %o"
videoConvertExtension="mp4"
symlink="true"

# generic
brackets="false"

[screenscraper]
# userCreds="user:pass"
userCreds="myUserName:myPassword"

[esgamelist]
# import customized ES textual data
cacheRefresh="true"
unattend="true"
cacheCovers="false"
cacheScreenshots="false"
cacheWheels="false"
cacheMarquees="false"
videos="false"

When I scrape using:

Skyscraper -p arcade -s esgamelist

It (1) still asks if I want to skip already existing game list entries unless I also use --flags unattend, and (2) does not update with my latest edits to those existing entries, unless I also use --cache refresh. The remaining settings appear to be working correctly (unless -s esgamelist never caches artwork or videos? In any case, there are none in the cache for the esgamelist module.) Skyscraper built from source Mon 8 Mar 2021. Setup script tells me it is the latest version.

[muldjord]

@sleve_mcdichael

Is this a setting that can be disabled somewhere, or just something I'll have to cope with?

It can't currently be disabled, but I'll add a --flags theinfront to the next release.

It (1) still asks if I want to skip already existing game list entries

It can currently not be used under a scraping module specific section. But there's no reason it shouldn't be able to, so I'll add that to the next release. Thanks for noticing.

and cacheRefresh="true" don't seem to be working, in the [esgamelist] section, at least.

This is working, so I'm thinking it's something else.

(unless -s esgamelist never caches artwork or videos? In any case, there are none in the cache for the esgamelist module.)

According to your settings, you've disabled them, so that's the reason it doesn't cache them.

[muldjord]

@sleve_mcdichael Skyscraper 3.6.8 out now. Please update.

[sleve_mcdichael]

@muldjord thank you so much! Re: cacheRefresh in the module section, here is a lightly-edited log from this morning:

# Skyscraper installed from binary (built 23 Feb 2021):

pi@retropie:~ $ cat /home/pi/.emulationstation/gamelists/arcade/gamelist.xml | grep \<name\> | grep Simpsons
                <name>The Simpsons</name>
pi@retropie:~ $ Skyscraper -p arcade --cache edit /home/pi/RetroPie/roms/arcade/simpsons2p.zip | grep Title
Title:          'Simpsons, The' (esgamelist)
q
pi@retropie:~ $ cat /home/pi/.skyscraper/config.ini
[main]

# define paths
gameListFolder="/home/pi/.emulationstation/gamelists"
mediaFolder="/home/pi/.emulationstation/downloaded_media"
#inputFolder="/home/pi/RetroPie/roms"
#cacheFolder="/home/pi/.skyscraper/cache"
#importFolder="/home/pi/.skyscraper/import"

# video settings
videos="true"
videoSizeLimit="42"
videoConvertCommand="videoconvert.sh %i %o"
videoConvertExtension="mp4"
symlink="true"

# generic
brackets="false"

[screenscraper]
# userCreds="user:pass"
userCreds="REDACTED:REDACTED"

[esgamelist]
# import customized ES textual data
cacheRefresh="true"
#unattend="true"
cacheCovers="false"
cacheScreenshots="false"
cacheWheels="false"
cacheMarquees="false"
videos="false"
pi@retropie:~ $ Skyscraper -p arcade -s esgamelist --flags unattend

# scroll scroll scroll

---- Resource gathering run completed! YAY! ----
Writing quick id xml, please wait... Done!
Writing 1705 (0 new) resources to cache, please wait... Done!

---- And here are some neat stats :) ----
Total completion time: 00:00:00

Average search match: 100%
Average entry completeness: 60%

Total number of games: 73
Successfully processed games: 73
Skipped games: 0 (Filenames saved to '/home/USER/.skyscraper/skipped-arcade-esgamelist.txt')

pi@retropie:~ $ Skyscraper -p arcade --cache edit /home/pi/RetroPie/roms/arcade/simpsons2p.zip | grep Title
Title:          'Simpsons, The' (esgamelist)
q
pi@retropie:~ $ sudo /home/pi/RetroPie-Setup/retropie_setup.sh 

# ...Installed Skyscraper from source (built: 31 Mar 2021)

pi@retropie:~ $ Skyscraper -p arcade -s esgamelist --flags unattend

# scroll scroll scroll

---- Resource gathering run completed! YAY! ----
Writing quick id xml, please wait... Done!
Writing 1705 (0 new) resources to cache, please wait... Done!

---- And here are some neat stats :) ----
Total completion time: 00:00:00

Average search match: 100%
Average entry completeness: 75%

Total number of games: 73
Successfully processed games: 73
Skipped games: 0 (Filenames saved to '/home/USER/.skyscraper/skipped-arcade-esgamelist.txt')

pi@retropie:~ $ Skyscraper -p arcade --cache edit /home/pi/RetroPie/roms/arcade/simpsons2p.zip | grep Title
Title:          'The Simpsons' (esgamelist)
q
pi@retropie:~ $

Play-by-play:

• Gamelist: The Simpsons
• Cache: Simpsons, The (esgamelist)
• Refresh enabled in module via config.ini
• Scrape gamelist:
• Cache: Simpsons, The (esgamelist)
• Update from source, scrape gamelist:
• Cache: The Simpsons (esgamelist)

Something seems to have changed between 23 Feb and today (n.b., before today, I was installed from source built ca. 7 Mar, still getting the "before" behavior.) In any case, it works now, so hooray!

unattend="true" in the module section is working now as well, thank you!

Re: flag theinfront -- as I understand it, when building a gamelist, this flag if enabled will try and manipulate the title to put "The" in front, and if disabled it will try and manipulate the title to put "The" in back. In both cases, it's manipulating the cached title before writing the gamelist. Would it make sense to have an option that says "don't manipulate anything, just write it exactly how it says in the cache"?

(Aside: what about other articles like "a/an," "in," "to," "of," or "from"? I'm not too sure how often these would come up, but I can think of at least one instance in "A Boy and His Blob.")

One suggestion:

articleFix="true" - attempts to fix article placement in game titles. If disabled, returns title as-is.

articleFirst="true" - puts the article first, only if articleFix is also true. When disabled, puts the article last, only if articleFix is true. No effect if articleFix is false.

(Edit: does it have to be "true/false" or would articleFix=<front|back|none> be a way to get it done in one go?)

What do you think?

[muldjord]

@sleve_mcdichael said in Skyscraper two ways: games beginning with "The", and config.ini settings not working:

In both cases, it's manipulating the cached title before writing the gamelist. Would it make sense to have an option that says "don't manipulate anything, just write it exactly how it says in the cache"?

Screenscraper tends to always return "Game, The" while TheGamesDb tends to return "The Game". Sometimes they will do some with one and some with the other. I can't think of instances where you wouldn't want to align those when creating a gamelist. So unless you have a strong argument for it, I'll leave it as it is.

(Aside: what about other articles like "a/an," "in," "to," "of," or "from"? I'm not too sure how often these would come up, but I can think of at least one instance in "A Boy and His Blob.")

Not sure about this. I don't think you would normally move those around for sorting or presentation. So I'll leave it as is. Thanks for the suggestion though.

[sleve_mcdichael]

@muldjord said in Skyscraper two ways: games beginning with "The", and config.ini settings not working:

I can't think of instances where you wouldn't want to align those when creating a gamelist. So unless you have a strong argument for it, I'll leave it as it is.

I suppose the "for" argument is just that it could be frustrating to have your meticulously-edited cached data be subjected to manipulation before writing to the gamelist. If you then re-scrape that manipulated gamelist, those changes have now wormed their way back into your cache, as well (though I do admit that with theinfront available, I see this being less of an issue.)

I guess it's just the control freak in me believes that since you can manually edit the cache to say whatever you want, it should then be able to pass what you've manually entered, at least with the option of no further manipulation. As customizable and controllable as it is, and then there's this one instance where it's just like "nah, we do this one thing my way instead." It's like having an incredible endless smorgasbord with everything you can imagine and then at the end they're like, "Coke or Pepsi? No tea."

[muldjord]

@sleve_mcdichael

I suppose the "for" argument is just that it could be frustrating to have your meticulously-edited cached data be subjected to manipulation before writing to the gamelist.

I feel like that frustration you mention is a thought up example. We are just talking about "The" which is a normal thing to move around when sorting or generating alphabetical lists. I'm not a fan of adding functionality for the sake of it. The CLI option is a flag which can be enabled or disabled. Supporting tristate would mess with that and complicate things needlessly.

So, please bear with me, thank you for your suggestions, but you will have to live with this one. :)

[sleve_mcdichael]

@muldjord yes, you are correct. Now that theinfront is available, the remaining frustration is very much imagined, not actual. I don't really expect it to be an issue in implementation, just in principle ;)

However, I've now got another issue.

Today I am getting "video conversion: failed" errors when scraping a new game. I tried refreshing one that worked previously, and got the same error. Reverted to binary 23 Feb 2021 and both games worked fine ("video conversion: success!"). Reinstalled today's source, and:

pi@retropie:~ $ Skyscraper -p arcade -s screenscraper /home/pi/RetroPie/roms/arcade/1942.zip --verbosity 3
------------------------------------------
Running Skyscraper v3.6.8 by Lars Muldjord
------------------------------------------
Platform:           'arcade'
Scraping module:    'screenscraper'
Input folder:       '/home/pi/RetroPie/roms/arcade'
Game list folder:   '/home/pi/.emulationstation/gamelists/arcade'
Covers folder:      '/home/pi/.emulationstation/downloaded_media/arcade/covers'
Screenshots folder: '/home/pi/.emulationstation/downloaded_media/arcade/screenshots'
Wheels folder:      '/home/pi/.emulationstation/downloaded_media/arcade/wheels'
Marquees folder:    '/home/pi/.emulationstation/downloaded_media/arcade/marquees'
Videos folder:      '/home/pi/.emulationstation/downloaded_media/arcade/videos'
Cache folder:       'cache/arcade'

DID YOU KNOW: You can create reports for all files missing certain resources in the cache using the '--cache report:missing=<GROUP or RESOURCE1,RESOURCE2,...> option. Great ways to add missing resources that the online sources are having trouble providing are the '--cache edit' option or the '-s import' module.

Fetching limits for user 'REDACTED', just a sec...
Setting threads to 2 as allowed for the supplied user credentials.

Reading and parsing quick id xml, please wait... Done!
Reading and parsing resource cache, please wait... Done!
Successfully parsed 1718 resources!

Resource cache stats for selected platform:
'esgamelist' module
  Titles       : 73
  Platforms    : 73
  Descriptions : 73
  Publishers   : 73
  Developers   : 72
  Ages         : 0
  Tags         : 73
  Ratings      : 72
  ReleaseDates : 73
  Covers       : 0
  Screenshots  : 0
  Wheels       : 0
  Marquees     : 0
  Videos       : 0
'import' module
  Titles       : 0
  Platforms    : 6
  Descriptions : 0
  Publishers   : 0
  Developers   : 0
  Ages         : 0
  Tags         : 0
  Ratings      : 0
  ReleaseDates : 0
  Covers       : 2
  Screenshots  : 2
  Wheels       : 2
  Marquees     : 2
  Videos       : 6
'screenscraper' module
  Titles       : 74
  Platforms    : 74
  Descriptions : 74
  Publishers   : 74
  Developers   : 73
  Ages         : 11
  Tags         : 74
  Ratings      : 73
  ReleaseDates : 74
  Covers       : 73
  Screenshots  : 74
  Wheels       : 74
  Marquees     : 74
  Videos       : 73

Looking for optional 'priorities.xml' file in cache folder... Found!
Priorities loaded successfully!

Adding files from subdir: '/home/pi/RetroPie/roms/arcade/fbneo'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/memcard'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/nvram'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/ctrlr'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/hi'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/diff'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/comment'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/ini'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/input'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/image'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/cfg'

Trying to parse and load existing game list metadata... Success!

Starting scraping run on 1 files using 2 threads.
Sit back, relax and let me do the work! :)

Warning! Couldn't add resource to cache. Have you run out of disk space?
cover: '2149208' > '323270', choosing resize for optimal result!
marquee: '158970' > '107347', choosing resize for optimal result!
#1/1 (T1) Pass 1 ---- Game '1942' found! :) ----
Scraper:        screenscraper
From cache:     NO
Search match:   100 %
Compare title:  '1942'
Result title:   '1942' ()
Platform:       'Capcom Classics' ()
Release Date:   '1984-01-01' ()
Developer:      'Capcom' ()
Publisher:      'Capcom' ()
Players:        '2' ()
Ages:           '1+' ()
Tags:           'Shoot'em Up, Shoot'em up / Vertical' ()
Rating (0-1):   '0.7' ()
Cover:          YES ()
Screenshot:     YES ()
Wheel:          YES ()
Marquee:        YES ()
Video:          YES ()

Description: ()
'"1942" is a vertically-scrolling shoot-em-up set in the Pacific theatre during World War II, in which the goal is to reach Tokyo and destroy the entire Japanese air fleet. 

The player pilots a plane dubbed the "Super Ace" (its appearance is that of a Lockheed P-38 Lightning) and has to shoot down waves of enemy planes while avoiding incoming enemy fire. The 'Super Ace' can perform a limited number of rolls or 'loop-the-loops' to evade enemy planes and shots. 

During the game, waves of red enemy planes periodically appear. If the player manages to destroy a red wave, a power-up, in the form of a 'POW' symbol, will be dropped and can be picked up. The power-up will award either increased fire-power (doubling the player's guns from two to four), a smart bomb that destroys all on-screen enemy planes the instant it's collected, two wingmen planes that flank the Super Ace and increase fire power, or simply extra points.

At the end of each stage the 'Super Ace' lands on an aircraft carrier and bonus points are awarded based on player performance. "1942" differs from other games in that its levels are numbered in reverse order, so the game begins at stage 32 and ends at stage 1.'

Cache output:
Video conversion: 
%i: '/opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4'
%o: '/opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/tmpfile_769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4'
Running command: 'videoconvert.sh /opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4 /opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/tmpfile_769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4'


Failed! (set higher '--verbosity N' level for more info)

Debug output:
Tried with: 'crc=F52D7D3C&md5=89625F073887A5915C3CE2E2E4DA8875&sha1=BF332AC22C7FDB5470AFF4172A9051555E6E1266&romnom=1942.zip&romtaille=106788'
Platform: arcade

#1/1, (1/0)
Elapsed time   : 00:00:22
Est. time left : 00:00:00

---- Resource gathering run completed! YAY! ----
Writing quick id xml, please wait... Done!
Writing 1717 (-1 new) resources to cache, please wait... Done!

---- And here are some neat stats :) ----
Total completion time: 00:00:22

Average search match: 100%
Average entry completeness: 100%

Total number of games: 1
Successfully processed games: 1
Skipped games: 0 (Filenames saved to '/home/USER/.skyscraper/skipped-arcade-screenscraper.txt')

pi@retropie:~ $ 

I see where it asks if I might be out of disk space. I am not; I have 20G available:

pi@retropie:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  8.1G   20G  30% /
devtmpfs        805M     0  805M   0% /dev
tmpfs           934M   12K  934M   1% /dev/shm
tmpfs           934M  9.6M  925M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           934M     0  934M   0% /sys/fs/cgroup
/dev/mmcblk0p1  256M   46M  210M  18% /boot
tmpfs           187M     0  187M   0% /run/user/1000
pi@retropie:~ $

Here is my conversion script:

pi@retropie:~ $ ls -l /home/pi/.skyscraper/videoconvert.sh 
-rwxr-xr-x 1 pi pi 437 Mar 12 22:52 /home/pi/.skyscraper/videoconvert.sh
pi@retropie:~ $ cat /home/pi/.skyscraper/videoconvert.sh 
#! /bin/bash
# Script provided by jgcobra

echo Converting video...
echo Source: $1
echo Dest: $2

# Check if the video is already in 420p format
c=$(ffprobe "$1" 2>&1 | grep -c yuv420p)

if [ "$c" -eq 0 ]
then
        echo Incorrect source video format detected. Converting video to 420p format...
        ffmpeg -i "$1" -y -c:v libx264 -pix_fmt yuv420p -acodec copy "$2"
else
        echo Video is in correct 420p format. Copying as is...
        cp "$1" "$2"
fi

...and current config.ini:

pi@retropie:~ $ cat /home/pi/.skyscraper/config.ini
[main]

# define paths
gameListFolder="/home/pi/.emulationstation/gamelists"
mediaFolder="/home/pi/.emulationstation/downloaded_media"
#inputFolder="/home/pi/RetroPie/roms"
#cacheFolder="/home/pi/.skyscraper/cache"
#importFolder="/home/pi/.skyscraper/import"

# video settings
videos="true"
videoSizeLimit="42"
videoConvertCommand="videoconvert.sh %i %o"
videoConvertExtension="mp4"
symlink="true"

# generic
brackets="false"
theInFront="true"

[screenscraper]
# userCreds="user:pass"
userCreds="REDACTED:REDACTED"

[esgamelist]
# import customized ES textual data
cacheRefresh="true"
unattend="true"
cacheCovers="false"
cacheScreenshots="false"
cacheWheels="false"
cacheMarquees="false"
videos="false"

Edit: here's the log from the working binary version:

pi@retropie:~ $ Skyscraper -p arcade -s screenscraper /home/pi/RetroPie/roms/arcade/1942.zip --verbosity 3
------------------------------------------
Running Skyscraper v3.6.7 by Lars Muldjord
------------------------------------------
Platform:           'arcade'
Scraping module:    'screenscraper'
Input folder:       '/home/pi/RetroPie/roms/arcade'
Game list folder:   '/home/pi/.emulationstation/gamelists/arcade'
Covers folder:      '/home/pi/.emulationstation/downloaded_media/arcade/covers'
Screenshots folder: '/home/pi/.emulationstation/downloaded_media/arcade/screenshots'
Wheels folder:      '/home/pi/.emulationstation/downloaded_media/arcade/wheels'
Marquees folder:    '/home/pi/.emulationstation/downloaded_media/arcade/marquees'
Videos folder:      '/home/pi/.emulationstation/downloaded_media/arcade/videos'
Cache folder:       'cache/arcade'

DID YOU KNOW: You can turn off these hints using the '--flags nohints' command line flag.

Fetching limits for user 'REDACTED', just a sec...
Setting threads to 2 as allowed for the supplied user credentials.

Reading and parsing quick id xml, please wait... Done!
Reading and parsing resource cache, please wait... Done!
Successfully parsed 1680 resources!

Resource cache stats for selected platform:
'esgamelist' module
  Titles       : 72
  Platforms    : 72
  Descriptions : 72
  Publishers   : 72
  Developers   : 71
  Ages         : 0
  Tags         : 72
  Ratings      : 71
  ReleaseDates : 72
  Covers       : 0
  Screenshots  : 0
  Wheels       : 0
  Marquees     : 0
  Videos       : 0
'import' module
  Titles       : 0
  Platforms    : 6
  Descriptions : 0
  Publishers   : 0
  Developers   : 0
  Ages         : 0
  Tags         : 0
  Ratings      : 0
  ReleaseDates : 0
  Covers       : 2
  Screenshots  : 2
  Wheels       : 2
  Marquees     : 2
  Videos       : 6
'screenscraper' module
  Titles       : 72
  Platforms    : 72
  Descriptions : 72
  Publishers   : 72
  Developers   : 71
  Ages         : 11
  Tags         : 72
  Ratings      : 71
  ReleaseDates : 72
  Covers       : 71
  Screenshots  : 72
  Wheels       : 72
  Marquees     : 72
  Videos       : 70

Looking for optional 'priorities.xml' file in cache folder... Found!
Priorities loaded successfully!

Adding files from subdir: '/home/pi/RetroPie/roms/arcade/fbneo'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/memcard'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/nvram'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/ctrlr'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/hi'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/diff'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/comment'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/ini'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/input'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/image'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/cfg'

Trying to parse and load existing game list metadata... Success!

Starting scraping run on 1 files using 2 threads.
Sit back, relax and let me do the work! :)

cover: '2149208' > '323270', choosing resize for optimal result!
marquee: '158970' > '107347', choosing resize for optimal result!
#1/1 (T1) Pass 1 ---- Game '1942' found! :) ----
Scraper:        screenscraper
From cache:     NO
Search match:   100 %
Compare title:  '1942'
Result title:   '1942' ()
Platform:       'Capcom Classics' ()
Release Date:   '1984-01-01' ()
Developer:      'Capcom' ()
Publisher:      'Capcom' ()
Players:        '2' ()
Ages:           '1+' ()
Tags:           'Shoot'em Up, Shoot'em up / Vertical' ()
Rating (0-1):   '0.7' ()
Cover:          YES ()
Screenshot:     YES ()
Wheel:          YES ()
Marquee:        YES ()
Video:          YES ()

Description: ()
'"1942" is a vertically-scrolling shoot-em-up set in the Pacific theatre during World War II, in which the goal is to reach Tokyo and destroy the entire Japanese air fleet. 

The player pilots a plane dubbed the "Super Ace" (its appearance is that of a Lockheed P-38 Lightning) and has to shoot down waves of enemy planes while avoiding incoming enemy fire. The 'Super Ace' can perform a limited number of rolls or 'loop-the-loops' to evade enemy planes and shots. 

During the game, waves of red enemy planes periodically appear. If the player manages to destroy a red wave, a power-up, in the form of a 'POW' symbol, will be dropped and can be picked up. The power-up will award either increased fire-power (doubling the player's guns from two to four), a smart bomb that destroys all on-screen enemy planes the instant it's collected, two wingmen planes that flank the Super Ace and increase fire power, or simply extra points.

At the end of each stage the 'Super Ace' lands on an aircraft carrier and bonus points are awarded based on player performance. "1942" differs from other games in that its levels are numbered in reverse order, so the game begins at stage 32 and ends at stage 1.'

Cache output:
Video conversion: 
%i: '/opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4'
%o: '/opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/tmpfile_769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4'
Running command: 'videoconvert.sh /opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4 /opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/tmpfile_769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4'
Converting video...
Source: /opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4
Dest: /opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/tmpfile_769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4
Video is in correct 420p format. Copying as is...


Success!

Debug output:
Tried with: 'crc=F52D7D3C&md5=89625F073887A5915C3CE2E2E4DA8875&sha1=BF332AC22C7FDB5470AFF4172A9051555E6E1266&romnom=1942.zip&romtaille=106788'
Platform: arcade

#1/1, (1/0)
Elapsed time   : 00:00:11
Est. time left : 00:00:00

---- Resource gathering run completed! YAY! ----
Writing quick id xml, please wait... Done!
Writing 1681 (1 new) resources to cache, please wait... Done!

---- And here are some neat stats :) ----
Total completion time: 00:00:11

Average search match: 100%
Average entry completeness: 100%

Total number of games: 1
Successfully processed games: 1
Skipped games: 0 (Filenames saved to '/home/USER/.skyscraper/skipped-arcade-screenscraper.txt')

pi@retropie:~ $ 

I notice I actually see output from the videoconvert script, which seems to be missing entirely from the first one, with the error.

[muldjord]

@sleve_mcdichael

Dammit, I was afraid of that. I had to convert the processes due to deprecated functions in Qt (the framework I use) that run those commands and tested some of them, but not all. And that's probably why this doesn't work. Don't know why it wouldn't save the data to the cache though, I haven't touched that code.

I'll have to look into it. Thanks.

[muldjord]

@sleve_mcdichael said in Skyscraper two ways: games beginning with "The", and config.ini settings not working:

I see where it asks if I might be out of disk space. I am not; I have 20G available:

I can't reproduce this error. Did you perhaps accidentally run Skyscraper as root at some point (with sudo Skyscraper ...)? In that case some of the cache files might be owned by root and Skyscraper can't overwrite them. If that is the case you need to fix permissions of the /home/pi/.skyscraper folder for it to work again (owner of all files should be pi:pi).

[muldjord]

@muldjord 3.6.9 released. Video conversion should be fixed. I tested it with quite an elaborate command line and it worked. Also fixed --flags unpack which also broke with 3.6.8. This is also tested and now works again.

The cache saving issue I believe to be a problem on your end. As mentioned I haven't touched that code, and it works fine here.

[sleve_mcdichael]

@muldjord said in Skyscraper two ways: games beginning with "The", and config.ini settings not working:

The cache saving issue I believe to be a problem on your end. As mentioned I haven't touched that code, and it works fine here.

I believe the cache-saving error was closely related to the video-conversion error. The resource that couldn't be saved to the cache was the video that didn't get converted/copied. I only saw them together, and only in 3.6.8. Neither were present in 3.6.7, nor after today's 3.6.9 update, thank you!

pi@retropie:~ $ Skyscraper -p arcade -s screenscraper /home/pi/RetroPie/roms/arcade/1942.zip --verbosity 3
------------------------------------------
Running Skyscraper v3.6.9 by Lars Muldjord
------------------------------------------
Platform:           'arcade'
Scraping module:    'screenscraper'
Input folder:       '/home/pi/RetroPie/roms/arcade'
Game list folder:   '/home/pi/.emulationstation/gamelists/arcade'
Covers folder:      '/home/pi/.emulationstation/downloaded_media/arcade/covers'
Screenshots folder: '/home/pi/.emulationstation/downloaded_media/arcade/screenshots'
Wheels folder:      '/home/pi/.emulationstation/downloaded_media/arcade/wheels'
Marquees folder:    '/home/pi/.emulationstation/downloaded_media/arcade/marquees'
Videos folder:      '/home/pi/.emulationstation/downloaded_media/arcade/videos'
Cache folder:       'cache/arcade'

DID YOU KNOW: With the '--cache show' command line option, you can check the status of the resources that are cached for the platform you are scraping.

Fetching limits for user 'REDACTED', just a sec...
Setting threads to 2 as allowed for the supplied user credentials.

Reading and parsing quick id xml, please wait... Done!
Reading and parsing resource cache, please wait... Done!
Successfully parsed 1700 resources!

Resource cache stats for selected platform:
'esgamelist' module
  Titles       : 72
  Platforms    : 72
  Descriptions : 72
  Publishers   : 72
  Developers   : 71
  Ages         : 0
  Tags         : 72
  Ratings      : 71
  ReleaseDates : 72
  Covers       : 0
  Screenshots  : 0
  Wheels       : 0
  Marquees     : 0
  Videos       : 0
'import' module
  Titles       : 0
  Platforms    : 7
  Descriptions : 0
  Publishers   : 0
  Developers   : 0
  Ages         : 0
  Tags         : 0
  Ratings      : 0
  ReleaseDates : 0
  Covers       : 3
  Screenshots  : 3
  Wheels       : 3
  Marquees     : 2
  Videos       : 7
'screenscraper' module
  Titles       : 73
  Platforms    : 73
  Descriptions : 73
  Publishers   : 73
  Developers   : 72
  Ages         : 11
  Tags         : 73
  Ratings      : 72
  ReleaseDates : 73
  Covers       : 72
  Screenshots  : 73
  Wheels       : 73
  Marquees     : 73
  Videos       : 72

Looking for optional 'priorities.xml' file in cache folder... Found!
Priorities loaded successfully!

Adding files from subdir: '/home/pi/RetroPie/roms/arcade/fbneo'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/memcard'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/nvram'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/ctrlr'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/hi'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/diff'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/comment'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/ini'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/input'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/image'
Adding files from subdir: '/home/pi/RetroPie/roms/arcade/mame2010/cfg'

Trying to parse and load existing game list metadata... Success!

Starting scraping run on 1 files using 2 threads.
Sit back, relax and let me do the work! :)

cover: '2149208' > '323270', choosing resize for optimal result!
marquee: '158970' > '107347', choosing resize for optimal result!
#1/1 (T1) Pass 1 ---- Game '1942' found! :) ----
Scraper:        screenscraper
From cache:     NO
Search match:   100 %
Compare title:  '1942'
Result title:   '1942' ()
Platform:       'Capcom Classics' ()
Release Date:   '1984-01-01' ()
Developer:      'Capcom' ()
Publisher:      'Capcom' ()
Players:        '2' ()
Ages:           '1+' ()
Tags:           'Shoot'em Up, Shoot'em up / Vertical' ()
Rating (0-1):   '0.7' ()
Cover:          YES ()
Screenshot:     YES ()
Wheel:          YES ()
Marquee:        YES ()
Video:          YES ()

Description: ()
'"1942" is a vertically-scrolling shoot-em-up set in the Pacific theatre during World War II, in which the goal is to reach Tokyo and destroy the entire Japanese air fleet. 

The player pilots a plane dubbed the "Super Ace" (its appearance is that of a Lockheed P-38 Lightning) and has to shoot down waves of enemy planes while avoiding incoming enemy fire. The 'Super Ace' can perform a limited number of rolls or 'loop-the-loops' to evade enemy planes and shots. 

During the game, waves of red enemy planes periodically appear. If the player manages to destroy a red wave, a power-up, in the form of a 'POW' symbol, will be dropped and can be picked up. The power-up will award either increased fire-power (doubling the player's guns from two to four), a smart bomb that destroys all on-screen enemy planes the instant it's collected, two wingmen planes that flank the Super Ace and increase fire power, or simply extra points.

At the end of each stage the 'Super Ace' lands on an aircraft carrier and bonus points are awarded based on player performance. "1942" differs from other games in that its levels are numbered in reverse order, so the game begins at stage 32 and ends at stage 1.'

Cache output:
Video conversion: 
%i: '/opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4'
%o: '/opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/tmpfile_769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4'
Running command: 'videoconvert.sh /opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4 /opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/tmpfile_769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4'
Converting video...
Source: /opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4
Dest: /opt/retropie/configs/all/skyscraper/cache/arcade/videos/screenscraper/tmpfile_769ccf601a52d89cec19168a8bf3df6351a9a0f5.mp4
Video is in correct 420p format. Copying as is...


Success!

Debug output:
Tried with: 'crc=F52D7D3C&md5=89625F073887A5915C3CE2E2E4DA8875&sha1=BF332AC22C7FDB5470AFF4172A9051555E6E1266&romnom=1942.zip&romtaille=106788'
Platform: arcade

#1/1, (1/0)
Elapsed time   : 00:00:11
Est. time left : 00:00:00

---- Resource gathering run completed! YAY! ----
Writing quick id xml, please wait... Done!
Writing 1700 (0 new) resources to cache, please wait... Done!

---- And here are some neat stats :) ----
Total completion time: 00:00:11

Average search match: 100%
Average entry completeness: 100%

Total number of games: 1
Successfully processed games: 1
Skipped games: 0 (Filenames saved to '/home/USER/.skyscraper/skipped-arcade-screenscraper.txt')

pi@retropie:~ $

No apparent errors. Thanks again!