*nix based VM (VMWare) as RetroPie play and test environment

Folly

reserved

Folly

reserved

Folly

Are you open into changing the topic name a bit ?
I would like to change it into :
*nix based VM (VMWare) as RetroPie play and test environment

I think the play part will definitely attract more people to use VM's.
I certainly do play games on my VM's when it comes to it ;-)

Using (VMWare) only in the name makes us a bit more flexibel in the discussion about what programs we use.
For example, I use VMware fusion on OSX. ( Can you add "Fusion" in your first post too ? )

Do you agree for these changes ?

Folly

@Ashpool said in *nix based VM (VMWare Player) as retropie test environment:

@Folly So here we go.... I am not really sure how to summarize/condensate in the reserved posts (vmware general/ubu-based/debian(pure)-based/???

I think it would be nice to summarize in the reserved posts on how to install a specific OS and how to overcome the issues that are faced.
Basically that would make these posts very interesting as a tutorial for each specific OS.

Let's see what will go on in the discussion.

For vmware in general it would be recommend 8GB minimum (at least for compiling (current) mame(/mess?)).

I did several tests on different VM's and compiling will work with 8GB depending a bit on how much cpu cores you use.
4 cpu cores is probably the maximum with 8GB minimum if computer allows these specs.

If not :
Mame standalone will be a bit more forgiving as it will add 8192 MB swap when compiling on a 64 bit OS.
However lr-mess/lr-mame will add only a swap of 4096 MB when compiling on a 64 bit OS
So when using 5632 MB and 3 cpu cores compiling lr-mess/lr-mame froze.
Compiling mame standalone compiles fine with using 5632 MB and 3 cpu cores.

So if you have a computer with 8 GB it's recommended to use a minimum 5632 MB and 3 cpu cores.
For lr-mess/lr-mame to compile you need to increase the swap space in the module-scripts to 8192 in :

~/RetroPie-Setup/scriptmodules/libretrocores/lr-mame
~/RetroPie-Setup/scriptmodules/libretrocores/lr-mess

( requirements is probably something to add in the fist post )

Edit :
Changed 5192 into 5632

Ashpool

@Folly said in *nix based VM (VMWare Player) as retropie test environment:

I think the play part will definitely attract more people to use VM's.
Using (VMWare) only in the name makes us a bit more flexibel in the discussion about what programs we use.

agreed on both

@Folly said in *nix based VM (VMWare Player) as retropie test environment:

So if you have a computer with 8 GB it's recommended to use a minimum 5192 MB and 3 cpu cores.
...
( requirements is probably something to add in the fist post )

Assuming the Host has that many cores to start with ;] My main work-pc is still an Dualcore AMDX2 (3.00GHz) with 12GB (VMWare 16.2.5 installed), whereas my main gaming machine is an Octacore Ryzen7 @3.2GHz with 16GB (VMWare 17.0.2 *).
Yeah, requirements for vmware in the 1st post sounds good (will do that later), though I am inclined to use the 1st reserved post for vmware/(open-)vm-ware-tools topics.

Btw. so far I haven't thought about it and used the default settings for "display", so no accelerated 3D - If you're using it, would you recommend it to be used and if yes, what should the shared memory setting be?

Edit: @Folly as wednesday is our established pub meeting day since '91, i leave it for that so far ;>

Ashpool

@Folly said in *nix based VM (VMWare) as RetroPie play and test environment:

Changed 5192 into 5632

Btw. is that one vmwares recommended max. on 8 GB Host machines, or why 5.5 GB (instead of 5 or 6 - so far I've always prefered multiples of 1024 (out of habbit and thinking in terms of memory modules)?

Edit: And I am also unsure of the difference needed to run emulator-cores vs. compiling them. None of my (host-)systems available for testing this has less than 8 GB available (ok just to add: Some of those systems aren't able to run VMWare Player Version > 14, and both my *nix machines are only on 8 GB and are using an onboard GPU's (so, 8 GB - shared memory with GPU)).
Maybe it would be possible/use-case-scenario-to-be-considered (play case) to setup/compile cores on a machine that is capable to do it and later on to play that VM on another host with edited/downgraded ~memory&|core limits for the VM (.?.).

Besides, as you are using Fusion on a Mac, and I am just on Windows/*buntu Systems (though I have yet to see if I can run any recent VMWare Player after 14 on any of my *nix System), maybe Mac-specific experiences/tips/issues would better be placed in one of your reserved posts, though I don't have any idea how to exactly split guest-system related topics in regards to Host OS.

'nother thought I had and not that I am aware of any: Does anyone know if there are emulators utilizing AMD-V/Intel VT-X (maybe not gaming consoles, more likely computer systems ?) and would therefore benefit from enabling that for the VM?

Folly

@Ashpool said in *nix based VM (VMWare) as RetroPie play and test environment:

@Folly said in *nix based VM (VMWare) as RetroPie play and test environment:

Changed 5192 into 5632

Btw. is that one vmwares recommended max. on 8 GB Host machines, or why 5.5 GB (instead of 5 or 6 - so far I've always prefered multiples of 1024 (out of habbit and thinking in terms of memory modules)?

I am using it on my Apple mac mini m1 which only has 8 GB of memory.
5632 is the maximum what VMWare recommends to use.

Edit: And I am also unsure of the difference needed to run emulator-cores vs. compiling them.

The difference is that almost all emulators just use 1 core to emulate stuff.
Only thing is that if you have a multicore cpu then other tasks can go to a secondary core, etc so the load is less on the core that is used for emulation.

Only a few emulators are trying to use multiple cores.
For what I read it's very difficult to use multiple cores with emulation because everything has to be in sync to get a fluid emulation experience.
So syncing is the problem here.

Whereas compiling can be done over multiple cores more easily because syncing isn't important.
Basically, if a program needs compiling and it has, for example, 100 files then the workload can be slit up over the cores.
So with 4 cores every core can compile 25 files which can be done simultaneously so it is eventually 4 times quicker.
The disadvantage is that compiling needs memory so if you have 4 cores compiling the you will get the picture.
4 cores have to use the same memory so basically 4 cores need 4 times the memory than with 1 core.

So if you don't have enough memory in you Host or VM than you can reduce the cores in the VM configuration so less memory is consumed.

None of my (host-)systems available for testing this has less than 8 GB available (ok just to add: Some of those systems aren't able to run VMWare Player Version > 14, and both my *nix machines are only on 8 GB and are using an onboard GPU's (so, 8 GB - shared memory with GPU)).
Maybe it would be possible/use-case-scenario-to-be-considered (play case) to setup/compile cores on a machine that is capable to do it and later on to play that VM on another host with edited/downgraded ~memory&|core limits for the VM (.?.).

Indeed, basically it would be possible.
However when compiling stuff on one specific cpu will not always work on an other specific cpu.
Most programs, when compiling, detect the features of the cpu and use the best available.
For example the Intel I7 is better than an I5 or I3.
Compiling a program on an I7 transferring it to a I5 or I3 and it will not work because it misses specific specs.
The other way around can sometimes be possible because then you compile your program with less specs running it on a better cpu with backwards-compatible specs.

A solution can be cross-compiling a program on a CPU with better specs compiling on that CPU for a CPU with lesser specs.
However cross-compiling is very difficult and you have to have quite some knowledge to accomplish.
I have done this once in a hackish way, compiling mame for an older 32 bit x86 cpu pc.
Compiling it in a x86-32bit VM with a changed module-script.
With this solution I was able to compile mame in 1:30 hours whereas it would take more that 24 hours on that older x86 32 bit PC.

Besides, as you are using Fusion on a Mac, and I am just on Windows/*buntu Systems (though I have yet to see if I can run any recent VMWare Player after 14 on any of my *nix System), maybe Mac-specific experiences/tips/issues would better be placed in one of your reserved posts, though I don't have any idea how to exactly split guest-system related topics in regards to Host OS.

Good idea,
I can use my preserved ones for Fusion specific stuff.
If needed, you can just link to these preserved posts in one of you posts.

'nother thought I had and not that I am aware of any: Does anyone know if there are emulators utilizing AMD-V/Intel VT-X (maybe not gaming consoles, more likely computer systems ?) and would therefore benefit from enabling that for the VM?

Processors with AMD-V/Intel VT-X can be used to improve the experience of running virtual machines.
If available and enabled in VMWare then the Virtual Machine uses hardware of the Host more directly and so the speed will drastically improve.
I don't think it can be enabled and used again within a Virtual Machine.

Hope everything makes sense in this post ;-)

Ashpool

@Folly said in *nix based VM (VMWare) as RetroPie play and test environment:

Processors with AMD-V/Intel VT-X can be used to improve the experience of running virtual machines.
If available and enabled in VMWare then the Virtual Machine uses hardware of the Host more directly and so the speed will drastically improve.
I don't think it can be enabled and used again within a Virtual Machine.

Ok, I allways thought, that as hardware-assisted-virtualization was a requirement for running 64bit guest systems vmware will always utilize it if available and enabling Virtualize Intel VT-x/EPT or AMD-V/RVI actually meant it to be virtualized within the guest (for nested solutions).
Just to be sure of that/verify it: I've just installed VMWare-Player 15 within a Windows7Pro(x64)-Guest (virtualize hardware-assisted-virtualization enabled) and then an XPx64 Guest within it, which went fine and the VM within the VM was usable. Then I've deactivated the virtualization for Intel VT-X/AMD-V for the Win7 VM ("Host"-"Guest") and when trying to start the XPx64 VM ("Guest"-"Guest") again, it failed with "Error while powering on: This host does not support AMD-V …".

Hope everything makes sense in this post ;-)

It does! And I feel lucky now, that for my past usage of windows guest systems, that system specific compilation wasn't a problem when moving a VM between my various Intel/AMD Based Host Systems over all those years (Some of my VMs I kept age back to VMWare Player 12 times (older ones simply became obsolete or where migrated to VMP12 by mounting and cloning the old "HDD".vmdk to a new VM.vmdk (don't remember what they where based upon, propably +-~ 6 or 7 ones [1]))) and "installed" programms always utilize what they find and are cabable of utilizing.

1: (P.S.:) This ain't in violation of my answer/comment in the "module-script-thread", it was just that trying out whether virtual-box may had been an alternative to vmware (and haven't used VMs for some while in between):

I've tried virtualbox in pre-Win7 times, but since vmware(-player) offered native support for xpmode vms, i've switched to that player and well, stayed with it so far (though for legacy systems as win9x, i would prefer QEMU/PCem/DOSBox-x))

Folly

@Ashpool said in *nix based VM (VMWare) as RetroPie play and test environment:

and then an XPx64 Guest within it, which went fine and the VM within the VM was usable. Then I've deactivated the virtualization for Intel VT-X/AMD-V for the Win7 VM ("Host"-"Guest") and when trying to start the XPx64 VM ("Guest"-"Guest") again, it failed with "Error while powering on: This host does not support AMD-V …".

Interesting, I never tried it.
So you proved you can still utilize Intel VT-x/EPT or AMD-V/RVI running a "VM" within a "VM".

that system specific compilation wasn't a problem when moving a VM between my various Intel/AMD Based Host Systems

Aha, indeed you are lucky.
But than again it could be rationally explained by the fact that compilation took place on older cpu's.
Running the software on cpu's from the same period or on newer cpu's can work as the cpu's used can have the backwards compatible specs.

Virtual box

It has also been a very long time since I used it.
Not sure how it is right now.

Ashpool

Ok, I've tried to gather some data:
I installed Debian 11 with just one Desktop Environment of choice & Ubuntu flavoured DE Installs with the basic setup of 30GB virtual HDD (non-split)/8GB/4 Cores/3d Acceleration with max. 4GB shared and after (manually installing open-vm-tools(-desktop) where needed) doing a sudo apt update && sudo apt upgrade (really did nothing on the deb installs, just on the *ubu ones), setting the screen to 1280x960 and after restarting the VM, calling 'top' from within a terminal and then having sampled after ~5 min idle time the memory usage and also, after shutting down the vm, the size of its .VMDK (HDD) file.

System             | Idle Mem | .VMDK size
-------------------+----------+------------
DEB11-XFCE         |  ~435 MB |    4.62 GB
DEB11-KDE (PSB)    |  ~640 MB |    6.7  GB
DEB11-Cinnamon     |  ~770 MB |    6.82 GB
DEB11-LXQT         |  ~440 MB |    5.23 GB
DEB11-Mate         |  ~430 MB |    5.15 GB
-------------------+----------+------------
Raspian OS (32bit) |  ~320 MB |   11.1  GB
-------------------+----------+------------
DEB12-KDE (PSB)    |  ~820 MB |   10.5  GB
-------------------+----------+------------
Kubu LTS (PSB)     | ~1000 MB |   15.7  GB
Xubu LTS           |  ~890 MB |   12.66 GB

KDE really proved problematic (ok, for this 'venture it ain't the DE of choice, but it is my preference for real installs), for the Version(s) of Plasma used in (around) Debian 11 , there is a known issue (which won't be resolved for those OS, and the workaround available ain't worth to be considered in our context).
...
And Kubuntu LTS was another mess too, its version of plasma don't have the will not resize-bug, but with VMs Display: 3D Hardware acceleration enabled set it will blank whence opening settings and even without it it has some glitches that are voiding the use of KDE as a DE in a VM Guest OS.

Just recently Debian 12 "Bookworm" was released, but as VMWare aren't aware of it by now, it ain't an option for now (open-vm-tools are unavailable, it has to be installed as deb11x64 and there may be more hazards to be encountered by doing so - experimental, nothing to discuss atm).

For some Ubuntu Installs (e.g. not Lubuntu) VMWare offers "easy install" - and I am not sure if that could be recommended, it will auto install open-vm-tools (what it will also do on recognized OS as Debian11 where it ain't using easy install)... but from my experience so far, in cases where it will use "easy install" it is IMHO better to opt for "I will install the OS later" during setting up the VM and later install open-vm-tools manually (and I still don't understand why they are still offering their vmware-tools.iso disc image to install the tools on *nix guests, if vmware itself is advising the use of open-vm-tools(-desktop) :insane:.

P.S.: @Folly I haven't forgotten/abandoned this endeavour so far ;) But as I had bought the DoW master collection at GoGs discounted 10€ launch sale... well, I distracted (and still am distracting) myself once again in serving the God-Emperor ;] But I think that next Midweek+ I may have some skeletal guide entries within my reserved posts (fingers crossed). And from above Data, I am inclined to go primary with pure Debian XFCE Installs as a base (though I haven't tried installing RetroPie on any of those for now).

@mitu If I am refering to external websites, I guess linking to a wayback-machine archive is frowned upon in this forum (because of being identical/so close to....) .?.

Folly

@Ashpool said in *nix based VM (VMWare) as RetroPie play and test environment:

sudo apt update && sudo apt upgrade (really did nothing on the deb installs, just on the *ubu ones)

sudo in Debian OSes only works if you add your user name (pi for example) to the /etc/sudoers file.

# User privilege specification
root    ALL=(ALL:ALL) ALL
pi    ALL=(ALL:ALL) ALL

Looking at the /etc/sudoers file on raspberry pi OS it also could be like this :

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

You can edit the file like this :

pi@raspberrypi:~ $ su
Password:  (enter your password)
root@raspberrypi:/home/pi# nano /etc/sudoers

Now you can use 'sudo apt .......`.

Will have a look at the other stuff later, when I can.

Ashpool

@Folly said in *nix based VM (VMWare) as RetroPie play and test environment:

sudo in Debian OSes only works if you add your user name (pi for example) to the /etc/sudoers file.

Sorry, wasn't clear on that (will end up in the 1st. setup guide) -> it worked, but the system was up-2-date after the install (which it ain't for the various ubuntu ones I tried (they had lots of stuff upgradeable)). And as long as you are not providing an admin password during debians setup dialogue, the created user will be a sudoer ;) Pretty convenient and I don't think that we are in need of any hardened security settings within a VM of this context. But thanks for the reminder, I guess I would have forgotten to add that info (which of course should be in here) otherwise!

Edit: @add atm I trying to figure out what went wrong on my try with installing just system-tools without a DE and afterwards doing that from the console (for a manual xfce install compared to it being installed via the debian setup, that brought the initial vmdk size down from around 5 to just 3 GB, so it may be worth it. But whence trying to do it again with the lxde-core package, the vm had no sound and I had to set sound.VirtualDevice to hdaudio, but that really shouldn't be needed as debian has drivers for the creative AudioPCI which seem to be the default VMWare emulates for the other installs (and only adds hdaudio in vmx files for windows7 or later) (though I hadn't tried to add sound.VirtualDevice = "es1371" to that lxde vmx (and the VM where I installed lxde with the regular setup worked well without the extra vmx line and used the creative driver).

Folly

@Ashpool

Sorry that I a not around that much.
Tried to make images and sort these on how to install Debian in VMware.
Want to make an animated gif of it.
However, everything seems to take an awful lot of time (which I don't have).
Hope I will manage in the near future, just to let you know ;-)
Otherwise you know why.

Ashpool

@Folly said in *nix based VM (VMWare) as RetroPie play and test environment:

Hope I will manage in the near future, just to let you know ;-)
Otherwise you know why.

No Problem at all! Main reason I haven't posted anything in recent times was: at first I had once again underestimated the quirks and hazards encountered in wording/drafting/layouting and then (last week+) spend some time away from home again, and whilst I had gathered all data I needed to work on on an external HDD and thought to prepare some text, the bloody murphy handicap (I had to live with) was: I had left my AC Adapter for my thinkpad at home :( (And since some years it has no working battery pack attached anymore) ... I hope that I may come up with some, at least rudimentary, text in regards to a minimalistic debian install [Edit: but this week without access to the data/mindmap/thoughts has now left me pretty much departed from it and I may need some time again to dig into the stuff again (as you hinted at: it is a PITA to setup/query/verify stuff by setups that each in itself takes hours/whole evenings) :( ].

Ashpool

@Folly BTW.: As I was just haggling again (for maybe 1-2 hours - every time I try again) with the Debian Website to find the download area for the old-stable branche, i have it finally bookmarked for now... but times they are a-changing, and so far I haven't found a suitable/recommandable/mentionable path that could be followed on the Debian Pages to reach a specific version/iso ... IMHO they are just a PITA, but sadly the more I experiment with Debian and (#)ubuntu installs, the more I am inclined to opt for a minimal DEB install without a DE (just system tools) and manual DE install afterwards (not only for VMs, but also for my regular *nix based Desktops) and so the more I am saddened that their website is such a mess to navigate (and even if DEB 12 may be installable in VMWare *, as long as they are not officially supporting it as a guest, 11 will remain my target for now). ... Sorry, just a short rant I had to make now - navigating the debian pages ain't really no fun at all (unless there is some small-print/link I always overlook)!
Edit: In fact, I only came up with the link above via my browsers history and today I really wasn't able to reach it by a regular access of/from debians main website :(

mitu

@Ashpool said in *nix based VM (VMWare) as RetroPie play and test environment:

so far I haven't found a suitable/recommandable/mentionable path that could be followed on the Debian Pages to reach a specific version/iso ...

https://cdimage.debian.org/cdimage/archive/, linked from https://www.debian.org/CD/faq/#old.

Folly

@Ashpool said in *nix based VM (VMWare) as RetroPie play and test environment:

Sorry, just a short rant I had to make now - navigating the debian pages ain't really no fun at all (unless there is some small-print/link I always overlook

Indeed, have that same problem once in a while.

Good to hear that you find the minimal install the best way.
Seems indeed obvious if we want to use it mostly for RetroPie.

Ashpool

Ok_ I know I am focusing on diversions from the main topic, but I am also notorious known for my focus on data as such :/ So, no (what I had hoped for), guide to a minimal debian (bullseye) install within a vmware vm. but just those notes below...

So for the state of the previous day (230705), this is what I spend my time upon to gather some data for the available Desktop Environments (may sound worse then it actually is: for the debian installs, the base VM without installing a DE was a good copy&paste base towards later DE installs).

Nevertheless... here we go with current data samples, if there are discrepancies towards my previous table , consider it to be due to updates made available since that time!

Data gathered on: 20230705
VM-Settings:
30 GB HDD
8192 GB RAM
4 Cores, Virtualize Vt-x/* enabled
Network: NAT
GFX: Accelerate 3D, 4 GB shared

Just to be sure: right after each install sudo apt update and sudo apt upgrade (if update mentioned the need) where performed.

If not done by the install, open-vm-tools(-desktop) are installed via sudo apt install afterwards.
Debian: on xfce4 minimal installs xfce4-terminal was also apt installed next to the DE.

Install Type: Minimal Install means:
UBU-based -> minimal option during install
DEB-based -> just standard system utilities, no desktop environment during initial setup
             Desktop Environment (and open-vm-tools-desktop) installed via CLI

Install Medias used:
(d1) debian-11.7.0-amd64-DVD-1iso      (CRC32: 0EE03455)
(d2) debian-11.7.0-amd64-netinst.iso   (CRC32: EB19C571)
(u1) ubuntu-22.04.2-desktop-amd64.iso  (CRC32: 1A29639F)
(u2) xubuntu-22.04.2-desktop-amd64.iso (CRC32: D8ABF8D5)
(u3) lubuntu-22.04.2-desktop-amd64     (CRC32: B6382DFC)

Idle Mem   -> memory usage read from top after +-~ 5min IDLE time (rounded up to multiple of 5)
Disk usage -> OS partition reported via File-Manager (max. space omitted, depends on the installers default partitioning scheme) [converted from GB to GiB by hand where needed]
.vmdk size -> VMWares .vmdk HDD image size (reported from Windows Explorer)

[keep in mind that this ain't a scientific report, nor a valid statistical sampling! So don't read those numbers by their value, but by their approximation & range they may represent]


System/Install    | Idle Mem | Disk usage |   .vmdk  | Install | Note
------------------+----------+------------+----------+---------+------
Ubuntu       (u1) |  900 MiB |  17.97 GiB | 12.5  GB | Minimal |
xUbuntu      (u2) |  530 MiB |  17.76 GiB | 12.4  GB | Minimal |
xUbuntu      (u2) |  610 MiB |  17.06 GiB | 12.8  GB | Full    |
lUbuntu      (u3) |  510 MiB |  19.4  GiB | 12.8  GB | Full    | *1, *2
DEB xfce     (d1) |  380 MiB |  25.0  GiB |  2.88 GB | Minimal |
DEB xfce     (d2) |  370 MiB |  24.86 GiB |  3.06 GB | Minimal |
DEB xfce     (d2) |  445 MiB |  23.36 GiB |  5.33 GB | Full    |
DEB Cinnamon (d2) |  670 MiB |  24.16 GiB |  3.95 GB | Minimal |
DEB LXDE     (d2) |  285 MiB |  24.3  GiB |  2.99 GB | Minimal |
DEB LXQt     (d2) |  390 MiB |  24.3  GiB |  3.66 GB | Minimal | *3
DEB Mate     (d2) |  365 MiB |  24.36 GiB |  3.74 GB | Minimal | *3

----
1: LUbuntu doesn't offer a minimal install option during its setup
2: LUbuntu ain't recognized by VMWare as an Ubuntu based Install (XUbuntu is!)
3: The DE install doesn't included a DisplayManager, lightdm was installed via cli.

Ashpool

@Ashpool (memo to self & @Folly FYI) some notes/details I've encountered during the manual setup of the Desktop Environments (Debian 11).

some are (pre-)populating their panels/menus with icons to applications which aren't installed by the minimal DE setup itself (e.g. webbrowser in xfce) [note: removing orphaned start menu entries].
even if the network itself works fine, cinnamon integrates the network-manager into the desktop but that one is only managing interfaces not setup in /etc/network/interfaces and therefore always showing not connected/no connection [which may irritate most users] (?options?: any way to remove the network-interface from cinnamon? comment out the interface in above config and let the network-manager handle the connection?)
some Desktop Environments aren't preinstalling a display manager (lxqt, mate). Stick to lightdm (as all other installs are using it), or is lxdm for lxqt a reasonable alternative?
At least configuring an autologin for lightdm is pretty easy and sticking to it would eventually avoid the need for different (2be documented) approaches towards that option.
weird issues with the soundcard/sound-device: on most(/all manual installed) DEs it is muted on 1st startup and with the exception of mate simply unmuting it and raising the volume solve the no sound issue. For mate, the install of the package mate-media was neccessary [edit: and raising pcm value slider in alsamixer afterwards]: Further research needed why installing alsa-tools wasn't sufficient.
Any benefits by using VMWares "shared folders" capability? IMHO SSH and RetroPies Samba shares are far more convinient ways of transfering files between host&guest. (VMWare Tools copy&pase somewhat the worst option as it is leaving most of the time its junk within the hosts temp folder(s).