RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Install and configure precompiled emulator binaries

    Scheduled Pinned Locked Moved
    Ideas and Development
    scriptmodule
    3
    14
    2.4k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      Menion
      last edited by

      Hi all
      It is becoming more and more complicated to get the compilation of recent emulators, at least mame, on the Rpi due to the memory limitation.
      Toggling swap may help, but still it endups in hours of compilation, and the scripts, as far as I understand are quite blind, forcing the user either to start again from a fresh source tree or issue the make command directly in the tmp/build/lr-**** directory.
      There is a way that I normally use to compile big source code targeting embedded system without going crazy to make cross-compilation to work: it is the emulation of the embedded system via chroot+qemu.
      It works wonderful, it is like you are on the embedded system but with full ram and a good percentage of the CPU power (reduced by the emulation but still way bigger than the embedded system) of the host computer.
      However it is clear that once you got the binary output (.so or executable) you need to adapt the scripts so they just do the installation and configuration part, basically the functions install_lr*** and configure_lr***
      Is there a fast way to install the emulators without having to tweak the scripts by empting the git clone and compile functions and use absolute path instead of variables such $md_build?
      Bye

      1 Reply Last reply Reply Quote 0
      • mituM
        mitu Global Moderator
        last edited by mitu

        The approach you're describing is partially used to assemble the binary builds and the images. However, I've found that raw compilation in the qemu based chroot being as slow as the real hardware and it's prone to crashing. The biggest benefit is the RAM usage, not using a swap file has a big advantage.

        You can run the separate build steps by using retropie_packages.sh and invoking the different steps directly:

        sudo ./retropie_packages <scriptmodule> sources # gets the sources
sudo ./retropie_packages <scriptmodule> build # builds the module
sudo ./retropie_packages <scriptmodule> install # installs the module
# etc.
        M 1 Reply Last reply Reply Quote 0
        • M
          Menion @mitu
          last edited by

          @mitu thank you so much for the hint on how invoke the steps from the build system.
          So you already tried with qemu? I used it heavily in the past when I ran the server on an allwinner based SBC, where for instance was impossible to compile libtorrent, and worked perfectly at 300% of the target speed, using as host a virtual box Linux image on a i5 pc, so emulation in emulation.

          1 Reply Last reply Reply Quote 0
          • M
            Menion
            last edited by

            I have setup everything for qemu using a fresh, never booted, retropie image
            When trying to launch the setup script, I get:

            pi@Menionubuntu:~/RetroPie-Setup $ sudo ./retropie_setup.sh .
Unknown host QEMU_IFLA type: 47
Unknown host QEMU_IFLA type: 48
Unknown host QEMU_IFLA type: 43
Unknown host QEMU_IFLA type: 47
Unknown host QEMU_IFLA type: 48
Unknown host QEMU_IFLA type: 43
Unknown host QEMU_IFLA type: 47
Unknown host QEMU_IFLA type: 48
Unknown host QEMU_IFLA type: 43
Unknown host QEMU_IFLA type: 47
Unknown host QEMU_IFLA type: 48
Unknown host QEMU_IFLA type: 43
Unknown QEMU_IFLA_INFO_KIND vlan
Unknown host QEMU_IFLA type: 47
Unknown host QEMU_IFLA type: 48
Unknown host QEMU_IFLA type: 43
Unknown QEMU_IFLA_INFO_KIND tun
Unknown platform - please manually set the __platform variable to one of the following: armv7-mali generic-x11 imx6 odroid-c1 odroid-c2 odroid-xu rpi1 rpi2 rpi3 tinker x86

            The unknowns are just warnings, I have specified __platform in the system.sh script instead of calling os_platform function, but maybe there is a better place to put it?

            1 Reply Last reply Reply Quote 0
            • BuZzB
              BuZz administrators
              last edited by BuZz

              can ignore the warnings- must have set the platform flag in the wrong place - just put it after sudo - btw building binaries on a qemu chroot is built into retropie-setup including a cross compiler via distcc on the host machine. Note that this functionality was built for myself to easily automate building of binaries.

              on an Ubuntu PC: (18.04 for example)

              sudo ./retropie_packages.sh builder chroot_build module lr-mame

              will build binaries for rpi1/rpi2 on jessie and stretch in an emulated chroot. I am going to remove jessie from this though as I am discontinuing building binaries for Raspbian Jessie.

              sudo ./retropie_packages.sh crosscomp setup stretch

              will build and install a cross compiler to be used via distcc. The first command will use it if it's available. Also sudo ./retropie_packages.sh crosscomp setup jessie for the jessie cross compiler.

              with the emulated chroot using the cross compiler on the host, compilation can be very quick and requires no modifications to any of the retropie-setup modules.

              To help us help you - please make sure you read the sticky topics before posting - https://retropie.org.uk/forum/topic/3/read-this-first

              1 Reply Last reply Reply Quote 0
              • M
                Menion
                last edited by

                question, maybe stupid: how the module makefiles/cmake/whatever will pickup native cross compiler in chroot, instead of the emulated compiler? Will the crosscomp setup stretch rename the crosscompiler in gcc/g++ or?

                mituM 1 Reply Last reply Reply Quote 0
                • mituM
                  mitu Global Moderator @Menion
                  last edited by

                  @Menion The 'native' compiler is never called. distcc transfers the compilation to the host system, where a cross compiler runs. The system in the chroot calls the normal binaries (cc/gcc/c++g++), but those are actually symlinks to distcc (your $PATH includes the distcc symlinks before the actual compilers' binaries in /usr/lib/distcc).

                  1 Reply Last reply Reply Quote 0
                  • M
                    Menion
                    last edited by

                    so I need to install distcc in the host system, or in the chroot-ed environment? Also, the

                    sudo ./retropie_packages.sh crosscomp setup stretch

                    shall be called in chroot-ed environment on on the host from the /hostdir/retropie/home/pi/RetroPie-Setup/scriptmodules ?

                    BuZzB 1 Reply Last reply Reply Quote 0
                    • BuZzB
                      BuZz administrators @Menion
                      last edited by BuZz

                      @Menion No - you don't need to create a chrooted environment or run anything on it. It's all done from the host.

                      The command you list sets up distcc and the cross compiler on the host machine.

                      The builder command creates a chroot (actually two - one for jessie and one for stretch), and does all the building etc. Note you need to make sure RetroPie-Setup is installed in $HOME on the host machine.

                      I use this to build and prepare binaries. I used to have a lot of standalone scripts for doing this, but I decided to implement it into RetroPie-Setup.

                      For more info look through the scriptmodules/admin/builder.sh / scriptmodules/admin/image.sh and scriptmodules/admin/crosscomp.sh files.

                      This code is also used for creating images for distribution (in this case image.sh is used but that's also used for the chroot creation for the builder.sh)

                      To help us help you - please make sure you read the sticky topics before posting - https://retropie.org.uk/forum/topic/3/read-this-first

                      1 Reply Last reply Reply Quote 0
                      • M
                        Menion
                        last edited by

                        Hi the scripts stopped because missing distcc (it checks for missing packages at the beginning but for some reason it does not check distcc itself) :

                        /1847]Running action 'switch_distcc' for 'crosscomp' : Create am arm cross compiler env - based on examples from http://preshing.com/20141119/how-to-build-a-gcc-cross-compiler
= = = = = = = = = = = = = = = = = = = = =                                    '/opt/retropie/admin/crosscomp/bin/cc' -> '/opt/retropie/admin/crosscomp/stretch/bin/arm-linux-gnueabihf-gcc'
'/opt/retropie/admin/crosscomp/bin/gcc' -> '/opt/retropie/admin/crosscomp/stretch/bin/arm-linux-gnueabihf-gcc'
'/opt/retropie/admin/crosscomp/bin/arm-linux-gnueabihf-gcc' -> '/opt/retropie/admin/crosscomp/stretch/bin/arm-linux-gnueabihf-gcc'
'/opt/retropie/admin/crosscomp/bin/c++' -> '/opt/retropie/admin/crosscomp/stretch/bin/arm-linux-gnueabihf-g++'                                            '/opt/retropie/admin/crosscomp/bin/g++' -> '/opt/retropie/admin/crosscomp/stretch/bin/arm-linux-gnueabihf-g++'                                            '/opt/retropie/admin/crosscomp/bin/arm-linux-gnueabihf-g++' -> '/opt/retropie/admin/crosscomp/stretch/bin/arm-linux-gnueabihf-g++'                        sed: can't read /etc/init.d/distcc: No such file or directory                Failed to restart distcc.service: Unit distcc.service not found.

                        Can I resume the job manually without having to start from scratch, invoking the remaining actions?

                        BuZzB 2 Replies Last reply Reply Quote 0
                        • BuZzB
                          BuZz administrators @Menion
                          last edited by

                          @Menion you need to manually run the depends for both modules. Sorry. I don't have time to go into more depth. You can manually install the dependencies also, but I don't recommend using this admin code if unfamiliar with debugging and perhaps having to adjust it.

                          To help us help you - please make sure you read the sticky topics before posting - https://retropie.org.uk/forum/topic/3/read-this-first

                          1 Reply Last reply Reply Quote 0
                          • BuZzB
                            BuZz administrators @Menion
                            last edited by BuZz

                            @Menion you will have to check the code in regards to continuing where it left off. Probably can just trigger the distcc switch function.

                            To help us help you - please make sure you read the sticky topics before posting - https://retropie.org.uk/forum/topic/3/read-this-first

                            1 Reply Last reply Reply Quote 0
                            • M
                              Menion
                              last edited by

                              No problem, I figured it out, it was really the last step, just add the compiled cross-compiler bin path to the init.d script of distcc. It is strange because there is a function that checks for distcc, maybe it is not called at all.
                              I don't want to bother you more, I don't pretend immediate answers don't worry. I am moving ahead now to the builder part. I have modified the script because I run an headless server, so there is no network manager and thus, no nmcli for detecting the DNS server.
                              I know that these scripts are not meant for end user, I just need some fast compiling environment for me if I need to make some experiments to give back to the community, like the fix for mame2016 I did (see my post in the General Discussion forum) without having to wait ages for compiling.

                              1 Reply Last reply Reply Quote 0
                              • M
                                Menion
                                last edited by

                                According to this process inspection, it worked :)

                                23595 ?        Ss     0:03 tmux
23596 pts/1    Ss     0:01  \_ -bash
24946 pts/1    S+     0:00      \_ sudo ./retropie_packages.sh builder chroot_build module lr-mame
24947 pts/1    S+     0:01          \_ bash ./retropie_packages.sh builder chroot_build module lr-mame
 6155 pts/1    Sl+    0:01              \_ /usr/bin/qemu-arm-static /usr/bin/sudo PATH=/usr/lib/distcc:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
 6157 pts/1    Sl+    0:10                  \_ /usr/bin/qemu-arm-static /bin/bash /home/pi/RetroPie-Setup/retropie_packages.sh builder module lr-mame
 7921 pts/1    Sl+    0:01                      \_ /usr/bin/qemu-arm-static /usr/bin/make OSD=retro RETRO=1 NOWERROR=1 OS=linux TARGETOS=linux CONFIG=libretro NO_USE_MIDI=1
 7984 pts/1    Sl+    0:00                          \_ /usr/bin/qemu-arm-static /usr/bin/make -R --no-print-directory -C 3rdparty/genie/build/gmake.linux -f genie.make
 8827 pts/1    Sl+    0:00                          |   \_ /usr/bin/qemu-arm-static /bin/sh -c gcc -O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard -pipe  -MMD -MP -MP -DND
 8833 pts/1    Sl+    0:00                          |   |   \_ /usr/bin/qemu-arm-static /usr/lib/distcc/gcc -O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard -pipe -MMD -MP
 8860 pts/1    Sl+    0:00                          |   \_ /usr/bin/qemu-arm-static /bin/sh -c gcc -O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard -pipe  -MMD -MP -MP -DND
 8863 pts/1    Sl+    0:00                          |       \_ /usr/bin/qemu-arm-static /usr/lib/distcc/gcc -O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard -pipe -MMD -MP
 8854 pts/1    Rl+    0:01                          \_ /usr/bin/qemu-arm-static /usr/bin/python scripts/build/msgfmt.py --output-file language/Vietnamese/strings.mo language
 8878 pts/1    Rl+    0:00                          \_ /usr/bin/qemu-arm-static /usr/bin/python scripts/build/msgfmt.py --output-file language/Burmese/strings.mo language/Bu
 2681 ?        SNs    0:00 /usr/bin/distccd --pid-file=/var/run/distccd.pid --log-file=/var/log/distccd.log --daemon --allow 127.0.0.1 --listen 127.0.0.1 --nice 10
 2682 ?        SN     0:00  \_ /usr/bin/distccd --pid-file=/var/run/distccd.pid --log-file=/var/log/distccd.log --daemon --allow 127.0.0.1 --listen 127.0.0.1 --nice 10
 2690 ?        SN     0:00  \_ /usr/bin/distccd --pid-file=/var/run/distccd.pid --log-file=/var/log/distccd.log --daemon --allow 127.0.0.1 --listen 127.0.0.1 --nice 10
 2691 ?        SN     0:00  \_ /usr/bin/distccd --pid-file=/var/run/distccd.pid --log-file=/var/log/distccd.log --daemon --allow 127.0.0.1 --listen 127.0.0.1 --nice 10
 2692 ?        SN     0:00  \_ /usr/bin/distccd --pid-file=/var/run/distccd.pid --log-file=/var/log/distccd.log --daemon --allow 127.0.0.1 --listen 127.0.0.1 --nice 10
 8848 ?        SN     0:00  |   \_ gcc -O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard -pipe -Wall -Wextra -Os -o /tmp/distccd_20ddb5c9.o -c /tmp/distccd_230bb5c9.i
 8849 ?        RN     0:02  |       \_ /opt/retropie/admin/crosscomp/stretch/libexec/gcc/arm-linux-gnueabihf/6.4.0/cc1 -fpreprocessed /tmp/distccd_230bb5c9.i -quiet -dumpbas
 8850 ?        SN     0:00  |       \_ /opt/retropie/admin/crosscomp/stretch/lib/gcc/arm-linux-gnueabihf/6.4.0/../../../../arm-linux-gnueabihf/bin/as -mcpu=arm1176jzf-s -mfl
 2693 ?        SN     0:00  \_ /usr/bin/distccd --pid-file=/var/run/distccd.pid --log-file=/var/log/distccd.log --daemon --allow 127.0.0.1 --listen 127.0.0.1 --nice 10
 8875 ?        SN     0:00  |   \_ gcc -O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard -pipe -Wall -Wextra -Os -o /tmp/distccd_befeb5cc.o -c /tmp/distccd_a1dab5cc.i
 8876 ?        RN     0:00  |       \_ /opt/retropie/admin/crosscomp/stretch/libexec/gcc/arm-linux-gnueabihf/6.4.0/cc1 -fpreprocessed /tmp/distccd_a1dab5cc.i -quiet -dumpbas
 8877 ?        SN     0:00  |       \_ /opt/retropie/admin/crosscomp/stretch/lib/gcc/arm-linux-gnueabihf/6.4.0/../../../../arm-linux-gnueabihf/bin/as -mcpu=arm1176jzf-s -mfl
 2694 ?        SN     0:00  \_ /usr/bin/distccd --pid-file=/var/run/distccd.pid --log-file=/var/log/distccd.log --daemon --allow 127.0.0.1 --listen 127.0.0.1 --nice 10
menion@Menionubuntu:/etc/default$

                                I have heard about distcc but I have never looked into it in the details, I have to study it :)

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post

                                Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.