Unreal Engine 4: use Swarm to distribute Lighting build on Local Network

This is a quick tutorial on how to install and configure Unreal Engine 4’s Swarm Agent and Coordinator to distribute Lighting build (that is, UE4’s Lightmass’s ray-tracing precomputations) across a Local Network of 64 bits Windows 10 machines. There are already a few good ressources on the subject, but I had to read through a bunch of them to pick all the needed information.

Existing online documentation on Swarm

  1. UDK’s documentation Unreal Swarm – Massive Application Distribution for Unreal Engine 3 is actually quite good, though not entirely up-to-date
  2. UE4’s documentation Lightmass Global Illumination
    is more high level, and lacking a section on Swarm Settings
  3. Brian Goodsell’s Tutorial: Setting Up Swarm for Multiple Machines is very good but slightly out-dated, not saying how to install the Swarm tools
  4. Niberspace’s Is UE4 Distributed Rendering Worth It? gives information on the performances you can expect of it
  5. Baking with Swarm gives a lot of in-details technical information (but far too much)

What is Swarm

The Swarm Agent is the process launched in the background (daemon) when you hit “Build Lighting” on your Unreal Engine Editor. That is, it is the application that bake your  lighting in the UE Lightmass. By default it uses your computer’s CPU cores to do so.

The Swarm Coordinator is a sibling process that you can launch manually to distribute your Lighting build on multiple Agents across the network.

Installation of Swarm Agent and/or Coordinator

Installation on a Windows 7/8/10 64 bits up-to-date is straightforward:

  1. You need the .NET 4.0 framework, which comes by default with any recent version of Windows.
  2. You should also deactivate the power saving/standby mode on every PC, else it will disable the Swarm Agent on the sleeping machines.

To install, you can either:

  1. Use existing installation of Unreal Engine on each computers of the network. All versions are cross-compatible so you can distribute your Lightmass work to Swarm Agents of outdated UE4 installation without any problem.
  2. Or, on computers without UE4, just copy the content of the Engine/Binaries/DotNET folder. You don’t need all this files, and no subfolders, but the whole is only a few MB in size. Put this on a fast disk with more than 10 GB free space for Agent cache.
Content of the DotNET folder

You need to choose an always-on machine, like a small/old server to act as the Swarm Coordinator. If you don’t have one, you can select any other Agent machine to host the Coordinator, but you should ensure that it stays online all the time you need it! On my use-case, I’ve selected my laptop (named “ASUS-I7”) to be the Coordinator, so that I can use it when away.

Lastly, you should ask the system to auto-start the Swarm Agent (and/or Coordinator on the selected machine) when the user log-in. To do so, put a shortcut to the executable on the Windows Startup folder, which is located here:

C:\Users\x\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\

Swarm Agent Settings

The Coordinator does not really need any settings. You need to accept the Windows Firewall request to let it communicate freely, though!

The Swarm Agent needs to be told how to distribute it’s work. To do that, launch the Swarm Agent if it is not already running, and double-click on it’s small icon hidden on the tray bar:

Go to the “Settings” tab of its UI:

Swarm Agent Settings (after full configuration)

Here, the only important Settings are (in bold those you need to change):

  • AgentGroupName: Default (is fine)
  • AllowedRemoteAgentGroup: Default (to match, instead of DefaultDeployed)
  • AllowedRemoteAgentNames: * (to match everything, instead of RENDER*)
  • CoordinatorRemotingHost: ASUS-I7 (name or IP of the Coordinator computer)

Whenever you set the last one, the Coordinator should start to show the new Agent as soon as it connects across the network:

Usage

Next time you build your lighting on any computer correctly configured,you should get your work distributed across multiple Agents, assuming that:

  1. the Coordinator is up,
  2. other Agents are running, connected and Available (not Busy, Working, nor Dead),
  3. and your workload is big enough to distribute (enough mesh to bake lighting onto)

At this stage, a look at the local Agent will tell show you the following Status:

Swarm Agent Status while Working

A look at the Coordinator will tell you the Agent involved in the work:

Swarm Coordinator with Working Agents

I hope this can help, and show how all this is incredibly easy and worthy to setup!

Android NDK r10e and Standalone Toolchain on Ubuntu 14.04

This is an update on my previous post about Android NDK Standalone Toolchain on Ubuntu, so have a look to it for better understanding.

We will see basic setup of the SDK & NDK. We will generate a standalone toolchain, for use with Android 4.1 Jelly Bean (target 90% of active devices). I then uses CMake to generate Makefiles. I am doing all this on the latest Ubuntu LTS (14.04).
ubuntu-logo
Ubuntu 14.04 desktop 64-bit:
Ubuntu download page
After standard installation, we need Java for the Android SDK, and a C/C++ development environment.
sudo apt-get install openjdk-7-jre ant
sudo apt-get install build-essential cmake

Android SDK (the Java part, to bundle C++ in an APK, see Native Activity):
Android SDK (r24.3.4) download page
SDK Tools Only, Linux
32 & 64-bit android-sdk_r24.3.4-linux.tgz
Uncompress under ~/android-sdk-linux/
Execute:
~/android-sdk-linux/tools/android
Add all packages from Android 4.1 – API Level16 (See Adding SDK Packages)

Android NDK (the C/C++ part) (Have a look at my standalone toolchain post):
Android NDK guide and API reference
Android NDK download page
Platform Linux 64-bit (x86) android-ndk-r10e-linux-x86_64.bin
chmod u+x android-ndk-r10e-linux-x86_64.bin
Execute into your $HOME to uncompress the NDK under ~/android-ndk-r10e/

Extract a standalone toolchain for API level 16 (with default GCC 4.8):
~/android-ndk-r10e/build/tools/make-standalone-toolchain.sh --arch)arm --platform=android-16 --install-dir=$HOME/android-standalone-toolchain-api16

Or package it with the following command:
~/android-ndk-r10e/build/tools/make-standalone-toolchain.sh --platform=android-16

Add SDK & Standalone toolchain to your PATH, with your ~/.bashrc file:
$ echo 'PATH=$PATH:$HOME/android-sdk-linux/tools' >> ~/.bashrc
$ echo 'PATH=$PATH:$HOME/android-sdk-linux/platform-tools' >> ~/.bashrc
$ echo 'PATH=$PATH:$HOME/android-standalone-toolchain-api16/bin' >> ~/.bashrc

$ echo ‘export ANDROID_STANDALONE_TOOLCHAIN=$HOME/android-standalone-toolchain-api16’ >> ~/.bashrc

Android CMake (Have a look at my previous post on CMake for Android):
Download the android-cmake now maintained on Github

Note: This is a work in progress, as I will need to refresh all my knowledge to switch to Android Studio in the following months.

CMake for Android

Last time, we’ve seen how to install and test the standalone toolchain of the NDK to build a native shared library with a standard Makefile & make command under Linux.

CMake LogoBut nowadays, many open source libraries and projects does not directly come with a Makefile, instead using a build system to automate its generation. CMake is becoming more and more used for those kind of jobs, so being able to use it for Android is really interesting. And easy!

A CMake configuration file for Android (android.toolchain.cmake) is provided by the android-cmake Google project hosted on Mercurial (see my previous post on Mercurial). Let’s clone its repository in the home directory with the following commands:
hg clone https://android-cmake.googlecode.com/hg/ android-cmake
Then, add to your .bashrc the ANDROID_NDK_TOOLCHAIN_ROOT and ANDTOOLCHAIN environment variables pointing to NDK and to the cmake config file. I also added a simple alias for later use:

PATH=$PATH:$HOME/android-sdk-linux_x86/tools
PATH=$PATH:$HOME/android-sdk-linux_x86/platform-tools
PATH=$PATH:$HOME/android-ndk-r5b/standalone-toolchain-api5/bin/

ANDROID_NDK_TOOLCHAIN_ROOT=$HOME/android-ndk-r5b/standalone-toolchain-api5
ANDTOOLCHAIN=$HOME/android-cmake/toolchain/android.toolchain.cmake

alias cmake-android='cmake -DCMAKE_TOOLCHAIN_FILE=$ANDTOOLCHAIN'

Ok, let’s try to compile the “hello-cmake” sample provided by the Google project (see hello-cmake documentation):

cd ~/android-cmake/samples/hello-cmake/
mkdir build
cd build
export ANDROID_NDK_TOOLCHAIN_ROOT
cmake -DCMAKE_TOOLCHAIN_FILE=$ANDTOOLCHAIN ..

We can simplify the last command by using our alias, so next type we will use “cmake-android ..” instead.
At this stage, CMake has generated the Makefile for you. Use it to build the sample :
make
Here you are, the shared library has been build and is ready for integration in an Application. You can find it here:
~/android-cmake/samples/hello-cmake/libs/armeabi-v7a/libhello-cmake.so
Now we want to invoke this native library from a Java Application, using JNI. This is documented in the next provided sample, “hello-android-cmake“, which is the Java counterpart of the C application we’ve build.

Copy the above “libs” directory to ~/android-cmake/samples/hello-android-cmake. Then open Eclipse and start a “File->New…->Project…->Android Project”. Select “create project from existing source” and give it the ~/android-cmake/samples/hello-android-cmake path.

Build the application, and launch it on an emulator or an actual device. You should see an activity showing the multi-line text starting by “JNI — pi = 3.14159…” coming from our shared library !

Hello Android CMake

Here you are!

Standalone toolchain

Since the revision 5 of the Native Development Kit previously discussed, the NDK can be customized to obtain a standard GCC cross compiler toolchain. This means that you can use it with standard Makefile and make command, or with more sophisticated build tools like automake, CMake or Bakefile. No more Android.mk ant Application.mk proprietary format required ! This is great in many situation where you want to recompile a complex software with big libraries. Lets have a try !

For this experimental article, I’ll assume that you already have good knowledge of Android development, SDK, ADB and some good understanding of C/C++ compilation workflow (gcc, make and Makefile).

NDK Requirements :

  • Android SDK (yes, the Java part !) : I’ll use API-level 5 (Android 2.0) but the minimum is API-level 3 (Android 1.5): “~/android-sdk-linux_x86/”
  • Android NDK r5b (the C/C++ part): “~/android-ndk-r5b/”
  • GNU Make 3.81 or later (“make” package under Ubuntu or Debian)
  • GNU awk or equivalent (nawk says the documentation, I have mawk)

To make this work under Windows (XP or 7) you’ll need Cygwin 1.7 installed, but it’s not that simple (the standalone toolchain officially dos not support Cygwin…). For that reason I’ve decided to work on this entirely under Linux (latest Ubuntu in my case). I you want to try to make it work under Windows, you could exploit useful information from the following blog post http://www.pocketmagic.net/?p=1462 and from some other Internet sources.

So let’s start by reading the “/android-ndk-r5b/docs/STANDALONE-TOOLCHAIN.html“, and proceed directly with the instructions on section 3 “Invoking the compiler (the easy way)”. Under a Linux console, type:

~/android-ndk-r5b/build/tools/make-standalone-toolchain.sh --platform=android-5 --install-dir=$HOME/android-ndk-r5b/standalone-toolchain-api5

Then add this few lines to the end of your .bashrc file and restart a console (or type them directly on the console):

PATH=$PATH:$HOME/android-sdk-linux_x86/tools
PATH=$PATH:$HOME/android-sdk-linux_x86/platform-tools
PATH=$PATH:$HOME/android-ndk-r5b/standalone-toolchain-api5/bin/

Then, create a standard hello world “hello-ndk.c”:

#include <stdio.h>

int main(void)
{
    printf("Hello from NDK\n");
    return 0;
}

At this stage, compiling for Android works easily with the following command:

arm-linux-androideabi-gcc  hello-ndk.c -o hello-ndk

Before testing it, we would like to add a bit of android API in it. You can easily complete it with a minimal Android log (logcat for Eclipse) :

#include <stdio.h>
#include <android/log.h>

#define LOGI(...) ((void)__android_log_print(ANDROID_LOG_INFO, "hello-ndk", __VA_ARGS__))

int main(void)
{
    printf("Hello from NDK\n");
    LOGI("Hello from NDK");
    return 0;
}

For this, you need to link explicitly with a the “liblog” library :

  • arm-linux-androideabi-gcc  hello-ndk.c -l log -o hello-ndk

Here is a Makefile to handle this with a simple “make” command (or other “make clean” and “make clean all” variants):

CC	= arm-linux-androideabi-gcc
CFLAGS	= -Wall -g
LDFLAGS	= -llog
SRC	=hello-ndk.c
OBJ	=$(SRC:.c=.o)
EXE	=hello-ndk

all: $(SRC) $(EXE)

$(EXE): $(OBJ)
	$(CC) -o $@ $^ $(LDFLAGS)

%.o: %.c
	$(CC) -o $@ -c $< $(CFLAGS)

clean:
	rm -f *.o $(EXE)

This create the standalone executable hello-ndk, so if your device is rooted, you can use adb to copy and execute “hello-ndk” on it, like this:

% adb shell
$ su
# mkdir /data/tmp
# chmod 777 /data/tmp
# exit
$ exit
% adb push hello-ndk /data/tmp
% adb shell
$ /data/tmp/hello-ndk

You should get a “Hello from NDK” on the command line, and the same message on the logcat window of Eclipse (in green, info level).

Hello-NDK-LogcatFor normal users that are not root (no “su” command), there is no way (oops, see edit bellow) of launching directly a native application under Android: you have to package it in a standard Android application .apk and use it from Java code. See http://gimite.net/en/index.php?Run%20native%20executable%20in%20Android%20App.

edit: look at the comment from pitypang, the path /data/local/tmp should work for an unrooted phone !

So we need to build instead a shared library, a “.so” file to be loaded by a Java application (through Java Native Interface, JNI). For this, we need to modify the Makefile like this :

CC	= arm-linux-androideabi-gcc
CFLAGS	= -Wall -g
LDFLAGS	= -llog -shared
SRC	=hello-ndk.c
OBJ	=$(SRC:.c=.o)
EXE	=libhello-ndk.so

With this you will get a libhello-ndk.so shared library. This will do nothing interesting by itself; we need to invoke the native function from Java, or to use the NativeActivity.

I will update this post later to make this works.

Edit:  look also at my post on CMake for Android