Unreal Engine 4: use Swarm to distribute Lighting build on Local Network

This is a quick tutorial on how to install and configure Unreal Engine 4’s Swarm Agent and Coordinator to distribute Lighting build (that is, UE4’s Lightmass’s ray-tracing precomputations) across a Local Network of 64 bits Windows 10 machines. There are already a few good ressources on the subject, but I had to read through a bunch of them to pick all the needed information.

Existing online documentation on Swarm

  1. Edit: There is now a brand new official documentation for Swarm: Swarm Overview!
  2. UDK’s documentation Unreal Swarm – Massive Application Distribution for Unreal Engine 3 is actually quite good, though not entirely up-to-date
  3. UE4’s documentation Lightmass Global Illumination
    is more high level, and lacking a section on Swarm Settings
  4. Brian Goodsell’s Tutorial: Setting Up Swarm for Multiple Machines is very good but slightly out-dated, not saying how to install the Swarm tools
  5. Niberspace’s Is UE4 Distributed Rendering Worth It? gives information on the performances you can expect of it
  6. Baking with Swarm gives a lot of in-details technical information (but far too much)

What is Swarm

The Swarm Agent is the process launched in the background (daemon) when you hit “Build Lighting” on your Unreal Engine Editor. That is, it is the application that bake your  lighting in the UE Lightmass. By default it uses your computer’s CPU cores to do so.

The Swarm Coordinator is a sibling process that you can launch manually to distribute your Lighting build on multiple Agents across the network.

Installation of Swarm Agent and/or Coordinator

Installation on a Windows 7/8/10 64 bits up-to-date is straightforward:

  1. You need the .NET 4.0 framework, which comes by default with any recent version of Windows.
  2. You should also deactivate the power saving/standby mode on every PC, else it will disable the Swarm Agent on the sleeping machines.

To install, you can either:

  1. Use existing installation of Unreal Engine on each computers of the network. All versions are cross-compatible so you can distribute your Lightmass work to Swarm Agents of outdated UE4 installation without any problem.
  2. Or, on computers without UE4, just copy the content of the Engine/Binaries/DotNET folder. You don’t need all this files, and no subfolders, but the whole is only a few MB in size. Put this on a fast disk with more than 10 GB free space for Agent cache.
Content of the DotNET folder

You need to choose an always-on machine, like a small/old server to act as the Swarm Coordinator. If you don’t have one, you can select any other Agent machine to host the Coordinator, but you should ensure that it stays online all the time you need it! On my use-case, I’ve selected my laptop (named “ASUS-I7”) to be the Coordinator, so that I can use it when away.

Lastly, you should ask the system to auto-start the Swarm Agent (and/or Coordinator on the selected machine) when the user log-in. To do so, put a shortcut to the executable on the Windows Startup folder, which is located here:

C:\Users\x\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\

Swarm Agent Settings

The Coordinator does not really need any settings. You need to accept the Windows Firewall request to let it communicate freely, though!

The Swarm Agent needs to be told how to distribute it’s work. To do that, launch the Swarm Agent if it is not already running, and double-click on it’s small icon hidden on the tray bar:

Go to the “Settings” tab of its UI:

Swarm Agent Settings (after full configuration)

Here, the only important Settings are (in bold those you need to change):

  • AgentGroupName: Default (is fine)
  • AllowedRemoteAgentGroup: Default (to match, instead of DefaultDeployed)
  • AllowedRemoteAgentNames: * (to match everything, instead of RENDER*)
  • CoordinatorRemotingHost: ASUS-I7 (name or IP of the Coordinator computer)

Whenever you set the last one, the Coordinator should start to show the new Agent as soon as it connects across the network:


Next time you build your lighting on any computer correctly configured,you should get your work distributed across multiple Agents, assuming that:

  1. the Coordinator is up,
  2. other Agents are running, connected and Available (not Busy, Working, nor Dead),
  3. and your workload is big enough to distribute (enough mesh to bake lighting onto)

At this stage, a look at the local Agent will tell show you the following Status:

Swarm Agent Status while Working

A look at the Coordinator will tell you the Agent involved in the work:

Swarm Coordinator with Working Agents

I hope this can help, and show how all this is incredibly easy and worthy to setup!

Why I switched from Mercurial to Git

After a few month using Mercurial (see my blog post from a year ago Mercurial to replace SVN) I switched to Git.

I’ve always been tempted to use Git, since this Tech Talk by Linus Torvalds in May 3, 2007.
[iframe src=”http://www.youtube.com/embed/4XpnKHJAok8″ width=”420″ height=”315″]
I even tried it a few times, but Mercurial commands seem more familiar to a Subversion user, and at this time, Mercurial GUI was more user friendly.

But then half a year ago, I wanted to convert to Mercurial our SVN code base at work, which is something like 15 man-years of C++.
I used the Mercurial ConvertExtension. The process took hours, used a few gigabytes on disk, but failed after using too much memory.

This failure was a clear signal: I started with the Git – SVN Crash Course.

I used the git-svn clone command, which also took a few hours to run, but succeeded after a shorter time, only using 600 megabytes. This was not even twice the size of the SVN working copy, but with the full history of our project!

Since then, I started using git for all my private repositories: I converted losslessly my private repositories, thanks to the marvelous Hg-Git Mercurial plugin.

I also registered to GiHub, and discovered a whole new dimension to open-source development!
GitHub Logo

There I truly discovered the power of a typical open-source Git workflow: the way to clone (fork) a repository, to make a correction, and then to make a pull request, fully integrated in the issue tracker.

Now, I am beginning to use the hosting facilities of the GitHub Pages, and I must say that I am perfectly happy with their automatic Pages Generator that helps me advertise my open source work.

And for the tools? GitHub help a lot creating, cloning and merging repositories. Under Linux I’ve become used to the good git command line. Under Windows, I am using the wonderful TortoiseGit GUI that I would recommend to you, at least to start.

Now I can tell you: if you try Git, and take the time to learn it subtleties, you will love it!

CMake for Android

Last time, we’ve seen how to install and test the standalone toolchain of the NDK to build a native shared library with a standard Makefile & make command under Linux.

CMake LogoBut nowadays, many open source libraries and projects does not directly come with a Makefile, instead using a build system to automate its generation. CMake is becoming more and more used for those kind of jobs, so being able to use it for Android is really interesting. And easy!

A CMake configuration file for Android (android.toolchain.cmake) is provided by the android-cmake Google project hosted on Mercurial (see my previous post on Mercurial). Let’s clone its repository in the home directory with the following commands:
hg clone https://android-cmake.googlecode.com/hg/ android-cmake
Then, add to your .bashrc the ANDROID_NDK_TOOLCHAIN_ROOT and ANDTOOLCHAIN environment variables pointing to NDK and to the cmake config file. I also added a simple alias for later use:



alias cmake-android='cmake -DCMAKE_TOOLCHAIN_FILE=$ANDTOOLCHAIN'

Ok, let’s try to compile the “hello-cmake” sample provided by the Google project (see hello-cmake documentation):

cd ~/android-cmake/samples/hello-cmake/
mkdir build
cd build

We can simplify the last command by using our alias, so next type we will use “cmake-android ..” instead.
At this stage, CMake has generated the Makefile for you. Use it to build the sample :
Here you are, the shared library has been build and is ready for integration in an Application. You can find it here:
Now we want to invoke this native library from a Java Application, using JNI. This is documented in the next provided sample, “hello-android-cmake“, which is the Java counterpart of the C application we’ve build.

Copy the above “libs” directory to ~/android-cmake/samples/hello-android-cmake. Then open Eclipse and start a “File->New…->Project…->Android Project”. Select “create project from existing source” and give it the ~/android-cmake/samples/hello-android-cmake path.

Build the application, and launch it on an emulator or an actual device. You should see an activity showing the multi-line text starting by “JNI — pi = 3.14159…” coming from our shared library !

Hello Android CMake

Here you are!

Standalone toolchain

Since the revision 5 of the Native Development Kit previously discussed, the NDK can be customized to obtain a standard GCC cross compiler toolchain. This means that you can use it with standard Makefile and make command, or with more sophisticated build tools like automake, CMake or Bakefile. No more Android.mk ant Application.mk proprietary format required ! This is great in many situation where you want to recompile a complex software with big libraries. Lets have a try !

For this experimental article, I’ll assume that you already have good knowledge of Android development, SDK, ADB and some good understanding of C/C++ compilation workflow (gcc, make and Makefile).

NDK Requirements :

  • Android SDK (yes, the Java part !) : I’ll use API-level 5 (Android 2.0) but the minimum is API-level 3 (Android 1.5): “~/android-sdk-linux_x86/”
  • Android NDK r5b (the C/C++ part): “~/android-ndk-r5b/”
  • GNU Make 3.81 or later (“make” package under Ubuntu or Debian)
  • GNU awk or equivalent (nawk says the documentation, I have mawk)

To make this work under Windows (XP or 7) you’ll need Cygwin 1.7 installed, but it’s not that simple (the standalone toolchain officially dos not support Cygwin…). For that reason I’ve decided to work on this entirely under Linux (latest Ubuntu in my case). I you want to try to make it work under Windows, you could exploit useful information from the following blog post http://www.pocketmagic.net/?p=1462 and from some other Internet sources.

So let’s start by reading the “/android-ndk-r5b/docs/STANDALONE-TOOLCHAIN.html“, and proceed directly with the instructions on section 3 “Invoking the compiler (the easy way)”. Under a Linux console, type:

~/android-ndk-r5b/build/tools/make-standalone-toolchain.sh --platform=android-5 --install-dir=$HOME/android-ndk-r5b/standalone-toolchain-api5

Then add this few lines to the end of your .bashrc file and restart a console (or type them directly on the console):


Then, create a standard hello world “hello-ndk.c”:

#include <stdio.h>

int main(void)
    printf("Hello from NDK\n");
    return 0;

At this stage, compiling for Android works easily with the following command:

arm-linux-androideabi-gcc  hello-ndk.c -o hello-ndk

Before testing it, we would like to add a bit of android API in it. You can easily complete it with a minimal Android log (logcat for Eclipse) :

#include <stdio.h>
#include <android/log.h>

#define LOGI(...) ((void)__android_log_print(ANDROID_LOG_INFO, "hello-ndk", __VA_ARGS__))

int main(void)
    printf("Hello from NDK\n");
    LOGI("Hello from NDK");
    return 0;

For this, you need to link explicitly with a the “liblog” library :

  • arm-linux-androideabi-gcc  hello-ndk.c -l log -o hello-ndk

Here is a Makefile to handle this with a simple “make” command (or other “make clean” and “make clean all” variants):

CC	= arm-linux-androideabi-gcc
CFLAGS	= -Wall -g
LDFLAGS	= -llog
SRC	=hello-ndk.c
OBJ	=$(SRC:.c=.o)
EXE	=hello-ndk

all: $(SRC) $(EXE)

$(EXE): $(OBJ)
	$(CC) -o $@ $^ $(LDFLAGS)

%.o: %.c
	$(CC) -o $@ -c $< $(CFLAGS)

	rm -f *.o $(EXE)

This create the standalone executable hello-ndk, so if your device is rooted, you can use adb to copy and execute “hello-ndk” on it, like this:

% adb shell
$ su
# mkdir /data/tmp
# chmod 777 /data/tmp
# exit
$ exit
% adb push hello-ndk /data/tmp
% adb shell
$ /data/tmp/hello-ndk

You should get a “Hello from NDK” on the command line, and the same message on the logcat window of Eclipse (in green, info level).

Hello-NDK-LogcatFor normal users that are not root (no “su” command), there is no way (oops, see edit bellow) of launching directly a native application under Android: you have to package it in a standard Android application .apk and use it from Java code. See http://gimite.net/en/index.php?Run%20native%20executable%20in%20Android%20App.

edit: look at the comment from pitypang, the path /data/local/tmp should work for an unrooted phone !

So we need to build instead a shared library, a “.so” file to be loaded by a Java application (through Java Native Interface, JNI). For this, we need to modify the Makefile like this :

CC	= arm-linux-androideabi-gcc
CFLAGS	= -Wall -g
LDFLAGS	= -llog -shared
SRC	=hello-ndk.c
OBJ	=$(SRC:.c=.o)
EXE	=libhello-ndk.so

With this you will get a libhello-ndk.so shared library. This will do nothing interesting by itself; we need to invoke the native function from Java, or to use the NativeActivity.

I will update this post later to make this works.

Edit:  look also at my post on CMake for Android

Mercurial to replace SVN

Mercurial Logo

For what I’ve seen in the last few month, it seems that Mercurial (Hg) is a Distributed Version Control System (DVCS, or Distributed Source Code Management) that is becoming to replace Subversion (SVN) in many Open Source projects… at least when Git is not already in the place !

While I am in it, Hg and Git have both been started on april 2005, shortly after the announcement that the free version of BitKeeper will be stopped.

Google Project and SourceForge are providing Mercurial as a choice of DVCS, and many tools are compatible with it. I’am using TortoiseHg, as I’ve used TortoiseSVN for many years now (in the same way, there is TortoiseGit).

For Android development, what I appreciate in Mercurial is that it does not cluster the working copy of my projects with “.svn” subdirectories like Subversion ! Instead, there is a single “.hg” directory, and this help a lot when working with Eclipse (no need to exclude explicitly “.svn” in the “src/” directory) as well as for performance/responsiveness!

I’ve quickly learned to use Mercurial with the famous “Hg Init” tutorial, which start by a “Subversion Reeducation“!

Here you can find some articles I’ve read about Hg and Git (and choosing among them) :

Please note that I did NOT really make a choice: I only started to use Hg because its used on some open source projects I am following!

Edit: for installation instruction, see Hg installation on a webserver (HgWebDirStepByStep) and in my case, installation for Debian 5.0. I use a simple “.htaccess” file to restrict access to my server.

Edit 2: please, have a look at my newer post Why I switched from Mercurial to Git

Android Market on the Web

The Android Market is now fully on the Web.
It allows you to browse, to buy, but also to push applications directly on your mobile!

Android Web Market
Android Market on the Web

From a developer perspective, this means that:

Take a look at the market to see how publishers are using this new media to advertise their applications !

Edit: give a look at Publishing on Android Market on the Android Developer website.