Unreal Engine 4: use Swarm to distribute Lighting build on Local Network

This is a quick tutorial on how to install and configure Unreal Engine 4’s Swarm Agent and Coordinator to distribute Lighting build (that is, UE4’s Lightmass’s ray-tracing precomputations) across a Local Network of 64 bits Windows 10 machines. There are already a few good ressources on the subject, but I had to read through a bunch of them to pick all the needed information.

Existing online documentation on Swarm

  1. UDK’s documentation Unreal Swarm – Massive Application Distribution for Unreal Engine 3 is actually quite good, though not entirely up-to-date
  2. UE4’s documentation Lightmass Global Illumination
    is more high level, and lacking a section on Swarm Settings
  3. Brian Goodsell’s Tutorial: Setting Up Swarm for Multiple Machines is very good but slightly out-dated, not saying how to install the Swarm tools
  4. Niberspace’s Is UE4 Distributed Rendering Worth It? gives information on the performances you can expect of it
  5. Baking with Swarm gives a lot of in-details technical information (but far too much)

What is Swarm

The Swarm Agent is the process launched in the background (daemon) when you hit “Build Lighting” on your Unreal Engine Editor. That is, it is the application that bake your  lighting in the UE Lightmass. By default it uses your computer’s CPU cores to do so.

The Swarm Coordinator is a sibling process that you can launch manually to distribute your Lighting build on multiple Agents across the network.

Installation of Swarm Agent and/or Coordinator

Installation on a Windows 7/8/10 64 bits up-to-date is straightforward:

  1. You need the .NET 4.0 framework, which comes by default with any recent version of Windows.
  2. You should also deactivate the power saving/standby mode on every PC, else it will disable the Swarm Agent on the sleeping machines.

To install, you can either:

  1. Use existing installation of Unreal Engine on each computers of the network. All versions are cross-compatible so you can distribute your Lightmass work to Swarm Agents of outdated UE4 installation without any problem.
  2. Or, on computers without UE4, just copy the content of the Engine/Binaries/DotNET folder. You don’t need all this files, and no subfolders, but the whole is only a few MB in size. Put this on a fast disk with more than 10 GB free space for Agent cache.
Content of the DotNET folder

You need to choose an always-on machine, like a small/old server to act as the Swarm Coordinator. If you don’t have one, you can select any other Agent machine to host the Coordinator, but you should ensure that it stays online all the time you need it! On my use-case, I’ve selected my laptop (named “ASUS-I7”) to be the Coordinator, so that I can use it when away.

Lastly, you should ask the system to auto-start the Swarm Agent (and/or Coordinator on the selected machine) when the user log-in. To do so, put a shortcut to the executable on the Windows Startup folder, which is located here:

C:\Users\x\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\

Swarm Agent Settings

The Coordinator does not really need any settings. You need to accept the Windows Firewall request to let it communicate freely, though!

The Swarm Agent needs to be told how to distribute it’s work. To do that, launch the Swarm Agent if it is not already running, and double-click on it’s small icon hidden on the tray bar:

Go to the “Settings” tab of its UI:

Swarm Agent Settings (after full configuration)

Here, the only important Settings are (in bold those you need to change):

  • AgentGroupName: Default (is fine)
  • AllowedRemoteAgentGroup: Default (to match, instead of DefaultDeployed)
  • AllowedRemoteAgentNames: * (to match everything, instead of RENDER*)
  • CoordinatorRemotingHost: ASUS-I7 (name or IP of the Coordinator computer)

Whenever you set the last one, the Coordinator should start to show the new Agent as soon as it connects across the network:

Usage

Next time you build your lighting on any computer correctly configured,you should get your work distributed across multiple Agents, assuming that:

  1. the Coordinator is up,
  2. other Agents are running, connected and Available (not Busy, Working, nor Dead),
  3. and your workload is big enough to distribute (enough mesh to bake lighting onto)

At this stage, a look at the local Agent will tell show you the following Status:

Swarm Agent Status while Working

A look at the Coordinator will tell you the Agent involved in the work:

Swarm Coordinator with Working Agents

I hope this can help, and show how all this is incredibly easy and worthy to setup!

Git Plugin v2.2 for Unreal Engine 4.17

I have just released a new v2.2 of the Git Source Control Provider plugin for Unreal Engine 4.17 with many bugfixes backported from upcoming UE versions!

I am often asked if I still work on it, so I figured that I would publish some news here.

Many Pull Requests!

This is my 16th release of the Git Plugin, with 181 commits across 17 Unreal Engine version (since 4.0 in early 2014 🙂

I have never stopped working on the Git plugin for UE4. I have regularly submitted small or large Pull Requests to the official Unreal Engine’s Github repository (private but free).

To this day, I’ve provided 49 PR to Epic Games, and around 44 have already been merged 🙂. The bulk of them are bugfixes on the Git plugin itself, but I’ve also added many features.

Release v2.2 for UE4.17

  • Fix action icons in History log Window UI (“add”, “delete”, “branch”).
    Before this, every revision of any file history was an “edit” icon!
  • Implement GetBranchSource() so that History now displays the previous name of a file that has been renamed
    Detects special case of move (“branch” in Perforce term) and point to the previous revision
  • Fix “UpdateStatus” not picking renamed assets, nor removed/missing/untracked ones
    Git rename detection require to do “git status” command on the whole sub-directory (not on the explicit list for current asset files)
    Detection of deleted assets require to do an extra pass of parse in this “git status” results specifically for searching such asset files
  • Fix global “UpdateStatus” operation not finding modified assets
  • Run an “UpdateStatus” at “Connect” time to populate the Source Control cache
    This fix a bug when right clicking on a directory of the Content Browser :
    the UI does not know the state of all contained assets and as such present the “Add” entry, but clicking on it can lead to a “no asset to add” message
  • Revert Cleanup RunDumpToFile(): WaitForProc() hang indefinitely!
  • Expand the size of the Button “Initialize project with Git”
    To help remind the user that Git Source Control is not possible without a Git repository initialized.
  • Fix #45 Add logs to help identify problem with git.exe and git lfs
  • Ignore “.vs” folder of Visual Studio 2017

Future

Since Git LFS 2.0 was release earlier this year, I’ve been working on integrating it into the Unreal Engine plugin.

The problem is that it requires quite some intensive rework because it does change the whole workflow and hypothesis on how Git works (DVCS local vs centralized LFS).

So it will come at some point in time, but it will require more work on some smaller features, like support for “remote origin” and associated “git push”.

Support my work

If you want to support my work, small tips are welcome  🙂

Unreal Engine 4.11 Plastic SCM Source Control Provider

I have been working hard for the past month to develop a new Unreal Engine 4 Source Control plugin for Plastic SCM (think modern Perforce with efficient and understandable GUI and very powerful branching management, with some Git interoperability). 
I’ve been using my work on the Git Plugin for Unreal Engine 4 to get the infrastructure work in a short time. Then I had to refactor a lot of it since the workflow of Plastic SCM is nothing like Git.
So here I am with a third alpha release, now already stable and useful:
– status icons for assets
– check-out files
– add, rename, delete asset files
– check-in
– history log of an asset
– Visual Diff of Blueprints

Disclaimer: I’ve done this work for Codice Software, the company behind Plastic SCM.

Edit: Related announcement post in the Community Content section of Unreal Engine forums. This thread has been updated with each subsequent releases of the plugin.

Joining Unreal Engine 4 team for GDC 2016 in San Francisco!

I am so excited to announce that I am joining the Unreal Engine 4 team to attend the Game Developers Conference (GDC, Moscone Center in San Francisco, March 14-18 2016) as a community member (with 9 other great people of this great community)!

I am a senior C++ developer, with 12+ years in the industry, currently working in the transportation industry at ENGIE INEO Systrans.

I’ve embark with Unreal Engine 4 immediately when UE 4.0 came out two years ago, mainly for the opportunity to dive into its C++ code base! I’ve developed the Git Source Control Provider Plugin as a free open-source Github project in my spare time.

I had the opportunity to integrate it officially into the Engine in 4.7 a year ago (just before GDC 2015) !

And now, I’ve been invited to join the team of Epic Games for this mythic event, which feels like a dream to me 🙂

Unreal Engine 4.7 Released with my Git Source Control plugin!

Great news, Unreal Engine 4.7 has been released with my Git Source Control Provider plugin!

Epic would like to especially thank community member Sébastien Rombauts for contributing his Git source control plugin to Unreal Engine for this release. Amazing work!

Thanks to community member Sébastien Rombauts for graciously contributing his excellent Git source control plugin to Unreal Engine!!

(emphasized are not mine :))

I’ve started working on this open source C++ Git plugin not long after the UE 4 “open source” release in early 2014.
I’ve released a first rough alpha version of the plugin in May 2014. Since then, I’ve poured dozens of hours of work to maintain and improve it.

 

Lately they asked me for official integration, and I am proud of it!

Unreal Engine 4.1 Git Plugin alpha

So at last I’ve started implementing an open source C++ Git Source Control Plugin (Github) for Unreal Engine 4.1.

For know it is barely a Windows only “pre-alpha v0.0” version. It does only provide overlay icons to show status of your project content assets.

But it’s a start, and it will soon become more useful, I’ll keep you informed on the forums.

Edit 2014/05/17: UE4 Git Plugin alpha v0.1 released, with history log, revert, add and delete. Still no diff (next step) nor commit.

Edit 2014/05/20: UE4 Git Plugin alpha v0.2 released, with diff fully working. Still no commit (next/last big step) but know really useful for anyone using Git.

Edit 2014/05/21: UE4 Git Plugin alpha v0.3 released, with commit working for small batch of files.
As its quickly reaching completion, I’ve started a dedicated new thread on the “Engine Source & GitHub” section (still not ready for the more general “Tools” section I think).

Edit 2014/05/30: UE4 Git Plugin alpha v0.4 released, with many bug fixes and improvements to usability.
I’ve started to use it “for real” in my own small projects.
As I see it, I will move on “Beta” for the next version, to reflect this increased stability.
Please let me know if you try it, an report any bug or missing feature.