Unreal Engine 4: use Swarm to distribute Lighting build on Local Network

This is a quick tutorial on how to install and configure Unreal Engine 4’s Swarm Agent and Coordinator to distribute Lighting build (that is, UE4’s Lightmass’s ray-tracing precomputations) across a Local Network of 64 bits Windows 10 machines. There are already a few good ressources on the subject, but I had to read through a bunch of them to pick all the needed information.

Existing online documentation on Swarm

  1. UDK’s documentation Unreal Swarm – Massive Application Distribution for Unreal Engine 3 is actually quite good, though not entirely up-to-date
  2. UE4’s documentation Lightmass Global Illumination
    is more high level, and lacking a section on Swarm Settings
  3. Brian Goodsell’s Tutorial: Setting Up Swarm for Multiple Machines is very good but slightly out-dated, not saying how to install the Swarm tools
  4. Niberspace’s Is UE4 Distributed Rendering Worth It? gives information on the performances you can expect of it
  5. Baking with Swarm gives a lot of in-details technical information (but far too much)

What is Swarm

The Swarm Agent is the process launched in the background (daemon) when you hit “Build Lighting” on your Unreal Engine Editor. That is, it is the application that bake your  lighting in the UE Lightmass. By default it uses your computer’s CPU cores to do so.

The Swarm Coordinator is a sibling process that you can launch manually to distribute your Lighting build on multiple Agents across the network.

Installation of Swarm Agent and/or Coordinator

Installation on a Windows 7/8/10 64 bits up-to-date is straightforward:

  1. You need the .NET 4.0 framework, which comes by default with any recent version of Windows.
  2. You should also deactivate the power saving/standby mode on every PC, else it will disable the Swarm Agent on the sleeping machines.

To install, you can either:

  1. Use existing installation of Unreal Engine on each computers of the network. All versions are cross-compatible so you can distribute your Lightmass work to Swarm Agents of outdated UE4 installation without any problem.
  2. Or, on computers without UE4, just copy the content of the Engine/Binaries/DotNET folder. You don’t need all this files, and no subfolders, but the whole is only a few MB in size. Put this on a fast disk with more than 10 GB free space for Agent cache.
Content of the DotNET folder

You need to choose an always-on machine, like a small/old server to act as the Swarm Coordinator. If you don’t have one, you can select any other Agent machine to host the Coordinator, but you should ensure that it stays online all the time you need it! On my use-case, I’ve selected my laptop (named “ASUS-I7”) to be the Coordinator, so that I can use it when away.

Lastly, you should ask the system to auto-start the Swarm Agent (and/or Coordinator on the selected machine) when the user log-in. To do so, put a shortcut to the executable on the Windows Startup folder, which is located here:

C:\Users\x\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\

Swarm Agent Settings

The Coordinator does not really need any settings. You need to accept the Windows Firewall request to let it communicate freely, though!

The Swarm Agent needs to be told how to distribute it’s work. To do that, launch the Swarm Agent if it is not already running, and double-click on it’s small icon hidden on the tray bar:

Go to the “Settings” tab of its UI:

Swarm Agent Settings (after full configuration)

Here, the only important Settings are (in bold those you need to change):

  • AgentGroupName: Default (is fine)
  • AllowedRemoteAgentGroup: Default (to match, instead of DefaultDeployed)
  • AllowedRemoteAgentNames: * (to match everything, instead of RENDER*)
  • CoordinatorRemotingHost: ASUS-I7 (name or IP of the Coordinator computer)

Whenever you set the last one, the Coordinator should start to show the new Agent as soon as it connects across the network:

Usage

Next time you build your lighting on any computer correctly configured,you should get your work distributed across multiple Agents, assuming that:

  1. the Coordinator is up,
  2. other Agents are running, connected and Available (not Busy, Working, nor Dead),
  3. and your workload is big enough to distribute (enough mesh to bake lighting onto)

At this stage, a look at the local Agent will tell show you the following Status:

Swarm Agent Status while Working

A look at the Coordinator will tell you the Agent involved in the work:

Swarm Coordinator with Working Agents

I hope this can help, and show how all this is incredibly easy and worthy to setup!

Git Plugin v2.2 for Unreal Engine 4.17

I have just released a new v2.2 of the Git Source Control Provider plugin for Unreal Engine 4.17 with many bugfixes backported from upcoming UE versions!

I am often asked if I still work on it, so I figured that I would publish some news here.

Many Pull Requests!

This is my 16th release of the Git Plugin, with 181 commits across 17 Unreal Engine version (since 4.0 in early 2014 🙂

I have never stopped working on the Git plugin for UE4. I have regularly submitted small or large Pull Requests to the official Unreal Engine’s Github repository (private but free).

To this day, I’ve provided 49 PR to Epic Games, and around 44 have already been merged 🙂. The bulk of them are bugfixes on the Git plugin itself, but I’ve also added many features.

Release v2.2 for UE4.17

  • Fix action icons in History log Window UI (“add”, “delete”, “branch”).
    Before this, every revision of any file history was an “edit” icon!
  • Implement GetBranchSource() so that History now displays the previous name of a file that has been renamed
    Detects special case of move (“branch” in Perforce term) and point to the previous revision
  • Fix “UpdateStatus” not picking renamed assets, nor removed/missing/untracked ones
    Git rename detection require to do “git status” command on the whole sub-directory (not on the explicit list for current asset files)
    Detection of deleted assets require to do an extra pass of parse in this “git status” results specifically for searching such asset files
  • Fix global “UpdateStatus” operation not finding modified assets
  • Run an “UpdateStatus” at “Connect” time to populate the Source Control cache
    This fix a bug when right clicking on a directory of the Content Browser :
    the UI does not know the state of all contained assets and as such present the “Add” entry, but clicking on it can lead to a “no asset to add” message
  • Revert Cleanup RunDumpToFile(): WaitForProc() hang indefinitely!
  • Expand the size of the Button “Initialize project with Git”
    To help remind the user that Git Source Control is not possible without a Git repository initialized.
  • Fix #45 Add logs to help identify problem with git.exe and git lfs
  • Ignore “.vs” folder of Visual Studio 2017

Future

Since Git LFS 2.0 was release earlier this year, I’ve been working on integrating it into the Unreal Engine plugin.

The problem is that it requires quite some intensive rework because it does change the whole workflow and hypothesis on how Git works (DVCS local vs centralized LFS).

So it will come at some point in time, but it will require more work on some smaller features, like support for “remote origin” and associated “git push”.

Support my work

If you want to support my work, small tips are welcome  🙂

Unreal Engine 4.11 Plastic SCM Source Control Provider

I have been working hard for the past month to develop a new Unreal Engine 4 Source Control plugin for Plastic SCM (think modern Perforce with efficient and understandable GUI and very powerful branching management, with some Git interoperability). 
I’ve been using my work on the Git Plugin for Unreal Engine 4 to get the infrastructure work in a short time. Then I had to refactor a lot of it since the workflow of Plastic SCM is nothing like Git.
So here I am with a third alpha release, now already stable and useful:
– status icons for assets
– check-out files
– add, rename, delete asset files
– check-in
– history log of an asset
– Visual Diff of Blueprints

Disclaimer: I’ve done this work for Codice Software, the company behind Plastic SCM.

Edit: Related announcement post in the Community Content section of Unreal Engine forums. This thread has been updated with each subsequent releases of the plugin.

Joining Unreal Engine 4 team for GDC 2016 in San Francisco!

I am so excited to announce that I am joining the Unreal Engine 4 team to attend the Game Developers Conference (GDC, Moscone Center in San Francisco, March 14-18 2016) as a community member (with 9 other great people of this great community)!

I am a senior C++ developer, with 12+ years in the industry, currently working in the transportation industry at ENGIE INEO Systrans.

I’ve embark with Unreal Engine 4 immediately when UE 4.0 came out two years ago, mainly for the opportunity to dive into its C++ code base! I’ve developed the Git Source Control Provider Plugin as a free open-source Github project in my spare time.

I had the opportunity to integrate it officially into the Engine in 4.7 a year ago (just before GDC 2015) !

And now, I’ve been invited to join the team of Epic Games for this mythic event, which feels like a dream to me 🙂

Android NDK r10e and Standalone Toolchain on Ubuntu 14.04

This is an update on my previous post about Android NDK Standalone Toolchain on Ubuntu, so have a look to it for better understanding.

We will see basic setup of the SDK & NDK. We will generate a standalone toolchain, for use with Android 4.1 Jelly Bean (target 90% of active devices). I then uses CMake to generate Makefiles. I am doing all this on the latest Ubuntu LTS (14.04).
ubuntu-logo
Ubuntu 14.04 desktop 64-bit:
Ubuntu download page
After standard installation, we need Java for the Android SDK, and a C/C++ development environment.
sudo apt-get install openjdk-7-jre ant
sudo apt-get install build-essential cmake

Android SDK (the Java part, to bundle C++ in an APK, see Native Activity):
Android SDK (r24.3.4) download page
SDK Tools Only, Linux
32 & 64-bit android-sdk_r24.3.4-linux.tgz
Uncompress under ~/android-sdk-linux/
Execute:
~/android-sdk-linux/tools/android
Add all packages from Android 4.1 – API Level16 (See Adding SDK Packages)

Android NDK (the C/C++ part) (Have a look at my standalone toolchain post):
Android NDK guide and API reference
Android NDK download page
Platform Linux 64-bit (x86) android-ndk-r10e-linux-x86_64.bin
chmod u+x android-ndk-r10e-linux-x86_64.bin
Execute into your $HOME to uncompress the NDK under ~/android-ndk-r10e/

Extract a standalone toolchain for API level 16 (with default GCC 4.8):
~/android-ndk-r10e/build/tools/make-standalone-toolchain.sh --arch)arm --platform=android-16 --install-dir=$HOME/android-standalone-toolchain-api16

Or package it with the following command:
~/android-ndk-r10e/build/tools/make-standalone-toolchain.sh --platform=android-16

Add SDK & Standalone toolchain to your PATH, with your ~/.bashrc file:
$ echo 'PATH=$PATH:$HOME/android-sdk-linux/tools' >> ~/.bashrc
$ echo 'PATH=$PATH:$HOME/android-sdk-linux/platform-tools' >> ~/.bashrc
$ echo 'PATH=$PATH:$HOME/android-standalone-toolchain-api16/bin' >> ~/.bashrc

$ echo ‘export ANDROID_STANDALONE_TOOLCHAIN=$HOME/android-standalone-toolchain-api16’ >> ~/.bashrc

Android CMake (Have a look at my previous post on CMake for Android):
Download the android-cmake now maintained on Github

Note: This is a work in progress, as I will need to refresh all my knowledge to switch to Android Studio in the following months.

Unreal Engine 4.7 Released with my Git Source Control plugin!

Great news, Unreal Engine 4.7 has been released with my Git Source Control Provider plugin!

Epic would like to especially thank community member Sébastien Rombauts for contributing his Git source control plugin to Unreal Engine for this release. Amazing work!

Thanks to community member Sébastien Rombauts for graciously contributing his excellent Git source control plugin to Unreal Engine!!

(emphasized are not mine :))

I’ve started working on this open source C++ Git plugin not long after the UE 4 “open source” release in early 2014.
I’ve released a first rough alpha version of the plugin in May 2014. Since then, I’ve poured dozens of hours of work to maintain and improve it.

 

Lately they asked me for official integration, and I am proud of it!

Unreal Engine 4.1 Git Plugin alpha

So at last I’ve started implementing an open source C++ Git Source Control Plugin (Github) for Unreal Engine 4.1.

For know it is barely a Windows only “pre-alpha v0.0” version. It does only provide overlay icons to show status of your project content assets.

But it’s a start, and it will soon become more useful, I’ll keep you informed on the forums.

Edit 2014/05/17: UE4 Git Plugin alpha v0.1 released, with history log, revert, add and delete. Still no diff (next step) nor commit.

Edit 2014/05/20: UE4 Git Plugin alpha v0.2 released, with diff fully working. Still no commit (next/last big step) but know really useful for anyone using Git.

Edit 2014/05/21: UE4 Git Plugin alpha v0.3 released, with commit working for small batch of files.
As its quickly reaching completion, I’ve started a dedicated new thread on the “Engine Source & GitHub” section (still not ready for the more general “Tools” section I think).

Edit 2014/05/30: UE4 Git Plugin alpha v0.4 released, with many bug fixes and improvements to usability.
I’ve started to use it “for real” in my own small projects.
As I see it, I will move on “Beta” for the next version, to reflect this increased stability.
Please let me know if you try it, an report any bug or missing feature.

Why I switched from Mercurial to Git

After a few month using Mercurial (see my blog post from a year ago Mercurial to replace SVN) I switched to Git.

I’ve always been tempted to use Git, since this Tech Talk by Linus Torvalds in May 3, 2007.
[iframe src=”http://www.youtube.com/embed/4XpnKHJAok8″ width=”420″ height=”315″]
I even tried it a few times, but Mercurial commands seem more familiar to a Subversion user, and at this time, Mercurial GUI was more user friendly.

But then half a year ago, I wanted to convert to Mercurial our SVN code base at work, which is something like 15 man-years of C++.
I used the Mercurial ConvertExtension. The process took hours, used a few gigabytes on disk, but failed after using too much memory.

This failure was a clear signal: I started with the Git – SVN Crash Course.
Git-logo

I used the git-svn clone command, which also took a few hours to run, but succeeded after a shorter time, only using 600 megabytes. This was not even twice the size of the SVN working copy, but with the full history of our project!

Since then, I started using git for all my private repositories: I converted losslessly my private repositories, thanks to the marvelous Hg-Git Mercurial plugin.

I also registered to GiHub, and discovered a whole new dimension to open-source development!
GitHub Logo

There I truly discovered the power of a typical open-source Git workflow: the way to clone (fork) a repository, to make a correction, and then to make a pull request, fully integrated in the issue tracker.

Now, I am beginning to use the hosting facilities of the GitHub Pages, and I must say that I am perfectly happy with their automatic Pages Generator that helps me advertise my open source work.

TortoiseGit-logo
And for the tools? GitHub help a lot creating, cloning and merging repositories. Under Linux I’ve become used to the good git command line. Under Windows, I am using the wonderful TortoiseGit GUI that I would recommend to you, at least to start.

Now I can tell you: if you try Git, and take the time to learn it subtleties, you will love it!

Objectives of this blog

My goals for this blog has never been really clear for me. I bought the domain name when I registered as a Google Android Developer only because a Web site address was required for application publishing, and I though it would be fun to share some experience. I started to write it in French

At the beginning, I did not use this blog frequently, and not the way I had expected. Lately I’ve became more active on it since I’ve discovered that what I was seeking was not sharing general though on Android, but only on some development related topic. What I like most to share are Howtos, Tutorials or easy to use Samples, and more specifically on subject that are not well documented or explored elsewhere (I like “terra incognita”)

During the last few months I’ve realized that, since I like C and love working with C++, the NDK would be the next world to discover (after playing with SDK last year). Since December, I was waiting for Gingerbread and its NativeActivity, but I’ve started to use the new revision of the NDK, r5b that goes a step further in supporting C++. I’ve also been interested on being able to porting some existing (open- or closed-)  source code, like Ogre 3D, with build tools like CMake for Android on Standalone Toolchain.

Since all the resource I use for my research and all the forum I follow are in English, I’ve switched in April to a full English blog, and rewrote everything (not much at this time).

And trying to improve the quality of my posts is teaching me a lot of tips and tricks and give me some additional fun! Now, I would like to improve the visibility of all this materials I’ve assembled, to get some feedback on my work on it 🙂

Now that I’ve switched to Gingerbread, stay tuned for more experiments!

Cheers,
SRombauts

edit: how, and one more thing, this blog is not intended to be a step by step introduction on development on Android for beginners, there are plenty of official resources for that.

Cyanogen Mod 7 – Android 2.3.3 Gingerbread

It has been a long time since the announcement of Gingerbread, last year in December, and still no official update for our phones here in Europe. I definitely want to test the NativeActivity I’ve spoken before, and I would like to get root access on my phone to go further in my testing (and update relative posts on this blog).

So yesterday evening I made the big jump to bring to my Nexus One the power of the recently released Cyanogen Mod 7. I just followed the provided wiki tutorial to update the Nexus One, but here is my experience about it.

CyanogenMod Logo

So I started by backuping my data: SMS with SMS Backup & Restore, log calls with Call Log Backup & Restore, Dolphin Browser HD links with the Bookmarks to SD plug-in, and some savegames, that I all copied from my SD Card to my computer hard drive (given that other important data like Contacts, Email, Calendar and GTasks are already synchronized by Google). I forgot to save my pictures…

Then I unmounted and formated the SD Card, rebooted to the bootloader to wipe all data and restore factory settings.

I installed fastboot on my SDK tools directory, and unlocked my bootloader with:
fastboot oem unlock

I first wanted to flash the ClockworkMod 3.0.0.5 recovery for later use with the famous ROM Manager application, but this version failed to unpack the radio-image I had to flash before Cyanogen (for script compatibility error), so I fall back on the Amon_Ra’s Recovery manual method.
fastboot flash recovery /path/to/recovery-RA-nexus-v2.1.1-CM.img

I had no more trouble flashing the radio-image (passion.5.08.00.04.zip), and then the cyanogen-mod7-update (update-cm-7.0.0-N1-signed.zip) and the Google applications (gapps-gb-20110307-signed.zip). I had to wipe all data once more to boot into the new OS, but after I have been able to restore all my data, to re-download all my favorite applications. Now I can enjoy the new eye-candy tools and other gadgets from Cyanogen, I can play with the new Java and Native API or improvement from Gingerbread, and I can use “root enabled application” like ROM Manager or the “su” command line tool.

This morning, I add the good surprise to see that a ROM update was out, so I used successfully the ROM Manager application to make a full backup of my phone to the SD Card (need to reboot on ClockworkMod recovery to proceed, 350Mo written in a few minutes), to download and to flash the OS update. Quick and smooth.

So now on, stay tuned for more tips and tricks about Android development!

Adding a “What’s New” screen to your Android application

In some case, you want users of your application to explicitly approve a License before being able to use your software. Adding an EULA (End User License Agreement) screen (a Dialog) to pop at the first start of an Android Java Application, is really simple and well documented on the web.

If you want to enforce the EULA at each new release of your application (which could become quickly annoying, beware!), give a look at this linked blog post a “SimpleEula” sample code showing how to do that simply. Something like this is needed as well if you want to update the term of the license.

But what I want is a simple “What’s New” screen to show at each new release, but without the EULA annoyance and without”cancel button”. The kind of dialog we are seeing  more and more nowadays in application of the market. From the above linked example, I’ve made a really simple “What’s New” dialog, here is the WhatsNewScreen.java sample code:

public class WhatsNewScreen {
    private static final String LOG_TAG                 = "WhatsNewScreen";

    private static final String LAST_VERSION_CODE_KEY   = "last_version_code";

    private Activity            mActivity;

    // Constructor memorize the calling Activity ("context")
    public WhatsNewScreen(Activity context) {
        mActivity = context;
    }

    // Show the dialog only if not already shown for this version of the application
    public void show() {
        try {
            // Get the versionCode of the Package, which must be different (incremented) in each release on the market in the AndroidManifest.xml
            final PackageInfo packageInfo = mActivity.getPackageManager().getPackageInfo(mActivity.getPackageName(), PackageManager.GET_ACTIVITIES);

            final SharedPreferences prefs = PreferenceManager.getDefaultSharedPreferences(mActivity);
            final long lastVersionCode = prefs.getLong(LAST_VERSION_CODE_KEY, 0);

            if (packageInfo.versionCode != lastVersionCode) {
                Log.i(LOG_TAG, "versionCode " + packageInfo.versionCode + "is different from the last known version " + lastVersionCode);

                final String title = mActivity.getString(R.string.app_name) + " v" + packageInfo.versionName;

                final String message = mActivity.getString(R.string.whatsnew);

                // Show the News since last version
                AlertDialog.Builder builder = new AlertDialog.Builder(mActivity)
                        .setTitle(title)
                        .setMessage(message)
                        .setPositiveButton(android.R.string.ok, new Dialog.OnClickListener() {

                            public void onClick(DialogInterface dialogInterface, int i) {
                                // Mark this version as read
                                SharedPreferences.Editor editor = prefs.edit();
                                editor.putLong(LAST_VERSION_CODE_KEY, packageInfo.versionCode);
                                editor.commit();
                                dialogInterface.dismiss();
                            }
                        });
                builder.create().show();
            } else {
                Log.i(LOG_TAG, "versionCode " + packageInfo.versionCode + "is already known");
            }

        } catch (PackageManager.NameNotFoundException e) {
            e.printStackTrace();
        }
    }

}

You have to call it on the onCreate of the first activity of your application:

    /** Called when the activity is first created. */
    @Override
    protected void onCreate (Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        // Layout of the main activity
        setContentView(R.layout.main);

        // Show the "What's New" screen once for each new release of the application
        new WhatsNewScreen(this).show();

        ...
    }

Here is an example of the XML string resource used res/values/strings.xml:

<?xml version="1.0" encoding="utf-8"?>
<resources>
    ...

    <string name="whatsnew">
        - Removed many SQLite exception stack\n
        - Improved the way the application launch on notification (no more multi-activity stacking)\n
        - Added this What\'s new screen\n
    </string>

    ...
</resources>

Enjoy!

CMake for Android

Last time, we’ve seen how to install and test the standalone toolchain of the NDK to build a native shared library with a standard Makefile & make command under Linux.

CMake LogoBut nowadays, many open source libraries and projects does not directly come with a Makefile, instead using a build system to automate its generation. CMake is becoming more and more used for those kind of jobs, so being able to use it for Android is really interesting. And easy!

A CMake configuration file for Android (android.toolchain.cmake) is provided by the android-cmake Google project hosted on Mercurial (see my previous post on Mercurial). Let’s clone its repository in the home directory with the following commands:
hg clone https://android-cmake.googlecode.com/hg/ android-cmake
Then, add to your .bashrc the ANDROID_NDK_TOOLCHAIN_ROOT and ANDTOOLCHAIN environment variables pointing to NDK and to the cmake config file. I also added a simple alias for later use:

PATH=$PATH:$HOME/android-sdk-linux_x86/tools
PATH=$PATH:$HOME/android-sdk-linux_x86/platform-tools
PATH=$PATH:$HOME/android-ndk-r5b/standalone-toolchain-api5/bin/

ANDROID_NDK_TOOLCHAIN_ROOT=$HOME/android-ndk-r5b/standalone-toolchain-api5
ANDTOOLCHAIN=$HOME/android-cmake/toolchain/android.toolchain.cmake

alias cmake-android='cmake -DCMAKE_TOOLCHAIN_FILE=$ANDTOOLCHAIN'

Ok, let’s try to compile the “hello-cmake” sample provided by the Google project (see hello-cmake documentation):

cd ~/android-cmake/samples/hello-cmake/
mkdir build
cd build
export ANDROID_NDK_TOOLCHAIN_ROOT
cmake -DCMAKE_TOOLCHAIN_FILE=$ANDTOOLCHAIN ..

We can simplify the last command by using our alias, so next type we will use “cmake-android ..” instead.
At this stage, CMake has generated the Makefile for you. Use it to build the sample :
make
Here you are, the shared library has been build and is ready for integration in an Application. You can find it here:
~/android-cmake/samples/hello-cmake/libs/armeabi-v7a/libhello-cmake.so
Now we want to invoke this native library from a Java Application, using JNI. This is documented in the next provided sample, “hello-android-cmake“, which is the Java counterpart of the C application we’ve build.

Copy the above “libs” directory to ~/android-cmake/samples/hello-android-cmake. Then open Eclipse and start a “File->New…->Project…->Android Project”. Select “create project from existing source” and give it the ~/android-cmake/samples/hello-android-cmake path.

Build the application, and launch it on an emulator or an actual device. You should see an activity showing the multi-line text starting by “JNI — pi = 3.14159…” coming from our shared library !

Hello Android CMake

Here you are!

Ogre3D to Android

Ogre3D Logo

I’ve started to read a lot about Ogre3D, the open-source Object Oriented 3D Engine, which I’ve always considered to be a really interesting subject. This was triggered by a comment about ongoing effort to port of Ogre3D to Android !

This interest is going far beyond what I expected… I’ve begun by following the Ogre Basic Tutorials, and I’ve already done some tests with elements from the more advanced Tutorials, regarding the in-code mesh generation, and then terrain management and paging management! I’ve followed this path into looking to environment generators, like Ogre SkyX (sky) and Hydrax (water). I was sorry not to found any open-source outstanding project for plants, foliage and vegetation management 🙁

I was disappointed to see that the Ogre port to Android does not seems to be in a good shape; a least there was not much discussion on this recently, and there is no good documentation on how to proceed with the Android NDK :

For me, its kind of a very interesting challenge: I would like to improve the documentation on all this, a least to the point where I can bring it (I do not have much time for this).

First of all, I needed to read some more about the standard process for building Ogre with CMake:

My first step will be to compile a minimal (monothreaded without Boost) Ogre 1.8 (unstable) from source for Linux:

Standalone toolchain

Since the revision 5 of the Native Development Kit previously discussed, the NDK can be customized to obtain a standard GCC cross compiler toolchain. This means that you can use it with standard Makefile and make command, or with more sophisticated build tools like automake, CMake or Bakefile. No more Android.mk ant Application.mk proprietary format required ! This is great in many situation where you want to recompile a complex software with big libraries. Lets have a try !

For this experimental article, I’ll assume that you already have good knowledge of Android development, SDK, ADB and some good understanding of C/C++ compilation workflow (gcc, make and Makefile).

NDK Requirements :

  • Android SDK (yes, the Java part !) : I’ll use API-level 5 (Android 2.0) but the minimum is API-level 3 (Android 1.5): “~/android-sdk-linux_x86/”
  • Android NDK r5b (the C/C++ part): “~/android-ndk-r5b/”
  • GNU Make 3.81 or later (“make” package under Ubuntu or Debian)
  • GNU awk or equivalent (nawk says the documentation, I have mawk)

To make this work under Windows (XP or 7) you’ll need Cygwin 1.7 installed, but it’s not that simple (the standalone toolchain officially dos not support Cygwin…). For that reason I’ve decided to work on this entirely under Linux (latest Ubuntu in my case). I you want to try to make it work under Windows, you could exploit useful information from the following blog post http://www.pocketmagic.net/?p=1462 and from some other Internet sources.

So let’s start by reading the “/android-ndk-r5b/docs/STANDALONE-TOOLCHAIN.html“, and proceed directly with the instructions on section 3 “Invoking the compiler (the easy way)”. Under a Linux console, type:

~/android-ndk-r5b/build/tools/make-standalone-toolchain.sh --platform=android-5 --install-dir=$HOME/android-ndk-r5b/standalone-toolchain-api5

Then add this few lines to the end of your .bashrc file and restart a console (or type them directly on the console):

PATH=$PATH:$HOME/android-sdk-linux_x86/tools
PATH=$PATH:$HOME/android-sdk-linux_x86/platform-tools
PATH=$PATH:$HOME/android-ndk-r5b/standalone-toolchain-api5/bin/

Then, create a standard hello world “hello-ndk.c”:

#include <stdio.h>

int main(void)
{
    printf("Hello from NDK\n");
    return 0;
}

At this stage, compiling for Android works easily with the following command:

arm-linux-androideabi-gcc  hello-ndk.c -o hello-ndk

Before testing it, we would like to add a bit of android API in it. You can easily complete it with a minimal Android log (logcat for Eclipse) :

#include <stdio.h>
#include <android/log.h>

#define LOGI(...) ((void)__android_log_print(ANDROID_LOG_INFO, "hello-ndk", __VA_ARGS__))

int main(void)
{
    printf("Hello from NDK\n");
    LOGI("Hello from NDK");
    return 0;
}

For this, you need to link explicitly with a the “liblog” library :

  • arm-linux-androideabi-gcc  hello-ndk.c -l log -o hello-ndk

Here is a Makefile to handle this with a simple “make” command (or other “make clean” and “make clean all” variants):

CC	= arm-linux-androideabi-gcc
CFLAGS	= -Wall -g
LDFLAGS	= -llog
SRC	=hello-ndk.c
OBJ	=$(SRC:.c=.o)
EXE	=hello-ndk

all: $(SRC) $(EXE)

$(EXE): $(OBJ)
	$(CC) -o $@ $^ $(LDFLAGS)

%.o: %.c
	$(CC) -o $@ -c $< $(CFLAGS)

clean:
	rm -f *.o $(EXE)

This create the standalone executable hello-ndk, so if your device is rooted, you can use adb to copy and execute “hello-ndk” on it, like this:

% adb shell
$ su
# mkdir /data/tmp
# chmod 777 /data/tmp
# exit
$ exit
% adb push hello-ndk /data/tmp
% adb shell
$ /data/tmp/hello-ndk

You should get a “Hello from NDK” on the command line, and the same message on the logcat window of Eclipse (in green, info level).

Hello-NDK-LogcatFor normal users that are not root (no “su” command), there is no way (oops, see edit bellow) of launching directly a native application under Android: you have to package it in a standard Android application .apk and use it from Java code. See http://gimite.net/en/index.php?Run%20native%20executable%20in%20Android%20App.

edit: look at the comment from pitypang, the path /data/local/tmp should work for an unrooted phone !

So we need to build instead a shared library, a “.so” file to be loaded by a Java application (through Java Native Interface, JNI). For this, we need to modify the Makefile like this :

CC	= arm-linux-androideabi-gcc
CFLAGS	= -Wall -g
LDFLAGS	= -llog -shared
SRC	=hello-ndk.c
OBJ	=$(SRC:.c=.o)
EXE	=libhello-ndk.so

With this you will get a libhello-ndk.so shared library. This will do nothing interesting by itself; we need to invoke the native function from Java, or to use the NativeActivity.

I will update this post later to make this works.

Edit:  look also at my post on CMake for Android

Android 2.3 NativeActivity

At each platform API revision, the C native APIs are growing, giving access to new native library, and bringing each time more functionalities to the low level world of C/C++. At this point, they are starting to look mature enough for me to want to dive into them.

As discussed earlier, Android 2.3 (API-9) brings to the C/C++ developers a new Java Class called “NativeActivity“. This allow us to write an entire Android application without using the Java API, managing inputs, internal sensors and the application window entirely with the so called native C APIs of the platforms.

The provided “native-activity” sample (“android-ndk-r5/samples/native-activity“, or in the online documentation of the NativeActivity Class) gives you an example of making use of this new API through a threaded “app-glue” helper. The web page What is the NDK does explain how to create a project from sources of this sample, with or without Eclipse. I made it work flawlessly under the emulator (as I am currently stuck with Android 2.2, API level 8).

The “native_app_glue” wrapper (“android-ndk-r5/sources/android/native_app_glue“) is designed to abstract the complexity of creating a native pthread and synchronizing it with the NativeActivity Lifecyle callbacks. It thus manage for you most of the work needed to handle the complexity of the classical transition (onCreate, onStart, onSaveInstanceState, onResume…) but also more dirty tasks like managing the application window (onNativeWindowCreated…) or the input events looper (onInputQueueCreated…)

But if you want to really understand all the magic behind this app glue, if like me you want to make your own by yourself (for instance in C++), it will need time to dig in it, as there is no comments to help the reader ! I will update this post with a commented version of my C++ implementation, in a step by step process with explanations.

Mercurial to replace SVN

Mercurial Logo

For what I’ve seen in the last few month, it seems that Mercurial (Hg) is a Distributed Version Control System (DVCS, or Distributed Source Code Management) that is becoming to replace Subversion (SVN) in many Open Source projects… at least when Git is not already in the place !

While I am in it, Hg and Git have both been started on april 2005, shortly after the announcement that the free version of BitKeeper will be stopped.

Google Project and SourceForge are providing Mercurial as a choice of DVCS, and many tools are compatible with it. I’am using TortoiseHg, as I’ve used TortoiseSVN for many years now (in the same way, there is TortoiseGit).

For Android development, what I appreciate in Mercurial is that it does not cluster the working copy of my projects with “.svn” subdirectories like Subversion ! Instead, there is a single “.hg” directory, and this help a lot when working with Eclipse (no need to exclude explicitly “.svn” in the “src/” directory) as well as for performance/responsiveness!

I’ve quickly learned to use Mercurial with the famous “Hg Init” tutorial, which start by a “Subversion Reeducation“!

Here you can find some articles I’ve read about Hg and Git (and choosing among them) :

Please note that I did NOT really make a choice: I only started to use Hg because its used on some open source projects I am following!

Edit: for installation instruction, see Hg installation on a webserver (HgWebDirStepByStep) and in my case, installation for Debian 5.0. I use a simple “.htaccess” file to restrict access to my server.

Edit 2: please, have a look at my newer post Why I switched from Mercurial to Git

Android Market on the Web

The Android Market is now fully on the Web.
It allows you to browse, to buy, but also to push applications directly on your mobile!

Android Web Market
Android Market on the Web

From a developer perspective, this means that:

Take a look at the market to see how publishers are using this new media to advertise their applications !

Edit: give a look at Publishing on Android Market on the Android Developer website.

Android Native Development Kit

Android 2.3 Gingerbread has been announced, and what I find the most interesting in it is related to the Native Development KIT (NDK) coming with it; the NDK r5.
Android-NDK

The new API level 9 of Android 2.3 brings with it a “Native access to Activity lifecycle” : it allows developers to write an Android Application with absolutely no Java coding, by the use of a provided Java Class “NativeActivity“. A “native-activity” sample gives you an example of making use of it through the “native_app_glue”, a C-only wrapper (see also my experiments on NativeActivity).

At the same time this revision r5 of the NDK is becoming interesting for me, with the first official support of C++ exception and RTTI. There is also a way to get a “standalone toolchain” with GCC 4.3, which give the ability to use only standard cross-compiler command (make) instead of the specific Android.mk files.

All this is not really well documented… You get plenty of example and sources, but there is not much in-code comments nor any “Native API Reference” available. In fact, even the documentation is not available directly on Internet: it is only provided on the NDK archive!

I share it on this web server : here is the NDK-r5b HTML documentation (http://www.srombauts.fr/android-ndk-r5b/)

Edit: look at this Android Developer Blog post about Gingerbread NDK
Edit2: there is also a android-ndk Google group
Edit3: some other people share my enthusiasm about Gingerbread and the new NDK revision !

Mind The Robot

I’ve discovered, through the Android Dev Blog, this great Android Blog called Mind The Robot

Through all his tutorials, Ivan provides us with a lot of explanation, some good advices and many useful insights of the Android platform. I’ve followed successfully the Android UI: Making a Live Wallpaper (Fire Simulation) tutorial to make an animated wallpaper, great !

I also love his idea of making vintage controls, look at his beautiful Vintage Thermometer :

Mind The Robot - Vintage Thermometer
Mind The Robot - Custom UI: Vintage Thermometer

Take a look at this website, you will love it !

Edit: the 11th of February, Ivan announced that he has no more time for his blog.

Getting Started with Android Development

I’ve been asked many time to explain how I’ve learned Android Development. And I’ve always answered that the official ressources on the Android Developers site associated with the “Professional Android 2 Application Development” book have been perfect for me.

But here is the nearly perfect blog post you should read : Getting Started with Android Development, by Jean Hsu.

And to begin with, got to the Android SDK download page.

Enjoy!
SRombauts

edit: fixed URL, added direct link to SDK download page