Project in Computer Graphics

Final Report

Fishing Game Using Android Tablet and 3D Input Device

Author: Dany Rybnikov

This project was developed as part of cooperation effort between Computer Graphics and Geometry Center at the Technion Computer Science faculty and Qualcomm Israel.


Project Goal

The goal of the project is to develop a game application using a 3D input device (e.g: ultrasound stylus) which has at least 3 degrees of freedom and allows to monitor in real time a stylus position on top of the device screen by providing relative X,Y and Z coordinates. In particular the game makes use of Android tablet and the hovering stylus. Game’s objective is to create fishing-like experience for the player, while the stylus is hovering above the tablet. While the fish are rendered on the screen as swimming on on various depths, the player must adjust the stylus position and height above the screen in order to catch the swimming fish. Once all the fish is caught the game is over.

Background

Since the release of iPhone the “smartphone” mobile world is growing at extremely fast pace. Along with iOS, Android has become a major player in the mobile OS world. As of today at least 75% of the smartphones worldwide are shipped with Android. Along with the huge number of the smartphone devices a large developer community has emerged, producing a vast number of mobile applications with games being one of the most popular application types.

One of the main obstacles for the mobile game developer is a lack of “standard” set of input controls e.g.: mouse and keyboard. Though on screen controls are quite a common possibility on the touch screen enabled devices, these are only suitable for a certain type of games which does not require more complex user interaction.

In spite (or due to) that fact, many smartphone games are gaining popularity by using a novice set of controls. As an example we can take popular titles such as “Temple Run”, which uses accelerator sensor along with the touch controls to enrich playing experience. Another example are popular “100 doors” series which makes use of almost ALL phone sensors by requiring a player to use them in different ways while solving a door-open puzzles.

In that context, having a new input control introduced to the mobile platform gives a rise to a number of opportunities and mobile gaming is one of them. This project will demonstrate usage of new type of control in gaming and explore the challenges and opportunities introduced by it.

HW platform and 3D input device

3D Input device

The heart of this project lies in usage of an ultrasound emitting digital pen. The pen contains a number of ultrasound (US) emitting transmitters. Those speaker emit a high pitch which cannot be heard by the human ear, but can be reliably detected by the microphones on the tablet and analyzed to provide an exact 3D position of the pen relative to the tablet.

The tablet used was a Qualcomm development tablet device “Liquid” MSM8960 with Snapdragon S4 and Adreno 225 graphics processor. The tablet screen resolution is 1364x710

and it supports OpenGL ES 2.0.

Game Engines for Android Overview

One of the major components in game development is choosing the right game engine. Game engine serves as framework for game development and provides all the majour services so the  the game developers can focus on the game design. The services that a capable game engine usually provides include: graphics engine, sound engine, assets management and a framework for implementing the game logic. A good game engine will also provide an IDE to allow game levels design and support the overall game development: logic implementation and debugging, testing, and ultimately even deployment process.

An adequate game engine for this project will have to meet the following requirements:

  1. Support for 3D graphics on Android platform
  2. Extensibility for new input devices (3D stylus)
  3. Very low cost per engine usage/copy

Thorough research on the internet revealed that the number of good game engine which support 3D gaming on Android is quite limited. Most of them are only a limited graphics frameworks, which require a lot of heavy lifting before reaching a suitable game quilty. Finally after reviewing the list of game engines http://mobilegameengines.com/android/game_engines, the two finalists were selected: Unity 3D and open source gamekit http://www.gamekit.org game engines.

While Unity was the best candidate since it provided all the required capabilities for the project it had a too high price tag which came with the Unity Pro version. The Unity Pro includes among other things support for the Android platform and extensibility through plugins. However, due to relatively high cost and a requirement to purchase a new copy for every student/computer that will use the Unity engine, it was decided to select gamekit game engine as a development platform for the project.

Gamkit - an open source cross-platform game engine

Gamekit Engine Overview

Gamekit is a cross-platform open source 3D game engine for Windows, Linux, Mac, Android, iPad and iPhone. Gamekit development started around year 2009, while the initial goal was to provide a cross-platform open source game engine which has Blender 3D http://www.blender3d.org  application serving as a gameengine IDE and Blender files serving as game assets containers. Eventually gamekit grew to be the game engine providing most of the essential services to the game developer by incorporating other open source projects:

  1. OrgreKit - is a wrapper for Open source Ogre3D graphics engine http://www.ogre3d.org/ 
  2. Bullet - an open source game physics simulator http://www.bulletphysics.com/
  3. OpenAL - audio library http://connect.creativelabs.com/openal/default.aspx
  4. OIS - Object Oriented Input System. http://sourceforge.net/projects/wgois/

The core of gamekit is called “Engine” and is written in C++. It has an extensive C++ APIs, but also provides a LUA bindings. LUA is a script language which can be used in the Blender game logic and exposes a subset of gamekit capabilities directly to the Blender.

As mentioned above, Blender application is the core component of the engine, though it is not used in run-time, all the character and level design operation and also most of the game logic implementation are done in the Blender application. It is also possible to include gamekit Blender add-on (included with gamekit sources) which will allow launching the gamekit runtime from within the Blender, and by this allow a better development experience.

The development process is mostly done inside Blender applications and is very similar to using the native Blender game engine (BGE).

  1. Design the game components
  1. Character mesh/skeleton
  2. Level design
  1. Place Lightning / Camera
  2. Create animations/actions
  3. Implement the basic logic using Blender Logic Bricks
  1. Simple actions / animations
  2. Physics simulations
  1. For advanced game logic implement a LUA scripts(s)
  1. Implement classes supporting your game logic
  2. Work directly with gamekit APIs via LUA bindings
  3. Advanced actions, state machines and physics
  1. On Windows:
  1. Lunch the game by using the gamekit runtime executable
  1. On Android:
  1. Put the blender file (*.blend) inside the android demo project (gk_android.blend)
  2. Compile and load the APK to the device

Though the original intentions were to make game development on gamekit as similar as possible to using the BGE, many discrepancies exists between the two and due to complete lack of documentation for the gamekit these are very hard to overcome.

Required tools

Blender

Blender can be downloaded from http://www.blender3d.org. Blender v2.63 was used for this project, but hopefully future versions will be also compatible with the gamekit and the the game .blend file.

Blender is a very capable 3D modelling and rendering environment. Though it might be hard to learn at first, once you get the essentials it becomes a natural choice for any 3D-related work. There are numerous tutorials on the internet describing the modelling and rendering techniques. But BGE (Blender game engine) tutorials are the most suitable ones for gamekit game development.

Recommended links:

http://www.youtube.com/watch?v=tczC2URHRao (12 part series)

http://wiki.blender.org/index.php/Doc:2.6/Manual/Game_Engine (Blender manual with examples)

Android SDK

There is a need to download and install latest Android SDK in order to compile the android gamekit demo. The SDK can be downloaded from: http://developer.android.com/sdk/index.html

Eclipse

Eclipse can be downloaded from http://www.eclipse.org. Eclipse is the natural choice as an IDE  for Android. There is an Android SDK plug-in for eclipse which allows you to code, compile and debug Android apps.

In order to compile the project game into an APK you will need to install Eclipse with Java support and make sure that it has the following add-ons:

  1. Android SDK
  2. eGit (to get the game source directly from github)

Other tools

Optional - needed for changing and compiling the gamekit source

  1. Visual Studio Express  - used to compile and debug the gamekit engine on windows http://www.microsoft.com/visualstudio/eng/products/visual-studio-express-products
  2. Android NDK - compile the gamekit native library for Android http://developer.android.com/tools/sdk/ndk/index.html
  3. Git for windows - extract the gamekit source from GitHub  http://git-scm.com/download/win

Getting and Building the Code

Getting and building the Fishing Game

The game sources are available from repository on the GitHub server (http://www.github.com). The repository link is:

git://github.com/danyr/Fishing.git

This link provide a read only access to repository, however you can use a fork feature of github to get your own copy of the repository with the read/write access.

The game sources contain an eclipse project, Android Java sources, blender game file and precompiled native gamekit library for android. The steps to get and build the game:

  1. Open Eclipse
  2. Select import of the GIT project and enter the repository url.
  3. In the relevant dialog make sure ‘master’ branch is selected

After the process is finished you will have an Eclipse project which is ready to be built into Android APK. The project is built automatically once its download from repository is complete.

Loading the APK into Android device should be quite a simple task: Press the “Run” button on the Eclipse toolbar. This will initiate the project Run & Load, and if the phone/tablet is correctly connected to the computer you should get a dialog to select on which device to load the project’s APK. Once your device is selected the project is loaded and started on the device.

For more details please refer to Google pages describing Eclipse and Android SDK usage.

Getting and Building the Gamekit Engine

Since the precompiled runtime is included with the game, gamekit engine sources are required only if there is a need to make changes in a gamekit or change a game itself and use windows gamekit run-time for debug purpose.

Current version of the game requires special version of the game it runtime. The main reasons are addition of the 3D stylus support and various bug fixes which are required to run the game logic correctly. Due to this reason getting the gamekit sources should be done from special repository on the GitHub server which contains the gamekit sources with all the above changes.

In order to get the sources you will need a windows command line git client. Once ready use the following command line get the sources:

Use the visual studio command line

  1. git clone git://github.com/danyr/gamekit.git
  2. set NDK=C:\<android-ndk-path>
  3. configure-android.bat c:\where_to_where_to_build_gamekit
  4. cd c:\where_to_where_to_build_gamekit
  5. nmake

For further build instruction you can refer to the gamekit original build procedure for Android or other operating systems: https://code.google.com/p/gamekit/wiki/Building

Development and Debug

When using gamekit the game development usually at first takes place in Blender environment on windows and only later moves to the mobile device. The main reason for that is convenience and debuggability of the windows version of the gamekit. Thought Blender Game Engine is very similar to the gamekit capabilities there are many differences, thus it is not advisable to develop BGE game and then use gamekit, rather it is best to use gamekit from the start. For this purpose you should enable the gamekit add-on in Blender. The add-on is provided as part of the gamekit sources and can added and enabled in Blender using the standard method. https://code.google.com/p/gamekit/wiki/BlenderAddon provides more details. Once addon is loaded you should change to the gamekit engine and provide the webkit run-time executable path in the render dialog so that Blender will be able to launch the game on windows. Note, that in order to use the above functionality with the gamekit version used by this project you will have to compile gamekit sources from github on windows. Please refer to https://code.google.com/p/gamekit/wiki/Building for more details on windows run-time compilation.

Developing in Blender and using the gamekit runtime on windows provides numerous advantages, such as quick turnaround when you would like to check small changes to the game scene or character, and Visual Studio debugger if you require debugging the gamekit code itself.

Fishing Game

The goal of developing the fishing game was to have a proof of concept of using tablet and the 3D stylus for gaming. But as all games this also required a game to have a acceptable graphics and a certain level of playability. This required designing a game characters and a full scale environment. It also allowed to experience benefits and limitations of the development tools in use. (In particular Blender)

Scene and Character Design

Game design included one main character - the fish. The game should have multiple fish swimming in the pool which behaviour and appearance resembles a real fish. To achieve this two major steps were required: creating the fish character model and adding the logic which will controls its behaviour. The following sections cover creation of the fish character, but the same can be applied to other parts of the water scene, such as algae.

A game scene rendered in Blender

Creating a mesh in Blender - modeling

The fish model was created completely by hand based on the images of the Blue Surgeon fish The beldner tutorial for creating similar character was used as a bsis for the process: http://www.linuxforu.com/2011/11/learn-to-animate-with-blender-part-1-modelling-the-goldfish/

First stage included creating a rough mesh as described in tutorial and refining the mesh later to achieve a fish like appearance.

The Mesh of the fish character and the rendered body in Blender

Applying Texture

In order to achieve realistic appearance the model must be textured with the realistic fish image. Blender provides great facilities to achieve this task. The real image of the Blue Surgion was used as a texture and the main challenge was to achieve a correct mapping of the image onto the mesh. Gamekit requires the mesh to have texture mapping coordinates on every vertex. Using blender U/V image Editor it is possible to quite easily map the mash onto the texture image visually and then assign the mapping values for each of the vertices automatically. Note, that although Blender allows textures of any size, gamekit requires to have only 512x512 (or any other power of two) textures only. This limitation can be a source of a great inconvenience while developing a game, as no error is reported, but rather the texture is not drawn. Closer inspection of the gamekit logs reveals that image with an incorrect dimensions was loaded. Also the setting of the Blender material provides a lot of options which are irrelevant for the gamekit. Sadly there is no way to know which one works and which one don’t, but by pure trial and error.

The Fish mesh and the image with the mesh mapped onto it.

Blender rednered 3D image of the with the texture


Character Animations with Armature

In order to animate the Fish Blender armature mechanism can be used. In summary a skeleton is created consistent of bones (Armature). then the character is parented to the armature and each vertex is given a weight of influence for the specific relevant bone. While the armature and bone creation is in general an simple process, weight painting is quite time consuming and hard to make perfect. (Though Blender weight painting mode is a great help) The fish (and algae) character in the game are all animated using the armature method. Each bone of the armature is then positioned in the correct position and key frame is recorded (using the Blender action view). There are plenty of tutorials describing Blender animation, however gamekit provides very limited and often completely different support for armature animation than an average Blender user would expect. For example, parenting requires the parented object positioning to be “applied” in Blender, if not done correctly it will cause an unpredictable results in animation, such as unwanted deformations. More, key frames are working best when all the bone transformations are keyed at once. All this requires a lot of trial and error and makes it hard to create complex animations in Blender without breaking some of the rules which are used by gamekit.

Bone armature of the fish. Bone transformation also deforms the mesh.

Weight painting in Blender for the specific bone.
Mesh vertices affected by the bone are in red/

Game Logic and Lua

One of the major parts of every game engine is the game logic. In BGE (Blender Game Engine) the concept of the logic bricks serves as a basis, and more complex actions are performed by Python scripts. In gamekit the concept is the same, but the init script required to be present and as in BGE logic bricks are supported (in a more limited way) and any action beyond the logic bricks had to be done directly in scripts. In case of the gamekit Lua is used as a scripting language.

Basic Game logic with bricks

Gamekit support most kind of the logic bricks just as in BGE, however not all options are active, and some will have slightly different effect on the game. Again this can be learned only by trial and error. An example for the brick logic in the game is an algae animation: when the scene is loaded the animation started and goes forever. This is shown by the brick logic example below:

Although brick logic seems very convenient as it does not require coding and visualizes the logic it is almost always failing to achieve more complex actions, beyond simple responses to the timer events user input events. In that case Lua script is used to enrich logic behaviour and allow performing complex actions in scripts. Although Lua script can be executed directly by the init script, sometimes it is convenient to execute as a result of the brick logic. For example:

Script to detect fishing hook and fish mouth proximity based on radar sensor

Physics usage for collisions

Ability to use built in physics engine is a great help help during the game design. In the fishing game physics engine plays an important role in preventing fish from colliding one with another and also providing invisible walls. Though other methods are available to prevent fish from colliding, physics engine allows the behaviour of the fish to be much more natural. The usage of physics engine in the game also controls fish speed and angular momentum. Thus swimming fish behaviour is mostly automated by the physics engine, until a decision need to be made by the game logic.

Though bullet physics is used by BGE as well in gamekit there is a lot of differences in how the physics are handled. For example when the objects are parented there is no way of distinguishing collision of any of the objects, only the parent object is available for collision detection. This and other limitations and differences makes the usage of the collision and physics a very unnatural process for any user.

Fish with the cone of radar sensor and the collision boundaries.

LUA and advanced game scripting

In order to create a reasonable character behaviour the fish had to be responding to different situation in the game. This is achieved by introducing fish states. In each state the fish behaviour is well defined as well as transitions to other states.

Since Lua is much better environment to implement complex behaviours, the FSM class was implemented which provided all the needed transition logic based on the input from the brick logic sensors. Code example:


Fish = BaseClass(OgreKit.FSM)

function Fish:constructor(object,mouth_obj,hook_target)

        .......

          -- INITIAL

        self:setState(self.states.IDLE)

        -- IDLE TRANSITIONS

        self:addTransition(self.states.IDLE,    self.states.IDLE)

        self:addTransition(self.states.IDLE,    self.states.SWIM):when(self, Fish.IsRestless)

        self:addEvent(self.states.IDLE, self,   Fish.OnIdle)

        self:addStartTrigger(self.states.IDLE, self,   Fish.EnterIdle)

        self:addEndTrigger(self.states.IDLE, self,   Fish.LeaveIdle)

        -- SWIM TRANSITIONS

        self:addTransition(self.states.SWIM,    self.states.SWIM)

        self:addTransition(self.states.SWIM,    self.states.SEARCH_HOOK):when(self,Fish.HookIsNear)

        self:addTransition(self.states.SWIM,    self.states.IDLE):when(self, Fish.IsTired)

    self:addTransition(self.states.SWIM,    self.states.TURN,10):when(self, Fish.HasCollision)

        self:addEvent(self.states.SWIM, self,   Fish.OnSwim)

                ......

Along with the advantages Lua usage has a number of disadvantages. The most prominent is that BGE is using python and Lua is not a native part of the Blender environment. Thus no syntax highlighting and debugging is possible. Also runtime error reporting is really weak, especially for the scripts executed from within the logic bricks. This can cause hours of debugging just by turning things on and off line by line.

Connecting 3D input device to the gamekit

Design overview

The main challenge in connecting the 3D stylus to the gamekit is to propogate the infomation provided by Android OS to the lower (C++) layers of gamekit. This requires adding support for the new interface on all the layers of the application. The Java part fo the Android APK is the only place were the 3D data can be actually read. JNI layer is the glue between the Java and the C++ gamekit code, and the gamekit engine of course should be updated to support new kind of input.

gamekit APK software layers

Inside gamekit there is a OIS components responsible for providing the input from various devices to the gamekit engine. By extending the OIS component to support 3D stylus and updating relevant parts of gamekit engine to support feeding tha 3D data back to the scripting and logic layers we achive the goal of integrating new input device into the webkit code.

Android API

Android API in use is supported by the SDK API level 12: MotionEvent::GetAxisValue

This API provides direct readings from the stylus 3D sensor. The readings are then passed to the JNI layer of the game engine using a newly added sendStylus function.


Summary

In short, the experience of designing the game on top of the gamekit engine was not positive. Many of the function which were expected to work were broken, or required some tedious trial and error sessions to discover the correct usage of one feature or another. The best examples are limited animation support and completely different interpretation of the Blender settings for almost any aspect of gamekit.

Despite the above, gamekit is still the most comprehensive free 3D game engine available for Android, and at the end it was possible to create a working game prototype using gamekit alone. Below is the short list of pros and cons for using the gamekit engine for android 3D graphics or game development.

Pros:

  1. Free of charge, cross-platform game engine (including Android)
  2. Integrated into Blender
  3. Supports skeleton animations, particle systems and brick logic

Cons:

  1. Inferior to commercial game engines
  2. Too large and heavy
  3. Almost entirely undocumented
  4. Community is not large enough,
  5. Android parts require a major rewrite, at prototype level right now
  6. No sound support on android
  7. Sluggish performance
  8. Lots of bugs and/or unexpected behaviour while using Blender/Bullet data
  9. Very limited support for 2D on screen graphics for Android
  10. No shadows support on Android

Finally, gamekit aims to provide a capable open source and free cross-platform game engine. However due to its being a conglomerate of components which are not working well together it crumbles under its own weight and take huge amount of time to achieve even the most simple tasks. However, if no other option is available gamekit is capable to provide a framework for creating a 3D game or simulation at the cost of time spent in reverse engineering almost any aspect of it.

GAME OVER!