Mittwoch, 10. April 2013

Zusammenfassung meiner Arbeit mit und an der Blender Game Engine

Im Verlauf des letzten Semester durfte ich mit und an der Blender Game Engine austoben. Dies konnte man schon in meinen letzten Blogeinträgen lesen. Für das Fach Independent Coursework 2 habe ich nun mein Arbeit zusammengefasst. Hier ist die Dokumentation in PDF Form:
https://docs.google.com/file/d/0B-DgjzoFZxxwQmxyTGk2cTd0ZU0/edit?usp=sharing

Warum solltest ausgerechnet du dir dieses PDF angucken?
Wahrscheinlich eher weniger um zu sehen, was ich gemacht habe. Ein Großteil davon steht ja schon hier ausführlicher im Blog.

Falls du allerdings an der Blender Game Engine interessiert bist und einen kleinen Überblick erhaschen willst, dann könnte ein Blick in diese PDF dir vielleicht weiterhelfen. Nicht zuletzt wegen der Auswahl an Bildmaterial, die dich eventuell beeindrucken könnten ;-)


Viel Spaß beim Stöbern!

Donnerstag, 28. März 2013

Why I prefer "native" games

I like the movement of HTML5 and WebGL.
But I still prefer native gaming.
A simple argument: performance.
Just compare those two implementations.

BananaBread.
HTML5, JS, WebGL
 

Sauerbraten.
C++, OpenGL
http://sauerbraten.org/



I know that the comparison has some flaws. It's not the same map and not the same graphic settings. In BananaBread I choose "Low" settings and in Sauerbraten I have moderate settings.
So I have better quality and better FPS inside the "native" game.

Dienstag, 26. März 2013

[BGE, Qt5] Coding Blender Game Engine

The Blender Game Engine lacks features. But as it's Open Source you can improve it as you like.

I thought so myself and tried to do it. Here is a short log on what I did.

While working with BGE I found some points that could need improvement or would be nice to have implemented. Here is a little list I came up with:
- irgendwie müsste man auch Qt GUI Elemente in die GameEngine rendern und nutzen können.
- dyn. add&rem. lights

- custom glsl material node ?

Just some improvements that are floating through my mind:
→ multithreading
→ easy (especially on the documentation part) usage of dynamic libraries and low-level engine improvements written in other languages (particularly C++)
→ easy usage of non-python scripts/modules, like C++, Java, C#, ... (e.g. create a dyn. lib. and use it via C++-Script-/Module-Controller)
→ OpenGL 3+4 support
→ integration of a high-level shading language like surface-shaders from Unity3D or OpenShadingLanguage
→ code-completion in editor (e.g. bge python and shader stuff)
→ independent game window so you can tweak stuff in Blender
→ ability to tweak variables while game is running
→ shop integration for plugins and assets
→ integration of a version control system, version controlable blend-files
→ more detailed profiling
→ better lighting (I know it's being worked on in candy/harmony)
→ ability to export/deploy to different platforms and not just to yourself
→ better/easier system it insert and keep track of project-assets and prefabs/linked objects (also those that are not in the blend file but in project path)
→ hardware particle system (I heard that someone is also working on that)
→ special game screen layouts
→ game logic sets ( putting different SCAs in a set/group, loading and saving those, adding sets on different objects, having different sets on an object,,, e.g. loading a first person set on your main-char-object)
→ same 3D-view when I switch screen layouts
→ easy system to do shadow- and lightmapping for e.g. all statics objects in scene
→ easy/simple GUI system [seen that around here]
→ easy/simple multiplayer system (if not integrated then e.g. a simple system to talk to e.g. RakNet) [yeah, here is already flying something around]
→ (deployment of an HTML5 version)
...


Additions to the wish-list:
→ Controller to call binary modules written in C++/Java/C#/…
→ flag to module-controller to start module asynchronous/threaded
→ adding a module-controller that doesn't already exist will create that including code for bound in- and outlets
→ "Next" / "Follow-Up" Actuator & Sensor - combination
e.g. || S: OnCollision → C: CheckConditions → A: Next & S: Next + S: OnKey[W] → C: AND → A: Motion ||
the Actuator should have a flag to say if the follow-up should be evaluated in the same or the next round
→ "State"-Sensor: reacting on entity:: creation, initialization, awake, update, sleep, deinitialization, destruction(, …?)
→ Option for S,C,A to select on which object/objects to act on (like having a control-object to control many equal object, you don't wanna put bricks on all of them, you could put logic on the controller and then on options tell on which objects/groups to act/react on,
e.g. a group of fireflies, S:OnHit(any of group FireFlies) → C: And → A: Glow(the one) | or A: Glow(all) | or A: Death(pony) )

[I know that most of those stuff is already doable via SCAs and Python. But having it a bit cleaner and packed together seems nice.
Also I don't mind having different Sensors being able to do the same stuff. As long as it's easier to use/understand or more consistent.]

→ grouping of SCAs
→ selection of SCAs → drag & drop to other object → option to move or copy over
→ saving of SCAs into & creation of SCAs from code-files


→ ability to pause game → continue normally or do just one step
→ ability to change properties (especially when paused)
→ visualize "flow" of commands through SCAs ("which why what when")
→ profiling of SCAs
→ profiling of modules

- UV scrolling
- Texture nodes
- OSL (Open Shading Language)
- timer in shader nodes
- Particle system (mokazon)
- Hive node GUI
- change vars while running
- OpenGL 3/4
- Prefabs
- VCS support for blend
- PureData for sound effects?
- C++ modules to call from nodes
- dynamic/procedural textures
- Threading
- group logic-bricks (hive)
- profiling
- VBOs
- HW Skinning
- Team-Development - Connected via network, working on one file(?)


glGetUniformLocation
glUseProgram
http://www.opengl.org/sdk/docs/man3/xhtml/glUseProgram.xml
Currently the shader program is part of BL_Shader. And it is capsuled pretty good. So BL_Shader should be exchangeable. So some "setShader" would be nice inside KX_BlenderMaterial.

2) RAS_IPolyMaterial just has one texture. BL_Material has access to more. Same goes for KX_BlenderMaterial. As KX_BM is a RAS_IPM there might be a way to query the the texturename with a virtual method.

 3) Create a state that disables the output of those messages.



My last works with BGE mostly had something to do with shader:
[BGE+GLSL] Finding interest points
[BGE + GLSL] Py GL Rant
[BGE + GLSL] GLSL Shader Repository Addon

That's why I started digging around in the C++ BGE shader code.
To get into the code, and because I like to work with the new features of C++11, I refactored the code in BL_Shader.
Nothing special.
Replacing plain pointers with shared pointer.
Using constant variables and expressions instead of defines.
Strings and not char pointer.
Arrays, templates and 'move'ment against void pointer.

I like it much better in this way.
You can find the code here in the cpp11 branch:
https://bitbucket.org/Urfoex/blender/



The first real coding was the integration of a shader manager.
Having just some objects around it might not matter so much. But using lots of objects with a custom shader on each this might be helpful.
Currently a shader is placed on a material. Many objects with all having the same material will also have the same shader on it. But if you have many objects with unique materials but each using the same shader, each material will have its own shader.
10k objects with the same material → one shader
10k objects with different materials and same shader → 10k shader

Additional nice little feature: Every shader needs to be compiled and linked. And every time you do that you'll get some nice message lines saying that it worked, or not.  Getting 10k messages isn't nice.

That's why I made a little shader manager.
It is rather simple:
Each shader-program gets a unique name and is stored in a map. When you create a shader-program you can give it a name. If it is already taken then that one is returned, else it will give you a new shader-program. (A shader-program is a combination of vertex and fragment shader.)
The map contains a shared pointer to the shader-program. When you remove a shader from the material the shared pointer will be decreased automatically.
Normally closing the game engine would also remove all shader. But using the in Blender embedded game engine the shader manager will stay and the shader in it too. So I included a check if the reference counter got to a minimum, meaning no object is using the shader anymore, then the shader would be removed.
Still one little thing bothers me a bit:
if one object has a shader and is the only object having this shader and I remove it, the shader gets removed. When I want to used the shader on an object again, it needs to be recreated. A possible way around could be to "store" the shader in another material.
With the shader manager I also put methods to python to query e.g. 'useShader', 'hasSource' and 'hasUniform'.



From OpenGL 3.2 geometry shader are a core part. I have a graphics card that can use those. So my next move was to make geometry shader available in BGE. That in itself isn't that hard. Just like with vertex and fragment shader you create the shader object, load the code into it and link it to the shader program. Because I dislike duplicated code I made a little refactoring first. But as I tested my changes it didn't seem to work. Up to this point I still don't know why. I guess it's BGE and its buggy OpenGL code.
Just to check if my drivers and my code are OK I created a little OpenGL sample with Qt5:
Qt5 OpenGL GLSL Source Code


Vertex shader:
attribute vec4 qt_Vertex;
attribute vec4 qt_Color;

uniform mat4 qt_ModelViewProjectionMatrix;

void main(void)
{
    gl_Position = qt_ModelViewProjectionMatrix * qt_Vertex;
}
Geometry shader:
 #version 330

layout(triangles) in;
layout(triangle_strip, max_vertices = 6) out;

uniform float time;

void main(void)
{

    gl_Position = vec4(abs(time),0,0,1);
    EmitVertex();
    gl_Position = vec4(0,time,0,1);
    EmitVertex();
    gl_Position = vec4(0,0,time,1);
    EmitVertex();
    EndPrimitive();

    gl_Position = vec4(-abs(time),0,0,1);
    EmitVertex();
    gl_Position = vec4(0,-time,0,1);
    EmitVertex();
    gl_Position = vec4(0,0,time,1);
    EmitVertex();

    EndPrimitive();
}
Fragment shader:
void main(void)
{
    gl_FragColor = vec4(1,0,0,1);
}
Simple as that.
And as you can see, it works.
(Coding Qt5 is a nice thing. But you currently are mostly on your own. Tutorials aren't yet ported over (it seems). And to make geometry shader work I needed to use the older QGLShader classes instead of the new QOpenGLShader classes. The extra bits and pieces should arrive with Qt5.1 and 5.2. At least that is what the man sad:
QtDD12 - OpenGL with Qt 5 - Dr. Sean Harmer
QtDD12 - Modern Shader-based OpenGL Techniques - Dr. Sean Harmer
)

But in Blender I get this:
You see the missing cube?
That one has an active geometry shader.
The console output says: All is fine:
source/gameengine/Ketsji/BL_Shader.cpp:295:---- Vertex Shader Error ----
source/gameengine/Ketsji/BL_Shader.cpp:296:Vertex shader was successfully compiled to run on hardware.

source/gameengine/Ketsji/BL_Shader.cpp:295:---- Geometry Shader Error ----
source/gameengine/Ketsji/BL_Shader.cpp:296:Geometry shader was successfully compiled to run on hardware.

source/gameengine/Ketsji/BL_Shader.cpp:295:---- Fragment Shader Error ----
source/gameengine/Ketsji/BL_Shader.cpp:296:Fragment shader was successfully compiled to run on hardware.

source/gameengine/Ketsji/BL_Shader.cpp:363:---- GLSL Program ----
source/gameengine/Ketsji/BL_Shader.cpp:364:Vertex shader(s) linked, fragment shader(s) linked, geometry shader(s) linked.
Btw.:
First time I tried the geometry shader in BGE it just crashed.
http://blenderartists.org/forum/showthread.php?286152-FGLRX-vs-BGE-Geometry-Shader-on-Linux-on-ATI-SIGSEGV

Digging through code I saw a place saying: "Don't use displaylist with VBO"
As I thought BGE would use VBOs I looked in the property panel and found displaylists enabled. So with that on it wouldn't use VBOs.
(As far as I know VBOs should be currently the best way to get your vertex data to your graphics card: http://www.opengl.org/wiki/Vertex_Specification#Vertex_Buffer_Object)
But in the storage-section of the property-panel there is no VBO to choose from.
Just because someone disabled them –.–

Oh men!

OK. Uncommenting was easy. I re-enabled VBOs. But that didn't help. The object still didn't show up.
Why?
I don't know. Without any error messages it is really hard to figure out.
Because it is working with VBOs and Qt5 I would say that there is an implementation problem inside BGE.



If OpenGL is working better within Qt5 I thought about moving the BGE somehow in to Qt5. The first thing I wanted to try was to show a Qt window when pressing play and having both windows, the Blender one and the Qt one, show the game. "No problem" I thought. BGE uses GLEW for OpenGL access currently. To render to different windows I would need to switch the context. There is a special GLEW_MX version for that (http://glew.sourceforge.net/advanced.html).
So the first step was to replace GLEW with GLEW_MX.
Next step: integrate Qt5.
I found a spot where I thought it would be a good place to start the Qt window. I got it open. And closed. And closing Blender gave me a crash somewhere in the window handling. Even with that error I've put in the Qt window a simple OpenGL context and tried to show that.
Didn't work.
Qt tells you that you can't mix GLEW and Qt-OpengGL functions.
And my system hang up sometimes when trying to start the game.

So I stopped.



One way to make this work could be to put the blenderplayer in Qt.
But the code isn't nice.
I mean the code of BGE, Blender and the glue between.
I have nothing against C and C++. But please don't mix…
Finding the code entry where BGE starts was a heck of a search. You know that starting a C or C++ program has always something to do with a  main function. You'll find some. And one is the one for Blender and BGE. And it calls sweet C functions. I didn't find the right entry into BGE here.
There is the game start button in the properties panel. And you'll find some lines of code for that in a python file. But as it looks like it doesn't start the BGE. A suspicious looking string will get you to a C function. Still not starting anything.
Per luck I used a debugger, started the game and did a break in the debugger. There I saw a function history that lead me to the start of the BGE.

I'm not someone that wants classes and object orientation anywhere. But having easy traceable and capsuled code is nicer then this extern C mess.
The game engine itself is written in C++. It's easier to jump around and find what you are looking for. But the code is also a mix of C and C++. That makes it smell not so good. Having clean C++, especially with C++11, would be very nice.
And a documentation. Please document your code :-(
Walking around the code it wasn't that hard to follow the concepts. But all in all documented code looks much nicer.

As I said above the glue code is also mind bowing. For Qt and GLEW_MX I needed to change some CMake files. But what the heck is that?
They made macros for like everything. Changing GLEW to GLEW_MX was the easy part here. Just search for the string in all cmake files and replace it. But for Qt I needed to put in the finding and includes and linking and such. As I couldn't figure out where to really put it I just placed it where the corresponding executable would be build.



A Qt-OpenGL powered BGE would be really nice. I'd like to do things like this:

With a package like Qt it should be much easier to develop and extend a big application like Blender and BGE.
Not just easier access to OpenGL and shader. Also a very nice way to put GUI into your game. And probably instant portability to the major systems: Mac, Windows, Linux, Android and iOS (last two should arrive in Qt in the near future).



The funny thing about all this is:
You can still create great stuff with BGE.
Just look through some projects here:
http://blenderartists.org/forum/forumdisplay.php?34-Game-Engine

or search on http://www.google.com/videohp for Blender GE or BGE or game.

But also the engine needs an upgrade to 2013.
Really.
Look at http://devmaster.net/devdb/engines and you will see lots of open source game engines that perform much better then BGE.

Here are some:
http://www.garagegames.com/products/torque-3d
http://irrlicht.sourceforge.net/
http://www.crystalspace3d.org/main/Main_Page
http://sauerbraten.org/
http://softpixelengine.sourceforge.net/screenshots.html
http://www.cafu.de/
https://bitbucket.org/gongminmin/klayge
http://code.google.com/p/urho3d/
http://www.iddevnet.com/doom3/
https://github.com/id-Software/DOOM-3-BFG/
http://www.maratis3d.org/
http://gameplay3d.org/

The big winning points BGE currently have are:
→ very easy to use
→ nice integration with Blender (even when you can't take all things over 1 to 1)



I'd really like to see and help BGE grow…

Mittwoch, 20. März 2013

[BGE + GLSL] GLSL Shader Repository Addon

The Blender Game Engine can make use of shader written in GLSL. In the standard version you can easily use vertex and fragment shader for your objects and fragment shader for post-processing (also known as 2D Filter).

Simple way to add 2D Filter
Adding post-processing effects is very simple. In the logic brick section you'll find an Actuator called "Filter 2D" where you set up the filter you want to use. You can choose from pre-built filters or one of you own from a text file.

To just GLSL shader on your objects you need to do a bit more:
Use GLSL shader on object
To use a GLSL shader on your object you need to write a bit of python code.
My code does it in a very simple way. I assume that my object is just one mesh. Therefore I put the code only on the first mesh. A mesh can have different materials. Because I don't care I put my code on all attached materials. Currently I can only set the vertex and fragment shader code. That can come from a string defined somewhere or loaded into a string from some file. Because I like to keep code separate I have put the shader code into own files and load those into strings at the beginning of the function. You can also push some variables, also known as uniforms, to the shader. In my case the are a texture, some matrices and some floats. (Read more about those here: http://www.blender.org/documentation/blender_python_api_2_66a_release/bge.types.BL_Shader.html?highlight=shader#bge.types.BL_Shader)

Recent development in the harmony branch (https://svn.blender.org/svnroot/bf-blender/branches/ge_harmony/ ) is making using shader much easier, and also brings support for geometry shader.
Improvements inside the harmony branch
You don't need python code anymore. Just add a custom shader, choose its type, its source and you are ready to go.


A big problem remains:
Where do I get all those fancy shader from or do I have to code them all by myself?

Yes. And No.

For most things you don't need a custom shader. There are built-in post-processing shader and for materials and objects you can use the material nodes most of the time.
But there are cases where you need custom code. If you are lucky someone already posted the answer to your question on some forum, probably on http://blenderartists.org/ . Have fun searching…

There was a discussion about integrating more shader into Blender:
http://blenderartists.org/forum/showthread.php?283250-Shader-Trunk-Integration

And because I had time and an idea I made a little addon:
GLSL Shader Repository

Basically there is a repository at https://bitbucket.org/Urfoex/bge-shader/
It has some shader sorted in folders.
The addon "GLSLShaderRepository.py" can be downloaded from the top directory. (Click on the file in the source view: https://bitbucket.org/Urfoex/bge-shader/src and select RAW from top right. Then save it e.g. via File → "Save page as …" or by pushing Ctrl + S ) Inside Blender go to
File → User Preferences… → Addons → Install from File…
and select the fresh downloaded file GLSLShaderRepository.py
You should get something that looks like this:
Addon loaded and activated
 You don't need to change a thing. If you disable "Use ZIPped Version" the addon will try to download the files using mercurial. Now you'll find a new option under File → Import:

Import shader from repository
That will download all shader from the repository and modify some files. Because of the modification you need to restart Blender now. Just then the new options will appear. They will be in the editor under Templates:
Vertex shader
Fragment shader
Post Processing shader

Click on one of the shader and it gets loaded as an internal file.
So currently the post-processing shader are easiest to use with the Filter 2D Actuator. 

Most of those shader I found on http://blenderartists.org/
I haven't tried them so they might not work. Please post issues (best on the repository site) so they can be fixed.

It would be very nice if you could also give me more shader to put in the repository. There is also a reference folder in the repository containing lots of shader code that would need some revision to make it compatible with BGE.


Now have fun with shader!

Mittwoch, 6. März 2013

Humble Bundle for Android 5 on Debian 64Bit

There is currently a new "Humble Bundle for Android 5" at https://www.humblebundle.com/

You can get:
  • Beat Hazard Ultra
  • Dynamite Jack
  • Solar 2
  • NightSky HD
  • Super Hexagon
  • Dungeon Defenders

Dynamite Jack:
It started fine. But no sound was there. Reading the output from starting the game on the terminal I saw a line about a missing:
/usr/lib32/alsa-lib/libasound_module_pcm_pulse.so
There is a "LINUX.txt" coming with Dynamite Jack telling something about Alsa. It wasn't exactly what I was looking for but it made me thinking.
The missing file is normally there:
apt-file search libasound_module_pcm_pulse.so
libasound2-plugins: /usr/lib/x86_64-linux-gnu/alsa-lib/libasound_module_pcm_pulse.so
Multiarch. As you see. So the corresponding bit for 32Bit is
libasound2-plugins:i386: /usr/lib/i386-linux-gnu/alsa-lib/libasound_module_pcm_pulse.so
Making a link to it like this:
  sudo ln -s /usr/lib/i386-linux-gnu/alsa-lib/ /usr/lib32/
should do the trick. (Be sure to have libasound2-plugins:i386 installed. If you don't know then just do a: sudo aptitude install libasound2-plugins:i386 )
It worked for me. And with that I had sound.



Solar 2:
Worked. No problems.



Beat Hazard Ultra:
Hello libc ...
./BeatHazard
./all/BeatHazard_Linux2: /lib/i386-linux-gnu/i686/cmov/libc.so.6: version `GLIBC_2.15' not found (required by ./all/hge_lib/liballegro.so.5.0)
On Debian  you currently just have 2.11 and 2.13 if you walk on the save side:
http://packages.debian.org/search?keywords=libc6&searchon=names&suite=all&section=all

On the experimental path there actually is a 2.17 available. I haven't tried that yet.
So you have some options now:
  1. Install from experimental. But don't blame people if your system doesn't work right after that move. It's not called experimental just for fun ;-)
  2. Get working libc6 from somewhere and do the LD_LIBRARY_PATH dance.
I took the second step because I already had that setup. Steam got the same libc6 problem but luckily someone made a little script :
http://steamcommunity.com/app/221410/discussions/0/882965118613928324/

After that somewhere around:
~/.local/share/Steam/ubuntu12_32/
the file should be.
You can also download the libc6 e.g. from here:
http://packages.debian.org/experimental/i386/libc6/download
and extract it.
Either way you should have a valid libc6 with you.
Then you can start telling the game where that file should be:
LD_LIBRARY_PATH=~/.local/share/Steam/ubuntu12_32/ ./BeatHazard

Super Hexagon:
libc6 again.
./SuperHexagon
./x86_64/superhexagon.x86_64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by ./x86_64/superhexagon.x86_64)
But this time there might be just one option. You need to upgrade your libc6. I tried using a newer libc6 with the LD_LIBRARY_PATH but it didn't work. It then just gives me different errors like this one:
./x86_64/superhexagon.x86_64: error while loading shared libraries: __vdso_time: invalid mode for dlopen(): Invalid argument
OK.
You have another option: Use the 32Bit version.
With that you can do the trick as with Beat Hazard. For me I start the game with this line:
LD_LIBRARY_PATH=~/.local/share/Steam/ubuntu12_32/:x86/ ./x86/superhexagon.x86

Nightsky HD:
Should it still be the same version as coming with Humble Indie Bundle 4 then it should be no problem.


Dungeon Defenders:
I just tried the version of Humble Indie Bundle 7 and that didn't work, giving me:
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
      after 207 requests (207 known processed) with 0 events remaining.
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
      after 18 requests (18 known processed) with 0 events remaining.
Google helped finding this:
The problem is because the launcher tries to find
/usr/bin/gnome-screensaver-command and fails after it.

So the actual fix for this problem if making the launcher doesn't find it or
continues if it was not found, as obviously a lot of us are not gnome users..

For a workaround, just use this command:
# ln -s /bin/true /usr/bin/gnome-screensaver-command

source: phoronix forun
http://forums.trendyent.com/showthread.php?88020-Linux-Version-XIO-Error&s=95cddeb9959b8a65de2ca62e06afb852&p=746999&viewfull=1#post746999

Really?
Seems to be a bad joke.
But it helps.

I could play a bit.

I guess the current version on HBfA5 is a bit better. And there is an update coming:
https://twitter.com/humblesupport/status/309017063173804033

Hopefully the Steam-version works soon too. I don't like to pull the 5GB for each nasty update...



So 2 out of 6 are working out of the box. Seeing those other being just minor fixable problems is good. Hopefully they patch those but I guess that they won't fix the libc6 issues. Just like "it works on Ubuntu LTS - have fun". At least we have some ways to still get the games working.



Have a nice time playing!

Donnerstag, 28. Februar 2013

[SVN + Mercurial] Mirroring and using an SVN repository with Mercurial

For trying some code changes in the Blender Game Engine I needed its code. It is accessible via SVN from https://svn.blender.org/svnroot/bf-blender/trunk/
With Git and Mercurial around I'm used to do local changes, commit them locally and maybe undo them if I made mistakes. With SVN this is not possible.

Why do they still use SVN?
http://wiki.blender.org/index.php/Dev:Doc/FAQ#Why_not_switch_to_distributed_version_control.3F_.28git.2Fmercurial.2Fbazaar.29

There is even a discussion on the mailinglist:
http://lists.blender.org/pipermail/bf-committers/2011-May/032082.html

And?
Still SVN.
The Git mirror mentioned in the Wiki seems outdated.
But there are some unofficial ones on github:
https://github.com/ldo/blender-main
https://github.com/nicholasbishop/blender/

But compiling them gives you messages about missing locale, addons and addons_contrib. Those folders are normally pulled via SVN externals. So if you are using the Git mirror you still need to get those extra folders from somewhere. A way might be Git-Submodules.

Looking around I found hgsubversion: http://mercurial.selenic.com/wiki/HgSubversion
A simple one:
  • add hgsubversion to your extensions list
  • hg clone svn+repository
You should have plenty of time or a PC in the back to do that job because it really takes a long time for big repositories. It reads like every SVN commit and puts it into the Mercurial repository. You will have a nice version history afterwards.
The same I did for
"Locale" did again take a long time because most files in it are binary. That's why in the end it had a size of about 300MB. The SVN co of it is just about 50MB.
In the meantime I pushed the repositories to new created repositories on https://bitbucket.org/

After that I had the following:
The next step was to include the former SVN externals via Mercurial subrepositories http://mercurial.selenic.com/wiki/Subrepository
Just create a .hgsubs file in your hg root folder and insert the appropriate lines. Just like this:
release/datafiles/locale = [svn]https://svn.blender.org/svnroot/bf-translations/trunk/locale
release/scripts/addons = https://bitbucket.org/Urfoex/addons
release/scripts/addons_contrib = https://bitbucket.org/Urfoex/addons_contrib
You can use Mercurial, Git and Subversion repositories here.
I'm not sure right now if I had to checkout / clone those repositories myself for it to initially work. At least for SVN HG complained about locale "is not a working copy". But right now if I clone my repository and go to the right branch Mercurial pulls all subrepositories in without my help. It works just like with SVN and SVN externals.


Sounds easy so far.

Looking at the commit graph shows that I had some problems.
I simply can't delete branches. I made some to test if I can put some commits in the branch and then put commits from default to it to, like "rebase". Didn't really work as I thought and now I have some branches "just because". This also includes some commits that I made that I can't remove. Not reverting. That works. But real deletion.
Branches could be removed in 3 ways: http://mercurial.selenic.com/wiki/PruningDeadBranches
None worked for me. I finally merged and closed them but they are still there.  The clone-method somehow didn't work with hgsubversion. It technically worked but doing a "hg pull" that normally pulled new SVN changes in resulted in an error.
Another commit and another branch was just locally. "hg strip" did fine to remove those. (http://webcache.googleusercontent.com/search?q=cache:jYnn0rwFlkIJ:mercurial.selenic.com/wiki/EditingHistory+&cd=2&hl=de&ct=clnk&gl=de)
Somehow I managed to accidentally put a commit in the default branch. And hgsubversion created a new head for the new SVN incomings. Now I have two heads for the default branch and don't now how to safely change that. Every time I pull the changes from SVN in it create a new head. If I don't mind I can push that and have two heads. Or I merge those two heads together. That creates a commit and looks a bit ugly. But at least the last commit will unsure that the specific subrepo-changes are in.

One thing I would like to know is: What happens when my local hgsubversion repository that I use to pull SVN changes and push them to Mercurial gets deleted? How do I then push SVN changes? Can I simply create a new local hgsubversion repository and my pushes will work on top?


At least it works for now.

Btw.: If you gonna push to a different path then you pull you can tell hg to do so in the .hg/hgrc file. Just like this:
[paths]
default = svn+https://svn.blender.org/svnroot/bf-blender/trunk/blender
default-push = https://bitbucket.org/Urfoex/blender

I also create a very simple script for updating the  repositories:
$ cat update.sh
#!/bin/sh
echo "+==============================================+"
echo "-------------------- ADDONS --------------------"
echo "+==============================================+"
cd addons
hg pull
hg push
cd ..

echo "+==============================================+"
echo "---------------- ADDONS_CONTRIB ----------------"
echo "+==============================================+"
cd addons_contrib
hg pull
hg push
cd ..

echo "+==============================================+"
echo "------------------ BLENDER-HG ------------------"
echo "+==============================================+"
cd blender-hg
hg pull
echo "+==============================================+"
echo "!! Don't forget to merge changes for subrepos !!"
echo "+==============================================+"
hg push
cd ..

echo "+==============================================+"
echo "----------------- BL_HG_CLONE ------------------"
echo "+==============================================+"
cd bl-hg-clone
hg pull
hg push
cd ..

echo "+==============================================+"
echo "-------------------- LOCALE --------------------"
echo "+==============================================+"
cd locale
hg pull
hg push
cd ..

:-D
Very simple.


In the end it is possible to move a SVN repository to Mercurial or just use one that way. Moving it over is a nice opportunity.
But using it as a wrapper is a bit painful. It feels much better with just Mercurial or just Git.


Back to changing BGE code…

Dienstag, 26. Februar 2013

[BGE + GLSL] Py GL Rant

Let's do this.

Blender Game Engine.
It is nice.

But it also has its problems. Just like everything else.
This time I had a hard time doing some GLSL shaders. Not the normal ones.

GLSL viewer
Just for testing I did a little GLSL viewer in BGE:
https://docs.google.com/file/d/0B-DgjzoFZxxwV1lhZ20ySTFrZzQ/edit?usp=sharing

Just edit the simple.fs as you like. The code inside is taken from http://glsl.heroku.com/e#6952.0

You can find more nice codes here:
http://glsl.heroku.com/
https://www.shadertoy.com/


But with this little sample I got a problem:
How do I find out if an uniform is active?

Answer: glGetUniformLocation
http://www.opengl.org/sdk/docs/man3/xhtml/glGetUniformLocation.xml
This should return -1 if the uniform isn't active.
The function is not in the python api.
So currently I have no choice but to just pass all uniforms I know and get lots of messages about some not being in the shader.


RenderToTexture
Here I tried using the RenderToTexture feature to create a texture for an object on the fly:
https://docs.google.com/file/d/0B-DgjzoFZxxwYzd6TjcxVS1zQWs/edit?usp=sharing

Short answer: I couldn't get it working.
The problems:
1) How do I switch between shaders without recreating them all the time?
2) How do I use another texture to render on then just the first?
3) How can I hide shader compile messages?

Don't wanna talk about the performance issue…
Or that it won't work as I thought it could.

So to get some answers:
1) glUseProgram
http://www.opengl.org/sdk/docs/man3/xhtml/glUseProgram.xml
Currently the shader program is part of BL_Shader. And it is capsuled pretty good. So BL_Shader should be exchangeable. So some "setShader" would be nice inside KX_BlenderMaterial.

2) RAS_IPolyMaterial just has one texture. BL_Material has access to more. Same goes for KX_BlenderMaterial. As KX_BM is a RAS_IPM there might be a way to query the the texturename with a virtual method.

3) Create a state that disables the output of those messages.


Framebuffer objects
My first intend was to improve the interest detection I talked about last time [http://urfoex.blogspot.de/2013/02/bgeglsl-finding-interest-points.html]. Therefor I thought about using framebuffer objects.
http://www.songho.ca/opengl/gl_fbo.html
http://www.opengl.org/wiki/Framebuffer_Object

Simply said it is the same as RenderToTexture. You create a buffer and then render to it instead of to the screen. When done you can just the buffer e.g. as a texture for the real screen rendering.
It is not the thing I had hoped for but at least it is something that could have helped a bit. My real intention was to have a buffer in GLSL that I could read and write to. The buffer should be on GPU. The shader could access and modify it each round.
The framebuffer could do the trick. The steps would be the following:
  1. create as much framebuffers as needed
  2. bind first fb, bind shader for first fb
  3. render just the object to the fb
  4. repeat step 2 and 3 until all framebuffers are used, consider using previous fbs as samplers
  5. set shaders, fb-samplers and other things for real rendering
  6. for each following rendering: skip step 1
(Idea: That could also help to create dynamic textures. )

Here are my first tries:
https://docs.google.com/file/d/0B-DgjzoFZxxwNVpMT2dRQW5SZHM/edit?usp=sharing
https://docs.google.com/file/d/0B-DgjzoFZxxwbW43bnJPRTE2Q1E/edit?usp=sharing
https://docs.google.com/file/d/0B-DgjzoFZxxwaXpETU5MeDMzR0k/edit?usp=sharing

They don't work.
Why?
I'm not quite sure. Using PyOpenGL is simple. But somehow it is not.
Or at least not in combination with BGE.

I had found a sample on how to include it:
http://www.blender.org/documentation/blender_python_api_2_60_1/bge.types.html?highlight=kx_polygonmaterial#bge.types.KX_PolygonMaterial

But "yeahy!" that just is for single-texture-mode (see http://blenderartists.org/forum/showthread.php?277025-KX_PolygonMater*ial-and-KX_BlenderMater*ial-question&highlight=KX_PolygonMaterial).

Writing some code I could get some framebuffer-code but for using the FB afterwards I would need to set it as a sampler. But how to do this without proper access the the shader? By building the shader via PyOpenGL. If it would give me more then just an error:
"Shader.py", line 68, in load
  File "/usr/local/lib/python3.3/dist-packages/PyOpenGL-3.0.2-py3.3.egg/OpenGL/GL/shaders.py", line 220, in compileShader
    glCompileShader( shader )
  File "/usr/local/lib/python3.3/dist-packages/PyOpenGL-3.0.2-py3.3.egg/OpenGL/latebind.py", line 41, in __call__
    return self._finalCall( *args, **named )
  File "/usr/local/lib/python3.3/dist-packages/PyOpenGL-3.0.2-py3.3.egg/OpenGL/GL/VERSION/GL_2_0.py", line 137, in GLSLCheckError
    description= glGetInfoLog( cArguments[0] )
OpenGL.error.GLError: GLError(
        description = b"Vertex shader failed to compile wit...,
        baseOperation = glCompileShader,
        cArguments = (14,)
)
So no PYOGL shader for me.


Afterthoughts
It's a pain if you just want to code and get things done and it just doesn't work as you want. Python moves forward but its modules doesn't follow. PyOpenGL just has experimental support for Python 3.2. Especially its acceleration support isn't working with Python 3.3. On the other hand: OpenGL has moved to version 4.3 over a long period. But you can't just use it via Python in the BGE. It's not even in the C++ code.
Looking at those problems I have they mostly arise from trying to code only using Python. Extending the engine would do the trick and might help others too.

Most time in game making you probably won't notice those problems I had. Shaders don't change on the fly. RenderToTexture is expensive and so used just like one or to times for e.g. mirrors.
Most shaders from e.g. shader-site like those above simply work inside the BGE. As far as I can see most of them should work too:
https://github.com/id-Software/DOOM-3-BFG/tree/master/base/renderprogs

One thing that could also be easily implemented, but for now just is a bit annoying: creating an own GLSL shader.
It's not the shader itself but the corresponding lighting and shadowing and such things. If you use material-nodes to do your tricks everything is just fine. But if you create your shader you loose everything for the corresponding object or better 'material'. Every extra like light and shadow you need to insert for yourself.
As an idea I thought about providing the main shader that should come with BGE and having some includes and functions in there. Via python you could set the right file to include and so the right function to call.

E.g. something like this:
mainFrag.glsl:
#include "mainShader.glsl"
#include "myShader.glsl"

out vec4 glFragColor;

void main(){
    vec4 diffuse = myDiffuseShader();
    vec4 specular = mySpecularShader();
    vec4 normal = myNormalShader();
    vec4 ambient = myAmbientShader();
    glFragColor = mainShader(diffuse, specular, normal, ambient);
}
myShader.glsl:
vec4  myDiffuseShader(){
    return vec4(0,0,0,1);
}
vec4  mySpecularShader(){
    return vec4(0,0,0,1);
}
vec4  myNormalShader(){
    return vec4(0,0,0,1);
}
vec4  myAmbientShader(){
    return vec4(0,0,0,1);
}

myOtherShader.glsl:
vec4  myDiffuseShader(){
    return vec4(1,0,0,1);
}
vec4  mySpecularShader(){
    return vec4(0,1,0,1);
}
vec4  myNormalShader(){
    return vec4(0,0,1,1);
}
vec4  myAmbientShader(){
    return vec4(0,0,0,1);
}
 myShader.py:
def useShader():
    fragShader = read(mainFrag.glsl)
    fragShader.replace("myShader.glsl", "myOtherShader.glsl")
    object.setShader(fragShader)
Shouldn't that work?
I think I try that another day.


Random rants:
→ nullptr instead of 0
→ no goto
→ shared_ptr instead of normal *
→ mercurial / git instead of svn
→ no "spit"ting around. usage of clog and cerror or even better logging
→ newest Python 3.3 with nearly no module working but really old OpenGL
→ C++ code that needs refactoring