New Home

The Jade blog has a new home. Thanks to Kartones for giving us space on his own geek community. Point your links, RSS and everything else over there or you'll probably miss the big announcement we are going to do in the following days ;)

Jade Virtual File System (VFS)

The 13 September 2006 I posted in this blog:

"Nothing more I think, probably the next post will be about the new virtual file system of Jade."

And then the nexts posts were not about the VFS :p They were more reports about how the project is going. I'll do a quickly one and move on: 1.1 is really close, most features are there (collada is there, animation is there, new input is there, scripting is there, VFS is there,...). But polishing takes a lot of time :( But you can download the latest check-ins and play with all the new things ;)

And now about the VFS. Back to when I started thinking about this subject (severals months ago), I asked in Stratos if someone had already done it (just to avoid common mistakes and to get some guidelines). I got several very useful answers, and a link to a very good article in old Flipcode (here). I also got several good answers from Jade forums.

Most of my objectives for the VFS were pretty similar to those of the article:

- it should be able to handle archives (a big file with lots of files inside, like a .zip file)
- it should allow plugins to read or write in archives (encryption, compression,...)
- accessing normal files and archives should be transparent to the user
- it should allow several file paths

The idea was to write the new VFS and replace the old class of the engine that managed that (JPath). I had a more or less idea of what JPath was doing and I though switching from one thing to the other wouldn't be very hard (BIG mistake, more on this later) .

So, with that in mind, I started to code. I took the decision to leave the plugins for the end (not a great idea, but not too bad either) because I wanted to have first a bare minimum functionality. The main code was written pretty fast (it was quite easy), and were I spent most time was writing the tool to create the "archives" (a small winforms application). I was lucky that Fernando (delahermosa) was not too busy during that time and he helped me a lot in several parts of the tool.

The structure for the VFS at the start was like this:

- there was a JVFS class that represented the file system. The engine or the user would ask for files to this class. This class would hold internally a dictionary of FileSources (places where you can find files) and pass the request to them. One of the features from JPath that the engine used a lot was that you could ask for a file without knowing where it is exactly and then JPath would search for it.

- I wrote to FileSources implementations: HardDiskSources and StorageSources. HardDiskSources is normal IO, and StorageSources are the so called archives. A StorageSource holds virtual directories and files inside it.

- The engine used to read files the .NET methods called FromFile(string path,...), but now it would have to use the FromStream(Stream stream,...), because there´s not a real file when reading from an archive file. To allow this a new type of stream class is created: a VirtualStream. A virtual stream works over a file stream (the stream for the archive file) and controls that the reading is done the right way (you read the file you want to, and not more or less than you should). This way, the archive is only open once really and close at the end (so you avoid lot of costly IO operations).

And that was it more or less. It sounded great, but when I started merging the VFS with the engine, things weren't so great ;)

First, I decided to do the plugins support. And well, it was hell to write that. Plugins are small classes that modify how a file is written or read, and you can apply several of them to a file. This is their final interface:

Stream ApplyForRead(Stream input);
Stream ApplyForWrite(Stream input);

In theory it is pretty easy: most times filters will be streams wrapping streams. In practice it was quite complicated. Why? Things go more or less like this: we are writing our new archive (remember, lots of files together like a .zip), so we have one stream open and a binary writer writing on it. But now, when writing files with filters applied, we have to modify the stream the writer uses to write (filters modify the stream). So:

- we cache the position where we are in the stream
- we close our stream
- we reopen it and apply the filters to write
- we write the file
- we get the new position in the stream. Now, we can calculate the number of bytes we wrote :) We can't use the Length property of a file stream to know the number of bytes we wrote because we can be reading a compressed file or writing a compressed file, and compressed streams don't support the Length property (System.IO.Compression).
- we close the stream
- we reopen it again to continue doing work with it (file header, and other things)

Sure you loved it :p But then things get better. After finishing that part (I can tell you it took me a while to write those 40 lines of code), now I needed to give support for the filters in the visual tool. It involved writting a custom attribute so filters can tell the type that edits them in a visual environment (like VS visual designers) and a lot of strange errors with reflection and assembly loading.

And the last headache I got was when testing the encryption filters, because for some strange reason, when using AES, DES or TDES symmetric cyphers with PaddingMode.ISO10126 the output ended truncated. It took me a while to realize it was the algorithm padding, so well, I set it to PaddingMode.Zeros and at last, filters were working :)

And things got worse :p

Now I started replacing JPath for JVFS. I deprecated all the JPath code and started searching the warnings around to replace it. I had to change also some configuration stuff, but nothing too serious.

After looking how JPath was working I found the harsh reality: my JVFS interface wasn't prepared to manage it. JPath allowed the user to get all files from a directory, write a file (I didn't allow to write files! You can't save your game, you have to finish it at once :p), set special directories (like for example, Shaders directories where all shaders would be), ignore extensions,... Well, a lot of stuff I haven't thought. So I had to stop the merge and rework the VFS.

This rework involved mostly the FileSources interface. Before this change, the VFS had methods to ask for a file to a specific FileSource (all of them have an unique name) or to all of them at the same time. But now, the VFS class was reworked to be also a FileSource. So, I took out the methods to ask for a specific file or to a generic file. Now the method was something like this:

public abstract Stream GetFile(string path, string fileName, bool recurse, System.IO.FileAccess access);

So, if you do VFS.GetFile, the VFS implementation would search in all its FileSources, but if you do VFS["FileSourceName"].GetFile, it will be use whatever that specific FileSource implementation is. Everything is much clearer like that. FileSources also got some new methods (all files from one directory and minor things needed).

Next step was to allow for defined directories. The old way of putting directories for Jade was something like this:

X:\Whatever\InHouse
X:\Whatever\Base

InHouse is for internal engine files. Base is for your game files. But the engine assumes some directories always exist. For example, it loads all shaders in the Shaders defined directory, that is that the engine searches for InHouse\Shaders and Base\Shaders. So I had to allow a way to get all files from a specific path from all the FileSources. And the defined directories where born.

A defined directory is a pair of key (name, for example Shaders) and value (path, for example MyWork\Shaders). What the engine does when it finds a request to a defined directory is to ask every FileSource if the defined directory key exists, and if it does exist, it performs a normal search using the path associated to that key. So every user can put the Shaders wherever he likes the most as long as the key is Shaders.

All of this is defined in the configuration file, like this:

[VFS]
[FilesSource Type="HardDisk" Name="Base" Path="../../../../Base"]
[DefinedPath Name="Materials" Path="Materials"/]
[DefinedPath Name="Models" Path="Models"/]
[DefinedPath Name="Particles" Path="Particles "/]
[DefinedPath Name="Scenes" Path="Scenes"/]
[DefinedPath Name="Textures" Path="Textures"/]
[/FilesSource]
[FilesSource Type="HardDisk" Name="InHouse" Path="../../../../InHouse"]
[DefinedPath Name="Effects" Path="Effects"/]
[DefinedPath Name="PostProcess" Path="Effects/PostProcess"/]
[DefinedPath Name="Shaders" Path="Shaders"/]
[DefinedPath Name="Geometry" Path="Shaders/Geometry"/]
[DefinedPath Name="Internal" Path="Shaders/Internal"/]
[DefinedPath Name="Lighting" Path="Shaders/Lighting"/]
[DefinedPath Name="Materials" Path="Materials"/]
[DefinedPath Name="Textures" Path="Textures"/]
[DefinedPath Name="Gizmos" Path="Textures/Gizmos"/]
[/FilesSource]
[/VFS]

(Note: it's a normal XML File, I just replaced it with brackets [] to avoid some formatting problems here in the blog)

Quite easy: you define a FilesSource, with its type (HardDisk is normal files, Storage is an archive), its name and its path (relative to the configuration.xml file or absolute, as you wish). And then inside it you can add defined paths with their name and their real path.

With everything done, I went back to replace JPath for the VFS again: now things were fitting much better as there weren't strange things around that I hadn't thought. But, when I finished, the engine didn't work at all :( It would load and start, but it would not render correctly, probably related to a shaders loading problem, but I was unable to find it, so in the end I had to go back and undo all the changes in the engine (I backed up everything first). See changesets 14554, 14572 and 14574 comments to see what I mean (I broke the build compilation on the first check-in).

And so, there I'm at the moment, trying (for the 3rd time) to replace JPath for JVFS, but this time, instead of changing all at the same time, I'm going to have both systems living together and then I'll remove JPath little by little.

Let's hope this time it works ;)

Edit: I forgot to say (and I shouldn't have) that all of this work was possible thanks to the support, advice and comments from Jader, Reed, Quimbo and delahermosa. They helped me a lot to catch bugs, write some parts of the code, test, ideas,... Thks team! ;)

Getting closer to 1.1

We are getting closer and closer to 1.1. Jader is hunting bugs in the engine to help Gonzo with a new planet tutorial. It will be quite a complete tutorial that will show you step by step how to use Jade to render a full planet. This is an early screenshot:


I´m aware it looks more like a hedgedog right now ;) But it will get prettier with some effects and shaders that are coming.

Meeshoo continues with his Collada work, and he has being able to import information to the engine from Collada, and now is working in the animation part (importing the Collada data and then playing the animations correctly).

The JadEd team has finished most of the core services of JadEd (plugin services, messaging services, undo/redo services,...). They are now going to start wrapping the engine so JadEd can talk to Jade and preparing the UI to work with Jade. They have being a little out during this last days, but they are back to work.

And myself I have finished the VFS once for all and uploaded to the main code. The VFS and the Storage Packager tool are now functional and working. The next step is to replace the JPath class for the VFS class to handle file requests. This change will be a little more heavy than expected in the end because I plan to change another thing also. Usually, when people get to try Jade for the first time, the "strange" directory structure we use gives them some problems to setup the engine and start using it.

So, as I have to change the JPath class (the one that was in charge of this feature also), I'm going to change how the engine starts up: you'll be able to define paths for files in the configuration.xml file, so it will be way easier to start playing with the engine and change resource locations (no need to compile anymore).

After this is ended, I´ll write two different tutorials, once about how the VFS internals (more oriented to understand how it works, how to extend it,...) and another about using the VFS.

And not much more, there are some interesting threads in the boards that maybe you should read if you haven´t done it: Jade GUI, DX10 vx XNA,... Check them, because they are the ones that will decide where Jade will be headed in the future.

Have a nice x-mas!

Advance Forward - Jade and JadEd

Long time since last post, and lots of things happened along the way :)

Jade 1.1 it´s coming out nicely, and it´s going to package more features than expected. At the moment the Jade binary format is completed and Jader has moved to tweak with the engine internals: his changes now allow to support multiple render views and scenes. Also, he did some changes in the engine shaders.

The collade import is advancing at a very good rate, with most of the import now finished (meeshoo is now giving it finishing touches and testing). Animation should closely come after he finishes with this.

The new filesystem is finished, except for the filters and total integration, but it´s working and the packager tool performs it work too.

The only point we haven´t being able to touch yet is the buffered input, that part will come probably after all the other features are finished. But on the other hand we are getting a very nice new mathematic library from Reed, who has finished several classes for interpolation and geometry operations. He also cleaned most warnings from the development code, so it´s much more neat now :)

But one of the most exciting things that have being happening to Jade in these last days are related to the Jad Engine Editor, codenamed JadEd (we are so clever with word jokes :p). After several people expressed their interest in JadEd, we have decided to create a new project in codeplex:

http://www.codeplex.com/Wiki/View.aspx?ProjectName=jaded

This project will be exclusively for JadEd, and it´s mainly managed by gonzo, Quimbo, Reed and matias. There´s some code now available to download, and I must admit that the level of architectural design they are giving to it will make it a very good piece of software. If you don´t believe me, you can join them at the new JadEd forum:

http://forums.jadengine.de/forumdisplay.php?fid=16

Also, if you have read the latest news in the boards, you will probably know that we have now some IRC channels where you´ll be able to chat with us and see how the development goes. You can grab the info at the JadEngine webpage (www.jadengine.com).

Hope to see you on IRC!

Moving towards Jade 1.1

Today the Jade Team announced the new features that will come with Jade 1.1. The new features are:

- Jade binary format: it will be able to load and save a full scene (scenegraph, meshobjects, meshes, materials, physics & animation).
- Collada converter: it will be able to convert collada exported scenes into Jade binary format with all the information needed to build a Jade scene. It will be a tool for the engine.
- Animation system: it will support morphing and skeletal animation.
- Virtual file system: it will be able to load files from real paths and virtual paths (storage files). It will come with a tool to create storage files.
- Input: buffered input and action based input (through scripting or another solution, not clear yet).

I must admit I´m very happy with the new features that are coming to the engine, as most people seems to want animation and importing/exporting facilities (and with a good reason, the engine is much less usable without them). Also, a lot of minor improvements are coming for all parts of the engine: refactoring, comenting, editing,... To be truth, with 1.1 I really hope more people will start considering Jade for their own game development needs using .NET.

The importance of profiling

We love to optmize code, all programmers do. We find joy and pleasure in writing more optimal and faster algorithms and routines. The problem is that most times, we just do it and think that the new code will be better, without taking measures or doing some tests to confirm that the new code is in fact faster than the old one.

Since the last update, I´ve spent most of my time profiling Jade. While I´m not able to make architectural changes to improve performance, I can find some functions that are not well written and improve them. I´ve had a lot of fun fighting with the Hashtable and Dictionary classes.

When I did my first profiles, I found two dark points:

- JEffectBuilder
- Building shadow volumes in the light pass

Both parts were using a lot of time calling several methods of the Hashtable class (Contains, [], Add,...). I started looking first at the JEffectBuilder class. There the Hashtable was declared as:

Hashtable effects = new Hashtable;

I browsed the code to search what it was using as keys and values: it was using JUniqueEffectID (struct) for the keys and JEffect (a class) for the values.

My first move was to change it to the Dictionary class. I like a lot generics and generic collections (they save casts), so a Dictionary should work better than a Hashtable. I also changed the keys from JUniqueEffectID to int because that struct has a field called Global that is like a hash value. It wasn´t going to save me from the call to GetHashCode, but well, I felt it was more natural to put it that way.

I profiled again the code and well, I was amazed: the time used by the JEffectBuilder diminished a lot. I was very happy with this, but I couldn´t believe that changing from Hashtable to Dictionary would yield such a great speed improvement. I rushed to do the same change in the shadow volumes, because they were well... slow as hell to say the least.

I re-ran the profiler again, and I was shocked by the results: the shadow volumes were as slow as usual. I started to think what had happened, because I was really surprised that I had changed the Hashtable of the shadow volumes to a Dictionary and the speed was the same... And then I realized: the problem was in the GetHashCode method. I don´t know the implementation, but I´m pretty sure that calculating the hash code of a struct is a lot slower than calculating the code of an int. That´s why changing to int keys from structs gave me such a big improvement in speed in the JEffectBuilder while in the shadow volumes, as the keys were already int values, I didn´t notice the boost.

I then had another strange result: Dictionaries have a method called TryGetValue. In the .NET 2.0 framework there are a lot of methods like that (like TryParse in float, int, double,...) that help to make code faster and easier to read. In Jade, the TryGetValue would save me from calling first Contains to see if a key existed in the dictionary, and then using the indexer to get the value. I was reducing the number of calls in half :)

I did some test projects, and they confirmed my idea: using TryGetValue was nearly 2 times faster than using Contains and the Indexer. But then the strange thing happened: when implemented in Jade, it was slower :(

I will continue studying this result, because it doesn´t make any sense at all (or I haven´t found it yet). I must admit it can be also that I haven´t run enough tests: my laptop heats quite fast when running Jade and then the performance of my ATI card drops a lot. I´ll investigate this one further.

Appart from my rambling in profiling and optimization, this last check in has several other aditions:

- documentation and translation fixes (seems they aren´t going to end any time soon).
- a third person camera class written by gonzo. Very nice addition for the default camera classes of Jade (and much more useful later in his editor JadEd).
- fixed some error messages in the code: the engine wasn´t reporting some errors as it should. Sweenie and timbann wrote some code that fixes that.
- fixed the dammed resize bug: Haddd found it at the end, after fighting with it since nearly the start of the engine coding. This will help a lot in the work of the editor.
- the old Haddd editor ported to Jade. Haddd and I ported it to the new code so people can use it as a testbench for their own experiments.

Nothing more I think, probably the next post will be about the new virtual file system of Jade. Or maybe about a surprise, but I don´t know where it´ll be ready (I´m not writting it ;)

Jade Comunity Contributions

One of the best ideas when thinking about Haddd future was converting it into an Open Source project. In very little time some people have submitted code in the form of new features or bug fixes that are very valuable for the engine future.

The first updates that have being merged in the engine code are:

- XBox360 controller and XInput wrapper by plonkman
- Jostick support by ThunderMusic
- JApplication with integrated states by meeshoo
- Reporting delegate on .haddd files loading by Ronnie Barnard
- Bug in input classes from MindGames

They are small features, but nice ones to be truth. And much more important things are on the works:

- Networking by ThunderMusic
- Collada support by meeshoo
- Viewer/Editor by gonzo

This projects when finished will allow Jade to be a much more rounded engine for game development and more useful for the community in general. So, that´s why I feel very happy having opened the code of Haddd, because if not, that code would have never appeared in Jade and no matter how small or big it is, it´s useful :)

Well, I said I was going to return to steering behaviors after this, but I´m going to move to profile the engine, I´ve seen very strange things in the performance, and I´m confident that some small changes can be done to improve it. The ideas is to rework some inner workings of the engine that will allow it to gain 10% speed more or less without breaking any interface or anything (so no method name or signature will change).

Let´s see what happens in the end, I´ll post some points that will be changed when I´m more sure that I´m right ;)