Re: File Access: cache-on-HDD, more

[Christian Reiniger <warewolf@mayn.de>]

Adrian Ratnapala wrote:

>> Another feature that more or less goes hand in hand with that is
>> transparent aliasing, i.e. aliasing file names/access paths at runtime,
>> e.g. to faciliate i18n or - to redirect accesses to cached files.
>
>Hmm, maybe I don't see what it has to do with i18n since users
>should be isolated from filenames (except for savegames). 

I'm thinking of game code like this:

switch (SelectedLanguage)
{
case lang_english: 
	ppFAliasPath ("sounds/speech/default/", "sounds/speech/english/");
        ppFAliasPath ("gfx/textures/withtext/default/",
                      "gfx/textures/withtext/english/");
 	break;
case lang_german:
	ppFAliasPath ("sounds/speech/default/", "sounds/speech/german/");
        ppFAliasPath ("gfx/textures/withtext/default/",
                      "gfx/textures/withtext/german/");
...
}

This would tell the PP file access code to redirect all matching ppFOpen ()
request to the localized files, allowing the game coders to use generic
paths throughout the game. I don't know if this is really the best
solution, however. (Of course this stuff won't be implemented via path
string pattern matching *grin* - some symlink thing for the internal lookup
tables could be an approach)

>hdd cache: should we be doing it or the app?

Hmmm, we don't know anything about the data being accessed, so caching
would have to be done via some hint system (ppFOpen ("myfile", "r",
cache_please);). Such a system is needed for mmapping etc anyway (see below
for more), so this shouldn't be the problem. This works with simple
cache-files-from-cd-on-hdd tasks, but not for cache-on-hdd-after-decode
(that would require a complexer system...).
Hmmmm, don't know if it's worth the effort...

>How is our pakfile going to help us with speed?  As I see it, it
>will mostly be acting as a fast directory cache.  We read the pakfile
>and store an in memory map of path<->offset.  Does this make a really
>big difference?  Should we be worrying more about things like the cache.
>
>I am out of my depth here, where can we get info about he performance
>effects of a PAK file, maybe they are just a hack to get around the 
>stupidity of FAT.

I bet they were a hack around FAT first, but now they'we moved to other
benefits - protecting game data from being ripped off, easing installation
of addons ("Just copy the pakfile to addons/ ...") etc.

The File access system as a whole can speed things up however. That's the
remaining points on my list - the ones that need most discussion (so I
posted the others first). Ok, let's start work on that:

(1) Memory mapping of files.
Either single freestanding files or on a per-PakFile basis. This will
propably give the greatest speed gain if applied to many small,
often-accessed files.

(2) File system cache manipulation.
Telling the filesystem not to cache certain files (e.g. streaming
video/audio) to conserve memory for better use, perhaps influencing the
prefetch buffer size for some files (again streaming stuff) etcetc.

(3) A mid level cache system.
Allowing for cache-after-decode (caching in RAM this time). Perhaps this
can be combined with the cache-on-hdd thing to store the decoded data on
the hdd and provide mmapped access to it...

(4) Perhaps an optional "master" index file containing the directory trees
of all PakFiles to speed up initialization and file finding.

(5) Accessing files via number.
Mainly to avoid the costly path string parsing. The alternatives for
that (as I see it now) are:

a) Automatically building header files containing macros resembling the
path/filename, e.g. 
#define gfx_textures_wall_darkbrick_rgb  123456
where 123456 is the offset of file "gfx/textures/wall/darkbrick.rgb".
Benefits: Access via this system is *fast*
Problems: Bound to specific builds of the PakFiles,
difficult to handle with multiple pakfiles (it's the offset - but in which
PakFile?)

b) Assigning numbers to paths at runtime, e.g.
FileNumber1 = ppFGetFileNumber ("gfx/textures/wall/darkbrick.rgb");
MyFile = ppFOpen_n (FileNumber1, PPFREADONLY);
Benefits: versatile, has no problems with different PakFile builds and
multiple PakFiles, also supports freestanding files
Problems: Still needs the number-to-offset lookup, requires the
ppFGetFileNumber query first.

Dan also mentioned that it could be beneficial to support cross-referencing
files via number, e.g. file "3d/meshes/mech/lefthand.3d" referencing to
"3d/meshes/mech/thumb.3d" etc. This is mostly beneficial in cases where the
path strings simply take up too much space.
But it's IMHO quite difficult to implement (mainly because the numbers have
to be known at PakFile build time - when the referenced PakFiles have not
neccessarily been built yet)..


>> AAAAA - American Association Against Acronym Abuse
>> 
>Australian
>Association
>Avidly
>Advocating
>Accurate
>And 
>Appropriate
>Acronyms
>

LOL!! ;)

>That's only 8, I got it up to 10 once.  (Besides the `And' and `Avidly'
>are cheating).

Doesn't matter *grin*


Cu
	Christian
--

I see no reason to stand here and be insulted. -Spock