Vs: PenguinFile

["Bjarke Hammersholt Roune" <asmild@post6.tele.dk>]

This began as a private Email, so all it of maybe won't make sense to you...

<<About the webmaster thing: We'll see. He hasn't been at it for very long yet. If he really is too slow/unresponsive/whatever, then someone else will have to do it. But until then... >>

I didn't mean replace him, just help him out if he needs any help. I probably should talk to him about that instead, though... Do you know his Email address? (can't remember who it is)

<< << You really should state somewhere that as a pointing device really is an input device, it should be handeld by the input team. This is not at all >> >>

<< Well, what else should it be? >>

<< << Ahh, yes, what I mean was, though, what is the head positioning TIME? What range will it lie in on a medium-size computer today? >> >>

<< Well, in the single-digit millisecond area ;) >>

Sometimes my brain just blanks out... :) It usually happens between 3 and 4 in the nigth if I haven't got any sleep the previous night.

[on ppFILE size]

<< Well, some positions, sizes etc - that easily sums up. But 40 bytes really aren't that much.  >>

I looked it over, you'r right, it really is needed, and it's really only a little less than 40k memory for a 1000 open files. Most systems should be able to cope... :)

<< << Not really, you would only need a single overload; one that took a  number, and one that took a string. The string version, I take it, is >> >>

<< The thing is that C does not support function overloading at all. >>

Hmm... I wasn't aware of that. The function that took the numbers could be made to take paths rather than numbers in all but release builds, though.

<< << Depends on what you mean by "hardcoded". They wouldn't be hardcoded in that they coudln't change, so they could be updated. >> >>

<< Only by recompiling the game. With "hardcoded" I mean that the names of all (or most) of the used files are specified in the code, not in some data file. >>

The third file contained in the pak file will still be the third file contained in the pak file regardless of wheter the size of a file changes or not.

<< A good handling would e.g. be to have a fixed main configuration file ("mygamerc") that contains the name of the levels' main map files, which in turn contain the names of the mesh/texture/soundeffect/... files they use. So you can exchange all game data by simply copying some files around. >>

You could also have suchs a file that just contained numbers rather than paths.

<< The important thing is that these things require that the file names are *not* compiled into the game. >>

The numbers would be hardcoded into the game, but what those numbers would retrive from the pak file is not. The game can also learn to access new files it didn't know about previously if given it's use and number.

I would say that with a little consideration on the art of the game programmer, anything that can be done with paths can also be done with numbers: It's just two different ways of saying where the needed data is. A string is also a number, just a very long one. It's like using URLs instead of paths, with the (important) exception that these numbers does not make much sense to a human, and that it's therefore a little harder to implement, and extra utilities is nessecary to ensure that files in pak files keep their numbers. 

These are not minor obstacles, and taking into consideration that the performance of ppfOpen() is so fast that the effect hardly will be noticeable to the user, I must agree that it's not a very good idea in this context.

>I don't know if there is suchs an API, but I couldn't find it (I didn't

<< Of course, because it's only in my head yet. >>

Well, I can't be expected to know what is in your head... ;)

<< ppfCreatePak (const char *Name, <all the header parameters>);
ppfCopy (const char *source, const char *dest, int flags);
ppfCreatePak (NewPakName, HeaderIfo);
ppfMount (NewPakName, TmpMountPoint, ppfMF_write);
ppfCopy (TheCommonMountpoint, TmpMountpoint, ppf??_recursive);
ppfUmount (everything); >>

What do you mean by ppfUnmount(everything), do you mean like an ppfUnmountAll() API, or what?

<< That means updating a pak will require roughly that code:

while (there_is_another_update)
{
ppfMount (TheUpdatePak, TheCommonMountpoint, ppfMF_override_files);
} >>

I thought mounting was strictly a temporary memory thing (?). So what happens here is that the update pak is mounted with the pak-overloading feature on the HARDDISK, rather than just altering some memory structures? It would seem to me that that practically IS an ppfUpdate()... :)

However, I think I have a good idea on how to write paks of format 0 (I take it that is what we are talking about here: format 0). I'm going to explain it in C++, but converting to C is merely a matter of having an additional struct argument to the functions.

First, we need a PakImage or PakMemImage or some suchs class. This class would then have these functions: (the failure and success returns values is of course somewhat unimportant)

** ppBool SetWorkingDir(const char* pDirUrl)
Not really nessecary, but I thought it would be a nice feature. The classes working directory is seperate from the global working directory. Returns 0 on failure, non-0 on success.

** ppBool AddFile(const char* pFileUrl)
Adds a file to the memory image of the pak file. Can be absolute or can be relative to the working directory. Returns 0 on failure, non-0 on success.

** ppBool AddDir(const char* pDirUrl)
Adds a directory to the memory image of the pak. Even if this directory exists, the files it contains is not added. Returns 0 on failure, non-0 on success.

** ppBool MapDirToImage(const char* pDirUrl, const char* pImageAddPoint)
Adds the files in the dir specified in the pDirUrl to the image at the point specified by pImageAddPoint. This method does not add the directories specified by either argument; these must be there already. Neither does this method add sub-directories. Returns 0 on failure, non-0 on success.

** ppBool MapDirStructureToImage(const char* pDirUrl, const char* pImageAddPoint)
Adds the directory specified by pDirUrl, and any sub-directories it may have (recursively), to the image at the point specified by pImageAddPoint. This method does not add files, and the directories specified by either argument must already exist. Returns 0 on failure, non-0 on success.

** ppBool MapFileStructureToImage(const char* pDirUrl, const char* pImageAddPoint)
Adds the directory specified by pDirUrl, and any sub-directories (recursively) or files it contain, to the image at the point specified by pImageAddPoint. The directories specified by either argument must already exist. Returns 0 on failure, non-0 on success.

** ppBool WriteImage(const char* pDestUrl)
Writes the image to the file specified by pDestUrl. If this proves to be a lengthy process, it could be an idea to move some of this functionality to a PrepareWrite() method. Returns 0 on failure, non-0 on success.

** void ResetImage()
Resets the pak file image so it is empty, thereby deallocating the memory used to hold this image.



If you think that these are too many methods, only AddDir(), AddFile() and WriteImage() is strictly nessecary. Writing a complete pak file to the image is merely a matter of calling MapFileStructureToImage(), specifying the mountpoint and map it to "." (ie, the working directory).

Doing it like this is a good idea because writing format 0 pak files require (as of the documentation) that file names be stored in optimized structures, and that can't be done before the complete architecture of the pak has been established.




<< << Besides the POSIX file handling, is there any other incompatibilities in PenguinFile towards WIN32 I should know about? >> >>

<< Not yet. The memory mapping stuff will be however. >>

Well, I implemented all the POSIX directory handling functions used by PenguinFile, and it still doesn't compile. I've noticed that all over the code, delete is rutinely called on const pointers. I don't know about your compiler, but my compiler does not accept this. What about including config.h (what does it do, anyway?) in PenguinFile.h rather than in each file. It seems to me it's used all over (?).

64 bit integers is also supported in VS Visual C++. The type is called __int64 and can be signed or unsigned. If the compiler being used is MS VC++, the preprocessor symbol _MSC_VER is defined. It would be nice to get this functionality into PenguinPlay.h like gcc has it's special functionality in PenguinPlay.h


[does gcc do templates]

<< Yes (except for a very minor exception) >>

exception?


>Isn't there any integrated development environments? IE, do you then
>have to write make files and invoke the compiler, builder and debugger from
>the command line?

<< But even if a real superduper-mega-killer-IDE was available, I'd still
prefer the cmdline. IDEs always put all kinds of junk in your source dir,
makefiles etc. And I haven't found a single IDE with a decent text editor yet ;) >>

MS VC++ has an *EXCELLENT* text editor. I simply couldn't imagine one better. It places 2 project files in you source dir, and a temporary directory debug aswell as a temporary directory release. No make files (though it can export them). The text editor (which btw colors comments, C/C++ keywords and other text in each their own unique color), the compiler, linker and debugger is all integrated into the environment. There's context-sensitive help on any keyword or error message 1 click away. Plugins can be written to expand functionality in any number of programming languages.

That's why I think a transition will be quite drastic for me, especially considering I've never written a makefile...

<< Well, VC++ make is primitive compared to GNU make ;) >>

There is a very good reason for that: VC++ normally do not use makefiles.

<< And our makefiles also make use of autoconf, so converting them is even harder. >>

What's autoconf?

<< You'll have to create your own makefiles/project files from scratch. >>


It takes about 10-20 seconds, so that's no problem... :)

<< << Btw, why are you using POSIX-specific functions? Why not just do everything through the standard library? >> >>

<< Because the standard library has no functions for directory reading.>>

Yes, I realised that when I tried to find the standard library's support for directories... Seems I ask alot of stupid questions... :)

<< We have a simpler procedural event system for the lower levels >>


Any documentation for that?

<< << pps. Is the mailinglist usually so quiet >> >>

<< It is rather quiet, but not *this* much. Did you respond to the mail you got from ezmlm? Unless you do that you won't get anything.>>

I got your "My holiday" post from the list, so it's ok.

On another note, is the POSIX time_t type the exact same type as the standard library time_t type? And is it encoded the same way? (ie, the amount of seconds elapsed since january 1, 1970) I need this information to read the modification times for windows files, as those are encoded as the amount of 100 nanosecond intervals that has elapsed since january 1, 1601.

btw, why are you testing the return value of strcpy() to make sure that it is the same as the destination string? It would seem to me that that there is no case in which these would not be the same...