lua file IO

douchebagatron

<div class="IPBDescription">in NS2</div>I've been working on a little project to create a profiler to work with NS2 so I can see what functions are taking up the most time. I tried using a standard lua profiler but ns2 crashes when I try to include that .dll for some reason.

So i'm currently writing a program to loop over all lua files and add some file i/o stuff at the beginning and end of each function to specify how much time was spent in it, and possibly the stack layer and whatnot.

The problem i'm running into is that io functions (i.e. io.write, io.read, file:write) and even os functions (os.execute) don't work from inside ns2's gamecode, but they work in the standard lua. anyone have any idea why?

I'm getting this error when I try to call them:

"lua/Main.lua:31: attempt to index global 'os' (a nil value)"


also another weird thing: I see calls to Shared.GetTime(), but there's no GetTime() function in Shared.lua

player

They are disabled for security reasons (giving a mod free access to a client's filesystem is not wise). Though I would imagine the Server-VM might have these functions enabled again in due time. Until that day, you'll have to load up an external module (DLL) to get access to the filesystem, or write your own implementation in C\C++.

douchebagatron

what about writing out to a log? I was looking into Print and PrintDetailed, but those just print to the console. I could probably dump everything to the console if it came to that, then sort through it later to only get what i want, but I'd still need to get everything into a log file.

player

Actually you make a good point, as printing to the console does exactly that, writing it to a log-file. You can find them at "AppData\Roaming\Natural Selection 2", it will store the last 3 sessions in log-files.

fsfod

Have you seen the builtin profiler for the engine just type profile in the console(it got updated a bit this build to show totals for the core systems). also max talked a bit about his way implementing profiling for the engines lua <a href="http://www.m4x0r.com/blog/2010/11/pre-processing/" target="_blank">http://www.m4x0r.com/blog/2010/11/pre-processing/</a>

wulf 21

I thought of writing a record plugin, so you could record in-game first and use fraps while playing the recorded file later. However I would need access to the file system for that. Couldn't they just allow access to the mod folder but block the rest of the file system?

player

I imagine they intent on exposing Lua's filesystem-IO functionality with the restrictions you suggest, but haven't come around to doing it yet.

douchebagatron

If they were to do that, it probably wouldn't be Lua's file i/o system, but a wrapper for c++ file i/o. I'm sure they did something similar for the output to logfile.

player

They probably do have their own IO-mechanism in their engine, which is used to write the console-output to a logfile. The Shared.Message-function is simply an exposed engine-rountine to output data to the console, which the engine in turn writes away to the logfile. They _could_ expose this IO-mechanism to Lua, but it would involve more work and will break\render useless all of the (non NS2-specific, but Lua-compliant) scripts\addons currently in existence, forcing NS2-scripters to write things from scratch.

wulf 21

Hey, now I actually wrote my own simple lua file io with C and I'm going to share it. It allows to sequentially write a new file or read a file. More advanced stuff like file handles for more than one file or append file or update a file without completely overwriting it is not included (yet?).

<a href="http://www.file-upload.net/download-3474566/fileiolight.zip.html" target="_blank">Download here</a>.

edit: uploaded new version 2011-6-1 12:12 GMT. See end of post.

<!--sizeo:4--><span style="font-size:14pt;line-height:100%"><!--/sizeo-->Description<!--sizec--></span><!--/sizec-->

In Order to use the Functions in the .dll within LUA you have to use

<i><function name for Lua>=package.loadlib("<path to fileiolight.dll>","<internal function name>")</i>

Contrary to the paths for lua Scripts, the <b>paths are relative to the Natural Selection 2 main folder</b>, not relative to the mod folder. The following internal functions can be used: Create, Open, Close, Write, Read. For your convenience I recommend to define a class as a container for all these functions and use the same names as the internal function names but you can of course name your functions as you like it.

For example if you put the functions in <i>myclass</i> and have put fileiolight.dll in the folder <i>mymod</i>, the definition for the function <i>Create</i> would look like this:

<i>myclass.Create=package.loadlib("mymod\\fileiolight.dll","Create")</i>

<!--sizeo:3--><span style="font-size:12pt;line-height:100%"><!--/sizeo-->Functions<!--sizec--></span><!--/sizec-->

<b>Create(<filename>)</b>

Creates a new file with the filename given as a string and opens the file for writing. If it already exists it will be overwritten with a new file.

<b>Open(<filename>)</b>

Opens an existing file. Read only. Like before the filename must be supplied as a string.

Note that, like mentioned before, paths are relative to the Natural Selection 2 main folder. If you don't add any folder to the name and e.g. call <i>myclass.Create("my.txt")</i> (assuming you use the naming convention above), <i>my.txt</i> will be created in the Natural Selection 2 main folder, not in the mod folder.

<b>Close()</b>

Closes the currently opened file. Note that a call of Open or Create will automatically close the last file, so you only need to call this if you for some reason don't want to have any file open, e.g. if you want to edit or delete the last file while NS2.exe is still running.

<b>Write(<arguments>)</b>

Writes any number of arguments into the current file in the order they are given. Will work only after you called Create(). Supported types are number, boolean and string. Unsupported types including nil values simply will be discarded.

Theoretically userdata is supported, too, however I haven't tested this. According to the lua reference manual, userdata doesn't have any native operators defined in lua, so you would have to write your own operators by modifying Luas source files. However I don't know if the dev's have implemented any use of userdata in the engine. If they haven't, this is obsolete.

Tables are not supported but it would certainly make more sense to write a function lua that can write tables which wraps around the Write function and writes the element names and types into the file.

Note that Write won't do any type conversions, e.g. the bits representing a lua number (which is a double value in C) will be directly written into the file. These will most likely be unprintable characters if you view the file in a text editor.

In order to write a human-readable file you would have to convert every type into a string manually. You can use tabs (<i>myclass.Write("\t")</i>) and new line (<i>myclass.Write("\r\n")</i>), too. Lua will automatically convert these to the according control characters before calling Write.

<b>Read(<type>,<count>)</b>

Reads count values from the file and returns them as the given type. So you either have to know the exact pattern how your file is built or you include the information how many variables of which type are written at the beginning of the file.

The type is supplied as a string and can be one of the following: "number", "boolean", "string", "userdata". The count is given as an integer and has the following meanings:

In case of number or boolean it is the count how many booleans or numbers should be read. They will be returned in the order they are read (function has multiple return values). For example <i>a,b=myclass.Read("number",2)</i> would read 2 numbers and store the first one in a, the second one in b. Or <i>nums={myclass.Read("number",4)}</i> would read 4 numbers and put them into a Table that is then assigned to the variable nums.

In case of string count means the number of characters to read. All the characters are then returned as one string, even if it should contain embedded zeros. Userdata works very similar, in that case count is the number of bytes to read. They are then returned as one block of userdata.

Note that you can currently only read 1kB of data of any type at once (size of my read buffer). That will be either 128 numbers (double values are 8 bytes), 512 booleans (yes, they are a waste of space, but C doesn't have a shorter value than short int to represent them) or 1024 characters or bytes of userdata.

edit: Out for a few hours and already a patch necessary (updated download link). I just noticed that Read() had undefined behaviour if you try to read past the end of file (would return random values). With the new version Read() will return only as much data until the end of file is reached. You can use this to read your file in a loop and check if the end of file has been reached.

In case of number and boolean nothing is returned past end of file. For example if <i>a,b=myclass.Read("number",2)</i> would return nothing, a and b would contain nil values. Or if <i>nums={myclass.Read("number",4)}</i> would return nothing nums would be an empty table (the check would look like <i>nums=={}</i>).

In case of string and userdata an empty string (ckeck if the string equals "") or an empty block of userdata will now be returned past eof.

player

<!--quoteo(post=1849397:date=Jun 1 2011, 02:26 AM:name=wulf 21)--><div class='quotetop'>QUOTE (wulf 21 @ Jun 1 2011, 02:26 AM) <a href="index.php?act=findpost&pid=1849397"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->That will be either 256 numbers (double values are 4 bytes), 512 booleans (yes, they are a waste of space, but C doesn't have a shorter value than short int to represent them) or 1024 characters or bytes of userdata.<!--QuoteEnd--></div><!--QuoteEEnd-->
(unsigned) short int -> 2 bytes.
(unsigned) char -> 1 byte.

If you really wanted to you could shove 8 booleans into a single (unsigned char) byte.

I should also point out that there are many (DLL-based) File-IO frameworks for Lua already in existence, so if you created this out of necessity I suggest you have a look at those as it'll save you quite a bit of time. However if you're just having a blast writing things for Lua and learning along the way, then go for it.