2010-04-28 - Debugging Multitasking

It would be nice to review some of the UI stuff that I have lying around, but I really should continue to work on the library loading parts first, because then I can work on the UI stuff as a separate library, copy it onto the machine, and have it run.

Not only would the code then be separate and compiled independently, I could also use it as a testbed to test out the Library loading parts and start planning out the API layers that will have to exist between the different areas of the OS.

Ok, the first part that I am missing is the filesystem delete code.  As I'm currently working with a FAT filesystem in a single-threaded system, a delete procedure should be fairly straightforward.  I know, I don't really NEED this in place to be able to proceed with UI devlopment, but I've been putting it off.

Still reviewing the current codebase, and was pleasantly surprised to find that the process system is almost entirely implemented.  A process command initialises the system by hooking in the interrupt 0 handler, and a second processtest command initialises two process functions which each do nothing other than loop for a "long" time, then print a character on the screen.  All of this works perfectly.  If I were to work on this aspect of the OS, it would probably be good to produce a "Top" command to list running processes and also modify the Timer_Delay() function (which currently just loops until the tick counter is correct) so that it suspends the task for a given amount of time.  This would be much more efficient as my processes are spinlocking because they woudn't use any CPU time when not running.  They can't use the current delay function as they may well miss the expected tick count because they weren't running.

Ok, so that's TOP written.  It's horrible code, but it should work for now.

Next thing I need is a state value for the processes.  Each processes will either be the current task (running), be waiting to run (waiting) or suspended (sleeping).  Once I have this, I can then provide functions to change the state of a task.

It seems there is still some instability in the process subsystem.  I introduced a third task to observe the behaviour, and the system threw an Int13 after about 4 "A" iterations.  The Code Segment in the exception must be a mistake because it's reading 0xECD7, but nothing I have changes the CS pointer, so how has that happened?

A quick change to the child Processes and they now load their registers with specific values when they start processing.  Hopefully, this should not only allow me to test that their register values are being properly maintained, but also see which process it is that is causing it to crash (not that it matters, they are trivial and almost identical).  An added bonus is that I should be able to see if there's a phase error in the exception stack frame which would mean that the value displayed as CS is actually from a different register ... if it crashes again.

I wonder what happens if you perform an iret operation to a hlt instruction?  Does it go back to being halted?  Does the IP advance beyond the hlt?  If I have an "idle task" which infinitely halts the processor, can I just leave that running.  The processor will awaken as required, do things, then halt again.  Maybe have to experiment.

Ok, the processing code failed again, another Int13.  The CS pointer is still 0xECD7 oddly enough, suggesting that the failure is the same (or at least similar) as the one before.  The full detail is :

EAX = 0x00011816  EBX = 0xFFFFFFFF  ECX = 0x000A6DB0  EDX = 0x00011AA0
ESP = 0x0020040F  EBP = 0x00200443  ESI = 0x00000001  EDI = 0x00000048
EIP = 0x00000020  CS  = 0x0000ECD7  Flg = 0x00000008  Cde = 0x00010202

The Stack and Base pointers look perfectly valid for the values I'm using, which suggests that the interrupt information does not have a phase error, however the A, B, C and D registers do not have the expected values in them.  The command that was running immediately before this was interrupted by a character paint event (which happily completed within its time slice).

2010-02-25 - Debugging

A bit of debugging today and found why it was broken ... it helps if you don't overwrite your own stack when reading data.  Fixed that and I now have "OS-in-a-Bochs".

Considering how to implement a directory listing system, I can load files if I know what they're called, but it would be nice to be able to enumerate them.  Also finishing the FAT implementation of Delete would be handy so that I can overwrite files.  Oh, and the streams, always the streams.

2010-02-24 - Bootloading from a FAT Filesystem

Finished entering the code that I came up with last night, added a few tweaks to improve it, and also fixed some invalid pushes and pops.  First test suggests that it was successful, but an examination of the memory says it wasn't.  This problem was quickly tracked down to not having reset BX (the buffer address) back to 0, so it should have gone to 0x0600:0600 (or 0x06600).  After fixing that and a quick test, it loads the block successfully.  \ o /

Next I need to put a disk to the bootblock so that it is complete.  I think actually putting a kernel on may be a good start, and a load of other files just so that I can tell if sectors have loaded (instead of all zeroes).

The calculations for the start and length in blocks of the root directory have been successful, but the load is failing.  Going to examine the parameters being passed to the BIOS INT 0x13 call

The Drive and Head parameters were the wrong way round in DX.  Fixed that.  It's now loading the Root directory correctly, searching for the file, then loading the FAT correctly.  I don't think the search is quite correct though, it found the file even when I changed the filename to something wrong.

With one small exception, the first draft is complete, using 440 bytes of 512

First block of the file is being loaded correctly, but the FAT read to find the next cluster seems to have yielded incorrect results.

Fixed up the FAT reading code, there were a number of errors, and it seems to have read all the sectors of the Kernel into memory.  Just need to get the counter incrementing correctly so that they are not all written on top of each other, and it will be done.

Also need to test reading beyond Cylinder 0

\ o /  The counter is now incrementing correctly, and the kernel had been loaded, and I've put in the call to jump to the Kernel, and it's loaded up ... with 40 bytes of bootblock to spare

I am limited to the first 8Gb for loading the kernel by the BIOS, and I am also limited to 43'690 clusters by the cluster computation algorithm.  In fairness though, FAT12 is going to give up long before my code does.

As for performance, not so much.  The primary problem here is that I load the entire root directory into memory, scan that, then load the entire FAT into memory and scan that whilst loading the kernel.  Given that loading sectors from a disk is going to be quite slow, this may well be a significant performance hit.

Having done "the math" against a Windows XP formatted 1.44Mb floppy, the FAT occupies 9 sectors and the Root Directory occupies 14, so the "entire data" load that I was worried about clocks in at just under 12Kb, less than 0.5% of the disk ... so meh.

I still need to add retry code and a few failure paths, so I'll probably eat my 40 bytes fairly quickly.

In any case, I'm happy, I have a full FAT12 file finder and loader in 389 bytes of code (with some more for data)

The other avenue that this presents is the possibility that I could create a small FAT12 Partition on the Hard drive in my test machine and put my boot block in the first block of the partition, then have my OS booting from the hard drive (with a few tweaks).  I could then PXE boot a new kernel, and then get the kernel to save itself out of memory onto the disk.  It's no Windows Update ... but then again, this would work

Looks like I have some more debugging to do, just dropped the latest (~85Kb) kernel onto the disk image and it's failing to load.  Could be either filesize, or the location of the sectors, or both. :S  Back to the debugging ...

It has loaded 66 sectors, it is about to try to load cluster 0x034C, which I think is block 875, but I have tried to test them individually, and them seem to work.

2010-02-23 - Bootblock Development

Bootblock:
1)  Correct the origin issue.  My code has to run with an Origin of 0 so that I can relocate it to 0x7A00 with ease
2)  Set the DS register correctly, it is needed for accessing the memory.
3)  Remember that sector 0 is invalid
4)  Read to the end of the chain, not to the end of the file.  File size is irrelevant, truncation is futile, the sectors will be assimilated
5)  The teletype function is INT 0x10,AH=0x0E
6)  Add LBA-CHS conversion to the sector loader.  Means one register for sector numbers up to 65536 (32Mb disk)


Because I am loading the code at 7C00 then copying it to 7A00 out of the way, there is no meaningful value I can use for the Origin of the file except zero.  Anything else and I either need to add an offset to each jump and memory reference before the 7A00 switch, or to everything after it.

I have gotten the code to work now using an Origin of zero.  The code segment I was loading was a power of 16 out.  Never can remember that Segment:Address syntax.

If I use a single 16-bit register (in real mode) to represent my sector address on disk, that gives me only 65536 sectors which means a limit of a 32Mb disk with a common 512-byte sector size, or rather, that we can only read the first 32 Mb of the disk.

Given that the CHS translation of the BIOS allows us to read the first 8Gb, I'd like to extend the LBA mapping to use two 16-bit registers to store the value.

However, in order to then compute the CHS values from the LBA value, I need to perform "long division" and "long modulo" on the two registers separately and then combine the result.

So just trying to compute the algorithm now.

Hmm, upon further investigation, the DIV instruction actually works against a 32-bit register pair anyway, specifically DX:AX, and would allow me to address 2Tb of drive, not that FAT12 can effectively support a drive that big, nor that the BIOS CHS can actually address a drive that big.

Anyway, given that the INT 0x13 command requires specific values in specific registers, and that the 32-bit DIV instruction requires the use of other specific registers, I shall try to write a LBA-CHS conversion algorithm that takes advantage of these whilst being efficient.

2010-02-22 - Bootblock Development

Ok, change of tac for today ... using assembly to find and load a file from a FAT drive.  Should allow me to create kernel files and drop them onto a disk to be able to demonstrate on Bochs or a real machine, whilst also being able to take the same file and PXE boot it still.

Going to assume a file in the root directory with a known name that is within the reach of the BIOS functions

Wow, didn't realise how rusty I am on Assembly, four hours of development and I have fewer than 100 lines to show for it

2010-02-21 - DosDevices and Filesystems

Work is progressing on the Dos Abstraction layer.  The mountlist parser is finished, and the filesystem and device are being looked up by the library loader and accessed.  If found, they are being put into a DosDevice structure and added to the list of devices.

I have just finished the abstracted version of LoadFile (which used to be implemented by the FAT driver dierctly to load from the "current" FAT volume) so that it looks up the dos device in question("BOOT:"), find the Filesystem attached to it, and call the LoadFile routine on that filesystem, passing it the path of the file.

Onoes!  /o\  A memory leak has developed somewhere within the Dos_Loadfile routine.  It's leaking 32 bytes, but it is cause the available space for the file to drift by 544 bytes each time.  Something must be done!  One scheme is to modify MM_Malloc() so that it records the return address of the call to MM_Malloc(), that way, the address of the instruction that allocated the space will be on record.

Success!  With a little inline assembly, I have been able to read the address of the instruction that is calling MM_Malloc() and store it along with the used block entry.  When my used block is full (and the system is therefore going into a controlled fail) it writes out the top entries in the memory list so that I can see what the block size is and where it was allocated, and can track it down.

My file searching function (that finds the first directory entry that matches the filename) was being called, but the returned entry was not being freed in all cases.  That was fixed, and now it is looping the load quite happily with no leak.  \ o /

Memory leaks and allocators aside, I need to finish off changing the Fat driver so that it uses the Context and not any global variables.  A good place to start would be to remove all the global variables and then see where we are :)

Ok, so I have spent most of the evening re-writing the Fat driver to use a structure for its state instead of global variables.  This way it will be able to handle multiple Fat volumes at once.

Damn, that's why it doesn't work.  I need to implement the mountlist parsing within the Ram driver.  It needs to read the parameters and make sense of them.  However, on the up side, it did remind me the LoadFile routine needs to test for a DosDevice being passed, a FileSystemContext within that DosDevice, and for the Context to be flagged as valid.

\ o /  A quick implementation of the versatile strtol() string-to-long converter and I'm able to finish parsing the mountlist values into the ram driver, and it has found the disk image in memory correctly, and FAT has deemed that it is valid.

Now ... why is it throwing an interrupt 6 (Invalid OpCode) when I try to read the drive?

Fixed the invalid opcode ( was passing 0 instead of the context ) and put in the appropriate exception path for that too, and then fixed the general protection fault that it hit after that.  I was using a ternary operator without the correct parenthesis.  Instead of allocating 1024 bytes, it was allocating 2, then using 1022 bytes of other allocation's space with "predictable" results.

You can now request that the Dos layer load a file such as "Boot:test~1.lib" and it will look up "BOOT:" in the device list, find the FileSystem attached to it and tell it to open "test~1.lib", which will then use the BlockDeviceDriver in the DosDevice to read the "disk".

2010-02-20 - Streams and DosDevices

Investigated streams more, and I have created a "FILE" structure that actually just creates a queue buffer similar to those I already have.  It means that I can now use generic functions to interact with them, and improve their complexity later on.

Looking now at expanding the definitions of the device and file system drivers to head toward my previously designed Dos abstraction layer.

If I'm going to have to parse this mountlist in order to derive all the required information, it would make sense to implement the C library strtok function, which is what I'm doing now.

Ok, String_strtok() seems to work, and I copied the example mountlist in to test in on that.  A few tweaks and I should have the routine to parse the mountlist for the devices.

Dropped strtok because it won't work in this situation.  It would be valid to have no space between the name and the equals sign, which would leave us no room for the null terminator.  I have instead parsed the mountlist by hand into a "sanitised" mountlist string.  The sanitised string is then being parsed again and copied into the appropriate Dos device structure.  Now I need to change the Library loader so that it can return a reference to the library if it is already loaded, then get the Dos system to find the library (filesystem driver, device driver) specified in the entry and initialise the device.

I've updated the Fat driver and the Ram driver to expose their interface structures with a name that won't clash with other things, and I've added them both as entries in the Kernel library so that they can be found by name when referenced by the mountlist

The Dos system is now pulling the device and the filesystem out of the mountist definition and looking them up in the library.  I also have them calling the CreateContext routines and passing the definition.  Now I need to get the CreateContext functions to create the appropriate context element from the settings in the mountlist.