Before I can fully test the Memory Manager, I'm writing the memdmp() function to display a section of memory on the kprintf console. Unfortunately this means working out more of an implementation of printf than I had bothered before.
In a way, this is a good thing because it's the one part of the my OS that, until now, was not written by me and subject to third-party copyright and license.
Also thought, I wonder if Bochs can have an additional output stream that I could (ab)use to give me the kernel output on a separate stream/window. I could make the OS output it through the serial port, which I guess a real machine would use.
Another thought I had earlier, create a macro for malloc when allocating structures. The macro would call a version of malloc that wrote the name and length of the structure somewhere so that memdmp() could display a lot more. Could this be extended to the definition of structs so that the kernel had a table of all the fields in each struct so knew how to interpret "all" the data in memory?
2013-07-05
2013-07-05 - Memory Manager Progress
Finally finished typing up the memory manager implementation and fixing some of the issues in it. It now compiles and appears to be OK.
2013-07-01
2013-07-01 - Memory Manager (Version 2)
My recent long train journey provided an opportunity to write down the implementation of a memory manager, one which uses a linked list of free blocks and where every block has a header and a footer. I read about something like this somewhere, but I don't remember where. It should work very well for lots of things, however, for small allocations, the size of the header and footer introduce a significant space inefficiency.
Anyway, I'll get the implementation in and see how it goes. I think I have a problem with my footer structure not being a multiple of 16 bytes, and I'm not sure if I should worry about putting the used blocks into a linked list of their own, or only worrying about the free blocks. If I need a map of the memory, I can walk the structures.
Anyway, I'll get the implementation in and see how it goes. I think I have a problem with my footer structure not being a multiple of 16 bytes, and I'm not sure if I should worry about putting the used blocks into a linked list of their own, or only worrying about the free blocks. If I need a map of the memory, I can walk the structures.
2013-06-21
2013-06-21 - How Programs Talk to the Kernel
So I've decided how I'm going to implement the mechanism by which the newly created executables find the kernel library.
The original Linux kernel used software interupts for system calls which, while being a bit slow, were easily accessible from everywhere. Later on, the linux kernel moved to using memory mapped kernel functions which are faster.
What I'm going to do is have a single interrupt based system call which provides the (process-specific?) address for a FindSymbol() function. Using this mechanism, a new process can find this function, then use it to search for other symbols such as "write". This way, as the program starts up, it can find the addresses of symbols it requires and set itself up. This may cause an overhead on programs which are frequently started and stopped such as console tools, but this should be acceptable for now.
In the meantime, I still haven't actually put the interrupt handling back in yet. Should probably do that, and may as well use the List structure to combine the handlers. I guess I should create a list for each interrupt number, then just do an AddTail() for each handler in turn. I should also put in the kmalloc() and kfree() using a slightly upgraded technology from last time.
The original Linux kernel used software interupts for system calls which, while being a bit slow, were easily accessible from everywhere. Later on, the linux kernel moved to using memory mapped kernel functions which are faster.
What I'm going to do is have a single interrupt based system call which provides the (process-specific?) address for a FindSymbol() function. Using this mechanism, a new process can find this function, then use it to search for other symbols such as "write". This way, as the program starts up, it can find the addresses of symbols it requires and set itself up. This may cause an overhead on programs which are frequently started and stopped such as console tools, but this should be acceptable for now.
In the meantime, I still haven't actually put the interrupt handling back in yet. Should probably do that, and may as well use the List structure to combine the handlers. I guess I should create a list for each interrupt number, then just do an AddTail() for each handler in turn. I should also put in the kmalloc() and kfree() using a slightly upgraded technology from last time.
2013-06-18
2013-06-18 - Hello World ELF
As well as pulling in sections of code from previous iterations of the OS, I've been thinking more and more about the library, driver, and executable system. Between the system calls from Linux, the POSIX specification and the functions required by NewLib, I've been able to whittle down a short list of system calls which I need to develop, and a rough order in which to do them.
My plan is to create a Hello World! program, but this program will not have a crt0.o, and will not include a LibC. Instead it will use the OS method of getting a handle on the kernel.library (which will provide methods for opening libraries and finding symbol locations), from there it will find the "write" symbol, and call it to write the string (for the moment, it can assume that the stdout is at a specific file number).
Once this program is ready, I'll need to finish all the kernel code to underpin this. For the moment, the following simplifications will be made:
The workflow needed in order to achieve this, including required components is:
Distracted again, but Bochs has implemented a basic beep capability, and I've put the start of a "music" player based loosely on the the ZX Spectrum +2 PLAY command in so that I can program Greensleeves ...
My plan is to create a Hello World! program, but this program will not have a crt0.o, and will not include a LibC. Instead it will use the OS method of getting a handle on the kernel.library (which will provide methods for opening libraries and finding symbol locations), from there it will find the "write" symbol, and call it to write the string (for the moment, it can assume that the stdout is at a specific file number).
Once this program is ready, I'll need to finish all the kernel code to underpin this. For the moment, the following simplifications will be made:
- The program binary will be loaded into memory by Grub (to save the filesystem code yet)
- The program will be linked to start at 4Mb, and will be loaded into the kernel memory space at this address.
- The process model will not create a separate process for the program.
The workflow needed in order to achieve this, including required components is:
- Create enough of the process table to store the open file list.
- Create the Open File list (including file operations).
- Create a basic console as previous versions, except using the stdin and stdout streams via read and write. The read() call will spinlock inside the process subsystem until keys are pressed.
- Get the kernel process to call into the console.
- Configure an exit command from the console to return to the kernel.
- Define the method of finding the kernel library and then finding other symbols.
- Reimplement the
- Pull in the ELF loader
Distracted again, but Bochs has implemented a basic beep capability, and I've put the start of a "music" player based loosely on the the ZX Spectrum +2 PLAY command in so that I can program Greensleeves ...
2013-06-14
2013-06-14 - Bootloader Progress
Ok, so I have persuaded NetBeans to start a new project, and I've rewritten the example OS's multiboot header code, and wired in my Grub-Bochs image.
It now compiles the (noddy) kernel, links it into an ELF binary, and loads Bochs so Grub can load the kernel.
I'm considering how modules, libraries and executables will load as I want to take this opportunity to separate them from the kernel completely.
I think I understand now what crt0.o does in the grand scheme of things. When you write a program, you start your program with a "void main()" and stuff like "stdout" already exists and stuff like "printf" works. When your executable is first loaded, the OS actually passes control to the crt0.o section in your program (the start() function). This routine (which is part of the toolchain which builds programs specifically for that platform) then uses some platform-specific voodoo to get references to things, it gets a reference to the environment somehow, it gets some library to power write() and open(), and generally sets up stuff that the C library will need to operate.
Now, I imagine I'm going to end up with two different types of entity, Executables and Libraries.
An Executable will be loaded into memory, a new thread (or process) will be created, and sent in to the start address of the executable. If this is a C toolchain program, it'll land in crt0.o and start the process. If it's a seat-of-the-pants assembly marvel, it will do something else.
When the program exists, either by calling an Exit() function or by the main thread leaving the building, the process will end, and the memory for the program will be freed.
This can all be achieved by me creating a crt0.o as part of my C library, and compiling programs as normal. They'll run and won't know the difference.
A Library on the other hand, is slightly different. It will be loaded into memory, and will present a set of entry points. It won't have a main thread or process given to it, and it won't have a defined exit point.
I imagine that all libraries will need some form of Open() entry point so that the C library linked into the library can connect all of its functions to the basic kernel functions. I guess it must be like a crt0.o, but it won't set up stdin or stdout, and it might not look to call the main() function once it's finished (although it may call a further initialise() in the library code itself, in case it needs to open further libraries or do other set-up).
Beyond that, the libraries could just be ELF .o files (which have no unresolved symbols) and just has symbols exposed which are its entry points.
(I'd have to investigate how to make a linked .o file with specific symbols exposed and others not (public vs. private) )
This all comes down to whether the device drivers should be executables or libraries. I suspect the answer is that they should be libraries, and will be loaded in the kernel memory space (for the moment) but some of the drivers may launch a new kernel thread in their initialise.
So. A device driver is a library. The kernel will have to be told to load it at some point (possibly by a driver list file, possibly by some form of probe program). The library will be loaded and will have its Open() method called (as all libraries will). I guess the open code will register the device driver with the kernel, or I guess it could start probing the system for compatible devices.
In the Amiga world, the device driver library would be a normal library but with some specific entry points to make it into a driver and then it's loaded especially as a driver ( http://amigadev.elowar.com/read/ADCD_2.1/Libraries_Manual_guide/node029E.html ).
I think the device drivers are going to work in a similar manner. The Device driver will be a library (ELF .o file) and will be loaded into the library list, .... (thought interrupted) ...
It now compiles the (noddy) kernel, links it into an ELF binary, and loads Bochs so Grub can load the kernel.
I'm considering how modules, libraries and executables will load as I want to take this opportunity to separate them from the kernel completely.
I think I understand now what crt0.o does in the grand scheme of things. When you write a program, you start your program with a "void main()" and stuff like "stdout" already exists and stuff like "printf" works. When your executable is first loaded, the OS actually passes control to the crt0.o section in your program (the start() function). This routine (which is part of the toolchain which builds programs specifically for that platform) then uses some platform-specific voodoo to get references to things, it gets a reference to the environment somehow, it gets some library to power write() and open(), and generally sets up stuff that the C library will need to operate.
Now, I imagine I'm going to end up with two different types of entity, Executables and Libraries.
An Executable will be loaded into memory, a new thread (or process) will be created, and sent in to the start address of the executable. If this is a C toolchain program, it'll land in crt0.o and start the process. If it's a seat-of-the-pants assembly marvel, it will do something else.
When the program exists, either by calling an Exit() function or by the main thread leaving the building, the process will end, and the memory for the program will be freed.
This can all be achieved by me creating a crt0.o as part of my C library, and compiling programs as normal. They'll run and won't know the difference.
A Library on the other hand, is slightly different. It will be loaded into memory, and will present a set of entry points. It won't have a main thread or process given to it, and it won't have a defined exit point.
I imagine that all libraries will need some form of Open() entry point so that the C library linked into the library can connect all of its functions to the basic kernel functions. I guess it must be like a crt0.o, but it won't set up stdin or stdout, and it might not look to call the main() function once it's finished (although it may call a further initialise() in the library code itself, in case it needs to open further libraries or do other set-up).
Beyond that, the libraries could just be ELF .o files (which have no unresolved symbols) and just has symbols exposed which are its entry points.
(I'd have to investigate how to make a linked .o file with specific symbols exposed and others not (public vs. private) )
This all comes down to whether the device drivers should be executables or libraries. I suspect the answer is that they should be libraries, and will be loaded in the kernel memory space (for the moment) but some of the drivers may launch a new kernel thread in their initialise.
So. A device driver is a library. The kernel will have to be told to load it at some point (possibly by a driver list file, possibly by some form of probe program). The library will be loaded and will have its Open() method called (as all libraries will). I guess the open code will register the device driver with the kernel, or I guess it could start probing the system for compatible devices.
In the Amiga world, the device driver library would be a normal library but with some specific entry points to make it into a driver and then it's loaded especially as a driver ( http://amigadev.elowar.com/read/ADCD_2.1/Libraries_Manual_guide/node029E.html ).
I think the device drivers are going to work in a similar manner. The Device driver will be a library (ELF .o file) and will be loaded into the library list, .... (thought interrupted) ...
2013-04-27
2013-04-27 - Bootloader Design
So, my plan is this ...
The 512-byte boot sector loads a small 16-bit loader. This loader will then load a text file detailing modules to load, the first being the 32-bit kernel. The 16-bit loader will then load all of these 32-bit modules. Its last act will be to switch to 32-bit mode and start the 32-bit kernel proper.
The downside to this is that the 16-bit loader will have to understand the various file-systems it may encounter, but this can be mitigated by requiring a small boot partition which contains the modules.
The upside is that the config file can be modified to account for many different boot devices, including, but not limited to, the following situations:
The bigger upside is that the 16-bit bootloader will be able to request blocks from any (bootable) block device using the BIOS routines and will be able to load any file by name from a PXE network boot using the BIOS routines. That way, we're not doing the same thing as a certain other OS of booting from a USB drive using the BIOS, jumping into protected mode, then finding that our boot device went away.
One downside is that we only have 640k of space to load the modules into (which should be OK as they can be read-only drivers for the various drives)
The 16-bit loader will also have to create an identifier from the boot device in order to pass this to the 32-bit kernel so that I can find the same boot device using the 32-bit drivers (when the BIOS is no longer available).
It could even be made "intelligent" to the point of identifying (guessing) what device drivers it will need, but it is dependant on the BIOS if it's even possible to get any information about what the booted device is.
Most BIOSes just pretend that a USB hard drive is a normal hard drive, leaving a surprise for the OS when they jump to 32-bit armed with an ATA driver and find no drive to boot from
The 512-byte boot sector loads a small 16-bit loader. This loader will then load a text file detailing modules to load, the first being the 32-bit kernel. The 16-bit loader will then load all of these 32-bit modules. Its last act will be to switch to 32-bit mode and start the 32-bit kernel proper.
The downside to this is that the 16-bit loader will have to understand the various file-systems it may encounter, but this can be mitigated by requiring a small boot partition which contains the modules.
The upside is that the config file can be modified to account for many different boot devices, including, but not limited to, the following situations:
- Booting from a floppy (which will need a floppy driver)
- Booting from a hard drive (which will need a S/PATA driver)
- Booting from a CD-ROM drive (which will need an ATAPI driver)
- Booting from a network (which requests file by name)
- Booting from a USB device (which requires PCI, USB, and class drivers)
The bigger upside is that the 16-bit bootloader will be able to request blocks from any (bootable) block device using the BIOS routines and will be able to load any file by name from a PXE network boot using the BIOS routines. That way, we're not doing the same thing as a certain other OS of booting from a USB drive using the BIOS, jumping into protected mode, then finding that our boot device went away.
One downside is that we only have 640k of space to load the modules into (which should be OK as they can be read-only drivers for the various drives)
The 16-bit loader will also have to create an identifier from the boot device in order to pass this to the 32-bit kernel so that I can find the same boot device using the 32-bit drivers (when the BIOS is no longer available).
It could even be made "intelligent" to the point of identifying (guessing) what device drivers it will need, but it is dependant on the BIOS if it's even possible to get any information about what the booted device is.
Most BIOSes just pretend that a USB hard drive is a normal hard drive, leaving a surprise for the OS when they jump to 32-bit armed with an ATA driver and find no drive to boot from
Subscribe to:
Posts (Atom)