Operating System Architecture

There are three broad designs for operating systems that I'm going to talk about. These are monolithic architecture, microkernel, and hybrid microkernel architecture. The key differences among these is how the operating system handles separating which processes run in user mode and which run in privileged, kernel mode. As will be shown, the decision has a strong impact on security and performance.

Monolithic architecture

In a monolithic architecture, the kernel is made up of all the operating system processes. This means that any service provided by the operating system (file management, interprocess communication, I/O management, etc) is run in a privileged state. Essentially, the operating system behaves as a software layer between the user applications and the hardware. This is efficient for the processor as it requires very few mode transitions to be made. A mode transition must be executed anytime a process with a different PWS (privilege word setting) is executed. It takes time to make this transition, so avoiding them is generally good for performance.

There are, however, several problems with this design as well. Processes within a monolithic architecture often communicate on an ad hoc basis which provides little for access control. It also results in very complex code which can make debugging or updating very difficult. And because the operating system works directly with the hardware, it can be very difficult to port the operating system to other systems.

There is one monolithic architecture model that helps to tackle some of these problems. That is the layered operating system model. In a layered operating system model, all of the operating system's processes are still part of the kernel, but they have been compartmentalized and have structured ways of interacting.

The image to the left shows the general structure of a layered operating model. Each layer has the ability to communicate with the layer directly above and below it. This provides a more structured way of communicating, and provides data hiding which means instructions and data at various levels do not have access to the instructions and data at other levels. And since the code is now more modular, it is much easier to upgrade or perform maintenance. There are still security concerns with this model, however, because so much code is given kernel privileges.

Microkernel and Hybrid Microkernel Architecture

Under the microkernel model, much of the code was stripped from the kernel and placed within the user space. This increased security because it meant that less code would be able to run in privileged mode. All that was left in the kernel space was key operating procedures such as memory management and interprocess communication. The overall goal of the microkernel model was to limit the processes that run in kernel mode to improve security, reduce complexity, and increase portability.

Engineers found that the microkernel suffered a major setback in performance, however. Because so many of the processes were now in user space, the CPU would constantly have to perform mode transitions. In response, the hybrid microkernel architecture was created.

In a hybrid model, the microkernel still exists. And as before, it is responsible for carrying out interprocess communication and memory management. The other operating system processes that had previously been relegated to user space were moved to an area contained within the kernel known as executive services. These services, such as file management, I/O management, power management, and so on, now operate on a client/server model. This means that whenever a user application wants to make use of one of these services, they must request it using the service API. The service will then carry out the request and return the result when finished. A sample hybrid microkernel model is below. This picture illustrates the architecture of the Windows NT operating system.