Writing an Operating System in Odin

Setting up to writing an OS kernel is always somewhat tough comparing to writing user-space applications, mostly because the compilers and standard libraries make a few assumptions about the execution environment that aren't necessary true in the space of kernel development.

These assumptions need to be disabled on the compiler level before your kernel can be ran, and what's hideous is that sometimes it can lead to subtle bugs that are really hard to track if you don't know much about what the compilers do with your code.

Today I want to walk you through the starting steps of writing an OS kernel in Odin, covering what you should do in Odin to set up an environment that doesn't rely on the core: collection and can still use the best parts of Odin to make the development process fun.

Bootloader

The process of OS booting starts with a bootloader. You could, of course, write your own bootloader, but I would advise against that, because if you want to write a kernel -- just write a kernel, don't write a bootloader. Though it can be a learning experience on its own.

For this article I will be using version 3 of the limine bootloader. Why version 3? Because it seemed stable and I didn't want to re-write the bindings if some of the ABI changed.

The bootloader understands a couple of boot protocols that we can chose from, but we'll chose the limine-v3 boot protocol, because it's somewhat simpler and provides a bit of functionality that's going to be useful for us.

We'll need to create a disk image, compile our kernel into an ELF file and put it into the disk image and then we can run the emulator on the disk image to run our kernel. But before writing our kernel entry point, let's look at what the default entry points in core:runtime do.

Entry points in `core:runtime`.

We can see a bunch of entry points there, all of which have platform-specific signatures, but they all do similar things. Let's first look at one example of an entry point.

foreign {
  @(link_name="__$startup_runtime")
  _startup_runtime :: proc "odin" () ---
  @(link_name="__$cleanup_runtime")
  _cleanup_runtime :: proc "odin" () ---
}

default_context :: proc "contextless" () -> Context {
  c: Context
  // Sets up allocators and assertion failure procedure
  __init_context(&c)
  return c
}

@(link_name="main", linkage="strong", require)
main :: proc "c" (argc: i32, argv: [^]cstring) -> i32 {
  args__ = argv[:argc]
  context = default_context()
  #force_no_inline _startup_runtime()
  intrinsics.__entry_point()
  #force_no_inline _cleanup_runtime()
  return 0
}

Here's what's happening here:

Get the arguments from the OS (we don't care about this step).
Initialize the context.
Call __$startup_runtime.
Call the application main function.
Call __$cleanup_runtime.

What's __$startup_runtime and __$cleanup_runtime? Those are functions, generated by the Odin compiler, that initialize all the global variables in the program, and call all the @(init) functions defined by our application. Correspondingly __$cleanup_runtime performs the cleanup tasks.

As our kernel is not meant to ever shut down or do any sort of memory management our entry point will not call __$cleanup_runtime.

These functions are defined in core:runtime like this:

foreign {
  @(link_name="__$startup_runtime") _startup_runtime :: proc "odin" () ---
  @(link_name="__$cleanup_runtime") _cleanup_runtime :: proc "odin" () ---
}

Basically we just link our program to itself to obtain symbols with these weird names that we wouldn't have been able to call otherwise.

You can also see that these functions have the "odin" calling convention, which basically means they use/require the context functionality. So we need to initialize the context before calling into these functions. In hosted implementations it's as simple as saying context = {} inside the entry point, but in our case it is a bit more involved.

Enabling SSE

Odin requires SSE to be enabled on x86 processors in order to be able to run. When your kernel starts, the SSE functionality is disabled on your processor by default. In fact, if you did not do anything about it, you would just crash on one of the first instructions of your kernel.

Here's the disassembly for a single line of code, context = {}.

xorps  xmm0,xmm0
movaps XMMWORD PTR [rsp],xmm0
movaps XMMWORD PTR [rsp+0x90],xmm0
movaps XMMWORD PTR [rsp+0x80],xmm0
movaps XMMWORD PTR [rsp+0x70],xmm0
movaps XMMWORD PTR [rsp+0x60],xmm0
movaps XMMWORD PTR [rsp+0x50],xmm0
movaps XMMWORD PTR [rsp+0x40],xmm0
lea    rdi,[rsp+0x40]
mov    QWORD PTR [rsp+0x18],rdi
call   runtime.__init_context-900
movaps xmm0,XMMWORD PTR [rsp]
mov    rdi,QWORD PTR [rsp+0x18]
movaps XMMWORD PTR [rsp+0x90],xmm0
movaps XMMWORD PTR [rsp+0x80],xmm0
movaps XMMWORD PTR [rsp+0x70],xmm0
movaps XMMWORD PTR [rsp+0x60],xmm0
movaps XMMWORD PTR [rsp+0x50],xmm0
movaps XMMWORD PTR [rsp+0x40],xmm0

In short it's using SSE registers to zero-fill a context struct, then calls runtime.init_context, and then zero-fills the struct again. Don't ask, just accept this. It's a quirk that might get fixed in the future.

There are legitimate arguments why you might want to disable SSE in your kernel if you had used a different compiler, in fact the Linux kernel compiles without SSE, because it saves some time when task switching between processes that don't use floating-point math. However with Odin you can not disable SSE, so we'll have to enable it on the CPU ourselves.

Enabling SSE involves a bit of assembly, which I provide below.

cpu x86-64
bits 64

global enable_sse
global halt_catch_fire

section .text

enable_sse:
    ;; Clear CR0.EM and set CR0.MP
    mov rax, cr0
    and ax, 0xfffb
    or  ax, 0x0002
    mov cr0, rax
    ;; Set CR4.OSFXSR and CR4.OSXMMEXCPT
    mov rax, cr4
    or  ax, 3<<9
    mov cr4, rax
    ret

halt_catch_fire:
    cli
.loop:
    hlt
    jmp .loop

I also provide the other function, halt_catch_fire which I'll explain later. But yeah, we'll just have to call enable_sse before initializing the context and we're set.

Our entry point

So let's write some code before getting into the next part of this "tutorial". We'll write our kernel entry point.

foreign import cpu "cpu/cpu.asm"

foreign cpu {
    enable_sse      :: proc "sysv" () ---
    halt_catch_fire :: proc "sysv" () -> ! ---
}

foreign {
    @(link_name="__$startup_runtime") _startup_runtime :: proc "odin" () ---
    @(link_name="__$cleanup_runtime") _cleanup_runtime :: proc "odin" () ---
}

@(export, link_name="_start")
kmain :: proc "sysv" () {
    enable_sse()
    context = {}
    #force_no_inline _startup_runtime()
    // TODO: Do the kernel stuff here.
    halt_catch_fire()
}

You'll notice that we are foreign-importing cpu.asm, although with the current version of the Odin compiler that doesn't actually work because Odin doesn't output a special sections telling the linker that we want the object file for the asm file during linking. It's just there for readability so that we know where these functions are coming from.

Custom `runtime:` library.

Odin comes with some great functionality like run-time type information and bounds checking that may be useful for our kernel. Along that list are also map and [dynamic] arrays. You can't use these directly in freestanding targets, but you can write a custom runtime library and re-define core: collection to a directory inside your kernel.

Basically what you gotta do is copy the files in core:runtime and provide your implementations of them. You don't have the freedom to choose the layout of Context structure etc, because the compiler relies on the exact size and order of the structure fields. What you can change though are the contents of procedures, like the bounds checking procedures or assertion failure procedures. Feel free to hook them to limine's terminal printing and subsequent calling of cpu.halt_catch_fire.

Compiler flags

Here's a list of compiler flags you need to be aware of and use in your kernel:

-target:freestanding_amd64_sysv: Builds the kernel as a freestanding target.
-no-crt: Avoids the usage of libc (C runtime library) by the compiler.
-no-thread-local: Threads @(thread_local) as if it wasn't there. You might want to disable this one if you're actually planning to write a multiprocessing kernel. Just don't use thread locals until you've initialized fs and gs segment registers, okay?
-no-entry-point: Allows us to define a custom entry point.
-reloc-mode:pic: Compiles our code as Position-Independent code. I forgot whether it's actually needed or you can use static as well, guess one more thing you can check!
-disable-red-zone: This one is pretty important -- it disables the System V red zone, a 128-byte space below the stack that functions can use without allocating. Some things like interrupts can run on the kernel stack so we absolutely don't want a random interrupt smashing something compiler put onto the stack, so we just tell the compiler to not put anything on the stack there.

Okay! That's pretty much everything you might need to know about Odin to start writing a kernel.

Sample source

This sample implements a simple kernel entry point that overrides some runtime functions and purposefully does a write out of bounds. You are supposed to see a slice error, which you can remove also.

The source code is on my github.

Writing an OS kernel in Odin