The debate between C and Rust in the Linux community is heating up. The battle has begun to spread beyond Linux circles, with Rust—armed with its promise of memory safety—making an aggressive push, while the old-school stalwart C remains calm and steadfast, holding its unshakable position in production environments.
In the recently influential blog post "The Path to Memory Safety is Inevitable", the author explains the necessity of memory safety and explores practical paths forward—all from the perspective of an old-school hacker. However, the article ends by raising an important question: "If you have thoroughly considered this approach and decide to proceed, please consider rewriting them in Lisp/Scheme."
The article ends abruptly, leaving readers wanting more. Why not simply consider a full rewrite? Why not C++? What are the real pros and cons of writing an operating system in Lisp or Scheme? And most importantly—shouldn’t we just rewrite that seemingly irreplaceable operating system in Rust?
Of course, many rational minds would argue that rewriting an operating system is not a practical solution. It involves countless technical details, significant engineering effort, and massive financial cost. However, there's no harm in imagining the possibilities through a thought experiment. It doesn’t cost anything, and it doesn’t hurt anyone. So in this article, we’ll explore a simple question: What if, just what if, we rewrote the operating system in a language other than C? Setting aside all social, economic, and political concerns, and focusing purely on the technical side—what might that look like?
Has the first principle of computer science really been overshadowed by nothing more than a so-called “benevolent dictator”? Or could it be that the benevolent dictator understands the first principles better than most?
What are those "first principles"? Open the wizard book, The Structure and Interpretation of Computer Programs (SICP) by Harold Abelson and Gerald Jay Sussman. There're three main abstracts introduced in this book:
- Data abstraction
- Procedural abstraction
- Machine abstraction
Those are first principles of computer science, and they are essential for understanding how any programming language operates at a fundamental level. All these abstractions are crucial for dealing with the complexities of programming and system design, and they form the foundation upon which modern programming languages are built. It’s important to recognize that the essence of computer science’s first principles lies in the fight against complexity. With this perspective, the existence of serious programming languages starts to make sense.
Hopefully, by the end of this article, you'll have arrived at an answer you’re satisfied with.
Now that someone has brought up the long-forgotten Lisp family, let’s start from there.
What if we rewrite OS in Lisp/Scheme?
There were many implementations of operating systems in Lisp/Scheme in the history, so let's not doubt if it's possible or any benefits of doing so. The real question is: what are the pros and cons of writing an operating system in Lisp/Scheme? And why would we even consider it in the first place?
What is Lisp/Scheme?
Lisp, short for "LISt Processing," is one of the oldest high-level programming languages, dating back to the late 1950s. It is known for its unique parenthetical syntax and powerful macro system, which allows developers to manipulate code as data. Lisp's flexibility and expressiveness make it a favorite among researchers and enthusiasts in artificial intelligence and symbolic computation.
However, Lisp itself once stood as a systems-level language, which means it's entirely feasible to use it for operating system development. Lisp's dynamic nature and garbage collection capabilities can be advantageous for certain types of applications, but they also introduce challenges in terms of performance and low-level hardware interaction. Of course, this doesn’t mean that Lisp inherently has performance issues. Rather, it means that extra effort is needed in compilation and optimization to narrow the performance gap between Lisp and lower-level languages like C.
Scheme is a dialect of Lisp that emphasizes simplicity and minimalism. It retains many of Lisp's core features while introducing a more streamlined syntax and semantics. Scheme is often used in academic settings and is known for its clean design and powerful functional programming capabilities. However, modern Scheme has far surpassed what people once understood it to be—though that’s a topic that likely deserves an article of its own.
The architecture of Lisp/Scheme machine
When we consider Lisp/Scheme as systems-level languages, we must first confront an important concept: the Lisp machine. A Lisp machine isn’t a physical device, but rather an abstract Turing machine designed to implement lambda calculus.
Why do we need to understand a bit about the architecture of the Lisp machine before talking about operating systems? Because when you're writing an OS, you have to consider a fundamental question: how does your programming language interact with the underlying hardware?
C doesn’t have this problem. From the very beginning, it was designed almost as a thin abstraction over the Turing machine model, with a bit of structured programming syntax layered on top. That’s why C is naturally well-suited for low-level, hardware-oriented programming.
Lisp, on the other hand, represents a completely different approach to language design. It’s based on lambda calculus, yet it must run on a Turing machine model—unless you're using dedicated Lisp machine hardware. When running on architectures like x86 or ARM, Lisp essentially operates within the abstraction of a Lisp machine, which then maps lambda calculus onto the Turing model.
As a result, Lisp becomes a high-level language when viewed from the perspective of a Turing machine. The additional layers of abstraction introduced by this design must be carefully optimized away during compilation in order to ensure efficient execution on conventional hardware.
However, even if you can optimize Lisp to run efficiently on a Turing machine, you still face the challenge of implementing low-level primitives. This is where the real complexity lies.
What is primitives?
In the context of programming languages, "primitives" refer to the basic building blocks or fundamental operations that a language provides. These are typically low-level operations that can be directly mapped to hardware instructions or system calls. In Lisp/Scheme, primitives might include operations like arithmetic calculations, memory allocation, and basic data structure manipulations.
When designing an operating system in Lisp/Scheme, understanding and defining these primitives is crucial. They serve as the foundation upon which higher-level abstractions and functionalities are built. The efficiency and performance of the OS will heavily depend on how well these primitives are implemented and optimized.
The point is, you can't implement primitives in Lisp/Scheme without considering the underlying hardware architecture. You have to write them with C or assembly language, or at least with a language that can directly interface with the hardware. This is because Lisp/Scheme, being high-level languages, do not provide direct access to low-level hardware operations.
Pros and cons of writing an OS in Lisp/Scheme
Pros:
- Lisp/Scheme's powerful macro system allows for expressive and flexible code generation, which can lead to more concise and maintainable code.
- The dynamic nature of Lisp/Scheme can facilitate rapid prototyping and development, enabling developers to quickly iterate on ideas and concepts.
- The garbage collection capabilities of Lisp/Scheme can help manage memory automatically, reducing the risk of memory leaks and dangling pointers.
Cons:
- Performance can be a significant concern, as Lisp/Scheme's dynamic features and garbage collection may introduce overhead compared to lower-level languages like C. That said, modern Lisp/Scheme implementations have made significant strides in performance optimization. This is big challenge for the compiler writers. But sounds like it's a sweet challenge for them and not a real cons maybe ......
- Low-level hardware interaction can be cumbersome, as Lisp/Scheme does not provide direct access to hardware operations. This means that developers must rely on C or assembly language for implementing low-level primitives, which can complicate the development process.
Conclusion
You still need C or assembly language to implement low-level primitives in Lisp/Scheme. This is a significant drawback, IN THIS ARTICLE, as it means that you cannot fully leverage the benefits of Lisp/Scheme without relying on lower-level languages for critical parts of the operating system.
Please remember, we are here to compare the languages in implementing an operating system, Lisp would have to be mixed with C in order to build a complete operating system—which clearly goes against the spirit of this article. After all, how could we call it a rewrite if we're still relying on C? Let’s not forget: our goal here is to eliminate C entirely.
So, let's forget about Lisp/Scheme, what about C++?
What if we rewrite OS in C++?
C++ is a powerful, high-level programming language that builds upon the foundations of C while introducing object-oriented programming (OOP) features. It retains the low-level capabilities of C, allowing for direct hardware manipulation, while also providing abstractions that can lead to more organized and maintainable code.
Linus on C++
It’s inevitable to bring up Linus’s criticism of C++, as it’s a first-hand opinion from someone who has built a real, production-grade operating system.
Briefly, The core issue is that the very features C++ prides itself on were never designed with operating system development in mind. So from Linus’s perspective, these otherwise excellent design choices amount to nothing more than bullshit. Clearly, this is not an emotional outburst, but rather a rational critique from the standpoint of kernel-level development. Personally, I believe there’s nothing unreasonable about his entire commentary—aside from perhaps the word "bullshit", which some may find disrespectful. But even then, it’s worth noting that he wasn’t the first to use the term in the context.
Zero-cost abstraction
C++ is often praised for its "zero-cost abstraction" philosophy, which means that the abstractions provided by the language do not incur additional runtime overhead compared to lower-level constructs. This is achieved through features like inline functions, templates, and operator overloading, which allow developers to write high-level code that compiles down to efficient machine code.
However, this philosophy can be a double-edged sword. While it allows for powerful abstractions, it can also lead to complex and hard-to-understand code. In the context of operating system development, where performance and predictability are paramount, the potential for subtle performance pitfalls in C++ code can be a significant concern.
In addition, the complexity of C++ can make it more difficult to reason about code behavior, especially in a concurrent environment like an operating system. This can lead to bugs that are hard to track down and fix, which is a critical issue when developing software that must run reliably on a wide range of hardware.
However, zero-cost abstractions are primarily designed to accommodate the frequent changes found in large-scale business software. They provide a balance between flexibility and performance, allowing such systems to evolve without breaking down. But when it comes to operating system kernel design, this philosophy holds far less value.
To meet the strict demands of stability and performance required at the kernel level, many of these features are essentially unusable. For instance, exception handling is simply not an option in low-level development. If you were to write an OS in C++, you'd have to rely on what's known as freestanding C++, and you’d need to disable features like exception handling and RAII via compiler flags. Because these features relies on stack unwinding and runtime support, which is missing when you're in the bearmetal world of operating system development.
-ffreestanding -fno-exceptions -fno-rtti
Object-oriented programming
Let’s be blunt: object-oriented programming is of little to no value in operating system kernel development. OS development is not game development—you don’t need to abstract everything into objects just to make things easier to manage, especially at the cost of added abstraction layers.
The strength of object-oriented design lies in its ability to encapsulate entities and manage complex interactions or unpredictable changes between them. This is extremely useful in large-scale business systems, and particularly in game development, where flexibility and dynamism are crucial.
But OS development is a different beast. The interactions between components are strictly defined and must not change arbitrarily. In fact, OS design is necessarily constrained by rigid principles to ensure safety and performance.
Seen in this light, Linus’s criticism of C++ from the perspective of kernel development makes perfect sense—it’s not that C++ is a bad language, but that its features simply don’t align well with the needs of OS-level programming.
Modern C++ is pretty good, but ...
Modern C++ has made significant strides in improving safety and performance, with features like smart pointers, type inference, and improved template meta-programming. These advancements can help mitigate some of the issues associated with traditional C++ programming, such as memory leaks and complex template code.
Ironically, the direction modern C++ is evolving in feels like it’s trying to reinvent a more powerful version of Scheme—just with a lot more complexity.
And how many of these features can you actually use in operating system development? The answer is: not many. Most of the modern C++ features are designed for high-level application development.
But there's still one important feature, unique_ptr. It provides a way to manage resources automatically, reducing the risk of memory leaks and dangling pointers. However, this feature is not sufficient to overcome the fundamental limitations of C++ in the context of operating system development.
If you're using a C++ subset in kernel space, you have to avoid exceptions, RTTI, and dynamic casts, these were mentioned in our previous section. However, make_unique throws on failure. If exceptions are disabled (as is typical in kernel), you must not rely on make_unique.
See the contradiction here? You can’t use the very feature that makes C++ safer in the context of operating system development. This is a significant drawback, as it means that you cannot fully leverage the benefits of modern C++ without relying on lower-level languages for critical parts of the operating system.
Wait, someone may have told you that unique_ptr and Rust’s linear (or more accurately, ownership-based) type system are conceptually similar. Here's a simple comparison:
So how about to use Rust?
What if we rewrite OS in Rust?
What is Rust?
Rust is a systems programming language that aims to provide memory safety, concurrency, and performance without sacrificing low-level control. It was designed to address many of the shortcomings of C and C++, particularly in terms of memory management and safety.
Rust's ownership model, which enforces strict rules about how memory is accessed and managed, allows developers to write safe and efficient code without the need for garbage collection or manual memory management. This makes Rust particularly well-suited for operating system development, where performance and reliability are critical.
Really?
Yes, it is—at least in theory. But in this article, we're exploring the idea of rewriting an operating system in an imaginary world, where even theoretical perfection counts as actual perfection. And since we've already set aside the social, economic, and political concerns of rewriting an OS, we might as well ignore the complexity of compilers, tooling and software supplychain too.
Memory safety
Rust's memory safety guarantees are one of its most significant advantages. The language's ownership model ensures that memory is automatically managed without the need for a garbage collector, which can introduce latency and unpredictability in performance. This is particularly important in operating system development, where low-level control over hardware resources is essential.
Yes, no doubt it is true in both theory and practices. Rust win a point due to its strong compile time checking for memory safety. This is a significant advantage over C and C++, since these languages are lacking of intrinsic features to do so. You have to take advantage of external tools like Sanitizer or static analysis tools to achieve the same results. So yes, Rust is a clear winner here.
However, compile-time checks doesn't mean full memory safety. In general, compile time checking prevents issues like use-after-free, double free, null pointer, dereference, data races (via Send/Sync traits), etc. But for runtime issues like buffer overflow, out-of-memory (OOM) conditions, indexing and panics, etc, Rust still relies on runtime checks. This means that while Rust can catch many memory-related issues at compile time, it cannot guarantee complete memory safety in all scenarios.
Unfortunately, OS development needs to care about runtime issues especially when it comes to buffer overflow and OOM conditions. These are critical issues that can lead to system crashes or security vulnerabilities. While Rust's runtime checks can help mitigate these risks, they do not eliminate them entirely.
Unsafe
Rust’s safe subset is designed to isolate you from undefined behavior. That includes:
- Arbitrary pointer arithmetic
- Access to hardware registers (MMIO)
- Inline assembly
- Interrupt handlers
- Direct memory access (DMA)
- Syscall trampolines
- Paging, segmentation, GDT/IDT manipulation
Well, all these are essential for operating system development. You can’t write an OS without them. So, you have to use the unsafe subset of Rust to implement these low-level primitives.
Linus on Rust
Linus Torvalds has expressed skepticism about Rust's suitability for operating system development, particularly in the context of the Linux kernel. His concerns primarily revolve around the complexity and potential performance overhead introduced by Rust's safety features, as well as the challenges of integrating Rust with existing C code in the kernel.
But let's be clear: Linus is not against Rust per se. He has acknowledged the potential benefits of Rust, particularly in terms of memory safety and concurrency. However, he remains cautious about its adoption in the kernel, emphasizing the need for careful consideration and thorough testing before integrating Rust into the Linux codebase.
What if we rewrite OS in Assembly?
All those concerns are off the table—completely. In theory, this is the most perfect approach we could imagine in this world.
After all, this article is about exploring perfection on paper, and nothing more.
Nice, but when?
It seems that even on paper, a perfect language for OS development remains elusive. So let’s accept reality—and dance with the wolves.
If you dream of rewriting an operating system without C, then don’t wait for the perfect language to arrive. It won’t. Stop waiting. Start building. Language features won't save you—only your resolve will.
LispOS? Go for it!
C++OS? Go for it!
RustOS? Go for it!
But what about AI? What about languages like Haskell, or even Java?
Perhaps the answer is still out there. I still believe that with the right ideas, we can one day confront complexity—not with fear, but with elegance. There is still hope for clarity, even in the most chaotic depths of system software.
