How can I run assembly code by bypassing the operating system?

Question

First of all, I don't know about modern CPUs and operating systems. For this reason, I will explain my explanation over the intel 8085 processor. And of course, I would like you to imagine that there is an operating system that can run on the intel 8085.

we have such assembly code:

MVI A,16
MVI B,16

ADD B

HLT

This code is very simple. When this code runs, it does the following: It loads the number 16 into registers a and b of the intel 8085 processor. And then it adds the value of these two registers.

Of course, when we try to run this code in our operating system, most likely nothing will happen.

What I want to ask is: How can I run a code that does not contain any system calls (or anything operating system specific) on the operating system (by-passing the operating system)? And I don't want the operating system to crash while doing this.

Answer 1

Clearly you are running an operating system, so why not try it yourself?

Processors only know how to run their instruction set and there are countless programming languages you can use to generate those instructions, including assembly language.

so.c

int fun ( int );

int main ( void )
{
    return(fun(5));
}

fun.c

int fun ( int x )
{
    return(x+3);
}

./so.elf ; echo $?
8

so

objdump -d fun.o

fun.o:     file format elf64-x86-64


Disassembly of section .text:

0000000000000000 <fun>:
   0:   8d 47 03                lea    0x3(%rdi),%eax
   3:   c3                      retq   

So now let's use assembly language:

.globl fun
fun:
lea 0x7(%rdi),%eax
retq

./so.elf;  echo $?
12

But now let us move it up a level:

int main ( int argc, char *argv[] )
{
    return(argc+7);
}


0000000000001040 <main>:
    1040:   8d 47 07                lea    0x7(%rdi),%eax
    1043:   c3                      retq   

Switching architectures, because I feel like it...

.globl _start
_start:
adds r0,#4
bx lr

and I get a seg fault (same with x86 doing a retq). As pointed out in the comments some will let you just return but you probably have to make an exit system call. Note I did not crash the operating system. I just crashed that program.

.globl _start
_start:
adds r0,#4
mov r7,#1
swi #0

./fun.elf; echo $?
4

Assembly language is not the real problem but getting the program into the OS and running it. And how you do that normally is creating a binary file that is a file format supported by the operating system (including various linker specific things as to memory space, entry point, etc). Otherwise you have to try to hack the operating system to shove it into memory and then convincing the operating system to run that code.

You could create or choose an operating system for your 8085 that allows a simple return as a way to exit a program, or modify the operating system to allow that, then you can just perform a return from subroutine/function call and it is just a few pure instructions with no system calls.

There is no magic to assembly language, just more freedom. The processor can only run its instructions, it does not know how to run C language or C++, or rust, or python, etc. Just its own instructions. And assembly language is just one programming language you can use. Most of the problem is not the program but the file format, the operating systems rules, and how to exit cleanly.

Answer 2

What I want to ask is: How can I run a code that does not contain any system calls (or anything operating system specific) on the operating system (by-passing the operating system)? And I don't want the operating system to crash while doing this.

Simple computers (such as computers using the 8085 CPU)

On such computers, the operating system is simply a set of functions that are called - either using the call operation or using specific system call operations (on 8086 CPUs: int x).

If you call int 10h (as an example) on an MS-DOS computer, a function (the so-called interrupt handler) is called. This function accesses the graphics card using the out operaration and the video RAM (which is written like any normal RAM; on the 8085, the corresponding instruction would be named stax d).

Your program can "simply" do the same steps the operating system (the interrupt handler) would do instead of calling int 10h.

However, it is not really "simple": I think on an MS-DOS computer, writing some text to the screen might take about 300 instructions (depending if you want to support cursor movement, scrolling, line break handling etc...).

Modern desktop computers (running a modern OS)

Simple answer: You can't.

Modern CPUs have security and protection features:

Such CPUs contain a register that contains the information if the application, an interrupt or the OS is running.

When the application is running, the CPU knows that the application is running; when the application performs a system call (for example using the int x instruction), the CPU knows that the OS is now running.

The protection features of the CPU (such as the memory management unit) do not allow you to access the hardware (such as the display) when the application is running.

For this reason, you must do a system call if you want to access the hardware. (Because the OS is allowed to access the hardware.)