There are several vim clone packages for Emacs (Evil, Viper..). These packages have a special built in functionality where they remap the Escape key for tty users. I'm not sure how this is done (With a timeout?).
I'd like this functionality as well however I don't want to use the entire Evil / Viper package, I only want the Escape key functionality. This is because I use God Mode, basically Vim's normal / Insert mode with emacs keybindings instead. It doesn't however have a solution for the TTY Escape key like Evil / Viper do. My Elisp isn't really good enough to implement this myself.
So if there is a stand alone package for the escape key issue I'd love to hear about it.
This doesn't /really/ answer my question but I've found something nice that solves my problem.
Xcape is a tool that you can use change the behavior of a key. For example if you run this command:
When pressing Left Control and releasing right away, it will act as Control-G. If you hold down Control it will simply act as Control normally does.
Now if I remap C-g to active God-mode I get the expected behavior in the terminal as well. And you can change the keys to however you want.
As I said this doesn't really answer my question but it does solve the problem. So I'll just leave this answer for other people to find.