First allow me to say that I don't have any experience developing drivers for OSX, nor drivers for Windows. So, there are a lot of things that I don't understand about how drivers work; I'm sure it'll be evident in my question.
I have a modem that is able to open and close TCP/UDP sockets using AT commands. I would like to create some kind of program (kernel extension? driver?) that implements a network driver, converting the network interface calls into AT command serial messages.
That's the basic jist of it. I'm essentially asking if anybody can point me in the right direction / give me a high level overview of how they would approach it and what Apple guides to focus on.
The XNU networking stack -- like most network stacks -- expects network devices to send and receive IP packets directly. It isn't tooled to work with network devices that handle part of the network stack (like TCP or UDP) internally -- it won't be possible to implement a network driver which uses this device.
You might have more luck exposing this device as a SOCKS proxy. You will need to write a userspace daemon which listens on a TCP port on localhost (on the computer) and relays traffic to the serial device; once that's done, you can set the computer to use that device as a SOCKS proxy in the Networking control panel.
(As an aside: most devices that implement this type of interface have a very low limit on the number of open sockets -- often fewer than 10. They're unlikely to be able to handle the network load generated by a desktop OS.)