How to hook Native process whose export functions are not available?

Question

I need to solve a reverse engineering problem of a native process. I am having unmanaged .exe of having some controls on it ( e.g TextBox, Buttons, TextAreas, ComboBoxes). After filling all the data on controls User will press "Open".

Actually it will open the modem port and will send the AT commands. I want to check the format of the data and the message which it will send to modem COM port.

So some how I need to reverse engineer the process and hook the functions( most probably the function which "Open" the modem port, and when user clicks "Open", it will be called).

Suggestions?? My directions are right and do I need to hook its functions then after injection, my goal will be achieved.

Note:

No EXPORTED function is detected. I used CFF/PE Explorer for that.

Regards Usman

Answer 1

OllyDbg can help you. You also need some basic knowledge of assembly language and calling conventions. And, of course, the principles of installing interceptor functions (detours).

Answer 2

I see two possibilities. One would be to hook a monitor up to the com port, and just look at what comes out there. This avoids RE of the code at all, and will generally be quite a bit quicker and easier.

[Edit: there are two forms of this: one is hardware -- a logic analyzer (or something on that order) connected to the COM port hardware. It decodes and displays the serial data stream as it goes across the wire. The other form would be software that used the IAT to hook into the call to WriteFile and showed what files were opened, and what data written to each. The Microsoft Detours library can be helpful for this.]

If you decide to RE the code anyway, you'll probably need a decent disassembler such as IDAPro. You'll probably be able to find the parts that deal with the COM ports by starting from places that call CreateFile and WriteFile (assuming you're dealing with Win32, based on the mention of "PE"). You'll probably have something like "\\.\COM " (or at least "COM ") as a literal string somewhere, and you'll want to find a call to CreateFile that uses that string (or a copy of it).

From there, you'll have to work backwards to find the code that puts together the strings that actually get written to the COM port. Hard to guess how complex that will be though.