Kdb+ data fomat when writing to a file

Question

I'm trying to understand what happen when someone try to write to a file the next way:

q)h:hopen `:out
q)h (1 2;3)
3i
q)hclose h
q)read1 `:out
0x07000200000001000000000000000200000000000000f90300000000000000

this is not the same as a binary representation:

q)-8!(1 2;3)
0x010000002d00000000000200000007000200000001000000000000000200000000000000f90300000000000000

• in which data format (1 2;3) was written into :out?
• is there a way to read data from it back? - like -9!-8!(1 2;3)
• how set/get related to these binary data formats? - do they use another 3-rd different binary data format?

Answer 1

Technically you can read the object if you have a little prior knowledge about the object, such that you can fabricate a header:

q)read1`:out
0x07000200000001000000000000000200000000000000f90300000000000000
q)-9!read1`:out
'badmsg
  [0]  -9!read1`:out
         ^
q)-9!{0x01,0x000000,(reverse 0x0 vs `int$count[x]+1+3+4+1+1+4),0x00,0x00,(reverse 0x0 vs 2i),x}read1`:out
1 2
3

The header here is comprised of:

0x01 - little endian
0x000000 - filler
message length (count of raw `x` plus the header additions)
0x00 - type (generic list = type 0) ... you have to know this in advance
0x00 - attributes .... none here, you would have to know this
length of list .... here we knew it was a 2-item list
x  - the raw bytecode of the object without header 

As rianoc pointed out, there are better ways to write such objects such that they can be more easily read without requiring advanced knowledge

Answer 2

-8! returns the IPC byte representation.

On disk the binary format is different. To read this data use get.

To stream data to a log file you need to create the empty file first https://code.kx.com/q/kb/replay-log/#replaying-log-files

q)`:out set () /This important part adds a header to the file to set the type
`:out
q)h:hopen `:out
q)h (1 2;3)
600i
q)hclose h
q)get `:out
1 2
3

Note if you wish to write your item as a single element to the list then use enlist

q)`:out set ()
`:out
q)h:hopen `:out
q)h enlist (1 2;3)
616i
q)hclose h
q)get `:out
1 2 3

Binary and text data can also be written to a file which is what you are doing.

https://code.kx.com/q/ref/hopen/#files

The intention is to write specific pieces of data

q)h 0x2324 /Write some binary
q)h "some text\n" /Write some text

In your code the binary representation of the raw KX object does get written but no header is added (it is neither IPC nor disk format). So when you read back the data it cannot be interpreted correctly by either -9! or get.

Valid file binary created with `:out set () has a file header: (Can be read by get)

q)read1 `:out
0xff0100000000000000000200000007000200000001000000000000000200000000000000f90300000000000000

Valid IPC binary with IPC header: (Can be read by -9!)

q)-8!(1 2;3) 
0x010000002d00000000000200000007000200000001000000000000000200000000000000f90300000000000000

Your raw object in binary - no header present to enable interpretation

q)read1 `:out
0x07000200000001000000000000000200000000000000f90300000000000000