I am looking for an explanation how to protect an algorithm say a function mysort(param x, param y) in the context of Intel SGX such that outside my enclave mysort() function is encrypted and it only decrypts inside enclave.
How could I achieve this ? Where should I write mysort function i.e. in the enclave project or app project and call through edl interface.
How should I encrypt it, where should I keep the key and the logic to decrypt it.
On
You can achieve this by sending an encrypted library after you established a secure encrypted channel with the remote enclave through attestation. Then decrypt the library within the enclave and run your algorithm. Since the code would only be accessible within the enclave your function logic will be protected.
Intel SGX does not guarantee protection against reverse engineering the signed libraries. They might be disassembled.
See: https://software.intel.com/en-us/forums/intel-software-guard-extensions-intel-sgx/topic/738239