How can one define a customized security domain on a smart card? Suppose that I have a GSM sim card in my mobile (I don't know operator key set to modify my sim)! Am I correct if I want to install a customized security domain on the same sim card in order to have an independent island on the sim to manage few applets of mine (not the operator)?
What properties should a security domain have? Is that possible to write one or use any pre-written online? Is that possible to install it on sim card without knowing the key set?
First of all, Java Card is a programming language to write applets that run on smartcards with a Java Card RE/VM. Global Platform is a standard for installation and management of smartcard applications, that includes the concept of Security Domains. Many smartcards out there combine both (although that is not necessary) and I assume this is your usecase as well.
There are two types of security domains, the (highest,top-level) Issuer Security Domain and Supplementary Security Domains. Firstly, if you not know the keys of the Issuer Security Domain, you cannot add another Security Domain or any other applets. The ISD is always preinstalled and usually there is a SSD load package on the card, that can be used to install and instantiate a SSD. The Security Domain is just another applet with the Security Domain privilege, but it has the functionality to load/install/manage applets and interact with the JCRE/VM directly. Technically you might be able to add the Security Domain privilege to your self-written applet.
All the information can be found in the freely available Global Platform specification and its needed to read through the pages if you want to dig deeper.