I have created a java application on http://wso2.com/. they have assigned a custom url
http://roaster.techgeeks.wso2apps.com
I tried to add me own domain name. but I always get "Fail to verify custom URL. " error.
How do I configure my domain to point to this application. do I have to set DNS or CNAME, if yes, what are they?
PS: do not ask to set url redirection. ;)
Assuming you are talking about apps hosted on WSO2 App Cloud:
Your screenshot seems to show wrong value specified as the CNAME record Name.
The name of the record should be your custom domain: e.g. myapp.mydomain.com. The target indeed should be the wso2 URL: in your case roaster.techgeeks.wso2apps.com.