Error "Connection not allowed by ruleset" with Tor socks proxy

Question

I am using tor socks proxy to retrieve webpage. It's working perfectly on using with some domain name like google.com, but on using with IP address it gives this error

Connection not allowed by ruleset

Is it because Tor proxy doesn't allow to enter IP address directly assuming that user may have resolved the domain's IP address without Tor and their identity can be revealed? If yes, how can I work around this? Or there is something else going on?

int port = onionProxyManager.getIPv4LocalHostSocksPort();
int proxyPort = port;
String proxyHost = "127.0.0.1";
String remoteHost = "https://97.X.X.X";//"google.com";
int remotePort = 8080;

Socks5Proxy socks5Proxy = new Socks5Proxy(proxyHost, proxyPort);
socks5Proxy.resolveAddrLocally(false);
Socket socket = new SocksSocket(socks5Proxy, 
InetAddress.getByName(remoteHost), remotePort);