Sigmoid function returns 1 for large positive inputs

Question

I wrote the following function in Python to calculate sigmoid function of a scalar, vector or matrix.

def sigmoid(z):
    sig = 1.0/(1.0 + np.exp(-z))
    return sig

For relatively large positive values of z, e^-z returns a very small value close to zero (0) and hence the value of sig is rounded to 1. My final objective is to determine cost function of logistic regression algorithm. Since sigmoid returns exactly 1, log(1-1) return 'nan'. How can I solve the problem in such a way that my sigmoid() function will return the proper value and not round the e^-z to 0?

When I normalized the input features using mean and standard deviation, it worked fine. But is there a way to make it work with larger values of z?

Also, I tried the same on Matlab and it worked fine without normalization.

>>> Z = np.array([[60, 100],[20, 80]])
>>> Z
array([[ 60, 100],
       [ 20,  80]])
>>> np.exp(-Z)
array([[8.75651076e-27, 3.72007598e-44],
       [2.06115362e-09, 1.80485139e-35]])
>>> 1.0/(1.0 + np.exp(-Z))
array([[1., 1.],
       [1., 1.]])

Answer 1

I overcame this issue by wrapping the sigmoid function with np.minimum & np.maximum:

def sigmoid(x):
    sig = 1 / (1 + np.exp(-x))     # Define sigmoid function
    sig = np.minimum(sig, 0.9999)  # Set upper bound
    sig = np.maximum(sig, 0.0001)  # Set lower bound
    return sig

As a result my losses started looking like this

But both training and test losses converged well & I received ~90% of accuracy on a tiny dataset using logistic regression.

Answer 2

As already mentioned by jdehesa, your issue is with precision limits. You can read more here: https://docs.python.org/2/tutorial/floatingpoint.html

You could try using the Decimal class which avoids rounding:

from decimal import Decimal
import numpy as np
import math

def sigmoid(z):
    sig = Decimal(1.0)/(Decimal(1.0) + Decimal(np.exp(-z)))
    return sig


math.log(Decimal(1)-sigmoid(60))
>>> -59.97257293350302

but this will only work up to a point (I see it already fails for 80). I did some reading and if you really need more precision, you can change the precision of the Decimal object by increasing the value of the following from the default of 28:

from decimal import *
getcontext().prec = 28 

More detail here:

https://docs.python.org/2/library/decimal.html

For most cases though, the suggestion of manually handling small results with a logic check will probably be best.