I took over some code from someone and my task was to reproduce the same model and performance in pytorch. I was given best hyper-parameters for that model as well. After playing around with it for quite sometime, I see that if I set drop out rate to zero, then my test performance is the best. it stops earlier, the smaller dropout is. In fact it outperforms previous model a bit. While the previous model had significant dropout rates.
The data size is about 33 million rows. And the neural net is like 4-5 layers. Total input embedding is ~1000.
Though I am happy to see the performance, I am wondering if it is some kind of red flag. Because without dropout i don't really have any other regularization. And it performing good with this may mean there might be some data leakage or something?
Looking for some wisdom around dropout in this context.
Neural net: no dropout gives the best test score. Is that bad?
527 Views Asked by obi wan At
1
There are 1 best solutions below
Related Questions in MACHINE-LEARNING
- How to cluster a set of strings?
- Enforcing that inputs sum to 1 and are contained in the unit interval in scikit-learn
- scikit-learn preperation
- Spark MLLib How to ignore features when training a classifier
- Increasing the efficiency of equipment using Amazon Machine Learning
- How to interpret scikit's learn confusion matrix and classification report?
- Amazon Machine Learning for sentiment analysis
- What Machine Learning algorithm would be appropriate?
- LDA generated topics
- Spectral clustering with Similarity matrix constructed by jaccard coefficient
- Speeding up Viterbi execution
- Memory Error with Classifier fit and partial_fit
- How to find algo type(regression,classification) in Caret in R for all algos at once?
- Difference between weka tool's correlation coefficient and scikit learn's coefficient of determination score
- What are the approaches to the Big-Data problems?
Related Questions in NEURAL-NETWORK
- How to choose good SURF feature keypoints?
- How to avoid overfitting (Encog3 C#)?
- Run out of VRAM using Theano on Amazon cluster
- Calculating equation from image in Java
- Print output of a Theano network
- Torch Lua: Why is my gradient descent not optimizing the error?
- How can I train a neural (pattern recognition) network multiple times in matlab?
- Using Convolution Neural Net with Lasagne in Python error
- Random number of hidden units improves accuracy/F-score on test set
- Matlab example code for deep belief network for classification
- Pybrain Reinforcement Learning Example
- How to speed up caffe classifer in python
- Opencv mlp Same Data Different Results
- Word2Vec Data Setup
- How can I construct a Neural Network in Matlab with matrix of features extracted from images?
Related Questions in DROPOUT
- Tensorboard and Dropout Layers
- Can't visualize the dropout layer in model summary using this method
- What happens if my Dropout is too high? what Dropout to use on my 2048-neuron-dense layer? (very little data)
- Adding Dropout layer after every activation layer in a pre-trained resnet model in tensorflow-2
- Neural net: no dropout gives the best test score. Is that bad?
- ReLu and Dropout in CNN
- Cannot generate random ints inside TensorFlow2.0 "call" function
- Tensorflow Model Underpredicts Values with Dropout
- Randomly set some elements in a tensor to zero (with low computational time)
- PyTorch: Why create multiple instances of the same type of layer?
- Why does Dropout not change my input tensor?
- How can I use dropout in keras
- Pytorch: nn.Dropout vs. F.dropout
- calculating accuracy for Monte carlo Dropout on pytorch
- What layers are affected by dropout layer in Tensorflow?
Related Questions in REGULARIZED
- In Keras what is the difference between weight_regularizer and activity_ regularizer
- How to implement regularization / weight decay in R
- Efficient Way to Choose The Regularization Parameter
- How do I exclude bias from regularization using TensorFlow's `matrix_solve_ls`?
- TensorFlow: How to change keep_prob for Dropout without using a feed_dict
- Spatial Regularization
- Can't seem to implement L2 regularization correctly in Python — low accuracy scores
- Neural net: no dropout gives the best test score. Is that bad?
- How can I break down large grid searches?
- Choosing the regularization parameter
- Understanding Lasso and Ridge Regression
- LSTM Regularizers
- Regularized regression using glmnet: No difference between groups?
- Changing regularization factor during training in Tensorflow
- How to perform weight regularization in pytorch?
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Sometimes, these things happen. Once my neural-net was not working and so I was advised to add Batch Normalization layers in it, and then it worked so well. But then in another problem, Batch Normalization made my neural-net worse. This is all because of backpropagation. Sometimes, adding some layer makes the neural-net get stuck in a local-minimum, while sometimes it helps to get out of it. I am not quite sure why is this, but I think it is because of BackPropagation.
The answer is no. Its just because of backpropagation.
NOTE - If you feel I am wrong anywhere in this post, then please comment it.