I see that people often remove special characters like () "" : [] from data before training translation machine. Could you explain for me the benefits of doing so?
Why special characters like () "" : [] are often removed from data before training translation machine?
798 Views Asked by phan-anh.tuan At
1
There are 1 best solutions below
Related Questions in NLP
- Seeking Python Libraries for Removing Extraneous Characters and Spaces in Text
- Clarification on T5 Model Pre-training Objective and Denoising Process
- The training accuracy and the validation accuracy curves are almost parallel to each other. Is the model overfitting?
- Give Bert an input and ask him to predict. In this input, can Bert apply the first word prediction result to all subsequent predictions?
- Output of Cosine Similarity is not as expected
- Getting an error while using the open ai api to summarize news atricles
- SpanRuler on Retokenized tokens links back to original token text, not the token text with a split (space) introduced
- Should I use beam search on validation phase?
- Dialogflow failing to dectect the correct intent
- How to detect if two sentences are simmilar, not in meaning, but in syllables/words?
- Is BertForSequenceClassification using the CLS vector?
- Issue with memory when using spacy_universal_sentence_encoder for similarity detection
- Why does the Cloud Natural Language Model API return so many NULLs?
- Is there any OCR or technique that can recognize/identify radio buttons printed out in the form of pdf document?
- Model, lexicon to do fine grained emotions analysis on text in r
Related Questions in TOKENIZE
- How to solve Config validation error when tokenizer is not callable in Python?
- SpanRuler on Retokenized tokens links back to original token text, not the token text with a split (space) introduced
- Altova Mapforce - How to use results from Tokenize at the same time in a database call?
- How do handle compound nouns (animal names) in word2vec (using tensorflow)?
- Tensorflow tokenizer question. What num_words does exactly?
- Issues with Training RoBERTa Model for Text Prediction with Fill Mask Task in Python
- Getting `ValueError: as_list() is not defined on an unknown TensorShape.` when trying to tokenize as part of the model
- Trying to run the LLama-2-7B-chat-GGUF on local machine but getting NotImplementedError
- why Tokenizer and TokenizerFast encode the same sentence get different result
- Compare vocabulary size of WordPiece and BPE tokenizer algorithm
- Why did I install Chinese tokenization package but the terminator kept saying I should install them?
- Pre-training or using the existing model of FastText?
- integrate `openai-whisper` tokenizer with spaCy
- Paytabs recurring payment request with token is giving [422 - Unable to process your request] error
- How to remove last N tokens in a string with XSLT?
Related Questions in MACHINE-TRANSLATION
- Is Speech-to-Text-to-Translation an Impossible Dream?
- Translate value before sending information
- State of the art language translation toolkit
- How to speedup mkcls step in mgiza++ or giza++, it taking up lots of time for word clustering?
- Tensorflow seq2seq `feed_previous' argument`
- How to differentiate between real improvement and random noise?
- What is the difference between ensembling and averaging models?
- What is the difference between mteval-v13a.pl and NLTK BLEU?
- finetuning tensorflow seq2seq model
- What do the counts in giza++ phrase-table mean?
- How to deal with punctuations in machine translation
- Aligning Parallel Sentences for Languages without a Written Form
- Sockeye WMT German to English news translation stopping criterium
- Why special characters like () "" : [] are often removed from data before training translation machine?
- Text language translation in android
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Date clean-up or pre-processing is performed so that algorithms could focus on important, linguistically meaningful "words" instead of "noise". See "Removing Special Characters":
Whenever this noise finds its way into a model, it can produce output at inference, that contains these unexpected (sequences of) characters, and even affect overall translations. It is a frequent case with brackets in Japanese translations.