GIZA++ output missing *.ti.final and *actual.ti.final files

Question

I am having issues understanding the basics of how to run the GIZA++.

I went through the discussion, here on StackOverflow (Is there a tutorial about giza++?) and through the links people provided there. I have downloaded and compiled the latest giza from the Moses-SMT Github.

git clone https://github.com/moses-smt/giza-pp.git
cd giza-pp
make

After the successful compilation I have written a simple script for testing purposes.

#!/bin/bash
SRC=french
TRG=english
PREFIX=out
GIZA=../giza-pp

# Cleaning from previous run ...
rm -f *.log
rm -f *.vcb
rm -f *.snt
rm -f *.vcb.classes
rm -f *.vcb.classes.cats
rm -f *.gizacfg
rm -f *.cooc
rm -f ${PREFIX}*

# Converting plain text into sentence format using the "plain2snt.out" tool ...
${GIZA}/GIZA++-v2/plain2snt.out ${SRC} ${TRG}

# Generating word clusters using the "mkcls" tool ...
${GIZA}/mkcls-v2/mkcls -p${SRC} -V${SRC}.vcb.classes
${GIZA}/mkcls-v2/mkcls -p${TRG} -V${TRG}.vcb.classes

# Generating coocurrence using the "snt2cooc" tool ...
${GIZA}/GIZA++-v2/snt2cooc.out ${SRC}.vcb ${TRG}.vcb ${SRC}_${TRG}.snt > ${SRC}_${TRG}.cooc

# Running "GIZA++" ...
${GIZA}/GIZA++-v2/GIZA++ -S ${SRC}.vcb -T ${TRG}.vcb -C ${SRC}_${TRG}.snt -CoocurrenceFile ${SRC}_${TRG}.cooc -o ${PREFIX} >> giza.log 2>&1

Now this is the content of the directory right after I run the script.

jakub@jakub-virtual-machine:~/Master/giza-pp_test$ ls
english                   french_english.snt       out.d3.final        out.perp
english_french.snt        french.vcb               out.d4.final        out.t3.final
english.vcb               french.vcb.classes       out.D4.final        out.trn.src.vcb
english.vcb.classes       french.vcb.classes.cats  out.Decoder.config  out.trn.trg.vcb
english.vcb.classes.cats  giza.log                 out.gizacfg         out.tst.src.vcb
french                    out.a3.final             out.n3.final        out.tst.trg.vcb
french_english.cooc       out.A3.final             out.p0_3.final      run_test.sh

The point is that the output is missing (for me important) files listed below.

out.ti.final
out.actual.ti.final

Now I've been looking to the GIZA's Main.cpp (lines: 260 - 273) and can see the lines that should be creating these files.

cerr << "writing Final tables to Disk \n";
  string t_inv_file = Prefix + ".ti.final" ;
  if( !FEWDUMPS)
    m1.getTTable().printProbTableInverse(t_inv_file.c_str(), m1.getEnglishVocabList(), 
                     m1.getFrenchVocabList(), 
                     m1.getETotalWCount(), 
                     m1.getFTotalWCount());
  t_inv_file = Prefix + ".actual.ti.final" ;
  if( !FEWDUMPS )
    m1.getTTable().printProbTableInverse(t_inv_file.c_str(), 
                     eTrainVcbList.getVocabList(), 
                     fTrainVcbList.getVocabList(), 
                     m1.getETotalWCount(), 
                     m1.getFTotalWCount(), true);

I am also having the "cerr" line printed in the log, but I just cannot find out why these files are not present within the output.

jakub@jakub-virtual-machine:~/Master/giza-pp_test$ cat giza.log | tail -n 25
p0_count is 4.0073 and p1 is 5.99635; p0 is 0.400584 p1: 0.599416
Model4: TRAIN CROSS-ENTROPY 0.80096 PERPLEXITY 1.74226
Model4: (10) TRAIN VITERBI CROSS-ENTROPY 0.801289 PERPLEXITY 1.74266
Dumping alignment table (a) to file:out.a3.final
Dumping distortion table (d) to file:out.d3.final
Dumping nTable to: out.n3.final

Model4 Viterbi Iteration : 10 took: 0 seconds
H3333344444 Training Finished at: Fri Oct 23 16:24:44 2015


Entire Viterbi H3333344444 Training took: 0 seconds
==========================================================
writing Final tables to Disk 
Writing PERPLEXITY report to: out.perp
Writing source vocabulary list to : out.trn.src.vcb
Writing source vocabulary list to : out.trn.trg.vcb
Writing source vocabulary list to : out.tst.src.vcb
Writing source vocabulary list to : out.tst.trg.vcb
writing decoder configuration file to out.Decoder.config

Entire Training took: 0 seconds
Program Finished at: Fri Oct 23 16:24:44 2015

==========================================================

Did someone please run into a similar problem please? Is this some kind of bug or I am doing something wrong?

Edit:

Now I have recompiled the whole GIZA++ without the -DBINARY_SEARCH_FOR_TTABLE option within the CFLAGS in a Makefile. And changed the script so that it won't generate and provide the coocurrence file to the GIZA++. After I have re-ran the script the output did contain the out.actual.ti.final and out.ti.final. Does anybody know how to explain this behaviour? I taught that I would get a better allignment and probability estimates using the coocurrence file, is it of any need please? Or is it only for improving the speed of the performance?

Answer 1

I faced the same issue before . I think the missing step is In the Makefile located at .\giza-pp\GIZA++-v2\, substitute the line: CFLAGS_OPT = $(CFLAGS) -O3 -funroll-loops -DNDEBUG -DWORDINDEX_WITH_4_BYTE -DBINARY_SEARCH_FOR_TTABLE -DWORDINDEX_WITH_4_BYTE

with the line: CFLAGS_OPT = $(CFLAGS) -O3 -funroll-loops -DNDEBUG -DWORDINDEX_WITH_4_BYTE -DWORDINDEX_WITH_4_BYTE

Check this put and good luck