Reproduce Constrained Hybrid Tree

2017-06-21

Reproduce Experiments with Constrained Hybrid Tree

I directly downloaded code from our official website and followed instruction in README.txt file to rerun the code. But for English corpus (600 instances for training, 280 for testing), I got around 82.1%, which is supposed to be 86.8% according to paper Constrained Semantic Forests for Improved Discriminative Semantic Parsing.

I tried a lot of ways to find out why I got this issue. Finally, Prof. reminded me that it may be caused by Prolog config if I got same results for both text and Geoquery execution accuracy, where I randomly choose 100 English instances as training samples and use the same 280 testing samples as before, like:

incorrect results

Well, in practically, the results for text accuracy and Geoquery execute accuracy should be different. And the results for the latter should be 62.8%.

Previously, the code applied SICSTUS for evaluation. But it has expired date after when it will be chareged, so thereafter, Raymond changed it to Prolog SWIPL which is totally free. Naturally, three are some configuration needed to be made for this modification.

After comparing codes for evaluation with Raymond’s configuration for Neural Hybrid Tree, in GeoqueryEvaluator class, I added some code which were for loop inside outer while loop, as figure shown below:
Code Modification in GeoqueryEvaluator
I feel this is the most important change.

Also some modification for geoquery.pl:
Before:
Before
After:
After

some modification for eval.pl to check “singleton” and “discontiguous”, which is to avoid warnings:
Before:
Before
After:
After
Otherwise, it also puts some warnings, like:

But results remains the same.

Finally, I got correct results.

BTW, do not forget to change the path of Prolog “SWIPL”.

Note: if we continue using “SICSTUS” for evaluation, we may need to set “locale” as default in linux, otherwise we still get incorrect results.