文本摘要评测工具ROUGE的搭建和测试

　　文本摘要任务中最常用的评价方法是ROUGE(Recall-Oriented Understudy for Gisting Evaluation)。ROUGE受到了机器翻译自动评价方法BLEU的启发，不同之处在于，采用召回率来作为指标。基本思想是将模型生成的摘要与参考摘要的n元组贡献统计量作为评判依据。

　　现在主要采用软件是PERL语言编写的版本，见地址：

（https://github.com/andersjo/pyrouge/tree/master/tools/ROUGE-1.5.5）

　　然而这个工具的搭建，相对还比较麻烦，因此把整个搭建的过程记录一下：

（1）安装PERL语言，一般的Ubuntu环境都具备

（2）安装PERL语言的相关库，主要是XML语言解析器

（3）对数据进行处理，主要是WordNet数据的处理，主要是原来给的文件会存在无法打开的问题，即报如下错误：（Cannot open exception db file for reading: data/WordNet-2.0.exc.db）

处理步骤如下：

cd pythonrouge/RELEASE-1.5.5/data/
rm WordNet-2.0.exc.db
./WordNet-2.0-Exceptions/buildExeptionDB.pl ./WordNet-2.0-Exceptions ./smart_common_words.txt ./WordNet-2.0.exc.db

然后进行测试：

./ROUGE-1.5.5.pl -e data -c 95 -2 -1 -U -r 1000 -n 4 -w 1.2 -a ROUGE-test.xml

　其中测试文件可以从如下网址下载：ROUGE-test.xml (https://raw.githubusercontent.com/summanlp/evaluation/master/ROUGE-RELEASE-1.5.5/sample-test/ROUGE-test.xml)

测试呈现的结果如下：

omnisky@omnisky:~/software/ROUGE-1.5.5$ ./ROUGE-1.5.5.pl -e data -c 95 -2 -1 -U -r 1000 -n 4 -w 1.2 -b 75 -m -a ROUGE-test.xml
---------------------------------------------
11 ROUGE-1 Average_R: 0.22536 (95%-conf.int. 0.18124 - 0.27016)
11 ROUGE-1 Average_P: 0.20359 (95%-conf.int. 0.17004 - 0.23980)
11 ROUGE-1 Average_F: 0.21027 (95%-conf.int. 0.17278 - 0.24804)
---------------------------------------------
11 ROUGE-2 Average_R: 0.03522 (95%-conf.int. 0.01812 - 0.05479)
11 ROUGE-2 Average_P: 0.02964 (95%-conf.int. 0.01698 - 0.04433)
11 ROUGE-2 Average_F: 0.03109 (95%-conf.int. 0.01669 - 0.04702)
---------------------------------------------
11 ROUGE-3 Average_R: 0.00243 (95%-conf.int. 0.00000 - 0.00774)
11 ROUGE-3 Average_P: 0.00171 (95%-conf.int. 0.00000 - 0.00545)
11 ROUGE-3 Average_F: 0.00201 (95%-conf.int. 0.00000 - 0.00640)
---------------------------------------------
11 ROUGE-4 Average_R: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
11 ROUGE-4 Average_P: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
11 ROUGE-4 Average_F: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
---------------------------------------------
11 ROUGE-L Average_R: 0.17144 (95%-conf.int. 0.14160 - 0.19972)
11 ROUGE-L Average_P: 0.15459 (95%-conf.int. 0.13442 - 0.17638)
11 ROUGE-L Average_F: 0.15979 (95%-conf.int. 0.13524 - 0.18293)
---------------------------------------------
11 ROUGE-W-1.2 Average_R: 0.10366 (95%-conf.int. 0.08657 - 0.12032)
11 ROUGE-W-1.2 Average_P: 0.14874 (95%-conf.int. 0.13072 - 0.16891)
11 ROUGE-W-1.2 Average_F: 0.12004 (95%-conf.int. 0.10260 - 0.13592)
---------------------------------------------
11 ROUGE-S* Average_R: 0.02919 (95%-conf.int. 0.01857 - 0.04092)
11 ROUGE-S* Average_P: 0.02341 (95%-conf.int. 0.01557 - 0.03158)
11 ROUGE-S* Average_F: 0.02400 (95%-conf.int. 0.01604 - 0.03186)
---------------------------------------------
11 ROUGE-SU* Average_R: 0.06207 (95%-conf.int. 0.04583 - 0.07847)
11 ROUGE-SU* Average_P: 0.05303 (95%-conf.int. 0.04009 - 0.06763)
11 ROUGE-SU* Average_F: 0.05310 (95%-conf.int. 0.04099 - 0.06553)
---------------------------------------------
12 ROUGE-1 Average_R: 0.24238 (95%-conf.int. 0.18967 - 0.29513)
12 ROUGE-1 Average_P: 0.25533 (95%-conf.int. 0.19308 - 0.31825)
12 ROUGE-1 Average_F: 0.24485 (95%-conf.int. 0.19150 - 0.30122)
---------------------------------------------
12 ROUGE-2 Average_R: 0.05210 (95%-conf.int. 0.02453 - 0.08236)
12 ROUGE-2 Average_P: 0.05569 (95%-conf.int. 0.02581 - 0.08922)
12 ROUGE-2 Average_F: 0.05265 (95%-conf.int. 0.02501 - 0.08296)
---------------------------------------------
12 ROUGE-3 Average_R: 0.01023 (95%-conf.int. 0.00114 - 0.02271)
12 ROUGE-3 Average_P: 0.01027 (95%-conf.int. 0.00125 - 0.02146)
12 ROUGE-3 Average_F: 0.00995 (95%-conf.int. 0.00119 - 0.02145)
---------------------------------------------
12 ROUGE-4 Average_R: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
12 ROUGE-4 Average_P: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
12 ROUGE-4 Average_F: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
---------------------------------------------
12 ROUGE-L Average_R: 0.18008 (95%-conf.int. 0.13709 - 0.22455)
12 ROUGE-L Average_P: 0.18728 (95%-conf.int. 0.14248 - 0.23337)
12 ROUGE-L Average_F: 0.18062 (95%-conf.int. 0.13810 - 0.22318)
---------------------------------------------
12 ROUGE-W-1.2 Average_R: 0.10847 (95%-conf.int. 0.08398 - 0.13339)
12 ROUGE-W-1.2 Average_P: 0.17875 (95%-conf.int. 0.13756 - 0.22048)
12 ROUGE-W-1.2 Average_F: 0.13289 (95%-conf.int. 0.10403 - 0.16220)
---------------------------------------------
12 ROUGE-S* Average_R: 0.03833 (95%-conf.int. 0.02085 - 0.05926)
12 ROUGE-S* Average_P: 0.04319 (95%-conf.int. 0.02107 - 0.06921)
12 ROUGE-S* Average_F: 0.03788 (95%-conf.int. 0.01997 - 0.05816)
---------------------------------------------
12 ROUGE-SU* Average_R: 0.07160 (95%-conf.int. 0.04882 - 0.09699)
12 ROUGE-SU* Average_P: 0.08071 (95%-conf.int. 0.05108 - 0.11638)
12 ROUGE-SU* Average_F: 0.07160 (95%-conf.int. 0.04794 - 0.09681)
---------------------------------------------
13 ROUGE-1 Average_R: 0.20161 (95%-conf.int. 0.15184 - 0.25908)
13 ROUGE-1 Average_P: 0.19956 (95%-conf.int. 0.14511 - 0.25978)
13 ROUGE-1 Average_F: 0.20030 (95%-conf.int. 0.14833 - 0.25923)
---------------------------------------------
13 ROUGE-2 Average_R: 0.04886 (95%-conf.int. 0.02609 - 0.07824)
13 ROUGE-2 Average_P: 0.04829 (95%-conf.int. 0.02445 - 0.07861)
13 ROUGE-2 Average_F: 0.04846 (95%-conf.int. 0.02523 - 0.07828)
---------------------------------------------
13 ROUGE-3 Average_R: 0.00887 (95%-conf.int. 0.00250 - 0.01758)
13 ROUGE-3 Average_P: 0.00909 (95%-conf.int. 0.00250 - 0.01804)
13 ROUGE-3 Average_F: 0.00897 (95%-conf.int. 0.00250 - 0.01758)
---------------------------------------------
13 ROUGE-4 Average_R: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
13 ROUGE-4 Average_P: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
13 ROUGE-4 Average_F: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
---------------------------------------------
13 ROUGE-L Average_R: 0.17044 (95%-conf.int. 0.12873 - 0.21975)
13 ROUGE-L Average_P: 0.16849 (95%-conf.int. 0.12400 - 0.22144)
13 ROUGE-L Average_F: 0.16919 (95%-conf.int. 0.12604 - 0.21969)
---------------------------------------------
13 ROUGE-W-1.2 Average_R: 0.10327 (95%-conf.int. 0.08048 - 0.12969)
13 ROUGE-W-1.2 Average_P: 0.16067 (95%-conf.int. 0.12237 - 0.20421)
13 ROUGE-W-1.2 Average_F: 0.12550 (95%-conf.int. 0.09682 - 0.15816)
---------------------------------------------
13 ROUGE-S* Average_R: 0.03974 (95%-conf.int. 0.02107 - 0.06491)
13 ROUGE-S* Average_P: 0.04116 (95%-conf.int. 0.01983 - 0.07039)
13 ROUGE-S* Average_F: 0.04018 (95%-conf.int. 0.02016 - 0.06705)
---------------------------------------------
13 ROUGE-SU* Average_R: 0.06653 (95%-conf.int. 0.04305 - 0.09595)
13 ROUGE-SU* Average_P: 0.06719 (95%-conf.int. 0.04110 - 0.10081)
13 ROUGE-SU* Average_F: 0.06650 (95%-conf.int. 0.04165 - 0.09775)
---------------------------------------------
14 ROUGE-1 Average_R: 0.23816 (95%-conf.int. 0.18633 - 0.28642)
14 ROUGE-1 Average_P: 0.20187 (95%-conf.int. 0.15801 - 0.24672)
14 ROUGE-1 Average_F: 0.21741 (95%-conf.int. 0.16959 - 0.26309)
---------------------------------------------
14 ROUGE-2 Average_R: 0.04832 (95%-conf.int. 0.02575 - 0.07404)
14 ROUGE-2 Average_P: 0.04008 (95%-conf.int. 0.02100 - 0.06148)
14 ROUGE-2 Average_F: 0.04350 (95%-conf.int. 0.02320 - 0.06550)
---------------------------------------------
14 ROUGE-3 Average_R: 0.00626 (95%-conf.int. 0.00129 - 0.01275)
14 ROUGE-3 Average_P: 0.00551 (95%-conf.int. 0.00125 - 0.01106)
14 ROUGE-3 Average_F: 0.00583 (95%-conf.int. 0.00129 - 0.01172)
---------------------------------------------
14 ROUGE-4 Average_R: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
14 ROUGE-4 Average_P: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
14 ROUGE-4 Average_F: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
---------------------------------------------
14 ROUGE-L Average_R: 0.18917 (95%-conf.int. 0.15208 - 0.22554)
14 ROUGE-L Average_P: 0.16072 (95%-conf.int. 0.12931 - 0.19636)
14 ROUGE-L Average_F: 0.17285 (95%-conf.int. 0.13969 - 0.20706)
---------------------------------------------
14 ROUGE-W-1.2 Average_R: 0.11376 (95%-conf.int. 0.09309 - 0.13418)
14 ROUGE-W-1.2 Average_P: 0.15239 (95%-conf.int. 0.12539 - 0.18210)
14 ROUGE-W-1.2 Average_F: 0.12955 (95%-conf.int. 0.10691 - 0.15301)
---------------------------------------------
14 ROUGE-S* Average_R: 0.04052 (95%-conf.int. 0.02377 - 0.05965)
14 ROUGE-S* Average_P: 0.03102 (95%-conf.int. 0.01668 - 0.05095)
14 ROUGE-S* Average_F: 0.03427 (95%-conf.int. 0.01952 - 0.05277)
---------------------------------------------
14 ROUGE-SU* Average_R: 0.07426 (95%-conf.int. 0.05233 - 0.09732)
14 ROUGE-SU* Average_P: 0.05627 (95%-conf.int. 0.03818 - 0.07912)
14 ROUGE-SU* Average_F: 0.06277 (95%-conf.int. 0.04369 - 0.08486)
---------------------------------------------
21 ROUGE-1 Average_R: 0.12268 (95%-conf.int. 0.09798 - 0.14879)
21 ROUGE-1 Average_P: 0.15320 (95%-conf.int. 0.12216 - 0.18730)
21 ROUGE-1 Average_F: 0.13279 (95%-conf.int. 0.10711 - 0.15971)
---------------------------------------------
21 ROUGE-2 Average_R: 0.01529 (95%-conf.int. 0.00592 - 0.02711)
21 ROUGE-2 Average_P: 0.02223 (95%-conf.int. 0.00779 - 0.04143)
21 ROUGE-2 Average_F: 0.01766 (95%-conf.int. 0.00648 - 0.03171)
---------------------------------------------
21 ROUGE-3 Average_R: 0.00146 (95%-conf.int. 0.00000 - 0.00387)
21 ROUGE-3 Average_P: 0.00189 (95%-conf.int. 0.00000 - 0.00500)
21 ROUGE-3 Average_F: 0.00165 (95%-conf.int. 0.00000 - 0.00436)
---------------------------------------------
21 ROUGE-4 Average_R: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
21 ROUGE-4 Average_P: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
21 ROUGE-4 Average_F: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
---------------------------------------------
21 ROUGE-L Average_R: 0.11136 (95%-conf.int. 0.08935 - 0.13337)
21 ROUGE-L Average_P: 0.14091 (95%-conf.int. 0.11120 - 0.17367)
21 ROUGE-L Average_F: 0.12123 (95%-conf.int. 0.09858 - 0.14549)
---------------------------------------------
21 ROUGE-W-1.2 Average_R: 0.07130 (95%-conf.int. 0.05835 - 0.08458)
21 ROUGE-W-1.2 Average_P: 0.14244 (95%-conf.int. 0.11389 - 0.17316)
21 ROUGE-W-1.2 Average_F: 0.09280 (95%-conf.int. 0.07623 - 0.10966)
---------------------------------------------
21 ROUGE-S* Average_R: 0.00977 (95%-conf.int. 0.00502 - 0.01464)
21 ROUGE-S* Average_P: 0.01637 (95%-conf.int. 0.00837 - 0.02539)
21 ROUGE-S* Average_F: 0.01103 (95%-conf.int. 0.00590 - 0.01624)
---------------------------------------------
21 ROUGE-SU* Average_R: 0.03016 (95%-conf.int. 0.02244 - 0.03836)
21 ROUGE-SU* Average_P: 0.05010 (95%-conf.int. 0.03553 - 0.06724)
21 ROUGE-SU* Average_F: 0.03441 (95%-conf.int. 0.02570 - 0.04326)
---------------------------------------------
22 ROUGE-1 Average_R: 0.16619 (95%-conf.int. 0.13350 - 0.20500)
22 ROUGE-1 Average_P: 0.15684 (95%-conf.int. 0.12675 - 0.19079)
22 ROUGE-1 Average_F: 0.15540 (95%-conf.int. 0.12640 - 0.18731)
---------------------------------------------
22 ROUGE-2 Average_R: 0.01970 (95%-conf.int. 0.00940 - 0.03235)
22 ROUGE-2 Average_P: 0.02285 (95%-conf.int. 0.00885 - 0.04183)
22 ROUGE-2 Average_F: 0.01963 (95%-conf.int. 0.00867 - 0.03326)
---------------------------------------------
22 ROUGE-3 Average_R: 0.00267 (95%-conf.int. 0.00000 - 0.00645)
22 ROUGE-3 Average_P: 0.00179 (95%-conf.int. 0.00000 - 0.00439)
22 ROUGE-3 Average_F: 0.00214 (95%-conf.int. 0.00000 - 0.00523)
---------------------------------------------
22 ROUGE-4 Average_R: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
22 ROUGE-4 Average_P: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
22 ROUGE-4 Average_F: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
---------------------------------------------
22 ROUGE-L Average_R: 0.14274 (95%-conf.int. 0.11681 - 0.17154)
22 ROUGE-L Average_P: 0.13564 (95%-conf.int. 0.11091 - 0.16389)
22 ROUGE-L Average_F: 0.13356 (95%-conf.int. 0.11087 - 0.15692)
---------------------------------------------
22 ROUGE-W-1.2 Average_R: 0.08851 (95%-conf.int. 0.07349 - 0.10502)
22 ROUGE-W-1.2 Average_P: 0.13443 (95%-conf.int. 0.10985 - 0.16234)
22 ROUGE-W-1.2 Average_F: 0.10269 (95%-conf.int. 0.08630 - 0.11951)
---------------------------------------------
22 ROUGE-S* Average_R: 0.02048 (95%-conf.int. 0.01204 - 0.03044)
22 ROUGE-S* Average_P: 0.01755 (95%-conf.int. 0.00929 - 0.02714)
22 ROUGE-S* Average_F: 0.01595 (95%-conf.int. 0.00957 - 0.02308)
---------------------------------------------
22 ROUGE-SU* Average_R: 0.04477 (95%-conf.int. 0.03270 - 0.05895)
22 ROUGE-SU* Average_P: 0.04262 (95%-conf.int. 0.02754 - 0.05872)
22 ROUGE-SU* Average_F: 0.03765 (95%-conf.int. 0.02812 - 0.04741)
---------------------------------------------
23 ROUGE-1 Average_R: 0.12235 (95%-conf.int. 0.08927 - 0.15829)
23 ROUGE-1 Average_P: 0.11503 (95%-conf.int. 0.08510 - 0.14914)
23 ROUGE-1 Average_F: 0.11823 (95%-conf.int. 0.08752 - 0.15313)
---------------------------------------------
23 ROUGE-2 Average_R: 0.00681 (95%-conf.int. 0.00000 - 0.01641)
23 ROUGE-2 Average_P: 0.00607 (95%-conf.int. 0.00000 - 0.01473)
23 ROUGE-2 Average_F: 0.00641 (95%-conf.int. 0.00000 - 0.01550)
---------------------------------------------
23 ROUGE-3 Average_R: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
23 ROUGE-3 Average_P: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
23 ROUGE-3 Average_F: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
---------------------------------------------
23 ROUGE-4 Average_R: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
23 ROUGE-4 Average_P: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
23 ROUGE-4 Average_F: 0.00000 (95%-conf.int. 0.00000 - 0.00000)
---------------------------------------------
23 ROUGE-L Average_R: 0.10965 (95%-conf.int. 0.08185 - 0.14105)
23 ROUGE-L Average_P: 0.10383 (95%-conf.int. 0.07819 - 0.13277)
23 ROUGE-L Average_F: 0.10635 (95%-conf.int. 0.07990 - 0.13597)
---------------------------------------------
23 ROUGE-W-1.2 Average_R: 0.06674 (95%-conf.int. 0.05082 - 0.08413)
23 ROUGE-W-1.2 Average_P: 0.10003 (95%-conf.int. 0.07684 - 0.12576)
23 ROUGE-W-1.2 Average_F: 0.07981 (95%-conf.int. 0.06101 - 0.10034)
---------------------------------------------
23 ROUGE-S* Average_R: 0.01001 (95%-conf.int. 0.00430 - 0.01689)
23 ROUGE-S* Average_P: 0.00899 (95%-conf.int. 0.00360 - 0.01568)
23 ROUGE-S* Average_F: 0.00939 (95%-conf.int. 0.00387 - 0.01613)
---------------------------------------------
23 ROUGE-SU* Average_R: 0.02865 (95%-conf.int. 0.01887 - 0.04049)
23 ROUGE-SU* Average_P: 0.02590 (95%-conf.int. 0.01739 - 0.03617)
23 ROUGE-SU* Average_F: 0.02692 (95%-conf.int. 0.01793 - 0.03771)
---------------------------------------------
24 ROUGE-1 Average_R: 0.28540 (95%-conf.int. 0.23134 - 0.34089)
24 ROUGE-1 Average_P: 0.27811 (95%-conf.int. 0.20830 - 0.35585)
24 ROUGE-1 Average_F: 0.26995 (95%-conf.int. 0.21567 - 0.32693)
---------------------------------------------
24 ROUGE-2 Average_R: 0.09309 (95%-conf.int. 0.05629 - 0.13592)
24 ROUGE-2 Average_P: 0.11162 (95%-conf.int. 0.05591 - 0.18063)
24 ROUGE-2 Average_F: 0.09350 (95%-conf.int. 0.05490 - 0.13769)
---------------------------------------------
24 ROUGE-3 Average_R: 0.03075 (95%-conf.int. 0.01081 - 0.05737)
24 ROUGE-3 Average_P: 0.04022 (95%-conf.int. 0.00900 - 0.08575)
24 ROUGE-3 Average_F: 0.03122 (95%-conf.int. 0.00986 - 0.05847)
---------------------------------------------
24 ROUGE-4 Average_R: 0.00860 (95%-conf.int. 0.00000 - 0.02009)
24 ROUGE-4 Average_P: 0.00703 (95%-conf.int. 0.00000 - 0.01639)
24 ROUGE-4 Average_F: 0.00774 (95%-conf.int. 0.00000 - 0.01805)
---------------------------------------------
24 ROUGE-L Average_R: 0.24161 (95%-conf.int. 0.19599 - 0.29178)
24 ROUGE-L Average_P: 0.24108 (95%-conf.int. 0.17423 - 0.31614)
24 ROUGE-L Average_F: 0.23010 (95%-conf.int. 0.18351 - 0.28161)
---------------------------------------------
24 ROUGE-W-1.2 Average_R: 0.14412 (95%-conf.int. 0.11828 - 0.17208)
24 ROUGE-W-1.2 Average_P: 0.22825 (95%-conf.int. 0.16623 - 0.30239)
24 ROUGE-W-1.2 Average_F: 0.16917 (95%-conf.int. 0.13670 - 0.20511)
---------------------------------------------
24 ROUGE-S* Average_R: 0.06608 (95%-conf.int. 0.04152 - 0.09971)
24 ROUGE-S* Average_P: 0.08502 (95%-conf.int. 0.04014 - 0.13988)
24 ROUGE-S* Average_F: 0.05949 (95%-conf.int. 0.03719 - 0.08882)
---------------------------------------------
24 ROUGE-SU* Average_R: 0.10433 (95%-conf.int. 0.07587 - 0.14102)
24 ROUGE-SU* Average_P: 0.12555 (95%-conf.int. 0.06607 - 0.19873)
24 ROUGE-SU* Average_F: 0.09434 (95%-conf.int. 0.06742 - 0.12844)
omnisky@omnisky:~/software/ROUGE-1.5.5$