神经机器翻译推断阶段信心校准研究 原

dahezi1

本论文由腾讯 AI Lab 和清华大学合作完成，作者提出了一种评估神经机器翻译模型在推断场景下信心校准偏差的方法，并发现 Transformer 模型的信心尽管在训练场景中校准较好，但是在推断场景中仍然存在较大的校准偏差。以下为论文的详细解读。

　　On the Inference Calibration of Neural Machine Translation

　　基于概率的机器学习模型在给出预测结果的同时，往往会输出一个对应的信心指数(i.e.， confidence)，该信心指数可以代表模型对自身预测结果的正确性的一个估计。在金融、医疗等安全等级较高的场景中，我们希望模型不但有较好的预测精度(i.e.， accuracy)，并且能够做到“知之为知之，不知为不知”，对预测结果的正确性有准确的估计。

　　我们可以设想一个场景：在一个共同抗击疫情的各国联合医疗队中，各国医护人员可以使用机器翻译系统进行交流。在涉及患者病情的关键性描述中，我们要求机器翻译系统要如实反映其对翻译结果的信心。对于模型不自信的翻译结果，我们可以请语言专家有针对性的进行后处理，对于大部分模型自信的结果，我们可以直接使用。由此可见，对自身输出结果是否有一个准确的信心估计，是衡量机器翻译模型能否实际部署的重要性质。

　　量化模型对预测结果信心校准偏差的前人工作大多是在分类任务上开展的。但是，不同于分类任务的单一输出，包括机器翻译在内的生成式自然语言任务的输出都是序列化的，并且往往具有潜在的语义结构。如何评估序列化生成模型的信心校准偏差依然是一个尚未解决的问题。

　　在本文中，我们对期望校准偏差(Expected Calibration Error, ECE)进行了扩展，使其能够应用到序列化生成任务中来。具体地，ECE 在计算方式如下：http://m.imdb.com/list/ls087039411
http://www.imdb.com/list/ls087039411
http://m.imdb.com/list/ls087039453
http://www.imdb.com/list/ls087039453
http://m.imdb.com/list/ls087039185
http://www.imdb.com/list/ls087039185
http://m.imdb.com/list/ls087039141
http://www.imdb.com/list/ls087039141
http://m.imdb.com/list/ls087022835
http://www.imdb.com/list/ls087022835
http://m.imdb.com/list/ls087022850
http://www.imdb.com/list/ls087022850
http://m.imdb.com/list/ls087022983
http://www.imdb.com/list/ls087022983
http://m.imdb.com/list/ls087022920
http://www.imdb.com/list/ls087022920
http://m.imdb.com/list/ls087022913
http://www.imdb.com/list/ls087022913
http://m.imdb.com/list/ls087022957
http://www.imdb.com/list/ls087022957
http://m.imdb.com/list/ls087022480
http://www.imdb.com/list/ls087022480
http://m.imdb.com/list/ls087022420
http://www.imdb.com/list/ls087022420
http://m.imdb.com/list/ls087022418
http://www.imdb.com/list/ls087022418
http://m.imdb.com/list/ls087022286
http://www.imdb.com/list/ls087022286
http://m.imdb.com/list/ls087022296
http://www.imdb.com/list/ls087022296
http://m.imdb.com/list/ls087022264
http://www.imdb.com/list/ls087022264
http://m.imdb.com/list/ls087022230
http://www.imdb.com/list/ls087022230
http://m.imdb.com/list/ls087022200
http://www.imdb.com/list/ls087022200
http://m.imdb.com/list/ls087022697
http://www.imdb.com/list/ls087022697
http://m.imdb.com/list/ls087022635
http://www.imdb.com/list/ls087022635
http://m.imdb.com/list/ls087022674
http://www.imdb.com/list/ls087022674
http://m.imdb.com/list/ls087022391
http://www.imdb.com/list/ls087022391
http://m.imdb.com/list/ls087022325
http://www.imdb.com/list/ls087022325
http://m.imdb.com/list/ls087022376
http://www.imdb.com/list/ls087022376
http://m.imdb.com/list/ls087022309
http://www.imdb.com/list/ls087022309
http://m.imdb.com/list/ls087022129
http://www.imdb.com/list/ls087022129
http://m.imdb.com/list/ls087022138
http://www.imdb.com/list/ls087022138
http://m.imdb.com/list/ls087022151
http://www.imdb.com/list/ls087022151
http://m.imdb.com/list/ls087022784
http://www.imdb.com/list/ls087022784
http://m.imdb.com/list/ls087022725
http://www.imdb.com/list/ls087022725
http://m.imdb.com/list/ls087022732
http://www.imdb.com/list/ls087022732
http://m.imdb.com/list/ls087022704
http://www.imdb.com/list/ls087022704
http://m.imdb.com/list/ls087022582
http://www.imdb.com/list/ls087022582
http://m.imdb.com/list/ls087022524
http://www.imdb.com/list/ls087022524
http://m.imdb.com/list/ls087022519
http://www.imdb.com/list/ls087022519
http://m.imdb.com/list/ls087022506
http://www.imdb.com/list/ls087022506
http://m.imdb.com/list/ls087022087
http://www.imdb.com/list/ls087022087
http://m.imdb.com/list/ls087022027
http://www.imdb.com/list/ls087022027
http://m.imdb.com/list/ls087022037
http://www.imdb.com/list/ls087022037
http://m.imdb.com/list/ls087022009
http://www.imdb.com/list/ls087022009
http://m.imdb.com/list/ls087026891
http://www.imdb.com/list/ls087026891
http://m.imdb.com/list/ls087026834
http://www.imdb.com/list/ls087026834
http://m.imdb.com/list/ls087026878
http://www.imdb.com/list/ls087026878
http://m.imdb.com/list/ls087026989
http://www.imdb.com/list/ls087026989
http://m.imdb.com/list/ls087026995
http://www.imdb.com/list/ls087026995
http://m.imdb.com/list/ls087026960
http://www.imdb.com/list/ls087026960
http://m.imdb.com/list/ls087026910
http://www.imdb.com/list/ls087026910
http://m.imdb.com/list/ls087026482
http://www.imdb.com/list/ls087026482
http://m.imdb.com/list/ls087026497
http://www.imdb.com/list/ls087026497
http://m.imdb.com/list/ls087026462
http://www.imdb.com/list/ls087026462
http://m.imdb.com/list/ls087026419
http://www.imdb.com/list/ls087026419
http://m.imdb.com/list/ls087026405
http://www.imdb.com/list/ls087026405
http://m.imdb.com/list/ls087026297
http://www.imdb.com/list/ls087026297
http://m.imdb.com/list/ls087026212
http://www.imdb.com/list/ls087026212
http://m.imdb.com/list/ls087026203
http://www.imdb.com/list/ls087026203
http://m.imdb.com/list/ls087026640
http://www.imdb.com/list/ls087026640
http://m.imdb.com/list/ls087026638
http://www.imdb.com/list/ls087026638
http://m.imdb.com/list/ls087026601
http://www.imdb.com/list/ls087026601
http://m.imdb.com/list/ls087026349
http://www.imdb.com/list/ls087026349
http://m.imdb.com/list/ls087026336
http://www.imdb.com/list/ls087026336
http://m.imdb.com/list/ls087026378
http://www.imdb.com/list/ls087026378
http://m.imdb.com/list/ls087026188
http://www.imdb.com/list/ls087026188
http://m.imdb.com/list/ls087026191
http://www.imdb.com/list/ls087026191
http://m.imdb.com/list/ls087026163
http://www.imdb.com/list/ls087026163
http://m.imdb.com/list/ls087026178
http://www.imdb.com/list/ls087026178
http://m.imdb.com/list/ls087026783
http://www.imdb.com/list/ls087026783
http://m.imdb.com/list/ls087026742
http://www.imdb.com/list/ls087026742