【论文复现赛】FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation

本文提出了一个新的联合上采样模块JPU(Joint Pyramid Upsampling)，将提取高分辨率特征图的任务映射成一个联合上采样问题。JPU模块可应用于各种语义分割模型中，并且提升模型的性能。本次复现的目标是ENCNet+JPU在ADE20K验证集上的miou达到42.5%，本次复现的miou为43.73%。本次复现基于PaddleSeg框架。

代码参考：https://github.com/wuhuikai/FastFCN
本项目git：https://github.com/justld/FastFCN_paddle

一、模型结构

network
上图为模型结构，在语义分割模型中（如PSPNet/ENCNet/FCN等）插入JPU模块，来提升语义分割的效果。

二、JPU(Joint Pyramid Upsampling)

JPU
上图为JPU模块，将backbone输出的不同大小的特征图（conv5、conv4、conv3）上采样至同样大小，concat后得到yc，接着使用dilation不同（分别为1/2/4/8）的卷积核进行卷积，从而输出特征图的感受野大小。

三、模型对比

compare
上图为ENCNet和ENCNet+JPU的预测结果（图片分别来自Pascal Context验证集和ADE20K验证集），上图可以明显看出JPU增加感受野的好处，山脉能够被正确识别。

四、实验结果

experiment
作者在多个数据集上测试了JPU的效果，上图为ADE20K验证集上的模型分割效果，可以看出JPU模块明显提高了模型的分割效果。

五、核心代码

JPU模块的核心代码如下所示，代码较为简单。


class JPU(nn.Layer):"""Joint Pyramid Upsampling of FCN.The original paper refers toWu, Huikai, et al. "Fastfcn: Rethinking dilated convolution in the backbone for semantic segmentation." arXiv preprint arXiv:1903.11816 (2019)."""def __init__(self, in_channels, width=512):super().__init__()self.conv5 = ConvBNReLU(in_channels[-1], width, 3, padding=1, bias_attr=False)self.conv4 = ConvBNReLU(in_channels[-2], width, 3, padding=1, bias_attr=False)self.conv3 = ConvBNReLU(in_channels[-3], width, 3, padding=1, bias_attr=False)self.dilation1 = SeparableConvBNReLU(3 * width,width,3,padding=1,pointwise_bias=False,dilation=1,bias_attr=False,stride=1,)self.dilation2 = SeparableConvBNReLU(3 * width,width,3,padding=2,pointwise_bias=False,dilation=2,bias_attr=False,stride=1)self.dilation3 = SeparableConvBNReLU(3 * width,width,3,padding=4,pointwise_bias=False,dilation=4,bias_attr=False,stride=1)self.dilation4 = SeparableConvBNReLU(3 * width,width,3,padding=8,pointwise_bias=False,dilation=8,bias_attr=False,stride=1)def forward(self, *inputs):feats = [self.conv5(inputs[-1]),self.conv4(inputs[-2]),self.conv3(inputs[-3])]size = paddle.shape(feats[-1])[2:]feats[-2] = F.interpolate(feats[-2], size, mode='bilinear', align_corners=True)feats[-3] = F.interpolate(feats[-3], size, mode='bilinear', align_corners=True)feat = paddle.concat(feats, axis=1)feat = paddle.concat([self.dilation1(feat),self.dilation2(feat),self.dilation3(feat),self.dilation4(feat)],axis=1)return inputs[0], inputs[1], inputs[2], feat

六、快速体验

运行一下代码，快速体验FastFCN在数据集ADE20K上的训练和验证。

step 1: 解压ADE20K数据集

step 2: FastFCN训练，复现使用Tesla V100 * 4,想要完整的复现结果请移步脚本任务

step 3: 使用训练好的权重在ADE20K验证集验证效果

step 4: 使用训练好的权重在ADE20K验证集翻转验证

step 5: 使用训练好的权重在ADE20K验证集多尺度翻转验证

# step 1: unzip data
%cd ~/data/data26423/
!unzip -oq ade20k.zip
%cd ~/

# step 2: training
%cd ~/FastFCN/
!python train.py --config configs/fastfcn/fastfcn_resnet50_os8_ade20k_480x480_120k.yml --num_workers 4 --do_eval --use_vdl --log_iter 100 --save_interval 10000

# step 3: val
%cd ~/FastFCN/
!python val.py --config configs/fastfcn/fastfcn_resnet50_os8_ade20k_480x480_120k.yml --model_path output/best_model/model.pdparams

/home/aistudio/PaddleSeg
2022-01-06 16:37:10 [INFO]	
---------------Config Information---------------
batch_size: 1
iters: 120000
loss:coef:- 1- 0.4- 0.2types:- type: CrossEntropyLoss- type: CrossEntropyLoss- type: SECrossEntropyLoss
lr_scheduler:end_lr: 0learning_rate: 0.01power: 0.9type: PolynomialDecay
model:add_lateral: trueaux_loss: truebackbone:output_stride: 8pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gztype: ResNet50_vdmid_channels: 512num_codes: 32type: FastFCNuse_jpu: trueuse_se_loss: true
optimizer:momentum: 0.9type: sgdweight_decay: 4.0e-05
train_dataset:dataset_root: /home/aistudio/data/data26423/ADEChallengeData2016/mode: traintransforms:- max_scale_factor: 2.0min_scale_factor: 0.5scale_step_size: 0.25type: ResizeStepScaling- crop_size:- 480- 480im_padding_value:- 0- 0- 0type: RandomPaddingCrop- type: RandomHorizontalFlip- brightness_range: 0.4contrast_range: 0.4saturation_range: 0.4type: RandomDistort- type: Normalizetype: ADE20K
val_dataset:dataset_root: /home/aistudio/data/data26423/ADEChallengeData2016/mode: valtransforms:- type: Normalizetype: ADE20K
------------------------------------------------
W0106 16:37:10.112859   783 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W0106 16:37:10.112905   783 device_context.cc:465] device: 0, cuDNN Version: 7.6.
2022-01-06 16:37:14 [INFO]	Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz
Connecting to https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz
Downloading resnet50_vd_ssld_v2.tar.gz
[==================================================] 100.00%
Uncompress resnet50_vd_ssld_v2.tar.gz
[==================================================] 100.00%
2022-01-06 16:37:23 [INFO]	There are 275/275 variables loaded into ResNet_vd.
2022-01-06 16:37:23 [INFO]	Loading pretrained model from output/best_model/model.pdparams
2022-01-06 16:37:24 [INFO]	There are 376/376 variables loaded into FastFCN.
2022-01-06 16:37:24 [INFO]	Loaded trained params of model successfully
2022-01-06 16:37:24 [INFO]	Start evaluating (total_samples: 2000, total_iters: 2000)...
2000/2000 [==============================] - 234s 117ms/step - batch_cost: 0.1170 - reader cost: 1.5749e-04s - batch_cost: 0.1170 - reader 
2022-01-06 16:41:19 [INFO]	[EVAL] #Images: 2000 mIoU: 0.4374 Acc: 0.8076 Kappa: 0.7930 Dice: 0.5773
2022-01-06 16:41:19 [INFO]	[EVAL] Class IoU: 
[0.745  0.8066 0.9372 0.7754 0.7208 0.8131 0.8158 0.8549 0.5986 0.65310.5689 0.6424 0.773  0.332  0.4218 0.5501 0.5841 0.4884 0.6756 0.54740.81   0.4592 0.6969 0.6264 0.3978 0.4191 0.446  0.5463 0.4915 0.28190.369  0.5631 0.3458 0.4537 0.3894 0.4501 0.5957 0.7278 0.3049 0.540.2025 0.2212 0.4444 0.3241 0.3992 0.2743 0.3133 0.6713 0.498  0.67310.638  0.407  0.1696 0.3026 0.6184 0.4454 0.8996 0.5376 0.4292 0.24330.1028 0.5304 0.4001 0.3601 0.5894 0.8081 0.3833 0.4164 0.1231 0.39820.5558 0.7378 0.5191 0.3173 0.571  0.4166 0.3316 0.1894 0.1841 0.42040.8377 0.5606 0.4784 0.3519 0.2219 0.6394 0.1883 0.2174 0.4548 0.66870.5063 0.0732 0.3426 0.1998 0.0073 0.1039 0.1758 0.4409 0.3437 0.32370.1685 0.0916 0.4238 0.1739 0.0514 0.6394 0.1297 0.6244 0.1974 0.53580.407  0.4615 0.2469 0.6013 0.9031 0.1345 0.6048 0.7609 0.3492 0.42850.3955 0.1561 0.3452 0.2539 0.4341 0.3856 0.524  0.5134 0.5083 0.67460.636  0.1091 0.4155 0.549  0.2858 0.3555 0.2771 0.0651 0.4047 0.52980.3904 0.0322 0.4277 0.0778 0.3629 0.006  0.4618 0.1152 0.2477 0.3207]
2022-01-06 16:41:19 [INFO]	[EVAL] Class Acc: 
[0.8345 0.861  0.9665 0.843  0.8215 0.8902 0.9051 0.9036 0.7432 0.7790.7483 0.7721 0.8486 0.5302 0.6345 0.6971 0.7393 0.6762 0.8098 0.72740.9002 0.6595 0.8051 0.734  0.5452 0.663  0.554  0.791  0.7802 0.42090.5678 0.6722 0.5855 0.6445 0.5493 0.646  0.7836 0.8657 0.5192 0.76170.4397 0.4533 0.6501 0.5246 0.5394 0.5499 0.4866 0.8419 0.7299 0.7990.78   0.5202 0.3198 0.5879 0.7143 0.5973 0.9641 0.7673 0.6897 0.41760.1653 0.8454 0.5436 0.6991 0.7019 0.8948 0.5658 0.5516 0.2746 0.69460.7083 0.8819 0.6251 0.412  0.8152 0.5758 0.528  0.6424 0.7616 0.65160.907  0.7407 0.783  0.6011 0.4543 0.776  0.4833 0.5236 0.8167 0.85930.7141 0.0957 0.4982 0.3761 0.0198 0.2625 0.5537 0.6707 0.529  0.70320.3318 0.2019 0.6512 0.3609 0.4271 0.8514 0.3312 0.9085 0.4599 0.81830.6643 0.7014 0.4142 0.7466 0.9363 0.3573 0.7655 0.8431 0.4491 0.8050.8297 0.2843 0.9189 0.6598 0.8819 0.7389 0.8312 0.6611 0.889  0.85780.8084 0.3986 0.5557 0.8412 0.698  0.5577 0.4893 0.139  0.6569 0.70730.462  0.0444 0.6447 0.3508 0.6634 0.0079 0.8371 0.4788 0.6483 0.8037]

# step 4: val flip
%cd ~/FastFCN/
!python val.py --config configs/fastfcn/fastfcn_resnet50_os8_ade20k_480x480_120k.yml --model_path output/best_model/model.pdparams --aug_eval --flip_horizontal

/home/aistudio/PaddleSeg
2022-01-06 16:41:22 [INFO]	
---------------Config Information---------------
batch_size: 1
iters: 120000
loss:coef:- 1- 0.4- 0.2types:- type: CrossEntropyLoss- type: CrossEntropyLoss- type: SECrossEntropyLoss
lr_scheduler:end_lr: 0learning_rate: 0.01power: 0.9type: PolynomialDecay
model:add_lateral: trueaux_loss: truebackbone:output_stride: 8pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gztype: ResNet50_vdmid_channels: 512num_codes: 32type: FastFCNuse_jpu: trueuse_se_loss: true
optimizer:momentum: 0.9type: sgdweight_decay: 4.0e-05
train_dataset:dataset_root: /home/aistudio/data/data26423/ADEChallengeData2016/mode: traintransforms:- max_scale_factor: 2.0min_scale_factor: 0.5scale_step_size: 0.25type: ResizeStepScaling- crop_size:- 480- 480im_padding_value:- 0- 0- 0type: RandomPaddingCrop- type: RandomHorizontalFlip- brightness_range: 0.4contrast_range: 0.4saturation_range: 0.4type: RandomDistort- type: Normalizetype: ADE20K
val_dataset:dataset_root: /home/aistudio/data/data26423/ADEChallengeData2016/mode: valtransforms:- type: Normalizetype: ADE20K
------------------------------------------------
W0106 16:41:22.066421  1150 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W0106 16:41:22.066468  1150 device_context.cc:465] device: 0, cuDNN Version: 7.6.
2022-01-06 16:41:26 [INFO]	Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz
2022-01-06 16:41:26 [INFO]	There are 275/275 variables loaded into ResNet_vd.
2022-01-06 16:41:26 [INFO]	Loading pretrained model from output/best_model/model.pdparams
2022-01-06 16:41:27 [INFO]	There are 376/376 variables loaded into FastFCN.
2022-01-06 16:41:27 [INFO]	Loaded trained params of model successfully
2022-01-06 16:41:27 [INFO]	Start evaluating (total_samples: 2000, total_iters: 2000)...
2000/2000 [==============================] - 331s 165ms/step - batch_cost: 0.1650 - reader cost: 1.5685e-04s - batch_cost:
2022-01-06 16:46:58 [INFO]	[EVAL] #Images: 2000 mIoU: 0.4411 Acc: 0.8106 Kappa: 0.7962 Dice: 0.5805
2022-01-06 16:46:58 [INFO]	[EVAL] Class IoU: 
[0.7485 0.8082 0.9397 0.7797 0.7276 0.8167 0.8194 0.8608 0.6061 0.65390.575  0.6508 0.7814 0.3318 0.4293 0.5546 0.5788 0.4963 0.6775 0.55390.8112 0.4674 0.7033 0.64   0.4091 0.4006 0.4496 0.5617 0.4939 0.28840.369  0.5615 0.3544 0.462  0.3813 0.4575 0.6046 0.7351 0.3083 0.55250.1979 0.2168 0.4514 0.3299 0.4035 0.2726 0.3221 0.6859 0.4887 0.690.6295 0.4135 0.163  0.3039 0.6246 0.4452 0.9012 0.5523 0.4486 0.24160.1053 0.5453 0.3964 0.3623 0.6056 0.8196 0.3914 0.4211 0.1107 0.40110.5576 0.7432 0.5304 0.3416 0.5791 0.4099 0.3293 0.1902 0.186  0.43040.8417 0.5676 0.4869 0.3592 0.1908 0.6434 0.2008 0.2228 0.4541 0.67690.5152 0.0764 0.3632 0.2057 0.0087 0.1101 0.1824 0.4414 0.3445 0.33920.1739 0.1069 0.3913 0.1655 0.0588 0.5915 0.1462 0.6378 0.1991 0.52350.4152 0.4741 0.2515 0.5948 0.9242 0.1265 0.6146 0.7569 0.3427 0.43360.3989 0.1391 0.3363 0.2547 0.4348 0.3821 0.5239 0.5278 0.5348 0.6790.6389 0.1006 0.4356 0.5638 0.2887 0.3482 0.2811 0.0715 0.4031 0.54070.3952 0.0315 0.4428 0.0825 0.3711 0.007  0.4821 0.1175 0.2258 0.3276]
2022-01-06 16:46:58 [INFO]	[EVAL] Class Acc: 
[0.835  0.8615 0.9677 0.8447 0.8257 0.893  0.9058 0.9062 0.7469 0.77750.7546 0.7806 0.852  0.5345 0.6515 0.6994 0.7294 0.6822 0.8135 0.7340.9    0.6672 0.8114 0.7432 0.5556 0.656  0.5598 0.8042 0.7891 0.42620.5641 0.6692 0.5956 0.646  0.5441 0.657  0.7924 0.8766 0.5225 0.77440.4419 0.4507 0.6601 0.5436 0.547  0.5575 0.4986 0.8539 0.7244 0.80640.7681 0.527  0.3156 0.5946 0.7201 0.5952 0.9638 0.7761 0.6731 0.41250.168  0.8637 0.539  0.7083 0.7159 0.9024 0.5777 0.5583 0.2625 0.71520.7038 0.8853 0.6358 0.4357 0.8207 0.5736 0.5339 0.6579 0.7888 0.66810.9102 0.7573 0.7961 0.6077 0.4095 0.7797 0.5179 0.5686 0.8037 0.85850.7307 0.1005 0.518  0.4255 0.0249 0.2956 0.5853 0.6801 0.5487 0.740.3508 0.2357 0.6296 0.3536 0.4493 0.8534 0.3589 0.9221 0.4796 0.79910.678  0.7347 0.4228 0.7442 0.9488 0.3472 0.7703 0.8367 0.4397 0.78110.8335 0.2587 0.8991 0.6834 0.8929 0.747  0.8301 0.6672 0.8696 0.86120.8156 0.3995 0.6059 0.8509 0.7342 0.5771 0.502  0.1504 0.6381 0.71120.4642 0.0435 0.6579 0.3839 0.6907 0.0094 0.8574 0.5058 0.6338 0.8341]

# step 5: val ms + flip
%cd ~/FastFCN/
!python val.py --config configs/fastfcn/fastfcn_resnet50_os8_ade20k_480x480_120k.yml --model_path output/best_model/model.pdparams --aug_eval --flip_horizontal --scales 0.75 1.0 1.25

/home/aistudio/PaddleSeg
2022-01-06 16:47:01 [INFO]	
---------------Config Information---------------
batch_size: 1
iters: 120000
loss:coef:- 1- 0.4- 0.2types:- type: CrossEntropyLoss- type: CrossEntropyLoss- type: SECrossEntropyLoss
lr_scheduler:end_lr: 0learning_rate: 0.01power: 0.9type: PolynomialDecay
model:add_lateral: trueaux_loss: truebackbone:output_stride: 8pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gztype: ResNet50_vdmid_channels: 512num_codes: 32type: FastFCNuse_jpu: trueuse_se_loss: true
optimizer:momentum: 0.9type: sgdweight_decay: 4.0e-05
train_dataset:dataset_root: /home/aistudio/data/data26423/ADEChallengeData2016/mode: traintransforms:- max_scale_factor: 2.0min_scale_factor: 0.5scale_step_size: 0.25type: ResizeStepScaling- crop_size:- 480- 480im_padding_value:- 0- 0- 0type: RandomPaddingCrop- type: RandomHorizontalFlip- brightness_range: 0.4contrast_range: 0.4saturation_range: 0.4type: RandomDistort- type: Normalizetype: ADE20K
val_dataset:dataset_root: /home/aistudio/data/data26423/ADEChallengeData2016/mode: valtransforms:- type: Normalizetype: ADE20K
------------------------------------------------
W0106 16:47:01.055764  1635 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W0106 16:47:01.055821  1635 device_context.cc:465] device: 0, cuDNN Version: 7.6.
2022-01-06 16:47:05 [INFO]	Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz
2022-01-06 16:47:05 [INFO]	There are 275/275 variables loaded into ResNet_vd.
2022-01-06 16:47:05 [INFO]	Loading pretrained model from output/best_model/model.pdparams
2022-01-06 16:47:06 [INFO]	There are 376/376 variables loaded into FastFCN.
2022-01-06 16:47:06 [INFO]	Loaded trained params of model successfully
2022-01-06 16:47:06 [INFO]	Start evaluating (total_samples: 2000, total_iters: 2000)...
2000/2000 [==============================] - 729s 365ms/step - batch_cost: 0.3644 - reader cost: 1.5749e-040s - batch_cost: 0.3644 - rea - ETA: 4s - batch_cost: 0.3645 - reader cost:
2022-01-06 16:59:15 [INFO]	[EVAL] #Images: 2000 mIoU: 0.4448 Acc: 0.8133 Kappa: 0.7992 Dice: 0.5837
2022-01-06 16:59:15 [INFO]	[EVAL] Class IoU: 
[0.7527 0.8097 0.9414 0.7868 0.7355 0.8209 0.8207 0.8674 0.6082 0.65390.5804 0.6463 0.782  0.3205 0.4471 0.5602 0.5773 0.4966 0.6849 0.55520.8131 0.4735 0.7035 0.647  0.4171 0.4013 0.461  0.5935 0.5196 0.27730.382  0.5613 0.3688 0.4587 0.4039 0.452  0.6097 0.7376 0.3171 0.56520.193  0.2286 0.4516 0.3269 0.4144 0.2382 0.3343 0.6679 0.4826 0.68830.6771 0.3886 0.1929 0.3088 0.6416 0.4324 0.9083 0.5537 0.4859 0.2940.1119 0.5132 0.4066 0.3783 0.6134 0.8137 0.405  0.4168 0.1201 0.42140.5591 0.7689 0.5229 0.3522 0.5708 0.4044 0.3517 0.1814 0.1813 0.4720.8464 0.559  0.4973 0.3436 0.1746 0.6528 0.2126 0.2439 0.4113 0.65520.4985 0.074  0.3616 0.19   0.0064 0.0866 0.1979 0.4782 0.3609 0.38060.1928 0.0763 0.3997 0.1436 0.1276 0.667  0.1506 0.6296 0.2117 0.49210.4215 0.5216 0.2695 0.5803 0.9271 0.0963 0.6495 0.7728 0.2964 0.43830.4349 0.1254 0.3443 0.2417 0.4412 0.411  0.5473 0.5382 0.3474 0.66030.6535 0.0792 0.4112 0.5583 0.3094 0.378  0.2878 0.055  0.4065 0.55930.4134 0.0351 0.4426 0.0837 0.3867 0.0111 0.5123 0.1057 0.2472 0.3119]
2022-01-06 16:59:15 [INFO]	[EVAL] Class Acc: 
[0.8406 0.8562 0.9692 0.8574 0.8349 0.8969 0.9062 0.9108 0.7461 0.780.757  0.7788 0.8546 0.5352 0.6675 0.6984 0.7263 0.6742 0.8119 0.72580.8952 0.6926 0.8036 0.742  0.5557 0.657  0.5732 0.8021 0.7732 0.39960.5757 0.6712 0.5944 0.6364 0.5521 0.6445 0.7951 0.8849 0.519  0.77610.4558 0.4609 0.6801 0.5578 0.5687 0.5608 0.5386 0.8239 0.7431 0.78850.8198 0.4817 0.3599 0.6377 0.7336 0.5772 0.9572 0.7745 0.7216 0.54090.1907 0.8687 0.5495 0.8249 0.7128 0.8986 0.5714 0.5834 0.2556 0.70270.6977 0.9005 0.6483 0.4329 0.7966 0.5609 0.5704 0.6234 0.744  0.66890.8925 0.7297 0.8129 0.5958 0.3428 0.794  0.5332 0.5379 0.6899 0.81070.7019 0.0961 0.5237 0.4292 0.0178 0.2641 0.7026 0.7275 0.5714 0.77020.4491 0.1522 0.6946 0.3302 0.5698 0.8987 0.3497 0.9316 0.4482 0.73660.7171 0.8771 0.5069 0.6875 0.9443 0.3283 0.7624 0.815  0.3729 0.70560.7882 0.2594 0.9448 0.692  0.8829 0.7768 0.8536 0.7001 0.4462 0.89760.8202 0.3823 0.6143 0.8352 0.6909 0.6335 0.4725 0.1329 0.6227 0.73210.4951 0.0504 0.6919 0.3802 0.6987 0.0163 0.8663 0.5677 0.6427 0.8729]