논문 요약
이전 SISR 알고리즘들은 깊은 네트워크 구조와 오랜 훈련시간이 필요했다. 더불어 SR을 위한 CNN은 multople scales에서의 feature를 이용하지 못했으며 같은 weight를 가지게 하였다. Densely Residual Laplacian Network (DRLN)는 residual 구조에서 cascading residual을 사용하여 low-frequency flow가 high, mid-level features의 기능을 학습하는 데 집중할 수 있도록 한다. 더불어 denslely concatenated redisual blocks는 high-level complex features를 학습하는데 도움을 준다. 또한, feature map 사이의 inter and intra-level dependencies를 학습하기 위한 Laplacian attention을 제안한다.
1. Paper Bibliography
논문 제목
Densely residual laplacian super-resolution
저자
Anwar et al.
출판 정보 / 학술대회 발표 정보
IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)
2. Problems & Motivations
Deep Convolutional Neural Network(CNN)을 이용한 SR이 발전되어 왔다. SRCNN의 레이어는 3개가 전부인 반면 RCAN은 무려 400개나 넘는 레이어를 가지고 있다. 그러나 deep한 네트워크는 무겁고 계산량이 많아 여러모로 활용되기 어려워 efficient한 네트워크를 만들 필요성이 있다. 가장 쉬운 방법은 depth를 줄이는 것이나 이는 퀄리티를 감소시킨다. 따라서, 계산된 features의 재사용 가능성에 초점을 맞춘 효율적인 네트워크를 설계하는 것이 필요하다.
Depth를 줄이는 방법대신 사용할 수 있는 방법은 recurrent한 구조를 이용하는 것이다. 이 구조를 통해 파라미터 수를 줄일 수 있고 일반적인 CNN보다 성능을 높일 수 있지만 역시 단점이 존재한다: 1) upsampled input, 2) increased depth, 3) increased width. 이는 많은 연산을 필요로 하고 inference시간이 오래 걸리게 한다.
한편 대부분의 CNN모델은 모든 features를 평등하게 취급하거나 한가지 scale에서만 다룬다. 이는 frequency level별로 충분한 표현력을 얻기 힘들게 한다.
3. Method
3.1 Network Architecture
목표: LR input image $x$를 super-resolved output $\hat{y}$로 만드는 것.
1. extract primitive feature
$$f_0 = H_{f}(x)$$
$H_{f}$: LR이미지에 적용되는 convolutional operator
2. $f_0$ is passed on to the cascading residual on the residual component
$$f_r = H_{crir}(f_0)$$
$f_{r}$: deep feature
$H_{crir}(\cdot)$: main cascading residual
3. extracted deep features from residual component are upscaled through the upsampling component
$$f_u = H_{u}(f_r)$$
$f_{u}$: upscaled feature
$H_{u}(\cdot)$: upsampling operator (pixel shuffle)
4. reconstruction component predict the super-resolved RGB color channels as an output
$$\hat{y} = H_{r}(f_u)$$
$\hat{y}$: estimated super-resolved image
$H_{u}(\cdot)$: reconstruction operator
class DRLN(nn.Module):
def __init__(self, args):
super(DRLN, self).__init__()
# n_resgroups = args.n_resgroups
# n_resblocks = args.n_resblocks
# n_feats = args.n_feats
# kernel_size = 3
# reduction = args.reduction
# scale = args.scale[0]
# act = nn.ReLU(True)
self.scale = args.scale[0]
chs = 64
self.sub_mean = ops.MeanShift((0.4488, 0.4371, 0.4040), sub=True)
self.add_mean = ops.MeanShift((0.4488, 0.4371, 0.4040), sub=False)
self.head = nn.Conv2d(3, chs, 3, 1, 1)
self.b1 = Block(chs, chs)
self.b2 = Block(chs, chs)
self.b3 = Block(chs, chs)
self.b4 = Block(chs, chs)
self.b5 = Block(chs, chs)
self.b6 = Block(chs, chs)
self.b7 = Block(chs, chs)
self.b8 = Block(chs, chs)
self.b9 = Block(chs, chs)
self.b10 = Block(chs, chs)
self.b11 = Block(chs, chs)
self.b12 = Block(chs, chs)
self.b13 = Block(chs, chs)
self.b14 = Block(chs, chs)
self.b15 = Block(chs, chs)
self.b16 = Block(chs, chs)
self.b17 = Block(chs, chs)
self.b18 = Block(chs, chs)
self.b19 = Block(chs, chs)
self.b20 = Block(chs, chs)
self.c1 = ops.BasicBlock(chs * 2, chs, 3, 1, 1)
self.c2 = ops.BasicBlock(chs * 3, chs, 3, 1, 1)
self.c3 = ops.BasicBlock(chs * 4, chs, 3, 1, 1)
self.c4 = ops.BasicBlock(chs * 2, chs, 3, 1, 1)
self.c5 = ops.BasicBlock(chs * 3, chs, 3, 1, 1)
self.c6 = ops.BasicBlock(chs * 4, chs, 3, 1, 1)
self.c7 = ops.BasicBlock(chs * 2, chs, 3, 1, 1)
self.c8 = ops.BasicBlock(chs * 3, chs, 3, 1, 1)
self.c9 = ops.BasicBlock(chs * 4, chs, 3, 1, 1)
self.c10 = ops.BasicBlock(chs * 2, chs, 3, 1, 1)
self.c11 = ops.BasicBlock(chs * 3, chs, 3, 1, 1)
self.c12 = ops.BasicBlock(chs * 4, chs, 3, 1, 1)
self.c13 = ops.BasicBlock(chs * 2, chs, 3, 1, 1)
self.c14 = ops.BasicBlock(chs * 3, chs, 3, 1, 1)
self.c15 = ops.BasicBlock(chs * 4, chs, 3, 1, 1)
self.c16 = ops.BasicBlock(chs * 5, chs, 3, 1, 1)
self.c17 = ops.BasicBlock(chs * 2, chs, 3, 1, 1)
self.c18 = ops.BasicBlock(chs * 3, chs, 3, 1, 1)
self.c19 = ops.BasicBlock(chs * 4, chs, 3, 1, 1)
self.c20 = ops.BasicBlock(chs * 5, chs, 3, 1, 1)
self.upsample = ops.UpsampleBlock(chs, self.scale, multi_scale=False)
# self.convert = ops.ConvertBlock(chs, chs, 20)
self.tail = nn.Conv2d(chs, 3, 3, 1, 1)
def forward(self, x):
x = self.sub_mean(x)
x = self.head(x)
c0 = o0 = x
b1 = self.b1(o0)
c1 = torch.cat([c0, b1], dim=1)
o1 = self.c1(c1)
b2 = self.b2(o1)
c2 = torch.cat([c1, b2], dim=1)
o2 = self.c2(c2)
b3 = self.b3(o2)
c3 = torch.cat([c2, b3], dim=1)
o3 = self.c3(c3)
a1 = o3 + c0
b4 = self.b4(a1)
c4 = torch.cat([o3, b4], dim=1)
o4 = self.c4(c4)
b5 = self.b5(a1)
c5 = torch.cat([c4, b5], dim=1)
o5 = self.c5(c5)
b6 = self.b6(o5)
c6 = torch.cat([c5, b6], dim=1)
o6 = self.c6(c6)
a2 = o6 + a1
b7 = self.b7(a2)
c7 = torch.cat([o6, b7], dim=1)
o7 = self.c7(c7)
b8 = self.b8(o7)
c8 = torch.cat([c7, b8], dim=1)
o8 = self.c8(c8)
b9 = self.b9(o8)
c9 = torch.cat([c8, b9], dim=1)
o9 = self.c9(c9)
a3 = o9 + a2
b10 = self.b10(a3)
c10 = torch.cat([o9, b10], dim=1)
o10 = self.c10(c10)
b11 = self.b11(o10)
c11 = torch.cat([c10, b11], dim=1)
o11 = self.c11(c11)
b12 = self.b12(o11)
c12 = torch.cat([c11, b12], dim=1)
o12 = self.c12(c12)
a4 = o12 + a3
b13 = self.b13(a4)
c13 = torch.cat([o12, b13], dim=1)
o13 = self.c13(c13)
b14 = self.b14(o13)
c14 = torch.cat([c13, b14], dim=1)
o14 = self.c14(c14)
b15 = self.b15(o14)
c15 = torch.cat([c14, b15], dim=1)
o15 = self.c15(c15)
b16 = self.b16(o15)
c16 = torch.cat([c15, b16], dim=1)
o16 = self.c16(c16)
a5 = o16 + a4
b17 = self.b17(a5)
c17 = torch.cat([o16, b17], dim=1)
o17 = self.c17(c17)
b18 = self.b18(o17)
c18 = torch.cat([c17, b18], dim=1)
o18 = self.c18(c18)
b19 = self.b19(o18)
c19 = torch.cat([c18, b19], dim=1)
o19 = self.c19(c19)
b20 = self.b20(o19)
c20 = torch.cat([c19, b20], dim=1)
o20 = self.c20(c20)
a6 = o20 + a5
# c_out = torch.cat([b1, b2, b3, b4, b5, b6, b7, b8, b9, b10, b11, b12, b13, b14, b15, b16, b17, b18, b19, b20], dim=1)
# b = self.convert(c_out)
b_out = a6 + x
out = self.upsample(b_out, scale=self.scale)
out = self.tail(out)
f_out = self.add_mean(out)
return f_out- optimization을 위해 L1 loss 사용
3.2 Cascading Residual on the Residual
각 cascading block은 medium skip-connection (MSC), cascading features concatenation, dense residual Laplacian modules (DRLM)으로 구성된다.
컴펙트하고 효율적인 모델을 위해 medium, long skip connections를 이용하여 블록의 정보를 계단식으로 전달한다
3.3 Dense Residual Laplacian Module
DRLM은 세가지 요소로 구성되어 있다: densely connected residual blocks unit, compression unit, Laplacian pyramid attention.
class Block(nn.Module):
def __init__(self, in_channels, out_channels, group=1):
super(Block, self).__init__()
self.r1 = ops.ResidualBlock(in_channels, out_channels) # conv-relu-conv-relu
self.r2 = ops.ResidualBlock(in_channels * 2, out_channels * 2) # conv-relu-conv-relu
self.r3 = ops.ResidualBlock(in_channels * 4, out_channels * 4) # conv-relu-conv-relu
self.g = ops.BasicBlock(in_channels * 8, out_channels, 1, 1, 0) # compression unit
self.ca = CALayer(in_channels) # pooling + attention
def forward(self, x):
c0 = x
r1 = self.r1(c0)
c1 = torch.cat([c0, r1], dim=1)
r2 = self.r2(c1)
c2 = torch.cat([c1, r2], dim=1)
r3 = self.r3(c2)
c3 = torch.cat([c2, r3], dim=1)
g = self.g(c3)
out = self.ca(g)
return outclass BasicBlock(nn.Module):
def __init__(self,
in_channels, out_channels,
ksize=3, stride=1, pad=1, dilation=1):
super(BasicBlock, self).__init__()
self.body = nn.Sequential(
nn.Conv2d(in_channels, out_channels, ksize, stride, pad, dilation),
nn.ReLU(inplace=True)
)
init_weights(self.modules)
def forward(self, x):
out = self.body(x)
return outclass CALayer(nn.Module):
def __init__(self, channel, reduction=16):
super(CALayer, self).__init__()
self.avg_pool = nn.AdaptiveAvgPool2d(1)
self.c1 = ops.BasicBlock(channel, channel // reduction, 3, 1, 3, 3)
self.c2 = ops.BasicBlock(channel, channel // reduction, 3, 1, 5, 5)
self.c3 = ops.BasicBlock(channel, channel // reduction, 3, 1, 7, 7)
self.c4 = ops.BasicBlockSig((channel // reduction) * 3, channel, 3, 1, 1)
def forward(self, x):
y = self.avg_pool(x) # 1x1xc
c1 = self.c1(y)
c2 = self.c2(y)
c3 = self.c3(y)
c_out = torch.cat([c1, c2, c3], dim=1)
y = self.c4(c_out)
return x * y
feature $f_c$는 global descriptor를 통해 $1\times1\timesc$형태로 바뀐다. Global descriptor는 AdaptiveAvgPool2d로 구형되었으며 https://underflow101.tistory.com/41에 있는 그림을 참고하면 이해하기 쉽다.
그 다음 $g_d$ features는 각 다른 스케일에서의 중요한 정보를 얻기 위해 Laplacian pyramid를 통과한다. Laplacian pyramid는 kernel dilation을 설정한 convolution layer 통해 얻을 수 있다. kernel dilation은 https://zzsza.github.io/data/2018/02/23/introduction-convolution/을 참고하자.
이를 통해 얻은 multi-level representations는 concat되고 upsampling-sigmoid를 거쳐 처음의 feature $f_c$와 곱해진다.
4. Experiments
Datasets
Train Set: DIV2K, Flicker2K
- LR size: 48 x 48
Test Set: SET5, SET14, URBAN100, B100, MANGA109
Results
Google Scholar Link
Google 학술 검색
S Anwar, N Barnes - IEEE Transactions on Pattern Analysis …, 2020 - ieeexplore.ieee.org Super-Resolution convolutional neural networks have recently demonstrated high-quality restoration for single images. However, existing algorithms often require ver
scholar.google.co.kr
GitHub
https://github.com/saeed-anwar/DRLN
GitHub - saeed-anwar/DRLN: Densely Residual Laplacian Super-resolution, IEEE Pattern Analysis and Machine Intelligence (TPAMI),
Densely Residual Laplacian Super-resolution, IEEE Pattern Analysis and Machine Intelligence (TPAMI), 2020 - GitHub - saeed-anwar/DRLN: Densely Residual Laplacian Super-resolution, IEEE Pattern Anal...
github.com
댓글