Joint Dictionary Learning for Example-based Image Super-resolution

0.8 MB·

by  Mojtaba Sahraee-Ardakan

JOINTDICTIONARYLEARNINGFOREXAMPLE-BASEDIMAGESUPER-RESOLUTION
MojtabaSahraee-Ardakan1,MohsenJoneidi2
1DepartmentofElectricalEngineering,SharifUniversityofTechnology,Iran.
2DepartmentofElectricalEngineering,FerdowsiUniversityofMashhad,Iran
ABSTRACT image. However the performance of these SR methods are
7
only acceptable for small upscaling factors (usually smaller
1 In this paper, we propose a new joint dictionary learning
0 method for example-based image super-resolution (SR), us- than 2). As the upscaling factor increases, the SR problem
2 ingsparserepresentation.Thelow-resolution(LR)dictionary becomes severely ill-conditioned and a large number of LR
images are needed to recover the HR image with acceptable
n istrainedfromasetofLRsampleimagepatches. Usingthe
a sparserepresentationcoefficientsoftheseLRpatchesoverthe quality.
J
LRdictionary, thehigh-resolution(HR)dictionaryistrained

2
byminimizingthereconstructionerrorofHRsamplepatches. To address this problem, example-based SR techniques
1
Theerrorcriterionusedhereisthemeansquareerror. Inthis weredevelopedwhichrequireonlyasingleLRimageasin-
] way we guarantee that the HR patches have the same sparse put[9].Inthesemethods,anexternaltrainingdatabaseisused
V representation over HR dictionary as the LR patches over tolearnthecorrespondencebetweenmanifoldsofLRandHR
C the LR dictionary, and at the same time, these sparse repre- imagepatches.Insomeapproaches,insteadofusinganexter-
. sentations can well reconstruct the HR patches. Simulation nal database, the patches extracted from the LR image itself
s
c resultsshowtheeffectivenessofourmethodcomparedtothe across different resolutions are used [10]. In [9] Freeman et
[ state-of-artSRalgorithms. al. used a Markov network model for super-resolution. In-

1 IndexTerms— Imagesuper-resolution,sparserepresen- spired by the ideas in locally linear embedding (LLE) [11],
v the authors of [12] used the similarity between manifolds of
tation,dictionarylearning
0 HR patches and LR patches to estimate HR image patches.
2
Motivated by results of compressive sensing [13], Yang et
4 1. INTRODUCTION
al. in [14] and [15] used sparse representation for SR. In
3
0 [16]theyintroducedcoupleddictionarytraininginwhichthe
Super-resolutionistheproblemofreconstructingahighreso-
. sparserepresentationofLRimagepatchesbetterreconstructs
1 lution1 imagefromoneorseverallowresolutionimages[1].
0 It has many potential applications like enhancing the image theHRpatches.
7
quality of low-cost imaging sensors (e.g., cell phone cam-
1
eras)andincreasingtheresolutionofstandarddefinition(SD) Recently,jointandcoupledlearningmethodsareutilized
:
v moviestodisplaythemonhighdefinition(HD)TVs,toname for efficient modeling of correlated sparsity structures [17,
i
X afew. 15]. However joint learning methods and the coupled learn-
PriortoSRmethods,theusualwaytoincreaseresolution ing methods proposed in [14, 15, 18] still does not guaran-
r
a ofimageswastousesimpleinterpolation-basedmethodssuch tee that the sparse representation of HR image patches over
asbilinear,bicubicandmorerecentlytheresamplingmethod theHRdictionaryisthesameasthesparserepresentationof
describedin[2]amongmanyothers.Howeverallthesemeth- LR patches over LR dictionary. To address this problem, in
odssufferfromblurringhigh-frequencydetailsoftheimage this paper we propose a direct way to train the dictionaries
especially for large upscaling factors (the amount by which that enforces the same sparse representation for LR and HR
theresolutionofimageisincreasedineachdimension).Thus, patches. MoreoversincetheHRdictionaryistrainedbymin-
overthelastfewyears,alargenumberofSRalgorithmshave imizing the final error in reconstruction of HR patches, the
beenproposed[3]. Thesemethodscanbeclassifiedintotwo reconstructionerrorinourmethodissmaller.
categories: multi-imageSR,andsingle-imageSR.
Since the seminal work by Tsai and Huang [4] in 1984,
Therestofthispaperisorganizedasfollows. Insection
manymulti-imageSRtechniqueswereproposed[5,6,7,8].
2, Yang’s method for super-resolution via sparse representa-
IntheconventionalSRproblem,multipleimagesofthesame
tion is reviewed. In section 3, a flaw in Yang’s method is
scene with subpixel motion are required to generate the HR
discussed,andourmethodtosolvethisproblemispresented.
1Inthisarticlebyresolutionwemeanspatialresolution. Finally,section4isdevotedtosimulationresults.
2. REVIEWOFSUPER-RESOLUTIONVIASPARSE asminimizingtherepresentationerrorofsignalsoverthedic-
REPRESENTATION tionary, while forcing these representations to be sparse by
adding a l -regularization to the error. Therefore λ can be
1
InSRviasparserepresentationwearegiventwosetsoftrain- usedasaparameterthatbalancesthesparsityandtheerror;a
ingdata: asetofLRimagepatches,andasetofcorrespond- largerλresultsinsparserrepresentationswithlargererrors.
ingHRimagepatches. Inotherwords,inthetrainingdatawe
have pairs of LR and HR image patches. The goal of SR is
2.2. DictionarylearningforSR
to use this database to increase the resolution of a given LR
image. Given the sets of LR and HR training patches, {y }N and
i i=1
Let {yi}Ni=1 be the set of LR patches (each patch is ar- {xi}Ni=1, by defining Y (cid:44) [y1 ··· yN] , and having (3) in
ranged into a column vector y ) and {x }N be the set of mind,Yangetal.in[15]proposedthefollowingjointdictio-
i i i=1
correspondingHRpatches.InSRusingsparserepresentation, nary learning to ensure that the sparse representation of LR
theproblemistotraintwodictionariesDlandDhfortheset patches over Dl is the same as sparse representation of HR
ofLRpatches(orafeatureofthesepatches)andHRpatches imagepatchesoverDh:
respectively, such that for any LR patch y , its sparse rep-
i min (cid:107)Y −D W(cid:107)2 +(cid:107)X−D W(cid:107)2 +λ(cid:107)W(cid:107)
resentation w over D , reconstructs the corresponding HR l F h F 1
i l Dl,Dh,W
patchx usingD :x ≈D w [15].Towardsthisend,first (4)
i h i h i
thedictionarylearningproblemisbrieflyreviewedinsection ThekeypointhereisthattheyhaveusedthesamematrixW
2.1. ThenthedictionarylearningmethodforSRproposedin forsparserepresentationofbothLRandHRpatchestomake
[15] is studied in section 2.2. Finally in section 2.3, it is surethattheirrepresentationisthesameoverthedictionaries
shownhowthesetraineddictionariescanbeusedtoperform D andD . IfwedefinetheconcatenatedspaceofHRand
l h
SRonaLRimage. LRpatches:
(cid:20) (cid:21) (cid:20) (cid:21)
Y D
Z = , D = l
2.1. Dictionarylearning X Dh
then joint dictionary training (4) can also be written equiva-
Given a set of signals {x }N , dictionary learning is the
i i=1 lentlyas
problem of finding a wide matrix D over which the signals
min (cid:107)Z−DW(cid:107)2 +λ(cid:107)W(cid:107) . (5)
have sparse representation [19]. This problem is highly re- F 1
D,W
latedtosubspaceidentification[20]. However,sparsityhelps
Thisformulationisclearlythesameas(3). Inotherwords,in
us to turn the subspace recovery to a well-defined problem.
the concatenated space, joint dictionary learning is the same
Thisapproachhasattractedlotofattentionsinthelastdecade
asconventionaldictionarylearning,andanydictionarylearn-
and found diverse applications [21, 22, 23]. If we denote
ingalgorithmcanbeusedforjointdictionarylearning.
thesparserepresentationofx overD byw ,thedictionary
i i
learningproblemcanbeformulatedas
2.3. Super-Resolution
N
min (cid:88)(cid:107)w (cid:107) s.t.(cid:107)x −Dw (cid:107)2 ≤(cid:15),i=1,...,N AftertrainingthetwodictionariesDl andDh,theinputLR
i 0 i i 2
D,{wi}Ni=1i=1 imagecanbesuper-resolvedusingthefollowingsteps:
(1)
1. TheinputLRimageisdividedintoasetofoverlapping
in which the (cid:107) · (cid:107) is the l -norm which is the number of
0 0
LRpatches: {yLR}M .
nonzero components of a vector and (cid:15) is a small constant i i=1
whichdeterminesthemaximumtolerableerrorinsparserep- 2. FromeachimagepatchyLR,subtractitsmean,η ,
resentations.Replacingthel -normbyl -norm,Yangetal.in i i
0 1
[15]usedthefollowingformulationforsparsecodinginstead yˆLR (cid:44)yLR−η ,
i i i
of(1)
andfinditssparserepresentationoverD
l
N N
(cid:88) (cid:88)
min (cid:107)x −Dw (cid:107)2+λ (cid:107)w (cid:107) . (2) w =argmin(cid:107)yˆLR−D α (cid:107)2+λ(cid:107)α (cid:107) .
i i 2 i 1 i i l i 2 i 1
D,{wi}Ni=1i=1 i=1 αi
By defining X (cid:44) [x ··· x ] and W (cid:44) [w ··· w ], it 3. Using the sparse representation of each LR patch and
1 N 1 N
canberewritteninmatrixformas itsmean,thecorrespondingHRpatchisestimatedby
min (cid:107)X−DW(cid:107)2F +λ(cid:107)W(cid:107)1, (3) xˆHi R =Dh·wi, xHi R =xˆHi R+ηi.
D,W
in which (cid:107)·(cid:107) stands for the Frobenius norm. (2) and (1) 4. CombiningtheestimatedHRimagepatches,theoutput
F
arenotequivalent,butcloselyrelated. (2)canbeinterpreted HRimageisgenerated.
3. OURPROPOSEDMETHOD
OurmethodforSRistoimprovethedictionarylearningpart
of Yang’s method, described in section 2.2. Having the dic-
tionaries trained, the rest of the method is the same as what
describedinsection2.3.
As mentioned earlier, in SR the dictionaries should be
trained in a way that the sparse representation of each LR
patch well reconstructs the corresponding HR patch. The
Yang’s method uses (4) to accomplish this. It uses the same
sparserepresentationmatrixW forbothLRandHRpatches (a)Original (b)Bicubicinterpolation
to ensure that each LR and HR patch, both have the same
sparserepresentation. Howeverascanbeseenfrom(5),this
joint dictionary learning is only optimal in the concatenated
spaceofLRandHRpatches,butifwelookatthespaceofLR
andHRpatchesseparately,wemayfindasparserrepresenta-
tionforsomepatchesthanthesparserepresentationfoundin
theconcatenatedspace.
To address this problem, note first that the SR method
described in section 2.3 consists of two distinct operations:
findingthesparserepresentationoftheLRpatch,andthere-
construction of the HR patch. Then, we note that the first (c)Yang’smethod (d)Proposedmethod
operation uses only D , and the second operation uses only
l
D . Therefore,insteadoftrainingthedictionariesjointlyas
h Fig. 1. Results of Lena image magnified by a factor of 2
in(4),weproposetotrainD forLRpatchessolely,andthen
l using: (b) Bicubic interpolation, (c) Yang’s method, (d) our
to train the HR dictionary by minimizing the reconstruction
proposedmethod. Theoriginalimageisalsogivenin(a)for
errorwhensparserepresentationofLRpatchesareused.
comparison.
Mathematically,weproposetotraintheLRdictionaryas
min (cid:107)Y −D W(cid:107)2 +λ(cid:107)W(cid:107) , (6)
l F 1 4. SIMULATIONRESULTS
Dl,W
which is a conventional dictionary learning problem. After
In this section we compare the performance of our method
trainingtheLRdictionary,foreachLRpatch,itssparserep-
with Yang’s method. The error criteria used here are Peak
resentationw isfoundoverD (notethatthisstepisalready
i l Signal to Noise Ratio (PSNR) and Structural SIMilarity
doneduringthedictionarytrainingin(6))
(SSIM)index[24]. PSNRcriterionisdefinedas
W =argmin(cid:107)Y −DlV(cid:107)2F +λ(cid:107)V(cid:107)1. (7) (cid:18) 255 (cid:19)
V PSNR=20log10 √ , (10)
MSE
Using the sparse representation of LR patches W, the HR
dictionary D is found such that the reconstruction error of whereMSEismeansquareerrorgivenby
h
thecorrespondingHRpatchesareminimized,thatis,
(cid:107)I −I (cid:107)2
MSE= SR g F, (11)
D =argmin(cid:107)X−D W(cid:107)2. (8) mn
h h F
Dh inwhichI istheoriginaldistortion-freeimage,andI is
g SR
This is an unconstrained quadratic optimization problem thesuper-resolvedimagederivedfromtheSRalgorithm,and
whichhasthefollowingclosed-formsolution: mandnaredimensionsoftheimageinpixels.
ForthedefinitionofSSIMreferto[24]. Fromthesedef-
(cid:16) (cid:17)−1
D =XW WWT =XW† (9) initionsitisclearthathigherPSNRmeanslessmeansquare
h
error, however it does not necessarily mean a better image
inwhich(·)T and(·)†representtransposeandpseudo-inverse qualitywhenperceivedbyhumaneye.Manyothererrorcrite-
ofamatrix,respectively. riahavebeenproposedtosolvethisproblemofPSNR.SSIM
NotethatunlikeYang’smethod, intheproposedmethod is one of these error criteria. But still PSNR is widely used
D isnottrainedinawaythatexplicitlyenforcesthesparsity becauseofitssimplemathematicalform. Thiscanbeseenin
h
ofrepresentationofHRpatchesoverit,ratheritistrainedto (8) where MSE is to train the HR dictionary, but in order to
minimizethefinalreconstructionerror. showtheeffectivenessofourmethod,hereweusebothSSIM
performing better than Yang’s method. The images super-
Table 1. PSNR and SSIM of some images magnified us-
resolved by the proposed method have on average a higher
ing bicubic iterpolation, Yang’s method and our proposed
SSIMthanimagesrecoveredbyYang’smethod. SinceSSIM
method. TheaveragePSNRsandSSIMsaregiveninthelast
is much more consistent with the image quality as it is per-
row. Bestperformanceineachrowiswritteninboldface.
ceivedbyhumaneyecomparedtoPSNR,higherSSIMofim-
Bicubic Yang proposed
ages recovered by our method suggests that they also have
PSNR 32.79 34.73 34.86 bettervisualquality.
Lena
SSIM 0.9012 0.9268 0.9283
5. CONCLUSIONANDFUTUREWORKS
PSNR 26.50 27.77 27.89
Parthenon
SSIM 0.8334 0.8737 0.8762
In this paper, we presented a new dictionary learning algo-
PSNR 24.66 25.30 25.39 rithm for example-based SR. The dictionaries were trained
Baboon
from a set of sample LR and HR image patches in order to
SSIM 0.9529 0.9872 0.9873
minimize the final reconstruction error. Simulation results
PSNR 27.93 28.61 28.59
Barbara on real images showed the effectiveness of our algorithm in
SSIM 0.9609 0.9852 0.9852 super-resolving images with less error compared to Yang’s
method.TheaveragePSNRandaverageSSIMofimagespro-
PSNR 30.51 33.24 33.36
Flower duced by our method were higher than images recovered by
SSIM 0.9230 0.9526 0.9538
Yang’smethod. Infuture, wecanextendthisworkbytrain-
PSNR 28.47 29.93 30.02 ing the HR dictionary using a better error criterion instead
Average
ofPSNR.Oneoftheadvantagesofourmethodisthattrain-
SSIM 0.9143 0.9451 0.9462
ing of D is separated from D in (6) and (8). We can use
h l
anothererrorcriterionthatbetterrepresentstheimagequality
likeSSIMin(8)withoutmakingthetrainingofD morecom-
l
andPSNRtocompareimagesproducedbyourmethodwith
plex. ChangingtheerrorcriterionineachofYang’smethods
Yang’smethod.
will make the optimizations in their algorithms much more
Tomakeafaircomparison,thesamesetof80000training
complex.
data patches sampled randomly from natural images is used
totraindictionariesforbothYang’sandourmethod. Thesize
6. REFERENCES
ofLRpatchesis5×5andtheyaremagnifiedbyafactorof2,
i.e.thesizeofgeneratedHRimagepatchesis10×10.TheLR
[1] Sung Cheol Park, Min Kyu Park, and Moon Gi Kang,
patchesextractedfromtheinputimagehavea4pixeloverlap.
“Super-resolution image reconstruction: a technical
Dictionarysizeisfixedat1024,andλ=0.15isusedforboth
overview,” SignalProcessingMagazine,IEEE,vol.20,
methodsasin[15].
no.3,pp.21–36,2003.
In Fig. 1, simulation results of Yang’s method and pro-
posed method on Lena image can be seen. The original [2] LeiZhangandXiaolinWu, “Anedge-guidedimagein-
image and the image magnified using bicubic interpolation terpolation algorithm via directional filtering and data
are also given as references. The PSNRs of these images fusion,” ImageProcessing, IEEETransactionson, vol.
are 32.79dB, 34.73dB and 34.86dB for bicubic interpola- 15,no.8,pp.2226–2238,2006.
tion, Yang’s method and ours respectively. It is clear that
the quality of images magnified by SR is much better than [3] Peyman Milanfar, Super-resolution imaging, CRC
the image magnified by bicubic interpolation and the details Press,2010.
are more visible, which has resulted in sharper images. But
the difference between image (c) and image (d) is not no- [4] RY Tsai and T.S. Huang, “Multiframe image restora-
ticeable visually, although the PSNR of image (d) which is tionandregistration,” Advancesincomputervisionand
super-resolvedbyourmethodisabout0.1dBhigher. ImageProcessing,vol.1,no.2,pp.317–339,1984.
In Table 1 the PSNRs and SSIMs of some images pro-
[5] Michael Elad and Arie Feuer, “Restoration of a single
duced by our method is compared with those of Yang’s
superresolution image from several blurred, noisy, and
method and bicubic interpolation. Almost all of the images
undersampled measured images,” Image Processing,
recovered by our method have higher PSNRs than images
IEEE Transactions on, vol. 6, no. 12, pp. 1646–1658,
recovered by Yang’s method. The average PSNRs given in
1997.
the last row show that our method performs slightly better
thanYang’smethodonaverage. [6] MichaelEladandArieFeuer, “Super-resolutionrecon-
The SSIMs in Table 1 also confirm that our method is struction of image sequences,” Pattern Analysis and
Machine Intelligence, IEEE Transactions on, vol. 21, [16] Jianchao Yang, Zhaowen Wang, Zhe Lin, Scott Co-
no.9,pp.817–834,1999. hen, andThomasHuang, “Coupleddictionarytraining
for image super-resolution,” Image Processing, IEEE
[7] DavidCapelandAndrewZisserman, “Super-resolution
Transactionson,vol.21,no.8,pp.3467–3478,2012.
from multiple views using learnt image models,” in
ComputerVisionandPatternRecognition,2001.CVPR [17] A. Taalimi, H. Qi, and R. Khorsandi, “Online multi-
2001.Proceedingsofthe2001IEEEComputerSociety modal task-driven dictionary learning and robust joint
Conferenceon.IEEE,2001,vol.2,pp.II–627. sparserepresentationforvisualtracking,” inAdvanced
VideoandSignalBasedSurveillance(AVSS),201512th
[8] SinaFarsiu, MDirkRobinson, MichaelElad, andPey-
IEEEInternationalConferenceon,Aug2015,pp.1–6.
manMilanfar, “Fastandrobustmultiframesuperreso-
lution,” Image processing, IEEE Transactions on, vol. [18] A.Taalimi,A.Rahimpour,C.Capdevila,Z.Zhang,and
13,no.10,pp.1327–1344,2004. H. Qi, “Robust coupling in space of sparse codes for
multi-view recognition,” in 2016 IEEE International
[9] WilliamTFreeman,ThouisRJones,andEgonCPasz-
ConferenceonImageProcessing(ICIP),Sept2016,pp.
tor, “Example-based super-resolution,” Computer
3897–3901.
GraphicsandApplications,IEEE,vol.22,no.2,pp.56–
65,2002. [19] Michael Elad, Sparse and redundant representations:
fromtheorytoapplicationsinsignalandimageprocess-
[10] DanielGlasner,ShaiBagon,andMichalIrani, “Super-
ing, Springer,2010.
resolution from a single image,” in Computer Vision,
2009 IEEE 12th International Conference on. IEEE,
[20] MostafaRahmaniandGeorgeAtia,“Innovationpursuit:
2009,pp.349–356.
Anewapproachtosubspaceclustering,” arXivpreprint
arXiv:1512.00907,2015.
[11] Sam T Roweis and Lawrence K Saul, “Nonlinear di-
mensionality reduction by locally linear embedding,”
[21] ShervinMinaee, AmiraliAbdolrashidi, andYaoWang,
Science,vol.290,no.5500,pp.2323–2326,2000.
“Screen content image segmentation using sparse-
smoothdecomposition,” in201549thasilomarconfer-
[12] HongChang,Dit-YanYeung,andYiminXiong,“Super-
ence on signals, systems and computers. IEEE, 2015,
resolution through neighbor embedding,” in Computer
pp.1202–1206.
VisionandPatternRecognition,2004.CVPR2004.Pro-
ceedings of the 2004 IEEE Computer Society Confer-
[22] Mahdi Abavisani, Mohsen Joneidi, Shideh Rezaeifar,
enceon.IEEE,2004,vol.1,pp.I–275.
and Shahriar Baradaran Shokouhi, “A robust sparse
[13] EmmanuelJCande`s, “Compressivesampling,” inPro- representationbasedfacerecognitionsystemforsmart-
ceedings oh the International Congress of Mathemati- phones,” in 2015 IEEE Signal Processing in Medicine
cians: Madrid, August 22-30, 2006: invited lectures, andBiologySymposium(SPMB).IEEE,2015,pp.1–6.
2006,pp.1433–1452.
[23] MJoneidi,MRahmani,HBGolestani,andMGhanbari,
[14] Jianchao Yang, John Wright, Thomas Huang, and “Eigen-gap of structure transition matrix: A new crite-
YiMa,“Imagesuper-resolutionassparserepresentation rionforimagequalityassessment,”inSignalProcessing
ofrawimagepatches,” inComputerVisionandPattern and Signal Processing Education Workshop (SP/SPE),
Recognition, 2008. CVPR 2008. IEEE Conference on. 2015IEEE.IEEE,2015,pp.370–375.
IEEE,2008,pp.1–8.
[24] ZhouWang,AlanCBovik,HamidRSheikh,andEeroP
[15] Jianchao Yang, John Wright, Thomas S Huang, and Simoncelli, “Imagequalityassessment: Fromerrorvis-
Yi Ma, “Image super-resolution via sparse representa- ibilitytostructuralsimilarity,” ImageProcessing,IEEE
tion,” ImageProcessing,IEEETransactionson,vol.19, Transactionson,vol.13,no.4,pp.600–612,2004.
no.11,pp.2861–2873,2010.