一種基于NSCT與GoogLeNet的多傳感器圖像融合算法（英文）

一種基于NSCT與GoogLeNet的多傳感器圖像融合算法(英文)

Abstract

Inrecentyears,therehasbeenagrowinginterestinthedevelopmentofmulti-sensorimagefusiontechnology.Theaimofthistechnologyistocombineinformationfromdifferentsensorstocreateanenhancedimagethatismoreinformativeandvisuallypleasing.Inthispaper,weproposeanewmulti-sensorimagefusionalgorithmbasedontheNon-SubsampledContourletTransform(NSCT)andtheGoogLeNetmodel.TheproposedalgorithmfirstextractsthefeaturesoftheinputimagesbyusingtheGoogLeNetmodel.Then,theNSCTisusedtofusethefeaturesextractedfromdifferentsensors.Theresultsdemonstratethattheproposedalgorithmoutperformsotherstate-of-the-artalgorithmsintermsofobjectiveandsubjectiveimagequalitymeasures.

Introduction

Multi-sensorimagefusionisbecomingincreasinglypopularinvariousfieldssuchasremotesensing,surveillance,andmedicalimaging.Themainaimofmulti-sensorimagefusionistocombineinformationfromdifferentsensorstocreateanenhancedimagethatismoreinformativeandvisuallypleasing.Thegoalistoproduceanimagethatprovidesabetterrepresentationofthescenethananyoftheindividualsensorimages.

Thereareseveralchallengesinmulti-sensorimagefusion,includingtheselectionoftheappropriatetransformforfeatureextraction,theselectionoffusionmethods,andtheevaluationoftheresults.Inrecentyears,theuseofdeeplearningmodelshasbecomeincreasinglypopularinfeatureextractionandfusion.

Inthispaper,weproposeanewmulti-sensorimagefusionalgorithmbasedontheNon-SubsampledContourletTransform(NSCT)andtheGoogLeNetmodel.Therestofthispaperisorganizedasfollows.Section2discussesrelatedworkinmulti-sensorimagefusion.Section3presentstheproposedalgorithm.Section4presentstheexperimentalresultsandcomparisonwithexistingalgorithms.Finally,Section5concludesthispaper.

RelatedWork

Thepastfewyearshaveseenmanydevelopmentsinmulti-sensorimagefusion.Varioustransform-basedanddeeplearning-basedalgorithmshavebeenproposed.Someofthepopulartransform-basedalgorithmsincludeWaveletTransform(WT),StationaryWaveletTransform(SWT),ContourletTransform(CT),andNon-SubsampledContourletTransform(NSCT).Thesealgorithmshavebeenusedinconjunctionwithtraditionalfusionmethodssuchasmaximum,minimum,andaverage.

Deeplearning-basedalgorithms,primarilyusingConvolutionalNeuralNetworks(CNN),havealsobeenproposedformulti-sensorimagefusion.TheuseofCNN-basedmodelshasshownconsiderableimprovementovertraditionaltransform-basedalgorithms.However,deeplearning-basedmethodsrequirealargedatasetandsignificantcomputationalresources.

ProposedAlgorithm

TheproposedalgorithmusestheGoogLeNetmodelforfeatureextractionandtheNSCTforfusion.TheGoogLeNetmodelisadeeplearningmodeldevelopedbyGoogleforimageclassification.Ithasshownremarkableperformanceinmanyimageclassificationtasks.Weusethepre-trainedGoogLeNetmodel,whichhasbeentrainedontheImageNetdataset,forfeatureextraction.

Intheproposedalgorithm,wefirstextractthefeaturesoftheinputimagesusingthepre-trainedGoogLeNetmodel.TheGoogLeNetmodelextractshigh-levelfeaturesfromtheinputimagesandreducesthedimensionofthefeaturespace.Theextractedfeaturesarethenresizedtothesamesizeastheoriginalimages.

Next,weusetheNSCTtofusethefeaturesextractedfromdifferentsensors.TheNSCTisasecond-generationmultiscaletransformthathasbeenshowntoprovidebetterperformancethanothertransform-basedmethods.TheNSCThasseveraladvantages,includingmultiscale,multidirectionalandshift-invariantproperties.

Intheproposedalgorithm,weapplytheNSCTtothefeaturesextractedfromdifferentsensors.TheNSCTcoefficientsofthefeaturesarecombinedbyusingthelocalenergycriteriontoobtainthefusedcoefficients.ThefusedcoefficientsarethenreconstructedusinginverseNSCTtoobtainthefinalfusedimage.

ExperimentalResults

Toevaluatetheproposedalgorithm,weconductedexperimentsonseveraldatasets,includingtheinfraredandvisiblelightsensordataset.

Wecomparedtheproposedalgorithmwithseveralstate-of-the-artalgorithms,includingtheSWT,CT,andCNN-basedalgorithms.

Toevaluatetheobjectivequalityofthefusedimages,weusedseveralevaluationmetrics,includingthePeakSignal-to-NoiseRatio(PSNR),StructuralSimilarityIndex(SSIM),andFusionQualityIndex(FQI).Theresultsdemonstratethattheproposedalgorithmoutperformsotherstate-of-the-artalgorithmsintermsofobjectivequalitymetrics.

Toevaluatethesubjectivequalityofthefusedimages,weconductedsubjectiveexperimentswithagroupofhumanobservers.Theobserverswereaskedtoratethequalityofthefusedimagesonafive-pointscale.Theresultsdemonstratethattheproposedalgorithmoutperformsotherstate-of-the-artalgorithmsintermsofsubjectivequality.

Conclusion

Inthispaper,weproposedanewmulti-sensorimagefusionalgorithmbasedontheNon-SubsampledContourletTransform(NSCT)andtheGoogLeNetmodel.TheproposedalgorithmfirstextractsthefeaturesoftheinputimagesusingtheGoogLeNetmodel.Then,theNSCTisusedtofusethefeaturesextractedfr