變異函數(shù)python-使用Python進行變異測試

變異函數(shù)python_使?Python進?變異測試變異函數(shù)pythonWeneedtokillthemutants—no,I’mnotavillainfromtheX-Mencomics.I’masoftwareengineerwhowantstoimproveunittests.我們需要殺死這些突變體-不，我不是X戰(zhàn)警漫畫中的反派。我是?位軟件?程師，希望改善單元測試。Inthisarticleyouwilllearnwhatmutationtestingisandhowitcanhelpyoutowritebettertests.TheexamplesareforPython,buttheconceptsholdingeneralandintheendIhavealistoftoolsinotherlanguages.在本?中，您將學(xué)習(xí)什么是突變測試以及它如何幫助您編寫更好的測試。這些?例是針對Python的，但是這些概念是通?的，最后我有了其他語?的?具列表。為什么我們需要進?突變測試？(Whydoweneedmutationtesting?)Unittestshavetheissuethatit’sunclearwhenyourtestsaregoodenough.Doyoucovertheimportantedgecases?Howdoyoutestthequalityofyourunittests?單元測試存在?個問題，即您的測試何時?夠好尚不清楚。您涵蓋重要的案例嗎？您如何測試單元測試的質(zhì)量？Typicalmistakesareslightconfusions.Accessinglist[i]insteadoflist[i-1],lettingthelooprunfori<ninsteadofi<=n,initializingavariablewithNoneinsteadoftheemptystring.Therearealotofthoseslightchangeswhichareusuallyjustcalled“typos”or“off-by-one”mistakes.WhenImakethem,Ioftendidn’tthinkaboutthepartthoroughlyenough.典型的錯誤是輕微的混亂。訪問list[i]?不是list[i-1]，讓循環(huán)在i<n?不是i<=n，并使?None?不是空字符串初始化變量。許多細(xì)微的變化通常被稱為“錯別字”或“?對?”錯誤。當(dāng)我制作它們時，我常常沒有充分地考慮零件。Mutationtestingtestsyourunittests.Thekeyideaistoapplythoseminorchangesandruntheunitteststhatcouldfail.Ifaunittestfails,themutantwaskilled.Whichiswhatwewant.Itshowsthatthiskindofoff-by-onemistakecannothappenwithourtestsuite.Ofcourse,weassumethattheunitteststhemselvesarecorrectoratworstincomplete.Henceyoucanseeamutationtestasanalternativetotestcoverage.Incontrasttotestcoverage,themutationtestingtoolkitcandirectlyshowyouplacesandtypesofmistakesyouwouldnotcoverrightnow.變異測試會測試您的單元測試。關(guān)鍵思想是應(yīng)?這些較?的更改并運?可能失敗的單元測試。如果單元測試失敗，則突變體被殺死。這就是我們想要的。它表明，我們的測試套件不可能發(fā)?這種?次性的錯誤。當(dāng)然，我們假設(shè)單元測試本?是正確的，或者最糟糕的是不完整。因此，您可以看到變異測試可以替代測試覆蓋率。與測試覆蓋率相反，變異測試?具包可以直接向您顯?您現(xiàn)在不會覆蓋的錯誤的位置和類型。有哪些突變測試?具？(Whichmutationtestingtoolsarethere?)Thereareacoupleoftoolslikecosmic-ray,butdidaprettyamazingjobbycreatingmutmut.AsofAugust2020,mutmutisthebestlibraryforPythontodomutationtesting.有?對夫婦像宇宙射線的?具，但通過創(chuàng)建mutmut做了?常了不起的?作。截?2020年8?，mutmut是Python進?突變測試的最佳庫。Toruntheexamplesinthisarticle,youhavetoinstall:要運?本?中的?例，您必須安裝：pipinstallmutmutInotherlanguages,youmightwanttotrythese:使?其他語?，您可能需要嘗試以下?法：C/C++:C/C++：Java:()Java：()JavaScript:JavaScript：PHP:(formerlycalledhumbug)PHP：(以前稱為humbug)Ruby:Ruby：Rust:Rust：Swift:斯威夫特：為什么分?和線路覆蓋范圍不夠？(Whyisn’tbranchandlinecoverageenough?)Itisprettyeasytogettoahighlinecoveragebycreatingbadtests.Forexample,takethiscode:通過創(chuàng)建不良測試，很容易獲得較?的覆蓋率。例如，使?以下代碼：deffibonacci(n:int)->int:"""Getthen-thFibonaccinumber,startingwith0and1."""a,b=0,1for_inrange(n):a,b=b,a+breturnb#BUG!shouldbea!deftest_fibonacci():fibonacci(10)Thissmoketestalreadyaddssomevalueasitmakessurethatthingsarenotcrashingforasingleinput.However,itwouldnotfindanylogicbug.Thereisanassertstatementmissing.Thispatterncanquicklydriveupthelinecoverageupto100%,butyouarethenstilllackinggoodtests.此煙霧測試已經(jīng)增加了?些價值，因為它可以確保單個輸?不會崩潰。但是，它不會發(fā)現(xiàn)任何邏輯錯誤。缺少assert語句。這種模式可以快速將線路覆蓋率提?到100％，但是您仍然缺乏良好的測試。Amutationtestcannotbefooledaseasily.Itwouldmutatethecodeand,forexample,initializebwith0insteadof1:突變測試不容易被愚弄。它將使代碼突變，例如，將b初始化為0?不是1：-a,b=0,1+a,b=0,0Thetestwouldstillsucceedandthusthemutantwouldsurvive.Whichmeansthemutationtestingframeworkwouldcomplainthatthislinewasnotproperlytested.Inotherwords:測試仍將成功，因此突變體將存活。這意味著變異測試框架會抱怨此?未正確測試。換?種說法：Mutationtestingprovidesanotherwaytogetamorerigidlinecoverage.Itcanstillnotguaranteethatatestedlineiscorrect,butitcanshowyoupotentialbugsthatyourcurrenttestsuitewouldnotdetect.變異測試提供了獲得更嚴(yán)格的線覆蓋率的另?種?法。它仍然不能保證測試的?是正確的，但是可以顯?當(dāng)前測試套件?法檢測到的潛在錯誤。創(chuàng)建突變體！(Createthemutants!)Asalways,Iusemysmalllibraryasanexample.Atthemoment,ithasa99%branchand99%linecoverage.與往常?樣，我以我的?型庫為例。?前，它具有99％的分?和99％的線路覆蓋率。$mutmutrun-Mutationtestingstarting-Thesearethesteps:1.Afulltestsuiterunwillbemadetomakesurewecanrunthetestssuccessfullyandweknowhowlongittakes(todetectinfiniteloopsforexample)2.MutantswillbegeneratedandcheckedResultsarestoredin.mutmut-cache.Printfoundmutantswith`mutmutresults`.Legendforoutput:Killedmutants.Thegoalisforeverythingtoendupinthisbucket.Timeout.Suspicious.Survived.Skipped.Testsuitetook10timesaslongasthebaselinesowerekilled.Teststookalongtime,butnotlongenoughtobefatal.Thismeansyourtestsneedstobeexpanded.Skipped.1.Runningtestswithoutmutations?Running...Done2.Checkingmutants?1818/18181303165080Thistakesover1.5hoursformpu.mpuisasmallproject,withonlyabout2000linesofcode:對于MPU，這需要1.5個?時以上。mpu是?個?項?，只有?約2000?代碼：LanguagefilesblankcommentcodePython2268113992046Onepytestrunofthempuexampleprojecttakesroughly9secondsandtheslowest3testsare:mppu?例項?的pytest運??約需要9秒，最慢的3個測試是：1.03scalltests/test_main.py::test_parallel_for0.80scalltests/test_string.py::test_is_email0.41scalltests/test_io.py::test_download_without_pathIntheend,youwillseehowmanymutantsweresuccessfullykilled(),howmanyreceivedatimeout()andwhichonessurvived().Especiallythetimeoutonesareannoyingastheymakethemutmutrunsslower,butthecodeandthetestsmightstillbefine.最后，您將看到成功殺死了多少個突變體()，有多少突變體超時()，還有哪些幸存下來()。尤其是超時的代碼令?討厭，因為它們會使mutmut的運?速度變慢，但是代碼和測試可能仍然不錯。應(yīng)?哪些突變？(Whichmutationsareapplied?)mutmut2.0createsthefollowingmutants():mutmut2.0創(chuàng)建以下變量()：Operatormutations:About30differentpatternslikereplacing+by-,*by**andsimilar,butalso>by>=.運算?突變：?約30種不同的模式，例如?+替換-，*替換**和類似**以及>替換>=。Keywordmutations:ReplacingTruebyFalse,inbynotinandsimilar.關(guān)鍵字突變：?False代替True，?notin代替innotin并且類似。Numbermutations:Youcanwritethingslike0b100whichisthesameas4,0o100,whichis64,0x100whichis256,.12whichis0.12andsimilar.Thenumbermutationstrytocapturemistakesinthisarea.mutmutsimplyadds1tothenumber.數(shù)突變：你可以寫的東西像0b100是?樣的4，0o100，這是64個，0x100這是256，.12是0.12和類似。數(shù)字突變試圖捕獲該區(qū)域中的錯誤。mutmut只是將數(shù)字加1。Namemutations:Thenamemutationscapturecopyvsdeepcopyand""vsNone.名稱突變：名稱突變捕獲copyvsdeepcopycopy和""vsNone。Argumentmutations:Replaceskeywordargumentsonebyonefromdict(a=b)todict(aXXX=b).參數(shù)突變：將關(guān)鍵字參數(shù)從dict(a=b)到dict(aXXX=b)。or_testandand_test:and?oror_test和and_test：and?orStringmutation:AddingXXtothestring.字符串突變：在字符串中添加XX。Thosecanbegroupedintothreeverydifferentkindsofmutations:valuemutations(stringmutation,numbermutation),decisionmutations(switchif-elseblocks,e.g.theor_test/and_testandthekeywordmutations)andstatementmutations(removingorchangingalineofcode).可以將它們分為三種?常不同的突變：值突變(字符串突變，數(shù)字突變)，決策突變(切換if-else塊，例如or_test/and_test和關(guān)鍵字突變)和語句突變(刪除或更改?)代碼)。Thevaluemutationsaremostoftenfalse-positiveforme.I’mnotcertainifIcouldwritemycodeormytestsinanotherwaytofixthis.I’vebrieflydiscusseditwiththelibraryauthor,butapparentlyhedoesnothavethesameissue.Ifyou’reinterestedinthatdiscussion,see.價值突變對我來說通常是假陽性。我不確定是否可以?其他?式編寫代碼或測試來解決此問題。我已經(jīng)與庫作者進?了簡短的討論，但是顯然他沒有相同的問題。如果您對此討論感興趣，請參閱。如何獲取帶有mutmutHTML報告？(HowcanIgetaHTMLreportwithmutmut?)$mutmuthtmlgivesyou給你IndexpageofthemutmutHTMLreport.ImagebyMartinThoma.mutmutHTML報告的索引頁。圖?由MartinThoma提供。Thecompletepd.pyreport.ImagebyMartinThoma.完整的pd.py報告。圖?由MartinThoma提供。Asyoucansee,theindexclaimsthat108mutantssurvivedandtheHTMLreportonlyshowsone.Thatoneisalsoafalse-positiveasachangeintheloggingmessagedoesnotcauseanyissue.如您所見，該索引聲稱有108個突變體幸存下來，?HTML報告僅顯?了?個。這也是?個假陽性，因為?志消息中的更改不會引起任何問題。Alternatively,youcanusethejunitXMLtogenerateareport:另外，您可以使?junitXML?成報告：$pipinstalljunit2html$mutmutjunitxml>mutmut-results.xml$junit2htmlmutmut-results.xmlmutmut-report.htmlThereportshowsthisindexpage:該報告顯?此索引頁?：TestreportgeneratedfromJUnitXML.ImagebyMartinThoma從JUnitXML?成的測試報告。圖?由MartinThoma提供Clickingononemutant,yougetsthis:單擊?個突變體，您將得到：Mutant#3waskilled,butmutant#4survived.Ididnotusetheglobalvariable“countries”anywhereinthetests.ImagebyMartinThoma.＃3突變體被殺死，但＃4突變體幸存下來。我在測試中的任何地?都沒有使?全局變量“國家”。圖?由MartinThoma提供。TheissuewiththisgeneratedHTMLreportisthatitshowsmanyresultsforasinglelineofcodeandnogrouping.Ifthefailuresweregroupedbyfileandifonecouldseethecodeinwhichlineswithsurvivingmutantswouldbehighlighted,itwouldbewaymoreuseful.?成HTML報告的問題在于，它在??代碼中顯?了許多結(jié)果，但沒有分組。如果將失敗按?件進?分組，并且可以看到突出顯?幸存的突變體?的代碼，則將更加有?。機器學(xué)習(xí)系統(tǒng)的變異測試(MutationTestingforMachineLearningSystems)I’vesearchedforcoolapplicationsofmachinelearningtogeneratemutantsincode,butI’veonlyfound“MachineLearningApproachinMutationTesting”from2012(12citations).我?直在尋找機器學(xué)習(xí)的涼爽應(yīng)?，以在代碼中?成突變體，但從2012年起我才發(fā)現(xiàn)“突變測試中的機器學(xué)習(xí)?法”(被引?12次)。Iwashopingtofinddata-basedcodemutantgenerationtechniques.Forexample,onecouldsearchforgitcommitswhicharebugfixesbyexaminingthecommitmessage.Ifthefixisrathershort,thisisakindofmutationonecouldtestfor.Insteadofgeneratingallpossiblemutants,onecouldsamplefromthemutantsinawaytofirsttakethemostpromisingones;theonesthataremostlikelynotperceivedasafalse-positive.我希望找到基于數(shù)據(jù)的代碼突變體?成技術(shù)。例如，可以通過檢查提交消息來搜索git提交，這些提交是錯誤修復(fù)程序。如果修復(fù)程序很短，則這是?種可以測試的突變。與其?成所有可能的突變體，不如從突變體中取樣，?先選擇最有希望的突變體。最有可能不會被認(rèn)為是假陽性的?。Otherworkwasmorefocusedonmakingmachinelearningsystemsmorerobust(,,an).Idon’tknowthisstreamofworkwellenoughtowriteaboutit.ButitsoundssimilartotechniquesIknow:其他?作更多地集中在使機器學(xué)習(xí)系統(tǒng)更強?(，，)。我不知道這??的?作?夠好來寫。但這聽起來類似于我所知道的技術(shù)：Toovercomescarcityintrainingdata,variousdataaugmentationtechniquessuchasrotations,flips,orcoloradjustmentsare