2024基于大模型的缺陷靜態(tài)檢查

基于大模型的缺陷靜態(tài)檢測(cè)LLM-basedStaticBugDetection

2024

StaticBugDetection

Software

StaticallyanalyzingthecodeIdentifyingbugs/vulnerabilities

Staticanalysistools

Buggy/correctcode

instance

ML/DLmodels

Data-driven&learning-

based

Recenttrend:LLM-basedbugdetection

Reseachersareveryinterestedin“howeffectivelydoLLMsdetectbugs?”

ItseemsquitepromisingthatLLMscanidentifybugssometimes,

Butstillnotalwaysperfectfortheprecisionandrecallinpracice.

Recenttrend:LLM-basedbugdetection

[1]ZhangC,LiuH,ZengJ,etal.Prompt-enhancedsoftwarevulnerabilitydetectionusingchatgpt.ICSE2

[2]Purba,MoumitaDas,etal.Softwarevulnerabilitydetectionusinglargelanguagemodels.ISSREW2023

[3]Fu,Michael,etal.Chatgptforvulnerabilitydetection,classification,andrepair:Howfararewe?APSE

Thereemergemanystudiesexploring

howdifferentpromptingstrategiescanhelpLLMinbugdection

Craftedinstructions

Fine-tuning

ProjectInformation

LLMs

CWEGeneralKnowledge

AdvancedPromptingStrategies

AST/CFGinPrompt

CoTreasoning

Recenttrend:advancedpromptstrategiesinLLM-basedbugdetection

[1]ZhangC,LiuH,ZengJ,etal.Prompt-enhancedsoftwarevulnerabilitydetectionusingchatgpt.ICSE2

[2]Purba,MoumitaDas,etal.Softwarevulnerabilitydetectionusinglargelanguagemodels.ISSREW2023

[3]Fu,Michael,etal.Chatgptforvulnerabilitydetection,classification,andrepair:Howfararewe?APSE

sapxItlrmuinleoreddswoLidpearfptcaemtoliachCraftedinstMructionsmredon iques

(i.e.,basedonstaticanalysis)?

ProjectInformation

AST/CFGinPrompt

CWEGeneralKnowledge CoTreasoning

Limitationsoftraditionaltechniques

HowLLMsaddressthelimitationsoftraditionaltechniques

Boundaryofknowledge

-E.g.,thespecificationsofAPIsarenotcomprehensivelyincluded

LLMsaregoodatsummarizingtheintentionofcode

Scalabilityissueofanalysismechanism

E.g.,pathexplosionininter-procedureanalysis

LLMscanavoiddivingintosomeprocedurebasedonAPIintention

Generalityissuetospecificdomains

E.g.,manuallyimplementthecheckingrulesforbusiness-relatedbugs

LLMscandetectbuggybehaviorsbasedonnaturallanguagedescription

Thistalkisabout

SynergyofLLMsandStaticAnalysis

UsingLLMstorefiningsouce/sinksandreachabilityanalysis

EnhanceLLMswithBugKnowledgeBase

UsingLLMstobuildandusebug-specificknowledgebase

Thistalkisabout

1.SynergyofLLMsandStaticAnalysis

UsingLLMstorefiningsouce/sinksandreachabilityanalysis

2.EnhanceLLMswithBugKnowledgeBase

UsingLLMstobuildandusebug-specificknowledgebase

CasestudyonResourceleakdetection(Background)

</>

LockManager.acquireLock()

…

If(LockManager!=null)LockManager.releaseLock()

…

Resourcereachabilityvalidation

Anunreachableresourcewouldnotcauseleaksevenwithouttherelease

RARpair:thepairoftheResourceAcquisitionmethodandtheResourceReleaseAPImethod

<res,acquire/release>

e.g.,<LockManager,acquireLock/relaseLock>

CasestudyonResourceleakdetection

ResourceRelease

ResourceAcquire

Unreachable

Constructcontrol-flowpaths

Identifypathsrelatedtoresource

Checktheresourcereachability

Checkiftheresourceisreleased

KeyChallenges Existingstaticanalysistools

AccurateRARpairsPool

PredefineasetofRARpairsandperformstringmatch

IncorrectRARpairsIncompleteRARpairs

PreciseResourceReachabilityValidation

Predefineseveralrules(e.g.,res!=null)

Falsepositive/negativeMissingunreachablepaths

Falsepositive

Challengesinprecisecontext-sensitiveandintuitivereasoning

Limitation1:Incomplete/IncorrectRARpairpool

PredefineasetofRARpairsandperformstringmatch

KeyChallenges

IncorrectRARpairsIncompleteRARpairs

CompleteRARPairPool

Falsepositive/negative

Existingstaticanalysistools

AhugenumberofRARpairsinopen-sourceprojects:e.g.,738RARpairsrelatedtotheLockresource

FalseNegative(lowrecall):

ItisinfeasibletodetectresourceleaksthatarerelatedtoRARundefinedintheinitialRARpairpool

Challenge1:howtobuildageneralresourceleakdetectiontoolthatcouldcoverawiderangeofRARparsindiverseprojects?

Limitation2:Mechanicalresourcereachabilityvalidation

PreciseResourceReachabilityValidation

Predefineseveralrules(e.g.,res!=null)

Missingunreachablepaths

FalsePositive

Misspotentialreachabilityvalidationchecke.g.,!res.Disabled()

FalsePositive(lowprecision):

Theunreachableresourcewithoutreleasewouldbeconsideredasresourceleak.

Challenge2:howtobuildageneralresourceleakdetectiontoolthatcouldpreciselyidentifytheresourcereachabilityvalidationindiverseprojects?

ResourceAcquire

ResourceRelease

Unreachable

FalseAlarms

Motivation:toimproveexistingstaticanalysisapproaches

Challenge2:howtobuildageneralresourceleakdetectiontoolthatcouldpreciselyperformtheresourcereachabilityvalidationindiverseprojects?

Predefined RulesDomainKnowledge

Challenge1:howtobuildageneralresourceleakdetectiontoolthatcouldcoverawiderangeofRARparsindiverseprojects?

Miningresource-relatedknowledgefromthemassivecorpusinopen-sourcesoftware.

Enhancingexistinganalysis-basedapproacheswiththeminedknowledgeforabetterunderstandingofthecodeintention.

Models

Resourcemanagementknowledgebase

(e.g.,RARpairs,reachabilitycheckingoperations)

OverviewofMIROK

Miningresource-relatedknowledgefromthemassivecorpusinopen-sourcesoftwaretoimproveresourceleakdetection

Evaluation:theimprovementoverbasicstaticanalysis

MIROKmines1,313newAbs-RARpairsfrom1,454,224Javamethods(89.2%arevalid)

MIROKinstantiates6,314RARpairsin2,261Mavenlibraries (93.3%arevalid)

OurminedRARpairsarereleasedforthecommunityandcouldbeintegratedintoexistingresourceleakdetectiontools.

MIROKdetects761leaksv.s.baselinesdetects168

73.4%(188)aremanuallycheckedastruepositive

Benchmark:46,389JavacodesnippetsinStackOverflow

OurMethod:Rule-basedmatchingbasedon1,197validAbs-RARpairs

Baseline:Rule-basedmatchingbasedon26seedAbs-RARpairs

Benchmark:10compilableJavaprojectsfromGitHub

OurMethod:Findbugs*=Findbugs+6,314RARpairsminedbyMIROK

Baseline:originalFindbugs

Results:

Findbugs*:15reports,7aretruebugs(PRwasaccepted)

Findbugs:9reports,4aretruebugs

OverviewofINFERROI

Step1:UseLLMtoidentifyresource-orientedcode

Step2:Providestaticanalysiswiththeidentifiedresource-orientedcodeforresourceleakdetection

INFERROI:LLM-basedintentioninference

PrompttemplateinINFERROI

TheanswerreturnedbyGPT-4

Formalized

ACQUIRE(client,167),RELEASE(client,186),VALIDATE(client,185)

Resource-orientedintention

INFERROI:enhancingstaticanalysiswithidentifiedintention

Alternatively: theinferredintentioncanberepresentedintheformatacceptedbyexistingstaticanalysistools(e.g.,representingasthesource/sinkspecificationqueryinCodeQL)

Evaluation:onexistingresourceleakdetectiondatasets

INFERROIcoveragesawiderangeofresourcetypes.

INFERROIachievesabesttrade-offbetweenbothdetectionrateandfalsealarms.

Evaluation:detectunknownresourceleaksonopen-sourceprojects

Inthe100methodssampledfromopen-sourceprojectsinGithub, InferROIreports16resourceleaksand12areannotatedastruebugs(7bugsareconfirmedbydevelopers)

AcceptedPRs

Evaluation:comparedtobasicGPT-4

DirectlyapplyingGPT-4withoutcombiningwithanalysistechniqueshasveryhighfalsepositives

Thistalkisabout

1.SynergyofLLMsandStaticAnalysis

UsingLLMstorefiningsouce/sinksandreachabilityanalysis

2.EnhanceLLMswithBugKnowledgeBase

UsingLLMstobuildandusebug-specificknowledgebase

UsingexistingbugstoboostLLM-basedbugdetection

Similarbugsrecurduringsoftwareevolutionoramongsimilarsoftware

SimilarCodeContext

SimilarRootCause

SimilarFixingSolutions

ProvidingrelevantexistingbugsintheinputconetxtofLLMs(Usingthein-contextlearningcapabilitiesofmodels)

UsingexistingbugstoboostLLM-basedbugdetectionviaRAG

[1]ASurveyonRAGMeetingLLMs:TowardsRetrieval-AugmentedLargeLanguageModels

Step2:Puttingtheretrievedinfointheinputpromt

RAGhasshownpromisingeffectivenessinmanysoftwareengineeringtasks

Assertiongeneration

Step1:Retrievingtherelevantinfofromtheknowledgebase

Codecompletion

Programrepair

…..

AclassicpipelineofRAG(Retrieval-basedAugmentationGeneration)[1]

Challenges:usingexistingbugstoboostLLM-basedbugdetectionviaRAG

[1]ASurveyonRAGMeetingLLMs:TowardsRetrieval-AugmentedLargeLanguageModels

HowtousetheretrievedbugstopromptLLMs?

Directlyappendintheinput?

InferenceMechanism

Howtofindthemostrelevantbugs?

Codesimilarity?

RetrievalMechanism

KnowledgeBase

KeyComponentsinRAG

Howtorepresentexistingbugsintheknowledgebase?

Justcodesnippets?

AclassicpipelineofRAG(Retrieval-basedAugmentationGeneration)[1]

Motivatingexamples

Betterretrievalstrategyisrequired

Whenonlyretrivingonlybasedoncodesimialrity

Itisverylikelytogetsemanticallyorfunctionallydifferentbugs

Motivatingexamples

Whenonlyputtingtherelevantbuggycodeintheprompt,It’shardforLLMstogetthecorrelationbetweentheretrievedbugandthegivencode

Betterin-contextpromptingstrategyisrequired

Ourinsight

RootCause

Represent

Functionality

ExistingBugs

Summarize

Knowledge-levelrepresentation

Cluster

FixingSolution

Insteadofstraightforwardcodesnippets

furtherrepresentingexistingbugswithinhigh-levelknowledgeof

naturallanguagedescriptions

Toretrievethelexically-differentbutsemantically-similarbugs

TofaciliatethecomprehensioncapabilitiesofLLMsfortheinput

Vul-RAGApproachPipeline:Knowledge-levelRAGforvulnerabilitydetection

Vul-RAGPipeline

Step1:ConstructingaknowledgebugofexistingCVEs

Step2:Retrievingtherelatedvulnerabilityknowledgeforthegivencode

Step3:reasoningwhetherthegivencodeisvulnerablebasedontheretrievedknowledge

Step1:Constructingknowledgebase(off-line)

Step2:Retrievingrelevantvulnerabilityknowledge

QueryGeneration:theabstractpurpose,detailedbehavior,andthecodeitself

CandidateKnowledgeRetrieval:three-dimensionsimilary

CandidateKnowledgeRe-ranking:re-rankcandidateknowledgeitemswith

theReciprocalRankFusion

Step3:DetectionReasoning

Ifthegivencode:

withthesimilarvulnerabilitycauses

withouttherelevantfixingoperationsitwillbeconsidered