使用GitHub Copilot生成的代碼性能評估

MeasuringthePerformanceofCodeProducedwithGitHubCopilot

by

DanielErhabor

Athesis

presentedtotheUniversityofWaterlooinfulfillmentofthe

thesisrequirementforthedegreeofMasterofMathematics

inComputerScience

Waterloo,Ontario,Canada,2022

?DanielErhabor2022

PAGE\*roman

viii

Author’sDeclaration

IherebydeclarethatIamthesoleauthorofthisthesis.Thisisatruecopyofthethesis,includinganyrequiredfinalrevisions,asacceptedbymyexaminers.

Iunderstandthatmythesismaybemadeelectronicallyavailabletothepublic.

Abstract

GitHubCopilotisanartificiallyintelligentprogrammingassistantusedbymanydevel-opers.WhileafewstudieshaveevaluatedthesecurityrisksofusingCopilot,therehasnotbeenanystudytoshowifitaidsdevelopersinproducingcodewithbetterperformance.WeevaluatetheperformanceofcodeproducedwhendevelopersuseGitHubCopilotver-suswhentheydonot.Tothisend,weconductedauserstudywith32participantswhereeachparticipantsolvedtwoC++programmingproblems,onewithCopilotandtheotherwithoutitandmeasuredtherunningtimeoftheparticipants’solutionsonourtestdata.OurresultssuggestthatusingCopilotcanproducecodewithasignificantlyslowerrunningtime.

Acknowledgements

Iwanttothankmysupervisors,MeiyappanNagappan,SamerAl-Kiswany,mycollab-oratoronthiswork,SreeharshaUdayashankar,theparticipantsforparticipatinginthestudy,membersofWASLandSWAGresearchgroups,andotherpeoplewhoadvisedonthingsrelatedtothiswork.

AsamemberoftheUniversityofWaterloo,IacknowledgethatthisworktookplaceonthetraditionalterritoryoftheNeutral,AnishinaabeandHaudenosauneepeoples.

Dedication

Idedicatethisthesistomyfriendsandfamily.

TableofContents

Author’sDeclaration

ii

Abstract

iii

Acknowledgements

iv

Dedication

v

ListofFigures

ix

ListofTables

x

Introduction

1

BackgroundandRelatedWork

3

ProgrammingProblemsSolvedbyParticipants

5

Problemselection

................................ 5

ProblemA

.................................... 6

ProblemB

.................................... 7

ModelSolutionstotheProblems

9

SolutionA

.................................... 9

Level0

.................................. 9

Level1

10

Level2

11

Level3

11

SolutionB

12

Level0

12

Level1

13

Participants

15

ParticipantRecruitment

15

DifficultiesRecruitingProfessionals

15

ParticipantSummary

16

ExperimentDesign

19

OrderofSolvingtheProblems

19

SessionIntroductionandTutorial

19

Tasks

20

Timing

21

AftertheProblem

21

BriefPost-sessionInterview

21

Evaluation

22

RQ0-DoesusingCopilotinfluenceprogramcorrectness?

22

RQ1-IstherearunningtimedifferenceincodewhenusingGitHubCopilot?

23

Approach

23

Results

24

Discussion

24

RQ2-DoCopilot’ssuggestionsswaydeveloperstoorfromcodewithfaster

runningtime?

25

Approach

25

StatementLevelOptimizations&Open-coding

26

VideoAnalysis

28

Results

28

Discussion

29

RQ3-DocharacteristicsofCopilotusersinfluencetherunningtimewhen

itisused?

36

Approach

36

Results

37

Discussion

37

Conclusion

40

Limitations

40

Takeaways

41

References

42

APPENDICES

46

AFullDescriptionoftheProblemsGiventoParticipants

47

ProblemA

47

ProblemB

52

Screeningsurvey

58

Tutorial

59

ProgrammingSurveys

61

FirstProgrammingSurvey

61

SecondProgrammingSurvey

62

ListofFigures

OverviewofMethodology

........................... 2

CopilotinAction

................................ 4

DistributionofParticipants’DeveloperExperiencefromScreeningSurvey

inAppendix

B

17

DistributionofParticipants’FamiliaritywithC++fromScreeningSurvey

inAppendix

B

18

PlotofProblemComprehensionforProblemAvsB.”x”isthemeanof

the6-PointLikert-ScaleinTable

7.4

andAppendix

D

38

ListofTables

6.1

FactorialMatrixofmodexproblem

....................

19

6.2

PossibleOrdersofmodexproblem

.....................

20

7.1

TableofInvalidRuns

..............................

23

7.2

TableofStatement-levelOptimizations&remarksforProblemA

.....

27

7.3

TableofStatement-levelOptimizations&remarksforProblemB

.....

28

7.4

TableofInputDatatoCorrelationMatrix

..................

36

7.5

CorrelationmatrixforproblemA

.......................

36

7.6

CorrelationmatrixforproblemB

.......................

37

B.1

ScreeningSurvey

................................

58

D.1

FirstProgrammingSurvey

...........................

62

D.2

SecondProgrammingSurveyPlusDemographics

...............

64

PAGE

10

Chapter1Introduction

Advancesinnaturallanguageprocessinganddeeplearninghaveresultedinlargelanguagemodels(LLMs)thatcangeneratecodefromfree-formtext.Withthis,languagemodelslikeGPT-3[

27

]havebeenextendedtowhatXuetal.[

35

]havetermedNatural-Language-to-Code(NL2Code)generators.Notably,OpenAI’sextensionoftheGPT-3languagemodel,Codex[

28

],andtheproduction-readyproductderivedfromit,GitHubCopilot[

1

],arepopularexamplesofNL2Codetoolsinusetoday.WhilesomestudieshaveshownthatdevelopersgenerallymayhaveapositiveexperienceusingGitHubCopilot,othershaveshownthatitcouldgeneratepotentiallyvulnerablecode.Wepresentthefirst-everevaluationofCopilotfromaperformanceperspectiveinsystemsprogramming.Wecon-ductedthefirstuserstudyonCopilottoevaluatetherunningtimeofthecodegeneratedwhendevelopersuseit.Withtheresultsfromourstudy,wehopetoanswerthefollowingresearchquestions:

RQ0:DoesusingCopilotinfluenceprogramcorrectness?

RQ1:IstherearunningtimedifferenceincodewhenusingGitHubCopilot??

RQ2:DoCopilot’ssuggestionsswaydeveloperstoorfromcodewithfasterrunningtime?

RQ3:DocharacteristicsofCopilotusersinfluencetherunningtimewhenitisused?

Toanswerourresearchquestions,weconductauserstudywith32participants,whereeachparticipantsolvedtwoprogrammingproblemsinC++,oneproblemwassolvedwith

Generatingtaskdesignfromproblems

A.cpp

B.cpp

runtime

surveydata

A

B.cpp

A.cpp

P1

B.cpp

A.cpp

P2P3

videoanalysis

P4

B

A.cpp

B.cpp

opencoding

RQ1

RQ3

RQ2

Problemselection

Assigningproblemstoparticipantstocounterbalanceorder-effects

Figure1.1:OverviewofMethodology

Copilotandtheotherwassolvedwithoutit.OurfindingsindicatethatusingCopilotresultedincodewithaslowerrunningtime.

Thethesisisorganizedinthefollowingway:WebrieflygooversomebackgroundrelatedtoGitHubCopilotandsomerelatedworkinChapter

2

.TheprocessofcreatingtheproblemstheparticipantswouldsolveandtherationalebehindchoosingtheproblemsisdescribedinChapter

3

.OurmodelsolutionstotheproblemsareelaboratedinChapter

4

givingcontexttotheproblems.AsummaryoftheparticipantrecruitmentprocessandtheparticipantsisdescribedinChapter

5

.WethenpresenttheexperimentdesignindetailinChapter

6

.AnoverviewofthemethodologycanbeseeninFigure

1.1

.Penultimately,weanalyzeanddiscusstheexperiment’sresults,answeringourresearchquestionsinChapter

7

.Finally,inChapter

8

,wetalkaboutthetakeawaysandlimitationsofourstudyandpotentialfuturedirections.

Chapter2

BackgroundandRelatedWork

GitHubCopilot,theproduction-readytoolbasedontheCodexmodelbyOpenAI,canbeusedasaVisualStudioCodeextensiontosuggestcodesnippetstouserswhentheextensionisactivated.Inthisway,userscanreceivesuggestionsbystartingtowritethecodeorbywritingcomments;eitherway,Copilotwillsuggestsomesnippets[

1

].SeeFigure

2.1

foranexampleofCopilotinaction.

OneoftheearlystudiesonCopilotwasbyPearceetal.[

31

],wheretheywantedtounderstandhowoftensuggestionsfromCopilotwerevulnerableandthecontextsthatmadeCopilotsuggestvulnerablecode.Toachievethis,theypromptedCopilottosuggestcodeinscenarioswheretheresultantsuggestionscouldhavebeenvulnerableormoresecure.Oftheprogramsproducedinresponsetothepotentiallyvulnerablescenarios,40%werevulnerable.

AstudybySandovaletal.[

32

]incollaborationwithPearcefrom[

31

]wantedtoassessthesecurityofcodewrittenbystudentprogrammerswhenassistedbyanNL2Codeassistant(OpenAI’scode-cushman-001model)likeCopilot.Theyconductedabetween-subjectsstudywith58computersciencestudentswhereparticipantsweretaskedwithimplementingoperationsofaSingly-LinkedListinC.Contraryto[

31

],theirresultsdidnotshowthatCopilothadaconclusiveimpactonsecurity.

Vaithilingametal.[

34

]conductedauserstudyon24participantstounderstandhowprogrammersperceiveanduseCopilot;theyfoundthatprogrammerspreferredtouseCopilotintheirday-to-dayprogrammingtasksandfoundithelpfulasastartingpoint.

Figure2.1:CopilotinAction

Chapter3

ProgrammingProblemsSolvedbyParticipants

FollowinginthesameveinasPearsonetal.[

31

],weprovided”incomplete”codeforpar-ticipantstoimplementasasolutiontoagivenproblem.By”incomplete”,wemeanthatweprovidedcodestubsandaccompanyingdocumentationforthestubsparticipantswereaskedtoimplementduringthestudy.Wecallthestubs”prompts”or”problems”andwilluseeitherofthosetermsinterchangeablythroughoutthiswork.ThesepromptswereprovidedtoparticipantsintheformofaCPPfilethatcontainedthefunctiondeclaration,theunimplementedfunctiondefinitionthatparticipantswereexpectedtoimplement,i.e.,theprimaryfunction,initializationfunctionsandsanitycheckstoverifycorrectness.Amainfunctionwasalsoprovidedasanentrypointtocalltheinitializationfunctions,theprimaryfunction,andthesanitychecksintheappropriateorder.

Problemselection

Wechosetwoproblemdomainsforourprogrammingproblems,file-systemoperationsandmultithreadedprogramming.Wechosethesetwoareasbecauseproblemsinthosedomainstendtohaveadirectimpactonapplicationperformance.WithfileI/Ooperationsaccountingforabout30%-80%ofinteractionsinnetworkedfilesystems[

29

],thereisaneedforfilesystemoperationstobefastonstoragedevices[

33

].Choosingaproblemrelatedtofilesystemsreflectsthisdemand.Additionally,sincemoderncomputingismovingtowardsamoreparalleldomain,thereisaneedtounderstandthebottlenecksofmultithreaded

applications[

30

]andoptimizeaccordingly.Toreflectthis,wechoseaproblemrelatedtofalsesharing,atypicalmulti-threadingoptimizationproblemthatisrelativelypopular[

2

].

Wechoseproblemsthatfitthefollowingcriteria:(1)theproblemmusthavemorethanonesolutionwhereeachsolutiondiffersnotincorrectnessbutperformance,(2)TheproblemshouldbesolvablewithorwithoutCopilotassistancein30minutes.ProblemAwasinthefile-systemoperationsarea,andproblemBwasinthemulti-threadingspace.

ProblemA

Forthisproblem,participantswereaskedtoreadmanyrecordsfromthree1GBlargetextfilesandwriteeachrecordtotheappropriatefilecombination.Afilecombinationisstructthatcontainedafileidentifier,abuffertowritetherecordto,andtheoffsetfortheassociatedfile.Arecordisasequenceof5000bytes.Forthisproblem,participantsreceivedaCPPfileforpromptAandthreelargetextfiles.WeprovideasummaryofrelevantdeclarationsformorecontexttotheprobleminListing

3.1

.ThefullfunctionsignaturesandtheentiretyoftheCPPfilewiththeaccompanyingdocumentationforpromptAgiventoparticipantsisinAppendix

A.1

.

1 #defineRECORD_SIZE5000

2 #defineNUM_RECORDS500000

3

4 conststd::vector<std::string>FILE_NAMES={

5 "large_file_1.txt","large_file_2.txt","large_file_3.txt"};

6

7 structFileCombo{

8 intfileId;

9 intoffset;

10 charbuffer[RECORD_SIZE+1];

11 };

12

13 voidreadFileCombos(std::vector<FileCombo>&fileCombos){

14 //YOURCODEGOESHERE

15 }

Listing3.1:SummaryofProblemA

ProblemB

Forthisproblem,participantswereaskedtouseacertainamountofthreadstosetallthevaluesinasourcearraybuffertozerowhilesettingallthevaluesinadestinationarraybuffertoaparticularvalue.However,theywerenotallowedtouseassignmentoperations,i.e.,moveandcopysemanticswerenotallowedoneitherthesourcearraybufferorthedestinationarraybuffer.Participantswereonlyallowedtoincrementordecrementthevaluesintherespectivearraybufferstosolvethetask.Thisrestrictionwasinplacebecausewewantedthreadstorepeatedlywritetoaniteminthearrayandthusshowthefalsesharingeffect(dependingontheimplementation).WeprovideasummaryofrelevantdeclarationsformorecontexttotheprobleminListing

3.2

.ThefullfunctionsignaturesandtheentiretyoftheCPPfilewiththeaccompanyingdocumentationforpromptBgiventoparticipantsareinAppendix

A.2

.

1 constintINIT_SRC_VAL=(1<<17);

2 constintSIZE=(1<<11);

3 constintTHREAD_COUNT=4;

4

5 structItem{

6 private:

7 intval;

8 Item(constItem&);

9 Item(Item&&);

10 Item&operator=(constItem&);

11 Item&operator=(Item&&);

12 public:

13 Item(){val=0;}

14 Item(inti){val=i;}

15

16 intget(){returnval;}

17

18 voidoperator++(){++val;}

19 voidoperator++(int){val++;}

20

21 voidoperator--(){--val;}

22 voidoperator--(int){val--;}

23 };

24

25 Itemsrc[SIZE];

26 Itemdst[SIZE];

27

28 voidschedule(){

29 //YOURCODEGOESHERE

30 }

Listing3.2:SummaryofProblemB

Chapter4

ModelSolutionstotheProblems

Wecreatedwhatweterm”model”solutionstotheproblems.Becausetherewasmorethanonesolutiontoeachproblem,eachsolutionwederiveddifferedonlyinperformanceandnotcorrectness.WeitemizeoursolutionshereandcategorizethemintoLevel0(L0),Level1(L1),Level2(L2),andLevel3(L3)forproblemAandLevel0(L0)andLevel1(L1)forproblemB.

SolutionA

Level0

Weconsideranaiveimplementationwhereincallstoopen,seek,read,andclosearemadeforeachfileComboinfileCombos.(SeeListing

4.1

)

1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){

2 for(auto&fileCombo:fileCombos){

3 ifstreamin;

4 in.open(FILE_NAMES[fileCombo.fileId],std::ios::binary);

5 in.seekg(fileCombo.offset);

6 in.read(fileCombo.buffer,RECORD_SIZE);

7 in.close();

8 }

9 }

Listing4.1:OurNaiveLevel0(L0)SolutiontoProblemA

Level1

Onestepfurtherfromthenaiveimplementationisacknowledgingthatonlythreefilesarebeinginteractedwith;thus,wedonotneedtoopenandcloseafileforeachfileComboinfileCombos.OuroptimizationinvolvesopeningallthefilesinFILE_NAMESfirst,thenprocessingeachfileComboinfileCombos,thenclosingallthefiles.(SeeListing

4.2

)

1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){

2 std::vector<ifstream>files(FILE_NAMES.size());

3 for(inti=0;i<FILE_NAMES.size();++i){

4 files[i].open(FILE_NAMES[i],std::ios::binary);

5 }

6 for(FileCombo&fc:fileCombos){

7 files[fc.fileId].seekg(fc.offset);

8 files[fc.fileId].read(fc.buffer,RECORD_SIZE);

9 }

10 for(ifstream&f:files){

11 f.close();

12 }

13

14 }

Listing4.2:OurLevel1(L1)SolutiontoProblemA

Level2

AsafurtherstepfromL1,inthisimplementation,wesortthefileCombosbyfileIdandbreaktiesbyoffset.Thisway,readingtherecordfromanoffsetinaspecificfilewillbesequentialandnotrandom.(SeeListing

4.3

).

1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){

2 std::sort(fileCombos.begin(),fileCombos.end(),

3 [](constFileCombo&a,constFileCombo&b){

4 if(a.fileId!=b.fileId){

5 returna.fileId<b.fileId;

6 }

7 returna.offset<b.offset;

8 });

9

10 for(FileCombo&fc:fileCombos){

11 ifstreamin;

12 in.open(FILE_NAMES[fileCombo.fileId],std::ios::binary);

13 in.seekg(fc.offset);

14 in.read(fc.buffer,RECORD_SIZE);

15 in.close();

16 }

17 }

Listing4.3:OurLevel2(L2)SolutiontoProblemA

Level3

AstepfurtherfromL2isacombinationoftheL1optimizationwedidin

4.1.2

andtheL2optimizationwedidin

4.1.3

.(SeeListing

4.4

).

1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){

2 std::vector<ifstream>files(FILE_NAMES.size());

3 for(inti=0;i<FILE_NAMES.size();++i){

4 files[i].open(FILE_NAMES[i],std::ios::binary);

5 }

6

7 std::sort(fileCombos.begin(),fileCombos.end(),

8 [](constFileCombo&a,constFileCombo&b){

9 if(a.fileId!=b.fileId){

10 returna.fileId<b.fileId;

11 }

12 returna.offset<b.offset;

13 });

14

15 for(FileCombo&fc:fileCombos){

16 files[fc.fileId].seekg(fc.offset);

17 files[fc.fileId].read(fc.buffer,RECORD_SIZE);

18 }

19

20 for(ifstream&f:files){

21 f.close();

22 }

23

24 }

Listing4.4:OurLevel3(L3)SolutiontoProblemA

SolutionB

Level0

Weconsideranaiveimplementationtobeasolutionwhereeachthreadstartsatthere-spectiveindices0,1,2,and3(whereTHREAD_COUNTis4)inthesrcanddstarrays.EachthreadthendecrementsandincrementstheIteminsrcanddst,respectively.(SeeList-ing

4.5

).AfterprocessingtherespectiveItem,eachthreadmovesTHREAD_COUNTstepsuntilthenextindex,i.e.,4,5,6,and7andprocessestheItemtherein.Weconsiderthisthe

naivesolutionbecausefalsesharingispresentbecauseeachthreadinvalidatesthesame64-bytecachelinewhendecrementingandincrementingtheItematsrcanddstarrays.

1 voidwork(intstart){

2 for(inti=start;i<SIZE;i+=THREAD_COUNT){

3 for(intj=0;j<INIT_SRC_VAL;++j){

4 --src[i];

5 ++dst[i];

6 }

7 }

8 }

9

10 voidschedule(){

11 std::threadthreads[THREAD_COUNT];

12 for(inti=0;i<THREAD_COUNT;++i){

13 threads[i]=std::thread(work,i);

14 }

15 for(inti=0;i<THREAD_COUNT;++i){

16 std::threads[i].join();

17 }

18 }

Listing4.5:OurNaiveLevel0(L0)SolutiontoProblemB

Level1

Oursecondoptimizationlevelistoavoidfalsesharingbydividingeacharray(srcanddst)intoTHREAD_COUNTslicesandassigningasinglethreadtoprocesseachIteminthatslice.WhileweacknowledgethataligningtheItemstructdefinitioninListing

3.2

to64bytes(thecachelinesize)couldbeanotherwayofavoidingfalsesharing,wechosenottogiveparticipantstheflexibilityofmodifyingthestructdefinitionandthuspotentiallyviolatingthetimelimitconstraintfortheproblem.(SeeListing

4.6

)

1

voidwork(intstart,intend){

2

for(inti=start;i<end;++i){

3

for(intj=0;j<INIT_SRC_VAL;++j){

4

--src[i];

5

++dst[i];

6

}

7

}

8

}

9

10

voidschedule(){

11

intslice=SIZE/THREAD_COUNT;

12

std::threadthreads[THREAD_COUNT];

13

for(inti=0;i<THREAD_COUNT;++i){

14

threads[i]=std::thread(work,i*slice,(i+1)*slice);

15

}

16

17

for(inti=0;i<THREAD_COUNT;++i){

18

threads[i].join();

19

}

20

}

Listing4.6:OurLevel1(L1)SolutiontoProblemB

Chapter5Participants

ParticipantRecruitment

Participantswererecruitedmainlyviathemailinglistforcomputersciencegraduatestu-dentsandsnowballedtootherinterestedparticipants.Wefocusedonsystemsdevelopers.Weconsiderparticipantsassystemsdevelopersiftheyhadtakenasystemscourseinclud-ingbutnotlimitedtoOperatingSystems,DistributedSystems,orComputerNetworking.Wealsoconsideredindividualsassystemdevelopersiftheywereinvolvedinsystemsde-velopmentprofessionally,withopen-sourcecontributionsincluded.

Tobeeligibleforthestudy,potentialparticipantsneededaccesstoaninternetbrowserandGitHubCopilotonVSCodeatthetime.Theyalsomustbeasystemdeveloperasdescribedabove,musthavehadatleastafewmonthsofprogrammingexperience,andmusthavehadsomefamiliaritywiththeC++programminglanguage.Additionally,tobeeligible,participantscouldnotbeemployedbyOpenAIorGitHuborinvolvedwiththedevelopmentofGitHubCopilotatthetime.

Tocheckifpotentialparticipantswereeligibletoparticipate,theyweresentaQualtricsscreeningsurveyaftertheyhadreadandsignedtheconsentformdeclaringtheirintenttoparticipate.DetailsofthescreeningsurveycanbefoundinAppendix

B

.

DifficultiesRecruitingProfessionals

Atthehalfwaypointofourdesiredparticipantgoal,wepausedparticipantrecruitmenttoanalyzethepreliminarydatawehadobtained.Onlookingatthesnapshotofparticipants’

solutionstoproblemA,wenoticedthatnotasingleparticipanthadimplementedanyofthethreelevelsofoptimizationswehadconsideredwhendesigningtheproblem.Atthetime,mostoftheparticipantshadbeengraduatestudentswithsoundsystemsbackgrounds,i.e.,theywerepartofaresearchgroupthatfocusedonsystems.However,wedecidedtodiversifyourparticipantpoolbyincludingprofessionalsystemsdevelopers.Theinitialprocessofattemptingtorecruitprofessionalsystemsdevelopersstartedwithcontactingalumsoftheaffiliateduniversitywhowereknowntobeworkingassystemsdevelopers.Additionally,welookedforcontributorstosystemsprojectsonGitHubthatwereprimar-ilyimplementedinC++.Theadvancedsearchfeaturewasusedtofindprojectsthatcontainedthekeywords”systems”,”operatingsystems,”or”database”.Wefine-grainedoursearchtoprojectswithadedicatedsocialplatformwhereinterestedpartiesconnect,i.e.,Discord[

4

]andInternetRelayChat(IRC)[

8

].ProjectssuchasSerenityOS[

14

]andSkiftOS[

15

]hadactiveDiscordcommunities;however,therewasapaucityofinterestedpotentialparticipantsinthestudy.

AttemptstogarnerinterestinthestudyfromsaidprojectcontributorsweremetwitheitherbacklashorsuggestionstoreachouttootherDiscordcommunitiessuchastheosdev(OperatingSystemsDevelopment)[

12

]discordandtheassociatedIRC.UponinteractingwiththeosdevcommunityontheDiscordandIRCplatforms,therewasageneralunwill-ingnesstoparticipateinthestudy,withcommunitymemberscitingpotentialcopyrightissueswithCopilotandothernegativeperceptionsofGitHubCopilot,GitHub,andMi-crosoft.Thankfully,recruitmenteffortspaidoffasafew(lessthanwewouldhaveliked)professionalswerewillingtoparticipateinourstudyandthusmetourdesiredparticipantgoal.

ParticipantSummary

Werecruitedatotalof32participantsforthestudy,where8wereprofessionalsinsystemsprogrammingorcontributorstoopen-sourcesystemsprojects.23weregraduatestudentswithasystemsresearchareaatthetimeofparticipatingandonewasasessionallecturerbutwaspreviouslyagraduatestudentwithasystemsresearchfocus.Thedistributionoftheparticipants’experienceisinFigure

5.1

andtheirfamiliaritywithC++isinFigure

5.2

.Participantswerecompensated$50fortheirtimeandthestudywasapprovedbyResearchEthicsBoard(REB#44162)attheaffiliateduniversity.

13

11

9

count

7

5

3

1

3.0 3.5 4.0 4.5 5.0

devExp

Figure5.1:DistributionofParticipants’DeveloperExperiencefromScreeningSurveyinAppendix

B

17

15

13

11

count

9

7

5

3

1

2 3 4 5

familiarCPP

Figure5.2:DistributionofParticipants’FamiliaritywithC++fromScreeningSurveyinAppendix

B

Chapter6ExperimentDesign

OrderofSolvingtheProblems

Givenourwithin-subjectsexperimentaldesignwhereoneparticipantsolvesoneproblemwithCopilotandthentheotherproblemwithoutit,weneededtoensurethatanyordereffectsarecounterbalancedacrossall32participants.Tothisend,wepresentafactorialmatrixwheretheprompts(AandB)arethecolumnlabels,andthemodes(CandNC)aretherowlabelswhichindicateusingCopilotandnotusingCopilotrespectively.(SeeTable

6.1

).Wefurtherexpandthismatrixtotheproductofpromptsandmodes.Fourpossibleordersofproblemxmodearegenerated(SeeTable

6.2

).

TheordersinTable

6.2

enforcedarequirementthatourparticipantpoolbeamultipleoffour.Hence,werecruited32participantsforthestudy.

axes

A

B

C

NC

CxA

NCxA

CxB

NCxB

Table6.1:FactorialMatrixofmodexproblem

SessionIntroductionandTutorial

Thesessionwasdoneremotelyonanonlineconferencingplatform.Itstartedwiththefacilitatorintroducingthestudy,confirmingtheparticipant’sconsenttoparticipate,and

#

first

second

ParticipantID

1

CxA

NCxB

P1

2

NCxB

CxA

P2

3

NCxA

CxB

P3

4

CxB

NCxA

P4

Table6.2:PossibleOrdersofmodexproblem

thenconfirmingtheparticipant’snumber(theparticipantIDgiventotheparticipantonceeligibilityandconsentweregivenbeforethesession).Thefacilitatorthencontinuedbyexplainingtheoverviewofparticipantresponsibility.Theythenrequestedtheparticipant’sconsenttorecordtheaudioandthescreenduringthesession.Finally,theexperimentergaveatutorialonwhatwasexpected,fromopeningtheprobleminVSCodetousingCopilottoaccept,reject,andviewallsuggestionsandzippingtheeditedcodefiles.(SeeAppendix

C

formoredetailsonthetutorial)

Tasks

ParticipantsweregiventwoC++programmingproblemstosolvewithin30minuteseach.Eachpromptwasself-containedwithinaCPPfilewithinacompressedfolder,