基于深度學習的人工智能模型來輔助 甲狀腺結(jié)節(jié)的診斷和治療：多中心 診斷研究

/digital-health

Vol3April2021

e

PAGE

250

e

PAGE

259

/digital-health

Vol3April2021

Articles

Deeplearning-basedartificialintelligencemodeltoassistthyroidnodulediagnosisandmanagement:amulticentrediagnosticstudy

SuiPeng*,YihaoLiu*,WeimingLv*,LongzhongLiu*,QianZhou*,HongYang,JieRen,GuangjianLiu,XiaodongWang,XuehuaZhang,QiangDu,FangxingNie,GaoHuang,YuchenGuo,JieLi,JinyuLiang,HangtongHu,HanXiao,ZelongLiu,FenghuaLai,QiuyiZheng,HaiboWang,YanbingLi,ErikKAlexander,WeiWang,HaipengXiao

Summary

BackgroundStrategiesforintegratingartificialintelligence(AI)intothyroidnodulemanagementrequireadditionaldevelopmentandtesting.Wedevelopedadeep-learningAImodel(ThyNet)todifferentiatebetweenmalignanttumoursandbenignthyroidnodulesandaimedtoinvestigatehowThyNetcouldhelpradiologistsimprovediagnosticperformanceandavoidunnecessaryfineneedleaspiration.

MethodsThyNetwasdevelopedandtrainedon18049imagesof8339patients(trainingset)fromtwohospitals(theFirstAffiliatedHospitalofSunYat-senUniversity,Guangzhou,China,andSunYat-senUniversityCancerCenter,Guangzhou,China)andtestedon4305imagesof2775patients(totaltestset)fromsevenhospitals(theFirstAffiliatedHospitalofGuangzhouUniversityofChineseMedicine,Guangzhou,China;theSixthAffiliatedHospitalofSunYat-senUniversity,Guangzhou,China;theGuangzhouArmyGeneralHospital,Guangzhou,China;theThirdAffiliatedHospitalofSunYat-senUniversity,Guangzhou,China;theFirstAffiliatedHospitalofSunYat-senUniversity;SunYat-senUniversityCancerCenter;andtheFirstAffiliatedHospitalofGuangxiMedicalUniversity,Nanning,China)inthreestages.Allnodulesinthetrainingandtotaltestsetwerepathologicallyconfirmed.ThediagnosticperformanceofThyNetwasfirstcomparedwith12radiologists(testsetA);aThyNet-assistedstrategy,inwhichThyNetassisteddiagnosesmadebyradiologists,wasdevelopedtoimprovediagnosticperformanceofradiologistsusingimages(testsetB);theThyNetassistedstrategywasthentestedinareal-worldclinicalsetting(usingimagesandvideos;testsetC).Inasimulatedscenario,thenumberofunnecessaryfineneedleaspirationsavoidedbyThyNet-assistedstrategywascalculated.

FindingsTheareaunderthereceiveroperatingcharacteristiccurve(AUROC)foraccuratediagnosisofThyNet(0·922[95%CI0·910–0·934])wassignificantlyhigherthanthatoftheradiologists(0·839[0·834–0·844];p<0·0001).Furthermore,ThyNet-assistedstrategyimprovedthepooledAUROCoftheradiologistsfrom0·837(0·832–0·842)whendiagnosingwithoutThyNetto0·875(0·871–0·880;p<0·0001)withThyNetforreviewingimages,andfrom0·862(0·851–0·872)to0·873(0·863–0·883;p<0·0001)intheclinicaltest,whichusedimagesandvideos.Inthesimulatedscenario,thenumberoffineneedleaspirationsdecreasedfrom61·9%to35·2%usingtheThyNet-assistedstrategy,whilemissedmalignancydecreasedfrom18·9%to17·0%.

InterpretationTheThyNet-assistedstrategycansignificantlyimprovethediagnosticperformanceofradiologistsandhelpreduceunnecessaryfineneedleaspirationsforthyroidnodules.

FundingNationalNaturalScienceFoundationofChinaandGuangzhouScienceandTechnologyProject.

Copyright?2021TheAuthor(s).PublishedbyElsevierLtd.ThisisanOpenAccessarticleundertheCCBY-NC-ND

4.0license.

LancetDigitHealth2021;3:e250–59

*Contributedequallytothiswork

ClinicalTrialsUnit

(ProfSPengPhD,YLiuMD,

QZhouMS,ProfHWangMPH),DepartmentofEndocrinology(YLiu,FLaiMM,QZhengMD,ProfYLiPhD,ProfHXiaoPhD),DepartmentofMedicalUltrasonics,InstituteofDiagnosticandInterventionalUltrasound(YLiu,JLiangPhD,HHuMD,HanXiaoMD,

ZLiuMD,ProfWWang),andDepartmentofBreastandThyroidSurgery

(ProfWLvPhD,JLiPhD),

TheFirstAffiliatedHospitalofSunYat-senUniversity,Guangzhou,China;DepartmentofUltrasound,SunYat-senUniversityCancerCenter,StateKeyLaboratoryofOncologyinSouthChina,Guangzhou,China(LLiuPhD);DepartmentofMedicalUltrasound,theFirstAffiliatedHospitalofGuangxiMedicalUniversity,Nanning,China(ProfHYangPhD);DepartmentofMedicalUltrasonics,

theThirdAffiliatedHospitalofSunYat-senUniversity,

Guangzhou,China

(ProfJRenPhD);DepartmentofMedicalUltrasonics,theSixthAffiliatedHospitalofSunYat-senUniversity,Guangzhou,China(GLiuPhD);DepartmentofMedicalUltrasonics,theFirstAffiliated

Introduction

Thyroidnodulesarefoundinupto68%ofasymp-tomaticadultsinthegeneralpopulation.1Approximately7–15%ofthyroidnodulesarethyroidcancer,whichisthemostrapidlyincreasingmalignancyinallpopulations.2Thelargenumberofthyroidnodules,withonlyafractionbeingcancerous,callsforareliablemethodtoaccuratelydifferentiatemalignantfrombenignnodules.

Routinedecisionmakingforpatientswiththyroidnodulesdependsonultrasoundorinvasivefineneedleaspiration.2However,theassessmentofultrasound

featuresistimeconsuming,subjective,andoftendependentonaradiologist’sexperienceandtheavailableultrasounddevices.3Ultrasoundconclusionsareofteninconsistentandevenwithfineneedleaspirations15–30%ofthesamplesstillyieldindeterminatecytologicalfindings.4Additionalrobustmethodsareneededtoimprovediagnosisandfineneedleaspirationstrategiestoadapttotheexponentialgrowthofpatientneedsandburdenonmedicalservices.

Artificialintelligence(AI)hasbeenreportedtomeetorexceedhumanexpertsinmedicalimaging.5–8Afew

HospitalofGuangzhouUniversityofChineseMedicine,Guangzhou,China(XWangMD);DepartmentofUltrasound,theGuangzhouArmyGeneralHospital,

Guangzhou,China

(XZhangMD);Xiaobaishiji,Beijing,China(QDuME,

FNieME,GHuangDE);InstituteforBrainandCognitiveSciences,TsinghuaUniversity,Beijing,China

(YGuoME);ThyroidSection,

Articles

Articles

Researchincontext

Evidencebeforethisstudy TheThyNet-assistedstrategynotonlyimprovedthe

WesearchedPubMedfromtheinceptionofthedatabaseto performanceofradiologistswhenreviewingimagesonly,

Sept20,2020,forresearcharticleswiththesearchterms“deep butalsowhenreviewingimagesandvideosinaclinicalsetting.learning”O(jiān)R“machinelearning”O(jiān)R“artificialintelligence”O(jiān)R OfnotethecombinationoftheAmericanCollegeof“convolutionalneuralnetwork”AND“thyroidcancer”O(jiān)R“thyroid RheumatologyThyroidImagingReportingandDataSystemnodule”O(jiān)R“thyroidcarcinoma”,withoutlanguagerestrictions. classificationwithAIassistanceimprovedthenegative

Weidentified15studiesonthedevelopmentandvalidationofpredictivevalueandpositivepredictivevalueofthyroidnoduleartificialintelligence(AI)modelsinthyroidnodulemanagement.differentiation,whichreducedthenumberofunnecessaryfineHowever,thesestudiescomparedtheperformanceofradiologistsneedleaspiration.

withthatoftheAImodel.Wefoundnopublicationsthat Implicationsofalltheavailableevidence

specificallyreportedhowdiagnosticdeep-learningormachine-

learningalgorithmscouldassistradiologistsperformancein ThyNet-assistedstrategycouldsignificantlyimprovethethyroidnodulemanagement.Theabsenceofmulticentretraining diagnosticperformanceofradiologistsandhelpreducethecohortsandasmallnumberofultrasounddevicesinprevious numberofunnecessaryfineneedleaspirationsforthyroidstudiesrestrictedtheirgeneralisabilityinclinicalpractice. nodules.Onthebasisofourfindings,AIdiagnosticprogrammes

shouldberolledouttoclinicalpracticeofthyroidnodule

Addedvalueofthisstudy management.Toourknowledge,thisstudyisthefirsttodevelopan

AI-assistedstrategyforthyroidnodulemanagement.

Brigham&Women’sHospital,HarvardMedicalSchool,

Boston,MA,USA

(ProfEKAlexanderMD)

Correspondenceto:ProfHaipengXiao,DepartmentofEndocrinology,TheFirstAffiliatedHospitalofSunYat-senUniversity,Guangzhou510080,

China

xiaohp@

or

ProfWeiWang,DepartmentofMedicalUltrasonics,InstituteofDiagnosticandInterventionalUltrasound,TheFirstAffiliatedHospitalofSunYat-senUniversity,Guangzhou510080,

China

wangw73@

or

ProfErikKAlexander,ThyroidSection,Brigham&Women’sHospital,HarvardMedicalSchool,Boston,MA02115,USA

ekalexander@

studieshavefocusedonacomparisonofthediagnosticperformanceofAIwithcliniciansinthyroidnoduledifferentiation.9–11Inourpreliminarystudy,amachinelearningsystemshowedabetterpredictivevalueformalignantthyroidnodulescomparedwithhumansusingAmericanCollegeofRheumatology(ACR)ThyroidImagingReportingandDataSystem(TI-RADS).7Theintroductionofdeeplearninginthyroidimaginghasalsoachievedabetterdiagnosticperformancethanexperiencedradiologists.12,13Previousstudiesapplyingdeep-learningalgorithmshavemainlyfocusedonthecomparisonofradiologistsanddeep-learningmodelsbyreadingultrasoundimages.However,inareal-worldsetting,thefinaldiagnosisshouldstillbemadebyradiologists.Therefore,evaluatingthediagnosticimprovementsprovidedbythecooperationbetweenradiologistsandAIsystemsismoresimilartotheclinicalsetting.Radiologistscouldimproveperformancebyreadingdynamicvideosinsteadofstaticimagesonly,butwhetheranAI-assistedmodelcanhelpradiologistsimprovediagnosticperformancebyprocessingbothimagesandvideosshouldbeinves-tigated.Moreover,fewstudiesdiscussedtheinfluenceofAIonfineneedleaspirationorthyroidectomytreatmentadvicegivenbyhealth-careprofessionals,leavingthisissuestillvague.

Wedevelopedadeep-learningAImodel(ThyNet)todifferentiatemalignanttumoursfrombenignthyroidnodules.WeinvestigatedwhetherradiologistscouldimprovetheirdiagnosticperformancewiththeassistanceoftheThyNetmodelwhenreadingultrasoundimagesandvideosandexploredthepotentialoftheThyNet-assistedstrategytohelpradiologistsavoidunnecessaryfineneedleaspirations.

Methods

Studydesignanddatasets

Thiswasamulticentre,diagnosticstudythatusedultrasoundimagesetsfromsevenhospitalsinChina.Patientsaged18yearsoldorolderwiththyroidnodulesatleast3mmindiameteridentifiedwithultrasoundwhohadadefinitivebenignormalignantpathologicalresult(surgicalspecimenorfineneedleaspiration[BethesdacategoryIIorVI])wereeligibleforinclusioninthetrainingsetandtestingsets.Thepathologicaldiagnosesweremadebytwopathologists,oneofwhomhadmorethan8years’experience.Allimageswereintiallyincluded,butlow-qualityultrasoundimages,suchassevereartifacts(eg,motionartifactsandspeedpropagationandrefractionartifacts)orlowimageresolution,wereexcludedafterscreening.

TheimagesofthetrainingsetwerecollectedfromtheFirstAffiliatedHospitalofSunYat-senUniversity,Guangzhou,ChinaandSunYat-senUniversityCancerCenter,Guangzhou,China(18049imagesof8339patients).FortestsetA,2185imagesof1424patientswiththyroidnoduleswereenrolledfromfourindependenthospitals(theFirstAffiliatedHospitalofGuangxiMedicalUniversity,Nanning,China;theFirstAffiliatedHospitalofGuangzhouUniversityofChineseMedicine,Guangzhou,China;theSixthAffiliatedHospitalofSunYat-senUniversity,Guangzhou,China;andtheGuangzhouArmyGeneralHospital,Guangzhou,China).FortestsetB,1754imagesof1048patientswiththyroidnoduleswereenrolledfromtheFirstAffiliatedHospitalofSunYat-senUniversity,andtheThirdAffiliatedHospitalofSunYat-senUniversity,Guangzhou,China.FortestsetC,366imagesof303patientswiththyroidnoduleswereenrolledfromtheFirstAffiliatedHospital

/digital-health

Vol3April2021

e

PAGE

252

e

PAGE

253

/digital-health

Vol3April2021

ofSunYat-senUniversity,SunYat-senUniversityCancerCenter,andtheFirstAffiliatedHospitalofGuangxiMedicalUniversity.

ThisstudywasapprovedbytheResearchEthicsCommitteeoftheFirstAffiliatedHospitalofSunYat-senUniversity.Informedconsentwaswaivedforretrospectivelycollectedultrasoundimages,whichwereannonymised.Writteninformedconsentwasobtainedfrompatientswhoseultrasoundimagesanddynamicvideoswereprospectivelycollected.

Outcomes

Theprimaryendpointofourstudywastheareaunderthereceiveroperatingcharacteristiccurve(AUROC)ofthyroidnodulediagnosis.Thesecondaryendpointsofourstudywereaccuracy,sensitivity,specificity,positivepredictivevalue,andnegativepredictivevalueofthyroidnodulediagnosis.Thepost-hocanalysisincludedthediagnosticaccuracyofThyNetindifferentpathologicalsubtypesandThyNet-assistedfineneedleaspirationstrategy.

Procedures

Forthetrainingset,ultrasoundimagesofconsecutivepatientswiththyroidnoduleswereretrospectivelyretrievedfromtheindividualthyroidimagingdatabaseattheFirstAffiliatedHospitalofSunYat-senUniversityandSunYat-senUniversityCancerCenter,betweenJan1,2009,andNov30,2018.Atotalof19312imagesfrom8339patientswereincludedinthetrainingset,with1263imagesexcludedduetopoorimagequality.

Therewasnooverlapbetweenpatientsinthetrainingandtestsetsandtherewasnooverlapbetweenthethreetestsubset.ThetestsetimagesforthecomparisonbetweenThyNetandradiologists(testsetA)andtheassessmentoftheThyNet-assistedstrategy(testsetB)wereretrospectivelyobtainedfromsixindependenthospitals(theFirstAffiliatedHospitalofGuangxiMedicalUniversity,theFirstAffiliatedHospitalofGuangzhouUniversityofChineseMedicine,theSixthAffiliatedHospitalofSunYat-senUniversity,theGuangzhouArmyGeneralHospital,FirstAffiliatedHospitalofSunYat-senUniversity,andtheThirdAffiliatedHospitalofSunYat-senUniversity)betweenJan1,2009,andJuly30,2019.Intheclinicalsettingtest(testsetC),bothimagesanddynamicvideosofnoduleswereprospectivelycollectedfrominpatientsattheFirstAffiliatedHospitalofSunYat-senUniversity,SunYat-senUniversityCancerCenter,andFirstAffiliatedHospitalofGuangxiMedicalUniversityfromOct1toNov30,2019(appendixp4).Atotalof6587patientsinthetrainingsetand1956patientsinthetestsetswereconfirmedashavingadefinitivebenignormalignantpathologicalresultbasedonasurgicalspecimen.1752patientsinthetrainingsetand819patientsinthetestsetswereconfirmedashavingadefinitivebenignormalignantpathologicalresultbasedonfineneedleaspiration(BethesdacategoryIIorVI).

AllthyroidultrasoundimagesextractedfromthethyroidimagingdatabasewereconvertedintoaJPEGformat.Variousmodelsofultrasoundequipmentproducedby13differentmanufacturers(GEHealthcare,Chicago,IL,USA;Philips,Amsterdam,theNetherlands;Siemens,Munich,Germany;Canon,Tokyo,Japan;Samsung,Seoul,SouthKorea;Esaote,Genoa,Italy;Mindray,Huntingdon,UK;SonoScape,Shenzhen,China;Aloka,Wallingford,CT,USA;BKMedical,Peabody,MA,USA;Supersonic,Aix-en-Provence,France;Vinno,Suzhou,China;andHitachi,Tokyo,Japan)wereusedtogeneratetheultrasoundimages(appendixp8).Imagequalitycontrolwasdoneforthetrainingsetandtestsets.Forthequalitycontrolofultrasoundimages,allthyroidimageswerescreenedandlow-qualityimagescontainingsevereartifactsorsignificantimageresolutionreductionswereremoved.Thescreeningfortheimageswasdonebytworadiologists(HanXandZL)whohadatleast1yearofultrasoundexperience.Iftherewasnoconsensusregardingnodulelocationbetweentheimageandthepathologicalreport,theimagewasremoved.2345imagesfrom1424patientsintestsetAforthecomparisonbetweenThyNetandradiologistsmetthecriteria,with160imagesexcluded.1896imagesfrom1048patientsmettheinclusioncriteriaandwereusedintheassessmentoftheThyNet-assisteddiagnosticstrategy,with142imagesexcludedafterimagequalitycontrol(testsetB).401imagesfrom303patientsintestsetCmettheinclusioncriteriaandwereusedintheassessmentoftheThyNet-assisteddiagnosticstrategyinareal-worldsetting,with35imagesexcludedafterimagequalitycontrol.Alldataweredeidentified(includingretro-spectivleycollecteddataforthetrainingsets)beforethedevelopmentandevaluationofthemodel.

TheThyNetdeep-learningalgorithmwasspecificallydesignedtodiagnosemalignancyfromthyroidultrasoundimages.Itisacombinedarchitectureofthreenetworks:ResNet,ResNeXt,andDenseNet(appendixp5).ResNetusesresiduallearningblockstoreducetheeffectofgradientvanishing.ResNeXtisamodifiedversionofResNet,developedbyrepeatingabuildingblockthataggregatesasetoftransformationswiththesametopology.ResNeXtadditionallyintroducedtheconceptofsparsityandgroupconvolutiontoenhancetheabilityoftheAItolearnthesemanticinformationwithlessparameters.DenseNetisanewnetworkarchitecturethatconnectseachlayertoeveryotherlayerinafeed-forwardfashion.14DenseNetmakesthenetworkdeeperbutreducesthenumberofparametersandpreventsoverfitting.Thethreebranchesofnetworksweretrainedseparatelyonthesametrainingsetandassembledthroughamajorityvotealgorithm.Tosearchfortheoptimalweightsforeachnetworkbranchandgettheensembledoutput,weusedthebrute-forcesearchmethodviacross-testinthetrainingsets.Thefinalweightingratiosare0·40forResNet,0·35forResNeXt,and0·25forDenseNet.

SeeOnlineforappendix

Trainingset

Testingsets

RadiologistsvsThyNet RadiologistsassistedbyThyNet

Prospectivecohortinclinicalpractice

1st

2nd

1st

2nd Final

vs

DeeplearningbasedThyNet

18049images

5122malignantand3217benignpathologicallyprovennodules

12radiologistsread

2185images

12radiologistsread1754imageswithThyNetassistance

12radiologistsread366imagesandvideoswithThyNetassistance

Figure1:Studyprofile

Usingdatasetsfromtwocentres,ThyNetwastrainedtodifferentiatethyroidnodules.ThyNetwasthentestedonthreedatasetswithnooverlap(testsetsA–C).

First,diagnosticperformancebetweenradiologistsandThyNetbasedonstaticimageswascompared.Second,diagnosticperformanceofradiologistsbefore

(firstdiagnosis)andafter(seconddiagnosis)theassistancebyThyNetwasassessedbasedonstaticimages.Third,thefirstdiagnosisbasedonstaticimagesandtheseconddiagnosisbasedondynamicvideoswasrecorded.Then,withtheassistanceofThyNet,thefinaldiagnosiswasobtainedandcomparedwiththeindependentdiagnosesmadebyradiologistswithoutThyNet.

Formoreontheratingplatform

see

Thenoiseinformation(eg,paramatersoftheultrasounddevice),whichwasdistributedmainlyintheperipheralareasoftheoriginalimages,wasmanuallyremovedbyoneradiologist(HH).Theimageswereresizedto256×256pixelsbeforebeingcroppedto224×224pixels.Standardimagepreprocessing(clipping,flipping,androtating)fordeeplearningtogeneratealarger,morecomplicatedanddiversedatasettoimproveaccuracyandgeneralisabilitywasthendone.Augmentationwasdoneindependentlybeforeeachepochwitharandomlyselectedalgorithmofthethreeaugmentationalgorithms.Ourmodeltooktheaugmentedimages(byoneaugmentationalgorithmforeachepoch;input)andcalculatedtheprobabilityofeachimagebeingamalignantdiagnosis(output)aftertrainingacertainnumberofepochs(appendixp6).

Weusedtheweightsofeachnetwork,pretrainedonImageNet,astheinitialisationofourmodel’sweights.Thesametrainingparameterswereappliedtoeachnetworkbranch.Stochasticgradientdescentandcross-entropylosswereusedfornetworkweighttuningandalgorithmoptimisation.Theinitiallearningratewas0·01,whichdecreasedbyone-tenthevery100epochs;thefinallearningratewas0·0001.Topreventoverfitting,batchnormalisationwasusedandtheweightdecayratewassetto0·0005.Weusedabatchsizeof128imagesandaRectifiedLinearUnitactivationfunction.Heatmapsweregeneratedbythegradcammethods.

12radiologists,includingsixjuniorradiologists(1–3yearsofexperience)andsixseniorradiologists(>8yearsofexperience),reviewedthetworetrospectivedatasetsandtheprospectivedataset.Radiologistsweremaskedtothepathologicalconfirmationofthenodulestatusandresearchaimsbeforethereviewingprocess.Theindependentreviewprocesswasmadeonaweb-basedrating

platform

.ThereviewofeachlesionincludedassigningpointsbasedontheACRTI-RADS15categories(composition,echogenicity,shape,margin,andechogenicfoci)anddeterminingamalignantorbenigndiagnosis(appendixpp17–24).

ThyNetwastestedinthreestages(figure1).First,thediagnosticperformanceofThyNetwascomparedwithradiologists(withtestsetA);second,improvementinthediagnosticperformanceofradiologistswhenassistedbyThyNetwasevaluated(withtestsetB);andthird,theapplicationofThyNetinactualclinicalpracticewasinvestigated(withtestsetC).

Forthefirststage,ultrasoundimagesfromfourindependenthospitalswereusedtocomparetheperformanceofThyNetwithradiologists.Radiologistswereinvitedtoreviewtheimagesandmakediagnosesindependently.Areviewprocesswasmadeonaweb-basedratingplatform,whichintegratedthedataofallvalidationdatasets.ThereviewofeachlesionincludedthefollowingassigningpointsbasedonfiveACRTI-RADS18categories(composition,echogenicity,shape,margin,andechogenicfoci)anddeterminingamalignantorbenigndiagnosis.Alldataweredeidentifiedbeforetransfertotheinvestigators,andtheradiologistswerealsomaskedtothepathologicalreports.Theradio-logistswereinformedoftheirdiagnosticperformancecomparedwithThyNetbeforethedeep-learningsystemwasusedtoaidtheirdiagnosis.

RadiologistsintwohospitalsusedThyNettoaidthediagnosticprocess.Initialindependentreviewanddiagnosisweremadebyradiologistsalone.TheradiologistdiagnosiswascomparedwithareferencediagnosisfromThyNet.Ifthetwodidnotmatch,theradiologistscouldthenchoosetoadheretotheirdiagnosisoradoptthediagnosisfromThyNetasthefinaldiagnosis.Boththeinitialandfinalassisteddiagnosiswererecorded.

ThyNetwastestedinareal-worldclinicalsettinginthreehospitals.Initialindependentreviewanddiagnosisweremadeby12radiologistsreviewingstaticimagesandaseconddiagnosiswasobtainedbasedondynamicvideosofthenodule.The12radiologistswerethesameindividualsthatassessedtheimagesintestsetsAandB.ThefinaldiagnosiswasmadeaftertheThyNet-assistedreferencediagnosis.Thethreeindependentdiagnostic

recordsofinitial,second,andfinaldiagnosisforeachradiologistwererecorded.

Inclinicalpracticeofthyroidnodulemanagement,acrucialdecisionfollowingACRTI-RADSscoringis

whethersubsequentfineneedleaspirationisindicated.AccordingtoACRTI-RADS,nodulesthatscore2pointsorlessdonotneedfineneedleaspiration,inwhichcasetheprobabilityofbeingbenign(negativepredictive

Sensitivity

1·00

Seniorradiologists

0·95withAIassistanceJuniorradiologistwithAIassistance

0·90

Seniorradiologist

0·85

0·80

0·75

Juniorradiologist

Radiologists

withAIassistance

0·95Juniorradiologist

1·00SeniorradiologistwithAIassistance

Seniorradiologist

withAIassistance withdynamicvideos

Seniorradiologist

0·90

withstaticimages

Juniorradiologist

0·85

0·80

0·75

Juniorradiologistwithdynamicvideoswithstaticimages

InitialdiagnosiswithimageSeconddiagnosiswithvideoFinaldiagnosiswithAI

A

B

PooledAUROC=0·922 Individualradiologists

GXMUAUROC=0·922 SeniorradiologistsAUROC=0·857

GUCMAUROC=0·928 JuniorradiologistsAUROC=0·821

SYSU06AUROC=0·924 AllradiologistsAUROC=0·839

GAGHAUROC=0·921

1·0

0·8

0·6

0·4

0·2

0·95

0·90

0·85

0·80

0·75

0·70

0·0

C D

SeniorradiologistsAUROC=0·855 SeniorradiologistswithstaticimagesAUROC=0·837

SeniorradiologistswithAIassistanceAUROC=0·885 SeniorradiologistswithdynamicvideosAUROC=0·871

JuniorradiologistsAUROC=0·819 SeniorradiologistswithradiologistsAIassistanceAUROC=0·881

JuniorwithAIassistanceAUROC=0·866 JuniorradiologistswithstaticimagesAUROC=0·809JuniorradiologistswithdynamicvideosAUROC=0·853JuniorradiologistswithAIassistanceAUROC=0·866

1·0

0·8

0·6

0·4

0·2

0·0

1·0

0·8

0·6

0·4

0·2

01·0

0·8

0·6

0·4

0·2

0

Specificity Specificity

1·00

0·95

0·90

0·85

0·80

0·75

0·70

Sensitivity

Figure2:DiagnosticperformanceofThyNetandradiologistsinserialtestfordiscriminationofmalignantfrombenignthyroidnodules

AUROCstoevaluatediagnosticperformanceofThyNetinthetotaltestsetandeachexternalinstitutioninthefirsttestcomparingThyNetwithradiologists.

DiagnosticperformanceofThyNetcomparedwitheachradiologistinthetotaltestset.Rounddotsindicatediagnosticsensitivitiesandspecificitiesofindividualradiologists,thetriangleindicatesthepooledsensitivitiesandspecificitiesofalljuniorradiologists,thestarindicatesthepooledsensitivitiesandspecificitiesofallseniorradiologists,andthesquareindicatespooledsensitivitiesandspecificitiesofallradiologists.(C)DiagnosticperformanceofradiologistsaloneandradiologistsassistedbyThyNet.Rounddotsindicatesensitivitiesandspecificitiesofthefirstdiagnosis,andthesquaresindicatesensitivitiesandspecificitiesofseconddiagnosiswithThyNetassistance.(D)DiagnosticperformanceofradiologistsassistedbyThyNetinaclinicalsetting.Rounddotsindicatesensitivitiesandspecificitiesofthefirstdiagnosisbasedonstaticimages,trianglesindicatetheseconddiagnosisbasedondynamicvideos,andthesquaresindicatefinaldiagnosisofradiologistwithThyNetassistance.AI=artificialintelligence.AUROC=areaunderthereceiveroperatingcharacteristiccurve.GAGH=theGuangzhouArmyGeneralHospital.GUCM=theFirstAffiliatedHospitalofGuangzhouUniversityofChineseMedicine.GXMU=theFirstAffiliatedHospitalofGuangxiMedicalUniversity.ROC=receiveroperatingcharacteristiccurve.SYSU06=theSixthAffiliatedHospitalofSunYat-senUniversity.

AUROC(95%CI)

pvalue

Accuracy(95%CI)

p