Gartner：管理人工智能信任、風險和安全（英文版）

Don’tLetYourAIControl

You:ManageAITrust,RiskandSecurity

MarkHorvath

?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.Thispublicationmaynotbereproducedordistributedinanyform

withoutGartner'spriorwrittenpermission.ItconsistsoftheopinionsofGartner'sresearchorganization,whichshouldnotbeconstruedasstatementsoffact.Whiletheinformationcontainedinthispublicationhasbeenobtainedfromsourcesbelievedtobereliable,Gartnerdisclaimsallwarrantiesastotheaccuracy,completenessoradequacyofsuchinformation.AlthoughGartnerresearchmayaddresslegalandfinancialissues,Gartnerdoesnotprovidelegalorinvestmentadviceanditsresearchshouldnotbeconstruedorusedassuch.YouraccessanduseofthispublicationaregovernedbyGartner’sUsagePolicy.Gartnerpridesitselfonitsreputationforindependenceandobjectivity.Itsresearchisproducedindependentlybyitsresearchorganizationwithoutinputor

influencefromanythirdparty.Forfurtherinformation,see"GuidingPrinciplesonIndependenceandObjectivity."

AI‘Misperformance’

CanThreatenHumanLife

HowAbout:

?Examproctoring

?Uncorroboratedinformation

?Failingself-drivingcars

?Deepfake-infusedfraud

?Employmentopportunity

?Imprisonment?

?Worse?

Source:FacialRecognitionLeadstoWeek-LongWrongfulImprisonment,TechSpot

2?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

KeyIssues

2

WhatYouNeed

toDoAboutNewAIRisks?

1

Where,WhenandHowCanAIBeCompromised?

3?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

KeyIssues

2

WhatYouNeed

toDoAboutNewAIRisks?

1

Where,WhenandHowCanAIBeCompromised?

4?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

PlentyofModelsto

CompromiseandAttack

NumberofAIModelsDeployedtoDate

73%

ofOrganizationsHave

HundredsorThousandsofModelsDeployed

n=324;Base:UsingAI(S08),excludesunsure

Q13A.HowmanyAImodelshasyourorganizationdeployedtodate?Source:2021GartnerP-21023AIinOrganizationsSurvey

5?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

Dozens

Hundreds

Thousands

HundredsofThousands

AISolutionsAretheTopEmergingTechnology

EmergingTechnologiesDeployedorPlannedtoDeployinNext12Months

48%

Artifical

Intelligence

DistributedSASE

Cloud

n=2,186;CIOsandtechnologyexecutives

Source:2023GartnerCIOandTechnologyExecutiveSurvey

6?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

Edge

Computing

Multiexperience

Development

Platform

MostRespondentsHaveAlreadyDeployedCloud

andAIorPlantoDeployintheNearTerm

StateofDeploymentforEmergingTechnologies

PercentageofRespondents

DistributedcloudArtificialintelligence/machinelearning

Edgecomputing Secureaccessserviceedge(SASE)MultiexperiencedevelopmentplatformDigitaltwin

ResponsibleAI

MLOps 5GBlockchain

nNointerestaWilldeployin2-3yearsaWilldeploybetween12to24monthsaWilldeploywithinnext12monthsaHavealreadydeployed

14%

21%

16%

15%

33%

6%

23%

21%

17%

32%

31%

25%

16%

11%

18%

20%

23%

23%

17%

18%

22%

31%

22%

13%

11%

41%

27%

13%

8%

10%

21%

37%

21%

12%

8%

24%

34%

20%

13%

10%

31%

28%

20%

11%

11%

46%

29%

11%

6%

8%

0%50%100%

n=2,186;CIOsandtechnologyexecutivesanswering

Q.Whatareyourenterprise'splansintermsofthefollowingdigitaltechnologiesandtrends?

Source:2023GartnerCIOandTechnologyExecutiveSurvey

7?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

RegulatoryCompliance

IsTopReasonWhy

Privacy,Securityand/orRiskAreBarriersto

AIImplementation

n=218;Base:AIprivacy,securityandriskmanagement(Q18)

Q19.Whatarethetop3reasonswhyprivacy,securityand/orriskarebarrierstotheimplementationofAItechniqueswithinyourorganization?

Source:2021GartnerP-21023AIinOrganizationsSurvey

8?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

Top3BarrierstothePrivacy,SecurityImplementation

SumofTop3FirstChoice

20%

15%

15%

22%

11%

38%

37%

9%

9%

AIdatabreachesormaliciouscompromises

Datacompromisebyinternalstafforconsultants

Modeltheftormaliciousmanipulation

Regulatorycompliance

WorriesaboutbiasinAImodels

Benignmistakesinmodelprogrammingortrainingdata

Unpredictablemodelperformance

50%

44%

43%

42%

39%

0%30%60%

Compromisesand

AttacksSpanAllStagesofAIOps

Compromises&Attacks:

?Datapoisoningorcompromises

(anystage)andprivacyconcerns.

?Modeloutcomemanipulationordeteriorationatruntime.

?Modelordatamisuse,compromiseortheft.

9?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

AILifeCycle

01

02

03

DevelopDeployRun

ManyAIBreachesAreCausedbyInsiders

ActualTypesofBreaches

Multipleresponsesallowed

Datacompromisebyinternalparty

Datacompromisebyexternalparty

MaliciousattackonourAIinfrastructure(otherthandatacompromise)

60%

56%

27%

0%50%100%

n=131;Base:TeamofAIprivacybreachorsecurity

Q26.WhattypesofAIprivacybreachesand/orsecurityincidentswerethose?Source:2021GartnerP-21023AIinOrganizationsSurvey

10?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

GreaterConcernsAboutOutsidersAreMisplaced

PerceptionsofBreaches:MostWorriedAboutOutsiders

Multipleresponsesallowed

WhichpartiesisyourorganizationmostworriedaboutwhenitcomestoAIprivacy,securityand/orrisk?

Competitors,partnersorotherthirdparties

outsideourorganizationthatweshareAImodelsorlearningwith

MalicioushackersandentitieswhoseektoharmourorganizationbyhackingourAI

Insiderssuchasdatascientists,AIdevelopers,ITstaff

50%

49%

39%

0%

50%

100%

n=218;Base:AIprivacy,securityandriskmanagement(Q18)

Q20.WhichpartiesisyourorganizationmostworriedaboutwhenitcomestoAIprivacy,securityand/orrisk?Source:2021GartnerP-21023AIinOrganizationsSurvey

11?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

DeepfakeSocial

Engineering

SeriousNationalSecurityImplications:

DeepfakeofUkrainianPresidentZelenskycalls

oncitizenstosurrendertoRussiaand“l(fā)aydowntheirarms”is

sharedonline.

Source:DeepfakeZelenskyySurrenderVideoIsthe"FirstIntentionallyUsed"inUkraineWar,Euronews.

12?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

In2024,

Predicts:

DeepfakesforSocial

Engineering

15%ofsuccessfulaccounttakeoverattackswilluse

deepfakestosociallyengineeruserstoturnoversensitivedata

ormovemoneyintocriminalaccounts.

13?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisargisidtrtmakofGatr,r,InIc..aanditisafifliialis..

Video

Text

Image

Voice

DigitalMediaTypes

PhysicalObject

Sensors

Camera

DataFeed

PhysicalVector

MaliciousInputstoAIModels;DigitalandPhysical

ManipulatedDigitalImages

CriminaladdsperturbationstodigitalimagetofoolAImodel

ManipulatedPhysicalSigns

Stickerchanges

signfrom

“speedlimit”to“noovertaking;”foolsself-drivingAImodel

Sources:iProov;DARTS:DeceivingAutonomousCarsWithToxicSigns,arXiv

14?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

QueryAttackAgainstAIModel:BlackBox

BlackBoxAttackDeterminesWhichInputstoUseforDesiredOutput

Attacker

?AttackGoal:

TrialInput

A,B,C

A,B,D

A,C,D

DesiredOutput?

No

No

Yes

Finduncommon,perturbatedinputexamplesthatresultinadesired

outcome,e.g.,forfinancialgainortoavoiddetection.

?AttackMethod:

Model

Repeatmodelqueriesuntilfeasibleinputanddesirableoutputpairshavebeenidentified.

BlackBoxAttack

15?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

QueryAttackAgainstAIModel:WhiteBox

WhiteBoxAttackRegeneratesTrainingDatasettoReproduceaSimilarModel

Attacker

?AttackGoal:

Generated

TrainingDataset

A,B,C

A,B,D

A,C,D

N

N

Y

Complete“reverseengineering”ofthemodel,e.g.,tostealIPorobtainsensitive/valuabletrainingdata.

?AttackMethod:

Model

WhiteBoxAttack

Manymodelqueriesto(re)generateatrainingdataset(input+label/target

rows),whichisthenusedtoreproduceasimilarmodel.

16?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

KeyIssues

2

WhatYouNeed

toDoAboutNewAIRisks?

1

Where,WhenandHowCanAIBeCompromised?

17?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

AITrustRisk&SecurityManagement(TRiSM)

UnmanagedRisks

AITRiSM

ManagedRisks

Explainability/

ModelMonitoring

AIApplicationSecurity

Privacy

ModelOps

TheAITRiSMMarketIncludesSolutionsforTheseFunctions

SupportsAI

?Governance

?Trustworthiness

?Fairness

?Reliability

?Privacy

?Security

?Compliance

18?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

FirstGetOrganized:AITRiSMIsaTeamSport

EnterpriseAIArchitects

InformationTechnology

Risk

Management

Privacy

Data

Analytics

LOB&

Operations

Compliance

Security

Legal

Ethics

19?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

CISOsNeedtoSpeakWithTheirAITeams

ThinkAIRiskIsConcerned

LikelytoMaterializeAboutAIRisk

CISO26%36%

AITeam53%51%

n=64;Base:CISOsinenterprisesatleastpilotingAIsolutionsn=49;Base:AIworkers

Q:WhatisthelikelihoodthatinformationriskstemmingfromAIsolutionswillmaterializeinthenext12-18monthsinyourenterprise?

Q:HowconcernedareyouaboutinformationriskinyourenterpriseassociatedwithAIsolutions?

Source:2021GartnerStateofAICyberRiskManagementStudy

20?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

OnceOrganized:SetYourPriorities

SuggestedTop5PrioritiesforAITRiSM

2

3

PrivacyandDataProtection

4

5

AISecurity

andResilience

AIRisk

Awareness

Robust

ModelOps

1

AIInventory:Explainability&Interpretability

21?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

FourReasonsWhyYouNeedAITRiSM

?Typicallyrequiresabest-of-breedtoolportfolioapproach

?Modelsanddatacandriftformanydifferentreasons,whichcancauseadverseconsequences

?AIposesconsiderabledatarisksassensitivedatasetsareoftenusedtotrainAImodels

?RegulatorsareissuinglawstoregulateAIindepth

22?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

ContinuousUseofAITRiSMSolutions

SafeguardsAIDelivery

UnmanagedRisks

AITRiSM

ManagedRisks

Explainability/

ModelMonitoring

Privacy

ModelOps

AIApplicationSecurity

23?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

By2026,

organizationsthatoperationalize

Predicts:

AITRiSM

ImprovesAIResults

AItransparency,trustand

securitywillseetheirAImodelsachievea50%result

improvementinterms

ofadoption,businessgoalsanduseracceptance.

Source:Gartner

registeredtrademarkofGartner,Inc.anditsaffiliates.

24?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

ExampleCaseStudy:TheDanishCancerSociety

?ThesocietyusesanAIproductthat

enablesitsresearcherstomoreeasilydiscoverinsightsintodata.

?Theproductusesmathematically

explainablemodelstoidentifygene

combinationsthatleadtohigher

incidenceofdeathfrombreastcancer.

?Thesediscoveriesareenabling

thesocietytodevelopmorepreciseandeffectivedrugs.

Explainability

25?2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

ExampleCaseStudy:UnityHealthToronto

?UnityHealthTorontousedAITRiSM

conceptsandtoolsinafour-stepprocesstomakeitsmodels

moretrustworthy.

?ThisimprovedtheresultsitachievedfromAI.

?UnityHealthTorontodrastically

reducedmortalityinitshigh-risk

patients,duringtheCOVID-19

pandemic,byfollowingahuman-

centric,credibility-focusedapproach.

TrustedAI

26?2023Gartner,Inc.and/oritsaf