2024微軟人工智能系統(tǒng)強(qiáng)化學(xué)習(xí)

1 signal

observation&reward

RealworldenvironmentAgent historyinfo

simulatorEachtimesteptAgentanaction????Worldupdatesgivenactionat,emitsobservationandAgentreceivesobservationandUseexperiencetoguidefuturedecisions(exploit)signal

observation&reward

RealworldenvironmentAgent historyinfo

simulatorHistory???=(??1,...,????,????,AgentchoosesactionbasedonhistoryisinformationassumedtodeterminewhathappensnextFunctionhistory=(???)Stateisifandonlyif p(????+1|,????)=p(????+1|???,????)Goalselectactionstomaximizetotalexpectedfuturerewardbalancingimmediate&long-termrewardsπdetermineshowtheagentchoosesactionsDeterministicpolicyStochasticpolicyfunctionexpecteddiscountedsumfuturerewardsunderapolicyπ initializeenvPolicymodelinitializeenvPolicymodelinitializepolicyPolicyinferenceinitializepolicyRolloutdataRolloutdataPolicyupdateUpdatepolicyUpdatepolicyHessel,Matteo,etal."Rainbow:Combiningimprovementsindeepreinforcementlearning."——給PPO帶來(lái)真正的性能上提升以及將policy約束在trustregion內(nèi)的效果，都不是通過(guò)PPO論文中提出的對(duì)新的policy和原policy的比值進(jìn)行裁切（clip）帶來(lái)的，而是通過(guò)code-level的一些技巧帶來(lái)的。Engstrom,Logan,etal."Implementationmattersindeeppolicygradients:AcasestudyonPPOandTRPO."Liang,Eric,etal."Rayrllib:Acomposableandscalablereinforcementlearninglibrary."Liang,Eric,etal."Rayrllib:Acomposableandscalablereinforcementlearninglibrary."新算法新算法新架構(gòu) 難以復(fù)用的強(qiáng)化學(xué)習(xí)代碼

可擴(kuò)展性的強(qiáng)化學(xué)習(xí)框架 TrainingDataMLModelTrainingDataMLModelTrainingsignalθ

observation&reward

RealworldenvironmentAgent historyinfo

simulator面臨的問(wèn)題面臨的問(wèn)題新的需求Horgan,Dan,etal."Distributedprioritizedexperiencereplay."可能傳輸大量的數(shù)據(jù)可能傳輸大量的數(shù)據(jù)GPUCPU面臨的問(wèn)題面臨的問(wèn)題可能的解決方案 通用的RL算法針對(duì)Env開發(fā)支持分布式Star數(shù)目RepoACME+Reverb2.1k/deepmind/acmeELF2k/facebookresearch/ELFRay+RLlib16.4k/ray-project/rayGym24.5k/openai/gymBaselines11.6k/openai/baselinesTorchBeast553/facebookresearch/torchbeastSeedRL617/google-research/seed_rlTianshuo？3.2k/thu-ml/tianshouKeras-RL5.1k/keras-rl/keras-rlRayisafastandsimpleframeworkforbuildingandrunningdistributedapplications./ray-project/ray Rayisafastandsimpleframeworkforbuildingandrunningdistributedapplications.AprocessexecutingtheuserprogramAstatelessprocessthatexecutesremotefunctionsinvokedbyadriverAstatefulprocessthatexecutesDistributedobjectIn-memorydistributedstoragetostoretheinputs/outputs,orstatelesscomputation.ImplementtheobjectstoreviasharedmemoryUseApacheArrowasdataformatsDistributedschedulerSubmittedfirsttolocalschedulerGlobalschedulerconsiderseachloadandconstraintstoschedulingdecisionsGlobalControlAkey-valuestorewithpub-subfunctionalityRLlibisanopen-sourcelibraryforreinforcementlearningthatoffersbothhighscalabilityandaunifiedAPIforavarietyofapplications.RayRayRLlib/ray-project/ray/tree/master/rllib distributedschedulerisanaturalfitforthehierarchicalcontrolmodel,asnestedcomputationcanbeimplementedinRaywithnocentraltaskschedulingbottleneck.Hierarchicalcontrol Actors/Workers RunscriptRemotedecoratorforruninremote InitrayRemotedecoratorforruninremoteInitrayExecutethetrainerandactorinremoteExecutethetrainerandactorinremoteStartthreadforasyncStartthreadforasynctrainingsignal

observation&reward

RealworldenvironmentAgent historyinfo

simulatorPolicyGraphPolicyModelPolicyOptimizerPolicyGraphPolicyModelPolicyOptimizerThepolicyoptimizerisresponsiblefortheperformance-criticaltasksofdistributedsampling,parameterupdates,andmanagingPolicyGraphPolicyModelPolicyOptimizerPseudocodeforfourRLlibpolicyoptimizerstepmethods.Eachstep()operatesalocalpolicygraphandarrayofremoteevaluatorreplicas. Serializationanddeserializationarebottlenecksinparallelanddistributedcomputing,especiallyinmachinelearningapplicationswithlargeobjectsandlargequantitiesofdata.Goalsefficientwithlargenumericaldata(e.g.NumpyandPandasdataframes)AsasPicklePythontypesCompatiblewithsharedmemory(allowingmultipleprocessestousethesamewithoutcopyingit)Deserializationshouldbeextremelylanguageindependent Makingdeserializationfastisimportant.AnobjectmaybeserializedonceandthendeserializedmanytimesAcommonpatternisformanyobjectstobeserializedinparallelandthenaggregatedanddeserializedoneatatimeonasingleworkermakingdeserializationthebottleneckDeserializationisfastandbarelyvisibleUsingonlytheschema,cancomputetheoffsetseachvalueinthedatablobwithoutscanningthroughthedatablob(unlikePickle,thisiswhatenablesfastdeserialization)copyingorotherwiseconvertinglargearraysandothervaluesduringdeserializat