基于Spark平臺的GPU調(diào)度研究[通用]

1、基于Spark平臺的GPU調(diào)度研究蘇鵬飛ICT-HPC目錄2Spark簡介目前工作動機現(xiàn)有編程模型（MapReduce）是一種非循環(huán)數(shù)據(jù)流抽象模型，不適用于需要重復使用數(shù)據(jù)集的應用：迭代計算（機器學習）交互式數(shù)據(jù)挖掘工具（R , Excel, Python）MapMapMapReduceReduceDiskInputDiskOutput特性分布式內(nèi)存抽象支持數(shù)據(jù)延遲查詢支持比Map和Reduce更多的函數(shù)提供交互式的Scala shell和Python shell保留MapReduce優(yōu)點容錯數(shù)據(jù)局部性可擴展性生態(tài)圈架構(gòu)數(shù)據(jù)存儲資源管理計算架構(gòu)sc = new SparkContextf =

2、sc.textFile(“”)f.filter() .count().programSpark client(app master/driver)Spark workerHDFS, HBase, Block managerTask threadsRDD graphSchedulerBlock trackerShuffle trackerClustermanagerRDD全稱: Resilient Distributed Datasets容錯的、只讀的、分布式數(shù)據(jù)集允許用戶指定數(shù)據(jù)的存儲級別(內(nèi)存/外設(shè))提供豐富的并行操作ReduceCollectCountRDD操作類型變換(Transform

3、ation)不計算，僅返回新RDD行動(Action)進行計算，返回新值給driver程序RDD一個例子：日志挖掘lines = spark.textFile(“hdfs:/.”)errors = lines.filter(_.startsWith(“ERROR”)messages = errors.map(_.split(t)(2)cachedMsgs = messages.cache()Block 1Block 2Block 3WorkerWorkerWorkerDrivercachedMsgs.filter(_.contains(“foo”).countcachedMsgs.filter

4、(_.contains(“bar”).count. . .tasksresultsCache 1Cache 2Cache 3Base RDDTransformed RDDCached RDDParallel operation對Wikipedia 全文搜索耗時 map.put(pair._1, pair._2) map Collect實現(xiàn)剖析def collect(): ArrayT = val results = sc.runJob(this, (iter: IteratorT) = iter.toArray) Array.concat(results: _*) Collect實現(xiàn)剖析rdd.collect()RDD Objectsbuild operator DAGDAGSchedulersplit graph into stages of taskssubmit each stage as readyDAGTaskSchedulerTaskSetlaunch tasks via cluster managerretry failed or straggling tasksClustermanagerWo