云計算-cloud l主要需求和目標

副研究員博士 主要需求和目標實現(xiàn)為程序員隱藏系統(tǒng)層細需要考慮的細節(jié)技術問題如何管理 數(shù)據(jù)？如何劃分數(shù)據(jù)如何調度計算任務并分配map和reduce節(jié)點如果節(jié)點間需要共享或交換數(shù)據(jù)如何如何掌控節(jié)點的執(zhí)行完成情況？如何收集中間和最終的結果節(jié)點失效如何處理？如何恢復數(shù)節(jié)點擴充后如何保證原有程序仍能正常運 系統(tǒng)性能提升 TextText1:theweatheris Text2:todayisText3:goodweatheris Text4:todayhasgoodString[]text=args[0];HashTableht=newHashTable();for(i=0;i<3;++i){StringTokenizerst=newStringTokenizer(text[i]);while(st.hasMoreTokens()){Stringword=if(!ht.containsKey(word)){ht.put(word,newInteger(1));}else{intwc((Integer)ht.get(word)).intValue()+1;//計數(shù)加1ht.put(word,new}}}for(Iteratoritr=ht.KeySet().iterator();itr.hasNext();{Stringword=System.out.print(word+“:”+ }輸出：good: has:1;is: the: today: weather:

鍵值

鍵值

海量數(shù)

鍵值

初始鍵值

Barrier：AggregationBarrier：Aggregationand

中間結

計算結MapReducemap:(k1;v1)→[(k2;reduce:(k2;[v2])→[(k3;)][最終的某種形式的結果輸出[(k3;v3)]輸出：最終輸出結果[(k3;使用4個map節(jié)點map節(jié)點輸入：(text1theweatheris輸出：(the1weather1(is1(goodmap節(jié)點輸入：(text2todayismap節(jié)點map節(jié)點reduce節(jié)點輸入：(good,1),(good,1),(good,1),(good1),(good(is,1),(is,輸出(is,1),(is,輸出good:is:3weather:reduce節(jié)點輸入：(has,1),輸出：(has,1),(is,輸入：(the,1),today1),(today,(weather,1),(weather,1),(weather,輸出：(the,1),(today,2),weatherreduce節(jié)點輸入：(good,1),(good,1),(good,2)(good,(is,1),(is,輸出(is,1),(is,輸出good:is:3weather:reduce節(jié)點輸入：(has,1),輸出：(has,1),(is,輸入：(the,1),today1),(today,(weather,1),(weather,1),(weather,輸出：(the,1),(today,2),weatherHadoop:Mapper&ReducerHadoop:DriverHadoop:Run HadoopMapReducesplitsplitsplitsplit

partpartpartpartsplitsplitReducershas3primaryphases:shuffle,sortandHadoop:HowmanyTasksetuptakesawhile,soitisbestifthemapstakeatleastaminuteto用戶可以顯示配置Configuration.set(MRjobConfig.NUM_MAPSintbutitonlyprovidesahinttotheframework10TBofinputdata,blocksizeisYouwillendupwith80KHadoop:HowmanyThenumberofreducersforthejobissetbytheuserTherightnumberofreducesseemstobe0.95or1.75multipliedby(<no.of<no. umcontainersperWith0.95allofthereducescanlaunch yandstarttransferringmapoutputsthemapsWith1.75thefasternodeswillfinishtheirfirstroundofreducesandlaunchasecondwave ngamuchbetterjobofloadIncreasingthenumberofreducesincreasestheframeworkoverhead,butloadbalancingandlowersthecostofHadoop:PartitionercontrolsthepartitioningofthekeysoftheintermediateThetotalnumberofpartitionsisthesameasthenumberofreducetasksforthejob.Hencethiscontrolswhichofthemreducetaskstheintermediatekey(andhencetherecord)issenttoforreductionHashPartitioneristhedefaultHadoop:JobJobrepresentsaMapReducejobJobistheprimaryinterfaceforausertodescribeaMapReducejobtoHadoopframeworkforJobistypicallyusedtoHadoop:JobOptionally,JobisusedtospecifyotheradvancedfacetsofthejobsuchFilestobeputintheWhetherintermediateand/orjoboutputsaretobecompressed(andWhetherjobtaskscanbeexecutedinaspeculativeumnumberofattemptsperUserscanuseConfiguration.set(String,String)/Configuration.get(String)toset/getarbitraryparametersneededbyapplicationsCountersrepresentglobalCounterscouldbedefinedMapReduceInthemapand/orreducemethods,applicationscandefinearbitraryCounters(oftypeEnum)andupdatethemvia:Counters.incrCounter(Enum,Counters.incrCounter(String,String,ThesecountersarethengloballyaggregatedbytheHadoop:CurrentlyFileInputFormat,TextInputFormat,SequenceFileInputFormat,CombineFileInputFormat,CombineSequenceFileInputFormat,CombineTextInputFormat,CompositeInputFormat,FixedLengthInputFormat,SequenceFileAsBinaryInputFormat,Seq