基于Hadoop的云存儲

1、1基于Hadoop的云存儲 隨時(shí)隨地訪問您的應(yīng)用2云存儲的數(shù)據(jù)管理特點(diǎn)與挑戰(zhàn)特點(diǎn)挑戰(zhàn)計(jì)算資源是可伸縮的數(shù)據(jù)具有備份數(shù)據(jù)存儲在大量分布的節(jié)點(diǎn)之上數(shù)據(jù)的自我管理和自調(diào)優(yōu)基于大量節(jié)點(diǎn)的查詢優(yōu)化算法基于大量節(jié)點(diǎn)的索引結(jié)構(gòu)資源調(diào)度和負(fù)載均衡多租戶情況3我們面臨的問題您如何來管理大量的應(yīng)用程序？運(yùn)行任務(wù)來處理100百萬兆字節(jié)的數(shù)據(jù)花費(fèi)11天在一臺電腦上讀取數(shù)據(jù)需要大量低價(jià)的計(jì)算機(jī)故障處理速度問題（15分鐘修復(fù)1000臺計(jì)算機(jī)），但可靠性問題在大型計(jì)算機(jī)集群中，每天都有計(jì)算機(jī)出現(xiàn)故障集群的規(guī)模不斷變化需要通用的基礎(chǔ)架構(gòu)必須是高效且可靠的4解決方案開源的Apache項(xiàng)目Hadoop主要包括:Distribut

2、ed 分布的數(shù)據(jù)Map/Reduce 分布的應(yīng)用程序使用Java開發(fā)運(yùn)行在 Linux, Mac OS/X, Windows, and Solaris廉價(jià)的硬件設(shè)備5Typically in 2 level architectureNodes are commodity PCs40 nodes/rackUplink from rack is 8 gigabitRack-internal is 1 gigabit Hardware Cluster of Hadoop6Distributed Single namespace for entire clusterManaged by a singl

3、e namenode.Files are single-writer and append-only.Optimized for streaming reads of large files.Files are broken in to large blocks.Typically 128 MBReplicated to several datanodes, for reliabilityAccess from Java, C, or command line.7Block PlacementDefault is 3 replicas, but settableBlocks are place

4、d (writes are pipelined):On same nodeOn different rackOn the other rackClients read from closest replicaIf the replication for a block drops below target, it is automatically re-replicated.8How is Yahoo using Hadoop?Started with building better applicationsScale up web scale batch applications (sear

5、ch, ads, )Factor out common code from existing systems, so new applications will be easier to writeManage the many clusters9Running Production WebMapSearch needs a graph of the “known” webInvert edges, compute link text, whole graph heuristicsPeriodic batch job using Map/ReduceUses a chain of 100 ma

6、p/reduce jobsScale1 trillion edges in graphLargest shuffle is 450 TBFinal output is 300 TB compressedRuns on 10,000 coresRaw disk used 5 PB10Terabyte Sort BenchmarkStarted by Jim Gray at Microsoft in 1998Sorting 10 billion 100 byte recordsHadoop won the general category in 209 seconds910 nodes2 quad

7、-core Xeons 2.0Ghz / node4 SATA disks / node8 GB ram / node1 gb ethernet / node40 nodes / rack8 gb ethernet uplink / rackPrevious records was 297 seconds11Hadoop clustersWe have 20,000 machines running HadoopOur largest clusters are currently 2000 nodesSeveral petabytes of user data (compressed, unreplicated)We run hundreds of thousands of jobs every month12Research Cluster Usage13Who Uses Hadoop?Amazon/A9AOLFacebookFox