高級(jí)篇第20節(jié)消息中間件在分布式系統(tǒng)中的作用介紹對(duì)比來(lái)源于

MQ對(duì)比基本信息對(duì)比主要關(guān)注前三個(gè)（標(biāo)紅）ActiveMQRabbitMQRocketMqJoramHornetQOpenMQMuleMQSonicMQZeroMQ關(guān)注度高高中中中中低低中成熟度成熟成熟比較成熟比較成熟比較成熟比較成熟新產(chǎn)品無(wú)成功案例成熟不成熟所屬社區(qū)/公司ApacheMozillaPublicLicenseAlibabaOW2JbossSunMuleProgress社區(qū)活躍度高高中中中低高低低文檔多多中多中中少少中特點(diǎn)功能齊全，被大量開(kāi)源項(xiàng)目使用由于Erlang語(yǔ)言的并發(fā)能力，性能很好各個(gè)環(huán)節(jié)分布式擴(kuò)展設(shè)計(jì)，主從HA；支持上萬(wàn)個(gè)隊(duì)列；多種消費(fèi)模式；性能很好在Linux平臺(tái)上直接調(diào)用操作系統(tǒng)的AIO，性能得到很大的提升性能非常好，與MuleESB無(wú)縫整合性能優(yōu)越的商業(yè)MQ低延時(shí)，高性能，最高43萬(wàn)條消息每秒授權(quán)方式開(kāi)源開(kāi)源開(kāi)源開(kāi)源開(kāi)源開(kāi)源商業(yè)商業(yè)開(kāi)源開(kāi)發(fā)語(yǔ)言JavaErlangJavaJavaJavaJavaJavaJavaC支持的協(xié)議OpenWire、STOMP、REST、XMPP、AMQPAMQP自己定義的一套(社區(qū)提供JMS--不成熟)JMSJMSJMSJMSJMSTCP、UDP客戶(hù)端支持語(yǔ)言Java、C、C++、Python、PHP、Perl、.net等Java、C、C++、Python、PHP、Perl等JavaC++（不成熟）JavaJavaJavaJavaJava、C、C++、.netpython、java、php、.net等持久化內(nèi)存、文件、數(shù)據(jù)庫(kù)內(nèi)存、文件磁盤(pán)文件內(nèi)存、文件內(nèi)存、文件內(nèi)存、文件內(nèi)存、文件內(nèi)存、文件、數(shù)據(jù)庫(kù)在消息發(fā)送端保存事務(wù)支持不支持支持支持支持支持支持支持不支持集群支持支持支持支持支持支持支持支持不支持負(fù)載均衡支持支持支持支持支持支持支持支持不支持管理界面一般好無(wú)社區(qū)有webconsole實(shí)現(xiàn)一般無(wú)一般一般好無(wú)部署方式獨(dú)立、嵌入獨(dú)立獨(dú)立獨(dú)立、嵌入獨(dú)立、嵌入獨(dú)立、嵌入獨(dú)立獨(dú)立獨(dú)立評(píng)價(jià)優(yōu)點(diǎn)：成熟的產(chǎn)品，已經(jīng)在很多公司得到應(yīng)用（非大規(guī)模場(chǎng)景）。有較多的文檔。各種協(xié)議支持較好，有多重語(yǔ)言的成熟的客戶(hù)端；缺點(diǎn)：根據(jù)其他用戶(hù)反饋，會(huì)出莫名其妙的問(wèn)題，切會(huì)丟失消息。其重心放到activemq6.0產(chǎn)品—apollo上去了，目前社區(qū)不活躍，且對(duì)5.x維護(hù)較少；Activemq不適合用于上千個(gè)隊(duì)列的應(yīng)用場(chǎng)景優(yōu)點(diǎn)：由于erlang語(yǔ)言的特性，mq性能較好；管理界面較豐富，在互聯(lián)網(wǎng)公司也有較大規(guī)模的應(yīng)用；支持amqp系誒，有多中語(yǔ)言且支持amqp的客戶(hù)端可用缺點(diǎn)：erlang語(yǔ)言難度較大。集群不支持動(dòng)態(tài)擴(kuò)展。優(yōu)點(diǎn)：模型簡(jiǎn)單，接口易用（JMS的接口很多場(chǎng)合并不太實(shí)用）。在阿里大規(guī)模應(yīng)用。目前支付寶中的余額寶等新興產(chǎn)品均使用rocketmq。集群規(guī)模大概在50臺(tái)左右，單日處理消息上百億；性能非常好，可以大量堆積消息在broker中；支持多種消費(fèi)，包括集群消費(fèi)、廣播消費(fèi)等。開(kāi)發(fā)度較活躍，版本更新很快。缺點(diǎn)：產(chǎn)品較新，文檔比較缺乏。沒(méi)有在mq核心中去實(shí)現(xiàn)JMS等接口，對(duì)已有系統(tǒng)而言不能兼容。阿里內(nèi)部還有一套未開(kāi)源的MQAPI，這一層API可以將上層應(yīng)用和下層MQ的實(shí)現(xiàn)解耦（阿里內(nèi)部有多個(gè)mq的實(shí)現(xiàn)，如notify、metaq1.x，metaq2.x，rocketmq等），使得下面mq可以很方便的進(jìn)行切換和升級(jí)而對(duì)應(yīng)用無(wú)任何影響，目前這一套東西未開(kāi)源支持的Protocols對(duì)比Thefollowingtableliststhesupportedprotocolsofeachbroker.ActiveMQRabbitMQRocketMQHornetQQpidZeroMQAMQP1.00-8,0-9,0-9-1-announced1.0-MQTT----OpenWire-----REST---STOMP---STOMPoverWebsockets---XMPPOverGateway----ClientInterfaces對(duì)比ActiveMQRabbitMQRocketMQHornetQQpidZeroQC--C++--Erlang----Haskell----JavaJMS--ActiveMQRabbitMQRocketMQHornetQQpidZeroQJavaproprietary--.NET---Objective-C-----Perl----PHP----Python---Ruby---mqbenchmark[文獻(xiàn)1]結(jié)論在傳統(tǒng)的MQ中rabbitmq的性能表現(xiàn)最好。kafka的性能優(yōu)于rabbitmq；傳統(tǒng)MQ對(duì)比場(chǎng)景設(shè)計(jì)：ScenarioA:Enqueuing20,000messagesof1024byteseach,thendequeuingthemafterwards.ScenarioB:Enqueuinganddequeuingsimultaneously20,000messagesof1024byteseach.ScenarioC:Enqueuinganddequeuingsimultaneously200,000messagesof32byteseach.ScenarioD:Enqueuinganddequeuingsimultaneously200messagesof32768byteseach.ScenarioAScenarioBScenarioCScenarioDKafkavsrabbitmq結(jié)論在kafka和rabbitmq的對(duì)比中，kafka的性能表現(xiàn)較好。rabbitmqkafkarocketmq設(shè)計(jì)初衷更多工作是保證消息遞交；認(rèn)為consumer是一直處于alive狀態(tài)去消費(fèi)消息的；broker設(shè)計(jì)的比較重，需要記錄很多狀態(tài)；充分考慮消息堆積因素，認(rèn)為consumer不一定處于alive狀態(tài)；考慮各個(gè)角色的分布式；為追求吞吐量設(shè)計(jì)；broker設(shè)計(jì)較輕，不保存consumer的消費(fèi)狀態(tài)同kafka；且考慮了事務(wù)特性；路由特性提供了豐富的routing，實(shí)現(xiàn)了amqp的exchange、binding、queuing等model，queue、topic等routing都支持；topicroutingonlytopicroutingonly集群特性對(duì)應(yīng)用透明的集群式實(shí)現(xiàn)；雖然是集群式的實(shí)現(xiàn)，但是集群對(duì)producer并不是完全透明，producer需要確認(rèn)消息發(fā)送到哪一個(gè)同kafkapartition；同時(shí)也利用這一點(diǎn)實(shí)現(xiàn)了消息的順序遞交特性（partition內(nèi)的消息是順序的）；HA通過(guò)mirrorqueue來(lái)支持（master/slave）master提供服務(wù)，slave僅備份通過(guò)replica來(lái)支持queue的master/slave的可以自動(dòng)切換（當(dāng)master宕掉后）支持，但需要人工切換master和slave的角色；消息處理量消息量~=20k/sec;消息量>100k/sec消息量>100k/sec隊(duì)列數(shù)目<1000支持上萬(wàn)隊(duì)列支持上萬(wàn)隊(duì)列順序投遞不支持支持支持事務(wù)性不支持不支持支持簡(jiǎn)介rocketmq的前身為metaq，metaq的第一個(gè)版本是可以看成是linkedin的kafka（scala）的java版本，并對(duì)其增加了事務(wù)的支持。rocketmq為metaq3.0，相比于原始kafka，其擅長(zhǎng)點(diǎn)出了原始的logcollecting之外，還增加諸如HA、事務(wù)等特性，使得從功能點(diǎn)上講，可以替代傳統(tǒng)大部分MQ。目前已經(jīng)在阿里開(kāi)始大規(guī)模應(yīng)用，并且預(yù)計(jì)未來(lái)會(huì)逐漸在阿里取代notify，meta2.x以前的所有隊(duì)列。由于rocketmq還比較新，且官方?jīng)]有給出相應(yīng)的benchmark。而rocketmq社區(qū)主要活躍者均是中國(guó)國(guó)內(nèi)人員，因此不便于直接對(duì)比rocketmq和其他隊(duì)列。在這里用kafka和其他隊(duì)列進(jìn)行對(duì)比，rocketmq的性能比kafka還要好一些。kafkavsrabbitmqindetailKafkaisageneralpurposemessagebroker,likeRabbItMQ,withsimilardistributeddeploymentgoals,butwithverydifferentassumptionsonmessagemodelsemantics.Iwouldbeskepticalofthe"AMQPismoremature"argumentandlookatthefactsofhoweithersolutionsolvesyourproblem.UseKafkaifyouhaveafirehoseofevents(100k+/sec)youneeddeliveredinpartitionedorder'atleastonce'withamixofonlineandbatchconsumers,youwanttobeabletore-readmessages,youcandealwithcurrentlimitationsaroundnode-levelHA(orcanusetrunkcode),and/oryoudon'tmindsupportingincubator-levelsoftwareyourselfviaforums/IRC.UseRabbitifyouhavemessages(20k+/sec)thatneedtoberoutedincomplexwaystoconsumers,youwantper-messagedeliveryguarantees,youdon'tcareaboutordereddelivery,youneedHAatthecluster-nodelevelnow,and/oryouneed24x7paidsupportinadditiontoforums/IRC.Neitheroffersgreat"filter/query"capabilities-ifyouneedthat,considerusingStormontopofoneofthesesolutionstoaddcomputation,filtering,querying,onyourstreams.OrusesomethinglikeCassandraasyourqueryablecache.Kafkaisalsodefinitelynot"mature"eventhoughitis"productionready".Details(caveat-myopinion,I'venotusedeitheringreatanger,andIhavemoreexposuretoRabbitMQ)Firstly,onRabbitMQvs.Kafka.Theyarebothexcellentsolutions,RabbitMQbeingmoremature,butbothhaveverydifferentdesignphilosophies.Fundamentally,I'dsayRabbitMQisbroker-centric,focusedarounddeliveryguaranteesbetweenproducersandconsumers,withtransientpreferredoverdurablemessages.WhereasKafkaisproducer-centric,basedaroundpartitioningafirehoseofeventdataintodurablemessagebrokerswithcursors,supportingbatchconsumersthatmaybeoffline,oronlineconsumersthatwantmessagesatlowlatency.RabbitMQusesthebrokeritselftomaintainstateofwhat'sconsumed(viamessageacknowledgements)-itusesErlang'sMnesiatomaintaindeliverystatearoundthebrokercluster.Kafkadoesn'thavemessageacknowledgements,itassumestheconsumertracksofwhat'sbeenconsumedsofar.BothKafkabrokers&consumersuseZookeepertoreliablymaintaintheirstateacrossacluster.RabbitMQpresumesthatconsumersaremostlyonline,andanymessages"inwait"(persistentornot)areheldopaquely(i.e.nocursor).RabbitMQpre-2.0(2010)wouldfalloverifyourconsumersweretooslow,butnowit'srobustforonlineandbatchconsumers-butclearlylargeamountsofpersistentmessagessittinginthebrokerwasnotthemaindesigncaseforAMQPingeneral.Kafkawasbasedfromthebeginningaroundbothonlineandbatchconsumers,andalsohasproducermessagebatching-it'sdesignedforholdinganddistributinglargevolumesofmessages.RabbitMQprovidesrichroutingcapabilitieswithAMQP0.9.1'sexchange,bindingandqueuingmodel.Kafkahasaverysimpleroutingapproach-inAMQPparlanceitusestopicexchangesonly.Bothsolutionsrunasdistributedclusters,butRabbitMQ'sphilosophyistomaketheclustertransparent,asifitwereavirtualbroker.Kafkamakesitexplicit,byforcingtheproducertoknowitispartitioningatopic'smessagesacrossseveralnodes,thishasthebenefitofpreservingordereddeliverywithinapartition,whichisricherthanwhatRabbitMQexposes,whichisalmostalwaysunordereddelivery(theAMQP0.9.1modelsays"oneproducerchannel,oneexchange,onequeue,oneconsumerchannel"isrequiredforin-orderdelivery).Putanotherway,Kafkapresumesthatproducersgenerateamassivestreamofeventsontheirowntimetable-there'snoroomforthrottlingproducersbecauseconsumersareslow,sincethedataistoomassive.ThewholejobofKafkaistoprovidethe"shockabsorber"betweenthefloodofeventsandthosewhowanttoconsumethemintheirownway--someonline,othersoffline-onlybatchconsumingonanhourlyorevendailybasis.Kafkacandeliver"atleastonce"semanticsperpartition(sincemaintainsdeliveryorder),justlikeRabbitMQ,butitdoesitinaverydifferentway.Performance-wise,ifyourequireordereddurablemessagedelivery,currentlyitlookslikethere'snocomparison:KafkacurrentlyblowsawayRabbitMQintermsofperformanceonsyntheticbenchmarks.Thispaperindicates500,000messagespublishedpersecondand22,000messagesconsumedpersecondona2-nodeclusterwith6-diskRAID10./en...OfcoursethiswaswrittenbytheLinkedInguyswithoutnecessarilyexpertRabbitMQinput,soYMMV.Finally,areminder:KafkaisanearlyApacheincubatorproject.Itdoesn'tnecessarilyhaveallthehard-learnedaspectsinRabbitMQ.Now,awordonAMQP.Frankly,itseemsthestandardisamess.Officiallythereisa1.0proposedspecificationthatisgoingthroughtheOASISstandardsprocess.Inpracticeitisaforkedstandard,one(0.9.1)supportedbyvendors,theother(1.0)supportedbytheworkinggroup.Asetofgenerallyavailable,widely-adopted,production-qualityAMQP1.0implementationsacrossthemajorreleases(QpidfromRedhat,RabbitMQ,etc.)won'texistuntil2013,ifever.Asanexternalobserverwithnoinsideknowledge,hereiswhatitlookslike:theworkinggroupspent5yearsonaspec,from2003to2008,culminatinginawidelyadoptedrelease(0.9.1).Thenasubsetofmorepowerfulworkinggroupmembersrewrotethespecbylate2011,completelyshiftingthefocusofthespecfromamessagingmodeltoatransportprotocol(sortoflikeTCP++),anddeclaredit1.0.So,wehavethestrangecasewherethe"mature"AMQPisthenon-standard0.9.1specificationandthe"immature"AMQPistheactual1.0standard.Thisisn'ttosuggest1.0isn'tgoodtechnology,itlikelyis,