畢業(yè)設計外文文獻-基于網絡爬蟲的信息挖掘系統(tǒng)設計與實現(xiàn)

附錄A外文翻譯—原文部分InformationMiningSystemDesignandmplementationBasedonWebCrawlerShanLin,You-mengLi,Qing-chengLiCollegeofInformationTechnicalScienceNankaiUniversityTianjin,300072,CHINAE-mail:lsskyshan@,solsikja@,liqch@Abstract–WiththeinformationexplosioncausingbytheWorldWideWebinrecentyears,theissueofhowtoexecutetheenormousinformationefficientlyatareasonablelosthasbecometheconcernofinformationproviders,serviceagenciesandendusers.Whenmanyresearchfocusonhowtodesignanefficientwebcrawler,wepayourattentiontohowtomakethebestoftheresultofwebcrawler.Inthispaper,wedescribethedesignandimplementationofaninformationminingsystemrunningontheresultsofwebcrawlertogainmoremetadatafromunstructureddocumentsforfocusedsearch(suchasRSSsearch).Wepresentthesoftwarearchitectureofthesystem,describeefficienttechniquesforachievinghighperformanceandreportpreliminaryexperimentalresultstoprovethatthissystemcanaddresstheissueofrobustness,flexibilityandaccuracyatalowcost.Keywords:Crawler,informationmining,RSS,lowcost.IntroductionTheexplosivegrowthoftheWorldWideWebgivespeopleamagicchangeoftheirlifestylesandworkingmanners.Astudyreleasedin2003[1]showedthatthevolumeofinformationontheWeb,whichisaccessibledirectly,isabout167terabytes,consistingabout2.5billionpages.Accordingtothelatestsurvey[2],byDecember2007,thetotalofnetizensintheworldhadincreasedto1,320million,withasharpincreaseof265.6%.Althoughexponentiallyincreasingamountsofmaterialareavailable,findingandmakingsenseofthismaterialispotentiallyuseful,butdifficultwithpresentsearchtechnology,HowtomakethebestofthehugedataandmanagethedocumentsontheInternetefficientlybecomeaveryimportanttasktoinformationprovidersandwebserviceagencies.Ouroverallaimistodesignafeasibleandflexibledistributedinformationminingsystem,whichcanmakethebestofthemetadataresultfromwebcrawlers,maximizethebenefitsobtainedperdownloadedpageandgetmoreby-productsatacomparativelylowcost.Weimplementthesystemarchitectureonthebasisofasimplebreadth-firstcrawlercalled‘WebSpider’,althoughthesystemcanbeadaptedtootherstrategies.Wereportpreliminaryexperimentalresultsinsection3,andtheconclusionanddirectionforfutureworkwillbepresentedattheendofthispaper.InformationMiningWebinformationminingtechniqueisaspecialexpandedapplicationofdataminingtechniquesonmanagingthehugeinformationontheInternet.WebinformationminingistheprocessofscratchingthemetadatafromtheInternet,analyzingfromdifferentperspectivesandsummarizingitintousefulinformation.Itincludesinformationextraction,informationretrieval,naturallanguageprocessinganddocumentsummarization.Informationminingcanadoptsomedataminingtechniques,buttherearesignificantdifferencesbetweenthem.Informationminingworkswithunstructureddata,suchasWebpagesandtextdocuments,incontrasttoDataMiningwhichisbasedonstructureddatalikerelationaldata.WebCrawlerThehugesizeofdataontheInternetgivethebirthofwebsearchengines,whicharebecomingmoreandmoreindispensableastheprimarymeansoflocatingrelevantinformation.Suchsearchenginesrelyonmassivecollectionsofwebpagesthatareacquiredbytheworkofwebcrawlers,alsoknownaswebrobotsorspiders.Awebcrawlerisaprogram,whichbrowsestheWorldWideWebinamethodical,automatedmanner.Webcrawlersaremainlyusedforautomatingmaintenancetasksbyscratchinginformationautomatically.Typically,acrawlerbeginswithasetofgivenWebpages,calledseeds,andfollowsallthehyperlinksitencountersalongtheway,toeventuallytraversetheentireWeb[3].GeneralcrawlersinserttheURLsintoatreediagramandvisittheminabreadth-firstmanner.Therehasbeensomerecentacademicinterestinnewtypesofcrawlingtechniques,suchasfocusedcrawlingbasedonsemanticweb[6,8],cooperativecrawling[10],distributedwebcrawler[7],andintelligentcrawling[9],andthesignificanceofsoftcomputingcomprisingfuzzylogic(FL),artificialneuralnetworks(ANNs),geneticalgorithms(GAs),androughsets(RSs)highlighted[11].Thebehaviorofawebcrawleristheoutcomeofacombinationofpolicies:1Aselectionpolicythatstatedwhichpagestodownload.2Are-visitpolicythatstateswhentocheck.3Apolitenesspolicythatstateshowtoavoidoverloadingwebsites.4Aparallelizationpolicythatstateshowtocoordinatedistributedwebcrawlers.[4]Intheridofrepeatedoperation,crawlersneedmakearecordofthewebpageswhichhavebeendownloadedbyHashTable.Thatmeansaftercrawlingsearchenginesstorenumerouspagesintheirdatabases.Thehardertaskisthatthecrawlingandstoringworkshouldrepeatinacertainperiod.TakingthemostpopularsearchengineGoogleasanexample,in2003,Google’scrawlercrawledineverymonth,butnow,crawlsevery2or3days.Socrawlingonthemassivepagesinsuchfrequency,thecostofnetresourceandstorageishuge.Itisexactlythemotivationofthispaperthatsincewehavetorunacrawlertofetchnumerouspagesofdataataenormouscostofmachinehourandstorage,whydon’twetakefulladvantageofitandtrytogetmoreusefulinformationintheformofmetadatawhichisdataaboutdata?RDFandRSSThispaperdescribesthedesignandimplementationofanoptimizeddistributedinformationminingsystem,takingtheapplicationofscratchingRSS(ReallySimpleSyndication)FeedfromnetasanexamplewhichisonthebasisofRDF.TheResourceDescriptionFramework(RDF)isageneral-purposelanguageforrepresentinginformationintheWeb.ThisdocumentdefinesanXML(ExtensibleMarkupLanguage)syntaxforRDFcalledRDF/XMLintermsofNamespacesinXML,theXMLinformationSetandXMLBase.[5]RDFallowsforrepresentationofrichmetadatarelationshipsbeyondwhatispossiblewithearlierflat-structuredRSS.TheReallySimpleSyndication(RSS)isastandardformattodescriptandsyndicatethewebinformation.ItisalightweightXMLformatdesignedforsharingheadlinesandhandingotherwebcontentsyndication,whichiswidelyusedinInternetnews,BlogandWiki.RSSisaformatusedtoindexinformationandmetadata.Forinstance,notalltheInternetnews’contentisalwaysfree.Butthemetadataofthearticlesisusuallyshared,suchastitle,author,linkandabstract.SoRSSbecometheinformationplatformofthesemetadata,andwecanregardRSSasanefficientwaytogetandsharewebinformation.Figure1asaboveshowsthemaintagsofstandardformatoftheRSS2.0document.BysubscribingtheRSSfeeds,Figure1.RSS2.0maintagtreerepresentation.youcanreceivethenewestinformationwithoutanyoperation.ThatisthemostimportantcharacterofRSS–SyndicationandAggregation.SoRSShasalreadybecomethemostpopularapplicationofXML.BecauseRSSfollowtheXMLstandardformat,wecanparseRSSSeeddocumentsbytheDOM(DocumentObjectModel).TheprocessofcertifyaRSSdocumentshouldbedividedintotwostepsasfollow:TheheadofthedocumentfollowtheRSSformat.ThedocumentcanbesetDOMandparsedsuccessfully.ThedetailedimplementationwillbepresentatSection3.DesignOverview3.1AssumptionsIndesigningawebinformationminingsystemforourneeds,wehavebeenguidedbyassumptionsthatofferbothchallengesandopportunities,whichareunderguidanceofsomepreliminaryobservation.1Theinformationminingsystemshouldstorehugedataandnumerousfilestemporarily.Asthelimitationofexperimentinstruments,weneednotconsiderthelimitofstorage.2Asthelimitationofbandwidth,wesetthelongestresponsetimefordownloadertoensurethesystemcanruncontinuallyandnormally.Buttheovertimewillreducethescratchingspeed.Sohighsustainedbandwidthismoreimportantthanlowlatency.3Thesystemshouldbebuiltfromseveralcomponents.Sinceitisnotthekeytosolveinthispaper,wedon’tconsidertheproblemoftoleratingandrecovering.3.2ArchitectureThisInformationMiningSystemconsistsoffourmajorkindsofcomponents–Crawler,InformationMiningMachine,FilterandDownloaderasshowninFigure2.Eachoftheseistypicallyacommoditycomputerrunningaser-levelserverprocess.Figure2.SystemarchitectureInthesystem,Crawlerisusedtoscratchingallkindsofwebpagessuchashtml,xml,asp,jspandsoonfromasetofseedpages.TheoutputofCrawlerisformattedintheattributesofnumber,URL,Text(abstractinformationaboutURL).Sincethecrawlerisnotessentialforourexperimentalsetup,wewon’tintroducethealgorithmanddetailedimplementationofcrawlerinthispaper.Notethatweonlyparseforhyperlinks,andnotforindexingtermsby‘WebSpider’,whichwouldsignificantlyslowdowntheapplication.ThenthedatawillbesendintoMiningMachinetoprocesswhichisthekeycomponentofthesystemwiththehelpofFilter.Thedetailedimplementationwillbedescribedinthefollowingsection.AtlastDownloadertakethechargeofdownloadingthewebpagesfollowingthelistfromInformationMiningMachine,scratchthemetadataandstoreintheserverdatabase.Inordertoachievehigh-performancewhichmeansdownloadhundredsoreventhousandsofpagespersecond,thedesignoftheclusterofDownloadersisquiteimportant.Forsystemflexibilityconsideration,thenumberoftheDownloaderisnotfixed.Thatmeanswecaninsertdownloadersintothesystemasweneedtoadapttodifferentexperimentconditionsandapplicationswithareasonableamountofwork.Beforedownloading,thesystemcandetectthenumberofthedownloadersautomatically,andtheitemsintheoutputlist.ToguaranteetheaccuracyofInformationMiningsystem,afterdownloadingthepagefilesuccessfully,theDownloaderchecksthefileagaintomakesurethatitisavalidRSSfeed.AsalltheworkofparsingaXMLfilecanbeimplementedbysetaDOM.SowecanjudgeaRSSfileinthemannerofcheckingwhetheritcanbestructuredasavalidDOMstructure.Atthesametime,thesystemscratchesthemetadatasuchastitle,link,dateandsoonfromDOMinterfacesandstoresinthedatabase.InformationMiningMachineTheminingmachinecomponenttraversestheitemslistedinthefile‘link.txt’inthedataflow,whichisimplementatedinC++.Itisconvenienttoscratchthelinkweneedbyregularexpression.Forexample,RSSisaspecialXMLfile,aXMLapplication,conformstotheW3C’sRDFSpecificationandisextensibleviaXMLnamespaceand/orRDFbasedmodularization[12].Sowedefinetheregularexpressionendingby‘.xml’atfirst:Exp(RSS)={,(.*)(?=\.xml),}(1)Aftersomeexperiments,wefindthat:1)Somewebpages(html,xml,asp,jsp,php…)aredirectedbytheirserverstojumpfromanon-RSSlinktoaRSSlinkautomatically.2)SomeURLdirectoryjumptoaRSSlinkdirectly.Forexample,theURLasfollowedactuallypointsataRSSfileaboutnews./rss2.aspAlthoughitseemstobeanaspwebpage,itisactuallydirectedtoaRSSfileacquiescently.SoifonlyscratchXMLfiles,wewillmissalotofRSSseeds.Thenweredefinetheregularexpressionasfollows:Exp(RSS)={,(.*)[(?=\.xml)|(?=\.asp)|(?=\.jsp)|(?=\.php)],}(2)IftheURL’sformatistallywiththeregularexpression(2)asabove,theinformationminingmachineinsertittothelistofpotentialhandlingtargets.Thenthishandling-listwillbesenttotheFilterthroughthedataflowsimultaneously.Experientially,executingtimealwaysinthelineargrowth,becausealltheworkshouldbedonebytraversingthewholedocument,andparsingitonthedifferentdetailedlevel.Here,thechallengeistoavoidtraversingandoverparsingasfaraspossible.Thusinoursystem,wedesignthecomponentcalledFiltertoco-operatewithinformationminingmachine,whichisinchargeofdealingwiththeproblem.Beforefetchingthevaluableinformationhiddeninunstructuredwebpages,theFilterofoursystemwillpreinspectthesedocuments,sendmetadatatotheInformationMiningMachinewhichismostpossiblytobeaRSSfile,andwhichisimpossible.Atfirst,theFilterdownloadfilestothesystemcacheandreadonly50bitsofeachpagerelatedtothelinkfromtheMiningMachine,thencheckoutwhetherthese50bitsdatafollowthestandardRSS1.0(moredetailsoftheRSS1.0referto[13]).InRSS1.0,alltheRSSfilesbeginwiththefollowingformat:<?Xmlversion="1.0"encoding="utf-8"?>Ofcourse,therearesomeothercodingstandardsuchasGB2312,UTF-16.Westilluseregularexpressiontocheckthebeginning50bitsofthefileswhetheritfollowsRSS1.0standard.IftheresultisTRUE,theFilterreturnsthelinkofthepagetotheInformationMiningMachine,ifnot,thislinkwillbeflittedoutwithoutnomoreunnecessaryoperation.ExperimentalResultandAnalysisWepresentthepreliminaryexperimentalresultsandexperiencehereanddosomebriefanalysisonit.Adetailedanalysisofperformancebottlenecksandscalingbehaviorisbeyondthescopeofthispaper,andwouldrequireafastsimulationtested,sinceitwouldnotbepossible(orappropriate)todosuchastudywithourcurrentInternet.ExperimentalResultonStep1SinceRSSiswidelyusedinwebnews,Blog,Wikiandsoon,ourexperimentalinitializingSeedLinkfortheCrawlershouldcoverasmanykindsofthsesaspectsaspossible.Becauseofourexperimentalcondition,thescopewecoveredontheInternetisverylimited.Soa‘right’seedlinkissignificantwhichcankeepthesystemrunningmoreefficiently.Asouranalysis,aseedlinkpagewhichisfulloflinkscanincreasethemininghitrate.OnStep1,wechoosethefollowingURLsastheseedlinkoftheCrawlerrunningrespectivelyincomparison:1B:ApopularBlogdiscoverysite.2Techcrunch:Oneofthemostfamousweblog.ExperimentalResultonStep2Wechoosethelink‘/p/articles/?sm=rss’ofBNETwhichispointedtothepageofaRSSresourcemapsiteandfullofInternetnews,bythestep1oftheexperiment.AfterthreeDownloadersrunning100hours,thenumberofhyperlinksin‘link.txt’requestlistis105025,including101872validURLs.ThetrendofthespeedofRSSinformationminingexecutedbyoneoftheDownloadersisshowninFigure3.ThegraphinFigure3revealsthatthenumberofvalidRSSSeedsscratchedbyInformationMiningMachineapproximatelypresentsalineargroethwiththeexcutingtime.Andtheflatpartofthetrendisrelatedtothelinkstructureofthewebsite.Atlastwescratch2312RSSFeeds,aftersendtoFilter,thereare2007validRSSFeeds.Theharvestrateisabout0.3345perminutewhichislimitedbythebrandwedth.FutureWorkWehavedescribedtheInformationMiningSystem,adistributedsystemforfindingoutvaluablestructuredmetadatahiddeninthethousandsofmillionsofunstructuredwebdocuments.Inaddition,wepresentpreliminaryexperimentsalongwithsomebriefanalysis.InthisInformationMiningSystem,thereareobviouslysomeimprovementscanbemade.Amajoropenissueforfutureworkisadetailedsolutiontoincreasetheharvestrateofourinformationminingsystem.Althoughtheharvestrateistightlyrelatedtothebrandwidth,wecanoptimizethesystemarchitecturetoimprove.Ascompletewebcrawlingcoveragecannotbeachieved,duetothevastsizeofthewholeinternetandtoresourceavailability,oursystemcan’tscratchalltheRSSFeeds.Sohowtoincreasethecoveragerateisanothertask.Forthefuturework,wewillmonitortheRSSSeeds,setmeasurementstandardssuchaslifecycleandfreshconditionwhichwerejustlikethemeasurementoftherealseedsinthenatureworld.ItwillbeacompletelynewideaaboutRSSSeeds,butabsolutelynecessarytohandlethemillionsofRSSSeeds.Inaddition,wewillimprovetheDownloaderconponentbythewayofsupervisedlearningtoincreasetheharvestrateofRSSscratching.Bysomeguidanceself-learedfromsampledata,thedownloadercanjudgeeasilytodownloadpagesselectively.Lastbutnottheleast,inordertomanagetheseRSSSeedswegetfromtheminingSystemefficiently,thewayofevaluatingshouldbeconsidered.Theimprovementsaboveallwillmakethissystemmorerealisticreliableandfriendliertousers.Figure3.Scratchingtrend

附錄B外文翻譯—譯文部分基于網絡爬蟲的信息挖掘系統(tǒng)設計與實現(xiàn)摘要-近年來，信息量突增。萬維網如何解決在一個點有效的執(zhí)行大量的信息，以及減少損失已經成為提供者、服務機構和用戶關注的焦點。當許多研究的重點是如何設計一個高效的網絡爬蟲，而我們的研究重點是如何使爬蟲的結果是最好的。在下文中，我們描述了信息挖掘系統(tǒng)的設計與實現(xiàn)過程。在Web爬蟲的結果中獲得更多的元數(shù)據。用于集中搜索非結構化文檔（如RSS搜索）。我們介紹了系統(tǒng)的軟件架構，描述了如何實現(xiàn)高績效性能的有效技術。Keywords:爬蟲，信息挖掘，RSS，低成本。1介紹萬維網的爆炸式增長使人們生活方式和工作方式都發(fā)生了很大變化。2003年發(fā)布的一項研究顯示，可直接訪問的網絡信息量，約167兆字節(jié)，約25億頁。根據最新調查，到2007年12月全，世界網民總數(shù)增加到13.2億，大幅增長265.6%。雖然數(shù)量呈指數(shù)增長。但發(fā)現(xiàn)和搜集這些有用的信息還是很難。如何充分利用龐大的數(shù)據并管理，成為了互聯(lián)網的一項重要任務。我們的總體目標是設計一個靈活可行性高的分布式信息挖掘系統(tǒng)，使爬蟲抓取到的數(shù)據最優(yōu)化，并且使得每一頁的下載都能帶來最大化的信息。我們WebSpider系統(tǒng)基于廣度優(yōu)先，并且可以適應其他策略。信息挖掘Web信息挖掘技術對于管理和拓展互聯(lián)網上的海量信息非常有效。信息挖掘是對互聯(lián)網的元數(shù)據進行抓取的一個過程，分析不同來源的信息，提取其中的有效信息。它包括信息提取、信息檢索、自然語言處理和文檔摘要。信息挖掘和數(shù)據挖掘看起來相似，但兩者存在著顯著的差異。信息挖掘使用非結構數(shù)據，而數(shù)據挖掘基于結構化數(shù)據。1.2信息挖掘互聯(lián)網上的龐大數(shù)據量使得搜索引擎越來越多，數(shù)據定位成為不可或缺的手段。這些搜索引擎依賴于大規(guī)模的網絡爬蟲，也稱為網絡機器人或蜘蛛。在擺脫重復操作上，爬蟲需要做一個網頁去重記錄，也稱url去重，這意味著在抓取搜索引擎后，他們的數(shù)據庫有大量的頁面信息，更艱巨的任務是爬蟲爬取重復的信息，使得數(shù)據重復量過于龐大。以最受歡迎的谷歌搜索引擎來說，以前是一個月爬一次，現(xiàn)在是2-3天爬取一次。所以如此頻繁的爬取網頁，網絡成本和資源存儲是巨大的，所以我們必須運行一個爬蟲來獲取其中的元數(shù)據而非整個頁面，并嘗試以新的形式去存儲這些元數(shù)據。2RDFandRSS本文介紹了以網絡為例分布式信息挖掘系統(tǒng)RSS的應用（ReallySimpleSyndication）和RDF基礎的設計和優(yōu)化。資源描述框架（RDF）是一個用于表示信息的通用網絡語言。本文檔定義了一個XML（可擴展的RDF的標記語言）語法，稱為RDF/XML中的命名空間術語，XML信息集和XMLBase。真正簡單的聚合（RSS）是一種用來描述和聯(lián)合Web信息的標準格式。它是一個輕量級XML，旨在分享標題并聯(lián)合處理其他網絡內容，廣泛應用于互聯(lián)網新聞，博客和維基。RSS是一個用于索引信息和元數(shù)據的格式。并非所有互聯(lián)網新聞的內容都是免費的。但文章的元數(shù)據通常是共享的，例如標題，作者，鏈接和摘要。所以RSS成了這些元數(shù)據的信息平臺，我們可以考慮RSS是獲取和共享Web信息的有效方式。通過查看RSS文檔，可以了解最新的信息操作。這是RSS最重要的特征-企業(yè)聯(lián)合組織和聚合。所以RSS已經成為了最流行的XML應用程序。因為RSS遵循XML標準格式，我們可以通過DOM解析RSS種子文檔（文檔對象模型）。驗證RSS文檔的過程應分為以下兩個步驟：1)文檔的頭部遵循RSS格式。2）文檔可以轉換為DOM并解析成功。具體實施將在第3節(jié)中介紹。3設計概述3.1假設為了設計一個web信息挖掘系統(tǒng)的需求，我們了假設了一些虛擬環(huán)境。1.信息挖掘系統(tǒng)應存儲大量數(shù)據和臨時的大量文件。由于實驗儀器限制，我們不需要考慮存儲限制。2.由于帶寬的限制，我們設置了最長的響應時間，以確保系統(tǒng)可以持續(xù)正常運行。但是響應過長會降低訪問頻率。因此高的持續(xù)性帶寬比低延遲更重要。3.這個系統(tǒng)應該由幾個組件組成。由于這不是本文要解決的關鍵問題，所以我們沒有考慮容量和恢復的問題。3.2體系結構該信息挖掘系統(tǒng)由四個主要部分組成各種組件-爬蟲，信息挖掘機器、過濾器和下載器，如圖所示。每一個都是典型的ser-level服務器運行進程。在該系統(tǒng)中，爬行器是用來爬取各種物體的從html、xml、asp、jsp等網頁中提取的種子頁集。格式化爬蟲程序的輸出屬性的數(shù)量，URL，文本(抽象信息關于URL)。因為爬行器對我們來說不是必需的實驗設置，我們不介紹算法和爬蟲的實現(xiàn)。請注意,我們只解析超鏈接，而不是索引項，這將大大減緩應用程序的能耗。然后數(shù)據將被發(fā)送到挖掘機械加工，這得于其中的關鍵部件系統(tǒng)與過濾器的幫助。詳細的實現(xiàn)將在下一節(jié)中描述。最后Downloader負責下載網頁，根據信息挖掘機的列表，擦除元數(shù)據并存儲在服務器數(shù)據庫中。在為了提高性能，這意味著下載每秒數(shù)百甚至數(shù)千頁的設計集群的下載器是非常重要的。為系統(tǒng)靈活性考慮，數(shù)量的下載程序不是固定的。這意味著我們可以插入下載到我們需要適應的系統(tǒng)中，不同的實驗條件和應用合理工的作量。在下載之前,系統(tǒng)可以檢測下載的數(shù)量以及輸出列表中的項。系統(tǒng)保證信息挖掘的準確性，成功下載頁面文件后，下載程序再次檢查文件以確保它是有效的RSS提要。因為解析XML文件的所有工作都可以通過設置DOM來實現(xiàn)。所以我們可以判斷一個RSS文件以檢查它是否可以結構化的方式有效的DOM結構。同時，系統(tǒng)劃痕標題，鏈接，日期等元數(shù)據來自DOM接口和數(shù)據庫中的存儲。3.3信息挖掘機的信息信息挖掘機組件遍歷項目列在”link.txt”的文件流中，用c++實現(xiàn)。提取鏈接很方便，我們只需要用正則表達式。例如，RSS是一個特殊XML文件，一個XML應用程序，符合W3C的RDF規(guī)范，可通過XMLnamespace擴展和/或基于RDF的模塊化。所以我們定義以”.xml”結尾的正則表達式:Exp(RSS)={,(.*)(?=\.xml),}如果URL的格式符合正則表達式,信息挖掘機將其插入到潛在處理目標列表。那么這份清單就可以同時通過數(shù)據流發(fā)送到過濾器了。從經驗上看，執(zhí)行時間總是線性的增加，因為所有的工作都應該通過遍歷來完成整個文檔，并對其進行了詳細的分析。這里的挑戰(zhàn)是盡可能地避免遍歷和過度解析。因此在系統(tǒng)中，我們進行設計稱為Filter的組件用于與過濾協(xié)作信息挖掘機。在獲取隱藏的有價值的信息之前在非結構化網頁中，系統(tǒng)的過濾器會進行預檢查這些文檔向信息發(fā)送元數(shù)據，挖掘機器最有可能是一個RSS文件，這是不可能的。首先，過濾器下載文件到系統(tǒng)緩存中，每個頁面