金融領(lǐng)域大數(shù)據(jù)管理與數(shù)據(jù)質(zhì)量分析

大數(shù)據(jù)管理與數(shù)據(jù)質(zhì)量

-美國金融業(yè)中的對策處理速度容量限制數(shù)據(jù)質(zhì)量Overview數(shù)據(jù)<=Data=信息(并非數(shù)字集合)數(shù)據(jù)科學(xué)(約)=信息科學(xué)為何研究大數(shù)據(jù)?因為相關(guān)產(chǎn)品(如硬盤,

memory,CPU等)價格指數(shù)下降因為信息爆炸因為大數(shù)據(jù)導(dǎo)致許多新問題大數(shù)據(jù)研究是多學(xué)科的綜合(IT,DM,BI,BA,…)實業(yè)界對大數(shù)據(jù)問題的對策(見下文)1.數(shù)據(jù)庫策略1.1Database(DB)performance1.2DBspace1.1DBperformanceAuditing–2tables:asmallactive&ahugepassivePartitionIndex(good/bad;Cluster;Global/Local)Locktype(whenapplyrowlock)Transaction:1-phaseor2-phaseNormalizationInternaloptimization

(e.g.ExecutionPlan=>hintinOracle)Constraints(e.g.Check)usagetoreplacetriggerTricks(e.g.Datefunction;Searchsmalltablefirst;…)1.2DBspaceSpacearrangementforevendistribution

(e.g.1hugetableusesafewdatafiles)CleaningprocedurewithdefragmentPartitiondesignwithcleaningplan2.Applications(軟件)

(Javaexample)Usingadvancedlanguage(e.g.JavaorC#)2.1Memory（內(nèi)存）2.2Disk/networkspace2.3Performance2.4Maintainability2.1MemoryMinimizebigobjectscreationandcoexistenceGC(GarbageCollection)ornullbigobjectsonceoutofscopeChooseappropriateGCtypegc()TrytosplitonebigobjecttosmallobjectsUsemutableclassforfrequentlychangedbigobjects(e.g.StringBuilder,insteadofString)2.2Disk/networkspaceSmartcleanandarchiveprocesses

e.g.archivezippedoldornotusedfilestolowspeednetworkspaceanddeleteveryoldfilesfromthatspaceSmartloggingsettingse.g.log4jsizerollinge.g.AvoidduplicatedortriviallogginginfoMonitorforspaces2.3PerformanceAvoidredundanttreatment(inbigloops)

MaximizereuseMulti-threadingDBaccessingLogging--avoidslowoptions(e.g.line#)2.4MaintainabilitySOAprinciplesLosecoupling,reusability,granularity,modularity,composability,componentization,interoperability,…JEEpatterns(DAO,DTO,BizDelegation,…)Designpatterns(23)andMVCCreationStructureBehavior(e.g.Visitor)OOPprinciplesAbstraction,encapsulation,polymorphism,…Open/Close3.數(shù)據(jù)質(zhì)量控制3.1Business3.2ProcessA.Failover&DR(DisasterRecovery)B.QA(QualityAssurance)

(see<軟件質(zhì)量管理點滴>

fordetails)C.UAT(UserAcceptanceTest)3.3Technology3.1BusinessReducemanualwork;IncreaseautomationCompleteapprovalsystemformanualwork

E.g.1level=>2levelsor3levelsapprovalExtendviewpointstoconfirmdataqualityReduceredundancysystems(e.g.duetomerge,duetovendors)ScheduleCleansing(seedetails)EnhanceReconciliation(seedetails)BuildTrustlevel(seedetails)Trytocoverallrarecases3.1.ECleansingWhenAtsystemmergeAtmajorchangeHowDevelopdetectionapplicationsDelivermismatchreportstoIT&businessFindsolutionsonbothIT&business3.1.FReconciliationWhere1+subsystemshavedataforsamecontents.1+subsystemshaveindependentdatechangefunctionality.WhatRun&improverecon.app.routinely.Categorizereportsbyurgency.Analyzereports.DebugoradjustbizruleorapplyCleansing.3.1.GTrustlevelWhenAt1+fixeddatainputsInputsareindependentMustdecidefinaldetailsfrominputsHow(basedon)Providerlevel(foradetaileddatagroup)DatahistorySamples:Bloomberg,Reuter,Telekurs,DTCC,…;Moody,S&P,Fitch.3.2.AFailover&DRFailoverDB:2+atdiff.locations;real-timereplicationAppActive-Active:ClusterwithLoadBalancingActive-PassiveAuto(viaSAN)Manual+AutoDRDB:e.g.dailyorhourlyorreal-timereplicationApp:Manualswitch3.3TechnologyDBdesignConstraint‘Check’(forsensitivetablevalues)Normalization(toreduceduplicatio