DDB-introduce.pptx

1、馬進(jìn) 馬進(jìn)app ,DDB introduce,DDB outline,Applicationscenarios Technology decides System architecture System design Applications and Users,applicationscenarios,Applications Netease Blog, yixin, cloud music, cloud reader Database demands Scalability TB,PB level； Cluster database nodes: less than 1000； Avai

2、lability take over in one minute when data nodes failed take over in seconds when sql proxy failed Reliablility Data never lose,applicationscenarios,Operation types OLTP: mainly CRUD Some complex query: join, group, agg etc. Distributed transactions Application demands Usability Drived from java jdb

3、c, easy to use Sql proxy 100% compatible with mysql Fast iteration to meet app needs,Related tech.,Solutions Oracle RAC cost：commercial prods functions：too much for us Middleware No mature solutions NoSQL products Lack availability, reliability. Lack support for join, complex query and transactions,

4、Technology decides,Independent development DDB: Middleware based on MySQL core：data sharding on MySQL for availability. Derived from stardard jdbc for usability. Sql parser and planner in middleware, app has no concern for sharding, just like using MySQL. Optionly, app use connector/J and DDBProxy t

5、o acess DDB,DBI Architecture,DBN1 MySQL,sysdn MySQL,DBI Module（JDBC）,DBAdmin,App,isql,DBA,Master,DBN2 MySQL,DBN3 MySQL,Control flow DDL，metadata sync,Data flow DML，metadata flush,Proxy Architecture,DBN1 MySQL,MetaStore MySQL,DBI Module（JDBC）,DBAdmin,Decoder,Encoder,DBA,Master,DBN2 MySQL,DBN3 MySQL,C

6、ontrol flow DDL，metadata sync,QueryServer(DDBProxy),App,standard jdbc connector,MySQL Client,App,Data flow DML，metadata flush,Entire Architecture,Entire architecture,DBI(Query Server) Distributed SQL parser, planner. Node connection pool. Distributed transaction manager. Sql, node connection Statist

7、ics collecting. Master & SYSDB Metadata(table, column), Configuration management and notification Data migration, xa, dbn monitoring DBN Master and slave DDB master is response for failing over(notify switching dbn url),System design,Data Sharding Availability and reliability Distributed transaction

8、 processing online scale out and data migration Global incremental ID,Sharding,Two mapping layer: hash, policy Hash is immutable, policy is mutable Change policy to implement data migration,Availability and reliability,Atomic fail over Master keep heartbeat from each dbn Pick a slave to switch on db

9、n failing No data loss VSR from innosql Mater high availability Use zookeeper to pick one leader and auto take over,Distributed transaction,Distributed transaction processing core： 2-Phase-Commit implemention DBI or QS as transaction coordinator； Sql parser and planner, dectect if trasaction cross m

10、ultiple dbns Use mysql xa interfaces Dbi and qs have xa logs (mainly prepare events) What if : dbi or qs crashed or restarted ? Transaction recovery based on local xa logs What if ： dbi or qs is removed ? Master asked about all suspended xa transactions from each DBN at a interval For those suspende

11、d time xx min, auto commit and report to DBA,online scale out and data migration,online scale out and data migration Core one：full copy and incremental replication Core two : incremental replication based on binlog (tungsten and ddb filter) full copy and incremental rep both depend on policy changin

12、g We develop a single tool “hamal” to implement full copy and incremental rep,Global incremental ID,Support global auto_increment id MSB: master allocate batch ids, default:1000 easy to implement and comprehend Master dependent, insert could be blocked. Bad incremental quality, Most apps use this implemention TSB：based on local and master timestamp Complex to implement, take care when changing master clock No master dependent, support multiple masters (versions) in one cluster For those need higher availability or incremental quality (NOS),Apps and Users,Appications