無服務(wù)器架構(gòu)設(shè)計模式和最佳實踐

1、無服務(wù)器架構(gòu)設(shè)計模式和最佳實踐技術(shù)創(chuàng)新 變革未來AWS所提供的服務(wù)類型AWSLambdaAmazonKinesisAmazon S3Amazon APIGatewayAmazon SQSAmazon DynamoDBAWS IoTAmazon EMRAmazon ElastiCacheAmazon RDSAmazon RedshiftAmazon Elasticsearch ServiceServerlessAmazon EC2Microsoft SQL ServerAmazon CognitoAmazonCloudWatch開源服務(wù)自動化、自有服務(wù)開放化?，F(xiàn)在？無服務(wù)器化！“On EC2”M

2、anagedAWS Serverless 客戶S3 Bucket(s)Amazon AthenaKeyAWSManagement CloudTrail ServiceAmazon CognitoAWS IAMAmazon Kinesis StreamsAmazon Kinesis FirehoseAmazon QuickSightAWS GlueAmazon ESAmazonDynamoDBAmazon MacieAmazon API GatewayAWS IAMAmazon Redshift SpectrumAWS Direct Connect攝入Catalog & 搜索安全 & 審計API

3、/UI分析 & 處理AWS GlueAWSLambdaAWS Serverless Data LakeAWS Lambda客戶案例 - YublAWS Lambda 客戶案例 - YublA Lambda function that processes Kinesis events should also:retry failed events X times depending on processing timesend failed events to a DLQ after exhausting X retries/2017/04/aws-lambda-3-pro-tips-for-w

4、orking-with-kinesis-streams/設(shè)計模式: Lambda優(yōu)化Amazon API GatewayAmazon AlexaAWS IoTAmazon KinesisAmazon SNSAmazon SESAWS Step Functions213Invocations Functions InteractionsAmazon S3Amazon DynamoDBCustom endpointsAmazonAmazonElasticsearch CloudWatchEC2 instanceAWS Lambda 優(yōu)化Your functionLanguageruntimeExe

5、cution EnvironmentCompute substrateLambda函數(shù)無服務(wù)器的設(shè)計模式建立在Functions之上Functions是最小的部署和擴展單元按照請求來自動擴展，用戶無需為這些關(guān)注這些擴展的開銷無需為idle的請求付費跳過了最無聊的部分，也是越過了最難的部分AWS Lambda并發(fā)問題 - 流量激增問題增加您的并發(fā)執(zhí)行函數(shù)/zh_cn/lambda/latest/dg/concurrent-executions.htmlAWS Lambda 將根據(jù)增加的流量動態(tài)擴展容量，具體取決于您的AWS賬戶的并發(fā)執(zhí)行限制。為了處理突增流量，Lambda 將立即根據(jù)預(yù)定量增加您

6、的并發(fā)執(zhí)行函數(shù)。3000直到達(dá)到賬戶的安全限制， 或并發(fā)執(zhí)行的函數(shù)數(shù)量足 以成功處理增加的負(fù)載Lambda函數(shù)時間軸（Lambda不在VPC里面）Bootstrapthe runtimeStart yourcodeFull cold startPartial cold startWarm startDownloadyour codeStart newcontainerAWS optimizationYour optimizationAWS X-Ray跟蹤Lambda冷啟動時間Before the function handler code executes, there is an also

7、additional 200ms of “Initialization”thats the time it takes the function to load its dependencies and run code outside of the handler function.AWS X-Ray跟蹤Lambda冷啟動時間In the second “warm start” case, the function handler code starts almost immediately after the service receives the request to invoke t

8、he functionand no initialization occurs. Minimizing time spent in the Lambda service and avoiding function initialization on requests can have significant performance implications.Lambda函數(shù)代碼Avoid “fat”/monolithic functionsControl the dependencies in your functions deployment packageSeparate Lambda h

9、andler (entry point) from core logicOptimize for your languageNode Browserfy, MinifyLambda的注意事項和最佳實踐你的Lambda Function能在冰天 雪地中幸存下來么?(Code Start)要注意初始化AWS客戶端連接或 者數(shù)據(jù)庫連接的時候的變量scope， 盡可能的復(fù)用連接利用CloudWatch Events來做預(yù)熱ENIs for VPC將會在Code Start時 候被加載import sys import loggingimport rds_config import pymysqlrds

10、_host = rds-instance db_name = rds_config.db_name try:conn = pymysql.connect( except:logger.error(ERROR: def handler(event, context):with conn.cursor() as:cur僅在調(diào)用時被執(zhí) 行冷啟動時執(zhí)行流失處理針對敏感的延時問題 - Cold StartLambda是否使用VPC？Should my Lambda function be in a VPC?Does my function need to access any specific reso

11、urces in a VPC?Does it also need to access resources or services in the public internet?Dont put the function in a VPCPut the function in a private subnetPut the function in a subnet with a NATd route to the internetYesYesNoNoLambda配合VPC的時間軸（Lambda在VPC內(nèi)）Downloadyour codeStart newcontainerStart yourc

12、odeCreateAttachVPC ENI VPC ENIFull cold startWarm startBootstrapruntimePartial cold startAWS optimizationYour optimizationALWAYS configure a minimum of 2 Availability ZonesGive your Lambda functions their own subnetsGive your Lambda subnets a large IP rangeto handle potential scaleIf your functions

13、need to talk to a resource onthe internet, you need a NAT!Lambda的注意事項和最佳實踐 - Lambda in VPC了解重試機制同步模式不會重試異步模式重試兩次Stream-Base模式會一直重試使用Dead Letter Queues（私信隊列）使用SQS 或者 SNS 來重試記住: 重試次數(shù)也算一次調(diào)用Lambda“重試”機制Lambda的注意事項和最佳實踐文件系統(tǒng)如何考量?別忘記我們還有/tmp (512 MB 空間)exports.ffmpeg = function(event,context)new ffmpeg(./t

14、humb.MP4, function (err, video)if (!err) video.fnExtractFrameToJPG(/tmp) function (error, files) if (!error) console.log(files); context.done();.設(shè)計模式: 批處理批處理的特征大批的數(shù)據(jù)量定期的或者被計劃的任務(wù)Extract Transform Load (ETL)的工作模式大部分情況下不會長期運行大部分問題都能用MapReduce的模式來解決無服務(wù)器架構(gòu) 批處理AWS Lambda: SplitterAmazon S3 ObjectAmazon S3

15、:Mapper ResultsAWS Lambda: Mappers.AWS Lambda: ReducerAmazon S3 Results無服務(wù)器架構(gòu) 批處理AWS Lambda: SplitterAmazon S3 ObjectAmazon DynamoDB:Mapper ResultsAWS Lambda: Mappers.AWS Lambda: ReducerAmazon S3 ResultsAWS Lambda處理map-reduce類型工作Lambda Counting Function 0S3 BucketUsersLambda Counting Function 0Lambd

16、a Counting Function nSource DataSource DataLambda CountingFunction nSource DataSource DataLambda CascadeFunction 0Lambda CascadeFunction nNode.js Web AppRequest ResponseRequestResponseRequest ResponseRequest ResponseRequest ResponseRequest ResponseQuery responds with URL for the event stream. App in

17、vokes Lambda and stream resultServer-Sent Event streaming responseWeb UICascadeProcessQueryAWS中國（北京）區(qū)域由光環(huán)新網(wǎng)運營注意事項和最佳實踐串聯(lián)mapper functionsLambda批處理模式 vs. SQL (Hadoop / Spark生態(tài))處理數(shù)據(jù)和Lambda function的并發(fā)和限制需要取得平衡使用DynamoDB/ElastiCache/S3來管理mapper functions的狀態(tài)Lambda MapReduce Reference ArchitectureLambda已經(jīng)

18、集成了SQS事件觸發(fā)Lambda已經(jīng)集成了SQS事件觸發(fā)設(shè)計模式: 流式處理流式處理的注意事項高吞吐近乎實時的處理能力爆發(fā)性的間歇性流量消息持久化消息處理的保序無服務(wù)器架構(gòu)流式處理SensorsAmazon Kinesis:StreamLambda:Stream ProcessorS3:Final Aggregated OutputLambda: Periodic Dump to S3CloudWatch Events: Trigger every 5 minutesS3:Intermediate Aggregated DataLambda: Scheduled DispatcherKPL:P

19、roducerLambda流式處理并發(fā)問題事件源Amazon Kinesis目的地 1Lambda目的地 2ShardsScale Amazon Kinesis by splitting or merging shardsPolls a batch Waits for responseLambda 自動擴展一個 Kinesis Shard:If no record, wait some time (1s)Sub-batch in-memory and format records into Lambda payloadInvoke Lambda with synchronous invokeL

20、ambda blocks on ordered processing for each individual shardIncreasing # of shards with even distribution allows increased concurrencyBatch size may impact duration if the Lambda function takes longer to process more recordsSourceAmazon KinesisDestination 1ShardsScale Amazon Kinesis by splitting or

21、merging shardsDestination 2Lambda 自動擴展LambdaPolls a batWchaits for responseLambda流式處理并發(fā)問題數(shù)據(jù)源ShardsTrim horizonCheckpointLatestCheckpointLambda流式處理并發(fā)問題Event received by Lambda function is a collection of records from the stream: Records: kinesis: partitionKey: partitionKey-3,kinesisSchemaVersion: 1.0

22、,data: SGVsbG8sIHRoaXMgaXMgYSB0ZXN0IDEyMy4=,sequenceNumber: 49545115243490985018280067714973144582180062593244200961 ,eventSource: aws:kinesis,eventID: shardId-000000000000:49545115243490985018280067714973144582180062593244200961,invokeIdentityArn: arn:aws:iam:account-id:role/testLEBRole,eventVersio

23、n: 1.0, eventName: aws:kinesis:record,eventSourceARN: arn:aws:kinesis:us-west-2:35667example:stream/examplestream,awsRegion: us-west-2 Lambda流式處理并發(fā)問題Lambda流式處理并發(fā)問題Lambda流式處理Polls and blocks on synchronous invocation per shardIf put/ingestion rate is greater than the theoretical throughput, your proc

24、essing is at risk of falling behind Maximum theoretical throughput # shards * 2 MB / Lambda function duration (s)Effective theoretical throughput # shards * batch size (MB) / Lambda function duration (s)Retries 流失處理的重試機制Will retry on execution failures until the record is expiredThrottles and errors

25、 impact duration and directly impact throughputBest practiceRetry with exponential backoffUse case for service integrationRealtime clickstream logs to Kinesis streamReal-time Clickstream Anomaly Detection Reference:/cn/blogs/big-data/real-time-clickstream-anomaly-detection-with-amazon-kinesis-analyt

26、ics/扇出模式（fade-out）Kinesis shards的數(shù)量等同于Lambda的調(diào)用并發(fā)數(shù)量高吞吐和低延時 vs 消息保序SensorsAmazon Kinesis:StreamKPL:ProducerLambda: ProcessorsIncrease throughput, reduce processing latency如果Kinesis在單個shard處理的時候超過了這個限制 級的處理能力，就需要考慮使用Fade out的模式了。分發(fā)者 Lambda: Dispatcher缺點？ 保序性！最佳實踐調(diào)教Kinesis每次調(diào)用 Lambda觸發(fā)器batch size的 值，默認(rèn)

27、是100，更大的值 能夠使得Lambda有更少的 調(diào)用次數(shù)調(diào)教Lambda的內(nèi)存 更快的處理速度盡量使用KPL的Batch Message功能來批量發(fā)送數(shù) 據(jù)，這樣可以更充分的利用Kinesis的高吞吐特性Lambda architectureData SourcesServing LayerSpeed LayerBatch Layer模式4: 自動化處理自動化處理所具備的特征處理報警和各類事件周期性的計劃任務(wù)審計和通知對AWS服務(wù)功能性上的擴展高可用 + 自動擴展自動化處理: 動態(tài)為EC2分配DNSAWS Lambda: Update Route53Amazon CloudWatch Eve

28、nts: Rule TriggeredAmazon EC2 Instance State ChangesAmazon DynamoDB:EC2 Instance PropertiesAmazon Route53: Private Hosted ZoneTag:CNAME = A 34/awslabs/route53-dynamic-dns-with-lambda自動化處理: 從S3加載生成圖片縮略圖AWS Lambda:Resize ImagesUsers upload photosS3:Source BucketS3:Destination BucketTriggered on PUTs圖片

29、打水?。↙ambda + S3）/bjsdemo/Lambda-watermark-demo/Lambda_watermark_demo.zip圖片壓縮/裁剪（Lambda + S3）/bjsdemo/Lambda-CreateThumbnail-demo/CreateThumbnail.zip根據(jù)輸入?yún)?shù)線上裁剪圖片（API Gateway + Lambda + S3）/bjsdemo/Serverless-Image-Resizing-Demo/serverless-image- resizing-demo-Danrong.zip自動化處理 - 配合Lambda還有很多常用場景EC2實例定時開機、關(guān)機：/awslabs/ec2-scheduler/blob/master/code/ec2-scheduler.pyEBS定時打快照，并記錄打快照的是否成