これは、なにをしたくて書いたもの？
先日、AWS SAM＋LocalStackを使ってAmazon SNSの通知を受け取るAWS Lambda関数を書きました。
AWS SAM＋LocalStackで、Amazon SNSの通知を受け取るAWS Lambda関数をTypeScriptで書いてみる - CLOVER🍀
今度は、Amazon SQSを対象にしてみたいと思います。
Amazon SQSキューに送信されたメッセージを、AWS Lambda関数で処理する
Amazon SQSキューに送信されたメッセージをAWS Lambdaで受け取る場合の、基本的な話はこちらに書かれています。
AWS Lambda 関数 (コンソール) をトリガーするキューの設定 - Amazon Simple Queue Service
サンプルコードなどは、こちらに。
Amazon SQS での Lambda の使用 - AWS Lambda
AWS SAMでイベントのタイプにSQSを指定すると、Amazon SQSキューにメッセージが登録された時にAWS Lambda関数をトリガー
するように設定してくれるようです。
このイベントタイプが設定されていると、SAM は AWS::Lambda::EventSourceMapping リソースを生成します。
SQS - AWS Serverless Application Model
AWS SAMのドキュメントにも、AWS SAMテンプレートの例が記載されています。
AWS SAMAmazon SQS アプリケーションの テンプレート - AWS Lambda
今回も、AWS SAMを使って試してみましょう。
環境
今回の環境は、こちらです。
$ python3 --version Python 3.8.10 $ localstack --version 1.0.1
LocalStackを起動。
$ LAMBDA_EXECUTOR=docker-reuse localstack start
AWS CLIおよびAWS SAM CLI、加えてLocalStackの提供ツール。
$ awslocal --version aws-cli/2.7.18 Python/3.9.11 Linux/5.4.0-122-generic exe/x86_64.ubuntu.20 prompt/off $ samlocal --version SAM CLI, version 1.53.0
アプリケーションを作成する際に使用するNode.jsのバージョン。
$ node --version v16.16.0 $ npm --version 8.11.0
AWS Lambda関数としても、Node.js 16で動かすことにします。
AWS SAMプロジェクトを作成する
まずは、AWS SAMプロジェクトを作成します。ランタイムはNode.js 16、テンプレートはquick-start-sqsを選択します。
$ samlocal init --name sam-lambda-sqs --runtime nodejs16.x --app-template quick-start-sqs --package-type Zip --no-tracing
プロジェクト内に移動して
$ cd sam-lambda-sqs
構成を確認。
$ tree . ├── README.md ├── __tests__ │ └── unit │ └── handlers │ └── sqs-payload-logger.test.js ├── buildspec.yml ├── events │ └── event-sqs.json ├── package.json ├── src │ └── handlers │ └── sqs-payload-logger.js └── template.yaml 6 directories, 7 files
生成された各ファイルも見ておきます。
package.json
{ "name": "replaced-by-user-input", "description": "replaced-by-user-input", "version": "0.0.1", "private": true, "dependencies": {}, "devDependencies": { "jest": "^26.6.3" }, "scripts": { "test": "jest" } }
src/handlers/sqs-payload-logger.js
/** * A Lambda function that logs the payload received from SQS. */ exports.sqsPayloadLoggerHandler = async (event, context) => { // All log statements are written to CloudWatch by default. For more information, see // https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-logging.html console.info(JSON.stringify(event)); }
__tests__/unit/handlers/sqs-payload-logger.test.js
// Import all functions from sqs-payload-logger.js const sqsPayloadLogger = require('../../../src/handlers/sqs-payload-logger.js'); describe('Test for sqs-payload-logger', function () { // This test invokes the sqs-payload-logger Lambda function and verifies that the received payload is logged it('Verifies the payload is logged', async () => { // Mock console.log statements so we can verify them. For more information, see // https://jestjs.io/docs/en/mock-functions.html console.info = jest.fn() // Create a sample payload with SQS message format var payload = { DelaySeconds: 10, MessageAttributes: { "Sender": { DataType: "String", StringValue: "sqs-payload-logger" } }, MessageBody: "This message was sent by the sqs-payload-logger Lambda function", QueueUrl: "SQS_QUEUE_URL" } await sqsPayloadLogger.sqsPayloadLoggerHandler(payload, null) // Verify that console.info has been called with the expected payload expect(console.info).toHaveBeenCalledWith(JSON.stringify(payload)) }); });
events/event-sqs.json
{ "DelaySeconds": 10, "MessageAttributes": { "Sender": { "DataType": "String", "StringValue": "sqs-payload-logger" } }, "MessageBody": "This message was sent by the sqs-payload-logger Lambda function", "QueueUrl": "SQS_QUEUE_URL" }
template.yaml
# This is the SAM template that represents the architecture of your serverless application # https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template-basics.html # The AWSTemplateFormatVersion identifies the capabilities of the template # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/format-version-structure.html AWSTemplateFormatVersion: 2010-09-09 Description: >- sam-lambda-sqs # Transform section specifies one or more macros that AWS CloudFormation uses to process your template # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/transform-section-structure.html Transform: - AWS::Serverless-2016-10-31 # Resources declares the AWS resources that you want to include in the stack # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resources-section-structure.html Resources: # This is an SQS queue with all default configuration properties. To learn more about the available options, see # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sqs-queues.html SimpleQueue: Type: AWS::SQS::Queue # This is the Lambda function definition associated with the source code: sqs-payload-logger.js. For all available properties, see # https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction SQSPayloadLogger: Type: AWS::Serverless::Function Properties: Description: A Lambda function that logs the payload of messages sent to an associated SQS queue. Runtime: nodejs16.x Architectures: - x86_64 Handler: src/handlers/sqs-payload-logger.sqsPayloadLoggerHandler # This property associates this Lambda function with the SQS queue defined above, so that whenever the queue # receives a message, the Lambda function is invoked Events: SQSQueueEvent: Type: SQS Properties: Queue: !GetAtt SimpleQueue.Arn MemorySize: 128 Timeout: 25 # Chosen to be less than the default SQS Visibility Timeout of 30 seconds Policies: # Give Lambda basic execution Permission to the helloFromLambda - AWSLambdaBasicExecutionRole
buildspec.yml
version: 0.2 phases: install: commands: # Install all dependencies (including dependencies for running tests) - npm install pre_build: commands: # Discover and run unit tests in the '__tests__' directory - npm run test # Remove all unit tests to reduce the size of the package that will be ultimately uploaded to Lambda - rm -rf ./__tests__ # Remove all dependencies not needed for the Lambda deployment package (the packages from devDependencies in package.json) - npm prune --production build: commands: # Use AWS SAM to package the application by using AWS CloudFormation - aws cloudformation package --template template.yaml --s3-bucket $S3_BUCKET --output-template template-export.yml artifacts: type: zip files: - template-export.yml
TypeScriptプロジェクトに変更する
ここから、TypeScriptに変えていきます。
$ npm i -D typescript $ npm i -D -E prettier $ npm i -D @types/node@v16 @types/aws-lambda
TypeScriptの設定。
tsconfig.json
{ "compilerOptions": { "target": "esnext", "module": "commonjs", "lib": ["esnext"], "baseUrl": "./src", "outDir": "dist", "strict": true, "forceConsistentCasingInFileNames": true, "noFallthroughCasesInSwitch": true, "noImplicitOverride": true, "noImplicitReturns": true, "noPropertyAccessFromIndexSignature": true, "esModuleInterop": true }, "include": [ "src" ] }
tsconfig.typecheck.json
{ "extends": "./tsconfig", "compilerOptions": { "baseUrl": "./", "noEmit": true }, "include": [ "src", "__tests__" ] }
.prettierrc.json
{ "singleQuote": true }
デフォルトでpackage.jsonに記載されているJestは古いので、アンインストール。
$ npm uninstall jest
あらためて、Jestをインストール。
$ npm i -D jest @types/jest esbuild esbuild-jest
ここまでで、依存関係はこんな感じになりました。
"devDependencies": { "@types/aws-lambda": "^8.10.101", "@types/jest": "^28.1.6", "@types/node": "^16.11.45", "esbuild": "^0.14.50", "esbuild-jest": "^0.5.0", "jest": "^28.1.3", "prettier": "2.7.1", "typescript": "^4.7.4" },
jest.config.js
module.exports = { testEnvironment: 'node', transform: { "^.+\\.tsx?$": "esbuild-jest" } };
package.jsonのscripts要素は、このように。
"scripts": { "build": "tsc --project . && cp package*.json dist", "typecheck": "tsc --project ./tsconfig.typecheck.json", "typecheck:watch": "tsc --project ./tsconfig.typecheck.json --watch", "format": "prettier --write src __tests__", "test": "jest" }
Amazon SQSキューに送信されたメッセージを受け取る、AWS Lambda関数。
src/handlers/sqs-payload-logger.ts
import { Context, SQSEvent } from 'aws-lambda'; export const sqsPayloadLoggerHandler = async ( event: SQSEvent, context: Context ): Promise<void> => { console.info(JSON.stringify(event, null, 2)); };
受け取ったメッセージは、標準出力に書き出しているだけです。
テストコード。
__tests__/unit/handlers/sqs-payload-logger.test.ts
import { Context, SQSEvent } from 'aws-lambda'; import { sqsPayloadLoggerHandler } from '../../../src/handlers/sqs-payload-logger'; test('Test for sqs-payload-logger, Verifies the payload is logged', async () => { console.info = jest.fn(); var payload: SQSEvent = { Records: [ { messageId: '19dd0b57-b21e-4ac1-bd88-01bbb068cb78', receiptHandle: 'MessageReceiptHandle', body: 'Hello from SQS!', attributes: { ApproximateReceiveCount: '1', SentTimestamp: '1523232000000', SenderId: '123456789012', ApproximateFirstReceiveTimestamp: '1523232000001', }, messageAttributes: {}, md5OfBody: '7b270e59b47ff90a553787216d55d91d', eventSource: 'aws:sqs', eventSourceARN: 'arn:aws:sqs:us-east-1:123456789012:MyQueue', awsRegion: 'us-east-1', }, ], }; await sqsPayloadLoggerHandler(payload, {} as Context); expect(console.info).toHaveBeenCalledWith(JSON.stringify(payload, null, 2)); });
Amazon SNSの時もそうだったのですが、テストコードに含まれているpayloadのサンプルの構造がちょっと違います。
// Create a sample payload with SQS message format var payload = { DelaySeconds: 10, MessageAttributes: { "Sender": { DataType: "String", StringValue: "sqs-payload-logger" } }, MessageBody: "This message was sent by the sqs-payload-logger Lambda function", QueueUrl: "SQS_QUEUE_URL" }
なので、samlocal local generate-event sqs receive-messageで生成して、こちらを今回作成したテストコードに反映しています。
$ samlocal local generate-event sqs receive-message
{
"Records": [
{
"messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
"receiptHandle": "MessageReceiptHandle",
"body": "Hello from SQS!",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1523232000000",
"SenderId": "123456789012",
"ApproximateFirstReceiveTimestamp": "1523232000001"
},
"messageAttributes": {},
"md5OfBody": "7b270e59b47ff90a553787216d55d91d",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
"awsRegion": "us-east-1"
}
]
}
ソースコードは書いたので、もともと生成されていたJavaScriptコードは削除します。
$ rm src/handlers/sqs-payload-logger.js __tests__/unit/handlers/sqs-payload-logger.test.js
テストを実行して確認。
$ npm test > replaced-by-user-input@0.0.1 test > jest PASS __tests__/unit/handlers/sqs-payload-logger.test.ts ✓ Test for sqs-payload-logger, Verifies the payload is logged (4 ms) Test Suites: 1 passed, 1 total Tests: 1 passed, 1 total Snapshots: 0 total Time: 0.375 s, estimated 1 s Ran all test suites.
できました。
LocalStackにデプロイする
ソースコードができあがったので、LocalStackにデプロイします。
その前に、template.yamlを修正。CodeUriを追加して、Handlerを修正。
SQSPayloadLogger: Type: AWS::Serverless::Function Properties: Description: A Lambda function that logs the payload of messages sent to an associated SQS queue. Runtime: nodejs16.x Architectures: - x86_64 CodeUri: dist/ #Handler: src/handlers/sqs-payload-logger.sqsPayloadLoggerHandler Handler: sqs-payload-logger.sqsPayloadLoggerHandler
Transformも修正しておきます。ここを修正しないと、LocalStackにデプロイできません。
#Transform: #- AWS::Serverless-2016-10-31 Transform: AWS::Serverless-2016-10-31
デプロイしたAWS Lambda関数とAmazon SQSキューのARNは、Outputsに含めておくことにしました。
Outputs: SQSPayloadLogger: Description: "SSQS Payload Logger Handler Function ARN" Value: !GetAtt SQSPayloadLogger.Arn SimpleQueue: Description: "SQS Queue ARN" Value: !GetAtt SimpleQueue.Arn
では、ビルドして
$ npm run build
デプロイ。
$ yes | samlocal sync --stack-name $(uuidgen) --region us-east-1 --no-dependency-layer
完了。
2022-07-27 01:12:20 - Waiting for stack create/update to complete CloudFormation events from stack operations (refresh every 0.5 seconds) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ResourceStatus ResourceType LogicalResourceId ResourceStatusReason ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- CREATE_COMPLETE AWS::CloudFormation::Stack SimpleQueue - CREATE_COMPLETE AWS::CloudFormation::Stack SQSPayloadLoggerRole - UPDATE_COMPLETE AWS::CloudFormation::Stack SimpleQueue - CREATE_COMPLETE AWS::CloudFormation::Stack SQSPayloadLogger - UPDATE_COMPLETE AWS::CloudFormation::Stack SQSPayloadLoggerRole - CREATE_COMPLETE AWS::CloudFormation::Stack SQSPayloadLoggerSQSQueueEvent - CREATE_COMPLETE AWS::CloudFormation::Stack SQSPayloadLogger - CREATE_COMPLETE AWS::CloudFormation::Stack SQSPayloadLoggerSQSQueueEvent - CREATE_COMPLETE AWS::CloudFormation::Stack fa1f97ea-38d6-4c04-8b65-ac9ab037f5bf - ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- CloudFormation outputs from deployed stack --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Outputs --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Key SQSPayloadLogger Description SSQS Payload Logger Handler Function ARN Value arn:aws:lambda:us-east-1:000000000000:function:fa1f97ea-38d6-4c04-8b65-ac9ab037f5bf-SQSPayloadLogger-18a5f785 Key SimpleQueue Description SQS Queue ARN Value arn:aws:sqs:us-east-1:000000000000:fa1f97ea-38d6-4c04-8b65-ac9ab037f5bf-SimpleQueue-22824c0d --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Stack creation succeeded. Sync infra completed.
作成された、Amazon SQSキューを確認。
$ awslocal sqs list-queues
{
"QueueUrls": [
"http://localhost:4566/000000000000/fa1f97ea-38d6-4c04-8b65-ac9ab037f5bf-SimpleQueue-22824c0d"
]
}
キューのURLを取得して、変数に入れておきます。
$ QUEUE_URL=$(awslocal sqs list-queues --query 'QueueUrls[-1]' --output text)
メッセージの送信。
$ awslocal sqs send-message --queue-url $QUEUE_URL --message-body 'Hello World'
{
"MD5OfMessageBody": "b10a8db164e0754105b7a99be72e3fe5",
"MessageId": "8d21f972-7505-4ade-93db-4a1e0d9b3754"
}
ログは、今はなぜかDockerコンテナのログを見る必要があります…。
$ docker container logs -f localstack_main_lambda_arn_aws_lambda_us-east-1_000000000000_function_fa1f97ea-38d6-4c04-8b65-ac9ab037f5bf-SQSPayloadLogger-18a5f785
Lambda API listening on port 9001...
START RequestId: 88fc5928-11e8-1826-ead9-a66cefa08448 Version: $LATEST
2022-07-26T16:13:07.459Z 88fc5928-11e8-1826-ead9-a66cefa08448 INFO {
"Records": [
{
"body": "Hello World",
"receiptHandle": "YWU3MDIxOTgtNjc0Ni00ZWIxLWJjZDEtNzkzYjE4YjI1NGQ3IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6ZmExZjk3ZWEtMzhkNi00YzA0LThiNjUtYWM5YWIwMzdmNWJmLVNpbXBsZVF1ZXVlLTIyODI0YzBkIDhkMjFmOTcyLTc1MDUtNGFkZS05M2RiLTRhMWUwZDliMzc1NCAxNjU4ODUxOTg0LjQyMjI0MDc=",
"md5OfBody": "b10a8db164e0754105b7a99be72e3fe5",
"eventSourceARN": "arn:aws:sqs:us-east-1:000000000000:fa1f97ea-38d6-4c04-8b65-ac9ab037f5bf-SimpleQueue-22824c0d",
"eventSource": "aws:sqs",
"awsRegion": "us-east-1",
"messageId": "8d21f972-7505-4ade-93db-4a1e0d9b3754",
"attributes": {
"SenderId": "000000000000",
"SentTimestamp": "1658851984121",
"ApproximateReceiveCount": "1",
"ApproximateFirstReceiveTimestamp": "1658851984422"
},
"messageAttributes": {},
"md5OfMessageAttributes": null,
"sqs": true
}
]
}
END RequestId: 88fc5928-11e8-1826-ead9-a66cefa08448
REPORT RequestId: 88fc5928-11e8-1826-ead9-a66cefa08448 Init Duration: 1417.90 ms Duration: 9.12 ms Billed Duration: 10 ms Memory Size: 1536 MB Max Memory Used: 43 MB
メッセージを受け取って処理していることは、確認できました。
もう1通、メッセージを送ってみます。
$ awslocal sqs send-message --queue-url $QUEUE_URL --message-body 'Hello SQS'
{
"MD5OfMessageBody": "5cf7495f9a4dc80d179fe9f6f201b486",
"MessageId": "b7ce6cc9-fdcf-44f7-977b-054f608f2627"
}
結果。
START RequestId: 7c320d0a-abc7-18af-32cb-ae39a4a46421 Version: $LATEST
2022-07-26T16:14:50.584Z 7c320d0a-abc7-18af-32cb-ae39a4a46421 INFO {
"Records": [
{
"body": "Hello SQS",
"receiptHandle": "YjY0ZmY0NmYtNDNiZi00NGFhLThiZjUtYjg1NTczNjBkNDI3IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6ZmExZjk3ZWEtMzhkNi00YzA0LThiNjUtYWM5YWIwMzdmNWJmLVNpbXBsZVF1ZXVlLTIyODI0YzBkIGI3Y2U2Y2M5LWZkY2YtNDRmNy05NzdiLTA1NGY2MDhmMjYyNyAxNjU4ODUyMDkwLjU2MzAxODM=",
"md5OfBody": "5cf7495f9a4dc80d179fe9f6f201b486",
"eventSourceARN": "arn:aws:sqs:us-east-1:000000000000:fa1f97ea-38d6-4c04-8b65-ac9ab037f5bf-SimpleQueue-22824c0d",
"eventSource": "aws:sqs",
"awsRegion": "us-east-1",
"messageId": "b7ce6cc9-fdcf-44f7-977b-054f608f2627",
"attributes": {
"SenderId": "000000000000",
"SentTimestamp": "1658852089895",
"ApproximateReceiveCount": "1",
"ApproximateFirstReceiveTimestamp": "1658852090563"
},
"messageAttributes": {},
"md5OfMessageAttributes": null,
"sqs": true
}
]
}
END RequestId: 7c320d0a-abc7-18af-32cb-ae39a4a46421
REPORT RequestId: 7c320d0a-abc7-18af-32cb-ae39a4a46421 Duration: 3.71 ms Billed Duration: 4 ms Memory Size: 1536 MB Max Memory Used: 43 MB
OKですね。
最後に、template.yaml全体を載せておきます。
template.yaml
# This is the SAM template that represents the architecture of your serverless application # https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template-basics.html # The AWSTemplateFormatVersion identifies the capabilities of the template # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/format-version-structure.html AWSTemplateFormatVersion: 2010-09-09 Description: >- sam-lambda-sqs # Transform section specifies one or more macros that AWS CloudFormation uses to process your template # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/transform-section-structure.html #Transform: #- AWS::Serverless-2016-10-31 Transform: AWS::Serverless-2016-10-31 # Resources declares the AWS resources that you want to include in the stack # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resources-section-structure.html Resources: # This is an SQS queue with all default configuration properties. To learn more about the available options, see # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sqs-queues.html SimpleQueue: Type: AWS::SQS::Queue # This is the Lambda function definition associated with the source code: sqs-payload-logger.js. For all available properties, see # https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction SQSPayloadLogger: Type: AWS::Serverless::Function Properties: Description: A Lambda function that logs the payload of messages sent to an associated SQS queue. Runtime: nodejs16.x Architectures: - x86_64 CodeUri: dist/ #Handler: src/handlers/sqs-payload-logger.sqsPayloadLoggerHandler Handler: sqs-payload-logger.sqsPayloadLoggerHandler # This property associates this Lambda function with the SQS queue defined above, so that whenever the queue # receives a message, the Lambda function is invoked Events: SQSQueueEvent: Type: SQS Properties: Queue: !GetAtt SimpleQueue.Arn MemorySize: 128 Timeout: 25 # Chosen to be less than the default SQS Visibility Timeout of 30 seconds Policies: # Give Lambda basic execution Permission to the helloFromLambda - AWSLambdaBasicExecutionRole Outputs: SQSPayloadLogger: Description: "SSQS Payload Logger Handler Function ARN" Value: !GetAtt SQSPayloadLogger.Arn SimpleQueue: Description: "SQS Queue ARN" Value: !GetAtt SimpleQueue.Arn
まとめ
LocalStack上で、Amazon SQSキューに送信されたメッセージを受け取るAWS Lambda関数を作成してみました。
