CLOVERšŸ€

That was when it all began.

Node.jsć§ćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ļ¼ˆcluster态child_processļ¼‰ć€ćƒžćƒ«ćƒć‚¹ćƒ¬ćƒƒćƒ‰ļ¼ˆworker_threadsļ¼‰ć‚’ä½æć£ć¦ćæ悋

恓悌ćÆ态ćŖć«ć‚’ć—ćŸćć¦ę›øć„ćŸć‚‚ć®ļ¼Ÿ

Node.jsć§ćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ć€ćć‚Œć‹ć‚‰Node.js 10.5.0ä»„é™ć§ć‚ć‚Œć°ćƒžćƒ«ćƒć‚¹ćƒ¬ćƒƒćƒ‰ćŒä½æ恈悋悈恆ćŖć®ć§ć€čˆˆå‘³ćŒć¦ć‚‰č©¦ć—ć¦ćæ悋恓ćØ恫
ć—ć¾ć—ćŸć€‚

Node.jsć§ćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ć€ćƒžćƒ«ćƒć‚¹ćƒ¬ćƒƒćƒ‰

Node.jsć§ćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ć‚’ę‰±ć†ć«ćÆ态clusterćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć¾ćŸćÆchild_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æ恆悈恆恧恙怂

Cluster | Node.js v18.15.0 Documentation

Child process | Node.js v18.15.0 Documentation

clusterćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ćÆć€å†…éƒØēš„恫ćÆchild_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æć£ć¦ć„ć‚‹ć‚ˆć†ć§ć™ć€‚

The worker processes are spawned using the child_process.fork() method, so that they can communicate with the parent via IPC and pass server handles back and forth.

Cluster / How it works

ćƒžćƒ«ćƒć‚¹ćƒ¬ćƒƒćƒ‰ć‚’ę‰±ć†ć«ćÆ态worker_threadsćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æć†ć‚ˆć†ć§ć™ć€‚ć“ć”ć‚‰ćÆ态Node.js 10.5.0ć‹ć‚‰åˆ©ē”ØåÆčƒ½ć«ćŖć£ćŸć‚‚ć®ć§ć™ć­ć€‚

Worker threads | Node.js v18.15.0 Documentation

child_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«

ęœ€åˆćÆćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ć‹ć‚‰ć€ćć—ć¦clusterćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ćÆchild_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ćƒ™ćƒ¼ć‚¹ć«ć—ć¦ć„ć‚‹ć‚ˆć†ćŖ恮恧态
ć¾ćšćÆchild_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’č¦‹ć¦ćæć¾ć—ć‚‡ć†ć€‚

child_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ćÆć€å­ćƒ—ćƒ­ć‚»ć‚¹ć‚’ē”Ÿęˆć™ć‚‹ćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć§ć™ć€‚

The node:child_process module provides the ability to spawn subprocesses in a manner that is similar, but not identical, to popen(3). This capability is primarily provided by the child_process.spawn() function:

Child process

ä½æć†ćƒ”ć‚½ćƒƒćƒ‰ćÆć„ćć¤ć‹ć‚ć‚Šć¾ć™ćŒć€ć–ć£ćć‚Šä»„äø‹ć®3恤恮ē³»ēµ±ć«åˆ†ć‹ć‚Œć¾ć™ć€‚

  • spawn ā€¦ ęŒ‡å®šć•ć‚ŒćŸć‚³ćƒžćƒ³ćƒ‰ć‚’å®Ÿč”Œć™ć‚‹ć€‚å¼•ę•°ćÆé…åˆ—ć§ęŒ‡å®š
  • exec ā€¦ OSć®ć‚·ć‚§ćƒ«ć‚’ä»‹ć—ć¦ęŒ‡å®šć•ć‚ŒćŸć‚³ćƒžćƒ³ćƒ‰ć‚’å®Ÿč”Œć™ć‚‹ć€‚å¼•ę•°ćÆć‚¹ćƒšćƒ¼ć‚¹åŒŗåˆ‡ć‚Šć§ęŒ‡å®š
  • fork ā€¦ å®Ÿč”Œć™ć‚‹ćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ęŒ‡å®šć—ć¦ć€ę–°ć—ć„Node.jsćƒ—ćƒ­ć‚»ć‚¹ć‚’ē”Ÿęˆć™ć‚‹ć€‚č¦Ŗćƒ—ćƒ­ć‚»ć‚¹ćØå­ćƒ—ćƒ­ć‚»ć‚¹ć®é–“ćÆIPCļ¼ˆćƒ—ćƒ­ć‚»ć‚¹é–“é€šäæ”ļ¼‰ćŒć§ćć‚‹ęŗ–å‚™ćŒę•“ćˆć‚‰ć‚Œć¦ć„ć‚‹

äøŠčؘćÆć‚µćƒ–ćƒ—ćƒ­ć‚»ć‚¹ćŒéžåŒęœŸć«å®Ÿč”Œć•ć‚Œć‚‹ć®ć§ć™ćŒć€spawnćØexec恫恤恄恦ćÆåŒęœŸē‰ˆć‚‚ć‚ć‚Šć¾ć™ć€‚

forkć§ä½œęˆć—ćŸå­ćƒ—ćƒ­ć‚»ć‚¹ćÆ态IPCļ¼ˆInternal Process Communicationć€ćƒ—ćƒ­ć‚»ć‚¹é–“é€šäæ”ļ¼‰ć§ćƒ”ćƒƒć‚»ćƒ¼ć‚ø悒送äæ”ć—åˆć†ć“ćØćŒć§ćć¾ć™ć€‚

Child process / subprocess.send(message[, sendHandle[, options]][, callback])

ćƒ”ćƒƒć‚»ćƒ¼ć‚øćÆć‚·ćƒŖć‚¢ćƒ©ć‚¤ć‚ŗ恗恦送äæ”恙悋恓ćØ恫ćŖć‚Šć¾ć™ć€‚ćƒ‡ćƒ•ć‚©ćƒ«ćƒˆć§ćÆJSONåž‹å¼ć®ę–‡å­—åˆ—ć§ć‚·ćƒŖć‚¢ćƒ©ć‚¤ć‚ŗć—ć¾ć™ć€‚

serialization Specify the kind of serialization used for sending messages between processes. Possible values are 'json' and 'advanced'. See Advanced serialization for more details. Default: 'json'.

Child process / child_process.fork(modulePath[, args][, options])

ć¾ćŸć€ć”ć‚‡ć£ćØå¤‰ć‚ć£ćŸć‚±ćƒ¼ć‚¹ćØ恗恦TCP恮Serverć‚Ŗ惖ć‚ø悧ć‚Æ惈悄Socketć‚Ŗ惖ć‚ø悧ć‚Æ惈悒ē›“ꎄęø”恙恓ćØć‚‚ć§ćć¾ć™ć€‚

ćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ćŖTCPć‚µćƒ¼ćƒćƒ¼ć‚’ä½œć‚‹ē”Øé€”ćŒę„č­˜ć•ć‚Œć¦ćć†ćŖꄟ恘恧恙恭怂

clusterćƒ¢ć‚øćƒ„ćƒ¼ćƒ«

clusterćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ćÆ态child_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć®fork悒ä½æć£ć¦å®Ÿē¾ć•ć‚Œć‚‹ć‚‚ć®ć§ć™ć€‚ć¤ć¾ć‚Šć€ę–°ć—ć„Node.jsćƒ—ćƒ­ć‚»ć‚¹ć‚’ē”Ÿęˆć™ć‚‹ć“ćØ恫
ćŖć‚Šć¾ć™ć€‚

The worker processes are spawned using the child_process.fork() method, so that they can communicate with the parent via IPC and pass server handles back and forth.

Cluster / How it works

ć‚ˆć£ć¦IPC悒ä½æć£ć¦å­ćƒ—ćƒ­ć‚»ć‚¹ćØć‚„ć‚Šå–ć‚Šć™ć‚‹ć“ćØćŒć§ćć¾ć™ć€‚

仄äø‹ć®ć‚ˆć†ć«ć€ćƒćƒƒćƒˆćƒÆćƒ¼ć‚Æꎄē¶šć‚’å—ć‘ä»˜ć‘ć‚‹ć‚ˆć†ćŖć‚¢ćƒ—ćƒŖć‚±ćƒ¼ć‚·ćƒ§ćƒ³ć§ä½æ悏悌悋恓ćØć‚’ę„č­˜ć—ć¦ć„ć¾ć™ć­ć€‚

The cluster module supports two methods of distributing incoming connections.

The first one (and the default one on all platforms except Windows) is the round-robin approach, where the primary process listens on a port, accepts new connections and distributes them across the workers in a round-robin fashion, with some built-in smarts to avoid overloading a worker process.

Although a primary use case for the node:cluster module is networking, it can also be used for other use cases requiring worker processes.

ē”Ÿęˆć•ć‚ŒćŸćƒ—ćƒ­ć‚»ć‚¹č‡Ŗä½“ć‚’ē®”ē†ć™ć‚‹ć®ćÆć‚¢ćƒ—ćƒŖć‚±ćƒ¼ć‚·ćƒ§ćƒ³å“ć®å½¹å‰²ć§ć€ć‚µćƒ–ćƒ—ćƒ­ć‚»ć‚¹ć®ę•°ć®ē®”ē†ć‚’Node.jsć§č”Œć†ć‚ˆć†ćŖ恓ćØćÆć‚ć‚Šć¾ć›ć‚“ć€‚

Node.js does not automatically manage the number of workers, however. It is the application's responsibility to manage the worker pool based on its own needs.

ć¾ćŸć€ć‚µćƒ³ćƒ—ćƒ«ć«ę›øć‹ć‚Œć¦ć„ć‚‹ć‚ˆć†ć«ć‚µćƒ¼ćƒćƒ¼ćŒåˆ©ē”Øć™ć‚‹ćƒćƒ¼ćƒˆć‚’ćƒ—ćƒ­ć‚»ć‚¹é–“ć§ē°”å˜ć«å…±ęœ‰ć™ć‚‹ć“ćØćŒć§ćć¾ć™ć€‚

import cluster from 'node:cluster';
import http from 'node:http';
import { availableParallelism } from 'node:os';
import process from 'node:process';

const numCPUs = availableParallelism();

if (cluster.isPrimary) {
  console.log(`Primary ${process.pid} is running`);

  // Fork workers.
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  cluster.on('exit', (worker, code, signal) => {
    console.log(`worker ${worker.process.pid} died`);
  });
} else {
  // Workers can share any TCP connection
  // In this case it is an HTTP server
  http.createServer((req, res) => {
    res.writeHead(200);
    res.end('hello world\n');
  }).listen(8000);

  console.log(`Worker ${process.pid} started`);
}

恓悓ćŖę„Ÿć˜ć§å­ćƒ—ćƒ­ć‚»ć‚¹å“ć§åŒć˜ćƒćƒ¼ćƒˆć‚’ćƒŖćƒƒć‚¹ćƒ³ć™ć‚‹Serverć‚’ä½œęˆć—ć¦ć„ć‚‹ć®ć§ć™ćŒć€ć“ć‚Œć§åŒć˜ćƒćƒ¼ćƒˆć‚’ćƒŖćƒƒć‚¹ćƒ³ć™ć‚‹č¤‡ę•°ćƒ—ćƒ­ć‚»ć‚¹ć®
Node.jsć‚¢ćƒ—ćƒŖć‚±ćƒ¼ć‚·ćƒ§ćƒ³ćŒå‹•ä½œć™ć‚‹ć‚“ć§ć™ć‚ˆć­ć€‚

  http.createServer((req, res) => {
    res.writeHead(200);
    res.end('hello world\n');
  }).listen(8000);
worker_threadsćƒ¢ć‚øćƒ„ćƒ¼ćƒ«

Node.jsć§ćƒžćƒ«ćƒć‚¹ćƒ¬ćƒƒćƒ‰ć‚’ę‰±ć†ć«ćÆ态worker_threadsćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æć„ć¾ć™ć€‚

Worker threads | Node.js v18.15.0 Documentation

ćƒÆćƒ¼ć‚«ćƒ¼ć‚¹ćƒ¬ćƒƒćƒ‰ćÆ态CPUćƒć‚¦ćƒ³ćƒ‰ćŖ処ē†ć‚’å®Ÿč”Œć™ć‚‹ć®ć«å½¹ē«‹ć”态IOćƒć‚¦ćƒ³ćƒ‰ćŖ処ē†ć«ćÆåŠ¹ęžœēš„恧ćÆćŖ恄ćØę›øć‹ć‚Œć¦ć„ć¾ć™ć€‚IOę“ä½œć«
恤恄恦ćÆ态Node.jsć®éžåŒęœŸIOć®ę–¹ćŒćƒÆćƒ¼ć‚«ćƒ¼ć‚¹ćƒ¬ćƒƒćƒ‰ć‚’ä½æć†ć‚ˆć‚Šć‚‚åŠ¹ēŽ‡ēš„ć ćć†ć§ć™ć€‚

Workers (threads) are useful for performing CPU-intensive JavaScript operations. They do not help much with I/O-intensive work. The Node.js built-in asynchronous I/O operations are more efficient than Workers can be.

Worker threads

ć¾ćŸclusterćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚„child_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ćØē•°ćŖ悊态worker_threadsćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć§ćÆćƒÆćƒ¼ć‚«ćƒ¼ć‚¹ćƒ¬ćƒƒćƒ‰ćØćƒ”ćƒ¢ćƒŖć‚’å…±ęœ‰ć§ćć¾ć™ć€‚

Unlike child_process or cluster, worker_threads can share memory. They do so by transferring ArrayBuffer instances or sharing SharedArrayBuffer instances.

ćŖćŠć€ć‚¹ćƒ¬ćƒƒćƒ‰ćƒ—ćƒ¼ćƒ«ć®ę©Ÿčƒ½ćÆć‚ć‚Šć¾ć›ć‚“ć€‚AsyncResource悒ä½æć£ć¦å®Ÿč£…ć—ć¦ć„ć‚‹ć‚µćƒ³ćƒ—ćƒ«ćŒć‚ć‚‹ć‚ˆć€ćØę›øć‹ć‚Œć¦ć„ć¾ć™ć€‚

When implementing a worker pool, use the AsyncResource API to inform diagnostic tools (e.g. to provide asynchronous stack traces) about the correlation between tasks and their outcomes. See "Using AsyncResource for a Worker thread pool" in the async_hooks documentation for an example implementation.

Worker threads

恓恔悉恧恙恭怂

Asynchronous context tracking / Class: AsyncResource / Using AsyncResource for a Worker thread pool

要恙悋恫č‡Ŗåˆ†ć§ä½œć£ć¦ć­ć€ćØā€¦ć€‚

ćŠé”Œ

ćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ć‚’ä½æć£ć¦ć‚‚ć€ćƒžćƒ«ćƒć‚¹ćƒ¬ćƒƒćƒ‰ć‚’ä½æć£ć¦ć‚‚ć€CPUę•°ć§ć‚¹ć‚±ćƒ¼ćƒ«ć™ć‚‹ć“ćØ悒ē¢ŗčŖć§ćć‚Œć°ć‚ˆć„ć®ć§ć€ä»Šå›žćÆćƒ•ć‚£ćƒœćƒŠćƒƒćƒę•°ć®
č؈ē®—ć‚’č”Œć†HTTPć‚µćƒ¼ćƒćƒ¼ć‚’ćć‚Œćžć‚Œć®ćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æć£ć¦č©¦ć—ć¦ćæ恟恄ćØę€ć„ć¾ć™ć€‚

HTTPć‚µćƒ¼ćƒćƒ¼č‡Ŗ体ćÆ态恓恔悉悒ä½æć£ć¦ä½œęˆć—ć¾ć™ć€‚

HTTP | Node.js v18.15.0 Documentation

ć¾ćŸć€ćƒ—ćƒ­ć‚°ćƒ©ćƒ ćÆTypeScript恧ę›ø恏恓ćØć«ć—ć¾ć™ć€‚

ē’°å¢ƒ

ä»Šå›žć®ē’°å¢ƒćÆ态恓恔悉怂

$ node --version
v18.15.0


$ npm --version
9.5.0

ęŗ–å‚™

Node.jsļ¼‹TypeScript恮ē’°å¢ƒć‚’ä½œęˆć—ć¾ć™ć€‚

$ npm init -y
$ npm i -D typescript
$ npm i -D @types/node@v18
$ npm i -D prettier
$ mkdir src

依存関äæ‚ćÆ恓悓ćŖę„Ÿć˜ć«ćŖć‚Šć¾ć—ćŸć€‚

  "devDependencies": {
    "@types/node": "^18.15.3",
    "prettier": "^2.8.5",
    "typescript": "^5.0.2"
  }

scripts怂

  "scripts": {
    "build": "tsc --project .",
    "build:watch": "tsc --project . --watch",
    "format": "prettier --write src"
  },

čØ­å®šćƒ•ć‚”ć‚¤ćƒ«ć€‚

tsconfig.json

{
  "compilerOptions": {
    "target": "esnext",
    "module": "commonjs",
    "moduleResolution": "node",
    "lib": ["esnext"],
    "baseUrl": "./src",
    "outDir": "dist",
    "strict": true,
    "forceConsistentCasingInFileNames": true,
    "noFallthroughCasesInSwitch": true,
    "noImplicitOverride": true,
    "noImplicitReturns": true,
    "noPropertyAccessFromIndexSignature": true,
    "esModuleInterop": true
  },
  "include": [
    "src"
  ]
}

.prettierrc.json

{
  "singleQuote": true,
  "printWidth": 120
}

ć¾ćšćÆć‚·ćƒ³ćƒ—ćƒ«ć«

ć¾ćšćÆćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ć‚‚ćƒžćƒ«ćƒć‚¹ćƒ¬ćƒƒćƒ‰ć‚‚ä½æ悏ćŖć„ćƒćƒ¼ć‚øćƒ§ćƒ³ć‚’ä½œęˆć—ć¾ć™ć€‚

ć“ć®å¾Œć€ēØ®ć€…ć®ćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æć£ć¦ä½œęˆć™ć‚‹ćƒ—ćƒ­ć‚°ćƒ©ćƒ ć§ć‚‚ę‰±ć„ć‚„ć™ć„ć‚ˆć†ć«ć€ć„ćć¤ć‹ć®ćƒ•ć‚”ć‚¤ćƒ«ć«åˆ†å‰²ć—ć¦ę›øć„ć¦ć„ćć¾ć™ć€‚

ćƒ•ć‚£ćƒœćƒŠćƒƒćƒę•°ć®č؈ē®—ćÆ态http://localhost:8000?num=[ę•“ę•°]ć®ć‚ˆć†ćŖęŒ‡å®šć§č”Œć†ć‚‚ć®ćØć—ć¾ć™ć€‚

ćƒ•ć‚£ćƒœćƒŠćƒƒćƒę•°ć®č؈ē®—ć‚’č”Œć†é–¢ę•°ć€‚

src/fib.ts

export function fib(n: number): number {
  if (n === 0) {
    return 0;
  } else if (n === 1) {
    return 1;
  } else {
    return fib(n - 1) + fib(n - 2);
  }
}

QueryString悒Mapć«åˆ†č§£ć—ćŸć‚Šć€ćć®å†…å®¹ć‚’fibé–¢ę•°ć«ęø”ć™ćŸć‚ć®é–¢ę•°ć€‚

src/handler.ts

import { fib } from './fib';

export function fibRequestHandler(params: Map<string, string>): number | undefined {
  let result;

  if (params.has('num')) {
    result = fib(parseInt(params.get('num')!, 10));
  } else {
    result = undefined;
  }

  return result;
}

export function queryToMap(query: string | undefined): Map<string, string> {
  const map = new Map<string, string>();

  if (!query) {
    return map;
  }

  for (const pair of query.split('&')) {
    const [name, value] = pair.split('=');
    map.set(name, value);
  }

  return map;
}

惭悰å‡ŗåŠ›ć€‚

src/log.ts

export function log(message: string): void {
  console.log(`[${new Date().toISOString()}] ${message}`);
}

ć‚·ćƒ£ćƒƒćƒˆćƒ€ć‚¦ćƒ³ć€‚

src/shutdown.ts

import http from 'node:http';
import { log } from './log';

export function registerShutdown(httpServer: http.Server): void {
  process.on('SIGTERM', signalHandler(httpServer));
  process.on('SIGINT', signalHandler(httpServer));
}

function signalHandler(httpServer: http.Server): (signal: NodeJS.Signals) => void {
  return (signal) => {
    httpServer.close(() => log(`shutdown, signal: ${signal}`));
  };
}

ć“ć“ć¾ć§ćŒåŸŗęœ¬ēš„ćŖéƒØå“ć§ć™ć­ć€‚

恧ćÆć€ć‚µćƒ¼ćƒćƒ¼éƒØåˆ†ć‚’ę›øćć¾ć™ć€‚

src/simple-http-server.ts

import http from 'node:http';
import { fibRequestHandler, queryToMap } from './handler';
import { log } from './log';
import { registerShutdown } from './shutdown';

const httpServer = http.createServer((req: http.IncomingMessage, res: http.ServerResponse) => {
  const query = req.url?.split('?')[1];
  const params = queryToMap(query);

  log(`accept request, pid = ${process.pid}`);

  const result = fibRequestHandler(params);

  res.writeHead(200, { 'Content-Type': 'application/json' });
  res.end(
    JSON.stringify({
      result,
      url: req.url,
      pid: process.pid,
    })
  );
});

httpServer.listen(
  {
    host: '0.0.0.0',
    port: 8000,
  },
  () => registerShutdown(httpServer)
);

log(`http-server, pid: ${process.pid}`);

ćƒ“ćƒ«ćƒ‰ć—ć¦

$ npm run build

čµ·å‹•ć€‚

$ node dist/simple-http-server.js
[2023-03-21T13:59:12.290Z] http-server, pid: 27141

ē¢ŗčŖć€‚

$ curl localhost:8000?num=10
{"result":55,"url":"/?num=10","pid":27141}

恓悓ćŖꄟ恘恧čæ”ć£ć¦ćć¾ć™ć€‚

ę‰‹å…ƒć®ē’°å¢ƒć§ćÆ态42ćć‚‰ć„ćŒć”ć‚‡ć†ć©ć„ć„ę™‚é–“ćŒć‹ć‹ć‚Šć¾ć—ćŸć€‚

$ time curl localhost:8000?num=42
{"result":267914296,"url":"/?num=42","pid":27141}
real    0m3.698s
user    0m0.009s
sys     0m0.003s

恓恮Ꙃ态mpstat恧見悋ćØ

$ mpstat -P ALL 1

CPUć‚’ć²ćØ恤ä½æć„åˆ‡ć£ć¦ć„ć‚‹ć“ćØćŒć‚ć‹ć‚Šć¾ć™ć€‚

23Ꙃ02分08ē§’  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
23Ꙃ02分09ē§’  all   13.32    0.00    0.00    0.13    0.00    0.00    0.00    0.00    0.00   86.56
23Ꙃ02分09ē§’    0    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00
23Ꙃ02分09ē§’    1  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
23Ꙃ02分09ē§’    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
23Ꙃ02分09ē§’    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
23Ꙃ02分09ē§’    4    3.03    0.00    0.00    1.01    0.00    0.00    0.00    0.00    0.00   95.96
23Ꙃ02分09ē§’    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
23Ꙃ02分09ē§’    6    1.98    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.02
23Ꙃ02分09ē§’    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

恧ćÆć€ć“ć‚Œć‚’é€£ē¶šć§4å›žå®Ÿč”Œć—ć¦ćæć¾ć™ć€‚

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’å—ć‘ä»˜ć‘ćŸę™‚ć®ćƒ­ć‚°ć€‚

[2023-03-21T14:01:04.742Z] accept request, pid = 27141
[2023-03-21T14:02:06.498Z] accept request, pid = 27141
[2023-03-21T14:03:17.756Z] accept request, pid = 27141
[2023-03-21T14:03:21.562Z] accept request, pid = 27141

ēµęžœć€‚

$ {"result":267914296,"url":"/?num=42","pid":27141}
real    0m3.814s
user    0m0.004s
sys     0m0.005s


{"result":267914296,"url":"/?num=42","pid":27141}
real    0m7.285s
user    0m0.000s
sys     0m0.006s


{"result":267914296,"url":"/?num=42","pid":27141}
real    0m10.876s
user    0m0.007s
sys     0m0.000s


{"result":267914296,"url":"/?num=42","pid":27141}
real    0m15.284s
user    0m0.004s
sys     0m0.004s

ć»ć¼ē·šå½¢ć«é…恏ćŖć£ć¦ć„ć¾ć™ć€‚

ä½æ恈悋CPUć‚³ć‚¢ćÆć²ćØ恤恠恋悉恧恙恭怂

23Ꙃ04分25ē§’  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
23Ꙃ04分26ē§’  all   12.96    0.00    0.13    0.00    0.00    0.00    0.00    0.00    0.00   86.92
23Ꙃ04分26ē§’    0  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
23Ꙃ04分26ē§’    1    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00
23Ꙃ04分26ē§’    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
23Ꙃ04分26ē§’    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
23Ꙃ04分26ē§’    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
23Ꙃ04分26ē§’    5    1.01    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.99
23Ꙃ04分26ē§’    6    1.01    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.99
23Ꙃ04分26ē§’    7    0.00    0.00    0.99    0.00    0.00    0.00    0.00    0.00    0.00   99.01

Ctrl-cć§åœę­¢ć€‚

[2023-03-21T14:05:04.163Z] shutdown, signal: SIGINT

ć“ć‚Œć‚’ć€ćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ć‚„ćƒžćƒ«ćƒć‚¹ćƒ¬ćƒƒćƒ‰ć‚’ä½æć£ć¦å¤‰ćˆć¦ć„ćć¾ć—ć‚‡ć†ć€‚

ć¾ćŸć€ä»„é™ćÆTypeScriptć‚½ćƒ¼ć‚¹ć‚³ćƒ¼ćƒ‰ć‚’ćƒ“ćƒ«ćƒ‰ć™ć‚‹ć‚¹ćƒ†ćƒƒćƒ—ćÆēœē•„ć—ć¾ć™ć€‚

clusterćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć§ę›ø恏

ęœ€åˆćÆclusterćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æć£ć¦ę›ø恄恦ćæć¾ć™ć€‚

src/prefork-http-server.ts

import cluster from 'node:cluster';
import http from 'node:http';
import { fibRequestHandler, queryToMap } from './handler';
import { log } from './log';
import { registerShutdown } from './shutdown';

const processes = process.argv[2] !== undefined ? parseInt(process.argv[2], 10) : 4;

if (cluster.isPrimary) {
  // main process
  for (let i = 0; i < processes; i++) {
    const subProcess = cluster.fork();
    log(`fork process, pid: ${process.pid}, sub-process pid: ${subProcess.process.pid}`);
  }

  log(`prefork, ${processes} processes`);
} else {
  // sub process
  const httpServer = http.createServer((req: http.IncomingMessage, res: http.ServerResponse) => {
    const query = req.url?.split('?')[1];
    const params = queryToMap(query);

    log(`sub-process accept request, pid = ${process.pid}`);

    const result = fibRequestHandler(params);

    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(
      JSON.stringify({
        result,
        url: req.url,
        pid: process.pid,
      })
    );
  });

  httpServer.listen({
    host: '0.0.0.0',
    port: 8000,
  }, () => registerShutdown(httpServer));

  log(`start prefork http-server, pid: ${process.pid}`);
}

čµ·å‹•ć™ć‚‹ćƒ—ćƒ­ć‚»ć‚¹ę•°ćÆ态4ć¾ćŸćÆčµ·å‹•å¼•ę•°ć§ęŒ‡å®šć—ćŸę•°ć«ć—ć¾ć—ćŸć€‚

const processes = process.argv[2] !== undefined ? parseInt(process.argv[2], 10) : 4;

ęŒ‡å®šę•°ć®ćƒ—ćƒ­ć‚»ć‚¹ć‚’ć€ęœ€åˆć«fork怂

  for (let i = 0; i < processes; i++) {
    const subProcess = cluster.fork();
    log(`fork process, pid: ${process.pid}, sub-process pid: ${subProcess.process.pid}`);
  }

fork恙悋ćØ态Node.jsć‚’čµ·å‹•ć—ćŸę™‚ćØåŒć˜å‡¦ē†ćŒå‘¼ć³å‡ŗ恕悌悋悈恆恧ļ¼ˆforkå‘¼ć³å‡ŗć—ćŸć‚½ćƒ¼ć‚¹ć‚³ćƒ¼ćƒ‰ć§ćÆćŖ恏ļ¼‰ć€č‡Ŗåˆ†ćŒforkå…ƒć‹
恝恆恧ćŖ恄恋ćØisPrimaryć§åˆ¤å®šć™ć‚‹ć“ćØ恫ćŖć‚Šć¾ć™ć€‚

if (cluster.isPrimary) {
  // main process
  
  ...
} else {
  // sub process

  ...
}

Cluster / cluster.isPrimary

恓悌恠恑恧ē°”å˜ć«ćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ć‹ć™ć‚‹ć“ćØćŒć§ćć¾ć™ć€‚

å­ćƒ—ćƒ­ć‚»ć‚¹ć§HTTPć‚µćƒ¼ćƒćƒ¼ć‚’ćć‚Œćžć‚Œä½œć£ć¦åŒć˜ćƒćƒ¼ćƒˆć‚’ćƒŖćƒƒć‚¹ćƒ³ć™ć‚‹ć‚ˆć†ć«ę›øć„ć¦ć„ć¾ć™ćŒć€ć“ć‚ŒćÆćƒ—ćƒ­ć‚»ć‚¹é–“ć§å…±ęœ‰ć™ć‚‹ć‚‰ć—ć
問锌ćŖćå‹•ćć¾ć™ć€‚

  // sub process
  const httpServer = http.createServer((req: http.IncomingMessage, res: http.ServerResponse) => {
    const query = req.url?.split('?')[1];
    const params = queryToMap(query);

    log(`sub-process accept request, pid = ${process.pid}`);

    const result = fibRequestHandler(params);

    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(
      JSON.stringify({
        result,
        url: req.url,
        pid: process.pid,
      })
    );
  });

  httpServer.listen({
    host: '0.0.0.0',
    port: 8000,
  }, () => registerShutdown(httpServer));

ć”ć‚‡ć£ćØäøę€č­°ć§ć™ć­ć€‚

ćƒ“ćƒ«ćƒ‰ć—ć¦čµ·å‹•ć€‚

$ node dist/prefork-http-server.js
[2023-03-21T14:34:11.205Z] fork process, pid: 28781, sub-process pid: 28788
[2023-03-21T14:34:11.209Z] fork process, pid: 28781, sub-process pid: 28789
[2023-03-21T14:34:11.211Z] fork process, pid: 28781, sub-process pid: 28795
[2023-03-21T14:34:11.213Z] fork process, pid: 28781, sub-process pid: 28796
[2023-03-21T14:34:11.213Z] prefork, 4 processes
[2023-03-21T14:34:11.251Z] start prefork http-server, pid: 28788
[2023-03-21T14:34:11.257Z] start prefork http-server, pid: 28789
[2023-03-21T14:34:11.259Z] start prefork http-server, pid: 28795
[2023-03-21T14:34:11.263Z] start prefork http-server, pid: 28796

4ć¤ć®å­ćƒ—ćƒ­ć‚»ć‚¹ćŒčµ·å‹•ć•ć‚Œć¾ć™ć€‚ps恧ē¢ŗčŖć™ć‚‹ćØ态ē¢ŗć‹ć«å¢—ćˆć¦ć„ć¾ć™ć€‚

xxxxx   28781    9821  0 23:34 pts/4    00:00:00 node dist/prefork-http-server.js
xxxxx   28788   28781  0 23:34 pts/4    00:00:00 /path/to/node /path/to/dist/prefork-http-server.js
xxxxx   28789   28781  0 23:34 pts/4    00:00:00 /path/to/node /path/to/dist/prefork-http-server.js
xxxxx   28795   28781  0 23:34 pts/4    00:00:00 /path/to/node /path/to/dist/prefork-http-server.js
xxxxx   28796   28781  0 23:34 pts/4    00:00:00 /path/to/node /path/to/dist/prefork-http-server.js

先ēØ‹ć®ć‚ˆć†ć«ć€ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’4ć¤é€ć£ć¦ćæć¾ć™ć€‚

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’å—ć‘ä»˜ć‘ćŸę™‚ć®ćƒ­ć‚°ć€‚

[2023-03-21T14:37:23.300Z] sub-process accept request, pid = 28788
[2023-03-21T14:37:23.525Z] sub-process accept request, pid = 28789
[2023-03-21T14:37:23.741Z] sub-process accept request, pid = 28795
[2023-03-21T14:37:23.965Z] sub-process accept request, pid = 28796

ćć‚Œćžć‚Œåˆ„ć®ćƒ—ćƒ­ć‚»ć‚¹ćŒå—ć‘ä»˜ć‘ćŸć‚ˆć†ć§ć™ć€‚

ēµęžœć€‚

{"result":267914296,"url":"/?num=42","pid":28788}
real    0m4.196s
user    0m0.001s
sys     0m0.008s


{"result":267914296,"url":"/?num=42","pid":28789}
real    0m4.265s
user    0m0.006s
sys     0m0.000s


{"result":267914296,"url":"/?num=42","pid":28795}
real    0m4.207s
user    0m0.006s
sys     0m0.000s


{"result":267914296,"url":"/?num=42","pid":28796}
real    0m4.239s
user    0m0.007s
sys     0m0.000s

ćć‚Œćžć‚Œć€ć»ć¼åŒć˜å‡¦ē†ę™‚間恧čæ”ć£ć¦ćć‚‹ć‚ˆć†ć«ćŖć‚Šć¾ć—ćŸć€‚OK恝恆恧恙怂

mpstat恧ćÆ态恓悓ćŖę„Ÿć˜ć«č¤‡ę•°ć®CPU恌ä½æć‚ć‚Œć¦ć„ć‚‹ć“ćØ恌ē¢ŗčŖć§ćć¾ć™ć€‚

23Ꙃ37分26ē§’  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
23Ꙃ37分27ē§’  all   50.63    0.00    0.00    0.25    0.00    0.00    0.00    0.00    0.00   49.12
23Ꙃ37分27ē§’    0    2.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.00
23Ꙃ37分27ē§’    1   99.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    1.00
23Ꙃ37分27ē§’    2    1.01    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.99
23Ꙃ37分27ē§’    3  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
23Ꙃ37分27ē§’    4   24.24    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   75.76
23Ꙃ37分27ē§’    5   76.77    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   23.23
23Ꙃ37分27ē§’    6   32.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   68.00
23Ꙃ37分27ē§’    7   69.70    0.00    0.00    2.02    0.00    0.00    0.00    0.00    0.00   28.28

今回ćÆ4ć¤ć®å­ćƒ—ćƒ­ć‚»ć‚¹ć§čµ·å‹•ć—ćŸć®ć§ć€ćć‚Œä»„äøŠć®ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’åŒę™‚ć«é€äæ”恙悋ćØćć®å “åˆćÆé…å»¶ć™ć‚‹ć‚ˆć†ć«ćŖć‚Šć¾ć™ć€‚

恓悓ćŖę„Ÿć˜ć§ć€čµ·å‹•ć™ć‚‹ćƒ—ćƒ­ć‚»ć‚¹ę•°ć‚’ēµžć£ć¦ć‚‚ē¢ŗčŖć—悄恙恄ćØę€ć„ć¾ć™ć€‚

$ node dist/prefork-http-server.js 2
[2023-03-21T14:38:37.798Z] fork process, pid: 29087, sub-process pid: 29094
[2023-03-21T14:38:37.803Z] fork process, pid: 29087, sub-process pid: 29095
[2023-03-21T14:38:37.803Z] prefork, 2 processes
[2023-03-21T14:38:37.837Z] start prefork http-server, pid: 29094
[2023-03-21T14:38:37.843Z] start prefork http-server, pid: 29095

åœę­¢ć€‚

[2023-03-21T14:38:25.955Z] shutdown, signal: SIGINT
[2023-03-21T14:38:25.955Z] shutdown, signal: SIGINT
[2023-03-21T14:38:25.955Z] shutdown, signal: SIGINT
[2023-03-21T14:38:25.955Z] shutdown, signal: SIGINT

child_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æ恆

ꬔćÆ态child_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æć£ć¦ćæć¾ć™ć€‚

今回ćÆ単ē“”åŒ–ć®ćŸć‚ć«ć€ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’å—ć‘ä»˜ć‘ćŸć‚‰forkć™ć‚‹ć‚ˆć†ć«ć—ć¾ć—ćŸć€‚

src/fork-http-server.ts

import { fork } from 'node:child_process';
import http from 'node:http';
import { queryToMap } from './handler';
import { log } from './log';
import { registerShutdown } from './shutdown';

const httpServer = http.createServer((req: http.IncomingMessage, res: http.ServerResponse) => {
  const subProcess = fork(`${__dirname}/fork-handler`);

  subProcess.on('spawn', () => {
    log(`sub-process start, pid: ${subProcess.pid}`);
  });

  const query = req.url?.split('?')[1];
  const params = queryToMap(query);

  subProcess.send(JSON.stringify(Object.fromEntries(params)));

  subProcess.on('message', (result) => {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(
      JSON.stringify({
        result,
        url: req.url,
        pid: subProcess.pid,
      })
    );

    res.on('close', () => {
      subProcess.kill();
    });
  });

  subProcess.on('exit', (code, signal) => {
    log(`sub-process exit, pid: ${subProcess.pid}, code: ${code}, signal: ${signal}`);
  });
});

httpServer.listen({
  host: '0.0.0.0',
  port: 8000,
});

log(`start fork http-server, pid: ${process.pid}`);

registerShutdown(httpServer);

ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’å—ć‘ä»˜ć‘ćŸć‚‰ć€ć¾ćšćÆforkć—ć¾ć™ć€‚

const httpServer = http.createServer((req: http.IncomingMessage, res: http.ServerResponse) => {
  const subProcess = fork(`${__dirname}/fork-handler`);

å­ćƒ—ćƒ­ć‚»ć‚¹ć«ć€IPCć§å‡¦ē†ć‚’č”Œć†ćŸć‚ć®ćƒ”ćƒƒć‚»ćƒ¼ć‚ø悒送äæ”怂

  subProcess.send(JSON.stringify(Object.fromEntries(params)));

å­ćƒ—ćƒ­ć‚»ć‚¹ćŒč؈ē®—ć—ćŸå€¤ć‚’ć€ćƒ¬ć‚¹ćƒćƒ³ć‚¹ćØ恗恦čæ”ć—ć¾ć™ć€‚

  subProcess.on('message', (result) => {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(
      JSON.stringify({
        result,
        url: req.url,
        pid: subProcess.pid,
      })
    );

ćƒ¬ć‚¹ćƒćƒ³ć‚¹ć®é€äæ”ćŒå®Œäŗ†ć—ćŸć‚‰ć€å­ćƒ—ćƒ­ć‚»ć‚¹ć‚’ēµ‚äŗ†ć€‚

    res.on('close', () => {
      subProcess.kill();
    });
  });

恓悓ćŖꄟ恘恧态č؈ē®—処ē†ć®éƒØ分ćÆå­ćƒ—ćƒ­ć‚»ć‚¹ć§ć€ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚„ćƒ¬ć‚¹ćƒćƒ³ć‚¹ć®čŖ­ćæå‡ŗ恗ćŖ恩ćÆč¦Ŗćƒ—ćƒ­ć‚»ć‚¹ć§č”Œć†ć‚ˆć†ć«ć—ć¾ć—ćŸć€‚

ć¾ćŸć€forkć§čµ·å‹•ć™ć‚‹ćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć§ćÆč¦Ŗćƒ—ćƒ­ć‚»ć‚¹ć‹ć‚‰ćƒ”ćƒƒć‚»ćƒ¼ć‚øć‚’å—ć‘å–ć‚Šć€ćƒ”ćƒƒć‚»ćƒ¼ć‚ø悒čæ”恙åæ…č¦ćŒć‚ć‚‹ć®ć§ä»„äø‹ć®ć‚ˆć†ćŖ
ćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½œęˆć—ć¾ć—ćŸć€‚

src/fork-handler.ts

import { fibRequestHandler } from './handler';
import { log } from './log';

process.on('message', (paramsAsString: string) => {
  log(`sub-process accept request, pid = ${process.pid}`);

  const params = new Map<string, string>(Object.entries(JSON.parse(paramsAsString)));

  const result = fibRequestHandler(params);

  process.send!({ result });
});

ćƒ—ćƒ­ć‚»ć‚¹é–“ć®ćƒ”ćƒƒć‚»ćƒ¼ć‚ø恮送äæ”ćÆsend恧态

受äæ”ćÆon('message', handler)ć§č”Œć„ć¾ć™ć€‚

ć‚ć‹ć‚Šć«ćć„ć§ć™ćŒć€ä»„äø‹ć§ćÆč¦Ŗćƒ—ćƒ­ć‚»ć‚¹ć‹ć‚‰ćƒ”ćƒƒć‚»ćƒ¼ć‚øć‚’å—ć‘å–ć‚Šć€å‡¦ē†ćŒēµ‚ć‚ć£ćŸć‚‰č¦Ŗćƒ—ćƒ­ć‚»ć‚¹ć«ćƒ”ćƒƒć‚»ćƒ¼ć‚ø悒送äæ”ć—ć¦ć„ć¾ć™ć€‚

process.on('message', (paramsAsString: string) => {
  log(`sub-process accept request, pid = ${process.pid}`);

  const params = new Map<string, string>(Object.entries(JSON.parse(paramsAsString)));

  const result = fibRequestHandler(params);

  process.send!({ result });
});

č¦Ŗćƒ—ćƒ­ć‚»ć‚¹ć‹ć‚‰å­ćƒ—ćƒ­ć‚»ć‚¹ć«ęƒ…å ±ć‚’ęø”ć—ć€å­ćƒ—ćƒ­ć‚»ć‚¹ć‹ć‚‰ēµęžœć‚’å—ć‘å–ć£ć¦ć„ć‚‹ć®ćÆ仄äø‹ć®éƒØåˆ†ć§ć™ć­ć€‚

  subProcess.send(JSON.stringify(Object.fromEntries(params)));

  subProcess.on('message', (result) => {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(
      JSON.stringify({
        result,
        url: req.url,
        pid: subProcess.pid,
      })
    );

čµ·å‹•ć€‚

$ node dist/fork-http-server.js
[2023-03-21T14:39:00.351Z] start fork http-server, pid: 29124

ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’1ć¤é€ć£ć¦ćæć¾ć™ć€‚

$ time curl localhost:8000?num=42

ćƒ­ć‚°ć«ćÆ态恓悓ćŖꄟ恘恧forkć•ć‚Œć¦č؈ē®—処ē†ć‚’č”Œć„ć€ēµ‚ć‚ć£ćŸć‚‰ćƒ—ćƒ­ć‚»ć‚¹ć‚’ēµ‚äŗ†ć•ć›ć¦ć„悋恓ćØ悒ē¢ŗčŖć§ćć¾ć™ć€‚

[2023-03-21T14:39:37.740Z] sub-process start, pid: 29148
[2023-03-21T14:39:37.773Z] sub-process accept request, pid = 29148
[2023-03-21T14:39:41.418Z] sub-process exit, pid: 29148, code: null, signal: SIGTERM

恧ćÆ态ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’4ć¤é€ć£ć¦ćæć¾ć™ć€‚

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

ēµęžœć€‚

{"result":{"result":267914296},"url":"/?num=42","pid":29192}
real    0m4.192s
user    0m0.005s
sys     0m0.004s


{"result":{"result":267914296},"url":"/?num=42","pid":29201}
real    0m4.261s
user    0m0.004s
sys     0m0.002s


{"result":{"result":267914296},"url":"/?num=42","pid":29210}
real    0m4.251s
user    0m0.003s
sys     0m0.003s


{"result":{"result":267914296},"url":"/?num=42","pid":29219}
real    0m4.209s
user    0m0.003s
sys     0m0.004s

ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’å—ć‘ä»˜ć‘ć¦ć‹ć‚‰ć€å­ćƒ—ćƒ­ć‚»ć‚¹ćŒēµ‚äŗ†ć™ć‚‹ć¾ć§ć®ćƒ­ć‚°ć€‚

[2023-03-21T14:40:43.341Z] sub-process start, pid: 29192
[2023-03-21T14:40:43.377Z] sub-process accept request, pid = 29192
[2023-03-21T14:40:43.888Z] sub-process start, pid: 29201
[2023-03-21T14:40:43.923Z] sub-process accept request, pid = 29201
[2023-03-21T14:40:44.216Z] sub-process start, pid: 29210
[2023-03-21T14:40:44.251Z] sub-process accept request, pid = 29210
[2023-03-21T14:40:44.520Z] sub-process start, pid: 29219
[2023-03-21T14:40:44.559Z] sub-process accept request, pid = 29219
[2023-03-21T14:40:47.524Z] sub-process exit, pid: 29192, code: null, signal: SIGTERM
[2023-03-21T14:40:48.144Z] sub-process exit, pid: 29201, code: null, signal: SIGTERM
[2023-03-21T14:40:48.462Z] sub-process exit, pid: 29210, code: null, signal: SIGTERM
[2023-03-21T14:40:48.723Z] sub-process exit, pid: 29219, code: null, signal: SIGTERM

CPU恌複ꕰä½æć‚ć‚Œć¦ć„ć‚‹ć“ćØ悒ē¢ŗčŖć€‚

23Ꙃ40分46ē§’  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
23Ꙃ40分47ē§’  all   50.63    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   49.37
23Ꙃ40分47ē§’    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
23Ꙃ40分47ē§’    1  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
23Ꙃ40分47ē§’    2  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
23Ꙃ40分47ē§’    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
23Ꙃ40分47ē§’    4  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
23Ꙃ40分47ē§’    5    1.01    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.99
23Ꙃ40分47ē§’    6  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
23Ꙃ40分47ē§’    7    1.01    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.99

ćŖ恊态ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’å‡¦ē†ć—ć¦ć„ć‚‹é–“ć®ćƒ—ćƒ­ć‚»ć‚¹ć‚’č¦‹ć‚‹ćØ态仄äø‹ć®ć‚ˆć†ć«ćŖć£ć¦ć„ć¾ć™ć€‚

xxxxx   29124    9821  0 23:39 pts/4    00:00:00 node dist/fork-http-server.js
xxxxx   29301   29124 73 23:42 pts/4    00:00:02 /path/to/node /path/to/dist/fork-handler
xxxxx   29310   29124 97 23:42 pts/4    00:00:01 /path/to/node /path/to/dist/fork-handler
xxxxx   29319   29124 84 23:42 pts/4    00:00:01 /path/to/node /path/to/dist/fork-handler
xxxxx   29328   29124 73 23:42 pts/4    00:00:01 /path/to/node /path/to/dist/fork-handler

ćƒŖć‚Æć‚Øć‚¹ćƒˆćŒēµ‚äŗ†ę¬”ē¬¬ć€forkć—ćŸå­ćƒ—ćƒ­ć‚»ć‚¹ćÆēµ‚äŗ†ć—ć¦ć„ćć¾ć™ć€‚

ēµ‚äŗ†ć€‚

[2023-03-21T14:45:49.815Z] shutdown, signal: SIGINT

worker_threadsćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æ恆

ęœ€å¾ŒćÆ态worker_threadsćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æć£ć¦ćƒžćƒ«ćƒć‚¹ćƒ¬ćƒƒćƒ‰ć§å‹•ä½œć™ć‚‹ć‚ˆć†ć«ć—ć¾ć—ć‚‡ć†ć€‚

恓恔悉悂态ē°”å˜ć®ćŸć‚ć«ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’å—ć‘ä»˜ć‘ćŸć‚‰ę–°ć—ćć‚¹ćƒ¬ćƒƒćƒ‰ć‚’čµ·å‹•ć™ć‚‹ć‚ˆć†ć«ć—ć¦ć„ć¾ć™ć€‚

src/threaded-http-server.ts

import http from 'node:http';
import { Worker, isMainThread, threadId } from 'node:worker_threads';
import { queryToMap } from './handler';
import { log } from './log';
import { registerShutdown } from './shutdown';

log(`main-thread, pid = ${process.pid}, thread-id = ${threadId}, is-main-thread: ${isMainThread}`);

const httpServer = http.createServer((req: http.IncomingMessage, res: http.ServerResponse) => {
  const query = req.url?.split('?')[1];
  const params = queryToMap(query);

  const worker = new Worker(`${__dirname}/worker-handler`, { workerData: params });

  worker.on('online', () => {
    log(`worker-thread start, pid: ${process.pid}, thread-id: ${worker.threadId}, is-main-thread: ${isMainThread}`);
  });

  worker.on('message', (result) => {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(
      JSON.stringify({
        result,
        url: req.url,
        pid: process.pid,
        threadId: worker.threadId,
      })
    );
  });

  worker.on('exit', () => {
    log(`worker-thread exit, pid: ${process.pid}, thread-id: ${worker.threadId}, is-main-thread: ${isMainThread}`);
  });
});

httpServer.listen(
  {
    host: '0.0.0.0',
    port: 8000,
  },
  () => registerShutdown(httpServer)
);

log(`start threaded http-server, pid: ${process.pid}, thread-id: ${threadId}, is-main-thread: ${isMainThread}`);

ć‚¹ćƒ¬ćƒƒćƒ‰ćÆ态Worker恫åÆ¾č±”ć®ćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ęŒ‡å®šć™ć‚‹ć“ćØć§čµ·å‹•ć—ć¾ć™ć€‚

  const worker = new Worker(`${__dirname}/worker-handler`, { workerData: params });

ć“ć®ę™‚ć«ć€ć‚¹ćƒ¬ćƒƒćƒ‰ć«ęø”ć™ćƒ‡ćƒ¼ć‚æć‚‚ęŒ‡å®šć§ćć¾ć™ć€‚

ć‚³ćƒ³ć‚¹ćƒˆćƒ©ć‚Æć‚æä»„å¤–ć§ć‚‚ć€postMessage恧悂ęø”恛恝恆恧恙恭怂

Worker threads / new Worker(filename[, options]) / worker.postMessage(value[, transferList])

č؈ē®—処ē†ćÆčµ·å‹•ć—ćŸć‚¹ćƒ¬ćƒƒćƒ‰ć§č”Œć„ć€ćƒ”ćƒƒć‚»ćƒ¼ć‚øćØć—ć¦å—ć‘å–ć‚Šć¾ć™ć€‚

  worker.on('message', (result) => {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(
      JSON.stringify({
        result,
        url: req.url,
        pid: process.pid,
        threadId: worker.threadId,
      })
    );
  });

ć“ć®ę™‚ć«ć‚¹ćƒ¬ćƒƒćƒ‰ć®id悂čæ”ć™ć‚ˆć†ć«ć—ć¾ć—ćŸć€‚

ć“ć®å€¤ćÆć€ćƒ—ćƒ©ćƒƒćƒˆćƒ•ć‚©ćƒ¼ćƒ ć®ć‚¹ćƒ¬ćƒƒćƒ‰ć®idćØćÆ関äæ‚ćŒćŖ恕恝恆恧恙怂

Worker threads / worker.threadId

ć¾ćŸć€ē¾åœØć®ć‚¹ćƒ¬ćƒƒćƒ‰ćŒćƒ”ć‚¤ćƒ³ć‚¹ćƒ¬ćƒƒćƒ‰ć‹ć©ć†ć‹ćÆisMainThreadć§åˆ¤å®šć™ć‚‹ć“ćØćŒć§ćć¾ć™ć€‚ä»Šå›žćÆ惭悰å‡ŗåŠ›ć«ä½æć£ć¦ć„ć¾ć™ć€‚

Worker threads / worker.isMainThread

isMainThread恌true悒čæ”ć™å “合ćÆ态Workerå†…ć§å‹•ä½œć—ć¦ć„ćŖ恄恓ćØ悒ē¤ŗć—ć¾ć™ć€‚

čµ·å‹•ć—ćŸć‚¹ćƒ¬ćƒƒćƒ‰ćØćƒ”ćƒƒć‚»ćƒ¼ć‚øć®é€å—äæ”悒恙悋åæ…č¦ćŒć‚ć‚‹ć®ć§ć€child_processćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æć£ćŸę™‚ćØåŒć˜ć‚ˆć†ć«ćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ē”Øę„ć—ć¾ć—ćŸć€‚

src/worker-handler.ts

import { isMainThread, parentPort, threadId, workerData } from 'node:worker_threads';
import { fibRequestHandler } from './handler';
import { log } from './log';

log(`worker-thread accept request, pid = ${process.pid}, thread-id = ${threadId}, is-main-thread: ${isMainThread}`);

const params: Map<string, string> = workerData;

const result = fibRequestHandler(workerData);

parentPort?.postMessage(result);

Workerä½œęˆę™‚ć«ęø”ć—ćŸćƒ‡ćƒ¼ć‚æćÆ态workerDatać§å‚ē…§ć§ćć¾ć™ć€‚

Worker threads / worker.workerData

ćć—ć¦ć€č¦Ŗć®ć‚¹ćƒ¬ćƒƒćƒ‰ćøćƒ‡ćƒ¼ć‚æ悒ęø”恙恫ćÆ态parentPortēµŒē”±ć§postMessage悒ä½æ恆恓ćØć§č”Œćˆć¾ć™ć€‚

恓恔悉悒ä½æ恆恓ćØć§ć€čµ·å‹•å…ƒćƒ»å…ˆć®ć‚¹ćƒ¬ćƒƒćƒ‰é–“ć§ćƒ‡ćƒ¼ć‚æć®ć‚„ć‚Šå–ć‚Šć‚’č”Œć†ć“ćØćŒć§ćć¾ć™ć€‚

čµ·å‹•ć€‚

$ node dist/threaded-http-server.js
[2023-03-21T15:13:59.056Z] main-thread, pid = 30736, thread-id = 0, is-main-thread: true
[2023-03-21T15:13:59.061Z] start threaded http-server, pid: 30736, thread-id: 0, is-main-thread: true

ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’1ć¤é€ć£ć¦ćæć¾ć™ć€‚

$ time curl localhost:8000?num=42
{"result":267914296,"url":"/?num=42","pid":30736,"threadId":1}
real    0m3.793s
user    0m0.003s
sys     0m0.008s

ćƒ­ć‚°ć«ćÆ态恓悓ćŖę„Ÿć˜ć§åˆ„ć‚¹ćƒ¬ćƒƒćƒ‰ć§č؈ē®—処ē†ć‚’č”Œć„ć€ēµ‚ć‚ć£ćŸć‚‰ć‚¹ćƒ¬ćƒƒćƒ‰ć‚’ēµ‚äŗ†ć•ć›ć¦ć„悋恓ćØ悒ē¢ŗčŖć§ćć¾ć™ć€‚

[2023-03-21T15:14:38.375Z] worker-thread start, pid: 30736, thread-id: 1, is-main-thread: true
[2023-03-21T15:14:38.380Z] worker-thread accept request, pid = 30736, thread-id = 1, is-main-thread: false
[2023-03-21T15:14:42.099Z] worker-thread exit, pid: 30736, thread-id: -1, is-main-thread: true

ć‚¹ćƒ¬ćƒƒćƒ‰ēµ‚äŗ†ć®ć‚¤ćƒ™ćƒ³ćƒˆćƒćƒ³ćƒ‰ćƒŖćƒ³ć‚°ć®ę™‚ć«ćÆć€ć‚¹ćƒ¬ćƒƒćƒ‰ć®id恌悏恋悉ćŖ恏ćŖ悋ćæ恟恄恧恙恭怂

恧ćÆ态ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’4ć¤é€ć£ć¦ćæć¾ć™ć€‚

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

$ time curl localhost:8000?num=42

ēµęžœć€‚

{"result":267914296,"url":"/?num=42","pid":30736,"threadId":2}
real    0m4.369s
user    0m0.003s
sys     0m0.007s


{"result":267914296,"url":"/?num=42","pid":30736,"threadId":3}
real    0m4.356s
user    0m0.000s
sys     0m0.006s


{"result":267914296,"url":"/?num=42","pid":30736,"threadId":4}
real    0m4.365s
user    0m0.007s
sys     0m0.000s


{"result":267914296,"url":"/?num=42","pid":30736,"threadId":5}
real    0m4.369s
user    0m0.007s
sys     0m0.000s

複ꕰ恮ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’é€äæ”ć—ć¦ć‚‚ć€ćć‚Œć»ć©åŠ£åŒ–ć›ćšćć‚Œćžć‚Œć®ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’å‡¦ē†ć§ćć¦ć„ć¾ć™ć€‚

ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’å—ć‘ä»˜ć‘ć¦ć‹ć‚‰ć€å­ćƒ—ćƒ­ć‚»ć‚¹ćŒēµ‚äŗ†ć™ć‚‹ć¾ć§ć®ćƒ­ć‚°ć€‚

[2023-03-21T15:15:30.236Z] worker-thread start, pid: 30736, thread-id: 2, is-main-thread: true
[2023-03-21T15:15:30.241Z] worker-thread accept request, pid = 30736, thread-id = 2, is-main-thread: false
[2023-03-21T15:15:30.492Z] worker-thread start, pid: 30736, thread-id: 3, is-main-thread: true
[2023-03-21T15:15:30.497Z] worker-thread accept request, pid = 30736, thread-id = 3, is-main-thread: false
[2023-03-21T15:15:30.711Z] worker-thread start, pid: 30736, thread-id: 4, is-main-thread: true
[2023-03-21T15:15:30.718Z] worker-thread accept request, pid = 30736, thread-id = 4, is-main-thread: false
[2023-03-21T15:15:30.936Z] worker-thread start, pid: 30736, thread-id: 5, is-main-thread: true
[2023-03-21T15:15:30.941Z] worker-thread accept request, pid = 30736, thread-id = 5, is-main-thread: false
[2023-03-21T15:15:34.540Z] worker-thread exit, pid: 30736, thread-id: -1, is-main-thread: true
[2023-03-21T15:15:34.795Z] worker-thread exit, pid: 30736, thread-id: -1, is-main-thread: true
[2023-03-21T15:15:35.021Z] worker-thread exit, pid: 30736, thread-id: -1, is-main-thread: true
[2023-03-21T15:15:35.240Z] worker-thread exit, pid: 30736, thread-id: -1, is-main-thread: true

CPU恌複ꕰä½æć‚ć‚Œć¦ć„ć‚‹ć“ćØ悒ē¢ŗčŖć€‚

00Ꙃ15分30ē§’  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
00Ꙃ15分31ē§’  all   53.63    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   45.36
00Ꙃ15分31ē§’    0  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
00Ꙃ15分31ē§’    1    5.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00    0.00   93.00
00Ꙃ15分31ē§’    2   22.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   77.00
00Ꙃ15分31ē§’    3   80.20    0.00    1.98    0.00    0.00    0.00    0.00    0.00    0.00   17.82
00Ꙃ15分31ē§’    4  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
00Ꙃ15分31ē§’    5   13.13    0.00    3.03    0.00    0.00    0.00    0.00    0.00    0.00   83.84
00Ꙃ15分31ē§’    6    7.14    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   92.86
00Ꙃ15分31ē§’    7  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00

ćŖ恊态ćƒŖć‚Æć‚Øć‚¹ćƒˆć‚’å‡¦ē†ć—ć¦ć„ć‚‹é–“ć®ć‚¹ćƒ¬ćƒƒćƒ‰ć®ę§˜å­ćÆ恓悓ćŖę„Ÿć˜ć§ć”ć‚‡ć£ćØå¤šć‚ć§ć™ćŒć€å‡¦ē†ćŒēµ‚äŗ†ć™ć‚‹ćØć‚¹ćƒ¬ćƒƒćƒ‰ćÆć”ć‚ƒć‚“ćØ
ęø›ć‚Šć¾ć™ć€‚Node.jsćŒć„ćć¤ć‹ć‚¹ćƒ¬ćƒƒćƒ‰ć‚’ē®”ē†ć—ć¦ć„ćć†ćŖꄟ恘恧恙恭怂

$ ps aux -L | grep node | grep dist
xxxxx   31040   31040  0.2   11  0.5 1430956 93000 pts/4   Sl+  00:19   0:00 node dist/threaded-http-server.js
xxxxx   31040   31041  0.0   11  0.5 1430956 93000 pts/4   Sl+  00:19   0:00 node dist/threaded-http-server.js
xxxxx   31040   31042  0.0   11  0.5 1430956 93000 pts/4   Sl+  00:19   0:00 node dist/threaded-http-server.js
xxxxx   31040   31043  0.0   11  0.5 1430956 93000 pts/4   Sl+  00:19   0:00 node dist/threaded-http-server.js
xxxxx   31040   31044  0.0   11  0.5 1430956 93000 pts/4   Sl+  00:19   0:00 node dist/threaded-http-server.js
xxxxx   31040   31045  0.0   11  0.5 1430956 93000 pts/4   Sl+  00:19   0:00 node dist/threaded-http-server.js
xxxxx   31040   31046  0.0   11  0.5 1430956 93000 pts/4   Sl+  00:19   0:00 node dist/threaded-http-server.js
xxxxx   31040   31117  112   11  0.5 1430956 93000 pts/4   Rl+  00:19   0:02 node dist/threaded-http-server.js
xxxxx   31040   31120  184   11  0.5 1430956 93000 pts/4   Rl+  00:19   0:01 node dist/threaded-http-server.js
xxxxx   31040   31123  143   11  0.5 1430956 93000 pts/4   Rl+  00:19   0:01 node dist/threaded-http-server.js
xxxxx   31040   31126  101   11  0.5 1430956 93000 pts/4   Rl+  00:19   0:01 node dist/threaded-http-server.js

ć–ć£ćć‚Šć€ć“ć‚“ćŖćØ恓悍恧恗悇恆恋怂

ć‚Ŗćƒžć‚±

ć”ć‚‡ć£ćØ恗恟ć‚Ŗćƒžć‚±ćØ恗恦态clusterćƒ¢ć‚øćƒ„ćƒ¼ćƒ«ć‚’ä½æć£ć¦č¦Ŗćƒ—ćƒ­ć‚»ć‚¹ć§HTTPć‚µćƒ¼ćƒćƒ¼ć‚’čµ·å‹•ć—ćŸå “åˆćÆć©ć†ć—ćŸć‚‰ć„ć„ć®ć‹ćŖļ¼ŸćØ恄恆恮悒
č©¦č”Œć—ć¦ćæć¾ć—ćŸć€‚

src/prefork-http-server2.ts

import cluster, { Worker } from 'node:cluster';
import http from 'node:http';
import { queryToMap } from './handler';
import { log } from './log';
import { registerShutdown } from './shutdown';

const processes = process.argv[2] !== undefined ? parseInt(process.argv[2], 10) : 4;

cluster.setupPrimary({ exec: `${__dirname}/fork-handler` });

const subProcesses: Worker[] = [];

for (let i = 0; i < processes; i++) {
  const subProcess = cluster.fork();
  log(`fork process, pid: ${subProcess.process.pid}`);

  subProcesses.push(subProcess);
}

log(`prefork, ${processes} processes`);

let index = 0;

const httpServer = http.createServer((req: http.IncomingMessage, res: http.ServerResponse) => {
  const query = req.url?.split('?')[1];
  const params = queryToMap(query);

  if (index >= subProcesses.length) {
    index = 0;
  }

  const subProcess = subProcesses[index++];

  subProcess.send(JSON.stringify(Object.fromEntries(params)));

  subProcess.on('message', (result) => {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(
      JSON.stringify({
        result,
        url: req.url,
        pid: subProcess.process.pid,
      })
    );
  });
});

httpServer.listen(
  {
    host: '0.0.0.0',
    port: 8000,
  },
  () => registerShutdown(httpServer)
);

log(`start prefork http-server, pid: ${process.pid}`);

ęœ€åˆć«å­ćƒ—ćƒ­ć‚»ć‚¹ć®ćƒ—ćƒ¼ćƒ«ć‚’ä½œć£ć¦ć€ćƒŖć‚Æć‚Øć‚¹ćƒˆćŒę„ć‚‹ćØå­ćƒ—ćƒ­ć‚»ć‚¹ć«ćƒ”ćƒƒć‚»ćƒ¼ć‚ø送受äæ”ć™ć‚‹ę„Ÿć˜ć«ćŖć‚Šć¾ć—ćŸć€‚

恓恔悉悂ē°”ꘓēš„ćŖć‚‚ć®ć§ć™ćŒć€‚

ć¾ćØ悁

Node.jsć§ćƒžćƒ«ćƒćƒ—ćƒ­ć‚»ć‚¹ć€ćƒžćƒ«ćƒć‚¹ćƒ¬ćƒƒćƒ‰ć‚’ę‰±ć†ę–¹ę³•ć‚’ć„ćć¤ć‹čŖæć¹ć¦ćæć¾ć—ćŸć€‚

ć“ć‚Œć¾ć§Node.js恧ćÆć“ć®ć‚ćŸć‚Šć®č©±é”Œć«č§¦ć‚Œć¦ć“ćŖć‹ć£ćŸć®ć§ć€ć„ć‚ć„ć‚å¤§å¤‰ć§ć—ćŸćŒć„ć„å‹‰å¼·ć«ćŖć‚Šć¾ć—ćŸć€‚

ć‚¢ćƒ—ćƒŖć‚±ćƒ¼ć‚·ćƒ§ćƒ³å†…ć§ć®ćƒ—ćƒ­ć‚»ć‚¹čµ·å‹•ć€ć‚¹ćƒ¬ćƒƒćƒ‰čµ·å‹•ć€ć©ć”ć‚‰ć‚‚ä½æ恈悋悓恧恙恭怂