CLOVERšŸ€

That was when it all began.

Prometheusć®ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć®ćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆć‚’ć•ć‚‰ć£ćØčŖ­ć‚“恧ćæ恦态retention恮čØ­å®šć‚‚ć—ć¦ćæ悋

恓悌ćÆ态ćŖć«ć‚’ć—ćŸćć¦ę›øć„ćŸć‚‚ć®ļ¼Ÿ

Prometheusć®ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć¾ć‚ć‚Šć®ć€ćŠå‹‰å¼·ć«ć€ćØ怂

Prometheusのデータ(TSDB)のSnapshotを取得して、リストアまで - CLOVER🍀

恓恔悉恮ē¶šćć§ć€ä»Šåŗ¦ćÆć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć®ćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆć‚’čŖ­ćæ态ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ć«ć¤ć„ć¦č¦‹ć¦ć„ć“ć†ćØę€ć„ć¾ć™ć€‚

åÆ¾č±”ćØ恙悋Prometheusć®ćƒćƒ¼ć‚øćƒ§ćƒ³ćÆ态2.9.2ćØć—ć¾ć™ļ¼ˆćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆćÆ2.9ļ¼‰ć€‚

ć”ć‚‡ć£ćØčŖæć¹ć¦ćæ悋ćØć€éŽåŽ»ć®ęƒ…å ±ćØē¾åœØć®ćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆć®å†…å®¹ćÆ恝悌ćŖ悊恫ē•°ćŖ悋ē®‡ę‰€ćŒå¤šćć†ćŖć®ć§ć€ćć®ę™‚ē‚¹ć®
ćƒćƒ¼ć‚øćƒ§ćƒ³ć®ćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆć‚’ć”ć‚ƒć‚“ćØē¢ŗčŖć—ćŸę–¹ćŒč‰Æ恄ćØę€ć„ć¾ć™ć€‚

ć‚ćć¾ć§ć€ē¾ę™‚ē‚¹ļ¼ˆ2.9.2ļ¼‰ć§ć®č©±ćØ恗恦怂

Prometheusć®ć‚¹ćƒˆćƒ¬ćƒ¼ć‚ø

Prometheusć®ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć¾ć‚ć‚Šć®ćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆćÆ态恓恔悉恧恙怂

Storage | Prometheus

č¦‹ć¦ć„ććØć€ć–ć£ćć‚Šć“ć‚“ćŖę„Ÿć˜ć®ć“ćØ恌ę›øć‹ć‚Œć¦ć„ć¾ć™ć€‚

  • Prometheus恫ćÆć€ćƒ­ćƒ¼ć‚«ćƒ«ćƒ‡ć‚£ć‚¹ć‚ÆäøŠć«ęŒć¤ę™‚ē³»åˆ—ćƒ‡ćƒ¼ć‚æćƒ™ćƒ¼ć‚¹ćŒå«ć¾ć‚Œć¦ć„ć‚‹
  • ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ć§ć€ćƒŖćƒ¢ćƒ¼ćƒˆć®ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć‚·ć‚¹ćƒ†ćƒ ćØēµ±åˆć™ć‚‹ć“ćØ悂恧恍悋

ćƒŖćƒ¢ćƒ¼ćƒˆćØć®ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć‚·ć‚¹ćƒ†ćƒ ćØēµ±åˆć™ć‚‹ć“ćØ恫恤恄恦ćÆć€ć¾ćŸę©Ÿä¼šć‚’ę”¹ć‚ć¦č¦‹ć¦ć„ćć¾ć—ć‚‡ć†ć€‚

恓恓恋悉ćÆć€ćƒ­ćƒ¼ć‚«ćƒ«ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć«ć¤ć„ć¦č¦‹ć¦ć„ćć¾ć™ć€‚

ćƒ­ćƒ¼ć‚«ćƒ«ć‚¹ćƒˆćƒ¬ćƒ¼ć‚ø恫恤恄恦

Prometheusć®ćƒ­ćƒ¼ć‚«ćƒ«ć®ę™‚ē³»åˆ—ćƒ‡ćƒ¼ć‚æćƒ™ćƒ¼ć‚¹ćÆć€ćƒ‡ćƒ¼ć‚æ悒ē‹¬č‡Ŗć®ćƒ•ć‚©ćƒ¼ćƒžćƒƒćƒˆć§ćƒ‡ć‚£ć‚¹ć‚Æ恫äæå­˜ć—ć¾ć™ć€‚

ćƒ‡ć‚£ć‚¹ć‚ÆäøŠć®ćƒ¬ć‚¤ć‚¢ć‚¦ćƒˆć«ć¤ć„ć¦ć§ć™ćŒć€ä»„äø‹ć®ć‚ˆć†ćŖ要ē“ ć§ę§‹ęˆć•ć‚Œć‚‹ć‚ˆć†ć§ć™ć€‚

  • å–å¾—ć—ćŸćƒ‡ćƒ¼ć‚æćÆ态2Ꙃ間恔ćØć®ćƒ–ćƒ­ćƒƒć‚Æć«ć¾ćØ悁悉悌悋
  • 2Ꙃ間恔ćØć®å„ćƒ–ćƒ­ćƒƒć‚ÆćÆ态仄äø‹ć®å†…å®¹ć‚’å«ć‚€
    • ćć®ęœŸé–“ć®ć™ć¹ć¦ć®ćƒ‡ćƒ¼ć‚æć‚’å«ć‚€ć€ćƒćƒ£ćƒ³ć‚Æćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖļ¼ˆ1恤仄äøŠć®ćƒćƒ£ćƒ³ć‚Æćƒ•ć‚”ć‚¤ćƒ«ć‚’å«ć‚€ļ¼‰
    • 惔ć‚æćƒ‡ćƒ¼ć‚æćƒ•ć‚”ć‚¤ćƒ«
    • ć‚¤ćƒ³ćƒ‡ćƒƒć‚Æć‚¹ćƒ•ć‚”ć‚¤ćƒ«ļ¼ˆćƒ”惈ćƒŖć‚Æć‚¹åćØć€ćƒćƒ£ćƒ³ć‚Æćƒ•ć‚”ć‚¤ćƒ«å†…ć®ę™‚ē³»åˆ—ć«ćƒ©ćƒ™ćƒ«ć‚’ä»˜äøŽć—ćŸć‚‚ć®ćŒć‚¤ćƒ³ćƒ‡ćƒƒć‚Æć‚¹ć•ć‚Œć‚‹ļ¼‰
    • Tombstonećƒ•ć‚”ć‚¤ćƒ«
      • API悒ä½æē”Ø恗恦态Ꙃē³»åˆ—ćƒ‡ćƒ¼ć‚æć‚’å‰Šé™¤ć—ćŸå “åˆć«ć§ćć‚‹ćƒ•ć‚”ć‚¤ćƒ«
      • ćƒćƒ£ćƒ³ć‚Æćƒ•ć‚”ć‚¤ćƒ«ć‹ć‚‰ć™ćć«ćƒ‡ćƒ¼ć‚æćŒå‰Šé™¤ć•ć‚Œć‚‹ć‚ć‘ć§ćÆćŖ恄

ć¾ćŸć€PrometheusćÆåŽé›†ć—ćŸćƒ‡ćƒ¼ć‚æ悒恙恐恫ę°øē¶šåŒ–恙悋恮恧ćÆćŖćć€ć¾ćšćƒ”ćƒ¢ćƒŖäøŠć«äæęŒć•ć‚Œć¦ć„ć¾ć™ć€‚

Prometheus恌ć‚Æćƒ©ćƒƒć‚·ćƒ„ć—ćŸć‚Šć—ć¦å†čµ·å‹•ć—ćŸå “åˆćÆ态WALļ¼ˆWrite Ahead Logļ¼‰ć‚’ä½æ恆恓ćØ恧ć‚Æćƒ©ćƒƒć‚·ćƒ„ć‹ć‚‰äæč­·ć•ć‚Œć¦ć„ć¾ć™ć€‚

WALćÆ态仄äø‹ć®ē‰¹å¾“ć‚’ęŒć”ć¾ć™ć€‚

  • 怌walć€ćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖ恫128MB恔ćØ恫äæå­˜ć•ć‚Œć¦ć„ć‚‹
  • 怌walć€ćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖć«å«ć¾ć‚Œć‚‹ćƒ•ć‚”ć‚¤ćƒ«ćÆć€ć¾ć åœ§ēø®ć•ć‚Œć¦ć„ćŖ恄Rawćƒ‡ćƒ¼ć‚æćŒå«ć¾ć‚Œć¦ć„ć‚‹
    • ć“ć®ćŸć‚ć€é€šåøøć®ćƒ–ćƒ­ćƒƒć‚Æćƒ•ć‚”ć‚¤ćƒ«ć‚ˆć‚Šć‹ćŖć‚Šå¤§ćć„
  • PrometheusćÆęœ€ä½Ž3恤恮WALćƒ•ć‚”ć‚¤ćƒ«ć‚’äæęŒć—恦恄悋
    • é«˜ćƒˆćƒ©ćƒ•ć‚£ćƒƒć‚ÆćŖć‚µćƒ¼ćƒćƒ¼ć§ćÆć€å°‘ćŖ恏ćØ悂2ę™‚é–“åˆ†ć®Rawćƒ‡ćƒ¼ć‚æ悒äæå­˜ć™ć‚‹åæ…č¦ćŒć‚ć‚‹ćŸć‚ć€3ć¤ć‚’č¶…ćˆć‚‹WALćƒ•ć‚”ć‚¤ćƒ«ćŒć§ćć‚‹ć“ćØ恌恂悋

ć§ć€ćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆć«ę›øć‹ć‚Œć¦ć„ć‚‹ćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖę§‹é€ ćŒć“ć”ć‚‰ć€‚ć“ć“ć¾ć§ē™»å “ć—ćŸč¦ē“ ćŒę›øć‹ć‚Œć¦ć„ć‚‹ę„Ÿć˜ć§ć™ć­ć€‚

./data/01BKGV7JBM69T2G1BGBGM6KB12
./data/01BKGV7JBM69T2G1BGBGM6KB12/meta.json
./data/01BKGTZQ1SYQJTR4PB43C8PD98
./data/01BKGTZQ1SYQJTR4PB43C8PD98/meta.json
./data/01BKGTZQ1SYQJTR4PB43C8PD98/index
./data/01BKGTZQ1SYQJTR4PB43C8PD98/chunks
./data/01BKGTZQ1SYQJTR4PB43C8PD98/chunks/000001
./data/01BKGTZQ1SYQJTR4PB43C8PD98/tombstones
./data/01BKGTZQ1HHWHV8FBJXW1Y3W0K
./data/01BKGTZQ1HHWHV8FBJXW1Y3W0K/meta.json
./data/01BKGV7JC0RY8A6MACW02A2PJD
./data/01BKGV7JC0RY8A6MACW02A2PJD/meta.json
./data/01BKGV7JC0RY8A6MACW02A2PJD/index
./data/01BKGV7JC0RY8A6MACW02A2PJD/chunks
./data/01BKGV7JC0RY8A6MACW02A2PJD/chunks/000001
./data/01BKGV7JC0RY8A6MACW02A2PJD/tombstones
./data/wal/00000000
./data/wal/00000001
./data/wal/00000002

ć‚ˆć‚Šč©³ē“°ćŖå†…å®¹ć‚’ēŸ„悊恟恑悌恰态TSDBć®ćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆćø怂

tsdb/README.md at v0.7.1 · prometheus/tsdb · GitHub

恔ćŖćæć«ć€čµ·å‹•ē›“å¾Œć®dataćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖ恮äø­čŗ«ćÆ态恓悓ćŖꄟ恘恧恙怂

$ find data -type f
data/wal/00000000
data/lock

lockćØć„ć†ćƒ•ć‚”ć‚¤ćƒ«ćÆå‡ŗć¦ćć¦ć„ć¾ć›ć‚“ć§ć—ćŸć­ć€‚

$ ls -l data/lock 
-rw-r--r-- 1 xxxxx xxxxx 0 May  2 10:51 data/lock

0ćƒć‚¤ćƒˆć®ćƒ•ć‚”ć‚¤ćƒ«ć§ć™ć€‚

ć“ć®ćƒ•ć‚”ć‚¤ćƒ«ćÆć€ę–‡å­—é€šć‚ŠęŽ’ä»–ć«ä½æć†ćƒ•ć‚”ć‚¤ćƒ«ć®ć‚ˆć†ć§ć€ć™ć§ć«PrometheusćŒčµ·å‹•ć—ćŸēŠ¶ę…‹ć§ć€ćƒŖćƒƒć‚¹ćƒ³ćƒćƒ¼ćƒˆć‚’
å¤‰ćˆć¦čµ·å‹•ć—ćŸć‚Šć—ć‚ˆć†ćØ恙悋ćØ态惭惃ć‚ÆćŒå–ć‚Œćšć«čµ·å‹•ć«å¤±ę•—ć—ć¾ć™ć€‚

$ ./prometheus --web.listen-address="0.0.0.0:9091"

...

level=error ts=2019-05-02T10:53:06.753Z caller=main.go:717 err="opening storage failed: lock DB directory: resource temporarily unavailable"

https://github.com/prometheus/prometheus/blob/v2.9.2/vendor/github.com/prometheus/tsdb/db.go#L280-L283

å…ˆć«é€²ćæć¾ć—ć‚‡ć†ć€‚

ęœ€åˆć®2ę™‚é–“ć®ćƒ–ćƒ­ćƒƒć‚ÆćÆ态꜀ēµ‚ēš„恫ćÆ惐惃ć‚Æć‚°ćƒ©ć‚¦ćƒ³ćƒ‰ć§ć‚ˆć‚Šé•·ć„ćƒ–ćƒ­ćƒƒć‚Æć«åœ§ēø®ć•ć‚Œć¾ć™ć€‚

ćƒ­ćƒ¼ć‚«ćƒ«ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć®åˆ¶é™

ćƒ­ćƒ¼ć‚«ćƒ«ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć®åˆ¶é™ćÆ态ć‚Æćƒ©ć‚¹ć‚æåŒ–ć‚‚ć•ć‚Œć¦ćŠć‚‰ćšć€ć¾ćŸćƒ¬ćƒ—ćƒŖć‚«ć‚‚ęŒćŸćŖ恄恓ćØ恧恙怂

ćŖć®ć§ć€ćƒ‡ć‚£ć‚¹ć‚Æć‚„ćƒŽćƒ¼ćƒ‰éšœå®³åÆ¾ć—ć¦ć€č€ę€§ćŒć‚ć‚Šć¾ć›ć‚“ć€‚ć‚¹ć‚±ćƒ¼ćƒ©ćƒ–ćƒ«ć§ć‚‚č€ä¹…ę€§ć‚‚ćŖ恄态ē›“čæ‘恮ēŸ­å‘½ćŖ
ć‚¹ćƒ©ć‚¤ćƒ‡ć‚£ćƒ³ć‚°ć‚¦ć‚£ćƒ³ćƒ‰ć‚¦ćƒ‡ćƒ¼ć‚æćØć—ć¦ę‰±ć‚ć‚Œć‚‹ć¹ćć§ć™ć€‚

č€ä¹…ę€§ć®č¦ä»¶ćŒåŽ³ć—ććŖć‘ć‚Œć°ć€ćƒ­ćƒ¼ć‚«ćƒ«ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć§ć‚‚ęœ€å¤§ę•°å¹“ć®ćƒ‡ćƒ¼ć‚æ悒äæå­˜ć§ćć‚‹ć‹ć‚‚ć—ć‚ŒćŖć„ć€ćć‚‰ć„ć®ć“ćØ恌
ę›øć‹ć‚Œć¦ć„ć¾ć™ā€¦ć€‚

ć“ć®ć‚ćŸć‚Šć®ć“ćØć‹ć‚‰ć€ćƒ­ćƒ¼ć‚«ćƒ«ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć®ć‚¹ć‚±ćƒ¼ćƒ©ćƒ“ćƒŖćƒ†ć‚£ćØč€ä¹…ę€§ć«ć¤ć„ć¦ć®čŖ²é”Œć‚’č§£ę±ŗć™ć‚‹ćŸć‚ć«ć€ćƒŖćƒ¢ćƒ¼ćƒˆć®
ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć‚·ć‚¹ćƒ†ćƒ ćØēµ±åˆć™ć‚‹ä»•ēµ„ćæ恌恂悋态ćØ恄恆恓ćØćæ恟恄恧恙怂

Remote storage integrations

ćƒ­ćƒ¼ć‚«ćƒ«ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć«é–¢ć™ć‚‹ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³

ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć«é–¢ć™ć‚‹ćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆć®ä»„äø‹ć®éƒØåˆ†ć«ć€ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć«é–¢ć—ć¦ęŒ‡å®šć§ćć‚‹ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ćŒę›øć‹ć‚Œć¦ć„ć¾ć™ć€‚

Operational aspects

Prometheusč‡Ŗčŗ«ć®ćƒ˜ćƒ«ćƒ—ć‹ć‚‰ć€ć€Œstorageć€ć‚’å«ć‚€ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ć‚’č”Øē¤ŗ恗恦ćæ悋ćØ态

$ ./prometheus -h 2>&1 | grep storage
      --storage.tsdb.path="data/"  
                                 Base path for metrics storage.
      --storage.tsdb.retention=STORAGE.TSDB.RETENTION  
                                 storage. This flag has been deprecated, use
                                 "storage.tsdb.retention.time" instead
      --storage.tsdb.retention.time=STORAGE.TSDB.RETENTION.TIME  
                                 How long to retain samples in storage. When
                                 "storage.tsdb.retention". If neither this flag
                                 nor "storage.tsdb.retention" nor
                                 "storage.tsdb.retention.size" is set, the
      --storage.tsdb.retention.size=STORAGE.TSDB.RETENTION.SIZE  
      --storage.tsdb.no-lockfile  
      --storage.tsdb.allow-overlapping-blocks  
      --storage.remote.flush-deadline=<duration>  
      --storage.remote.read-sample-limit=5e7  
      --storage.remote.read-concurrent-limit=10  

怌--storage.remote.ć€œć€ć«ć¤ć„ć¦ćÆ态ćƒŖćƒ¢ćƒ¼ćƒˆć‚¹ćƒˆćƒ¬ćƒ¼ć‚øć‚·ć‚¹ćƒ†ćƒ ćØ恮ēµ±åˆć«é–¢ć™ć‚‹ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ćŖ恮恧态恓恓恧ćÆåÆ¾č±”å¤–ć€‚

ćć®ä»–ć®ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ć«ć¤ć„ć¦ć€čŖ¬ę˜Žć‚’見恦ćæć¾ć™ć€‚

      --storage.tsdb.path="data/"  
                                 Base path for metrics storage.
      --storage.tsdb.retention=STORAGE.TSDB.RETENTION  
                                 [DEPRECATED] How long to retain samples in storage. This flag has been deprecated, use "storage.tsdb.retention.time" instead
      --storage.tsdb.retention.time=STORAGE.TSDB.RETENTION.TIME  
                                 How long to retain samples in storage. When this flag is set it overrides "storage.tsdb.retention". If neither this flag nor
                                 "storage.tsdb.retention" nor "storage.tsdb.retention.size" is set, the retention time defaults to 15d.
      --storage.tsdb.retention.size=STORAGE.TSDB.RETENTION.SIZE  
                                 [EXPERIMENTAL] Maximum number of bytes that can be stored for blocks. Units supported: KB, MB, GB, TB, PB. This flag is experimental and can be
                                 changed in future releases.
      --storage.tsdb.no-lockfile  
                                 Do not create lockfile in data directory.
      --storage.tsdb.allow-overlapping-blocks  
                                 [EXPERIMENTAL] Allow overlapping blocks which in-turn enables vertical compaction and vertical query merge

恙恧恫DeprecatedćŖć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ćÆē„”č¦–ć—ć¾ć™ć€‚

  • --storage.tsdb.path ā€¦ PrometheusćŒćƒ‡ćƒ¼ć‚æ悒äæå­˜ć™ć‚‹éš›ć®ćƒ™ćƒ¼ć‚¹ćƒ‘ć‚¹ć€‚ćƒ‡ćƒ•ć‚©ćƒ«ćƒˆćÆ态怌data怍
  • --storage.tsdb.retention.time ā€¦ å¤ć„ćƒ‡ćƒ¼ć‚æć‚’ć„ć¤å‰Šé™¤ć™ć‚‹ć‹ć‚’ęŒ‡å®šć™ć‚‹ć€‚ćƒ‡ćƒ•ć‚©ćƒ«ćƒˆćÆ态怌15d怍
  • --storage.tsdb.retention.size ā€¦ ļ¼ˆå®Ÿéؓēš„ļ¼‰ć‚¹ćƒˆćƒ¬ćƒ¼ć‚ø惖惭惃ć‚ÆćŒęœ€å¤§ć§ä½æē”Øć§ćć‚‹ć‚µć‚¤ć‚ŗć‚’ęŒ‡å®šć™ć‚‹
  • --storage.tsdb.no-lockfile ā€¦ dataćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖå†…ć«ć€ćƒ­ćƒƒć‚Æćƒ•ć‚”ć‚¤ćƒ«ć‚’ä½œęˆć—ćŖ恄
  • --storage.tsdb.allow-overlapping-blocks ā€¦ ļ¼ˆå®Ÿéؓēš„ļ¼‰ć‚Ŗćƒ¼ćƒćƒ¼ćƒ©ćƒƒćƒ—ć™ć‚‹ćƒ–ćƒ­ćƒƒć‚Æ悒čرåÆć™ć‚‹ć€‚åœ§ēø®ć€ć‚Æć‚ØćƒŖćƒ¼ć®ēø¦ę–¹å‘ć®ćƒžćƒ¼ć‚ø恌åÆčƒ½ć«ćŖ悋

怌--storage.tsdb.retention.timeć€ć®ćƒ‡ćƒ•ć‚©ćƒ«ćƒˆå€¤ćÆ15ę—„ć§ć€ćƒ•ć‚©ćƒ¼ćƒžćƒƒćƒˆć«ćÆy态w态d态h态m态s态msćŒåˆ©ē”Øć§ćć¾ć™ć€‚

prometheus/db.go at v2.9.2 · prometheus/prometheus · GitHub

https://github.com/prometheus/prometheus/blob/v2.9.2/vendor/github.com/prometheus/common/model/time.go#L192-L208

ćƒ‡ćƒ¼ć‚æ恮äæęŒęœŸé–“ć«é–¢ć™ć‚‹ćƒŖćƒ†ćƒ³ć‚·ćƒ§ćƒ³ćƒćƒŖć‚·ćƒ¼ćÆꙂ間ćØ悵悤ć‚ŗ恮2ć¤ćŒć‚ć‚Šć¾ć™ćŒć€ä½æć‚ć‚Œć‚‹ć®ćÆęœ€åˆć«å‹•ä½œć—ćŸę–¹ć ćØ恋怂
ā€»ć‚µć‚¤ć‚ŗćÆć¾ć å®Ÿéؓēš„ćŖć®ć§ć€ć‚¹ćƒ«ćƒ¼ć—ć¾ć—ćŸćŒ

怌--storage.tsdb.allow-overlapping-blocksć€ć«ć¤ć„ć¦ćÆć€ć¾ć å®Ÿéؓēš„ćŖę„Ÿć˜ćŒć™ć‚‹ć®ć§ć€ć‚¹ćƒ«ćƒ¼ć—ć¾ć™ā€¦ć€‚

Implement vertical query merging and compaction · Issue #90 · prometheus/tsdb · GitHub

ćŖ恊态2Ꙃ間恔ćØć«ćƒ–ćƒ­ćƒƒć‚Æć«ć¾ćØ悁悉悌悋态ćØ恄恆恮ćÆć€ć“ć®ć‚ćŸć‚Šć‹ć‚‰ę„ć¦ć„ć‚‹ę°—ćŒć—ć¾ć™ć€‚

https://github.com/prometheus/prometheus/blob/v2.9.2/vendor/github.com/prometheus/tsdb/db.go#L51

ćØ恓悍恧恧恙恭态Prometheus恮Web Console恋悉怌Status怍ā†’ć€ŒCommand-Line Flagsć€ć‚’č¦‹ć‚‹ćØć€ć‚‚ć†å°‘ć—ęŒ‡å®šć§ććć†ćŖ
ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ćŒå¤šć„ć‚ˆć†ć«č¦‹ćˆć‚‹ć®ć§ć™ćŒā€¦ć€‚

f:id:Kazuhira:20190502222000p:plain

今回ćÆ态ćØć‚Šć‚ćˆćšę·±čæ½ć„恗ćŖ恄ā€¦ć€‚

ćƒ­ćƒ¼ć‚«ćƒ«ć‚¹ćƒˆćƒ¬ćƒ¼ć‚ø恧åæ…要ćØć™ć‚‹ćƒ‡ć‚£ć‚¹ć‚Æ容量

PrometheusćÆć€ć²ćØć¤ć®ć‚µćƒ³ćƒ—ćƒ«ć‚ćŸć‚Šć€1怜2ćƒć‚¤ćƒˆć»ć©ć‚’ä½æē”Ø恙悋恝恆恧恙怂

ćŖ恮恧态PrometheusćŒč¦ę±‚ć™ć‚‹ć‚µćƒ¼ćƒćƒ¼ć®ćƒ‡ć‚£ć‚¹ć‚Æ容量ćÆ态仄äø‹ć®č؈ē®—å¼ć§ē®—å‡ŗć§ćć¾ć™ć€ćØ怂

needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample
åæ…要ćŖćƒ‡ć‚£ć‚¹ć‚Æ悵悤ć‚ŗ = äæęŒęœŸé–“ļ¼ˆē§’ļ¼‰ Ɨ ē§’ć‚ćŸć‚Šć®å–å¾—ć‚µćƒ³ćƒ—ćƒ«ę•° * ć‚µćƒ³ćƒ—ćƒ«ć‚ćŸć‚Šć®ćƒć‚¤ćƒˆę•°

1ē§’ć‚ćŸć‚Šć«å–å¾—ć™ć‚‹ć‚µćƒ³ćƒ—ćƒ«ć‚’å°ć•ćć™ć‚‹ć«ćÆć€å–å¾—ć™ć‚‹ę™‚ē³»åˆ—ćƒ‡ćƒ¼ć‚æ恮ꕰ悒ęø›ć‚‰ć™ļ¼ˆåŽé›†ć™ć‚‹ć‚æćƒ¼ć‚²ćƒƒćƒˆć‚’ęø›ć‚‰ć™ć€
ć‚æćƒ¼ć‚²ćƒƒćƒˆć‚ćŸć‚Šć®å–å¾—é …ē›®ć‚’ęø›ć‚‰ć™ļ¼‰ć‹ć€å–å¾—é–“éš”ć‚’é•·ćčØ­å®šć™ć‚‹ć‹ć®ć©ć”ć‚‰ć‹ć§ć™ć€‚

ćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆć«ć‚ˆć‚‹ćØć€å–å¾—ć—ćŸćƒ‡ćƒ¼ć‚æćÆ圧ēø®ć•ć‚Œć‚‹ćŸć‚ć€å–å¾—ć™ć‚‹é …ē›®ę•°ć‚’ęø›ć‚‰ć™ę–¹ćŒåŠ¹ęžœēš„恠ćØ恋怂

ćŖćŠć€ćƒ­ćƒ¼ć‚«ćƒ«ć‚¹ćƒˆćƒ¬ćƒ¼ć‚ø恌ē “ęć—ćŸå “åˆćÆ态仄äø‹ć®åÆ¾å‡¦ć«ćŖ悋ćØ恋怂

  • Prometheusć‚’ć‚·ćƒ£ćƒƒćƒˆćƒ€ć‚¦ćƒ³ć—ć¦ć€ćƒ‡ćƒ¼ć‚æćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖć‚’å‰Šé™¤ć™ć‚‹ļ¼ˆć“ć‚ŒćŒęœ€å–„ć‚‰ć—ć„ļ¼‰
    • 当ē„¶ć€å…Øćƒ‡ćƒ¼ć‚æć‚’å¤±ć†
  • å€‹ć€…ć®ćƒ–ćƒ­ćƒƒć‚Æćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖć‚’å‰Šé™¤ć™ć‚‹
    • å‰Šé™¤ć—ćŸćƒ–ćƒ­ćƒƒć‚Æć«å«ć¾ć‚Œć‚‹ć€2ę™‚é–“ć®ćƒ‡ćƒ¼ć‚æćÆå¤±ć†

ć©ć”ć‚‰ć«ć›ć‚ˆć€Prometheusć®ćƒ­ćƒ¼ć‚«ćƒ«ć‚¹ćƒˆćƒ¬ćƒ¼ć‚øćÆé•·ęœŸé–“ć®ćƒ‡ćƒ¼ć‚æäæå­˜ć‚’ę„å›³ć—ć¦ć„ćŖ恄恓ćØćŒćƒ‰ć‚­ćƒ„ćƒ”ćƒ³ćƒˆć§ćÆå¼·čŖæć•ć‚Œć¦ć„ć¾ć™ć€‚

ćƒ‡ćƒ¼ć‚æ恮äæęŒęœŸé–“ć‚’ęŒ‡å®šć—ć¦ćæ悋

恝悌恧ćÆć€ćƒ‡ćƒ¼ć‚æ恮äæęŒęœŸé–“ć‚’ēŸ­ć‚ć«ć—ć¦ć€å‹•ä½œć‚’ē¢ŗčŖć—恦ćæć¾ć—ć‚‡ć†ć€‚

仄äø‹ć®ć‚ˆć†ćŖć€ćƒ‡ćƒ¼ć‚æć®å–å¾—é–“éš”ć‚’ēŸ­ć‚ć«ć—ćŸPrometheus恮čØ­å®šćƒ•ć‚”ć‚¤ćƒ«ć‚’ē”Øę„ć—ć€čµ·å‹•ć€‚
prometheus.yml

global:
  scrape_interval:     1s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 1s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

ćƒ‡ćƒ¼ć‚æć®å–å¾—åÆ¾č±”ćÆć€ćƒ‡ćƒ•ć‚©ćƒ«ćƒˆļ¼ˆč‡Ŗ分č‡Ŗčŗ«ļ¼‰ć®ć¾ć¾ć§ć™ć€‚

čµ·å‹•ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ćØć—ć¦ć€ę—©ć‚ć«ę¶ˆćˆć‚‹ć“ćØ悒ē¢ŗčŖć—ćŸćć¦ć€Œ--storage.tsdb.retention.time怍悒3åˆ†ć«čØ­å®šć—ć¾ć—ćŸć€‚
恂ćØ态怌--storage.tsdb.min-block-durationć€ć‚’ęŒ‡å®šć—ć¦ć„ć¾ć™ćŒć€ćŖć‚“ć§ć“ć‚Œć‚’ä»˜ć‘ćŸć‹ćÆć¾ćŸå¾Œć§ā€¦ć€‚

$ ./prometheus --storage.tsdb.retention.time 3m --storage.tsdb.min-block-duration 1m
level=info ts=2019-05-02T14:09:07.183Z caller=main.go:321 msg="Starting Prometheus" version="(version=2.9.2, branch=HEAD, revision=d3245f15022551c6fc8281766ea62db4d71e2747)"

Web Consoleć§ć€čµ·å‹•ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ć‚’čŖč­˜ć—恦恄悋恓ćØ悒ē¢ŗčŖć€‚

f:id:Kazuhira:20190502231638p:plain

ćØ悊恂恈恚态Web Console恧CPUä½æē”ØꙂ間ļ¼ˆprocess_cpu_seconds_totalļ¼‰ć‚’č”Øē¤ŗ怂10åˆ†é–“ć®č”Øē¤ŗć§č¦‹ć¦ć„ć¾ć™ć€‚

f:id:Kazuhira:20190502231018p:plain

1分ēµŒéŽć™ć‚‹ćØ态恓悓ćŖ惭悰恌å‡ŗåŠ›ć•ć‚Œ

level=info ts=2019-05-02T14:10:44.219Z caller=compact.go:499 component=tsdb msg="write block" mint=1556806153112 maxt=1556806200000 ulid=01D9WE45RQ73CF18Y6J2BPMN1Q duration=100.343317ms
level=info ts=2019-05-02T14:10:44.237Z caller=head.go:540 component=tsdb msg="head GC completed" duration=1.785374ms

惖惭惃ć‚ÆćŒć§ćć¾ć™ć€‚

$ find data -type f
data/wal/00000000
data/lock
data/01D9WE45RQ73CF18Y6J2BPMN1Q/meta.json
data/01D9WE45RQ73CF18Y6J2BPMN1Q/index
data/01D9WE45RQ73CF18Y6J2BPMN1Q/tombstones
data/01D9WE45RQ73CF18Y6J2BPMN1Q/chunks/000001

ć“ć®ćƒ­ć‚°ćŒ4回å‡ŗåŠ›ć•ć‚Œć‚‹ć¾ć§ćÆ态惖惭惃ć‚Æćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖ恌čæ½åŠ ć•ć‚Œć¦ć„ćć¾ć™ć€‚

level=info ts=2019-05-02T14:10:44.219Z caller=compact.go:499 component=tsdb msg="write block" mint=1556806153112 maxt=1556806200000 ulid=01D9WE45RQ73CF18Y6J2BPMN1Q duration=100.343317ms
level=info ts=2019-05-02T14:10:44.237Z caller=head.go:540 component=tsdb msg="head GC completed" duration=1.785374ms
level=info ts=2019-05-02T14:11:30.196Z caller=compact.go:499 component=tsdb msg="write block" mint=1556806200000 maxt=1556806260000 ulid=01D9WE5JP7Y6FFGZMKKS1KK2AD duration=77.507689ms
level=info ts=2019-05-02T14:11:30.217Z caller=head.go:540 component=tsdb msg="head GC completed" duration=1.549613ms
level=info ts=2019-05-02T14:12:30.229Z caller=compact.go:499 component=tsdb msg="write block" mint=1556806260000 maxt=1556806320000 ulid=01D9WE7D97CMB7R8RS12G9E9YW duration=109.711593ms
level=info ts=2019-05-02T14:12:30.251Z caller=head.go:540 component=tsdb msg="head GC completed" duration=1.421509ms
level=info ts=2019-05-02T14:13:30.197Z caller=compact.go:499 component=tsdb msg="write block" mint=1556806320000 maxt=1556806380000 ulid=01D9WE97W9WEKZCQ0957KVH2GX duration=75.948389ms
level=info ts=2019-05-02T14:13:30.215Z caller=head.go:540 component=tsdb msg="head GC completed" duration=1.492116ms

恓悓ćŖę„Ÿć˜ć«ć€‚

$ find data -type f
data/wal/00000000
data/01D9WE7D97CMB7R8RS12G9E9YW/meta.json
data/01D9WE7D97CMB7R8RS12G9E9YW/index
data/01D9WE7D97CMB7R8RS12G9E9YW/tombstones
data/01D9WE7D97CMB7R8RS12G9E9YW/chunks/000001
data/lock
data/01D9WE45RQ73CF18Y6J2BPMN1Q/meta.json
data/01D9WE45RQ73CF18Y6J2BPMN1Q/index
data/01D9WE45RQ73CF18Y6J2BPMN1Q/tombstones
data/01D9WE45RQ73CF18Y6J2BPMN1Q/chunks/000001
data/01D9WE5JP7Y6FFGZMKKS1KK2AD/meta.json
data/01D9WE5JP7Y6FFGZMKKS1KK2AD/index
data/01D9WE5JP7Y6FFGZMKKS1KK2AD/tombstones
data/01D9WE5JP7Y6FFGZMKKS1KK2AD/chunks/000001
data/01D9WE97W9WEKZCQ0957KVH2GX/meta.json
data/01D9WE97W9WEKZCQ0957KVH2GX/index
data/01D9WE97W9WEKZCQ0957KVH2GX/tombstones
data/01D9WE97W9WEKZCQ0957KVH2GX/chunks/000001

f:id:Kazuhira:20190502231415p:plain

恓恓恧态5回ē›®ć®ć‚³ćƒ³ćƒ‘ć‚Æć‚·ćƒ§ćƒ³ćŒčµ·ć“ć‚‹ćØ

level=info ts=2019-05-02T14:10:44.219Z caller=compact.go:499 component=tsdb msg="write block" mint=1556806153112 maxt=1556806200000 ulid=01D9WE45RQ73CF18Y6J2BPMN1Q duration=100.343317ms
level=info ts=2019-05-02T14:10:44.237Z caller=head.go:540 component=tsdb msg="head GC completed" duration=1.785374ms
level=info ts=2019-05-02T14:11:30.196Z caller=compact.go:499 component=tsdb msg="write block" mint=1556806200000 maxt=1556806260000 ulid=01D9WE5JP7Y6FFGZMKKS1KK2AD duration=77.507689ms
level=info ts=2019-05-02T14:11:30.217Z caller=head.go:540 component=tsdb msg="head GC completed" duration=1.549613ms
level=info ts=2019-05-02T14:12:30.229Z caller=compact.go:499 component=tsdb msg="write block" mint=1556806260000 maxt=1556806320000 ulid=01D9WE7D97CMB7R8RS12G9E9YW duration=109.711593ms
level=info ts=2019-05-02T14:12:30.251Z caller=head.go:540 component=tsdb msg="head GC completed" duration=1.421509ms
level=info ts=2019-05-02T14:13:30.197Z caller=compact.go:499 component=tsdb msg="write block" mint=1556806320000 maxt=1556806380000 ulid=01D9WE97W9WEKZCQ0957KVH2GX duration=75.948389ms
level=info ts=2019-05-02T14:13:30.215Z caller=head.go:540 component=tsdb msg="head GC completed" duration=1.492116ms
level=info ts=2019-05-02T14:14:30.248Z caller=compact.go:499 component=tsdb msg="write block" mint=1556806380000 maxt=1556806440000 ulid=01D9WEB2F4SWD5FDP5EBV7BD0E duration=132.373486ms
level=info ts=2019-05-02T14:14:30.270Z caller=head.go:540 component=tsdb msg="head GC completed" duration=1.201336ms

惖惭惃ć‚Æćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖćŒå¢—ćˆćŖ恏ćŖć‚Šć¾ć™ć€‚ć€Œ01D9WE45RQ73CF18Y6J2BPMN1Q怍ćØć„ć†åå‰ć®ćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖ恌恄ćŖ恏ćŖć‚Šć¾ć—ćŸć€‚

$ find data -type f
data/01D9WEB2F4SWD5FDP5EBV7BD0E/meta.json
data/01D9WEB2F4SWD5FDP5EBV7BD0E/index
data/01D9WEB2F4SWD5FDP5EBV7BD0E/tombstones
data/01D9WEB2F4SWD5FDP5EBV7BD0E/chunks/000001
data/wal/00000000
data/01D9WE7D97CMB7R8RS12G9E9YW/meta.json
data/01D9WE7D97CMB7R8RS12G9E9YW/index
data/01D9WE7D97CMB7R8RS12G9E9YW/tombstones
data/01D9WE7D97CMB7R8RS12G9E9YW/chunks/000001
data/lock
data/01D9WE5JP7Y6FFGZMKKS1KK2AD/meta.json
data/01D9WE5JP7Y6FFGZMKKS1KK2AD/index
data/01D9WE5JP7Y6FFGZMKKS1KK2AD/tombstones
data/01D9WE5JP7Y6FFGZMKKS1KK2AD/chunks/000001
data/01D9WE97W9WEKZCQ0957KVH2GX/meta.json
data/01D9WE97W9WEKZCQ0957KVH2GX/index
data/01D9WE97W9WEKZCQ0957KVH2GX/tombstones
data/01D9WE97W9WEKZCQ0957KVH2GX/chunks/000001

恓恮ēµęžœć€ęœ€åˆć®ę™‚é–“ć®ćƒ‡ćƒ¼ć‚æ恌ćŖ恏ćŖć‚Šć¾ć™ć€‚ć€Œ01D9WE45RQ73CF18Y6J2BPMN1Q怍ćØ恄恆恮ćÆć€ęœ€åˆć«ä½œć‚‰ć‚ŒćŸ
ćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖ恧恗恟怂

f:id:Kazuhira:20190502231541p:plain

ćØ恄恆悏恑恧态怌--storage.tsdb.retention.timeć€ć§ęŒ‡å®šć—ćŸäæęŒęœŸé–“ć‚’č¶ŠćˆćŸćƒ‡ćƒ¼ć‚æćÆå‰Šé™¤ć•ć‚Œć¦ć„ć‚‹ć“ćØ恌ē¢ŗčŖć§ćć¾ć—ćŸć€‚

ćØć“ć‚ć§ć€ä»Šå›žć€čµ·å‹•ć‚Ŗćƒ—ć‚·ćƒ§ćƒ³ć«ć€Œ--storage.tsdb.retention.time怍恠恑恧ćÆćŖćć¦ć€Œ--storage.tsdb.min-block-duration怍悂
ęŒ‡å®šć—ć¾ć—ćŸć€‚

$ ./prometheus --storage.tsdb.retention.time 3m --storage.tsdb.min-block-duration 1m

怌--storage.tsdb.min-block-durationć€ć‚’ęŒ‡å®šć—ćŸć“ćØ恧态惖惭惃ć‚Æ恮ēÆ„å›²ćŒ1åˆ†ć«ćŖć£ć¦ć„ć¾ć™ć€‚

ęœ€åˆć€ä»„äø‹ć®ć‚ˆć†ć«ć€Œ--storage.tsdb.retention.timeć€ć‚’ęŒ‡å®šć—ćŸć ć‘ć ćØćƒ‡ćƒ¼ć‚æćŒć¾ć£ćŸćę¶ˆćˆćšć€ć€Œć‚‚ć—ć‹ć—ć¦WAL恫恄悋間恮
ćƒ‡ćƒ¼ć‚æćÆåÆ¾č±”ć«å…„ć‚‰ćŖ恄恮恧ćÆļ¼Ÿć€ćØę€ć„ć€å¤‰ę›“ć«č‡³ć‚Šć¾ć—ćŸć€‚

$ ./prometheus --storage.tsdb.retention.time 3m

ć¤ć¾ć‚Šć€é•·ę™‚é–“å¾…ć¤ć®ćŒå«Œć§ć€ę„µē«Æ恫ēŸ­ć„Ꙃ間悒retention恫čØ­å®šć—ćŸć®ćŒå®Œå…Øć«č£ē›®ć«å‡ŗ恟悈恆恧恙怂

å®Ÿéš›ć€å‰Šé™¤åÆ¾č±”ćÆ惖惭惃ć‚Æå˜ä½ć®ć‚ˆć†ć§ć™ć€‚

https://github.com/prometheus/prometheus/blob/v2.9.2/vendor/github.com/prometheus/tsdb/db.go#L670

惖惭惃ć‚Æćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖå†…ć®meta.json悒見悋ćØ态minTimećØmaxTime恧态恩恮Ꙃ間恮ēÆ„å›²ć®ćƒ‡ćƒ¼ć‚æćŒå«ć¾ć‚Œć¦ć„ć‚‹ć‹ē¢ŗčŖć™ć‚‹ć“ćØ恌
ć§ćć¾ć™ć€‚
data/01D9WEEQN5CH9F53GAFCN3G14E/meta.json

{
    "ulid": "01D9WEEQN5CH9F53GAFCN3G14E",
    "minTime": 1556806500000,
    "maxTime": 1556806560000,
    "stats": {
        "numSamples": 27120,
        "numSeries": 452,
        "numChunks": 452,
        "numBytes": 77220
    },
    "compaction": {
        "level": 1,
        "sources": [
            "01D9WEEQN5CH9F53GAFCN3G14E"
        ]
    },
    "version": 1
}

恓恓恋悉悏恋悋恓ćØćÆ态怌--storage.tsdb.min-block-duration怍悒čØ­å®šć™ć‚‹ć“ćØćÆ恝恆ćŖć„ć‹ć‚‚ć—ć‚Œć¾ć›ć‚“ćŒć€
怌--storage.tsdb.retention.time怍ćÆ惖惭惃ć‚Æćƒ‡ć‚£ćƒ¬ć‚Æ惈ćƒŖ恧ꌁ恤ēƄ囲ļ¼ˆćƒ‡ćƒ•ć‚©ćƒ«ćƒˆ2Ꙃ間ļ¼‰ć®å€ę•°ć§ćŖć‘ć‚Œć°ę„å‘³ćŒćŖ恄
ćØ恄恆恓ćØ恧恙恭怂

č¦šćˆć¦ćŠćć¾ć—ć‚‡ć†ć€‚

恄恄ē¢ŗčŖć«ćŖć‚Šć¾ć—ćŸć€‚