Site up. Cards up.
19 of 84 pipes on time. 65 behind.
| pipe | newest data | limit | state | detail |
|---|---|---|---|---|
| RH archive: rows deleted from hot after R2 verify (nightly 02-04Z) | 40.7 h | 30.0 h | behind | rh_archive: last 24 h rh.curve_state in ~871,651 / out 375,170; rh.trade_tape in ~836,461 / out 252,477; 0.13 GiB ... |
| Solana archive: rows deleted from hot (nightly) | 41.0 h | 30.0 h | behind | sol_archive: last 24 h observe.curve_state in unknown / out 220,000; observe.trade_tape in ~11,108,028 / out 9,920,000; 1.24 GiB ... |
| Fix auditor (tools/gt_audit.py every 6 h; latest.json summary) | 41.6 h | 7.0 h | behind | 4 verified / 5 submitted / 11 open / 10 regressed |
| Base data spot-fill (pump.fun dev lists) | 97 s | 60 min | on time | queue 0; devs held 21470; pump.fun 6124 calls today (3 refused) of 20000 |
| Server disk free (/var/gt, floor 30 GiB) | 45 s | 5 min | on time | C: free 212.6 GiB (target 30, fail < 17); last 4 h -0.15 GiB/h (213.2 at 17:00Z); 24 ... |
| Server disk on course to cross 17 GiB within 24 h at the last day's rate | 45 s | 5 min | on time | C: free 212.6 GiB, -4.2 GiB/day over 25 h -> crosses at 17 GiB in 1118 h |
| C: housekeeping (every 4 h + after heavy jobs), 30 GB floor | never | 5.0 h | behind | BELOW FLOOR: C: free 18.2 GB (floor 30) - reclaimed 0.00 GB this run (gt_bg_run parity) |
| Sentinel candles (while positions are open) | 5 d | 5 min | behind | no open positions (nothing to candle) |
| Cloudflare D1 storage, all DBs (alarm > 5 GB billed) | never | 6.0 h | behind | 11.50 GB total: aa778e38 6.47 GB, f696f8c6 5.01 GB, c9aebfc7 0.01 GB |
| Cloudflare KV storage (alarm > 1 GB billed or growing) | never | 6.0 h | behind | 4.46 GB, 2,102,740 keys (2026-10-17), +0.0 MB vs the day before |
| Cloudflare KV writes / UTC day (alarm > 25k: 1M/month free) | 40.8 h | 6.0 h | behind | today 2,011 so far (~11,645 projected), 12,161 yesterday (2026-10-16) |
| Chainstack request units (alarm at 80% projected before reset) | 38.6 h | 60 min | behind | local meter: ~66,929 RU since 10-17 (estimate, 0.3% of 20,000,000; lower bound 66,873); guard 12,000,000; 4 callers unmetered |
| Chainstack RU this period, local meter (guard 16M, 12M while any caller is unmetered) | 40.8 h | 2.0 h | behind | ~48,228 RU since 10-17 (estimate, 0% of 20M; lower bound 48,159 from the 10-03 billing page + meter); ... |
| Claude layer timeouts (last hour; alarm over 10% of >= 20 calls) | 5 d | 2.0 h | behind | timeouts 0 of 0 calls in the last hour (0.0%); answered p50 - s, p95 - s |
| Creator backfill (first-look cards counted from the chain; R12) ran in the last 45 min | 38.7 h | 45 min | behind | last run read 1 creator(s), 13 queued |
| D1 rows written, Cloudflare's count (tools/gt_d1_rows.py, hourly) | 39.1 h | 2.0 h | behind | month to date 25,493,798 rows written (2026-10-01..2026-10-17, calendar month assumed: renewal day unknown); today pace 484,912/day; 7-day mean ... |
| D: mounted + writable (4 KiB write+fsync under 10 s) | 38.6 h | 5 min | behind | D: write+fsync 4 KiB in 82 ms |
| DAILY POSTS on TG channel: daily boards + trending + paired (tools/gt_trending_post.py) | never | 26.0 h | behind | 2026-10-16 boards refused: coverage_top is over the line (robinhood 3 of 50, solana 1 of 50, pumpf · trending HELD: ... |
| DAILY POSTS on TG group: daily boards + trending + paired (tools/gt_trending_post.py) | never | 26.0 h | behind | 2026-10-16 boards refused: coverage_top is over the line (robinhood 3 of 50, solana 1 of 50, pumpf · trending HELD: ... |
| DAILY POSTS on X (@groundtruth_now): daily boards + trending + paired (tools/gt_trending_post.py) | never | 26.0 h | behind | 2026-10-16 boards REFUSED: X posting is paused (D:\Users\leest\AI\SolBotv2\_agent\X_PAUS · trending HELD: $AI: ONE ANSWER: dev count 0 OF ... |
| DB integrity (nightly PRAGMA quick_check, one DB a night, C: first) | 3 d | 36.0 h | behind | last pipeline_state.db ok in 0.0s; next observe_capture.db |
| Disk reclaim completed (daily; alarm past 26 h) | 41.0 h | 26.0 h | behind | last vacuum that returned space 10-17 03:54Z; last run 10-17 03:51Z: vacuum 0.07 GiB, WAL truncate 0.01 GiB |
| Published figures | 2 d | 9.0 h | behind | [upstream_silent] /api/figures generated_at_utc |
| FIX-API-DOORS: Every paid / creator door sells the live card (one resolver), and no listing carries the withdrawn stamp claim | never | 7.0 h | behind | regressed -- L1 fail: rc 1, missing ['LISTINGS OK'] |
| FIX-ARCHIVE-SOL: The nightly Solana archive moves rows out of hot every night (status row archive_sol not 'upstream silent') | never | 7.0 h | behind | open -- S1 fail: rc 1, missing ['ARCHIVE_SOL OK'] |
| public audit published (every 6 h, alarm past 7 h) | 96 min | 7.0 h | on time | generated 2026-10-18T19:20:18Z |
| FIX-AUTOPSY-DAILY: Today's autopsy publishes every day (09-27's KV put failed silently) | never | 7.0 h | behind | regressed -- all checks pass |
| FIX-CARD-FRESH: The card is current at mint: last mint / launched / BONDED from the live feed, fresh ... | never | 7.0 h | behind | regressed -- B1 fail: rc 1, missing ['CARD_FRESH OK'] |
| FIX-CARDS-LIVE: Trending coins' cards tell the truth today (name, no dead headline on a live coin, pool depth, no error) | never | 7.0 h | behind | regressed -- L1 fail: 100 cards, 102 failing |
| FIX-CF-COST: Cloudflare cost watched: KV daily write cap 25k in gt_kv, KV/D1 usage rows fresh from Cloudflare GraphQL | never | 7.0 h | behind | submitted -- K1 fail: rc 1, missing ['CFCOST OK'] |
| FIX-COUNTED-TO: Home 'counted to' stays fresh without the pass: GT Counted To publishes the creator rap sheets every 30 ... | never | 7.0 h | behind | open -- TAMPER fail: acceptance changed |
| FIX-COVERAGE: The top 50 coins per chain, from an outside list, answer on GT as a visitor: dev ... | never | 7.0 h | behind | open -- TAMPER fail: acceptance changed |
| FIX-DISK: C: stops sliding: archive deletes land every day and the hot DBs stop growing | never | 7.0 h | behind | open -- D2 fail: 0 chunks |
| FIX-EVERY-BACKOFF: A gt_every stage backing off after a failure reads FAILED (exit 75), never DONE | never | 7.0 h | behind | regressed -- B1 fail: rc 1, missing ['BACKOFF TEST: PASS'] |
| FIX-EXT-104: Extension 1.0.4 is built with GMGN + Axiom on both chains and is in the Chrome Web ... | never | 7.0 h | behind | open -- V1 fail: rc 1, missing ['V1 OK'] |
| FIX-FEED-NO-USB-WAIT: No feed process holds a file on a USB drive (D: Maxone, E: Samsung flash, Z: GT2TB) | never | 7.0 h | behind | submitted -- U1 fail: rc 1, missing ['FEED_USB OK'] |
| FIX-FEEDS: Capture feeds stay under their SLA and the heal ladder, not Lee, fixes what it can | never | 7.0 h | behind | submitted -- F1 fail: hot_list 310 min, site_live 910 min, vps_claims 349 min, vps_rh_cover 14 min |
| FIX-FIGURES-behind: Published figures are fresh | never | 7.0 h | behind | open -- G1 fail: 3.6 h (2026-10-18T15:46:18Z) |
| FIX-GATE: No smoke-gate alarm lives past 48 h without Lee's written retirement | never | 7.0 h | behind | open -- S1 fail: 12 FAIL row(s); oldest 61 h |
| FIX-KV: Cloudflare KV write spend stays inside the budget | never | 7.0 h | behind | submitted -- R1 fail: unreadable |
| FIX-KV-WORKER: The API worker's per-request KV writes (usage/rate counters, TRUTHBOT caches) stay small | never | 7.0 h | behind | open -- W1 fail: unreadable |
| FIX-LANDING-C: Solana capture writes only the small C: landing DB; the hourly merge moves rows to GT2TB and ... | never | 7.0 h | behind | submitted -- L1 fail: rc 1, missing ['LANDING OK'] |
| FIX-NAV-390: Top nav fits a 390 px phone: every link readable, nav and page no wider than the screen | never | 7.0 h | behind | regressed -- N1 fail: rc 1, missing ['NAV390 OK'] |
| FIX-PIPS: The card envelope (hero fraction + pips, clock, snapshot row) renders on a WARRANT, a STUB ... | never | 7.0 h | behind | open -- P1 fail: rc 1, missing ['PIPS OK'] |
| FIX-POSTER: Exactly one X poster, never a slot posted twice | never | 7.0 h | behind | regressed -- P2 fail: unreadable |
| FIX-R28-KOL-503: /kol never 503s a visitor: the KOL board is on the store, board pages with their board ... | never | 7.0 h | behind | submitted -- TAMPER fail: acceptance changed |
| FIX-RESTART: Pipeline back on GT2TB: cold DBs resolve to GT2TB by label and a pass finished in the ... | never | 7.0 h | behind | submitted -- P1 fail: rc 1, missing ['RESTART OK'] |
| FIX-RH-TRADING-AGAIN: A resolved RH coin trading again reads 'recorded died <date> . trading again' with its real name, ... | never | 7.0 h | behind | regressed -- all checks pass |
| FIX-SENTINEL-EXITS: Sentinel paper books exits at the real quote: price guard, own-deposit reserve cap, no-bid at the reserves' ... | never | 7.0 h | behind | regressed -- X1 fail: rc 1, missing ['SENTINEL_A3 OK'] |
| FIX-SENTINEL-REMOVE: Sentinel gone: no process or task on the PC, /sentinel 301 to home, /api/sentinel 410 on the served site | never | 7.0 h | behind | regressed -- S1 fail: rc 1, missing ['SENTINEL_PC OK'] |
| FIX-SIGNALS: figures:signals:v1 republishes on its 8 h cadence | never | 7.0 h | behind | open -- all checks pass |
| FIX-SKEPTIC: Every visitor-facing surface survives a skeptic reading it beside its own chart, daily and before every auto-post | never | 7.0 h | behind | open -- K1 fail: rc 1, missing ['SKEPTIC OK'] |
| FIX-TG-DAILY: The daily card posts in the Telegram group every day | never | 7.0 h | behind | regressed -- T1 fail: FAIL |
| FIX-X-POSTS: The X poster posts every day (the 13:00 ET autopsy did not post 09-26 or 09-27) | never | 7.0 h | behind | regressed -- X1 fail: unreadable |
| HOT LIST: the 20 most traded coins on each chain name a dev on GT (tools/gt_hot_list.py, hourly) | 49 min | 2.3 h | on time | HOT LIST: RH 20/20, Solana 18/18 name a dev |
| Intel sweep (local, 07:52 + 15:52 ET; latest.md with a status line) | 2 d | 14.0 h | behind | GroundTruth sweep — 2026-10-16 15:52 ET — 4 items — 22 searches |
| capture landing (C:) -> big DB (GT2TB) merge, hourly | 50 min | 2.0 h | on time | observe_capture.db OK: trade_tape mark 549054439 moved 414747; curve_state mark 125541458 moved 123774; curve_registry mark 322518 moved 2216; meta ... |
| Card mint overlay (API cron every 5 s: pump.fun New board -> D1 mint_since_census; /v1/flag ticker_asof) | 38.6 h | 2 min | behind | /v1/flag ticker_asof (newest mint the feed has seen) |
| drop_dup_ix + db_stats ran tonight (checked 05:10Z) | 17.8 h | 26.0 h | on time | both ran tonight: drop_dup_ix 10-18 04:10Z · db_stats 10-18 03:10Z |
| ONE ANSWER: one fact, one value at every door (tools/gt_one_answer.py, every 30 min) | 14 min | 45 min | on time | ONE_ANSWER OK 15 subjects, 108 facts |
| PAGE SPEED: first byte, 9 nav pages + a held Solana and Robinhood pull, alarm over 3 s (hourly) | 43 min | 3.0 h | on time | home 0.8s · kol 0.1s · watchlist 0.2s · memestocks 0.1s · board 0.6s · extension 0.0s · ... |
| Paper publisher (/api/sentinel/live) | 5 d | 90 s | behind | [process_dead] /api/sentinel/live age |
| Paper runner, Robinhood Chain | 5 d | 90 s | behind | [process_dead] runner payload written |
| Paper runner, Solana | 6 d | 90 s | behind | [process_dead] runner payload written; it reports its tape behind since 17:10Z |
| One pipeline stage holding pipeline.lock (alarm past 3 h) | 40.6 h | 3.0 h | behind | stage db_stats holds pipeline.lock since 04:19Z (pass pid 28428) |
| Pipeline pass (end to end; passes end every ~78-104 min, measured 09-26) | 43.3 h | 3.0 h | behind | [upstream_silent] last pass ended DEGRADED |
| Chat -> Code relay watcher (_relay/status.json) | 40.6 h | 2 min | behind | [upstream_silent] watcher idle (pid 11912) |
| RH adapter write (events.jsonl) | 90 s | 5 min | on time | newest adapter write (events.jsonl) |
| RH spool backlog (C: -> events.jsonl on D:) | 40.6 h | 15 min | behind | [disk_io] 111 segment(s), 28.8 MB waiting on C: for events.jsonl on D: |
| RH tape (rh_capture.db) | 90 s | 7 min | on time | newest RH trade_tape row (rh_capture.db) |
| RH upstream (Chainstack 429s / errors, failover state) | 46 s | 15 min | on time | 429s 0 / errors 0 in 15 min; on chainstack; lag 430 blocks |
| v3/v2 SWEEP: every Robinhood coin's v3 / v2 pools read (tools/gt_rh_pools.py --v3_sweep, hourly) | 9 d | 3.0 h | behind | v3/v2 sweep: 1,054,545 of 1,054,545 coins · outcomes moved 7 (+4) · dev counts moved 6 (+4) |
| SITE LIVE: the gate-mode skeptic on live (0 causes, 0 unreviewed) | never | 26.0 h | behind | 36 surface(s) unreviewed (reviewer error: assistant_error:billing_error) |
| Solana tape | 46 s | 60 s | on time | newest Solana trade_tape row |
| Solana upstream (stream slot lag vs head) | 72 s | 5 min | on time | stream slot lag 0 vs publicnode head; getSlot errors 0/60 in 15 min; reconnects 2 since start |
| RH venues marked dormant (still watched; reactivate + DM on the next launch) | 38.6 h | 15 min | behind | rh-pools-trade reactivated 09-28 15:15 ET (launch 0x6280857b01) |
| Public coverage claims (VPS) | 11 min | 40 min | on time | 58 of 58 claims pass |
| RH coverage, 20 newest per venue (VPS) | 11 min | 40 min | on time | 220 of 220 covered on 11 venues |
| RH launch index (VPS) | 36 s | 30 min | on time | 1,447,983 launches; newest launch 0 min before head; +LONG (Doppler) 55,Pons 15,Uniswap v4 20 this run; 165 pushed; ... |
| RH outcome set age (VPS) | never | 2.0 h | behind | published_asof not read |
| RH top 20 by volume (VPS) | 11 min | 40 min | on time | source GeckoTerminal top pools by 24 h volume: 4 of 16 top-volume coins covered; not covered, launched via: ... |
| pump.fun cards, 20 newest graduates (VPS) | 11 min | 40 min | on time | 20 of 20 cards ok |
| X poster (live posts) | 38.6 h | 24.0 h | behind | not live (no _agent/xpost_LIVE) |
| D1 reads a visitor waits on (p95 <= 250 ms per route) | 0 s | 60 min | behind | p95 over 250 ms: /api/boards/kol GT_DATA p95 400 ms (n=3124); /v1/flag GT_DATA p95 400 ms (n=1784); /v1/flag LEDGER ... |
| pipe | times | behind for | now |
|---|---|---|---|
| page_speed | 2 | 164 min | fixed |
| vps_claims | 6 | 439 min | fixed |
| hot_list | 3 | 976 min | fixed |
| vps_rh_cover | 6 | 97 min | fixed |
| site_live | 1 | 1006 min | still behind |
| vps_rh_top | 1 | 16 min | fixed |
| fix_rh_trading_again | 1 | 2130 min | still behind |
| fix_every_backoff | 1 | 2130 min | still behind |
| fix_sentinel_remove | 1 | 2130 min | still behind |
| fix_nav_390 | 1 | 2130 min | still behind |
| vps_sol_cards | 1 | 29 min | fixed |
| c_free_24h | 2 | 139 min | fixed |
| c_free | 3 | 339 min | fixed |
| pipeline_pass | 4 | 2511 min | still behind |
| rh_spool | 1 | 2422 min | still behind |
| chainstack_quota | 52 | 100 min | fixed |
| rh_tape | 16 | 179 min | fixed |
| relay | 3 | 2436 min | still behind |
| sol_upstream | 41 | 91 min | fixed |
| x_poster | 1 | 0 min | fixed |
| archive_sol | 13 | 9 min | fixed |
| pipeline_lock | 3 | 2 min | fixed |
| disk_reclaim | 1 | 0 min | fixed |
| db_integrity | 1 | 2 min | fixed |
| archive_rh | 7 | 7 min | fixed |
| fix_r28_kol_503 | 1 | 2485 min | still behind |
| sol_tape | 18 | 31 min | fixed |
| figures | 1 | 2547 min | still behind |
| rh_upstream | 3 | 5 min | fixed |
| one_answer | 2 | 480 min | fixed |
| audit | 1 | 431 min | fixed |
| d1_rows | 1 | 61 min | fixed |
| creator_backfill | 1 | 9 min | fixed |
Checked 2026-10-18T20:56:02.584Z · refreshes every 30 s · JSON · Boring stuff →