/dev/yukarinoki

Building a Volatility Oracle Prototype: Computing Realized Volatility from Orderbook Data and Serving It On-Chain

Fri, 10 Apr 2026 08:45:00 GMT

日本語要約: DeFiに不足しているボラティリティデータを提供するオンチェーンOracleのプロダクトコンセプトです。オーダーブックデータからrealized volatilityを算出し、スマートコントラクトに配信する仕組みをTypeScriptプロトタイプとSolidityインターフェースのコード例とともに提案します。

I’ve been thinking about a gap in DeFi infrastructure that seems surprisingly under-served: volatility data. We have robust price oracles, but almost nothing purpose-built for serving volatility estimates on-chain. This post walks through a product concept I’ve been prototyping — a volatility oracle that computes realized vol from high-frequency orderbook data and makes it available to smart contracts.

The Problem: DeFi Needs Volatility, Not Just Price

Price oracles are table stakes at this point. Every lending protocol, every DEX, every derivatives platform relies on some form of price feed. But volatility? That’s a different story entirely.

Think about who actually needs vol data on-chain:

Options protocols are the obvious case. You can’t price options without implied volatility, and you can’t calibrate your vol surface without realized vol as a reference. Protocols that sell options on-chain are essentially flying blind or relying on off-chain computation that users have to trust.

Risk management in lending protocols is another big one. Liquidation thresholds are typically static — you set a collateral ratio of 150% and call it a day. But a 150% ratio means very different things when ETH is trading at 10% annualized vol versus 80%. A vol-aware liquidation engine could dynamically adjust thresholds, reducing unnecessary liquidations during calm markets and protecting the protocol during volatile ones.

Liquidation engines themselves could benefit from vol-adjusted parameters. If you know volatility is spiking, you can widen your liquidation buffers proactively rather than reacting after the damage is done.

The current state of affairs is that protocols either hardcode assumptions about volatility, compute it off-chain in trusted backends, or simply ignore it. None of these are great.

The Concept

The idea is straightforward: ingest high-frequency orderbook data from major exchanges, compute realized volatility using multiple estimators, and push the results on-chain as an oracle feed that any smart contract can consume.

Here’s the high-level architecture:

Binance WS → Aggregator (0.5s VWAP) → Vol Calculator → Oracle Contract → DeFi Consumers

The key insight is that orderbook microstructure data — the actual bids and asks updating dozens of times per second — gives us much better volatility estimates than the traditional approach of using daily OHLC candles. We’re capturing information about price dynamics that gets lost when you downsample to hourly or daily bars.

The aggregator computes volume-weighted average prices at 500ms intervals, giving us a clean price series that incorporates the full depth of the orderbook rather than just last-trade prices. This feeds into the volatility calculator, which maintains rolling windows and outputs annualized vol estimates. Those estimates get pushed to a Solidity oracle contract whenever they change by more than a configurable threshold.

Volatility Computation: Multiple Estimators

Not all volatility estimators are created equal. I’m implementing several and letting consumers choose which one suits their use case.

Standard Realized Volatility

The classic approach. Take a rolling window of log returns, compute the standard deviation, annualize it.

$$\sigma{realized} = \sqrt{\frac{252}{n} \sum{i=1}^{n} (r_i - \bar{r})^2}$$

Simple, well-understood, but inefficient — it only uses closing prices and throws away intra-period information.

Parkinson Estimator

Uses the high-low range within each period, which captures more information about price dynamics:

$$\sigma{parkinson} = \sqrt{\frac{1}{4n \ln 2} \sum{i=1}^{n} (\ln Hi - \ln Li)^2}$$

This is roughly 5x more efficient than the close-to-close estimator for the same number of observations. The tradeoff is that it assumes continuous trading (no gaps), which is fine for crypto markets.

Garman-Klass Estimator

Goes further by using full OHLC data:

$$\sigma{GK} = \sqrt{\frac{1}{n} \sum{i=1}^{n} \left[ \frac{1}{2}(\ln Hi - \ln Li)^2 - (2\ln 2 - 1)(\ln Ci - \ln Oi)^2 \right]}$$

Even more efficient, roughly 8x better than close-to-close for the same observation count.

Why Microstructure Data Matters

With traditional OHLC data from daily candles, you’re working with maybe 30 data points for a monthly vol estimate. With 500ms VWAP snapshots from orderbook data, you get ~170,000 observations per day. Even accounting for the autocorrelation in high-frequency data (which you need to correct for), the statistical precision of your vol estimate is dramatically better.

More importantly, you can detect vol regime changes in minutes rather than days. For DeFi protocols that need to react quickly — say, tightening liquidation parameters during a flash crash — this responsiveness is the whole point.

Live Dashboard Prototype

Here’s an interactive prototype of the volatility oracle dashboard. It simulates a real-time BTC price feed and computes all three volatility estimators continuously. Try adjusting the window size and update threshold, or hit “Inject Volatility Spike” to see how each estimator responds to sudden regime changes.

TypeScript Prototype: Real-Time Vol Computation

Here’s the core of the prototype. This service connects to a WebSocket feed, maintains a rolling window of VWAP prices, and computes realized vol on demand.

import WebSocket from "ws";

interface PricePoint {
  timestamp: number;
  vwap: number;
}

class VolatilityService {
  private prices: PricePoint[] = [];
  private readonly windowSize: number;
  private readonly aggregationMs: number;
  private currentBucket: { sumPQ: number; sumQ: number; ts: number } | null = null;

  constructor(windowSize = 3600, aggregationMs = 500) {
    this.windowSize = windowSize; // seconds of history to keep
    this.aggregationMs = aggregationMs;
  }

  connectToFeed(wsUrl: string, symbol: string): void {
    const ws = new WebSocket(wsUrl);
    ws.on("message", (data: string) => {
      const msg = JSON.parse(data);
      if (msg.s === symbol && msg.e === "trade") {
        this.ingestTrade(msg.T, parseFloat(msg.p), parseFloat(msg.q));
      }
    });
  }

  private ingestTrade(timestamp: number, price: number, qty: number): void {
    const bucketTs = Math.floor(timestamp / this.aggregationMs) * this.aggregationMs;
    if (!this.currentBucket || this.currentBucket.ts !== bucketTs) {
      if (this.currentBucket && this.currentBucket.sumQ > 0) {
        this.prices.push({
          timestamp: this.currentBucket.ts,
          vwap: this.currentBucket.sumPQ / this.currentBucket.sumQ,
        });
        this.pruneOldPrices();
      }
      this.currentBucket = { sumPQ: 0, sumQ: 0, ts: bucketTs };
    }
    this.currentBucket.sumPQ += price * qty;
    this.currentBucket.sumQ += qty;
  }

  computeRealizedVol(windowSeconds?: number): number {
    const window = windowSeconds || this.windowSize;
    const cutoff = Date.now() - window * 1000;
    const relevant = this.prices.filter((p) => p.timestamp >= cutoff);
    if (relevant.length < 2) return 0;

    const logReturns: number[] = [];
    for (let i = 1; i < relevant.length; i++) {
      logReturns.push(Math.log(relevant[i].vwap / relevant[i - 1].vwap));
    }

    const mean = logReturns.reduce((a, b) => a + b, 0) / logReturns.length;
    const variance =
      logReturns.reduce((sum, r) => sum + (r - mean) ** 2, 0) / (logReturns.length - 1);

    // Annualize: scale by observations per year
    const obsPerYear = (365.25 * 24 * 3600 * 1000) / this.aggregationMs;
    return Math.sqrt(variance * obsPerYear);
  }

  private pruneOldPrices(): void {
    const cutoff = Date.now() - this.windowSize * 1000;
    this.prices = this.prices.filter((p) => p.timestamp >= cutoff);
  }
}

The service exposes the current vol estimate via a simple REST endpoint (not shown here for brevity, but it’s a basic Express route returning { annualizedVol, timestamp, windowSeconds, sampleCount }). This is what the oracle updater process polls to decide whether to push a new value on-chain.

A key design choice: the 500ms VWAP aggregation smooths out individual trade noise while preserving meaningful price movements. If you compute returns on raw trade-by-trade data, you get a lot of microstructure noise (bid-ask bounce, etc.) that inflates your vol estimate. The VWAP bucketing handles this elegantly.

Solidity Interface: The Oracle Contract

On the smart contract side, I want something simple and composable. Here’s the interface:

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.19;

interface IVolatilityOracle {
    struct VolData {
        uint256 realizedVol;      // Annualized vol in basis points (e.g., 5000 = 50%)
        uint256 parkinsonVol;     // Parkinson estimator
        uint256 garmanKlassVol;   // Garman-Klass estimator
        uint256 timestamp;        // When this was computed
        uint32  windowSeconds;    // Observation window used
        uint16  confidence;       // Sample count / quality metric
    }

    function getVolatility(bytes32 pair) external view returns (VolData memory);
    function getRealizedVol(bytes32 pair) external view returns (uint256 vol, uint256 timestamp);
    function isStale(bytes32 pair, uint256 maxAge) external view returns (bool);

    event VolatilityUpdated(bytes32 indexed pair, uint256 realizedVol, uint256 timestamp);
}

contract VolatilityOracle is IVolatilityOracle {
    mapping(bytes32 => VolData) private _latestVol;
    mapping(address => bool) public authorizedUpdaters;
    uint256 public updateThresholdBps = 100; // Only update if vol changed by >1%

    modifier onlyUpdater() {
        require(authorizedUpdaters[msg.sender], "unauthorized");
        _;
    }

    function pushUpdate(bytes32 pair, VolData calldata data) external onlyUpdater {
        VolData storage current = _latestVol[pair];
        uint256 delta = _absDiff(current.realizedVol, data.realizedVol);
        require(
            delta * 10000 / (current.realizedVol + 1) >= updateThresholdBps,
            "change below threshold"
        );
        _latestVol[pair] = data;
        emit VolatilityUpdated(pair, data.realizedVol, data.timestamp);
    }

    function getVolatility(bytes32 pair) external view returns (VolData memory) {
        return _latestVol[pair];
    }

    function getRealizedVol(bytes32 pair) external view returns (uint256, uint256) {
        VolData storage d = _latestVol[pair];
        return (d.realizedVol, d.timestamp);
    }

    function isStale(bytes32 pair, uint256 maxAge) external view returns (bool) {
        return block.timestamp - _latestVol[pair].timestamp > maxAge;
    }

    function _absDiff(uint256 a, uint256 b) internal pure returns (uint256) {
        return a > b ? a - b : b - a;
    }
}

A few design decisions worth noting:

Push model with threshold. The updater only sends a transaction when vol has changed meaningfully (configurable threshold, defaulting to 1% relative change). This keeps gas costs manageable — during stable markets, updates might happen every few minutes; during volatile periods, more frequently.

Staleness check built in. Consumers can call isStale() to verify the data is fresh enough for their use case. A lending protocol might tolerate 5-minute-old data; an options protocol might want sub-minute freshness.

Multiple estimators in one struct. Different consumers have different needs. By publishing all three estimators in a single update, we amortize the gas cost across multiple use cases.

Basis points representation. Using uint256 with basis points (50% vol = 5000 bps) avoids floating point entirely while giving sufficient precision for any practical application.

Monitoring Dashboard

I’ve been building a simple dashboard to visualize the feed in real time — nothing fancy, just a React frontend with a chart showing the rolling vol estimate, the raw VWAP prices it’s computed from, and the on-chain update history. It’s useful for debugging and for demonstrating to potential integrators that the feed is responsive and accurate.

The most interesting visualization is overlaying the three estimators. During normal markets they track closely, but during dislocations (flash crashes, sudden liquidity events) the Parkinson estimator reacts fastest because it captures the intra-bucket range expansion immediately. Watching them diverge and reconverge is genuinely interesting from a market microstructure perspective.

Challenges and Future Work

This prototype works, but there’s a gap between “works on my laptop” and “production oracle that protocols depend on.” Here’s what I’m thinking about:

Gas Costs

Even with the threshold-based update mechanism, gas costs add up. On L1 Ethereum, pushing updates every few minutes for multiple pairs gets expensive quickly. The obvious answer is deploying on L2s (Arbitrum, Base, etc.) where updates cost fractions of a cent. You could also explore a pull-based model where consumers pay for their own reads from an off-chain data availability layer, but that adds complexity.

Multi-Source Aggregation

Relying on a single exchange’s orderbook is a centralization risk and a manipulation vector. The production version needs to aggregate across multiple venues — Binance, OKX, Bybit, Coinbase — and use some form of outlier detection to handle cases where one venue’s data diverges (possibly due to manipulation or technical issues).

I’m thinking a median-of-estimators approach across venues, with automatic exclusion of any source that deviates more than 2 standard deviations from the median. This is similar to how existing price oracle networks handle multi-source aggregation.

Manipulation Resistance

This is the hard problem. If someone knows that a DeFi protocol uses this vol oracle to set liquidation thresholds, they could theoretically manipulate the underlying orderbook data to spike the vol estimate, causing the protocol to widen thresholds, and then exploit the looser parameters.

Defenses include: using longer observation windows (harder to sustain manipulation), requiring consistent signals across multiple venues, implementing rate-of-change limits on the oracle output (vol can’t jump more than X% per update), and incorporating TWAP-style smoothing.

Integration with Oracle Networks

The end game isn’t running this as a standalone service. It’s getting vol data integrated into existing oracle networks so that it benefits from their existing security models, node operator sets, and consumer ecosystems. The computation methodology described here could run as a module within a decentralized oracle network, with multiple independent node operators each computing vol from their own data sources and reaching consensus on the output.

Further Estimator Research

There are more sophisticated approaches worth exploring: realized kernels (which handle microstructure noise more elegantly), jump-robust estimators (bi-power variation), and multi-scale realized volatility. Each has tradeoffs between noise sensitivity, computation cost, and the assumptions they make about the price process.

Wrapping Up

I think there’s a real product here. The DeFi options market is growing, lending protocols are getting more sophisticated about risk management, and everyone needs better vol data. The gap between “price oracle” and “volatility oracle” is mostly an engineering challenge at this point — the statistical methods are well-established, the data sources exist, and the smart contract interface is straightforward.

The prototype is running locally, ingesting Binance trade data and computing vol estimates that look reasonable when compared against Deribit’s implied vol surface. Next steps are multi-venue aggregation, a proper deployment on a testnet, and conversations with potential consumers about what API they actually want.

If you’re building something in the DeFi derivatives or risk management space and would find a vol oracle useful, I’d love to hear about your requirements. The interface design should be driven by actual consumer needs rather than my assumptions about what’s useful.

Analyzing Bid-Ask Spreads and Liquidity Shifts in Polymarket's 5-Minute Binary Option Orderbooks

Wed, 18 Mar 2026 16:30:00 GMT

日本語要約: Polymarketの5分BTCバイナリオプションのL2オーダーブックを分析した記事です。FastAPI + vanilla JSで構築したビジュアライザーを使い、スプレッドの推移、流動性のパターン、スナップショット間のサイズ変化を可視化。市場効率性への考察も行います。

I’ve been spending way too much time staring at Polymarket’s 5-minute BTC binary options lately. Not trading them (well, not much), but pulling apart their orderbooks to understand what’s actually happening in these ultra-short-duration prediction markets. Here’s what I found and the tool I built to see it all.

Why Orderbook Analysis Matters for Prediction Markets

Most people look at prediction markets from the price/probability angle: “What does the market think the odds are?” But if you dig into the L2 orderbook, you get a much richer picture. You can see:

Who’s confident (large resting orders near the mid)
Who’s uncertain (thin books with wide spreads)
When the market is about to move (liquidity getting pulled from one side)
Market maker behavior (symmetric adjustments, inventory management)

For 5-minute binary options on BTC price, this is especially interesting because the entire lifecycle of the market plays out in 300 seconds. You get to watch an orderbook be born, mature, and die in the time it takes to make coffee.

L2 Orderbook Structure on Polymarket

Polymarket runs a CLOB (Central Limit Order Book) on their hybrid exchange. For binary options, you have two outcome tokens — let’s call them YES and NO (or in the BTC case, UP and DOWN). Each outcome has its own orderbook with bids and asks.

Key things to understand:

Prices are probabilities: A bid of 0.65 on YES means someone is willing to pay $0.65 for a token that pays $1.00 if the event happens. That’s an implied probability of 65%.
Complementary pricing: YES at 0.65 and NO at 0.35 should be equivalent (minus spread). If YES bid is 0.60 and NO bid is 0.35, there’s a 0.05 gap representing the spread/profit for market makers.
Size represents conviction: A $5,000 order at 0.50 is very different from a $50 order at the same price.

The L2 data gives you multiple price levels deep on each side. For these 5-minute markets, you typically see 3-8 meaningful levels before the book gets sparse.

Building the Orderbook Visualizer

I wanted to see how the orderbook evolves over the life of a 5-minute contract. Not just the final state, but snapshots over time, side by side, with changes highlighted. So I built a simple tool.

Architecture: FastAPI + Vanilla JS

Nothing fancy. FastAPI backend that polls Polymarket’s API for orderbook snapshots, stores them, and serves them to a vanilla JS frontend that renders the books visually.

The market selector lets you pick which 5-minute contract to analyze. I filter for markets where there was actual user trading activity (not just market maker quotes sitting there).

Snapshot Caching with LRU

Since these markets are short-lived, I cache snapshots aggressively. Each market gets polled every 2 seconds during its active window, giving ~150 snapshots per contract.

from functools import lru_cache
from datetime import datetime
import httpx

class OrderbookSnapshotCache:
    def __init__(self, max_snapshots_per_market=200):
        self.snapshots: dict[str, list[dict]] = {}
        self.max_snapshots = max_snapshots_per_market

    async def capture_snapshot(self, market_id: str, token_id: str):
        async with httpx.AsyncClient() as client:
            resp = await client.get(
                f"https://clob.polymarket.com/book",
                params={"token_id": token_id}
            )
            book = resp.json()

        snapshot = {
            "timestamp": datetime.utcnow().isoformat(),
            "bids": [(float(o["price"]), float(o["size"])) for o in book["bids"]],
            "asks": [(float(o["price"]), float(o["size"])) for o in book["asks"]],
            "best_bid": float(book["bids"][0]["price"]) if book["bids"] else 0,
            "best_ask": float(book["asks"][0]["price"]) if book["asks"] else 1,
        }

        if market_id not in self.snapshots:
            self.snapshots[market_id] = []

        snaps = self.snapshots[market_id]
        snaps.append(snapshot)
        if len(snaps) > self.max_snapshots:
            snaps.pop(0)

        return snapshot

    def get_window(self, market_id: str, start_idx: int, window_size: int = 5):
        """Get a window of consecutive snapshots for side-by-side display."""
        snaps = self.snapshots.get(market_id, [])
        end_idx = min(start_idx + window_size, len(snaps))
        return snaps[start_idx:end_idx]

Computing Deltas Between Snapshots

This is where it gets interesting. I compute the difference between consecutive snapshots so you can see at a glance what changed — did size get added or pulled at a given level?

from dataclasses import dataclass
from typing import Optional

@dataclass
class LevelDelta:
    price: float
    prev_size: float
    curr_size: float
    delta: float
    is_new: bool
    is_removed: bool

    @property
    def direction(self) -> str:
        if self.is_new:
            return "new"
        if self.is_removed:
            return "removed"
        if self.delta > 0:
            return "increased"
        if self.delta < 0:
            return "decreased"
        return "unchanged"


def compute_book_deltas(
    prev_snapshot: dict,
    curr_snapshot: dict,
    side: str = "bids"
) -> list[LevelDelta]:
    """
    Compare two orderbook snapshots and return per-level deltas.
    Highlights new levels, removed levels, and size changes.
    """
    prev_levels = {price: size for price, size in prev_snapshot[side]}
    curr_levels = {price: size for price, size in curr_snapshot[side]}

    all_prices = sorted(
        set(prev_levels.keys()) | set(curr_levels.keys()),
        reverse=(side == "bids")
    )

    deltas = []
    for price in all_prices:
        prev_size = prev_levels.get(price, 0.0)
        curr_size = curr_levels.get(price, 0.0)

        deltas.append(LevelDelta(
            price=price,
            prev_size=prev_size,
            curr_size=curr_size,
            delta=curr_size - prev_size,
            is_new=(price not in prev_levels),
            is_removed=(price not in curr_levels),
        ))

    return deltas


def compute_spread(snapshot: dict) -> float:
    """Spread as percentage of mid price."""
    if not snapshot["bids"] or not snapshot["asks"]:
        return float("inf")
    best_bid = snapshot["bids"][0][0]
    best_ask = snapshot["asks"][0][0]
    mid = (best_bid + best_ask) / 2
    return (best_ask - best_bid) / mid * 100

The Frontend: Rendering Orderbook Levels

The frontend displays 5 consecutive snapshots side by side. Each level is color-coded based on whether size increased (green), decreased (red), is new (blue), or was removed (gray strikethrough).

<div id="orderbook-window" class="snapshots-container"></div>

<style>
  .level { display: flex; font-family: monospace; font-size: 12px; padding: 2px 4px; }
  .level.increased { background: rgba(0, 255, 0, 0.1); }
  .level.decreased { background: rgba(255, 0, 0, 0.1); }
  .level.new { background: rgba(0, 100, 255, 0.15); border-left: 2px solid #06f; }
  .level.removed { opacity: 0.4; text-decoration: line-through; }
  .price { width: 60px; text-align: right; margin-right: 8px; }
  .size { width: 80px; }
  .delta { width: 60px; color: #666; font-size: 11px; }
</style>

function renderOrderbookLevel(level, side) {
  const el = document.createElement('div');
  el.className = `level ${level.direction}`;

  const priceEl = document.createElement('span');
  priceEl.className = 'price';
  priceEl.textContent = level.price.toFixed(2);

  const sizeEl = document.createElement('span');
  sizeEl.className = 'size';
  sizeEl.textContent = `$${level.curr_size.toFixed(0)}`;

  const deltaEl = document.createElement('span');
  deltaEl.className = 'delta';
  if (level.delta !== 0 && !level.is_new && !level.is_removed) {
    const sign = level.delta > 0 ? '+' : '';
    deltaEl.textContent = `${sign}${level.delta.toFixed(0)}`;
    deltaEl.style.color = level.delta > 0 ? '#0a0' : '#c00';
  }

  if (side === 'bid') {
    el.append(deltaEl, sizeEl, priceEl);
  } else {
    el.append(priceEl, sizeEl, deltaEl);
  }

  return el;
}

function renderSnapshot(snapshot, deltas, container) {
  const header = document.createElement('div');
  header.className = 'snapshot-header';
  const ts = new Date(snapshot.timestamp);
  header.textContent = `${ts.getMinutes()}:${ts.getSeconds().toString().padStart(2, '0')}`;
  container.appendChild(header);

  const spread = ((snapshot.best_ask - snapshot.best_bid) * 100).toFixed(1);
  const spreadEl = document.createElement('div');
  spreadEl.className = 'spread-indicator';
  spreadEl.textContent = `spread: ${spread}c`;
  container.appendChild(spreadEl);

  // Render asks (reversed so lowest ask is closest to spread)
  const askDeltas = deltas.asks.slice().reverse();
  askDeltas.forEach(level => {
    container.appendChild(renderOrderbookLevel(level, 'ask'));
  });

  // Render bids
  deltas.bids.forEach(level => {
    container.appendChild(renderOrderbookLevel(level, 'bid'));
  });
}

The visualization above shows 5 consecutive L2 orderbook snapshots for both UP and DOWN outcomes. You can clearly see the book evolving — levels appearing and disappearing, sizes adjusting, spreads tightening and widening.

Observations From the Data

After watching a few hundred of these 5-minute contracts, some patterns become very clear.

Spread Dynamics: Tight at Start, Wide Near Expiry

At market open (T+0), spreads are typically 2-4 cents. Market makers are happy to provide liquidity because there’s plenty of time for the underlying (BTC) to move and create trading opportunities.

As expiry approaches (T+4:30 onwards), spreads blow out to 8-15 cents. This makes total sense — the market maker’s gamma risk is enormous in the last few seconds. A $10 move in BTC can flip the outcome, and you don’t want to be caught on the wrong side with tight quotes.

Liquidity Clustering Around Key Levels

Unsurprisingly, the thickest liquidity clusters around:

0.50: The “I have no idea” price. Early in the contract when BTC hasn’t moved much, both sides stack size at 50 cents.
Round numbers (0.20, 0.30, 0.70, 0.80): Psychological anchors. People set limit orders at these levels.
0.05 and 0.95: The “almost certain” levels. Large size here from people willing to bet on near-certainties at favorable odds.

Size Delta Patterns: Market Maker Inventory Management

The deltas reveal classic market-making behavior:

Symmetric pulls: When a market maker wants to reduce exposure, they simultaneously reduce size on both bid and ask. You see negative deltas on both sides in the same snapshot.
Leaning: When BTC starts trending in one direction, you can see market makers shift their quotes — adding size on the side they want to get filled, reducing on the side they don’t.
Refreshing: After a fill, new size appears at the same or adjacent level within 2-4 seconds. This is the MM replenishing their quote.

The “Close” Event: Last 10 Seconds

The final 10 seconds are wild. Here’s what consistently happens:

T-10s: Market makers start pulling quotes aggressively. Book thins out.
T-7s: Spreads hit maximum. Only stale quotes remain.
T-5s: Aggressive traders market-order into whatever’s left, knowing the outcome is nearly certain.
T-2s: Book is basically empty. Anyone still quoting is either a bot with a direct feed or someone who forgot to cancel.

The delta visualization makes this collapse very visible — you see a cascade of “removed” levels (gray strikethrough) rippling through the book in the final snapshots.

Market Efficiency Implications

A few takeaways about market efficiency in these ultra-short contracts:

These markets are surprisingly efficient mid-life. Between T+30s and T+4:00, the prices track BTC movements well. The orderbook adjusts within 2-4 seconds of BTC price changes.

But they’re inefficient at the edges. In the first 10 seconds and last 10 seconds, there are clear mispricings. At open, it takes time for the book to populate and find fair value. At close, the liquidity vacuum creates slippage that benefits anyone with a fast feed.

The spread IS the market’s uncertainty measure. Wider spreads = more disagreement about the outcome. You can actually use the spread time series as a proxy for realized volatility of the underlying during that 5-minute window.

Size matters more than price. A level with $5,000 behind it at 0.48 is much more informative than a level with $20 at 0.52. The orderbook depth weighted by size gives you a better probability estimate than the mid-price alone.

What’s Next

I’m planning to:

Add historical spread analytics per market (average spread curve over contract lifetime)
Build a simple signal from book imbalance (total bid size vs total ask size) to predict short-term direction
Overlay BTC price feed to correlate orderbook changes with underlying movements

The visualizer has been a great tool for building intuition about these markets. If you’re interested in prediction market microstructure, I’d highly recommend building something similar — there’s no substitute for watching the books move in real time.

The full code is a mess of FastAPI routes and vanilla JS spaghetti, but the core logic above should be enough to get you started. The Polymarket CLOB API is well-documented and rate limits are generous enough for 2-second polling on a handful of markets.

Processing 155 Million Orderbook Records: Building a Fast Data Pipeline for Crypto Microstructure Analysis

Fri, 20 Feb 2026 11:00:00 GMT

日本語要約: Binance WebSocketから取得した1億5500万件超のオーダーブックレコードを効率的に処理するデータパイプラインの構築記録です。バイナリサーチローダー、Numba JITによる高速集計、LOBシミュレーターの設計に加え、AR-MLPやQuant-GANなどのML手法も試した結果と考察を共有します。

I’ve been working on crypto microstructure analysis for a while now, and the hardest part isn’t the modeling — it’s wrangling the data. When you’re looking at tick-level BTC/USDT orderbook dynamics, you’re dealing with a firehose of information that will happily eat all your RAM and leave you staring at a frozen terminal.

This post covers how I built a data pipeline that can efficiently load and process 155+ million orderbook records, run aggregation at sub-second intervals, and feed the results into ML models and LOB simulators.

The Problem

Crypto microstructure research requires tick-level data. Not 1-minute candles, not even 1-second bars — actual individual bid/ask updates as they stream from the exchange. For BTC/USDT on Binance, that means:

155M+ bid/ask depth update records (~48GB raw CSV)
20M trade tick records (~6GB)
3.45M orderbook snapshot records (~12GB)

All of this collected over several months from the Binance WebSocket feed. Each record has a timestamp, and we need to slice arbitrary time windows, compute derived features (mid price, VWAP, volume profiles), and feed them into models.

The goal was simple: given any arbitrary time range, produce aggregated microstructure features at 0.5-second intervals in under 2 seconds. Sounds reasonable until you realize a naive approach takes 45+ minutes just to load the data.

Why Naive Approaches Fail

My first attempt was embarrassingly straightforward:

import pandas as pd

# Don't do this with 48GB of CSVs
df = pd.read_csv("binance_btcusdt_depth_updates.csv")

On my 64GB workstation, this OOM’d spectacularly. Even with chunksize, iterating through 155M rows to find a 30-minute window was painfully slow (~3 minutes per query). Parquet helped with compression but the full scan was still the bottleneck.

The key insight: these files are already sorted by timestamp. We don’t need to scan — we need to seek.

Binary Search Loader

The idea is simple. Since our CSV files are sorted by timestamp (they come from a WebSocket feed, after all), we can binary search for the starting position and only read what we need.

import os
import bisect
import numpy as np
import pandas as pd
from pathlib import Path
from typing import Tuple, Optional


class BinarySearchCSVLoader:
    """
    Loads time-sliced data from large sorted CSV files using binary search.
    Builds a sparse index on first access, then seeks directly to the
    relevant byte offset for subsequent queries.
    """

    def __init__(self, csv_path: str, timestamp_col: str = "timestamp",
                 index_granularity: int = 100_000):
        self.csv_path = Path(csv_path)
        self.timestamp_col = timestamp_col
        self.granularity = index_granularity
        self._index: Optional[np.ndarray] = None
        self._offsets: Optional[np.ndarray] = None
        self._header: Optional[str] = None
        self._build_index()

    def _build_index(self):
        """Build sparse index: sample every N-th row's timestamp and byte offset."""
        timestamps = []
        offsets = []

        with open(self.csv_path, 'r') as f:
            self._header = f.readline()
            row_count = 0

            while True:
                offset = f.tell()
                line = f.readline()
                if not line:
                    break

                if row_count % self.granularity == 0:
                    ts = int(line.split(',')[0])  # timestamp is first column
                    timestamps.append(ts)
                    offsets.append(offset)

                row_count += 1

        self._index = np.array(timestamps, dtype=np.int64)
        self._offsets = np.array(offsets, dtype=np.int64)
        print(f"Index built: {len(self._index)} entries covering {row_count:,} rows")

    def load_range(self, start_ts: int, end_ts: int) -> pd.DataFrame:
        """Load only rows within [start_ts, end_ts] using binary search."""
        # Find the index entry just before our start timestamp
        idx_start = max(0, bisect.bisect_left(self._index, start_ts) - 1)
        idx_end = min(len(self._index) - 1,
                      bisect.bisect_right(self._index, end_ts))

        byte_start = self._offsets[idx_start]
        # Read until well past the end to account for granularity
        if idx_end + 1 < len(self._offsets):
            byte_end = self._offsets[idx_end + 1]
        else:
            byte_end = os.path.getsize(self.csv_path)

        bytes_to_read = byte_end - byte_start

        # Read only the relevant chunk
        from io import StringIO
        with open(self.csv_path, 'r') as f:
            f.seek(byte_start)
            chunk = f.read(bytes_to_read)

        df = pd.read_csv(StringIO(self._header + chunk))
        # Filter to exact range
        mask = (df[self.timestamp_col] >= start_ts) & \
               (df[self.timestamp_col] <= end_ts)
        return df[mask].reset_index(drop=True)

The index build takes about 90 seconds for the 155M-row file (one-time cost), and after that, loading any 30-minute window takes 0.8-1.2 seconds instead of 3+ minutes. Memory usage drops from “all of it” to just the slice you need — typically 50-200MB for a 30-minute window depending on market activity.

Performance Numbers

Operation	Naive (pandas)	Binary Search Loader
Load 30-min window	185s	1.1s
Load 2-hour window	185s (same full scan)	3.8s
Memory (30-min)	48GB+ (OOM)	~150MB
Index build (one-time)	N/A	92s

Numba JIT for Hot-Path Aggregation

Once we have the raw tick data loaded, we need to aggregate it into features at regular intervals. Computing mid price, VWAP, and volume at 0.5-second intervals across millions of ticks is the hot path that runs on every query.

Pure pandas/numpy was taking 800ms+ for a 30-minute window. With Numba JIT, this drops to about 45ms.

import numba
import numpy as np
from numba import njit, prange


@njit(cache=True)
def compute_aggregated_features(
    timestamps: np.ndarray,    # int64, microseconds
    bid_prices: np.ndarray,    # float64
    ask_prices: np.ndarray,    # float64
    bid_volumes: np.ndarray,   # float64
    ask_volumes: np.ndarray,   # float64
    trade_prices: np.ndarray,  # float64
    trade_volumes: np.ndarray, # float64
    trade_timestamps: np.ndarray,  # int64
    interval_us: int = 500_000     # 0.5 seconds in microseconds
) -> tuple:
    """
    Compute mid price, VWAP, and volume at fixed intervals.
    All arrays must be sorted by timestamp.
    """
    t_start = timestamps[0]
    t_end = timestamps[-1]
    n_intervals = int((t_end - t_start) / interval_us) + 1

    mid_prices = np.empty(n_intervals, dtype=np.float64)
    vwaps = np.empty(n_intervals, dtype=np.float64)
    volumes = np.empty(n_intervals, dtype=np.float64)
    interval_times = np.empty(n_intervals, dtype=np.int64)

    tick_idx = 0
    trade_idx = 0

    for i in range(n_intervals):
        t_interval_start = t_start + i * interval_us
        t_interval_end = t_interval_start + interval_us
        interval_times[i] = t_interval_start

        # Advance tick pointer to end of this interval
        last_bid = 0.0
        last_ask = 0.0
        while tick_idx < len(timestamps) and timestamps[tick_idx] < t_interval_end:
            last_bid = bid_prices[tick_idx]
            last_ask = ask_prices[tick_idx]
            tick_idx += 1

        mid_prices[i] = (last_bid + last_ask) / 2.0

        # Compute VWAP and volume from trades in this interval
        vol_sum = 0.0
        pv_sum = 0.0
        while trade_idx < len(trade_timestamps) and \
              trade_timestamps[trade_idx] < t_interval_end:
            v = trade_volumes[trade_idx]
            p = trade_prices[trade_idx]
            vol_sum += v
            pv_sum += p * v
            trade_idx += 1

        volumes[i] = vol_sum
        vwaps[i] = pv_sum / vol_sum if vol_sum > 0 else mid_prices[i]

    return interval_times, mid_prices, vwaps, volumes

The first call takes ~2s due to JIT compilation, but subsequent calls with the same type signature hit the cache and run in 40-50ms. For interactive exploration this is a huge win — you get near-real-time iteration on feature engineering.

LOB Simulator

With the pipeline working, I built a Limit Order Book simulator to generate synthetic orderbook dynamics. The idea was to test whether ML models could learn microstructure patterns from simulated data before deploying on real data.

The simulator supports multiple price process models:

Geometric Brownian Motion (baseline)
Jump Diffusion (Merton model)
Self-Exciting Jump Diffusion (Hawkes process intensity)
AR-MLP (autoregressive neural network)
Quant-GAN (generative adversarial network)

Each model generates price paths, and the LOB simulator wraps these with realistic order flow dynamics — queue sizes, arrival rates, and cancellation patterns calibrated from the real Binance data.

ML Experiments

Here’s where things got interesting (and humbling).

AR-MLP: Autoregressive MLP

The AR-MLP takes a window of past mid-price returns and orderbook imbalance features, and predicts the next 0.5s return. Architecture is straightforward: 3 hidden layers (256, 128, 64), batch norm, dropout 0.3, trained on 2 months of data.

The problem is immediately visible: systematic drift. The model learns a slight directional bias from the training set and compounds it over time. Even with careful normalization and detrending, the generated paths diverge from realistic price behavior after about 5 minutes of simulation.

I tried several fixes:

Mean-centering predictions (helps but introduces weird mean-reversion artifacts)
Predicting log returns instead of price levels (same drift, just in log space)
Adding an explicit mean-reversion penalty to the loss (improves it but kills realistic momentum)

None fully solved it. The model captures short-term autocorrelation well but fails at longer horizons.

Quant-GAN

The Quant-GAN approach (based on the Wiese et al. paper) uses temporal convolutional networks as both generator and discriminator, operating on sequences of log returns.

This produced more visually interesting results. The distribution of log returns from the GAN actually matches the empirical distribution reasonably well — you can see the heavy tails are captured. But in practice:

Training was unstable (classic GAN problems, even with spectral normalization and gradient penalty)
Generated paths lacked the temporal structure of real prices — autocorrelation of absolute returns was wrong
Most importantly, for downstream tasks (strategy backtesting), it didn’t outperform calibrated jump-diffusion

Why Traditional Models Won

After weeks of ML experiments, I ended up using a self-exciting jump diffusion model for the LOB simulator. The reasons:

Calibration is fast and stable — fit parameters via MLE on historical data in seconds
Interpretable — you can directly see how jump intensity decays, what the baseline volatility is
Hawkes-driven clustering — captures the volatility clustering and trade arrival patterns that matter for microstructure
No drift problem — martingale property is built in by construction

The ML models are interesting academically, and I think with more data and better architectures (transformers, perhaps) they could eventually win. But for a practical LOB simulator where you need reliable synthetic data for strategy testing, a well-calibrated stochastic model is hard to beat.

Pipeline Architecture Summary

The full pipeline looks like:

Binance WebSocket --> Raw CSVs (sorted by timestamp)
                         |
                    Binary Search Index (sparse, in-memory)
                         |
                    Time-Slice Loader (seek + read)
                         |
                    Numba Aggregator (0.5s intervals)
                         |
               +--------------------+
               |                    |
         Feature Store         LOB Simulator
               |                    |
         ML Models            Strategy Backtester

Total latency from “give me features for this time range” to having aggregated data ready: ~1.5 seconds for a typical 30-minute window. This includes loading from disk, so if you’re iterating on features interactively, it feels snappy enough.

Lessons Learned

Binary search on sorted files is underrated. Everyone reaches for databases or Parquet partitioning, but if your data is already sorted and you need arbitrary range queries, a simple sparse index + seek gets you 90% of the way there with zero dependencies.

Numba is magic for numerical hot paths. The constraint is that you need to write “numpy-style” code inside the JIT function (no pandas, no Python objects), but for aggregation loops it’s a perfect fit. The cache=True flag is essential — without it you pay compilation cost on every restart.

ML models for price simulation are harder than they look. The drift problem in AR models and mode collapse in GANs are well-known, but experiencing them firsthand on real financial data is educational. The gap between “interesting paper results” and “useful tool for my workflow” is wider than I expected.

Start with the data pipeline, not the model. I spent the first two weeks just on the loader and aggregator, and it paid off massively. Every model experiment after that was fast to iterate on because the data was always ready in seconds.

The full pipeline code is roughly 2,500 lines of Python. Not a massive codebase, but it handles the core problem well: turning a firehose of raw WebSocket data into something you can actually work with for research.

Self-Exciting Jump-Diffusion for Crypto: Why Vanilla Models Miss Momentum and Volatility Clustering

Thu, 08 Jan 2026 09:15:00 GMT

日本語要約: BS/Mertonモデルの限界を踏まえ、自己励起型ジャンプ拡散モデル（Excited Kou v2）を開発した過程を解説します。Laplace分布ジャンプ、自己励起強度、方向性持続の仕組みにより、BTCの短期価格におけるモメンタムとボラティリティクラスタリングを捕捉します。Hawkesプロセスとの比較も行います。

Where We Left Off

In previous work, I explored pricing binary options on BTC using Black-Scholes (GBM) and Merton’s jump-diffusion. The takeaway was clear: Black-Scholes systematically misprice short-term crypto derivatives because it assumes continuous paths and constant volatility. Merton improved things by adding normally-distributed jumps, but still missed important structure in the data.

The core issue: both models assume independent increments. Each time step is memoryless. In reality, 5-minute BTC returns show strong evidence of:

Momentum (returns are positively autocorrelated at short lags)
Volatility clustering (big moves follow big moves)
Jump clustering (jumps don’t arrive uniformly — they come in bursts)

These aren’t just cosmetic failures. For binary option pricing on platforms like Polymarket, where you’re betting on whether BTC crosses a threshold within hours, getting the short-term dynamics wrong means your model’s fair value is wrong.

What the Data Actually Looks Like

I pulled 5-minute BTC/USDT candles and computed log returns. Here’s what jumps out from the empirical statistics:

Fat Tails (Excess Kurtosis)

The empirical kurtosis of 5-minute log returns sits around 8-12, far exceeding the Gaussian value of 3. This means extreme moves happen way more often than a normal distribution predicts. Merton’s model can match this with Gaussian jumps, but the shape of the tails matters — crypto jumps tend to be sharper and more asymmetric than a normal distribution captures.

Autocorrelation at Lag-1

This is the killer. Standard diffusion models produce returns with zero autocorrelation by construction. But empirical 5-minute BTC returns show ACF(1) around 0.10-0.12. That’s not huge in absolute terms, but it’s statistically significant and economically meaningful over many time steps. It means there’s genuine short-term momentum — an up move slightly increases the probability of the next move also being up.

Jump Clustering

When I detect jumps (returns exceeding 3 standard deviations), they aren’t uniformly distributed in time. The conditional probability of a jump given that the previous bar was also a jump is significantly higher than the unconditional jump probability. Jumps beget jumps.

Kou’s Double-Exponential Model

Before adding self-excitation, let’s upgrade the jump distribution. Kou (2002) proposed using a double-exponential (asymmetric Laplace) distribution for jump sizes instead of Gaussian. The density is:

$$fJ(x) = p \cdot \eta1 e^{-\eta1 x} \mathbf{1}{x \geq 0} + (1-p) \cdot \eta2 e^{\eta2 x} \mathbf{1}_{x < 0}$$

where:

$p$ = probability of an upward jump
$\eta_1 > 1$ = rate parameter for upward jumps (larger = smaller jumps)
$\eta_2 > 0$ = rate parameter for downward jumps

Why does this matter? The exponential distribution has memoryless properties that make analytical option pricing tractable, and its sharper peak + heavier tails better match crypto return distributions than Gaussian jumps. The asymmetry parameter lets us capture the empirical observation that downward jumps tend to be slightly larger than upward ones.

For my 5-minute BTC data, I estimated:

$\eta_1 \approx 60$ (upward jump mean size ~1.7%)
$\eta_2 \approx 50$ (downward jump mean size ~2.0%)
$p \approx 0.48$ (slight bias toward down jumps)

The Excited Extension: Self-Exciting Intensity

Here’s where it gets interesting. Standard Kou (like standard Merton) assumes jumps arrive via a homogeneous Poisson process — constant intensity $\lambda$, independent arrivals. But we just showed jumps cluster in time.

The fix: make the jump intensity depend on recent history. I call this the “Excited Kou” model. The key modifications:

Time-Varying Intensity

$$\lambdat = \lambda{base} + \alpha \cdot \mathbf{1}[\text{jump at } t-1]$$

After a jump occurs, the intensity spikes by $\alpha$, making an immediate follow-up jump more likely. This is a simplified version of a Hawkes process — instead of tracking the full history with exponential decay, I just look one step back. It’s crude but surprisingly effective for capturing the clustering at the 5-minute scale.

Directional Persistence

$$p{up,t} = 0.5 + \delta \cdot \text{sign}(r{t-1})$$

The probability of an upward jump is shifted based on the sign of the previous return. If the last move was positive, upward jumps become slightly more likely. This injects the momentum/persistence effect directly into the jump component.

5-State Markov Additive Process

Combining these, each time step falls into one of 5 states:

State	Description
0	No previous jump, diffusion only
1	Previous jump was up, no current jump
2	Previous jump was up, current jump
3	Previous jump was down, no current jump
4	Previous jump was down, current jump

The transition probabilities depend on $\lambdat$ and $p{up,t}$, which themselves depend on the current state. This makes it a proper Markov chain — the state at time $t$ determines the distribution at time $t+1$.

Parameter Estimation

I estimated parameters from ~2 weeks of 5-minute BTC/USDT data (about 4,000 observations). The approach was:

Jump detection: Flag returns with $|r_t| > 3\hat{\sigma}$ as jumps
Base intensity: $\lambda_{base}$ = unconditional jump frequency (~0.05 jumps per 5-min bar)
Excitation parameter: $\alpha$ estimated from the conditional probability $P(\text{jump}t | \text{jump}{t-1}) - \lambda_{base}$, giving $\alpha \approx 0.15$
Directional persistence: $\delta$ estimated from the correlation between sign of previous return and sign of current jump, giving $\delta \approx 0.08$
Diffusion volatility: Estimated from non-jump returns, $\sigma \approx 0.001$ per 5-min bar

I also ran Bayesian optimization over the parameter space to minimize the discrepancy between simulated and empirical ACF + kurtosis. The BO converged on similar values, which was reassuring.

The Simulation Loop

Here’s the core simulation in Python:

import numpy as np

def simulate_excited_kou(
    S0, mu, sigma, lambda_base, alpha, delta,
    eta1, eta2, p_base, n_steps, dt, n_paths
):
    """
    Simulate Excited Kou jump-diffusion paths.
    
    Self-exciting intensity + directional persistence.
    """
    paths = np.zeros((n_paths, n_steps + 1))
    paths[:, 0] = S0
    
    # Track state: did a jump happen at previous step, and its sign
    had_jump_prev = np.zeros(n_paths, dtype=bool)
    sign_prev = np.zeros(n_paths)
    
    for t in range(1, n_steps + 1):
        # Time-varying intensity: excite after jumps
        lambda_t = lambda_base + alpha * had_jump_prev.astype(float)
        
        # Directional persistence in jump direction
        p_up_t = p_base + delta * sign_prev
        p_up_t = np.clip(p_up_t, 0.1, 0.9)
        
        # Poisson jump arrivals
        jump_occurs = np.random.poisson(lambda_t * dt) > 0
        
        # Double-exponential jump sizes
        n_jumps = jump_occurs.sum()
        jump_sizes = np.zeros(n_paths)
        
        if n_jumps > 0:
            is_up = np.random.uniform(size=n_jumps) < p_up_t[jump_occurs]
            sizes = np.where(
                is_up,
                np.random.exponential(1.0 / eta1, size=n_jumps),
                -np.random.exponential(1.0 / eta2, size=n_jumps)
            )
            jump_sizes[jump_occurs] = sizes
        
        # GBM diffusion component
        dW = np.random.normal(0, np.sqrt(dt), size=n_paths)
        diffusion = (mu - 0.5 * sigma**2) * dt + sigma * dW
        
        # Combine
        log_return = diffusion + jump_sizes
        paths[:, t] = paths[:, t-1] * np.exp(log_return)
        
        # Update state for next step
        had_jump_prev = jump_occurs
        sign_prev = np.sign(jump_sizes)
        sign_prev[~jump_occurs] = 0.0
    
    return paths

The key insight is in lines where lambda_t and p_up_t depend on the previous step’s state. This is what breaks the independent-increments assumption and introduces the autocorrelation.

Interactive Demo

Play with the simulator below to see how self-excitation changes the price dynamics. Toggle between Vanilla GBM, Kou, and Excited Kou v2 — watch how the log return distribution and autocorrelation change as you crank up the α (self-excitation) and δ (persistence) parameters.

Results

The dashboard above compares fair value estimates from BS Theory, Polymarket’s market price, and our Excited Kou v2 model. The self-exciting model tracks closer to market prices, particularly during volatile regimes where momentum effects are strongest.

ACF Improvement

This is where the model really shines:

Model	ACF(1)	Target (Empirical)
Black-Scholes (GBM)	~0.000	0.120
Merton Jump-Diffusion	~0.007	0.120
Excited Kou v2	0.115	0.120

Going from 0.007 to 0.115 against a target of 0.120 is a massive improvement. The model now captures ~96% of the observed short-term autocorrelation.

Kurtosis

Model	Kurtosis	Target (Empirical)
Black-Scholes	3.0	~9.5
Merton	~6.2	~9.5
Excited Kou v2	~8.8	~9.5

The double-exponential jumps plus clustering naturally produce higher kurtosis because you get bursts of large moves rather than uniformly distributed jumps.

The figure above shows the simulation vs real data across three panels: mid price paths, log return distributions, and the return ACF. The Excited Kou model reproduces the empirical ACF structure that vanilla models completely miss.

Comparison: Hawkes Process Approach

An alternative way to model self-excitation is through a proper Hawkes process. Instead of my simplified one-step-back indicator, a Hawkes process uses exponential kernel decay:

$$\lambdat = \lambda0 + \sum{ti < t} \alpha \cdot e^{-\beta(t - t_i)}$$

Each past event contributes to current intensity with exponentially decaying influence. I implemented a 2-timescale variant with fast ($\beta1 \approx 20$, half-life ~2 bars) and slow ($\beta2 \approx 2$, half-life ~20 bars) components to capture both immediate clustering and longer regime effects.

The Hawkes approach produces slightly better calibration on the jump clustering statistics and handles multi-bar momentum streaks more naturally. However, it’s also harder to estimate (MLE for Hawkes processes is finicky with 5-minute data) and computationally more expensive in simulation since you need to track the full event history.

For the binary option pricing use case, the simplified Excited Kou gives 90%+ of the benefit at a fraction of the complexity. The Hawkes version is more theoretically satisfying but the practical improvement is marginal for short-horizon pricing.

Takeaways

Independent increments are wrong for short-term crypto. The data clearly shows autocorrelation and clustering. Any model that ignores this will misprice short-term derivatives.
You don’t need a full Hawkes process to get most of the benefit. A simple one-step-back excitation captures the bulk of the clustering effect in 5-minute data.
Jump distribution shape matters. The double-exponential (Kou) fits crypto tails better than Gaussian jumps, and gives sharper kurtosis matching.
Directional persistence in jumps is real and exploitable. The $\delta$ parameter is small (~0.08) but statistically significant and meaningful for pricing.
Bayesian optimization over simulation parameters works well when you have clear target statistics (ACF, kurtosis) to match. The objective landscape is surprisingly smooth.

Next steps: incorporating this into a real-time pricing engine that updates parameters as new data arrives. The stationarity assumption is still an issue — these parameters drift, especially $\lambda_{base}$ and $\sigma$, so some kind of rolling estimation window or regime-switching layer would help. But that’s for another post.

Comparing Black-Scholes and Merton Jump-Diffusion Models Against Real Polymarket Binary Option Prices

Sat, 15 Nov 2025 14:00:00 GMT

日本語要約: Black-ScholesモデルとMertonジャンプ拡散モデルを用いて、Polymarketの5分BTCバイナリオプションの理論価格を算出し、実際の市場価格と比較します。Monte Carloシミュレーションのコード例を交えながら、伝統的金融モデルが暗号通貨の超短期市場でどこまで通用するかを検証します。

I’ve been obsessed with Polymarket’s ultra-short-term BTC binary options lately. These are 5-minute contracts where you’re betting whether BTC/USDT will be above or below a strike price at expiry. The tick-by-tick nature of these contracts makes them a perfect playground for testing option pricing models against reality.

The question I wanted to answer: can classical Black-Scholes or the more sophisticated Merton jump-diffusion model actually predict what Polymarket traders are pricing in? Spoiler: the answer is nuanced and interesting.

Why Crypto Binary Options Are a Weird Beast

Binary options are the simplest derivative you can imagine. You get paid $1 if BTC is above the strike at expiry, $0 otherwise. The “fair” price is just $P(S_T > K)$ - the probability that the spot price exceeds the strike at expiration.

But crypto makes this deceptively hard to model:

5-minute expiries mean you’re in a regime where microstructure noise dominates
BTC volatility is extreme - annualized vol ranges from 40% to 120% depending on the regime
Jumps happen constantly - a single whale market order can move BTC 0.5% in seconds
Volatility clusters - calm periods alternate with chaos in a very non-Gaussian way

Polymarket runs these contracts with strikes spaced around the current spot price (e.g., if BTC is at $68,450, you’ll see strikes at $68,400, $68,500, $68,600). I pulled about 1,200 trades across multiple contract cycles to compare with model predictions.

Black-Scholes for Binary Options

The standard BS framework assumes BTC follows geometric Brownian motion (GBM):

$$dS = \mu S , dt + \sigma S , dW$$

where $\mu$ is the drift, $\sigma$ is the volatility, and $dW$ is a Wiener process.

For a cash-or-nothing binary call (pays $1 if $S_T > K$), the analytical price is:

$$C{binary} = e^{-rT} \cdot N(d2)$$

where:

$$d_2 = \frac{\ln(S/K) + (r - \frac{1}{2}\sigma^2)T}{\sigma\sqrt{T}}$$

and $N(\cdot)$ is the standard normal CDF.

For 5-minute expiries, $T$ is tiny (about $9.5 \times 10^{-6}$ years), $r$ is basically zero, and the whole thing reduces to:

$$C_{binary} \approx N\left(\frac{\ln(S/K) - \frac{1}{2}\sigma^2 T}{\sigma\sqrt{T}}\right)$$

The critical input is $\sigma$. I estimated it using rolling 1-hour realized volatility from Binance tick data, annualized. Typical values during my observation period were 45-65% annualized.

The BS dashboard above shows the model tracking real prices. It works… okay. The shape is right - prices near 0.5 when the spot is at the strike, approaching 1 or 0 as you move away. But there’s consistent mispricing at the tails.

Where Black-Scholes Breaks Down

BS assumes returns are log-normally distributed. In a 5-minute window for BTC, the actual return distribution has:

Fat tails: the kurtosis of 5-minute BTC returns is typically 8-15 (normal would be 3). Extreme moves are way more likely than BS predicts.
Jumps: BTC doesn’t move continuously. Liquidation cascades, large market orders, and news events create discrete jumps. In my dataset, about 3-4% of 5-minute windows had moves exceeding 3 standard deviations.
Asymmetry: negative jumps (crashes) are slightly more common and larger than positive jumps, giving negative skewness.
Volatility clustering: a big move in the last 5 minutes makes a big move in the next 5 minutes much more likely. BS assumes constant vol.

The practical effect: BS underprices deep out-of-the-money binaries. If the strike is $200 above spot and you have 5 minutes left, BS says “basically zero probability” but the market says “small but non-trivial” because traders know jumps happen.

Merton Jump-Diffusion Model

Merton (1976) extended GBM by adding a compound Poisson jump process:

$$dS = (\mu - \lambda k)S , dt + \sigma S , dW + S , dJ$$

where:

$\lambda$ is the jump intensity (average number of jumps per unit time)
$J$ is the jump size, typically $\ln(1+J) \sim N(\muJ, \sigmaJ^2)$
$k = E[e^J - 1] = e^{\muJ + \frac{1}{2}\sigmaJ^2} - 1$ is the expected percentage jump size

The key insight: between jumps, the price follows GBM with reduced drift (to compensate for the expected jump contribution). When a jump arrives, the price instantaneously moves by a log-normal factor.

For binary option pricing under Merton, the analytical formula is a series expansion:

$$C{binary}^{Merton} = \sum{n=0}^{\infty} \frac{e^{-\lambda’ T}(\lambda’ T)^n}{n!} \cdot N(d_2^{(n)})$$

where $\lambda’ = \lambda(1+k)$ and:

$$d2^{(n)} = \frac{\ln(S/K) + (r - \frac{1}{2}\sigman^2 - \lambda k)T + n\muJ}{\sigman\sqrt{T}}$$

$$\sigman = \sqrt{\sigma^2 + n\sigmaJ^2 / T}$$

I calibrated the jump parameters from the empirical 5-minute return distribution:

$\lambda = 500$ jumps/year (roughly 1 jump per 10 five-minute intervals)
$\mu_J = -0.0002$ (slight downward bias)
$\sigma_J = 0.003$ (jump magnitude)
$\sigma = 0.50$ (diffusion vol, annualized)

Interactive Simulator

Before diving into the code, try the interactive simulator below. You can adjust the model parameters (volatility, jump intensity, etc.) and run Monte Carlo simulations to see how BS and Merton pricing curves diverge in real-time.

Monte Carlo Simulation Approach

While the analytical Merton formula works, I also ran Monte Carlo simulations for both models. This lets me sanity-check the analytics and easily extend to more complex dynamics later.

Here’s the simulation code:

import numpy as np
from scipy.stats import norm

def simulate_bs_binary(S0, K, T, sigma, r=0.0, n_sims=100000):
    """
    Monte Carlo price for a binary call under GBM.
    Returns P(S_T > K).
    """
    Z = np.random.standard_normal(n_sims)
    S_T = S0 * np.exp((r - 0.5 * sigma**2) * T + sigma * np.sqrt(T) * Z)
    payoff = (S_T > K).astype(float)
    return np.exp(-r * T) * payoff.mean()


def simulate_merton_binary(S0, K, T, sigma, lam, mu_j, sigma_j,
                           r=0.0, n_sims=100000):
    """
    Monte Carlo price for a binary call under Merton jump-diffusion.
    Returns P(S_T > K) with Poisson jumps.
    """
    # Number of jumps in [0, T] for each path
    N_jumps = np.random.poisson(lam * T, n_sims)

    # Compensator for jump drift
    k = np.exp(mu_j + 0.5 * sigma_j**2) - 1
    drift = (r - 0.5 * sigma**2 - lam * k) * T

    # Diffusion component
    Z = np.random.standard_normal(n_sims)
    diffusion = sigma * np.sqrt(T) * Z

    # Jump component: sum of N_jumps log-normal jumps
    jump_component = np.zeros(n_sims)
    for i in range(n_sims):
        if N_jumps[i] > 0:
            jumps = np.random.normal(mu_j, sigma_j, N_jumps[i])
            jump_component[i] = jumps.sum()

    # Terminal price
    S_T = S0 * np.exp(drift + diffusion + jump_component)
    payoff = (S_T > K).astype(float)
    return np.exp(-r * T) * payoff.mean()


# Example: BTC at $68,450, strike at $68,500, 5 min to expiry
S0 = 68450.0
K = 68500.0
T = 5.0 / (365.25 * 24 * 60)  # 5 minutes in years
sigma = 0.55  # 55% annualized vol

# BS price
bs_price = simulate_bs_binary(S0, K, T, sigma, n_sims=500000)
print(f"BS Binary Price: {bs_price:.4f}")

# Merton price
lam = 500      # ~1 jump per 10 five-min intervals
mu_j = -0.0002
sigma_j = 0.003

merton_price = simulate_merton_binary(S0, K, T, sigma, lam, mu_j, sigma_j,
                                       n_sims=500000)
print(f"Merton Binary Price: {merton_price:.4f}")

# Analytical BS for comparison
d2 = (np.log(S0/K) + (0 - 0.5*sigma**2)*T) / (sigma*np.sqrt(T))
bs_analytical = norm.cdf(d2)
print(f"BS Analytical: {bs_analytical:.4f}")

The loop in simulate_merton_binary is intentionally naive for clarity. In practice you’d vectorize it (pre-allocate max jumps and mask), which gets you ~10x speedup. For 500k paths on a 5-minute contract, runtime is about 2 seconds on my machine.

Comparison Against Real Polymarket Data

This is where it gets interesting. I collected trade data from Polymarket’s BTC 5-minute binary contracts over several hours, tracking:

The contract strike price
Time remaining at each trade
The trade price (what the market thinks P(S_T > K) is)
Contemporaneous BTC/USDT spot from Binance

Then I computed BS and Merton model prices for each observed trade and plotted them together.

The chart shows two panels. The top panel is the BTC/USDT mid price over the observation window. The bottom panel overlays three series: Polymarket actual trade prices (what people paid), BS theoretical prices, and Merton Monte Carlo prices.

Key observations:

1. Merton tracks the market better than BS at the extremes. When the spot is far from the strike (contract is deep ITM or OTM), Merton prices are closer to what traders actually pay. BS snaps too aggressively to 0 or 1.

2. Both models track well near ATM. When the spot is close to the strike and there’s 2-3 minutes left, all three lines converge. Near ATM, the jump component doesn’t change the probability much since a small move in either direction is equally likely from diffusion alone.

3. The market has a persistent “crash premium.” Polymarket prices for OTM puts (binary calls struck above spot) are slightly higher than either model predicts. Traders are assigning extra probability to upward jumps that neither model fully captures. This might reflect order flow imbalance or informed trading.

4. Discrete price effects matter. Polymarket contracts trade at discrete prices (0.01 increments), so very OTM contracts have a floor at $0.01-0.02 even when models say the fair value is $0.005. This creates a systematic overpricing of tail events.

Where the Models Diverge from the Market

The biggest systematic errors I noticed:

Last 30 seconds: Both models become unreliable in the final half-minute. Microstructure effects dominate - the bid-ask spread on BTC itself creates uncertainty that’s not captured by either model. The market seems to use a “volatility bump” in the last few seconds.
During high-activity periods: When BTC is trending (3+ consecutive 5-min candles in one direction), the market prices in momentum that neither model accounts for. BS and Merton are both martingale models - they don’t believe in trends.
Jump clustering: The Merton model assumes jumps arrive independently (Poisson). In reality, one jump often triggers another (liquidation cascades). During my observation period, there was a sequence of 3 rapid moves within 2 minutes that pushed the spot $150. The market repriced instantly, but Merton’s independent-jump assumption meant it still underestimated tail probabilities during that episode.

What Would Actually Work Better?

Based on these results, a few extensions seem promising:

Self-exciting jump process (Hawkes process instead of Poisson): jumps beget jumps. This would capture the clustering effect.
Stochastic volatility + jumps (SVJ or SVJJ): let $\sigma$ itself be random and correlated with returns. Heston + jumps might nail the dynamics better.
Regime-switching vol: simple two-state model (calm vs. volatile) with Markov transitions. Computationally cheap and might capture the bimodal nature of 5-min BTC vol.
Just use the orderbook: for ultra-short expiries, the BTC orderbook depth within $100-200 of the spot price is probably more informative than any parametric model. If there’s a $2M bid wall $50 below, the probability of breaking through in 5 minutes is lower than any vol estimate would suggest.

Takeaways

For anyone building pricing models for crypto binary options:

BS is a reasonable starting point and surprisingly decent for ATM contracts with 2-5 minutes left
Merton adds real value at the tails - if you’re market-making these contracts, using BS alone will get you picked off on OTM strikes
Neither model captures the full picture - the market knows about orderbook dynamics, momentum, and jump clustering that parametric models miss
Monte Carlo is your friend for these short-dated contracts since path count needed is modest and you can layer in arbitrarily complex dynamics
Calibration window matters enormously - I got best results using the last 1-2 hours of realized vol rather than daily estimates

The code above is a starting point. The real alpha is in how you estimate $\sigma$ and the jump parameters in real-time, adapting to the current market regime. But that’s a topic for another post.

How Binary Options Work on Prediction Markets: A Developer's Introduction to Polymarket Contracts

Tue, 12 Aug 2025 10:30:00 GMT

日本語要約: Polymarketの予測市場におけるバイナリオプションの仕組みを開発者視点で解説する記事です。CLOB（中央指値注文帳）の構造、5分/15分BTCバイナリコントラクトのライフサイクル、APIやWebSocketを使ったデータ取得方法をPythonコード例とともに紹介します。

I’ve been digging into Polymarket’s binary options infrastructure for the past few weeks, specifically the short-duration BTC contracts. If you’re a developer looking to build trading tools, analytics dashboards, or just want to understand how these contracts actually work under the hood, this post covers what I’ve learned.

What Is Polymarket and Why Binary Options?

Polymarket is a prediction market platform built on Polygon. Users trade shares in the outcome of events - political elections, sports, crypto prices, whatever. Each market resolves to either YES or NO, and shares pay out $1 if correct, $0 if wrong.

Binary options on Polymarket take this concept and apply it to price movements. Instead of “Will candidate X win the election?”, you get markets like “Will BTC be above $67,500 at 14:05 UTC?” - a simple up/down bet on where price will be at a specific time.

The key insight: these are NOT traditional options with Greeks and continuous payoffs. They’re binary - you either get $1 or $0. The pricing reflects the market’s implied probability of the event occurring. A YES share trading at $0.62 means the market thinks there’s roughly a 62% chance BTC will be above the strike at expiry.

The CLOB: How Orders Get Matched

Polymarket runs a Central Limit Order Book (CLOB) - not an AMM. This is important because it means you get real bid/ask spreads, limit orders, and proper price discovery.

The CLOB is operated by Polymarket’s exchange operator but settlement happens on-chain via their CTF (Conditional Token Framework) contracts. Here’s the flow:

Order placement: You sign an order off-chain (EIP-712 typed data) specifying the token ID, side, price, and size
Matching: The exchange operator matches compatible orders
Settlement: Matched trades are settled on-chain - CTF tokens move between wallets

The order book for each binary option market has two sides: YES tokens and NO tokens. Since YES + NO = $1 always, buying YES at $0.60 is equivalent to selling NO at $0.40. The exchange handles this complementarity automatically.

What makes this interesting from a market microstructure perspective: the spreads on short-duration BTC binaries are often 2-5 cents wide. Market makers are actively quoting, and you can see the book update in real-time via WebSocket.

5-Minute and 15-Minute BTC Binary Contracts

The short-duration BTC markets are where things get fun. Polymarket runs rolling binary option contracts on BTC/USDT price with 5-minute and 15-minute durations.

Here’s how a typical 5-minute contract works:

Strike: Set at the current BTC mid-price at market creation (e.g., $67,482.50)
Duration: 5 minutes from creation to resolution
Resolution source: BTC/USDT price from a specified oracle feed
Outcome: YES if BTC >= strike at expiry, NO otherwise

The 15-minute contracts work identically but with a longer window, which generally means tighter spreads and more liquidity since there’s more time for price discovery.

The dashboard above shows the relationship between the underlying BTC/USDT mid price and the binary option price curves. You can see how the BO price responds non-linearly to moves in the underlying - this is basically the CDF of the implied distribution compressed into a 5-minute window.

One thing that tripped me up initially: the contract creation is staggered. New 5m contracts launch every 5 minutes, so at any given time you might have an active contract with 3 minutes remaining and a new one that just opened. The overlap creates interesting arbitrage dynamics.

Contract Lifecycle

Every binary option contract goes through distinct phases:

1. Creation

The market is created with a condition ID, strike price, and expiry timestamp. The CTF contract mints the initial token positions. At this point, YES and NO tokens exist but no one has traded yet.

2. Trading Period

Orders flow in. Market makers post bids and asks on both sides. As BTC moves relative to the strike, the BO price adjusts. The closer to expiry, the more sensitive the price becomes to small moves in the underlying (gamma increases, if you want to think in options terms).

3. Settlement

At expiry, the oracle reports the BTC price. The contract resolves: if BTC >= strike, YES tokens are redeemable for $1 each. NO tokens become worthless. Redemption happens on-chain through the CTF contract.

The fills chart above shows actual trades on a 5m binary market. Notice how trade frequency and size increase as we approach expiry - that’s participants reacting to last-second price moves in BTC.

Accessing Polymarket Data Programmatically

Polymarket exposes two main interfaces for developers:

REST API (CLOB API)

The CLOB API gives you access to markets, order books, and trade history. Base URL is https://clob.polymarket.com.

Key endpoints:

GET /markets - list available markets (filterable by tag, status)
GET /book?token_id={id} - current order book snapshot
GET /trades?market={id} - recent trade history
GET /prices?token_id={id} - price history with OHLC data

WebSocket

For real-time data, you connect to the WebSocket feed at wss://ws-subscriptions-clob.polymarket.com/ws/market. You subscribe to channels per market and get:

Order book updates (bids/asks added/removed)
Trade notifications
Market status changes

Here’s a basic Python script to fetch market data and inspect a BTC binary option:

import requests
import json
from datetime import datetime

CLOB_BASE = "https://clob.polymarket.com"

def get_btc_binary_markets():
    """Fetch active BTC binary option markets."""
    resp = requests.get(
        f"{CLOB_BASE}/markets",
        params={"tag": "btc-binary", "active": "true"}
    )
    resp.raise_for_status()
    markets = resp.json()
    return markets

def get_order_book(token_id: str):
    """Get current order book for a binary option token."""
    resp = requests.get(
        f"{CLOB_BASE}/book",
        params={"token_id": token_id}
    )
    resp.raise_for_status()
    return resp.json()

def get_recent_trades(condition_id: str, limit: int = 50):
    """Fetch recent trades for a market."""
    resp = requests.get(
        f"{CLOB_BASE}/trades",
        params={"market": condition_id, "limit": limit}
    )
    resp.raise_for_status()
    return resp.json()

# Fetch active BTC binary markets
markets = get_btc_binary_markets()

for market in markets[:3]:
    print(f"Market: {market['question']}")
    print(f"  Condition ID: {market['condition_id']}")
    print(f"  End Date: {market['end_date_iso']}")
    print(f"  Tokens: YES={market['tokens'][0]['token_id'][:12]}...")
    print(f"          NO={market['tokens'][1]['token_id'][:12]}...")

    # Get the order book for YES token
    yes_token_id = market['tokens'][0]['token_id']
    book = get_order_book(yes_token_id)

    if book.get('bids') and book.get('asks'):
        best_bid = float(book['bids'][0]['price'])
        best_ask = float(book['asks'][0]['price'])
        print(f"  Best Bid: ${best_bid:.3f}")
        print(f"  Best Ask: ${best_ask:.3f}")
        print(f"  Spread: ${best_ask - best_bid:.3f}")
    print()

Parsing Trade Data

Once you have trade data, you usually want to reconstruct what happened - who was aggressive, what the fill rate looked like, how price evolved. Here’s how I parse trade data from the API:

import pandas as pd
from datetime import datetime, timezone

def parse_trades_to_df(trades: list) -> pd.DataFrame:
    """Parse raw trade data into a DataFrame with derived fields."""
    records = []
    for trade in trades:
        ts = datetime.fromisoformat(
            trade['timestamp'].replace('Z', '+00:00')
        )
        records.append({
            'timestamp': ts,
            'price': float(trade['price']),
            'size': float(trade['size']),
            'side': trade['side'],  # 'BUY' or 'SELL'
            'fee_rate': float(trade.get('fee_rate_bps', 0)) / 10000,
            'notional': float(trade['price']) * float(trade['size']),
        })

    df = pd.DataFrame(records)
    if df.empty:
        return df

    df = df.sort_values('timestamp').reset_index(drop=True)

    # Compute running VWAP
    df['cum_notional'] = df['notional'].cumsum()
    df['cum_size'] = df['size'].cumsum()
    df['vwap'] = df['cum_notional'] / df['cum_size']

    # Tag aggressive side (taker)
    df['is_buyer_taker'] = df['side'] == 'BUY'

    # Time until expiry (assuming market_end is known)
    return df

# Example usage
trades = get_recent_trades(markets[0]['condition_id'], limit=100)
df = parse_trades_to_df(trades)

print(f"Total trades: {len(df)}")
print(f"Total volume: ${df['notional'].sum():.2f}")
print(f"VWAP: ${df['vwap'].iloc[-1]:.4f}")
print(f"Buyer-taker ratio: {df['is_buyer_taker'].mean():.1%}")
print(f"\nLast 5 trades:")
print(df[['timestamp', 'price', 'size', 'side']].tail())

A few things I noticed when working with this data:

Trades come back in batches - the API doesn’t guarantee strict ordering within the same block timestamp
Fee rates vary by maker/taker status and can be 0 for makers on certain markets
The size field is in shares (not USD) - multiply by price to get notional
During high-volatility moments, you’ll see clusters of fills at the same timestamp as the book gets swept

Practical Challenges

A few gotchas if you’re building on this:

Rate limits: The CLOB API has rate limits that aren’t always clearly documented. I found ~100 requests/minute to be safe for the REST endpoints. Use WebSocket for anything real-time.

Token ID vs Condition ID: Markets have a condition_id (the overall market) and individual token_ids for YES and NO positions. Some endpoints want one, some want the other. The order book endpoint takes token_id, while trades takes the market’s condition_id.

Timing: For 5-minute binaries, even 500ms of latency matters. If you’re building a trading bot, you need to account for the time between your price observation and order arrival at the exchange.

Resolution edge cases: What happens if the oracle price is exactly at the strike? In Polymarket’s implementation, >= resolves to YES. But double-check the specific market’s resolution rules - they’re in the market description.

Wrapping Up

Polymarket’s binary options are a surprisingly clean environment for building trading tools. The CLOB gives you proper market microstructure to analyze, the contracts are simple enough to model (binary payoff + time decay), and the API access is decent for a crypto platform.

If you’re coming from TradFi options or DeFi AMMs, the mental model shift is: think of these as extremely short-dated digital options with discrete settlement. The pricing behavior near expiry is where all the action happens - and where the data gets interesting.

I’m working on a follow-up post covering how to model the implied probability surface across multiple strikes and durations. If you’re building something similar, feel free to reach out.

swager-codegenのまともな使い方

Sun, 22 Jan 2023 22:12:03 GMT

swagger

SwaggerはAPIの仕様記述言語とツール群です。APIの仕様書をYAMLまたはJSONの形式で記述し、Swagger EditorやSwagger UIを使って可視化・共有することができます。

そして、ちょっとすごいところはSwaggerで書かれたAPI記述をサーバーのコード、クライアントのコードに変換することができます。そのためのツールがswagger-codegenです。

swagger-codegen

swagger-codegen https://github.com/swagger-api/swagger-codegen

swagger-codegenには、goやnodejsをはじめとした複数の言語やフレームワークを対象としたバックエンドを含んでいます。今回はtype-scriptのaxiosクライアント、nodejsのサーバーコードを生成してみました。

手順

cloneとbranchの切り替え

まず、swagger-codegen をクローンします。クローンできたら、branchを自分のopenapi記述のバージョンによって切り替えます。 openapi2.0ならmaster、openapi3.0なら3.0.0です。私は3.0.0のブランチを利用しています。私は、最新コミットでやりましたが、バージョンを固定したい方は適当なTagにcheckoutしてください。僕はcommit 8f191f46a853bed86debc74f10f242712ae4758eでやっています。

build swagger-codegen

いくつかのbuild方法がありますが、Dockerを使うのが最も簡単な方法だと思います。 Dockerのイメージのビルド、swagger-codegenのビルドを一括でやってくれるshellスクリプトが用意されています。

./run-in-docker.sh mvn package

5分ぐらいでビルド完了して以下の出力が出ます。

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for swagger-codegen-project 3.0.38-SNAPSHOT:
[INFO] 
[INFO] swagger-codegen-project ............................ SUCCESS [ 11.138 s]
[INFO] swagger-codegen (core library) ..................... SUCCESS [03:25 min]
[INFO] swagger-codegen (executable) ....................... SUCCESS [ 23.943 s]
[INFO] swagger-codegen (maven-plugin) ..................... SUCCESS [ 10.012 s]
[INFO] swagger-generator .................................. SUCCESS [01:19 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  05:31 min
[INFO] Finished at: 2023-01-21T04:56:44Z
[INFO] ------------------------------------------------------------------------

これが出れば成功です。

試しに、./run-in-docker.sh langsを実行してみましょう。利用可能なバックエンドの一覧が表示されるはずです。

{your host name}@ swagger-codegen % ./run-in-docker.sh langs                     
++ dirname ./run-in-docker.sh
+ cd .
+ maven_cache_repo=/Users/{your host name}/.m2/repository
+ mkdir -p /Users/nao/.m2/repository
++ id -u
++ id -g
+ docker run --rm -it -w /gen -e GEN_DIR=/gen -e MAVEN_CONFIG=/var/maven/.m2 -u 501:20 -v /Users/nao/test/swagger-codegen:/gen -v /Users/nao/.m2/repository:/var/maven/.m2/repository --entrypoint /gen/docker-entrypoint.sh maven:3-jdk-8 langs
Available languages: [dart, aspnetcore, csharp, csharp-dotnet2, go, go-server, dynamic-html, html, html2, java, jaxrs-cxf-client, jaxrs-cxf, inflector, jaxrs-cxf-cdi, jaxrs-spec, jaxrs-jersey, jaxrs-di, jaxrs-resteasy-eap, jaxrs-resteasy, micronaut, spring, nodejs-server, openapi, openapi-yaml, kotlin-client, kotlin-server, php, python, python-flask, r, ruby, scala, scala-akka-http-server, swift3, swift4, swift5, typescript-angular, typescript-axios, typescript-fetch, javascript]

openapi.yamlからコードをgenerate

今回利用するbackendは、clientがtypescript-axios、backendがnodejs-serverです。clientは複数の選択肢がありますが、serverはjavascript(typescript)ならnodejs-serverしか選択肢がないようです。

まず、自分の作成したopenapi.yamlをswagger-codegenの直下にコピーします。その上で、以下を実行します。

./run-in-docker.sh generate -i openapi.yaml \
    -l typescript-axios -o /gen/out/test-client -DpackageName=test-client # client

./run-in-docker.sh generate -i openapi.yaml \
    -l nodejs-server -o /gen/out/test-server -DpackageName=test-server # server

すると以下のような出力が出て、生成が完了します。

...
05:00:38.447 [Thread-1] INFO  i.s.codegen.v3.AbstractGenerator - writing file /gen/out/test-client/tsconfig.json
05:00:38.459 [Thread-1] INFO  i.s.codegen.v3.AbstractGenerator - writing file /gen/out/test-client/.swagger-codegen-ignore
05:00:38.470 [Thread-1] INFO  i.s.codegen.v3.AbstractGenerator - writing file /gen/out/test-client/.swagger-codegen/VERSION

swagger-codegen/out/test-{client, server}に出力されたコードがあるはずです。

実行

生成された各ディレクトリにREADME.mdがあります。そこにどうやって生成コードを実行すればいいのか書いてあります。今回の場合、どちらとも以下です。

npm install
npm start

これで実行できました。

まとめ

今回は、swagger-codegenを使ってopenapi記述から、javascriptのclient, serverのコードを生成してみました。もちろん、実装上必要なコード全てが生成できるわけではありませんが、参考にするには十分なコードが生成されていると思います。このコードをパーツとして切り貼りして開発すれば、実装の速度や正確性も向上するのではないかと思っています。

Anteをコンパイルするときの謎つまづきポイント

Sat, 18 Jun 2022 22:12:03 GMT

Anteという、新しめの言語がある。

Ante low-level functionnal language
https://antelang.org/

C++のような手続き型言語とRush, Haskelのような関数型言語の中間を目指しているらしい。Rustほど真面目にメモリ安全を目指さない分、自由度が高く書きやすい言語を目指しているようだ。特徴的なのは、Algebraic Effectかな、プロダクションレベルでこの仕様を使える最初の言語になりそう。（LISP・・・？なんですかそれ）

使うには自分でAnteのコンパイラ（Rust製）をコンパイルする必要があるようだ。コンパイラのリポジトリは以下である。

jfecher/ante
https://github.com/jfecher/ante

本記事はこのコンパイラをコンパイルするうえでつまづいたポイントを後世のために残しておくためのものである。

環境 & ビルド法

環境はWSL: Ubuntu20.04
build-essentialとかはすでに入れている。

ビルドは以下のコマンドを用いて行った。AnteのgithubのREADME.mdに書いてある方法だ。

$ cargo install llvmenv
$ llvmenv init
$ llvmenv build-entry -G Makefile -j7 13.0.0
$ llvmenv global 13.0.0
$ LLVM_SYS_130_PREFIX=$(llvmenv prefix)
$ cargo build

llvm 13.0.0 も自前でビルドする方式である。

ポイント1 error: a destructor cannot be ‘constexpr‘

問題

$ llvmenv build-entry -G Makefile -j7 13.0.0

の部分のコンパイルで、

error: a destructor cannot be ‘constexpr‘

というエラーが出る。

対処法

gcc, g++のバージョンが古いことが原因。現在のllvmはgcc-9ではコンパイルできない。aptからgcc-10, g++-10を入れて、update-altanativeでデフォルト化する。

sudo apt install gcc-10 g++-10

update-altanativeについては、ここを参照
https://students-tech.blog/post/change-gcc.html

ポイント2 mach-o/compact_unwind_encoding.h が無い

問題

$ llvmenv build-entry -G Makefile -j7 13.0.0

の部分のコンパイルで、

fatal error: mach-o/compact_unwind_encoding.h: No such file or directory

というエラーが出る。

対処法

ここに記述がある。
https://github.com/llvmenv/llvmenv/issues/115#issuecomment-1072951262
super hackyだがこれしかないようだった。つまり、mach-oディレクトリを作成し、そこにネットからググって取ってきたcompact_unwind_encoding.hを配置するという方法だ。環境にもよるかもしれないが、以下の位置にファイルを配置すれば良い。
/home/{ユーザー名}/.cache/llvmenv/13.0.0/tools/lld/MachO/mach-o/compact_unwind_encoding.h

私がネットから取ってきたcompact_unwind_encoding.hはこれ
https://github.com/JuliaLang/libosxunwind/blob/master/include/mach-o/compact_unwind_encoding.h
う～ん、super hacky!

ポイント3 sys/cdefs.h がない

問題

$ llvmenv build-entry -G Makefile -j7 13.0.0

の部分のコンパイルで、

sys/cdefs.h: そのようなファイルやディレクトリはありません

というエラーが出る。

対処法

ここに記述がある。
https://qiita.com/milmilk/items/e4a8821abbf04c3c942c

これでOK

sudo apt install libc6-dev-i386

ポイント4 なぜか対応するllvmがないといわれる。

問題

$ cargo build

の部分でllvm-sysのコンパイル中に対応するllvmのバージョンがないといわれる

対処法

使っていたshellがzshだったので、llvmenv のgitに書いてある、zsh integrationを行った。
zsh integration: https://github.com/llvmenv/llvmenv#zsh-integration

source <(llvmenv zsh)

を.zshrcに書き込み。

これでできるようになったが、単にshellに入りなおすだけでも良かったのかも。（llvmenv 関連の環境変数の読み込み？）

終わりに

コンパイル時間がかかった。コントリビュートできたらしたい、Rustかけないけど逆説的に勉強モチベがわきました。

UdonをC#で書く方法　初歩編

Mon, 03 Jan 2022 22:12:03 GMT

概要

近年メタバースの筆頭として注目されているVRChatでは、各ユーザーがワールドを自作することができます。単に建物や景観を作成するだけなら3DCGスキルで十分ですが、ユーザーの挙動に応じてワールドが変化するようなギミックを作るにはUdonを用いたプログラミングが必要になります。UdonはVRChat上で独自のギミックを作成するためのツールとしてはほとんど唯一の選択肢になっています。

Udonを最も簡易的に利用する方法はUdon Graphを用いた方法であり、この方法についての解説記事がネット上で大半を占めています。しかし、一定以上に複雑なギミックを作成する場合には、Udon Graphを用いた手法は複雑度・実装速度の面から見て現実的ではありません。例えば、大人気World AmongUsVRには、AmongUsのほぼすべての機能が実装されていますが、これをUdon Graphを用いてプログラムすることは不可能だろうと容易に想像出来ると思います。そのため、Udonを直接C#でコーディングできるUdonSharpを利用する方法が重要になります。

本稿は、Udon Sharpを用いたC#によるプログラミングの方法を、Udon Graphによる実装と対応させながら解説することで、現在Udon Graphを使用しているユーザーがC#実装に一歩踏み出すための手助けになることを目指しました。

事前知識

VRChatのワールド作成、Udon Graphによるプログラミングについての知識を仮定します。具体的には、

VRChatワールド作成

おくらすVRちゃんねる氏のワールド作成動画　再生リスト
https://www.youtube.com/watch?v=_0vgPjV4YZk&list=PLdywIhUp6C4o_mNKGedMFRlbckXWv_UpD

Udon Graphによるプログラミング

Udon のはじめかた　(toe氏)
https://note.com/toh_csecb/n/nc57eb0e73d27
VRCワールドのSDK3 UDON(graphのほう)でミラーをオンする (かくちゅ氏)
https://note.com/kakutyu50/n/nc8bbe6cfc1c2
sdk3のudonでミラーをオンオフする(ついでに同期させる) (かくちゅ氏)
https://note.com/kakutyu50/n/n12052eba0b39

以上すべてを実際に手を動かしてやって頂ければ、本稿を読むうえで必要十分の知識が得られます。

事前準備

本稿は「sdk3のudonでミラーをオンオフする(ついでに同期させる)」で作成したミラースイッチ実装を出発点とするので、それが実装された状態にしてください。
あと、C#によるコーディングにはUdon Sharpが必要なので、以下からパッケージをDownload & Importしてください。

Udon Sharp/release
https://github.com/MerlinVR/UdonSharp/releases

あと、必須かわからないのですが、Visual Studioをインストールしておいてください。

1. C#コードを動かすまで

出発点

「sdk3のudonでミラーをオンオフする(ついでに同期させる)」でミラースイッチ実装を作成していると、以下のようなオブジェクトとそれに付与されたUdon Graphがあるはずです。（僕の実装では、ユーザーがそのワールドにJoinした時の処理も追加してあります。画像はクリックで拡大してください。）

このUdon graphと同様の処理を行うC#コードの実装について解説していきます。

同じものを実装するので、このオブジェクトの隣に、同じような配置で鏡とスイッチのオブジェクトを作成してください。(1)

C#でコーディングするモード

スイッチのオブジェクトにUdon Behaviourを追加してください。New Programの下のドロップダウンからUdon C# Programing Assetを選択し、New Programをクリック、続けてCreate Scriptもクリックしてください。すると、ファイルを保存するディレクトリを聞かれるので、どこでもいいので保存してください（僕はプロジェクト直下/Assets/yukarinoki/interactive_mirrorというディレクトリに設定しました）。保存すると、おそらくVisual Studioが開くと思います。開いたら右側のソリューションエクスプローラーの一番上から、Assembly-Csharpを開いて自分が先ほど保存したファイル開いてください。（僕の環境では以下の場所にあります。）

すると左側に以下のコードが現れると思います。

using UdonSharp;
using UnityEngine;
using VRC.SDKBase;
using VRC.Udon;

public class mirror_switch : UdonSharpBehaviour
{
    void Start()
    {
        
    }
}

鏡スイッチのC#コード

上のコード削除して、以下のコードを貼り付けてください。ただ、mirror_switchの部分は自分のオブジェクト名にしてください。

using UdonSharp;
using UnityEngine;
using VRC.SDKBase;
using VRC.Udon;
using UnityEngine.UI;



public class mirror_switch : UdonSharpBehaviour
{
    [SerializeField] private GameObject mirror;

    private bool is_active = false;

    public override void Interact()
    {
        Debug.Log(is_active);
        if (is_active)
        {
            SendCustomNetworkEvent(VRC.Udon.Common.Interfaces.NetworkEventTarget.All, "MirrorOff");
        }
        else
        {
            SendCustomNetworkEvent(VRC.Udon.Common.Interfaces.NetworkEventTarget.All, "MirrorOn");
        }
    }

    public void MirrorOn()
    {
        mirror.SetActive(true);
        is_active = true;
    }
    public void MirrorOff()
    {
        mirror.SetActive(false);
        is_active = false;
    }

    public override void OnPlayerJoined(VRCPlayerApi player)
    {
        if (is_active)
        {
            SendCustomEvent("MirrorOn");
        }
        else
        {
            SendCustomEvent("MirrorOff");
        }
    }
}

コンパイル & オブジェクト設定

Unityに戻り、スイッチオブジェクトのUdon BehaviourのCompile All UdonSharpo Programsをクリックしてください。すると、UtilitiesのところにMirrorというオブジェクト設定項目で出ていると思います。ここにVRCMirrorのオブジェクトを設定してください。

動作確認

VRshat SDKからBuildテストをしてみてください。Udon Graphと同様にスイッチで動作することが確認できると思います。

2. コード解説

他オブジェクトの利用

以下の部分が、そのコードが使用する他オブジェクトの宣言になります。ここに宣言されるとコンパイル時にUdon Sharpが読み取り、変数名(mirror)と同名のオブジェクト設定項目(Mirror)が出てくるようになっています。先頭が大文字になるのは仕様です。

public class mirror_switch : UdonSharpBehaviour
{
    [SerializeField] private GameObject mirror; /*ここを見てUdonは対象オブジェクトを判断*/

    private bool is_active = false;

試しに、以下のように変更してコンパイルしてみましょう。すると、Sizeというパラメータを含んだ設定が出てきますね。複数同じようなオブジェクトを管理したいときには配列を使うのが便利です。

public class mirror_switch : UdonSharpBehaviour
{
    [SerializeField] private GameObject mirror; 
     [SerializeField] private GameObject[] tekitou; /*変更*/

    private bool is_active = false;

　Interact時の挙動の設定

以下の部分がInteract時の挙動の設定コードです。

 public override void Interact()
    {
        Debug.Log(is_active);
        if (is_active)
        {
            SendCustomNetworkEvent(VRC.Udon.Common.Interfaces.NetworkEventTarget.All, "MirrorOff");
        }
        else
        {
            SendCustomNetworkEvent(VRC.Udon.Common.Interfaces.NetworkEventTarget.All, "MirrorOn");
        }
    }

overrideでUdonSharpBehaviour内のInteract()を上書きしています。

is_activeに応じて条件分岐した上で、SendCustomNetworkEvent()で全員を対象に”MirrorOff”か”MirrorOn”というCustomEventを発火させています。

デバッグ

複雑なコードになると当然デバッグが必要になります。手軽なデバッグ方法として、ログ出力があります。それは以下の部分です。

    {
        Debug.Log(is_active);　// ここ
        if (is_active)
        {

これで変数がログに出力されます。

一体ログはどこでみるのかという話なんですが、VRChatを起動して、右SHIFT + バッククウォート + 3 を押してみるとログが立ち上がります。（日本語キーボードなら、右SHIFT + @ + 3ということです。ログが開かない人はVRChat内の設定画面を開いてから同じキーを入力してみるとうまくいくかも？）これで、スイッチにインタラクトすると以下のようなログ出力が確認できます。

CustomEventに対応する挙動の設定

CustomEventに対応するには、そのイベント名と同名の関数を宣言します。

    public void MirrorOn()
    {
        mirror.SetActive(true);
        is_active = true;
    }
    public void MirrorOff()
    {
        mirror.SetActive(false);
        is_active = false;
    }

オブジェクトの存在は.SetActive()で変更できます。

Eventに対応する挙動の設定

Udon Graphのときと同じ名前の関数をoverrideで宣言します。他は解説することがないと思います。

public override void OnPlayerJoined(VRCPlayerApi player)
    {
        if (is_active)
        {
            SendCustomEvent("MirrorOn");
        }
        else
        {
            SendCustomEvent("MirrorOff");
        }
    }

　あとがき

まず、パッと見ただけでも、C#のほうがUdon Graphより簡単だということがわかってもらえると思います。この複雑度でこの差ですから、もう少し複雑になるとUdonSharpの効用は計り知れないと思います。 C#実装をみて、「Udon Graphは意外と簡単にC#に変換できるんだな」ということだけでもわかってもらえると嬉しいです。

本稿の内容だけで概念としては、Udon Sharpの実装のほぼすべてみたいな感じになると思いますが、実際にはいろいろ複雑な部分もあります。
もう少し勉強したい方はaki_lua87氏が実装されたoxゲーム、リバーシのコードを確認してみると良いと思います。
https://aki-lua87.booth.pm/

WANDアルゴリズム、効率的なTop-kのために

Mon, 20 Dec 2021 22:12:03 GMT

概要

WANDやblock WANDについて日本語の資料が少なかったので、まとめておく。

情報検索

この記事において、情報検索とは、「複数の単語を入力し、大量のドキュメントの中から、入力単語と最も関連性が強いドキュメントを、関連性が高い順に出力するもの」をいう。

Top-k query

Top-kクエリとは、上の情報検索において、上位k個を求めるクエリのことである。Top-1なら、最も関連性の強い一つのドキュメントを出力する。

転置インデックス

転置インデックスとは、それぞれのドキュメントについて、単語ごとの関連スコアを計算し、それを隣接リストの形にまとめたものである。
例えば、my, search, wordの3つの単語について、d1からd7までの7つのドキュメントから、転置インデックスを求めたものは下図のようになる。

情報検索において、転置インデックスは事前計算しておくこととする。
このスコア(s: 2など)は、具体的にはドキュメント中のその単語の出現頻度などである。

ドキュメントのスコア

例えば、上図の転置インデックスを持つドキュメント群に対して、“my search word”という3つの単語で検索を行ったとする。この時、“my search word”に対するドキュメントd1のスコアをどう求めるかはいろいろな流儀がある。単純に和を取る方法（1 + 2 = 3がスコアになる）、もしくは単語の重要度を入力しスコアと内積を取る方法（例えば、{my: 1, search:3, word: 2}のように入力し、スコア 1 * 1 + 2 * 3 + 0 * 2 = 6を得る）などがある。

この記事では、単純に和を取る方法を採用した場合について記述する。

情報検索アルゴリズム

転置インデックスが与えられれば、ナイーブには、すべての転置インデックスを確認し、すべてのドキュメントについてスコアを求め、それをpriority queueに入れていけば、Top-kクエリに回答できる。

なので、情報検索アルゴリズムは「転置インデックスの読み取り回数をいかに少なくするか(= 転置インデックスをいかにスキップするか)」を工夫することになる。

DAAT(Document At A Time)

DAATは、ナイーブな情報検索アルゴリズムです。Document At A Timeという名前の通り、その時のドキュメントを見ていくアルゴリズムです。

まず、すべての検索ワードの転置インデックスの先頭にポインタを置きます。 DAATは各イテレーションで、ポインタが置かれているドキュメントのうち最もドキュメントIDの小さいドキュメントについてスコアを計算します。スコア計算に使われたポインタをその転置インデックスの次の要素に移動させます。これをすべてのポインタが転置インデックスの終端に着くまで行います。

ナイーブに毛が生えたアルゴリズム

todo

WANDアルゴリズム

todo

Dec 2020 読んだもの、やったこと

Wed, 02 Dec 2020 22:12:03 GMT

やったこと

VRChat UDON

VRChatのWorldが作りたいので、UDONを学習した。（参考:https://qiita.com/amamagi/items/ac497db1b20cb754a1e6) ここに書いてあるような方法ではとてもじゃないけど一定以上のリッチなinteractionを持ったワールドを作ることができないので、UDON Sharp (略称：U#)を用いるしかなさそうである・・・。

U#とは、C#をUDONの機能にコンパイルしてくれるプラグイン。

U# 入門 ①（by ハツェ氏） https://hatuxes.hatenablog.jp/entry/2020/04/05/013310

を読めば大体のことはわかりそう。読むかな。

追記：読んだんですが、このようなやり方でゲーム性を持ったworldを制作するのはものすごい大変だろうと思いました。たとえばAmong Us “Game”（というamong usのVRChat移植版）を作成するためには地獄のような作業量が必要そうなことがわかり、震えました。

Blenderでキャラクターモデリング

VRChatのAvatarは以前友人に作ってもらったものがあるんですが、やっぱり自分で作ったほうが愛着が湧くし、上の事もあって自分でモデリングする作業がどこかで必要になるので、練習ついでにキャラクターモデリングをしてみることにしました。

以下の動画がわかりやすく参考にしています。
【Blender 2.9 Tutorial】キャラクターモデリング解説 #1 -Character Modeling Tutorial #1
https://youtu.be/BBtgjTiL1XQ

モデリングの学習ができればよかったんですが、副産物としてアニメキャラの基本的な造形についても詳しくなれて良いです。
以下が#1の動画を見終わったあとの、途中経過です。

読んだもの

角居勝彦著　さらば愛しい競馬 (12/06)

https://amzn.to/2JRsuwH

いろいろなことが書いてあったし、調教の実態や調教師から見たジョッキー・競馬界のシステム・レースについての見解には（僕にとっての）目新しさがあった。ただ、調教に関する一貫した理論やシンプルな法則・原則が記述されていないことは残念だった。一方で、競馬界のシステムの変化（降級馬がなくなる・スーパー未勝利がなくなる等）がもたらす調教シーンの変化予測には説得力があり、確かに「出走ローテーションなどによって陣営の競走馬への期待値が測れる」というのはそのとおりだと思った。未勝利戦では、調教師の期待値は最も結果に影響を与える要素になるはずだし、また、この要素はあまり顧みられていなそうなので、とても興味深い観点だと思った。あと次競馬見るときには、上手に手前を変えているか、馬の利き足、右回り左回りの得意不得意、に注目したい。

Huge Puerto Rico radio telescope, already damaged, collapses (12/01)

https://apnews.com/article/technology-arecibo-observatory-puerto-rico-science—0da6abb251f455977bf0c752348e712e

こないだ言及した、アレシボ天文台が日本時間 2020年 12月1日 20:56 (現地時間 7:56) に崩壊した。 アーメン・・・ 。

gatsby-blogにtable of contentを付ける

Thu, 19 Nov 2020 22:12:03 GMT

概要

gatsbyで作られたブログに

リンクが機能していて
スクロールに追従する
目次をつける方法について

MDX pluginは使わずにgatsby-transformer-remarkの元で実現しています。

graphQLでtable of contents

gatsby-transformer-remarkを使うと、gatsbyのgraphQLにいろいろな項目が追加されますが、そのなかにtable of contentsがあります。

まず、これをblog postのtemplateから読み取りましょう。

template/blog-post.js

export const pageQuery = graphql`
  query BlogPostBySlug($slug: String!) {
    site {
      siteMetadata {
        title
      }
    }
    markdownRemark(fields: { slug: { eq: $slug } }) {
      id
      excerpt(pruneLength: 160)
      html
      tableOfContents //←これ追加
      frontmatter {
        title
        date(formatString: "MMMM DD, YYYY")
        description
      }
    }
  }
`

そして、得られたtableOfContentsを（ここらへんは個人の実装のより異なりますが）、自分の作ったtableOfContents用のcomponentに渡します。

templates/blog-post.js

const BlogPostTemplate = ({ data, pageContext, location }) => {
  const post = data.markdownRemark
  const siteTitle = data.site.siteMetadata.title
  const { previous, next } = pageContext

  return (
    <Layout 
      location={location} title={siteTitle}
      rightSide={
                  <>
                    <Bio/> 
                    <TOC tocitems={data.markdownRemark.tableOfContents}/> // <- これ
                  </>
                }
     > 
....................中略..................
)

うけとる、TOC componentはこんな感じ。tableOfContentsは生のhtmlなのでdengerouslySetInnerHTMLで渡します。

components/toc.js

import React from "react"
import styled from "styled-components"
import { rhythm, scale } from "../utils/typography"

const TOCInner = ({ className, tocitems }) => {
    return (
        <div classname={className} dangerouslySetInnerHTML={{__html: tocitems}} />
    )
}

const TOC = styled(TOCInner)`
padding: ${rhythm(0.5)} ${rhythm(0.25)};
padding-top: ${rhythm(0.5)}
font-style: italic;
color: var(--fg-demisub-color);
${scale(-3 / 8)};
& > a {
  line-height: 0;
}
`
export default TOC

この時点で見た目だけまともなのができますが、まだpage内リンクは機能していません　（ハリボテ・・・）

page内リンクが機能するようにする

上で作ったtable of contentsが動かないのは、page内の<h1> ~ <h6>にidが設定されていないからです。 なのでidを設定するためにpluginを入れます。

idを設定してくれる~~（と嘯いている）~~ pluginはいくつかありますが、
gatsby-remark-autolink-headers を使ってください。
https://www.gatsbyjs.com/plugins/gatsby-remark-autolink-headers/

gatsby-remark-slugやgatsby-remark-heading-slugはgraphQLのtableOfContentsがなにも返さなくなるので使わないように

npm i gatsby-remark-autolink-headers

gatsby-config.js

resolve: `gatsby-transformer-remark`,
      options: {
        plugins: [
          `gatsby-remark-code-titles`,
          `gatsby-remark-autolink-headers`, // <- これ追加
          {
            resolve: `gatsby-remark-images`,
            options: {
              maxWidth: 590
            }
          },

そうするとリンクが機能するようになります。

スクロールに目次が追従するようにする

（僕なんかからすると）モダンなデザインに思えますが、現在ではcssにposition: stickyを設定するだけでこのような動作を実現することができます。

toc.js

const TOC = styled(TOCInner)`
top: 0px;
padding: ${rhythm(0.5)} ${rhythm(0.25)};
padding-top: ${rhythm(0.5)};
margin-top: ${rhythm(0.5)};
margin-left: ${rhythm(0.20)};
margin-right: ${rhythm(0.20)};
position: sticky; // <-これ
........

しかし、単純にposition: stickyを設定するだけでは、うまく動かない場合がしばしばあります。

具体的には、

親要素のどれかに、overflow: hidden overflow: auto が設定されている。
その要素が動ける範囲がない
その要素が固定される位置が明示されていない
などの場合があります。

僕の場合も動かず、原因は3番目でした。

要素の位置を明示するために

toc.js

const TOC = styled(TOCInner)`
top: 0px; // <- これ
padding: ${rhythm(0.5)} ${rhythm(0.25)};
padding-top: ${rhythm(0.5)};
margin-top: ${rhythm(0.5)};
margin-left: ${rhythm(0.20)};
margin-right: ${rhythm(0.20)};
position: sticky; 
........

を指定したら動きました。

まとめ

CSSがうまく動かない系バグはエラーメッセージがないので原因究明が大変です。position: stickyが動かない理屈も分かり次第追記しようと思います。

per_cpu_variableとRELOC_HIDE()

Fri, 06 Nov 2020 05:12:03 GMT

Per cpu variable

linux kernelには、per cpu variableという機構があります。(Per-CPU Variablesとも書かれる。)
per cpu variableはその名前の通りCPU Coreごとの変数です。 coreごとの変数を用いないとキャッシュが汚れてしまうのでパフォーマンスが大幅に低下します。この点から、linux kernelでは多くのper cpu variableが使われています。

per cpu variableが用いられている例としては、バディシステムにおけるper_cpu_pagesなどが挙げられます。

Per-CPU Variablesの概要については、Linux Device Drivers, Third Edition Chapter8 p228 (https://lwn.net/Kernel/LDD3/) にも記述があります。参考にしてください。

注意: プログラム内のコードは解説のための疑似コードであり、動きません

Per cpu variableの配置

linuxはC, asm, ldsで書かれています。ちなみにCのstandard内だけで書くことはできません。このldsとはリンカスクリプトです、linuxはvmlinuxという形に実行形式をまとめますが、そのときのdataやtextなどのセクションの位置を記述しています。 per cpu variableはldsによって配置されアドレスは固定されています。
/arch/x86/kernel/vmlinux.lds.S
/include/asm-generic/vmlinux.lds.h
https://elixir.bootlin.com/linux/v5.9.4/source/include/asm-generic/sections.h#L42

/include/asm-generic/sections.h

extern char __per_cpu_load[], __per_cpu_start[], __per_cpu_end[];

こうして配置されたメモリ、data.per_cpu_area セクションがkernel起動時に初期化されます。
https://elixir.bootlin.com/linux/latest/source/arch/x86/kernel/setup_percpu.c#L168

/arch/x86/kernel/setup_percpu.c

void __init setup_per_cpu_areas(void) {
~~
for_each_possible_cpu(cpu) {
		per_cpu_offset(cpu) = delta + pcpu_unit_offsets[cpu];
		per_cpu(this_cpu_off, cpu) = per_cpu_offset(cpu);
		per_cpu(cpu_number, cpu) = cpu;
		setup_percpu_segment(cpu);
		setup_stack_canary_segment(cpu);
		/*
		 * Copy data used in early init routines from the
		 * initial arrays to the per cpu data areas.  These
		 * arrays then become expendable and the *_early_ptr's
		 * are zeroed indicating that the static arrays are
		 * gone.
		 */
#ifdef CONFIG_X86_LOCAL_APIC
		per_cpu(x86_cpu_to_apicid, cpu) =
			early_per_cpu_map(x86_cpu_to_apicid, cpu);
		per_cpu(x86_bios_cpu_apicid, cpu) =
			early_per_cpu_map(x86_bios_cpu_apicid, cpu);
		per_cpu(x86_cpu_to_acpiid, cpu) =
			early_per_cpu_map(x86_cpu_to_acpiid, cpu);
#endif
~~

Per cpu variablesの構造

per cpu variableは以下のような構造になっています。このCPU 0, CPU 1 という領域にいくつものオブジェクトが入っています。同じ変数(同名だがCPUごとに異なる変数の意味)なら各CPUの領域のstartからのoffsetは等しくなっています。
このため、

（CPUiの構造体Aのptr）= (CPU0の構造体Aのptr) + (i-1)×(各CPU領域のサイズ)

でそれぞれのCPUの変数アドレスを計算することができます。

このような実装はChristophさんの考案です。
参考: https://lwn.net/Articles/258238/

RELOC_HIDE()

offset加算の問題点

先ほど、（CPUiの構造体Aのポインタ）= (CPU0の構造体Aのポインタ) + (i-1)×(各CPU領域のサイズ) で当該CPUのper cpu variableのポインタを取得できると書きましたが、ここには一つの問題があります。
それは、このような操作はC standardを逸脱しておりundefined behaviorに該当するものだということです。
では、なにがUB(Undefined behavior)なのでしょうか？

(i-1)×(各CPU領域のサイズ) = offset  
（CPUiの構造体Aのポインタ）= ptri  
(CPU0の構造体Aのポインタ) = ptr0

とします。
ここで、上の操作は ptri = ptr0 + offsetなわけですが、このoffsetの加算は確実にptr0の指し示している構造体のサイズを超えています。この構造体の範囲を超えるptrの加算がUBになります。

例を示します。

example1.c

struct person {            
    char name[20];        
    int age;                
};

int main(){
  struct person A={"Nanoha", 8}, B={"Fate", 9};
  char* ptr = &A.name;
  int* age_ptr = (int *)(ptr + 20); // in C standard
  char* B_name_ptr = (int *)(ptr + 24); // UB!
  
  printf("vars: %d, %s\n", *age_ptr, B_name_ptr); // vars: 8, Fate
}

構造体の中から中へoffsetを加算してptrを計算するのはCstandardに含まれますが、構造体の外にoffsetを加算してptrを計算するのはUBです。

ただ、上のコードのように動く場合もあります。しかし、UBは「コンパイラがどう処理しても構わない」ということなので、UBを発見次第、「なのはちゃんの全力全開！」と出力してコンパイルを停止するという動作もコンパイラの裁量の範囲内になります。（ほんとか？）

具体的には、

struct person {            
    char name[20];        
    int age;                
};
int main(){
  struct person A={"Nanoha", 8}, B={"Fate", 9};
  struct person *ptr = &A; 
  if(ptr + offset > 0 && ptr + offset < ptr + sizeof(struct person)){
    ~~~~~
  }
  /*
  * ptr + offset > 0 && ptr + offset < ptr + sizeof(struct person)
  * は最適化によって、ptr + offset > 0に変換される可能性がある。
  * ptrに何を足してもその構造体のサイズを超えることがないと仮定されるため。
  */
}

という可能性があります。

RELOC_HIDE

以上の問題を解決するためにRELOC_HIDE()があります。 https://elixir.bootlin.com/linux/v4.18/source/include/linux/compiler-gcc.h#L50

include/linux/compiler-gcc.h

#define RELOC_HIDE(ptr, off)						\
({									\
	unsigned long __ptr;						\
	__asm__ ("" : "=r"(__ptr) : "0"(ptr));				\
	(typeof(ptr)) (__ptr + (off));					\
})

拡張アセンブリ

そもそも、この拡張インラインアセンブリが読めないかもしれないので解説します。
分かりやすい解説: http://caspar.hazymoon.jp/OpenBSD/annex/gcc_inline_asm.html　

__asm__ (
          "" :          // asm template, asmのコード
          "=r"(__ptr) : // 出力レジスタ(カッコ内は対応する変数)
          "0"(ptr)      // 入力レジスタ(カッコ内は対応する変数)
        );

をそれぞれ意味しています。(それぞれの間を:でつないでいる。) アセンブリはレジスタを陽に扱いますが、C言語ではレジスタは扱われません。そのため、単純にアセンブラを書くと、その瞬間レジスタにどの変数が入っているかによって、結果が変わってしまいます。その瞬間のレジスタがどの変数に対応しているかは簡単にはわかりません。この問題に対応するために拡張アセンブリ構文があります。

拡張アセンブリ

入力レジスタに基づいて、変数をレジスタに導入
asm templeteに記述されたasm code
出力レジスタに基づいて、変数をレジスタに導入というアセンブリコードを生成してくれます。

通常のインラインアセンブリ

asm templeteに記述されたasm code だけをやってくれます。

出力入力レジスタに制約というアノテーションをつけることもでき、RELOC_HIDE()の場合では

"=r" = は読み込み専用、rはどのレジスタでもいいという意味
"0"  0 は出力レジスタの0番目にこの変数を入れる

を意味しています。

RELOC HIDEがやっていること

include/linux/compiler-gcc.h

#define RELOC_HIDE(ptr, off)						\
({									\
	unsigned long __ptr;						\
	__asm__ ("" : "=r"(__ptr) : "0"(ptr));				\
	(typeof(ptr)) (__ptr + (off));					\
})

この拡張アセンブリはasm templeteが空文字列になっています。つまり、asmの操作は何もしないで、変数をレジスタに入れて、レジスタから変数に出すということだけを行っています。

入力レジスタに基づいて、変数をレジスタに導入
（asm templeteに記述されたasm code）<-これがない
出力レジスタに基づいて、変数をレジスタに導入

そして、どのように変数をレジスタに出し入れしているかというと、

ptr -> register -> __ptr 　// "0"によってptrは__ptrと同じ変数に対応付けられる

ということは

__ptr = ptr

をやっているだけなんですね。あとは、__にoffsetを足して返すだけ。

これは一見なんの意味もなさそうですが、
アセンブリを介すことによってコンパイラのoptimizerに__ptrがptrと同じ構造体のポインタであることを隠すことができます。 それによって、コンパイラはどんなサイズの構造体のポインタかわからず、offsetを足しても構造体のサイズ内なのかもしれないと考えてコード除去をすることができなくなります。

以上が RELOC_HIDE()の意味になります。

最後に

日本語で解説されているものがなかったので書きました。このptrの加算はアップキャストと似たようなもやっと感がありますね。こういうコードがほとんどのハードウェアの上でちゃんと動作するのはすごいことだと思いました。

Nov 2020 読んだもの、やったこと

Mon, 02 Nov 2020 22:12:03 GMT

やったこと

ハル本2章の最小リスクヘッジの導出

https://mathlog.info/articles/279 このサイト、画像の大きさが変更できない？

A complete React with GraphQL Tutorial

https://www.robinwieruch.de/react-with-graphql-tutorial/

GraphQLをまともに触ったこともなかった。これをみた上で、このブログもtable of contentsをつけた。

読んだもの

The Complete Idiot’s Guide to the Independence of the Continuum Hypothesis: Part 1 of <=Aleph_0

https://www.scottaaronson.com/blog/?p=4974

Aaronsonが書いている。連続体仮説の独立性の証明。GedelとCohenの強制法を含めて、何回かで解説するらしい。 Aaronsonは（コロナの影響で暇だったから？）ちょっと頑張って理解したらしい。研究者のバイタリティ＋理解力は異常。強制法とか日本で理解している人100人 ~ 200人ぐらいの話だと思う。僕も昔から興味はあったんだけども、駒場の本屋で開いた本には理論の本質に入る前の段階の記述が長すぎて、そっ閉じした記憶がある。

僕は知らなかったが、以下のような強制法の初心者向けガイドがあるらしい。いつか読みたい
A beginner’s guide to forcing
Timothy Y. Chow
https://arxiv.org/abs/0712.1320

Linux カーネルのなかに入り込む GCC

https://www.ibm.com/developerworks/jp/linux/library/l-gcc-hacks/

gcc拡張について書いてある。知らないものがいくつもあった。例えば、switchの範囲指定。

char calcGradeByScore(int score){
  switch(score) {
  case 0 ... 49:
    return 'F';
  case 50 ... 69:
    return 'C';
  case 70 ... 80:
    return 'B';
  case 80 ... 100:
    return 'A';
  default:
    return 'X';
  }
}

Deprecated scp

https://lwn.net/SubscriberLink/835962/ae41b27bc20699ad/

scpにはちょっとした脆弱性があって、deprecatedになるらしい。ifconfigは実はdeprecatedなのはそれなりに有名な話？scpもifconfigも手癖で使う人が多いから置換するのは難しそう。あと、直感的に使えて便利だからなぁ。

Loving twins Bitterly Divided by Politics

https://www.nytimes.com/2020/11/04/opinion/2020-election-trump-family.html　　

兄弟がトランプサポーターである(民主党支持の)記者が書いた文章。いい文章だと思う。こういうことがアメリカで沢山起きているんだろうなという話だった。twitterは良くない呟きにノーテーションつけるよりも、違う政治的考えの呟きをサジェスチョンする試みをした方が良いと思う。政治的な分断はアルゴリズムから生み出されている側面もあると思う。共和党支持者には、健康保険に加入できていない人も多い。ギリギリで作り上げられた「今」の生活を守るために、変化をもたらす民主党が嫌悪されるのはその通りなのかもしれない。排他的で属性の違う人を排除することは変化を恐れる心情から生まれているのだと思う。

キャラハンの時空の幾何学 5.1 ~ 5.3

https://arxiv.org/abs/1502.03808
を理解したい（というかweb上でブラックホールの画像を生成したい）という気持ちがあり、その実現のために、一般相対論的な幾何学を勉強していくことにした。この本は（数学的な部分は）とてもわかりやすかった。こういう物理と数学を足して二で割ったような本はちゃんと、数学としてのステートメントか物理としてのステートメントかちゃんと書いといてほしい。

Famed Arecibo telescope, on the brink of collapse, will be dismantled

https://www.sciencemag.org/news/2020/11/famed-arecibo-telescope-brink-collapse-will-be-dismantled

プエルトリコにあるアレシボ天文台の電波望遠鏡が解体されることが決まったらしい。少しさびしい気持ちではある。1963年の建設からいろいろな観測で利用されており、特にパルサーの研究で大きい貢献がある。しかし、アレシボで行われた最も有名な実験は、1974年のM13ヘ地球外知的生命体へのメッセージの送信だと思う（Arecibo message）。このちょっと特殊なエイリアンとの縁（？）からアレシボは様々なフィクション作品に登場している。僕の読んだことあるうちでは、テッド・チャンの「息吹」（必読！）に収録されている「The Great Silence」はアレシボの隣の熱帯雨林にいる絶滅寸前のヨウムとアレシボの関係（？、ヨウムの独白？）を描いている。あの、ヨウムたちはアレシボが解体されたら、さぞさびしいだろう

Why you should use `nproc` and not grep /proc/cpuinfo

https://www.flamingspork.com/blog/2020/11/25/why-you-should-use-nproc-and-not-grep-proc-cpuinfo/

nprocは現在のプロセスにとってavailableなcpuの数(hardware thread?)を返す。一方、/proc/cpuinfoはonlineなcpuの数を返す。違いは、dockerの中から見た時にわかりやすく現れる。

$ nproc
8

$ docker run --cpuset-cpus=0-1 --rm=true -it  amazonlinux:2
bash-4.2# nproc
2
bash-4.2# exit

$ docker run --cpuset-cpus=0-2 --rm=true -it  amazonlinux:2
bash-4.2# nproc
3

$ /usr/bin/lscpu -p | grep -c "^[0-9]"
8
$ grep -c 'processor' /proc/cpuinfo 
8

$ docker run --cpuset-cpus=0-1 --rm=true -it  amazonlinux:2
bash-4.2# yum install -y /usr/bin/lscpu
......
bash-4.2# /usr/bin/lscpu -p | grep -c "^[0-9]"
8
bash-4.2# grep -c 'processor' /proc/cpuinfo 
8
bash-4.2# nproc
2

自分用備忘録 CS自学者のために(Teach Your Self Computer Science)

Mon, 22 Jun 2020 22:12:03 GMT

概要

Teach Your Self Computer Science : https://teachyourselfcs.com/
のうちの自分が興味のある部分を自分用備忘録としてまとめておく。

言語・コンパイラ

インタプリタを作る crafting interpreters : https://craftinginterpreters.com/contents.html
ドラゴンブックをやる: https://smile.amazon.com/Compilers-Principles-Techniques-Tools-2nd/dp/0321486811
ドラゴンブックの補足として: https://www.edx.org/course/compilers

ドラゴンブックをわかれば、わかっておくべきことのほとんどすべてはわかっていると言っても良い。

分散システム

まずこの本を読むべき: Designing Data-Intensive Applications
動画がいいなら: MIT’s 6.824 https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39_tb2fErI4-WkMbsvGQk9_UB
動画のテキストは: https://pdos.csail.mit.edu/6.824/schedule.html

そしたら、論文を読めばいい
読むべき論文リストは: http://dsrg.pdos.csail.mit.edu/papers/

コンピュータネットワーク

まず、やる本: Computer Networking: A Top-Down Approach
この本のExcersizeだけでもやるべき（オンライン上で完璧に提供されている）: http://www-net.cs.umass.edu/wireshark-labs/

データベース

このコースを見よう: recordings of CS 186 https://www.youtube.com/user/CS186Berkeley/videos
RDBがなにやってるかはこの本をみる: Architecture of a Database System http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf

それが終わったら論文まとめ: the databases “Red Book” http://www.redbook.io/

Operating System

基本の教科書: http://pages.cs.wisc.edu/~remzi/OSTEP/

OSの個別論
Unix: Lions’ Commentary on Unix
MacOS: Mac OS X Internals: A Systems Approach
Linux: Linux Kernel Development

そしたら、xv6を実際にみて機能を追加してみる
xv6: https://pdos.csail.mit.edu/6.828/2016/xv6.html
追加する機能の例: http://pages.cs.wisc.edu/~remzi/OSTEP/lab-projects-xv6.pdf
追加する機能例はここでも見れる: https://github.com/remzi-arpacidusseau/ostep-projects

Jun 2020 読んだもの、やったこと

Sat, 20 Jun 2020 22:12:03 GMT

やったこと

libfuzzer tutorial

https://github.com/google/fuzzing/blob/master/tutorial/libFuzzerTutorial.md

fuzzingとは、ランダムな入力をプログラムに入力してバグを検出すること。特にメモリ周り(heap-overflow, stack-overflow)。コードカバレッジが100%になるまで上手いこと入力を変化させてくれる。
そんなん上手くいくのかと思うが、上手くいくときもあるし、上手くいかないときにプログラマが入力のヒントをいれることもできる。

実際、CVE-2014-0160　https://www.ipa.go.jp/security/ciadr/vul/20140408-openssl.html　　はfuzzingで検出できる。（できるだけで、fuzzingによって初めて発見されたかは自信がない）

僕はテストとか人間が書くべきなのか？と思っており、fuzzingに希望を見出している。（テストを書くのは（反例を作るのは）人間の脳には難しいと思う）
ただ、メモリリークとかを検出するのは難しいらしい。

やりたいひとはgitpodで環境構築したので下のリポジトリからどうぞ
https://github.com/yukarinoki/fuzzing-tutorial

仕組みが知りたい人は
https://github.com/google/fuzzing/blob/master/docs/structure-aware-fuzzing.md

読んだもの

Into Thin Error: Mountaineer Ed Viesturs on Making Mistakes

https://slate.com/news-and-politics/2010/06/into-thin-error-mountaineer-ed-viesturs-on-making-mistakes.html

登山家が極限状態でのミスの経験とかミスに対する考えとかを喋ってる。
山登ってみたいけど死ぬのはこわい。登山家の課題は、神様が（地球が）与えた形で自然に定まってるから、羨ましいなと思う。ピオレドールとかすごく夢あっていい。コンピュータ科学はマッチポンプみたいなところがあるので。

天才 Ueli Stechが死んだときからずっと思ってるけど、どれほど能力があっても山に登ってれば漸近的に100%死ぬから、そういう事実と登山家が生きたいと思う感情はどういう風に混ざり合っているのか、どこかの登山家にエッセイでも書いて欲しい。 https://www.youtube.com/watch?v=VUWBbepsdmY

日本人の｢給料安すぎ問題｣の意外すぎる悪影響

https://toyokeizai.net/articles/-/357011?page=4　　

「300円の牛丼が美味しいから日本人は幸せ」みたいな論法を聞くと、この記事に書いてあるような気持ちがむくむく湧いてくる。 monopsonyという言葉は覚えておこうと思った。

console.mihai()

Firefoxのコンソールを立ち上げ（F12）console.mihai()と入力すると表示されるURLから読める。

いい話、エンジニアっていい仕事ですね。

ライフタイムで参照を検証する

https://doc.rust-jp.rs/book/second-edition/ch10-03-lifetime-syntax.html

Rustを理解中。ライフタイムみたいな概念は初めて知ったが、C++で構造体・クラスにポインタを入れるときには「このポインタがいつまでもつかちゃんと管理し続けられるのかな・・・」と毎回思う。C++はそれをIQで何とかする設計なので、書くときはお腹に力をいれていたけど、Rustなら安心できそう。

Gastby + Netlifyで高速なblog

Wed, 03 Jun 2020 22:12:03 GMT

Gatsby + Netlifyが早いらしいと聞いたので作ってみた本当に速いのですごいぞ。