The Metric That Drifts to Zero: An Accounting Bug in Tor's Connection Gauge

By 1AEO Team • July 10, 2026 • Tor 0.4.8.x MetricsPort • Follow-up to our June DoS-wave post

While investigating the circuit-building DoS wave on our Guard-flagged relays, one metric kept telling a story the others contradicted. Tor's inbound connection gauge, tor_relay_connections{state="opened", direction="received"}, would fall steadily toward zero for a week or two, then jump back to a large value, then fall again. Our alerting could read it as a connection flood (the snap-back) or a total collapse (the decay) on the same relay in the same month.

The source contains a path that can make the inbound gauge undercount: an accepted connection can be marked for close before its matching open increment. The stored gauge is floored at zero and is not recounted when exported. The accept path and accounting functions establish this mechanism.

The Symptom

The replacement chart uses hourly historical samples from April 11, 00:00 through July 10, 00:00 UTC. Tor's series sum type="OR" gauges over IPv4 and IPv6; node_netstat_Tcp_CurrEstab counts TCP connections in ESTABLISHED or CLOSE-WAIT, in both directions and across all ports in the monitored network namespace. Each series is divided by its own median. They measure different populations:

• The direction="received" gauge spans about 104x from minimum to maximum in this recalculation. It repeatedly falls while the kernel TCP total changes much less.
• The kernel total is an independent cross-check. Its different states, directions and scope mean it cannot establish the true number of inbound Tor connections.
• Reporting populations vary from 408 to 981 Tor processes and three to seven kernel hosts. The lower panels show that coverage; dips and jumps cannot automatically be attributed to restarts or traffic changes.
• The direction="initiated" gauge varies less relative to its own median. This is consistent with different accounting paths, but does not measure how often the inbound failure path ran.

The Gauge Is an Accumulator, Not a Live Count

In src/feature/relay/relay_metrics.c, the fill function for this metric does not walk the list of live connections. It reads a stored counter:

// relay_metrics.c : fill_conn_gauge_values()
fill_single_connection_value(sentry, i, "received", "opened", AF_INET,
                             rep_hist_get_conn_opened(true, i, AF_INET));

A stored gauge can be accurate when its updates balance. Here, a specific failure path can subtract without the matching addition.

The Accounting Asymmetry

The backing counter lives in src/feature/stats/rephist.c. It is incremented in one function and decremented in another:

// rephist.c
void rep_hist_note_conn_opened(bool from_listener, unsigned type, int af) {
  ...
  conn_num_opened_v4[dir][type]++;          // OPEN: +1
}
void rep_hist_note_conn_closed(bool from_listener, unsigned type, int af) {
  ...
  if (conn_num_opened_v4[dir][type] > 0)    // the underflow floor
    conn_num_opened_v4[dir][type]--;        // CLOSE: -1 (never below 0)
}

The bug is where those two calls are reached from, in src/core/mainloop/connection.c:

• connection_mark_for_close_internal_() decrements the gauge the first time an object is marked for close, whether or not its matching increment happened. Some failure paths free the object directly instead.
• The increment for an inbound connection runs only at the very end of connection_handle_listener_read(), after the connection has been added to Tor's connection list and its handshake has been initialized:

// connection.c : connection_handle_listener_read()  (inbound accept path)
newconn = connection_new(new_type, conn->socket_family);   // object exists
...
newconn->from_listener = 1;
if (connection_add(newconn) < 0) { connection_free(newconn); return 0; }   // now on the list
if (connection_init_accepted_conn(newconn, ...) < 0) {
    if (!newconn->marked_for_close)
        connection_mark_for_close(newconn); // DECREMENT before the increment
    return 0;                             //     ...and we return before the increment
}
note_connection(true /* inbound */, newconn);   // inbound increment

There is a window after an inbound connection enters Tor's list but before note_connection() counts it. If connection_init_accepted_conn() fails, the close path can decrement the gauge without a matching increment. That establishes an undercount mechanism, not its frequency or the peer behavior that caused a particular failure.

The outbound path increments after connect() succeeds or reports an in-progress connection, and before adding it to the connecting list. An immediate connect() failure returns before the increment. That differs from the inbound path; it is not a proof that every outbound failure path balances.

Why the Error Can Persist

A normally counted connection adds one when opened and subtracts one when marked for close. An unmatched decrement can leave a persistent shortfall because the export path does not recount live connections. The zero floor prevents negative values. A process restart resets the bookkeeping, which then rebuilds as new connections are counted. Connection turnover and the floor mean the error is not necessarily permanent.

Reproducing It

For this correction, we compiled the open, close and getter functions copied from tor-0.4.8.16 into an isolated accounting model. We counted 40,000 simulated open connections, then called the close function without corresponding opens. After 40,000 unmatched calls, the exported gauge reached zero and stayed there. The caller schedule is synthetic; it demonstrates the accounting behavior rather than a network attack or production frequency.

A Proposed Fix

One proposed approach is to count eligible connections when exporting the gauge — walk Tor's connection array, bucket by direction, type and address family, and skip objects marked for close:

// relay_metrics.c : count live connections instead of reading the accumulator
SMARTLIST_FOREACH_BEGIN(get_connection_array(), const connection_t *, conn) {
    if (conn->marked_for_close) continue;
    unsigned dir = conn->from_listener ? 1 : 0;
    ... tally[dir][conn->type][af]++ ...
} SMARTLIST_FOREACH_END(conn);

This avoids the mutable gauge accumulator, but the connection-state definition, correctness and scrape cost still need testing. The created and rejected counters use separate cumulative bookkeeping. This is a code sketch, not a validated patch or a claim that an upstream fix has been filed, accepted or released.

If You Run Relays: What to Trust Instead

For the source version examined here, cross-check the inbound gauge before using it for alerts. Other metrics have their own scope:

• tor_relay_connections_total{state="created"} is cumulative creation bookkeeping. Its rate avoids the gauge's decrement drift, but does not count concurrency or every accepted socket: the early inbound failures above return before its increment. Account for process resets when calculating rates.
• node_netstat_Tcp_CurrEstab counts TCP connections in ESTABLISHED or CLOSE-WAIT, in both directions across the monitored network namespace. Match scope as closely as possible when comparing it with Tor.
• Cross-check any gauge against an independent source before interpreting a change as a traffic event. Compare states, directions, scrape coverage and process restarts as well as values.

As with the corrected listener-drop analysis, the useful question is what the metric actually measures. A kernel counter and an application gauge can disagree without either being an exact count of the population you meant to measure.

Join the Mission