mxhop

Reference · observability

Eighteen metrics, two routes, and a list of what is never logged.

A forwarder sees every message you receive, so what it records is a decision rather than an implementation detail. The whole of it is on this page, including the enforcement.

The endpoint

[observe]
listen = "127.0.0.1:9779"

Two routes, and nothing else — any other path is a 404. There is no framework behind it; two routes did not justify one.

GET /metricstext/plainPrometheus text exposition format. Always 200.
GET /healthztext/plain200 when well, 503 when not, with the reason in the body.

Bind it to localhost. It is unauthenticated, and the output says which domains this host handles and how much mail each gets. Put your monitoring agent on the same host, or a reverse proxy in front.

Omit the block and there is no endpoint, which also means mxhop health has nothing to ask and a container healthcheck cannot work.

Health

mxhop health          # or: curl -s localhost:9779/healthz
ok
listener: ok
spool: ok
queue: 0/100000
resolver: ok
LineNot ok when
listenerThe listener has stopped accepting — during a graceful shutdown, or after a bind that failed. This is the one that should take the host out of rotation.
spoolThe spool directory could not be read or written. Almost always a full disk or a permissions change. Mail is being refused while this says no, because mxhop will not accept what it cannot store.
queueDepth against the hard ceiling of 100,000. A number that climbs is the signal; the ceiling is a backstop, not a target.
resolverDNS is not answering. Nothing can be delivered without it, so everything defers.

The depth figure is refreshed every 30 seconds by a background sweep, so it can lag a sudden burst by that much. mxhop health exits 0 on a 200 and 1 on anything else, including no answer within five seconds.

Every metric

Names, types and help strings exactly as served. All counters are monotonic; use rate() on them.

Sessions

mxhop_sessions_totalcounterInbound SMTP connections served.
mxhop_sessions_tls_totalcounterInbound connections that reached STARTTLS. Against the total, this is your encryption rate — a number worth watching, since it should be high and a sudden drop means the certificate broke.
mxhop_sessions_failed_totalcounterInbound connections that ended in an error.
mxhop_sessions_opengaugeInbound connections being served right now.
mxhop_session_secondshistogramHow long an inbound session lasted.

Messages and recipients

A message has one end-of-DATA outcome; a recipient copy has its own. One message to three mailboxes is one messages_accepted and three recipients_*.

mxhop_messages_accepted_totalcounterMessages taken at end of DATA.
mxhop_messages_refused_totalcounterMessages refused at end of DATA.
mxhop_recipients_delivered_totalcounterRecipient copies a destination took.
mxhop_recipients_deferred_totalcounterRecipient copies a destination deferred.
mxhop_recipients_refused_totalcounterRecipient copies a destination refused for good.
mxhop_auth_refusals_totalcounterMessages refused by the DMARC rule. Forgeries turned away at the door — expect this to be non-zero on any public MX.
mxhop_delivery_secondshistogramHow long one destination group took.

Queue

mxhop_queue_depthgaugeMessages waiting in queue/ at the last sweep.
mxhop_queue_sweeps_totalcounterQueue sweeps run. Flat means the runner has stopped, which is worth an alert of its own.
mxhop_queue_retries_totalcounterDelivery attempts made from the queue.
mxhop_queue_frozen_totalcounterMessages given up on and moved to frozen/. Every increment is mail that did not arrive.

Delivery reports

mxhop_reports_queued_totalcounterDelivery reports written to the queue.
mxhop_reports_suppressed_totalcounterFailures no report was sent for, because the sender could not be proved. Not an error — each one is backscatter you did not send.

Histogram buckets, in seconds: 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, +Inf. Each observation is counted in every bucket it fits, as the format requires, and _sum and _count are served alongside.

No metric is labelled with an address, a subject or a message id. Cardinality is fixed: eighteen series plus histogram buckets, whatever volume arrives. That is a privacy decision as much as a performance one — a metric label is a log line that gets scraped and stored somewhere else.

Logging

mxhop run --log info                      # a level
mxhop run --log info,mxhop::queue=debug   # per module
mxhop run --log-json                      # one JSON object per line

Fields are the same in both formats. Every line about a message carries its queue id, so one grep gives you the whole story:

journalctl -u mxhop | grep 1795110377370-000003-0007

In a log line

queue idThe thread through everything
peer, heloThe connection's facts
envelopeMAIL FROM and resolved recipients
Message-IDSo a line here matches a line in the destination's log. Truncated at 200 characters.
size, hopsCounts, never content
verdictsSPF, DKIM, DMARC and the policy found
their replyThe destination's own words

Never, at any level

SubjectNot in a line, not in a label
BodyNot a byte
HeadersAny the sender wrote, apart from the Message-ID

This is enforced, not promised. A test parses every tracing call in the source and fails the build if one could print message content — so a length has to be computed and named before it is logged, where a reviewer sees it.

--log trace gives more lines about the same facts, not the mail.

What to alert on

Four things, in the order they matter. Everything else is a dashboard.

AlertExpressionWhy it is first
spool unwritable/healthz non-200Mail is being refused right now. Everything else can wait.
queue climbingderiv(mxhop_queue_depth[30m]) > 0 for an hourSomething is systematically not being accepted. Normal spikes drain.
messages frozenincrease(mxhop_queue_frozen_total[1h]) > 0Mail that did not arrive, 48 hours after it was sent. Always worth a look.
TLS rate droppedrate(sessions_tls_total) / rate(sessions_total) fallingThe usual cause is an expired certificate and a missed reload hook. Senders that require TLS are already deferring.

Do not alert on mxhop_auth_refusals_total or mxhop_reports_suppressed_total being non-zero. Both counting up is mxhop working: forgeries turned away, and backscatter not sent.