Reference · observability
Eighteen metrics, two routes, and a list of what is never logged.
A forwarder sees every message you receive, so what it records is a decision rather than an implementation detail. The whole of it is on this page, including the enforcement.
The endpoint
[observe] listen = "127.0.0.1:9779"
Two routes, and nothing else — any other path is a 404. There is no framework behind it; two routes did not justify one.
| GET /metrics | text/plain | Prometheus text exposition format. Always 200. |
| GET /healthz | text/plain | 200 when well, 503 when not, with the reason in the body. |
Bind it to localhost. It is unauthenticated, and the output says which domains this host handles and how much mail each gets. Put your monitoring agent on the same host, or a reverse proxy in front.
Omit the block and there is no endpoint, which also means mxhop health has nothing to ask and a container healthcheck cannot work.
Health
mxhop health # or: curl -s localhost:9779/healthz ok listener: ok spool: ok queue: 0/100000 resolver: ok
| Line | Not ok when |
|---|---|
| listener | The listener has stopped accepting — during a graceful shutdown, or after a bind that failed. This is the one that should take the host out of rotation. |
| spool | The spool directory could not be read or written. Almost always a full disk or a permissions change. Mail is being refused while this says no, because mxhop will not accept what it cannot store. |
| queue | Depth against the hard ceiling of 100,000. A number that climbs is the signal; the ceiling is a backstop, not a target. |
| resolver | DNS is not answering. Nothing can be delivered without it, so everything defers. |
The depth figure is refreshed every 30 seconds by a background sweep, so it can lag a sudden burst by that much. mxhop health exits 0 on a 200 and 1 on anything else, including no answer within five seconds.
Every metric
Names, types and help strings exactly as served. All counters are monotonic; use rate() on them.
Sessions
| mxhop_sessions_total | counter | Inbound SMTP connections served. |
| mxhop_sessions_tls_total | counter | Inbound connections that reached STARTTLS. Against the total, this is your encryption rate — a number worth watching, since it should be high and a sudden drop means the certificate broke. |
| mxhop_sessions_failed_total | counter | Inbound connections that ended in an error. |
| mxhop_sessions_open | gauge | Inbound connections being served right now. |
| mxhop_session_seconds | histogram | How long an inbound session lasted. |
Messages and recipients
A message has one end-of-DATA outcome; a recipient copy has its own. One message to three mailboxes is one messages_accepted and three recipients_*.
| mxhop_messages_accepted_total | counter | Messages taken at end of DATA. |
| mxhop_messages_refused_total | counter | Messages refused at end of DATA. |
| mxhop_recipients_delivered_total | counter | Recipient copies a destination took. |
| mxhop_recipients_deferred_total | counter | Recipient copies a destination deferred. |
| mxhop_recipients_refused_total | counter | Recipient copies a destination refused for good. |
| mxhop_auth_refusals_total | counter | Messages refused by the DMARC rule. Forgeries turned away at the door — expect this to be non-zero on any public MX. |
| mxhop_delivery_seconds | histogram | How long one destination group took. |
Queue
| mxhop_queue_depth | gauge | Messages waiting in queue/ at the last sweep. |
| mxhop_queue_sweeps_total | counter | Queue sweeps run. Flat means the runner has stopped, which is worth an alert of its own. |
| mxhop_queue_retries_total | counter | Delivery attempts made from the queue. |
| mxhop_queue_frozen_total | counter | Messages given up on and moved to frozen/. Every increment is mail that did not arrive. |
Delivery reports
| mxhop_reports_queued_total | counter | Delivery reports written to the queue. |
| mxhop_reports_suppressed_total | counter | Failures no report was sent for, because the sender could not be proved. Not an error — each one is backscatter you did not send. |
Histogram buckets, in seconds: 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, +Inf. Each observation is counted in every bucket it fits, as the format requires, and _sum and _count are served alongside.
No metric is labelled with an address, a subject or a message id. Cardinality is fixed: eighteen series plus histogram buckets, whatever volume arrives. That is a privacy decision as much as a performance one — a metric label is a log line that gets scraped and stored somewhere else.
Logging
mxhop run --log info # a level
mxhop run --log info,mxhop::queue=debug # per module
mxhop run --log-json # one JSON object per line
Fields are the same in both formats. Every line about a message carries its queue id, so one grep gives you the whole story:
journalctl -u mxhop | grep 1795110377370-000003-0007
In a log line
| queue id | The thread through everything |
| peer, helo | The connection's facts |
| envelope | MAIL FROM and resolved recipients |
| Message-ID | So a line here matches a line in the destination's log. Truncated at 200 characters. |
| size, hops | Counts, never content |
| verdicts | SPF, DKIM, DMARC and the policy found |
| their reply | The destination's own words |
Never, at any level
| Subject | Not in a line, not in a label |
| Body | Not a byte |
| Headers | Any the sender wrote, apart from the Message-ID |
This is enforced, not promised. A test parses every tracing call in the source and fails the build if one could print message content — so a length has to be computed and named before it is logged, where a reviewer sees it.
--log trace gives more lines about the same facts, not the mail.
What to alert on
Four things, in the order they matter. Everything else is a dashboard.
| Alert | Expression | Why it is first |
|---|---|---|
| spool unwritable | /healthz non-200 | Mail is being refused right now. Everything else can wait. |
| queue climbing | deriv(mxhop_queue_depth[30m]) > 0 for an hour | Something is systematically not being accepted. Normal spikes drain. |
| messages frozen | increase(mxhop_queue_frozen_total[1h]) > 0 | Mail that did not arrive, 48 hours after it was sent. Always worth a look. |
| TLS rate dropped | rate(sessions_tls_total) / rate(sessions_total) falling | The usual cause is an expired certificate and a missed reload hook. Senders that require TLS are already deferring. |
Do not alert on mxhop_auth_refusals_total or mxhop_reports_suppressed_total being non-zero. Both counting up is mxhop working: forgeries turned away, and backscatter not sent.