Running a benchmark
run drives the spec’s transaction mix against the data init loaded:
pgmeter [connection options] run --spec <DIR|FILE> [-c N] [-T N] [-r N] [-P N] [-j N] [--once] [--seed N] [--format FMT] [-q]
| Option | Default | Description |
|---|---|---|
--spec | required | The same spec init was given |
-c, --client | 1 | Concurrent clients, one connection each |
-T, --time | 1 | Measured duration in seconds, after the ramp-up |
-r, --rampup | 0 | Ramp-up in seconds: the clients start spread over this window, and nothing in it counts |
-P, --progress | 0 | Every N seconds, print a progress: line when stderr is not a terminal: the current throughput and average latency (over the last second) of a timed run, the transactions done and in flight of a --once run; on a terminal the progress bar shows them live |
-j, --threads | 0 | Worker threads of the async engine (see below); 0 is half the CPUs, at most one per client |
--once | off | Each client runs every transaction once, in its own random order, then stops (see below); not with -T |
--seed | none | Seed of the clients’ random draws, so that two runs issue the same transactions with the same parameters (see below) |
--format | table | table, json or csv for the results on stdout |
-q, --quiet | off | No progress output; --format json and csv imply it |
Use the same --spec for init and run: the spec is the single source of
truth for the schema and the workload, and run reads its scaling query and
transactions from it.
What a run does
- Reads the largest scale-unit id from the data (the spec’s
max_id_query), so the workload spreads over everythinginitloaded. - Starts the clients one by one over the ramp-up window. Each client runs transactions back to back, with no think time, picking the next one at random according to the spec’s weights.
- At the end of the ramp-up, starts counting. Committed and failed
transactions and their response times are collected for
-Tseconds. - Stops the clients, writes the results directory and prints the summary.
Every transaction is logged with its response time, ramp-up included, but the headline figures, the per-transaction statistics and the report’s histogram describe the steady-state window only. The time-series charts show the whole run with the ramp-up shaded.
The two engines
How the clients are driven depends on the spec, and the run header says which engine is in use.
- Simple-mode specs (every transaction is SQL with bindings, like PGM-B)
run on the async engine: a pool of
--threadsworker threads drives one task per client, like pgbench’s-j. A worker that finds its client waiting for the server moves on to the next client, so the process wakes up far less often than one thread per client would, and the load generator takes less CPU away from a server on the same machine. The default of half the CPUs suits a machine shared with the server; a dedicated client machine can use more. - Specs with a script transaction (Lua, like PGM-C and PGM-H) run one
thread per client on a blocking connection, because the Lua database bridge
is synchronous.
--threadsis ignored.
Running every transaction once
--once replaces the timed random stream with the query-stream model of the
TPC-H and TPC-DS power and throughput tests: each client gets its own random
permutation of the spec’s transactions, runs each of them exactly once, and
stops. The run ends when the last client is done, so -T is not used and
cannot be given. Transactions with a weight of 0 are skipped; the other
weights are ignored.
-c 1 --onceis a power test: every query once, in one stream.-c 4 --onceis a throughput test with four streams, each in a different order, concurrently.
The run is reported as a stream: where its time went, transaction by
transaction, with a timeline per client (see Reading the
results). duration_s and workload_secs in run.json are
set to the window the run took, and "once": true. The ramp-up
still staggers the clients’ start, and Ctrl-C still ends the run early.
Running the same transactions twice
Without --seed, every client draws its transaction order and the parameters
of its queries from a generator seeded from the operating system, so no two
runs execute the same statements. Averaged over a long run that does not
matter. It does matter when two runs are compared query by query: a TPC-DS
query whose parameter selects one of a few stores can take forty times longer
in one run than in the other for no reason but the draw, and the difference
looks like the server’s.
--seed N seeds each client’s generator from the seed and the client id. Two
runs with the same seed, the same spec and the same number of clients then run
the same transactions with the same parameters, client for client, on either
engine and whatever --threads: a --once run repeats exactly, and a timed
run repeats the same stream for as long as it lasts. The seed is echoed in
the run header and recorded as seed in run.json.
pgmeter -d db_a run --spec pgm-ds --once --seed 42
pgmeter -d db_b run --spec pgm-ds --once --seed 42 # the same 99 queries, same parameters
The seed covers what the clients draw: the order of the transactions and
every rand_* call of their bindings and scripts. It does not cover the data,
which init --seed covers, nor the timing of the clients against each other.
Stopping early
Ctrl-C ends the run at the end of each client’s current transaction. The
results and the report cover what ran so far and are marked as interrupted
("interrupted": true in run.json), and pgmeter exits normally. A second
Ctrl-C aborts immediately with exit status 130.
Errors during the run
A transaction that fails, whether the server rejected a statement or a Lua
script raised an error, is counted in the errors column and its message is
written to error.log. The client rolls back and carries on with the next
transaction; a failed transaction’s duration is not counted in the latency
statistics. Some workloads fail on purpose: PGM-C’s New-Order rolls back
about 1% of its orders, as TPC-C prescribes, and those show up as errors with
the message New-Order rollback: unknown item.
A client that cannot connect at all stops the run with an error and exit status 1, after the collector has written the logs of what ran.
Output formats
--format table prints the headline and the two tables described in
Reading the results. --format json prints one JSON document
with the same figures and --format csv one CSV table, both for scripting;
they imply --quiet so stdout holds only the result data.