pgmeter

pgmeter is a PostgreSQL benchmarking tool written in Rust. A benchmark is a YAML specification plus Lua scripts: the spec declares the schema, how to load the data and the transaction mix, and pgmeter initializes the database and drives the workload. Adding a workload never means writing Rust or recompiling anything.

Six reference workloads ship inside the binary, ready to run: PGM-B, PGM-C and its client-driven variant, PGM-H, PGM-DS and PGM-IoT, derived from the TPC-B, TPC-C, TPC-H, TPC-DS and TPCx-IoT specifications, plus a minimal template to start your own. They are not audited TPC benchmarks and their results are not comparable to official TPC results; see License and trademarks.

What it does

• Benchmarks as data. Schema, indexes, data loading and the workload live in one spec file. Start from a bundled workload with pgmeter spec new, or from a one-table template, and edit.
• Lua where logic is needed. Transactions and loaders can be Lua (Luau) scripts with a database bridge (exec, query, query_one, transaction control, COPY streaming) and fast random-data helpers.
• Two execution engines. Specs made of plain SQL transactions run on an async engine that drives every client from a small pool of worker threads, like pgbench; specs with Lua transactions run one thread per client.
• Measured properly. Concurrent clients with a ramp-up window that is excluded from the statistics, throughput in the unit the spec defines (TPM, TPS, queries per hour), response-time percentiles, error rates, live progress, and a self-contained HTML report with charts.
• Fast loading. Parallel COPY-based data loading with per-job id ranges.

Where to start

• Installation, then your first benchmark: a PGM-B run from an empty database to the report in five minutes.
• The user guide for every option of init and run, the results directory and the bundled workloads.
• Writing benchmarks when the bundled workloads are not what you need: every spec field, both loading styles, both transaction modes and a worked example.
• The reference for the command line, the Lua API and the output file formats.

A taste

export PGPASSWORD=secret
psql -h db -U postgres -c "CREATE DATABASE benchmark"

# Write an editable copy of the bundled PGM-B spec into ./pgm-b
pgmeter spec new pgm-b -o .

# Load it at scale factor 10 with 4 loader threads
pgmeter -H db -U postgres -d benchmark init --spec pgm-b -s 10 -j 4

# 8 clients for 60 seconds after a 10 second ramp-up
pgmeter -H db -U postgres -d benchmark run --spec pgm-b -c 8 -T 60 -r 10

pgmeter prints the throughput, the committed and failed transactions and the response-time statistics of the steady-state window, and writes a results directory with the raw logs, per-second CSVs and report.html.

View a sample report: throughput, a PGM-C run with 10 warehouses and 8 clients, or stream, PGM-DS at scale factor 1 as two query streams.

Installation

Requirements

• Rust 1.87 or later to build (declared as rust-version in Cargo.toml).
• PostgreSQL 12 or later to benchmark. pgmeter connects over the network like psql does; it does not need to run on the database server.
• Linux or macOS. pgmeter is developed and tested on Linux; Windows is not supported.

From source

git clone https://github.com/julmon/pgmeter.git
cd pgmeter
cargo build --release

The binary is target/release/pgmeter. Copy it anywhere on your PATH; it has no runtime dependencies. The bundled workloads, the Lua runtime and the report’s chart library are embedded in it, so a source checkout is not needed after the build.

TLS uses rustls with the system certificate store, so nothing has to be installed for --sslmode; see Connecting to PostgreSQL.

Checking the installation

pgmeter --version
pgmeter spec list

spec list prints the bundled workloads with their description. Next: your first benchmark.

Your first benchmark

This page runs the bundled PGM-B workload, a banking debit/credit transaction derived from TPC-B, against a database of your own, and shows what comes out. It takes about five minutes.

1. A database to benchmark

init drops and recreates the tables a spec declares, so use a database you can afford to wipe:

export PGPASSWORD=secret
psql -h db -U postgres -c "CREATE DATABASE benchmark"

pgmeter reads the same connection settings as psql: -H/PGHOST, -p/PGPORT, -U/PGUSER, -d/PGDATABASE, PGPASSWORD and --sslmode/PGSSLMODE. The examples below pass the host, user and database on the command line; exporting the environment variables once is shorter.

2. A copy of the spec

The bundled workloads are embedded in the binary. spec new writes an editable copy of one into a directory, which is what init and run take:

pgmeter spec new pgm-b -o .
ls pgm-b

The directory holds spec.yaml, the whole benchmark: the four tables and their indexes, how to fill them and the one transaction with its weights and SQL. Workloads with Lua scripts also get a scripts/ folder.

3. Load the data

pgmeter -H db -U postgres -d benchmark init --spec pgm-b -s 10 -j 4

-s 10 is the scale factor: PGM-B’s unit is a branch, with 10 tellers and 100,000 accounts each, so this creates one million accounts. -j 4 loads with four parallel connections. The steps print as they run: schema, pre-loading, loading, primary keys, foreign keys, indexes, vacuum, checkpoint. At scale 10 this takes well under a minute.

4. Run the workload

pgmeter -H db -U postgres -d benchmark run --spec pgm-b -c 8 -T 60 -r 10 -P 10

Eight clients run for 60 seconds after a 10 second ramp-up during which they start one by one; -P 10 prints the current throughput and average latency every 10 seconds when the output is not a terminal (on a terminal a progress bar shows them live). The ramp-up is excluded from every statistic. Press Ctrl-C to end a run early with the results of what ran so far.

5. Read the results

The console ends with a headline line (throughput in the spec’s unit, committed transactions, error rate, average response time), a results table with one row per transaction and a response-time table with the average, minimum, maximum, standard deviation and the 95th and 99th percentiles.

A results directory named after the start time, like pgmeter-20260101T120000Z/, holds report.html, a self-contained page with the same figures and charts of throughput, latency and errors over time, plus the raw transaction and error logs and per-second CSV files. Open the report in a browser; it needs no network.

Reading the results describes each file and figure.

Next

• Run the same spec against two configurations and compare the headline figures: that is what the tool is for.
• Bundled workloads lists the other five and what each models.
• Writing benchmarks explains the spec you just copied and how to write one for your own schema and queries.

Connecting to PostgreSQL

The connection options are global: they come before the subcommand and apply to init, run and spec alike. Each one has the environment variable the PostgreSQL tools use, so an exported PGHOST, PGUSER, PGDATABASE and PGPASSWORD are enough.

The password comes from PGPASSWORD when it is set, otherwise from the password file, PGPASSFILE or ~/.pgpass, matched on host, port, database and user as libpq does. There is no prompt and no --password option. The target shown in the run header has the password redacted.

PGOPTIONS passes server settings to every connection pgmeter opens, as libpq does: init, each client of run and the report’s own connection all start with them. For example, to run with a larger work_mem and no statement timeout without touching the server’s configuration:

PGOPTIONS="-c work_mem=256MB -c statement_timeout=0" pgmeter run --spec pgm-h.yaml

The run header shows them in the target, as the options parameter. Other libpq environment variables, such as PGAPPNAME or PGCONNECT_TIMEOUT, are not read.

TLS

--sslmode follows libpq’s meaning:

TLS is provided by rustls, so no OpenSSL is needed. The verify-* modes use the operating system’s certificate store; a custom CA file (libpq’s sslrootcert) and client certificates are not supported yet. A server without TLS and a require or verify-* mode fail with one error at connection time.

Connections pgmeter opens

• init: one connection for the schema, the pre-loading and the final checkpoint; -j connections for the data loading, the primary keys, the indexes and the vacuum; one for the foreign keys.
• run: one connection to read the id range, then one per client (-c) for the whole run. Make sure max_connections on the server allows a few more than the client count.

Initializing a benchmark

init creates the schema a spec declares and loads its data:

pgmeter [connection options] init --spec <DIR|FILE> [-s N] [-j N] [--no-fkey] [--post-create-table FILE] [--seed N] [-q]

init is destructive. It runs DROP TABLE IF EXISTS <table> CASCADE for every table in the spec’s schema.tables before creating it, then loads the data. Views, foreign keys from other tables and anything else that depends on those tables go with them. Point it at a database you can afford to wipe.

What happens, in order

1. Schema: the spec’s pre_ddls, then DROP TABLE and CREATE TABLE for every table, in one transaction, then the --post-create-table file if given.
2. Pre-loading: the spec’s pre_load_script, if any, on one connection. The bundled workloads use it for their small reference tables.
3. Loading: -j jobs, each loading its share of the 1..N unit ids with COPY, from either the spec’s row templates or its Lua loader. With --seed, each unit’s random generator is seeded from the seed and the unit id just before the unit loads, so the same seed and scale factor give the same rows whatever -j is; without it the data differs from one load to the next.
4. Primary keys, then foreign keys (unless --no-fkey), then indexes. They are created after the load, which is much faster than loading into indexed tables.
5. VACUUM FREEZE ANALYZE on every table and the spec’s post_ddls, dealt over the -j connections (with one job the post_ddls run after the vacuums; with more they can overlap).
6. A CHECKPOINT, so the run that follows does not pay for the load.

Each step prints its duration. The init lifecycle describes the same steps from the spec author’s side.

Choosing a scale factor

The scale factor sets the data size and the key space the workload spreads over, so it is part of what you compare. Two runs are comparable only at the same scale factor. The bundled workloads document their data volume per unit in Bundled workloads; as a rule of thumb, pick a scale factor whose data is larger than the server’s shared_buffers if you want to exercise storage, and smaller if you want a CPU-bound comparison.

Reloading between runs

A run modifies the data (PGM-B and PGM-C insert history and orders; PGM-C’s new orders grow the order tables). Running init again before each measurement puts every run on the same starting point. The first run after a load is usually a little slower than the next ones, as autovacuum and autoanalyze settle; treat it as a warm-up.

Running a benchmark

run drives the spec’s transaction mix against the data init loaded:

pgmeter [connection options] run --spec <DIR|FILE> [-c N] [-T N] [-r N] [-P N] [-j N] [--once] [--seed N] [--format FMT] [-q]

Use the same --spec for init and run: the spec is the single source of truth for the schema and the workload, and run reads its scaling query and transactions from it.

What a run does

1. Reads the largest scale-unit id from the data (the spec’s max_id_query), so the workload spreads over everything init loaded.
2. Starts the clients one by one over the ramp-up window. Each client runs transactions back to back, with no think time, picking the next one at random according to the spec’s weights.
3. At the end of the ramp-up, starts counting. Committed and failed transactions and their response times are collected for -T seconds.
4. Stops the clients, writes the results directory and prints the summary.

Every transaction is logged with its response time, ramp-up included, but the headline figures, the per-transaction statistics and the report’s histogram describe the steady-state window only. The time-series charts show the whole run with the ramp-up shaded.

The two engines

How the clients are driven depends on the spec, and the run header says which engine is in use.

• Simple-mode specs (every transaction is SQL with bindings, like PGM-B) run on the async engine: a pool of --threads worker threads drives one task per client, like pgbench’s -j. A worker that finds its client waiting for the server moves on to the next client, so the process wakes up far less often than one thread per client would, and the load generator takes less CPU away from a server on the same machine. The default of half the CPUs suits a machine shared with the server; a dedicated client machine can use more.
• Specs with a script transaction (Lua, like PGM-C and PGM-H) run one thread per client on a blocking connection, because the Lua database bridge is synchronous. --threads is ignored.

Running every transaction once

--once replaces the timed random stream with the query-stream model of the TPC-H and TPC-DS power and throughput tests: each client gets its own random permutation of the spec’s transactions, runs each of them exactly once, and stops. The run ends when the last client is done, so -T is not used and cannot be given. Transactions with a weight of 0 are skipped; the other weights are ignored.

• -c 1 --once is a power test: every query once, in one stream.
• -c 4 --once is a throughput test with four streams, each in a different order, concurrently.

The run is reported as a stream: where its time went, transaction by transaction, with a timeline per client (see Reading the results). duration_s and workload_secs in run.json are set to the window the run took, and "once": true. The ramp-up still staggers the clients’ start, and Ctrl-C still ends the run early.

Running the same transactions twice

Without --seed, every client draws its transaction order and the parameters of its queries from a generator seeded from the operating system, so no two runs execute the same statements. Averaged over a long run that does not matter. It does matter when two runs are compared query by query: a TPC-DS query whose parameter selects one of a few stores can take forty times longer in one run than in the other for no reason but the draw, and the difference looks like the server’s.

--seed N seeds each client’s generator from the seed and the client id. Two runs with the same seed, the same spec and the same number of clients then run the same transactions with the same parameters, client for client, on either engine and whatever --threads: a --once run repeats exactly, and a timed run repeats the same stream for as long as it lasts. The seed is echoed in the run header and recorded as seed in run.json.

pgmeter -d db_a run --spec pgm-ds --once --seed 42
pgmeter -d db_b run --spec pgm-ds --once --seed 42   # the same 99 queries, same parameters

The seed covers what the clients draw: the order of the transactions and every rand_* call of their bindings and scripts. It does not cover the data, which init --seed covers, nor the timing of the clients against each other.

Stopping early

Ctrl-C ends the run at the end of each client’s current transaction. The results and the report cover what ran so far and are marked as interrupted ("interrupted": true in run.json), and pgmeter exits normally. A second Ctrl-C aborts immediately with exit status 130.

Errors during the run

A transaction that fails, whether the server rejected a statement or a Lua script raised an error, is counted in the errors column and its message is written to error.log. The client rolls back and carries on with the next transaction; a failed transaction’s duration is not counted in the latency statistics. Some workloads fail on purpose: PGM-C’s New-Order rolls back about 1% of its orders, as TPC-C prescribes, and those show up as errors with the message New-Order rollback: unknown item.

A client that cannot connect at all stops the run with an error and exit status 1, after the collector has written the logs of what ran.

Output formats

--format table prints the headline and the two tables described in Reading the results. --format json prints one JSON document with the same figures and --format csv one CSV table, both for scripting; they imply --quiet so stdout holds only the result data.

Reading the results

A run reports in three places: the console, a results directory, and the HTML report inside that directory. All three describe the steady-state window, the -T seconds after the ramp-up; only the time-series charts and the raw logs include the ramp-up.

A run is reported in one of two layouts. A throughput run, the default, is judged by its rate over the measured window. A stream run is judged by where its time went, transaction by transaction: the spec picks it with metrics.report: stream, as PGM-H and PGM-DS do, and run --once always uses it.

The console

With --format table (the default), a throughput run ends with:

• The headline: overall throughput in the spec’s unit (TPM for PGM-C, TPS for PGM-B and PGM-IoT, queries per hour for PGM-H and PGM-DS), the number of committed transactions, the error rate, the average and the 99th-percentile response time. For example 24 040 TPM · 24 103 committed · 0.43% errors · 19.82 ms avg · 61.30 ms p99.
• The steady-state line of a timed run, described below. For example steady state: drifted · -13.3% second half · 17.9% variation per second.
• Results, one row per transaction: committed count, errors, error rate, throughput in the spec’s unit.
• Response times, one row per transaction: average, minimum, maximum, standard deviation, 95th and 99th percentiles, in milliseconds.

A stream run ends with:

• The headline: the elapsed time, the number of executions, the geometric mean of the transactions’ mean response times, the transaction that took the most time with its share, and the errors. For example 40.1 s elapsed · 44 executed · 976 ms geomean · slowest Q18-Large-Volume 25.46 s (32.5%) · 0 errors.
• Where the time went, one row per transaction, the most time-consuming first: executions, errors, total time and its share, and the average, minimum and maximum response times.

During a timed run, a progress bar on the terminal shows the elapsed time, the current throughput and the average latency; when stderr is not a terminal, --progress N prints the same figures as a progress: line every N seconds. The rate is the one of the last second, not an average since the start.

A --once run counts transactions instead: the bar shows how many of the planned ones are done (7/99), the elapsed time and what each client is running and for how long; every completed transaction is printed above it with the time of day, its name, its response time and its client, or its error. When stderr is not a terminal, the completion lines are printed as they come and --progress N adds a progress: 1234.5 s, 7/99 done, running Q16-Multi-Warehouse-Orders for 4m 12s line every N seconds, so a multi-hour stream run can be followed from its log. transaction.log is flushed at least every second, so tail -f on it works too.

--format json prints the same figures as one document (report, which is throughput or stream, duration_s, interrupted, throughput_unit, steady_state, stream and a transactions array with name, committed, errors, error_rate, throughput and a latency_ms object holding avg, min, max, p95, p99 and stddev); --format csv prints one row per transaction with the same columns. Both imply --quiet. steady_state holds verdict, seconds, mean, cv_pct, first_half, second_half and drift_pct, or is null when the run was not judged. stream holds elapsed_s, executed, errors and geomean_ms for a stream run, and is null otherwise; a stream run’s transactions also carry total_ms and share_pct.

Steady state

A throughput figure describes a run only if the rate held while it was measured. pgmeter checks this on the per-second commit counts of the measured window, without its first and last seconds, in which clients start and stop:

• Variation per second is the coefficient of variation of the per-second rate: its standard deviation as a percentage of its mean.
• Drift compares the mean rate of the window’s second half with its first half.

The verdict is steady under 10% variation and under 5% drift, unstable when only the variation is over, and drifted when the halves differ by 5% or more. A drifted run averages two regimes, such as a cache warming up or checkpoints starting, and a longer run may settle at a different rate. A window under 10 full seconds is not judged, and neither is a stream run.

The results directory

Each run creates a directory named after its start time in UTC, such as pgmeter-20260101T120000Z/, in the current directory:

Output files gives the exact format of each. The two logs are plain text so that a run can be re-analysed with other tools; rampup_end_ts in run.json is the transaction.log timestamp at which the steady-state window starts.

The HTML report

report.html opens offline in any browser: the chart library is inlined and nothing is fetched. It shows:

• An environment header: benchmark name and description, PostgreSQL version, key server settings (shared_buffers, work_mem, max_wal_size, max_connections, effective_cache_size) and the run parameters.
• The data set: tables and indexes with their sizes.

A throughput run’s report then shows:

• The headline figures: throughput, committed transactions, errors and error rate, and the average, 95th and 99th percentile response times.
• The steady-state verdict with its figures.
• The per-transaction results table.
• Charts: throughput over time with the mean of each half of the window dashed, latency over time as the mean and the 95th percentile of all transactions or of the one picked in the chart’s selector, the response-time distribution and errors over time; for specs with several transactions, also the response-time percentiles per transaction and the committed transaction mix. The ramp-up window is shaded.

A stream run’s report shows instead:

• The headline figures: elapsed time, executions, the geometric mean of the response times, the transaction that took the most time with its share, and errors.
• Where the time went: the total time of each transaction as a bar, the slowest first. The 16 slowest are drawn and the rest grouped, unless Show all is pressed; Log scale brings the short ones into view.
• The stream timeline: one lane per client and one bar per execution, where it ran, the slowest transaction highlighted. It is drawn up to 20,000 executions.
• The per-transaction table: rank, executions, errors, total time, share and cumulative share, and the average, minimum and maximum response times. Any column sorts, and a filter matches names.

View a sample throughput report: a PGM-C run with 10 warehouses and 8 clients for 60 s after a 10 s ramp-up, the first after the load, which the steady-state check flags as drifted.

View a sample stream report: PGM-DS at scale factor 1, two streams with --once.

Hover the charts; the pages have a light and a dark theme.

Comparing runs

The figures that matter for a comparison are the headline throughput and the per-transaction percentiles, at the same scale factor, client count and duration. Measuring well lists the habits that make such a comparison hold: reload before each run, discard the first run after a load, ramp up and measure long enough, repeat, and watch the server log. When the comparison goes down to single transactions whose cost depends on their parameters, give both runs the same --seed so they draw the same parameters (see Running a benchmark).

Measuring well

pgmeter reports what the server did during the measured window. Whether two such numbers can be compared depends on how the runs were set up. These are the rules that make a comparison hold, learned the hard way.

Keep everything the same but one thing

The scale factor, the client count, the ramp-up, the duration, the spec, the server settings, the machine the client runs on: change one of them between two runs, and the difference you see is that one. The run header printed at the start of each run and run.json in the results directory record the run parameters, so a results directory can always be matched to its setup.

Reload before each run

A run changes the data: PGM-B and PGM-C insert history and orders, PGM-IoT grows its measurement table, autovacuum rearranges pages. Run init again before each measured run so that every run starts from the same tables, the same statistics and a fresh checkpoint.

Discard the first run after a load

The first run after init is slower than the next ones by a few percent, sometimes more: the first autovacuum and autoanalyze pass lands in it, and the buffer cache and the file-system cache are cold. Either do a short throwaway run after each load, or run the same configuration twice and keep the second.

Ramp up, then measure long enough

Use a ramp-up (-r) long enough for every client to be connected and busy and for the caches to settle; 30 to 60 seconds is usual for an OLTP spec with a few dozen clients. The ramp-up is excluded from every figure. Then measure for at least a minute or two: short windows are noisy, and throughput that drifts during the window (a checkpoint, a vacuum, a growing table) shows in the time-series charts of the report, which are worth a look before trusting the headline.

Repeat

Run each configuration at least twice, three times when the difference is small, and look at the spread before reading anything into it. A difference smaller than the run-to-run spread is not a result.

Watch the server, not only the client

The server log tells you when a run was not measuring what you thought:

• Checkpoints every few seconds (“checkpoints are occurring too frequently”): max_wal_size is too small for the write rate. Both runs being compared suffer from it unevenly. Raise it, or at least keep it the same, and issue a CHECKPOINT before each run so none lands in the measured window by chance.
• Lock waits and deadlocks: a spec of your own with transactions that take their locks in different orders will show deadlocks in error.log and long tails in the latency table. PGM-C takes its locks in a fixed order; PGM-C-client updates stock rows in the order the lines were drawn and can deadlock under contention, like the application it models.
• Memory pressure and swapping on a shared machine show up as stalls of tens of seconds in the throughput chart, with nothing in the PostgreSQL log to explain them.

Where the client runs

The load generator uses CPU too. When it shares the machine with the server, every core-second it burns is taken from PostgreSQL, and the comparison between tools or between pgmeter versions becomes partly a comparison of load generators. For a server comparison, run pgmeter on another machine, or at least keep it on the same one for every run. On a shared machine, simple-mode specs cost less with the default --threads than with one worker per CPU, because the workers compete with the server.

Compare like with like

A result from a simple-mode spec that sends a transaction as one batch is not comparable with one that sends the statements one by one (see transactions), a PGM-C result is not comparable with a PGM-C-client result, and none of the bundled workloads’ figures are comparable with official TPC results. Comparing two servers or two configurations under the same spec is what the numbers are for.

Troubleshooting

Errors are printed on stderr as error: <message>, with the server’s own message and detail when there is one, and pgmeter exits with status 1. During a run, per-transaction failures go to error.log in the results directory instead and the run goes on.

Connecting

connection refused, no route to host, timed out: the host or port is wrong, or the server does not listen on the network. pgmeter reads PGHOST, PGPORT, PGUSER, PGDATABASE and PGPASSWORD like psql; if psql connects with the same settings, pgmeter will.

password authentication failed: the password comes from PGPASSWORD, or from the password file (PGPASSFILE, default ~/.pgpass) when it is not set; there is no prompt. A .pgpass entry must match the host, port, database and user exactly or with *, and the file must not be readable by others.

server does not support TLS with --sslmode require or verify-*: the server has no TLS. Use prefer or disable, or enable TLS on the server.

A TLS error naming the certificate (invalid peer certificate, an unknown issuer, a host name mismatch) with verify-ca or verify-full: the server’s certificate is not signed by a CA in the system store, or does not match the host name. A custom CA file is not supported yet; use require to encrypt without verifying, or install the CA in the system store. no CA certificate could be loaded from the system store means the store itself is empty or unreadable.

too many clients already: the server’s max_connections is below the client count plus the connections already open. Lower -c or raise the setting.

Initializing

max_id_query returned NULL: no data found, run `pgmeter init` first (or returned 0) on run: the tables exist but are empty, or run was given a different spec or database than init.

relation "..." does not exist on run: init was not run against this database, or the spec’s max_id_query or transactions name a table the schema does not create.

init reports a spec problem such as transaction 1 (read): mode `simple` requires a non-empty `sql` or transaction 3 (order): mode `script` requires `script` or `script_file` : the spec is invalid and nothing was touched. The message names the field; Writing benchmarks has the rules.

deadlock detected during init: two inits are running against the same tables at the same time, which happens when two benchmarks share table names in one database. Use one database per benchmark, or run them one at a time.

Loading is slow: raise -j up to the server’s core count, check that the loader uses COPY (template loaders and the copy_* API do) rather than INSERT in a loop, and look at max_wal_size and checkpoint_timeout on the server, which bound the write rate of a bulk load.

Running

Errors on New-Order with New-Order rollback: unknown item: expected. PGM-C and PGM-C-client roll back about 1% of their orders as TPC-C prescribes, and pgmeter counts every rollback as an error. Any other message in error.log is a real failure.

current transaction is aborted in error.log: a statement failed earlier in the same transaction and a later one ran on the aborted connection. pgmeter rolls back after a failed transaction before the next one; inside a script, a statement that fails raises a Lua error that ends the transaction, so this message means a script caught the error itself and went on. Look at the first error of that client instead.

Lua errors with a traceback: the message names the script and line. Only the first occurrence of an error carries the traceback; the same error from the same transaction is then logged with its message alone. The usual causes are a $n placeholder with no argument, a column name that does not match the query’s result (rows are keyed by column name), a 1-indexed table used as if it were 0-indexed, and scalefactor used in a transaction (it is 0 at run time; derive ranges from min_id, max_id and const.*).

Throughput collapses partway through a run: open report.html and look at the throughput chart for the moment it drops, then the server log at that time. The usual suspects are a checkpoint storm, a vacuum of a large table, the machine swapping, or a growing table that no longer fits the cache. Measuring well has the checklist.

The run header says “one thread per client” for a spec you expected to be fast: the spec has at least one script transaction, which selects the thread engine. --threads only applies when every transaction is in simple mode.

Ctrl-C does not stop it: one Ctrl-C ends the run gracefully at the end of each client’s current transaction and writes the results; a second one aborts immediately with status 130. A client stuck in a long statement finishes it first.

Results

error_rate is not errors / committed: it is errors over all transactions attempted, committed plus failed, in the measured window.

The CSVs have fewer rows than the duration: a per-second series has no row for a second in which the transaction did not run; pgmeter-stats-*.csv has one row per statistic, not per second.

An interrupted run’s throughput looks off: after a Ctrl-C the figures are computed over the window that actually ran, not over -T; run.json has "interrupted": true and workload_secs with the real length.

Bundled workloads

Seven specs are embedded in the binary: six workloads and a template. pgmeter spec list prints them; pgmeter spec new <name> -o <dir> writes an editable copy, which is what init and run take:

pgmeter spec list
pgmeter spec new pgm-c -o .          # creates ./pgm-c/spec.yaml and ./pgm-c/scripts/
pgmeter -d benchmark init --spec pgm-c -s 10 -j 4
pgmeter -d benchmark run  --spec pgm-c -c 8 -T 120 -r 30

spec new refuses to overwrite an existing directory unless --force is given. The copy is yours to edit: change the constants, the weights, the indexes or the SQL, and the same init and run apply.

Things to know before comparing numbers:

• The workloads are derived from the TPC specifications: they implement a representative subset of each, for relative comparison (two configurations, two servers, two table access methods under the same load). They are not audited, full-specification TPC benchmarks and their results must not be compared with or presented as official TPC results. See License and trademarks.
• PGM-C rolls back about 1% of its New-Orders on purpose, as TPC-C prescribes. They are reported as errors with the message New-Order rollback: unknown item; other messages in error.log are real failures.
• PGM-C and PGM-C-client load the same data and run the same mix. PGM-C puts the whole transaction in a server-side function, which is how HammerDB drives PostgreSQL; PGM-C-client sends every statement from the client, which is how most applications behave. Their throughput differs by the round trips, not by the work the server does.
• PGM-H and PGM-DS are query streams: run them with --once (each query once per client, a power test with one client and a throughput test with several) and read the per-query response times and the elapsed time, not the throughput headline.
• Customers, items and last names in the two PGM-C variants follow TPC-C’s non-uniform random distribution, so the hot spots are the ones the specification intends.

Each workload has a reference page with its data volumes, its transactions and the deviations from the TPC text: PGM-B, PGM-C, PGM-C-client, PGM-H, PGM-DS and PGM-IoT.

PGM-B

A banking debit/credit workload derived from TPC-B: one transaction that updates an account, a teller and a branch and records the operation in a history table. It is the simplest bundled workload and the one to use for raw OLTP throughput of small write transactions. Its metric is transactions per second.

pgmeter spec new pgm-b -o .
pgmeter -d benchmark init --spec pgm-b -s 10 -j 4
pgmeter -d benchmark run  --spec pgm-b -c 8 -T 60 -r 10

Scale and data

The scale unit is a branch. Per branch:

At scale factor 10 that is one million accounts. Primary keys on the three id columns, an index on branch_id in teller and account, and foreign keys from teller and account to branch (skipped with --no-fkey). The filler columns pad the rows to the widths the TPC-B text describes.

The transaction

The seven statements are sent one at a time, one round trip each, which is what pgbench’s built-in script does, so the two report comparable rates on the same server. The account, teller and branch are drawn uniformly over the whole loaded range, independently of each other.

Deviations from TPC-B

• No terminals and no think time: every client runs transactions back to back.
• The account and teller are not chosen within the transaction’s branch (TPC-B keeps 85% of accounts local to the branch); they are uniform over the whole dataset, like pgbench.
• The headline figure is committed transactions per second over the measured window, not the TPC-B tpsB metric with its residence-time constraints.

The spec is a short YAML file with no Lua, and a good first spec to read.

PGM-C

An order-entry workload derived from TPC-C: five transactions over warehouses, districts, customers, orders and stock, with the mix, the input distributions and the designed 1% rollback the TPC-C text prescribes. Each transaction runs as one PL/pgSQL function call, which is also how HammerDB’s TPROC-C drives PostgreSQL. The metric is committed transactions per minute.

PGM-C-client is the same workload driven statement by statement from the client.

pgmeter spec new pgm-c -o .
pgmeter -d benchmark init --spec pgm-c -s 10 -j 4
pgmeter -d benchmark run  --spec pgm-c -c 8 -T 120 -r 60

Scale and data

The scale unit is a warehouse. Per warehouse:

item has 100,000 rows shared by every warehouse, loaded once before the warehouses. At 10 warehouses the tables take about 1 GB and the indexes 200 MB more. Primary keys on every table but history, which TPC-C leaves without a key, plus the two TPC-C secondary indexes, customer (c_w_id, c_d_id, c_last, c_first, c_id) for the by-name lookups and orders (o_w_id, o_d_id, o_c_id, o_id) for Order-Status, and the ten foreign keys of TPC-C’s referential integrity. The foreign keys cost every New-Order about a dozen extra index probes; HammerDB’s TPROC-C schema declares none, so pgmeter init --no-fkey is the setting to compare with it. There are no other indexes: an extra one on stock quantities turned Stock-Level into a hash join over every low-stock item of the warehouse and was removed.

The transactions

Customers, items and last names are drawn with TPC-C’s non-uniform random function (NURand), so a few hot rows take most of the traffic as the specification intends. Each client picks the next transaction at random by weight, with no think time.

Each transaction is a SELECT pgmc_<name>(...) round trip; the five functions are created by the spec’s post_ddls at init time, in plain PL/pgSQL you can read and change in the spec. Delivery visits the districts in order and locks each district’s oldest order as it reads it, skipping one another delivery holds, and New-Order feeds its stock update from the lines sorted by key, so concurrent transactions take their row locks in the same order. No run has deadlocked so far; the sort is an ORDER BY in a CTE, which the planner has honoured in every plan seen, not a guarantee the server gives.

Errors you should expect

A correct run reports errors on New-Order only, about 1% of them, with the message New-Order rollback: unknown item in error.log. That is the rollback TPC-C requires, not a failure. Any other message is a real problem.

Deviations from TPC-C

• No terminals, no keying and think times: every client runs transactions back to back, so the throughput per warehouse is far above TPC-C’s 12.86 tpmC ceiling, and the warehouse count sets the data size rather than the load.
• Every transaction draws its warehouse and district at random. TPC-C attaches each terminal to one district and has it emulate a user there; the hot rows and the remote shares are the same, the per-terminal bookkeeping is not.
• Delivery runs inline as a normal transaction rather than through the deferred queue the specification describes, and the districts it skipped are counted, not reported.
• The reads a transaction makes only to display them are left out: New-Order does not fetch the item names nor compare the ORIGINAL markers of the item and stock data, Payment does not read the warehouse and district addresses, Order-Status does not read the customer’s name and balance. Stock-Level is the two statements the specification describes.
• The items of an order are distinct. TPC-C draws each line’s item on its own, so an order may name the same item twice; one set-based stock update per order needs them distinct.
• The constants of the non-uniform random function are fixed in pgmc_common.lua rather than drawn once per database, so every load is the same; the load-time and run-time constants for last names differ by 88, within the 65 to 119 the specification requires.
• The headline figure is committed transactions per minute over all five transactions. The results table has the committed New-Orders per minute for anyone who wants the TPC-C-style view; neither is tpmC.

PGM-C-client

The PGM-C order-entry workload, driven from the client instead of from PL/pgSQL functions: each transaction is a Lua script that sends every statement in its own round trip, reads the results it needs and decides what to do next, the way an application without stored procedures behaves.

pgmeter spec new pgm-c-client -o .
pgmeter -d benchmark init --spec pgm-c-client -s 10 -j 4
pgmeter -d benchmark run  --spec pgm-c-client -c 8 -T 120 -r 60

The schema, the data, the transaction mix, the input distributions, the transaction profiles and the designed 1% New-Order rollback are the same as PGM-C’s; the two specs share their loader and their Lua helpers. What differs is where the transaction logic runs:

On the same server PGM-C-client reports a much lower rate than PGM-C, and the ratio between the two is a direct measure of what the network and the protocol cost your application. Use PGM-C-client when that is what you want to see, or when the thing under test is a connection pooler, a proxy or a network path. Use PGM-C to measure the server itself.

The error you should expect is the same: about 1% of New-Orders end with New-Order rollback: unknown item, which is the specification’s rollback, counted as an error. The deviations from TPC-C listed on the PGM-C page apply here too.

Because the scripts read a value, compute on it and write it back, they lock what they read: New-Order takes its stock rows FOR UPDATE, in key order like the PL/pgSQL function, so concurrent orders neither lose a decrement nor deadlock, and Delivery picks each district’s oldest order with FOR UPDATE SKIP LOCKED, so two deliveries of the same warehouse take different orders.

PGM-H

A decision-support workload derived from TPC-H: the 8-table schema of parts, suppliers, customers, orders and line items, and 22 read-only analytical queries with random parameters. It is meant for a single client and read by the per-query response times; the headline figure is queries per hour.

pgmeter spec new pgm-h -o .
pgmeter -d benchmark init --spec pgm-h -s 1 -j 4
pgmeter -d benchmark run  --spec pgm-h -c 1 --once      # power test: each query once
pgmeter -d benchmark run  --spec pgm-h -c 4 --once      # throughput test: four streams

Scale and data

The scale unit is the TPC-H scale factor. Per unit:

region (5 rows) and nation (25) are loaded once. Scale factor 1 is about 2 GB with indexes and loads in a minute or two on a fast server; scale factor 10 is ten times that. At scale factor 1 there is a single unit, so the load itself runs on one connection; -j still parallelizes the keys, the indexes and the vacuum. Keys are integer and dates are date, as the TPC-H data types prescribe. Besides the primary keys and eight foreign keys, the spec creates the indexes an analytical PostgreSQL setup usually has: lineitem on ship date, order key, supplier key and (part key, supplier key); orders on customer key and order date; partsupp on supplier key; customer and supplier on nation key. There is no single-column index on lineitem’s part key: the (part key, supplier key) index starts with it and serves the same lookups at the same speed. The order-key index repeats the first column of the primary key, but it is narrower and the planner prefers it for the order-to-line-item joins (Q10 runs twice as fast with it). The indexes are listed in the spec and easy to remove if your comparison needs a narrower schema.

The queries

The 22 queries follow the TPC-H query definitions, with their parameters drawn at random within the ranges the specification gives (dates, regions, brands, discounts, quantities):

With --once each client runs the 22 queries once in its own random order, which is TPC-H’s stream model: one client is a power test, several clients are a throughput test. Without it the run is a continuous random stream with equal weights, for as long as -T says.

The queries live in scripts/pgmh_queries.lua as a dispatch table; each transaction’s script is one call into it, so changing a query or its parameter ranges is a Lua edit.

Reading a run

The spec reports a stream: the elapsed time, the geometric mean of the query times and where the time went, query by query, in the console and the HTML report, which adds a timeline per client. The elapsed time of a --once run is the stream’s result. The queries per hour headline is a plain rate over the queries executed; it is not the TPC-H QphH composite, which only an audited run may report.

Deviations from TPC-H

• Each client’s query order is a random permutation rather than the fixed permutations the specification assigns to its streams.
• No refresh functions: the data is read-only during a run.
• The data comes from the spec’s Lua loader, not from the official generator. The loader follows dbgen’s cardinalities, value ranges and distributions, including the ones the queries depend on: a third of the customers have no orders (Q13, Q22), the order status follows the lines shipped by mid-1995 (Q21), one part type in six is PROMO (Q14), and the comments carry the word pairs Q13 looks for and the Customer ... Complaints markers Q16 excludes, at dbgen’s rates. What differs: the text columns (addresses, comments, part names) are random words or characters rather than dbgen’s text grammar, the order keys are dense rather than dbgen’s sparse numbering (no query depends on it), and the random sequences are pgmeter’s, so the rows themselves are not dbgen’s rows and query results differ from a dbgen database of the same scale.
• Secondary indexes the specification leaves to the implementer, as listed above.

PGM-DS

A decision-support workload derived from TPC-DS: the full 24-table snowflake schema of a retailer selling through stores, catalogs and the web, its foreign-key graph, and the 99 analytical queries of the specification. Like PGM-H it is meant to be run as query streams with --once and read by the per-query response times; the headline figure is queries per hour.

pgmeter spec new pgm-ds -o .
pgmeter -d benchmark init --spec pgm-ds -s 1 -j 4
pgmeter -d benchmark run  --spec pgm-ds -c 1 --once      # power test: each query once
pgmeter -d benchmark run  --spec pgm-ds -c 4 --once      # throughput test: four streams

Scale and data

The scale unit is a scale factor. The dimensions that do not scale are loaded once; the rest grows with the scale factor.

Fixed dimensions: date_dim (every day from 1900 to 2100, about 73,000 rows), time_dim (86,400 seconds of a day), customer_demographics (1,920,800), household_demographics (7,200), catalog_page (11,718), promotion (300), reason (35), income_band (20), ship_mode (20), store (12), call_center (6), web_site (30), web_page (60), warehouse (5).

Scale factor 1 is about 2.4 GB with indexes and loads in a few minutes; start there. At scale factor 1 there is a single unit, so the fact tables load on one connection; -j still parallelizes the keys, the indexes and the vacuum. Primary keys on every table, the 106 foreign keys of the TPC-DS referential graph (skipped with --no-fkey), and secondary indexes chosen for the queries rather than one per foreign key: the sales tables’ sold-date keys, store_sales’ customer and store keys, store_returns’ customer and returned-date keys, inventory’s item key, customer’s address key, and the item attributes the queries filter on.

The queries

All 99 TPC-DS queries, numbered as in the specification, one transaction each with equal weights. The SQL follows the query templates, written for PostgreSQL, and the substitution parameters are drawn at random on every execution within the template ranges: years and dates within the loaded sales window, categories, colours, states, demographics and the other vocabularies the loader writes, and lists the specification takes from the data (zip codes, counties, cities, store states) sampled from the database once per client. The four multi-statement queries (14, 23, 24 and 39) run both of their parts.

They live in scripts/pgmds_queries.lua as a dispatch table, Q[1] to Q[99]; each transaction’s script is one call into it, so changing a query or its parameter ranges is a Lua edit. The query subjects are the specification’s: the description of each transaction in spec.yaml summarises it, and pgmeter spec list prints them.

Seven queries are restructured for PostgreSQL, with the same result rows. Q4, Q11 and Q74 share in the templates one year_total CTE over every channel and year that is then self-joined four or six times; PostgreSQL has no statistics for a materialized CTE and picked plans that ran for hours at scale factor 1, so the port aggregates each channel and year separately. Q1, Q6, Q30 and Q81 evaluate a correlated average subquery for every row in the templates, up to a minute each here; the port joins a per-group average instead. Everything else is the template’s shape.

With --once each client runs the 99 queries once in its own random order, which is TPC-DS’s stream model: one client is a power test, several clients are a throughput test. Without it the run is a continuous random stream with equal weights, for as long as -T says. A few queries (q95, q14, q10, q4, q39, q22, q23, q67, q74, q11) take far longer than the rest on PostgreSQL; they count once per stream like the others.

The spec reports a stream: the elapsed time, the geometric mean of the query times and where the time went, query by query, with a timeline per client in the HTML report. In one single-stream run at scale factor 1 on a test machine, q95 and q14 together took more than a quarter of the time.

Deviations from TPC-DS

• Each client’s query order is a random permutation rather than the fixed permutations the specification assigns to its streams, and there is no data maintenance run.
• The data is generated by the spec’s Lua loaders with TPC-DS cardinalities and the vocabularies the queries filter on, not by the official generator; text columns are synthetic and the distributions are uniform where the generator’s are skewed.
• Q4, Q11, Q74, Q1, Q6, Q30 and Q81 are restructured as described above.
• The headline figure is a plain rate over the queries executed, not the QphDS composite, which only an audited run may report.

PGM-IoT

A time-series workload derived from TPCx-IoT: substations with sensors that stream measurements into one wide table, queried while it grows. Seventy percent of the transactions are bulk inserts, the rest are the queries a monitoring system runs: recent aggregates per sensor, a sensor’s latest readings, unacknowledged alerts, threshold changes. The metric is committed transactions per second; each ingestion transaction inserts a batch of rows.

pgmeter spec new pgm-iot -o .
pgmeter -d benchmark init --spec pgm-iot -s 20 -j 4
pgmeter -d benchmark run  --spec pgm-iot -c 8 -T 300 -r 30

Scale and data

The scale unit is a substation. Per substation:

Primary keys, five foreign keys and five indexes: measurement on timestamp, on (sensor, timestamp) and on (substation, timestamp) with the sensor and value included, a partial index on unacknowledged alerts, and sensor on its substation. The measurement table grows by 100 rows per ingestion transaction, so a long run with many clients adds tens of millions of rows; size the scale factor and the duration together.

The transactions

The substation and sensor are drawn uniformly. Each transaction is a short Lua script in scripts/, one round trip per statement.

Deviations from TPCx-IoT

• TPCx-IoT defines a gateway workload run through a YCSB-like driver with prescribed record sizes and a fixed ingestion to query ratio measured over a warm-up and a measurement interval; PGM-IoT keeps the shape (bulk ingestion dominating, time-range and device queries alongside) on a relational schema with the queries a PostgreSQL deployment would run.
• The headline figure is committed transactions per second over the mix, not the IoTps metric.
• The data grows during the run, so throughput over a long run reflects the table’s growth as well as the server; keep the duration the same between the runs you compare.

Writing benchmarks

pgmeter runs a complete benchmark from a YAML specification file: the schema, the data loading and the transaction workload, with Lua for any logic that is too dynamic to express declaratively. Adding a benchmark never means modifying or recompiling pgmeter.

This section is the complete reference for benchmark authors. It covers every field of the spec file, the row-template mini-language, the binding mechanism, and, in the Lua API page, the full scripting API.

The bundled PGM-B, PGM-C, PGM-IoT, PGM-H and PGM-DS workloads, derived from the TPC-B, TPC-C, TPCx-IoT, TPC-H and TPC-DS specifications, are all expressed as specs of this kind; pgmeter spec new <name> -o . writes a copy to read alongside these pages. (They are not audited, full-specification TPC benchmarks; see the trademark notice.)

Anatomy of a spec file

A spec is a single YAML document with these top-level keys:

name:               # string  (required) - benchmark name shown in output
description:        # string  (optional)

scaling:            # required - scale factor unit, max-id query, constants
metrics:            # optional - throughput unit and report layout
schema:             # required - tables, foreign_keys, indexes, pre/post DDL

# --- choose ONE data-loading style ---
load:               # list    - declarative, template-based COPY loaders
pre_load_script:    # string  - Lua run once before per-id loading
load_script:        # string  - Lua run per parallel job for loading

transactions:       # required - the workload (one or more transactions)

scripts:            # optional - shared Lua modules preloaded into every VM

The pages that follow describe each of them, except metrics, described below.

metrics: how a run is reported

metrics:
  throughput_unit: "queries/hr"   # label of the throughput figures (default TPM)
  per: hour                       # second, minute (default) or hour
  report: stream                  # throughput (default) or stream

throughput_unit is only a label; per sets the time base the committed count is turned into a rate with. report picks the layout of the results:

• throughput judges the run by its rate over the measured window: the headline throughput and percentiles, and whether the rate held steady. This suits OLTP workloads such as PGM-B and PGM-C.
• stream judges the run by where its time went, transaction by transaction: the elapsed time, the geometric mean of the response times, each transaction’s share of the time and a timeline per client. This suits analytical workloads such as PGM-H and PGM-DS.

run --once always reports a stream. Reading the results shows both layouts.

A minimal spec

Create my-bench.yaml:

name: hello-bench
description: Smallest possible benchmark

scaling:
  unit: account
  max_id_query: "SELECT MAX(id) FROM account"

schema:
  tables:
    - name: account
      columns:
        - { name: id,      type: "INTEGER NOT NULL" }
        - { name: balance, type: "INTEGER" }
      primary_key: [id]

load:
  - table: account
    copy_columns: [id, balance]
    rows_per_id: 1
    row: "{{id}}\t0\n"

transactions:
  - id: 1
    name: read-account
    weight: 100
    mode: simple
    bindings:
      acct: "rand_int(min_id, max_id)"
    sql: "SELECT balance FROM account WHERE id = {{acct}};"

Initialize and run it:

# load 1000 accounts (scale factor 1000)
pgmeter -H localhost -d benchmark init --spec my-bench.yaml -s 1000 -j 4

# run 8 clients for 60s with a 10s ramp-up
pgmeter -H localhost -d benchmark run --spec my-bench.yaml -c 8 -T 60 -r 10

That is the whole loop: define the schema, how to fill it and what to run, then init and run. Everything else in this section is detail on those three parts.

scaling: scale factor and context

scaling:
  unit: branch
  max_id_query: "SELECT MAX(branch_id) AS max_id FROM branch"
  constants:
    tellers_per_branch: 10
    accounts_per_branch: 100000

The scale-factor model

pgmeter loads -s N units, numbered 1..N. The unit id is the partitioning key for everything:

• During loading, each unit id (id) is processed once, and you generate the rows that belong to it.
• During a run, every transaction picks a random unit id in [min_id, max_id] and operates on that unit’s data.

This is what makes data generation and the workload scale together: at scale factor 1000 you get 1000 branches, and the workload spreads across branches 1..1000.

Context variables

The following are injected into the Lua environment used for bindings, row templates, and scripts:

⚠️ scalefactor is 0 while a benchmark runs. It only carries the real value during init/loading. In transaction logic, derive ranges from min_id, max_id, and const.* instead.

constants must be scalars (numbers, strings, booleans). Nested lists/maps are ignored when injected into Lua.

schema: tables, keys, indexes

schema:
  pre_ddls:            # optional - SQL run before any CREATE TABLE
    - "CREATE SEQUENCE measurement_id_seq"
  tables:
    - name: account
      columns:
        - { name: id,      type: "INTEGER NOT NULL" }
        - { name: balance, type: "INTEGER" }
        - { name: filler,  type: "CHAR(84)" }
      primary_key: [id]
      access_method: heap          # optional
      storage:                     # optional - WITH (...) options
        fillfactor: 90
        autovacuum_enabled: false
  foreign_keys:
    - name: fk_x
      table: child
      columns: [parent_id]
      references: { table: parent, columns: [id] }
  indexes:
    - { name: idx_balance, table: account, columns: [balance] }
  post_ddls:           # optional - SQL appended to the vacuum step
    - "SELECT setval('measurement_id_seq', (SELECT MAX(id) FROM measurement))"

tables

Each table is generated as DROP TABLE IF EXISTS <name> CASCADE followed by CREATE TABLE, so re-running init is idempotent.

Column type is passed through verbatim. Whatever you write lands in the CREATE TABLE between the column name and the next comma. This is the escape hatch for defaults, generated columns, check constraints on a column, etc.

foreign_keys

Emitted as ALTER TABLE <table> ADD CONSTRAINT <name> FOREIGN KEY (<columns>) REFERENCES <references.table> (<references.columns>), created after data is loaded. Skipped entirely when init is run with --no-fkey.

indexes

indexes:
  - { name: i_ts,     table: measurement, columns: [ts] }
  - { name: i_uniq,   table: account,     columns: [email], unique: true }
  - { name: i_brin,   table: measurement, columns: [ts], type: brin }
  - { name: i_cover,  table: measurement, columns: [ss_id, ts], include: [sn_id, value] }
  - { name: i_unack,  table: alert,       columns: [ss_id], where: "NOT acknowledged" }

The generated statement is:

CREATE [UNIQUE] INDEX <name> ON <table> [USING <type>] (<columns>) [INCLUDE (<include>)] [WHERE <where>]

For a benchmarking tool, index shape directly affects the numbers you measure. type, include, and where exist so the YAML can reproduce the exact index definitions a hand-tuned schema would use.

pre_ddls / post_ddls

Free-form SQL statements. pre_ddls run before table creation (sequences, CREATE EXTENSION, etc.). post_ddls run at the end of the vacuum step (e.g. setval to align a sequence with loaded data).

Data loading

There are two mutually exclusive styles. Pick the one that fits:

If load_script is present it takes precedence and load: is ignored.

Template-based loading (load:)

Each entry is a loader for one table, executed once per unit id via PostgreSQL COPY.

load:
  - table: account
    method: copy                 # only "copy" is supported (default)
    copy_columns: [id, branch_id, balance, filler]
    rows_per_id: "{{const.accounts_per_branch}}"
    batch_size: 10000            # rows per COPY chunk (default 10000)
    row: "{{(id - 1) * const.accounts_per_branch + i}}\t{{id}}\t0\t\n"

The row-template mini-language

A row string is literal text with {{ lua_expression }} holes. The engine compiles it once into a Lua function and calls it in batches.

• Output must be tab-separated (\t) values in copy_columns order, ending with a newline (\n): this is PostgreSQL’s default COPY text format.
• {{expr}} is replaced by tostring(expr).
• Two loop variables are available:
  • id: the current unit id.
  • i: the row counter, running 1..rows_per_id.
• rand_int, rand_float, min_id, max_id, scalefactor, and const.* are all in scope. Database functions (query, exec, …) are not: template loading is pure generation.

Examples:

# one row per unit: "<id>\t0\n"
row: "{{id}}\t0\n"

# N rows per unit with a globally unique id, random price and stock
row: "{{(id - 1) * 100 + i}}\t{{id}}\tProduct-{{i}}\t{{rand_float(1.0, 999.99)}}\t{{rand_int(0, 10000)}}\n"

NULLs and empty values in templates

The default COPY text format treats an empty field as an empty string and the two-character sequence \N as SQL NULL. CHAR(n) columns left empty (as in the PGM-B filler columns) are stored blank-padded. If you need a real NULL, emit \N, or switch to a Lua loader where you control the COPY options.

Lua-based loading (load_script / pre_load_script)

For anything beyond a per-row formula, use Lua. Both fields accept either an inline script (a multi-line YAML block) or a path to a .lua file (resolved relative to the spec file). The rule: if the value contains a newline it is treated as inline code, otherwise as a file path.

pre_load_script: lua/setup_reference_tables.lua   # runs ONCE, before loading
load_script: lua/load_per_unit.lua                # runs per parallel job

• pre_load_script runs a single time on one connection before any per-unit loading. Use it to populate small shared/reference tables that are not partitioned by unit id (e.g. the PGM-C item table, PGM-H region/nation). It has the full DB API but ids is empty.

• load_script runs once per unit id, on the job that unit is assigned to (the engine round-robins the 1..scalefactor ids across -j jobs). The global ids is an array holding that unit id, so a script loops over ids and loads each unit’s data, typically via the copy_* streaming API:

for _, id in ipairs(ids) do
    copy_start("COPY account FROM stdin")
    for i = 1, const.accounts_per_unit do
        copy_write(string.format("%d\t%d\t0\n", (id - 1) * const.accounts_per_unit + i, id))
    end
    copy_end()
end

Both scripts have scalefactor, const.*, and the full Lua API (random, SQL, and COPY helpers). min_id/max_id are not injected into loaders: iterate ids instead.

Parallelism caveat: because each call sees one unit and the jobs run concurrently, a load_script must compute any global offsets from the id value itself (as above with (id - 1) * N + i), never from a running counter. If a table is not partitioned by unit (a single shared table), load it from pre_load_script, or guard it with if id == 1 then ... end inside load_script so only one job creates it. Loaders that draw every value with the rand_* functions are reproducible with init --seed, which reseeds the generator per unit; likewise the transactions’ draws are reproducible with run --seed, which seeds the generator per client.

transactions: the workload

Each transaction is one weighted entry in the workload mix. On every step a client picks a transaction at random, weighted by weight.

transactions:
  - id: 1
    name: New-Order
    description: Create a customer order
    weight: 45
    mode: script          # "simple" | "script"
    # for simple mode:
    bindings: { ... }
    sql: "..."
    # for script mode (one of):
    script: "Q[1]()"          # inline Lua
    script_file: lua/new_order.lua   # external Lua file

Only simple and script modes exist. (Some sample comments mention an extended mode: it is not implemented and will error.)

simple mode

Best for fixed SQL with a few random parameters. sql takes two shapes:

• One string: the whole block is sent to the server in one round trip (batch_execute). You may include several statements separated by ;; wrap them in BEGIN; ... COMMIT; if they must be atomic, otherwise each statement autocommits.
• A list of statements: each element is sent in its own round trip, in order, inside the same measured transaction. This is how pgbench runs its scripts, so use the list form when your numbers must be comparable with pgbench’s or when the client/server round trips are part of what you measure. On a fast network a 7-statement transaction runs about twice as fast in the batched form, which is why the two are not comparable.

In both shapes no result rows are returned to the client (this mode is fire-and-measure); use script mode when a later statement depends on an earlier result.

Bindings are evaluated just before the SQL is rendered. Each binding value is a Lua expression evaluated against the context (min_id, max_id, scalefactor, const.*, plus rand_int/rand_float). The result substitutes every {{name}} occurrence in sql.

- id: 1
  name: PGM-B
  weight: 100
  mode: simple
  bindings:
    branch_id:  "rand_int(min_id, max_id)"
    account_id: "rand_int((min_id - 1) * const.accounts_per_branch + 1, max_id * const.accounts_per_branch)"
    delta:      "rand_int(-5000, 5000)"
  sql: |
    BEGIN;
    UPDATE account SET account_balance = account_balance + {{delta}} WHERE account_id = {{account_id}};
    UPDATE branch  SET branch_balance  = branch_balance  + {{delta}} WHERE branch_id  = {{branch_id}};
    INSERT INTO history (account_id, branch_id, delta, ts)
      VALUES ({{account_id}}, {{branch_id}}, {{delta}}, CURRENT_TIMESTAMP);
    COMMIT;

Binding values are substituted as literals: integers and floats verbatim, strings single-quoted and escaped, booleans as true/false, and nil as NULL.

Bindings evaluate in declaration order. Each binding becomes a local in a single generated Lua function, in the order written, so a later binding may reference an earlier one (e.g. account_id can use branch_id). For anything beyond a one-line derivation (reads that feed writes, branching, loops), use script mode instead. See Limitations.

script mode

Best for multi-statement transactions with control flow, reads that feed writes, conditional rollback, etc. The Lua source (inline script or script_file) becomes the transaction body and runs with the full Lua API.

You manage the transaction boundary explicitly with begin() / commit() / rollback(), and use query/query_one/exec for SQL.

-- new_order.lua
local w_id = rand_int(min_id, max_id)
local d_id = rand_int(1, 10)

begin()

local d = query(
  "UPDATE district SET d_next_o_id = d_next_o_id + 1 WHERE d_w_id = $1 AND d_id = $2 RETURNING d_next_o_id",
  w_id, d_id)
local o_id = d[1]["d_next_o_id"] - 1

exec("INSERT INTO orders (o_id, o_d_id, o_w_id, o_entry_d) VALUES ($1, $2, $3, NOW())",
     o_id, d_id, w_id)

commit()

A common pattern is a dispatch table loaded via scripts.modules, with each transaction’s script just calling into it:

scripts:
  modules: [lua/queries.lua]     # defines a global table Q with Q[1]..Q[22]
transactions:
  - { id: 1, name: Q1, weight: 8, mode: script, script: "Q[1]()" }
  - { id: 2, name: Q2, weight: 4, mode: script, script: "Q[2]()" }

scripts.modules: shared Lua code

scripts:
  modules:
    - lua/helpers.lua
    - lua/queries.lua

Each listed file is loaded once per worker thread, at global scope, before any transaction runs, and also into the VM of pre_load_script and load_script, so loaders and transactions can share helpers. Use modules for:

• Helper functions shared across transactions.
• Dispatch tables (e.g. Q[1]..Q[22] for PGM-H).
• Constant lookup tables (syllable lists, type names, …).

Anything a module defines as a global is visible to every script-mode transaction on that thread.

The init lifecycle

init performs these steps in order (you can see each in the console output):

1. Database DDLs: pre_ddls, then DROP/CREATE TABLE for every table (with access_method / storage). Run in one transaction.
2. Pre-loading: pre_load_script, if present (single connection).
3. Data loading: -j parallel jobs. The 1..scalefactor unit ids are round-robin distributed across jobs; each job runs the template loaders or load_script for its ids.
4. Primary keys: ALTER TABLE ... ADD PRIMARY KEY.
5. Foreign keys: unless --no-fkey.
6. Indexes: all indexes.
7. Vacuum: VACUUM FREEZE ANALYZE per table, then post_ddls.
8. Checkpoint: forces a checkpoint so the run starts from a clean state.

Keys and indexes are created after loading for speed. This ordering is why your row templates / load_script only need to produce raw column data: the constraints are applied afterwards.

Worked example: a benchmark from scratch

Let’s build a small link-shortener benchmark from nothing. Scale unit = a “workspace”; each workspace owns 1000 links. The workload is 80% redirects (read a link, bump its hit counter) and 20% link creation.

Step 1: schema and scaling

name: linkshortener
description: Read-heavy URL shortener workload

scaling:
  unit: workspace
  max_id_query: "SELECT MAX(id) FROM workspace"
  constants:
    links_per_workspace: 1000

schema:
  tables:
    - name: workspace
      columns:
        - { name: id,   type: "INTEGER NOT NULL" }
        - { name: name, type: "VARCHAR(64)" }
      primary_key: [id]

    - name: link
      columns:
        - { name: id,           type: "BIGINT NOT NULL" }
        - { name: workspace_id, type: "INTEGER NOT NULL" }
        - { name: slug,         type: "VARCHAR(12) NOT NULL" }
        - { name: target,       type: "TEXT NOT NULL" }
        - { name: hits,         type: "BIGINT DEFAULT 0" }
        - { name: created_at,   type: "TIMESTAMP DEFAULT NOW()" }
      primary_key: [id]

  foreign_keys:
    - name: fk_link_workspace
      table: link
      columns: [workspace_id]
      references: { table: workspace, columns: [id] }

  indexes:
    - { name: i_link_ws,   table: link, columns: [workspace_id] }
    - { name: i_link_slug, table: link, columns: [slug], unique: true }

Step 2: load data (template style)

The slug must be unique, so we derive it from the globally-unique link id ((id-1)*1000 + i), and give each link a random target:

load:
  - table: workspace
    copy_columns: [id, name]
    rows_per_id: 1
    row: "{{id}}\tWorkspace-{{id}}\n"

  - table: link
    copy_columns: [id, workspace_id, slug, target, hits]
    rows_per_id: "{{const.links_per_workspace}}"
    batch_size: 5000
    # globally unique id and slug; random hit count
    row: "{{(id - 1) * const.links_per_workspace + i}}\t{{id}}\ts{{(id - 1) * const.links_per_workspace + i}}\thttps://example.com/{{rand_string(8, 16)}}\t{{rand_int(0, 9999)}}\n"

Step 3: the workload

A read transaction in simple mode, and a write in script mode (it needs the next id and a generated slug, which is easier in Lua):

transactions:
  - id: 1
    name: redirect
    weight: 80
    mode: simple
    bindings:
      ws:   "rand_int(min_id, max_id)"
      link: "rand_int((ws - 1) * const.links_per_workspace + 1, ws * const.links_per_workspace)"
    sql: |
      UPDATE link SET hits = hits + 1 WHERE id = {{link}} AND workspace_id = {{ws}};

  - id: 2
    name: create-link
    weight: 20
    mode: script
    script_file: lua/create_link.lua

Note the redirect binding link depends on ws. This works because bindings evaluate in declaration order (see Limitations), so ws is already in scope when link is computed. Keep each binding to a single expression; for anything with control flow, use script mode.

lua/create_link.lua:

local ws = rand_int(min_id, max_id)
local slug = rand_string(6, 10)
local target = "https://example.com/" .. rand_string(8, 20)

begin()
-- next id = current max + 1 (simple approach for the example)
local next_id = (query_one("SELECT COALESCE(MAX(id), 0) FROM link") or 0) + 1
exec("INSERT INTO link (id, workspace_id, slug, target) VALUES ($1, $2, $3, $4)",
     next_id, ws, slug, target)
commit()

Step 4: run it

pgmeter -d benchmark init --spec linkshortener.yaml -s 500 -j 4
pgmeter -d benchmark run  --spec linkshortener.yaml -c 16 -T 120 -r 15

That’s a complete, original benchmark: no Rust, no recompile.

Performance tips

• Load with COPY, not INSERT. Template loaders and the copy_* API both use COPY. Avoid exec("INSERT ...") in a tight loop for bulk loads.
• Batch your row generation. Template loaders already concatenate a whole batch into one string per COPY round-trip. In load_script, build a batch with string.format and a buffer, then copy_write once, rather than one copy_write per row where practical.
• Prefer simple mode for fixed SQL. It is a single round-trip with precompiled bindings and is the fastest path for pgbench-style transactions.
• Use script mode when logic demands it: reads feeding writes, conditional rollback, loops. The first call on each thread compiles the script (not counted in the measured time); later calls are direct function calls.
• Cache per-thread work in globals. Since the VM persists, you can memoize a formatted query string or a lookup table in a global on first use.
• rand_strings for bulk strings. When generating many random strings during loading, the batched rand_strings(count, min, max) reduces Lua↔Rust crossings.

Limitations and gotchas

• Modes: only simple and script exist. Any other mode value is rejected when the spec is loaded.
• TLS: --sslmode uses rustls with the system CA store; a custom CA file (libpq’s sslrootcert) and client certificates are not supported yet.
• scalefactor is 0 at run time. Only loaders see the real value. Derive ranges in transactions from min_id/max_id/const.*.
• simple-mode bindings evaluate in declaration order. Bindings compile into a single Lua function in the order written, so a later binding may reference an earlier one. Each binding is still a single expression, though: for multi-step logic, reads that feed writes, or control flow, use script mode.
• Bindings have no DB access. query/exec are only available in script mode and in loaders, not in simple-mode bindings or row templates.
• Loaders don’t get min_id/max_id. Iterate the ids array instead.
• Tables are 1-indexed (Luau): rows[1] is the first row, Q[1] the first query.
• Shared (non-partitioned) tables must be loaded from pre_load_script or guarded by if id == 1 in load_script, otherwise every parallel job will try to load them.
• Parameters are interpolated as literals (simple protocol), not bound server-side. String values are escaped, but treat $n as value placeholders only: don’t build identifiers from untrusted input.

Command line

pgmeter [connection options] <command> [options]

Commands: init, run, spec list, spec new. pgmeter --help and pgmeter <command> --help print the same options with their defaults.

Connection options

Global, placed before the command. Each has the environment variable the PostgreSQL tools use; see Connecting to PostgreSQL.

The password comes from PGPASSWORD, or from the password file (PGPASSFILE, default ~/.pgpass) when it is not set.

init

pgmeter [connection options] init --spec <DIR|FILE> [options]

Drops and recreates the tables the spec declares, loads the data, then adds keys and indexes, vacuums and checkpoints. See Initializing a benchmark.

run

pgmeter [connection options] run --spec <DIR|FILE> [options]

Drives the spec’s transaction mix. See Running a benchmark.

spec list

pgmeter spec list

Prints the bundled workloads with their title and description.

spec new

pgmeter spec new <BENCHMARK> [-o <DIR>] [--force]

Writes an editable copy of a bundled workload: its spec.yaml and, when it has any, its scripts/ folder.

Exit status

Errors are printed on stderr as error: <message>, with the server’s detail when there is one.

Lua API

Execution model

• A spec with at least one script transaction runs one thread per client, because the database bridge below is synchronous. A spec made of simple transactions only runs on the async engine instead: a pool of worker threads (--threads) drives one task per client, which costs much less CPU per transaction. The run header names the engine in use.
• Each benchmark client thread owns one Lua VM, created lazily on its first script-mode transaction. The VM lives for the whole run.
• On VM creation: context globals are injected, then scripts.modules are executed in order, then every transaction script is compiled into a named function. Subsequent executions just call the precompiled function: there is no re-parsing per transaction.
• Because the VM persists, globals set by one transaction survive into the next on the same thread. This is useful for caching (compile a prepared string once), but means you should not assume a clean slate each call.
• Data-loading scripts (pre_load_script / load_script) get a fresh VM per invocation.

Globals available to scripts

Plus the standard Luau library (string, table, math, tostring, ipairs, …). The engine uses Luau (via the mlua crate), so Luau semantics apply (notably: arrays/tables are 1-indexed).

Random data functions

SQL execution

All SQL uses PostgreSQL’s simple query protocol. Parameters are passed as extra arguments after the SQL string and interpolated into $1, $2, … as literals (not server-side bind parameters):

• integers / floats → written as-is
• strings → single-quoted, with ' escaped to ''
• booleans → true / false
• nil → NULL

Result values come back through the simple protocol as text, without type information, and are converted by shape, matching what PostgreSQL prints:

Because the shape is all there is, a text column whose value happens to be t, f or a bare number is converted too. If that matters, keep the value unambiguous on the SQL side (for example prefix it, '#' || col) or do the comparison in SQL. So:

local rows = query("SELECT c_discount, c_last FROM customer WHERE c_id = $1", cid)
local discount = rows[1]["c_discount"]   -- number
local last     = rows[1]["c_last"]       -- string

local total = query_one("SELECT count(*) FROM orders WHERE o_c_id = $1", cid)  -- number or nil

$n placeholders are substituted by simple text replacement (highest index first, so $10 is handled before $1). Use them only as value placeholders.

Transaction control

A script transaction that should be atomic must call begin() at the start and commit() at the end. To abort, call rollback().

Error handling

Raising a Lua error (via error("...")) aborts the current transaction; pgmeter records it as an error in the results (the per-transaction error count and rate). The standard pattern for an intentional abort is to roll back first, then error:

local item = query("SELECT i_price FROM item WHERE i_id = $1", iid)
if #item == 0 then
    rollback()
    error("invalid item")   -- counted as an error / rollback in the report
end

This is how PGM-C-client mirrors the ~1% New-Order rollback rate mandated by the TPC-C specification (PGM-C raises the same error from its PL/pgSQL function). Because pgmeter counts every Lua error as a transaction error, a correct run shows a New-Order error rate near 1% (about 0.45% of all transactions), each logged as New-Order rollback: unknown item in error.log; other messages there are real failures. Intentional rollbacks are not reported separately from errors yet.

Bulk COPY streaming (loaders)

For high-throughput loading inside load_script / pre_load_script:

Only one COPY stream is active at a time per thread. Any stream left open when a loader finishes is flushed automatically, but call copy_end() explicitly.

copy_start("COPY measurement FROM stdin")
for k = 1, n do
  copy_write(string.format("%d\t%d\t%.4f\n", id_base + k, sensor, rand_float(50, 150)))
end
copy_end()

Output files

Every run writes a directory named after its start time in UTC, pgmeter-<YYYYMMDD>T<HHMMSS>Z/, in the current directory. Its files are plain text so that a run can be re-analysed with other tools. Timestamps are Unix seconds; durations are milliseconds; <name> is the transaction’s name from the spec.

run.json

The run parameters and timing, one JSON object:

transaction.log

One line per committed transaction, ramp-up included, in the order the collector received them, flushed at least every second while the run goes:

<timestamp> <active_clients> <transaction_id> <response_time_ms> <client_id> <start_ms>
1793005977 1 3 4.893 1 10412

active_clients is the number of distinct clients seen so far, which grows during the ramp-up. transaction_id is the id from the spec. client_id is the client that ran the transaction, from 1 to -c. start_ms is when the transaction started, in milliseconds since the run’s collector started, so start_ms plus response_time_ms is when it ended. Lines with a timestamp before rampup_end_ts belong to the ramp-up.

error.log

One line per failed transaction:

<timestamp> <active_clients> <transaction_id> <message>

The message is the server error or the Lua error, with its detail and, for Lua, its traceback. Line breaks, tabs and backslashes inside it are escaped as \n, \r, \t and \\, so one line is always one error.

A script that fails the same way many times, as a workload with designed rollbacks does, gets its traceback once: the first error of a transaction with a given message is logged in full, the next ones with the message alone. To find where a short line comes from, look for the first line of that transaction with the same message. pgmeter remembers 4096 distinct messages per run; past that, new ones are logged in full every time.

pgmeter-throughput-<name>.csv and pgmeter-throughput-all.csv

Throughput per second over the whole run, one file per transaction and one for all of them together:

time_s,throughput
0,48480
1,105120

time_s counts from the first second of the run in the all file and from the first second in which that transaction committed in the per-transaction files; throughput is the second’s committed count expressed in the spec’s unit (the count times throughput_factor), so a TPM series reads as the rate the second would sustain over a minute.

pgmeter-response-time-<name>.csv

The average response time of the transactions committed in each second of the whole run:

time_s,response_time_ms
0,1.8445
1,1.8127

Seconds in which the transaction did not run are absent.

pgmeter-stats-<name>.csv

The statistical summary of the steady-state window, the same figures as the results tables:

describe,response_time_ms
count,19339
mean,7.1517
std,3.1767
min,2.094
95%,12.7685
99%,17.1971
max,46.398

count is the number of committed transactions in the window; the other rows are in milliseconds.

report.html

The report, self-contained: the chart library is inlined and the page opens offline. It reads nothing from the other files; the figures are embedded at generation time.

Cheat sheets

YAML keys

name                      benchmark name (required)
description               free text
scaling:
  unit                    label only (not used by engine)
  max_id_query            SQL -> single int (max unit id)
  constants: { k: v }     scalars -> const.k
schema:
  pre_ddls: [sql]         run before CREATE TABLE
  tables:
    - name
      columns: [{name, type}]
      primary_key: [col]
      access_method       USING <m>
      storage: { k: v }   WITH (k=v)
  foreign_keys:
    - { name, table, columns, references: { table, columns } }
  indexes:
    - { name, table, columns, unique, type, include, where }
  post_ddls: [sql]        run after vacuum
load:                     (template loaders; ignored if load_script set)
  - { table, method, copy_columns, rows_per_id, batch_size, row }
pre_load_script           Lua (inline or path), runs once
load_script               Lua (inline or path), runs per job over `ids`
transactions:
  - { id, name, weight, description, mode, bindings, sql (string | [statements]), script, script_file }
scripts:
  modules: [path]         Lua preloaded into every VM

Template variables (in row: and rows_per_id:)

id            current unit id
i             row counter 1..rows_per_id   (row: only)
min_id max_id scalefactor const.*
rand_int(min,max)  rand_float(min,max)

Lua functions

-- random
rand_int(min, max)                 -> int   [min, max]
rand_float(min, max)               -> num   [min, max)
rand_string(min_len, max_len)      -> string
rand_strings(count, min_len, max_len) -> array

-- transactions
begin()   commit()   rollback()

-- SQL (simple protocol; params -> $1,$2,... as literals)
exec(sql, ...params)               -> nil
query(sql, ...params)              -> array of {colname = value}
query_one(sql, ...params)          -> scalar or nil

-- bulk loading
copy_start(sql)   copy_write(data)   copy_end()

-- abort a transaction (counts as an error)
error("message")

Lua globals

transactions: min_id, max_id, const.*        (scalefactor == 0)
loaders:      ids[], scalefactor, const.*

Changelog

All notable changes to pgmeter are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning. Until 1.0.0, minor versions may change the YAML spec schema, the Lua API, the CSV layout or the CLI; such changes are listed under “Changed”.

Unreleased

First public release.

Added

• pgmeter init creates the schema, loads data in parallel (--jobs, --scalefactor), then adds primary keys, foreign keys (--no-fkey to skip), indexes, vacuums and checkpoints. --post-create-table runs an SQL script after table creation. --seed makes a load reproducible: the same seed and scale factor load the same rows, whatever --jobs.
• pgmeter run drives the workload with one connection per client (--client, --time, --rampup), prints live progress (--progress) and a results summary as a table, JSON or CSV (--format, --quiet). Specs made of simple transactions run on an async engine: a pool of worker threads (--threads, default half the CPUs) drives one task per client, which wakes the process up far less often than one thread per client did (PGM-B went from 87% to 94% of pgbench’s rate on the same box); specs with a script transaction run one thread per client. run --once replaces the timed random stream with the TPC-H/DS stream model: each client runs every transaction once in its own random order, then stops, and the run shows its progress as it goes: transactions done out of the planned total and the ones in flight on the bar or in the progress: lines, and one line per transaction as it completes. transaction.log is flushed at least every second, so it can be tailed. PGM-H’s queries have equal weights, as the specification’s streams give them. run --seed makes the clients’ draws reproducible: the same seed, spec and client count run the same transactions with the same parameters, so two servers or configurations can be compared query by query.
• Benchmarks are YAML specifications plus Lua (Luau) scripts. Transactions run in simple mode (SQL templates with Lua bindings, sent as one batch or, with the list form of sql, one round trip per statement like pgbench) or script mode (Lua with a database bridge: exec, query, query_one, begin, commit, rollback, copy_*, rand_*). Data loading uses either row templates or Lua load scripts, with shared modules and a pre-load script.
• Bundled reference workloads derived from the public TPC specifications: PGM-B, PGM-C, PGM-H, PGM-DS and PGM-IoT, plus a minimal custom-simple template. PGM-DS implements all 99 TPC-DS queries with per-execution random parameters. PGM-C runs its transactions as PL/pgSQL functions, one round trip each, like HammerDB’s TPROC-C; PGM-C-client drives the same workload from the client statement by statement. Both pick customers, items and last names with TPC-C’s NURand distribution. pgmeter spec list shows them and pgmeter spec new scaffolds a copy to edit.
• A documentation site at https://julmon.github.io/pgmeter/: getting started, the user guide, the benchmark authoring guide and the command-line, Lua API and output-file reference, built with mdBook from docs/. The README is a summary that points to it.
• Each run writes a result directory (pgmeter-<UTC timestamp>/) with run.json (run parameters, elapsed and ramp-up times, throughput unit, the ramp-up boundary), transaction.log (one line per commit with the client that ran it and its start time) and error.log (one record per line, line breaks in messages escaped, the Lua traceback of a script error that repeats written the first time only), per-transaction CSV files (pgmeter-throughput-*.csv in the spec’s unit, response times over time, statistics) and a self-contained HTML report with charts, percentiles, server and data-set information. Committed counts, throughput, statistics and the report’s histogram describe the steady-state window after ramp-up; the time-series charts show the whole run with ramp-up shaded. A timed run’s throughput is judged steady, unstable or drifted from its per-second variation and the drift between the halves of the window, in the console, the JSON output and the report; the headline adds the 99th percentile response time, and the report’s latency chart shows the 95th percentile next to the mean. A spec with metrics.report: stream (PGM-H, PGM-DS) and any run --once is reported as a stream instead: elapsed time, geometric mean of the response times and each transaction’s share of the time, in the console and the JSON output, and in the report a sorted time chart, a timeline per client and a sortable per-transaction table.
• Specs are validated when loaded: unknown keys, unknown modes or load methods, missing sql or script, duplicate or colliding transaction ids and names, a zero total weight, a zero batch_size, undeclared tables and load together with load_script are reported together before anything touches the database.
• A failed transaction rolls back and the connection stays usable; a script that fails mid-COPY has its stream aborted; Lua bridge and loader errors carry the PostgreSQL server message. Lua NaN and infinite numbers reach the server as float8 literals.
• Ctrl-C ends a run early: the clients stop, the results and the report are produced from the steady-state samples so far and marked as interrupted (interrupted in run.json); a second Ctrl-C aborts immediately with status 130.
• Connection settings follow libpq conventions: PGHOST, PGPORT, PGUSER, PGDATABASE, PGPASSWORD, ~/.pgpass and PGOPTIONS, whose server settings reach every connection. The default user name falls back to $USER, $LOGNAME and postgres when the UID has no passwd entry.
• TLS with --sslmode / PGSSLMODE (disable, prefer, require, verify-ca, verify-full, libpq semantics) through rustls; the verify-* modes check the server certificate against the system CA store.
• Released under the PostgreSQL License; third-party notices for Chart.js and Luau in THIRD_PARTY_NOTICES.md.
• Minimum supported Rust version 1.87; PostgreSQL 12 or later; Linux and macOS.
• Continuous integration on stable and the MSRV, with end-to-end tests against PostgreSQL 12 and 18.

Contributing

Issues and pull requests are welcome at github.com/julmon/pgmeter.

Building

cargo build            # debug
cargo build --release  # optimized, what you benchmark with

Rust 1.87 or later. There are no system dependencies: TLS is rustls, the Lua runtime (Luau) is compiled in, the chart library is vendored.

Checks

The CI runs these on every pull request, on stable Rust and on the minimum supported version:

cargo fmt --all --check
cargo clippy --locked --all-targets -- -D warnings
cargo test --locked

The end-to-end tests need a PostgreSQL server and are skipped without one. Point PGMETER_TEST_DSN at a database you can afford to wipe, for example a container:

docker run -d --name pgtest -p 5432:5432 -e POSTGRES_PASSWORD=pgmeter postgres:18
PGMETER_TEST_DSN=postgresql://postgres:pgmeter@localhost:5432/postgres cargo test

They run init and short runs of the bundled and ad-hoc specs against it, on PostgreSQL 12 and the current major in CI. Tests that create tables use their own table names, because they run in parallel.

Documentation

This site is an mdBook under docs/ and is deployed by a workflow on every push to main:

cargo install mdbook --version 0.5.4 --locked
mdbook serve docs --open

The changelog page includes CHANGELOG.md from the repository root.

Project layout

src/
├── main.rs                    # Entry point: dispatches the commands, maps errors to exit codes
├── args.rs                    # CLI argument parsing (clap)
├── spec.rs                    # Embedded benchmark specs (`spec list` / `spec new`)
├── executor.rs                # Executor: connection, spec, aggregation, results, report
└── executor/
    ├── init.rs                # The `init` pipeline (schema, parallel loading, keys, indexes)
    ├── run.rs                 # The `run` pipeline (both client engines, data collector, logs)
    ├── error.rs               # The executor error type and the server-message helper
    ├── tls.rs                 # rustls connector for `--sslmode`
    ├── benchmark.rs           # Transaction and counter types shared by the pipelines
    ├── yaml_types.rs          # Serde structs for the YAML spec
    ├── yaml_benchmark.rs      # Spec validation, schema DDL, loading and transaction execution
    ├── yaml_lua.rs            # Lua VM, DB bridge, binding/template compilation
    ├── data_agg.rs            # Per-transaction response-time statistics
    ├── summary.rs             # Throughput and error-rate math shared by every output
    ├── report.rs              # Self-contained HTML report generation
    ├── terminal.rs            # Terminal output formatting
    ├── txmessage.rs           # Inter-thread messaging
    └── assets/                # Vendored assets (inlined Chart.js)
samples/                       # The bundled benchmark specs, embedded in the binary
docs/                          # This site (mdBook)
tests/                         # End-to-end tests (need `PGMETER_TEST_DSN`)

Conventions

• One topic per pull request, with a commit message that says why.
• A change in behaviour comes with its test and its line in CHANGELOG.md under “Unreleased”.
• The bundled workloads stay derived from the public TPC specifications and keep the wording of the trademark notice; see License and trademarks.

License and trademarks

License

pgmeter is released under the PostgreSQL License.

The binary and the generated reports embed third-party components, Chart.js and Luau among them, under their own licenses. See THIRD_PARTY_NOTICES.md in the repository.

Trademark notice

pgmeter ships workloads (PGM-B, PGM-C, PGM-H, PGM-DS, PGM-IoT) derived from the corresponding TPC and TPCx benchmark specifications. They implement a representative subset of each specification for relative comparison (for example, comparing two table access methods under the same workload). They are not audited, full-specification TPC benchmarks, and their results must not be compared with, or represented as, official TPC results.

TPC, TPC Benchmark, TPC-B, TPC-C, TPC-H, TPC-DS, TPCx-IoT, tpmC, QphH and QphDS are trademarks of the Transaction Processing Performance Council. See www.tpc.org.