Pgbench results unstable

Hi,

I am doing some experiments for the database benchmarking. In order to get acquainted with pgbench, I ran some experiments using pgbench on the default postgres configurations. The experiment was run 8 times and there is extreme variability in the results, approximately more than 20%. Each workload experiment was run for 1000 sec. The database server is being restarted everytime to clear the postgres cache. Furthermore I am also dropping the os cache.

Here are the hardware and software configurations of my machine:

Operating System: 20.04 LTS (Focal Fossa)
Postgresql 12.2

#CPU

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 142
Model name: Intel® Core™ i5-7200U CPU @ 2.50GHz
Stepping: 9
CPU MHz: 3099.993
CPU max MHz: 3100,0000
CPU min MHz: 400,0000
BogoMIPS: 5399.81
Virtualization: VT-x
L1d cache: 64 KiB
L1i cache: 64 KiB
L2 cache: 512 KiB
L3 cache: 3 MiB
NUMA node0 CPU(s): 0-3

#Memory

total used free shared buff/cache available
Mem: 11850 2556 4002 835 5291 8163
Swap: 15623 0 15623

I wrote the following shell script:

psql -c ‘DROP DATABASE test’
pg_ctl -D /home/user/mylocal_db start
createdb test
pgbench -i -s 100 test
pg_dump test > pgbench
psql -c ‘DROP DATABASE test’
for i in $(seq 1 8)
do
pg_ctl -D /home/user/mylocal_db stop
sudo sh -c ‘echo 3 > /proc/sys/vm/drop_caches’
pg_ctl -D /home/user/mylocal_db start
createdb test
psql test < pgbench
pgbench -c 10 -T 1000 -j 2 test > output$i.txt
psql -c ‘DROP DATABASE test’

done

Here are the 8 results that I got from running the above script:

Run 1

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 100
query mode: simple
number of clients: 10
number of threads: 2
duration: 1000 s
number of transactions actually processed: 881602
latency average = 11.343 ms
tps = 881.583934 (including connections establishing)
tps = 881.587761 (excluding connections establishing)

Run 2

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 100
query mode: simple
number of clients: 10
number of threads: 2
duration: 1000 s
number of transactions actually processed: 889516
latency average = 11.243 ms
tps = 889.424605 (including connections establishing)
tps = 889.428028 (excluding connections establishing)

Run 3

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 100
query mode: simple
number of clients: 10
number of threads: 2
duration: 1000 s
number of transactions actually processed: 831564
latency average = 12.026 ms
tps = 831.547487 (including connections establishing)
tps = 831.555002 (excluding connections establishing)

Run 4

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 100
query mode: simple
number of clients: 10
number of threads: 2
duration: 1000 s
number of transactions actually processed: 812352
latency average = 12.311 ms
tps = 812.290472 (including connections establishing)
tps = 812.295065 (excluding connections establishing)

Run 5

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 100
query mode: simple
number of clients: 10
number of threads: 2
duration: 1000 s
number of transactions actually processed: 792560
latency average = 12.618 ms
tps = 792.535054 (including connections establishing)
tps = 792.538990 (excluding connections establishing)

Run 6

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 100
query mode: simple
number of clients: 10
number of threads: 2
duration: 1000 s
number of transactions actually processed: 779947
latency average = 12.822 ms
tps = 779.925794 (including connections establishing)
tps = 779.929973 (excluding connections establishing)

Run 7

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 100
query mode: simple
number of clients: 10
number of threads: 2
duration: 1000 s
number of transactions actually processed: 794905
latency average = 12.582 ms
tps = 794.812104 (including connections establishing)
tps = 794.815127 (excluding connections establishing)

Run 8

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 100
query mode: simple
number of clients: 10
number of threads: 2
duration: 1000 s
number of transactions actually processed: 766314
latency average = 13.052 ms
tps = 766.171848 (including connections establishing)
tps = 766.176399 (excluding connections establishing)

Kindly advice me on how I can control this extreme variability to measure the baseline performance under default configurations. What am I missing?.

Thanks in advance for your help,
Best.