=== 02:46:55 0 fresh lab (flat45/00-run-all.sh on server) === 03:26:20 1 fio on every level; the kinc nodes stopped, so the guests are idle === 03:27:25 L0 server: XFS LV (RAID 1 SSD), where f45-min's raw images live L0 server XFS LV run 1: fdatasync p50 0.40 ms p99 0.59 ms p99.9 0.85 ms max 1.41 ms L0 server XFS LV run 2: fdatasync p50 0.41 ms p99 0.56 ms p99.9 0.88 ms max 1.47 ms L0 server XFS LV run 3: fdatasync p50 0.42 ms p99 0.64 ms p99.9 0.88 ms max 2.41 ms L0 server XFS LV randread QD1 run 1: 7290 IOPS p99 0.194 ms L0 server XFS LV randread QD1 run 2: 7328 IOPS p99 0.194 ms L0 server XFS LV randread QD1 run 3: 7459 IOPS p99 0.191 ms L0 server XFS LV randread QD32 run 1: 87229 IOPS p99 0.569 ms L0 server XFS LV randread QD32 run 2: 87466 IOPS p99 0.610 ms L0 server XFS LV randread QD32 run 3: 87722 IOPS p99 0.594 ms L0 server XFS LV randwrite QD1 run 1: 9420 IOPS p99 0.200 ms L0 server XFS LV randwrite QD1 run 2: 9987 IOPS p99 0.194 ms L0 server XFS LV randwrite QD1 run 3: 9374 IOPS p99 0.191 ms L0 server XFS LV randwrite QD32 run 1: 50818 IOPS p99 0.913 ms L0 server XFS LV randwrite QD32 run 2: 50732 IOPS p99 0.938 ms L0 server XFS LV randwrite QD32 run 3: 50525 IOPS p99 0.938 ms === 03:31:48 L1 f45-min: root ext4 on raw (falloc, cache=none, io=native) L1 f45-min root ext4 on raw run 1: fdatasync p50 1.07 ms p99 1.56 ms p99.9 2.21 ms max 6.68 ms L1 f45-min root ext4 on raw run 2: fdatasync p50 0.97 ms p99 1.47 ms p99.9 1.81 ms max 2.42 ms L1 f45-min root ext4 on raw run 3: fdatasync p50 1.01 ms p99 1.43 ms p99.9 2.06 ms max 4.10 ms L1 f45-min root ext4 on raw randread QD1 run 1: 4163 IOPS p99 0.309 ms L1 f45-min root ext4 on raw randread QD1 run 2: 4122 IOPS p99 0.309 ms L1 f45-min root ext4 on raw randread QD1 run 3: 4130 IOPS p99 0.305 ms L1 f45-min root ext4 on raw randread QD32 run 1: 67909 IOPS p99 0.922 ms L1 f45-min root ext4 on raw randread QD32 run 2: 70334 IOPS p99 0.864 ms L1 f45-min root ext4 on raw randread QD32 run 3: 68237 IOPS p99 0.881 ms L1 f45-min root ext4 on raw randwrite QD1 run 1: 5876 IOPS p99 0.264 ms L1 f45-min root ext4 on raw randwrite QD1 run 2: 6007 IOPS p99 0.230 ms L1 f45-min root ext4 on raw randwrite QD1 run 3: 5880 IOPS p99 0.251 ms L1 f45-min root ext4 on raw randwrite QD32 run 1: 38028 IOPS p99 1.401 ms L1 f45-min root ext4 on raw randwrite QD32 run 2: 38234 IOPS p99 1.401 ms L1 f45-min root ext4 on raw randwrite QD32 run 3: 38656 IOPS p99 1.352 ms === 03:36:26 L1 f45-min: /storage XFS on raw, where the guests' qcow2 overlays live L1 f45-min /storage XFS on raw run 1: fdatasync p50 1.09 ms p99 1.58 ms p99.9 2.15 ms max 4.08 ms L1 f45-min /storage XFS on raw run 2: fdatasync p50 1.07 ms p99 1.47 ms p99.9 2.21 ms max 4.69 ms L1 f45-min /storage XFS on raw run 3: fdatasync p50 0.73 ms p99 1.03 ms p99.9 1.45 ms max 5.94 ms L1 f45-min /storage XFS on raw randread QD1 run 1: 4085 IOPS p99 0.330 ms L1 f45-min /storage XFS on raw randread QD1 run 2: 4190 IOPS p99 0.305 ms L1 f45-min /storage XFS on raw randread QD1 run 3: 4303 IOPS p99 0.305 ms L1 f45-min /storage XFS on raw randread QD32 run 1: 70481 IOPS p99 0.872 ms L1 f45-min /storage XFS on raw randread QD32 run 2: 72705 IOPS p99 0.840 ms L1 f45-min /storage XFS on raw randread QD32 run 3: 69766 IOPS p99 0.856 ms L1 f45-min /storage XFS on raw randwrite QD1 run 1: 6181 IOPS p99 0.220 ms L1 f45-min /storage XFS on raw randwrite QD1 run 2: 6217 IOPS p99 0.220 ms L1 f45-min /storage XFS on raw randwrite QD1 run 3: 6206 IOPS p99 0.226 ms L1 f45-min /storage XFS on raw randwrite QD32 run 1: 49586 IOPS p99 0.954 ms L1 f45-min /storage XFS on raw randwrite QD32 run 2: 50153 IOPS p99 0.930 ms L1 f45-min /storage XFS on raw randwrite QD32 run 3: 49800 IOPS p99 0.922 ms === 03:40:59 L2 saturn: root ext4 on a qcow2 overlay (cache=none, io=native), where etcd's data is L2 saturn root ext4 on qcow2 run 1: fdatasync p50 3.00 ms p99 5.60 ms p99.9 6.78 ms max 8.93 ms L2 saturn root ext4 on qcow2 run 2: fdatasync p50 3.59 ms p99 6.19 ms p99.9 7.44 ms max 9.20 ms L2 saturn root ext4 on qcow2 run 3: fdatasync p50 3.69 ms p99 6.19 ms p99.9 7.96 ms max 10.09 ms L2 saturn root ext4 on qcow2 randread QD1 run 1: 1520 IOPS p99 0.922 ms L2 saturn root ext4 on qcow2 randread QD1 run 2: 1517 IOPS p99 0.872 ms L2 saturn root ext4 on qcow2 randread QD1 run 3: 1520 IOPS p99 0.905 ms L2 saturn root ext4 on qcow2 randread QD32 run 1: 24207 IOPS p99 3.686 ms L2 saturn root ext4 on qcow2 randread QD32 run 2: 21990 IOPS p99 4.293 ms L2 saturn root ext4 on qcow2 randread QD32 run 3: 19241 IOPS p99 3.916 ms L2 saturn root ext4 on qcow2 randwrite QD1 run 1: 1779 IOPS p99 0.881 ms L2 saturn root ext4 on qcow2 randwrite QD1 run 2: 2072 IOPS p99 0.741 ms L2 saturn root ext4 on qcow2 randwrite QD1 run 3: 1874 IOPS p99 0.766 ms L2 saturn root ext4 on qcow2 randwrite QD32 run 1: 19305 IOPS p99 3.916 ms L2 saturn root ext4 on qcow2 randwrite QD32 run 2: 18627 IOPS p99 2.638 ms L2 saturn root ext4 on qcow2 randwrite QD32 run 3: 23769 IOPS p99 3.883 ms === 03:46:36 L2 jupiter: same stack L2 jupiter root ext4 on qcow2 run 1: fdatasync p50 2.97 ms p99 5.87 ms p99.9 7.18 ms max 16.41 ms L2 jupiter root ext4 on qcow2 run 2: fdatasync p50 3.06 ms p99 5.60 ms p99.9 7.18 ms max 16.27 ms L2 jupiter root ext4 on qcow2 run 3: fdatasync p50 1.34 ms p99 4.11 ms p99.9 4.75 ms max 6.76 ms L2 jupiter root ext4 on qcow2 randread QD1 run 1: 1477 IOPS p99 0.905 ms L2 jupiter root ext4 on qcow2 randread QD1 run 2: 1674 IOPS p99 0.897 ms L2 jupiter root ext4 on qcow2 randread QD1 run 3: 1493 IOPS p99 0.905 ms L2 jupiter root ext4 on qcow2 randread QD32 run 1: 20848 IOPS p99 2.671 ms L2 jupiter root ext4 on qcow2 randread QD32 run 2: 22448 IOPS p99 4.145 ms L2 jupiter root ext4 on qcow2 randread QD32 run 3: 21834 IOPS p99 3.686 ms L2 jupiter root ext4 on qcow2 randwrite QD1 run 1: 1575 IOPS p99 0.848 ms L2 jupiter root ext4 on qcow2 randwrite QD1 run 2: 1478 IOPS p99 0.872 ms L2 jupiter root ext4 on qcow2 randwrite QD1 run 3: 1490 IOPS p99 0.864 ms L2 jupiter root ext4 on qcow2 randwrite QD32 run 1: 19201 IOPS p99 4.227 ms L2 jupiter root ext4 on qcow2 randwrite QD32 run 2: 20419 IOPS p99 2.572 ms L2 jupiter root ext4 on qcow2 randwrite QD32 run 3: 19031 IOPS p99 4.080 ms === 03:51:57 2 restart the kinc nodes === 03:52:36 STORAGE REDO DONE