pts-disk-different-nvmes

AMD Ryzen Threadripper 1900X 8-Core testing with a Gigabyte X399 DESIGNARE EX-CF (F13a BIOS) and NVIDIA Quadro P400 on Debian 11 via the Phoronix Test Suite.

HTML result view exported from: https://openbenchmarking.org/result/2211127-NE-PTSDISKDI57&grr&rdt.

pts-disk-different-nvmesProcessorMotherboardChipsetMemoryDiskGraphicsAudioMonitorNetworkOSKernelCompilerFile-SystemScreen ResolutionZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVMEAMD Ryzen Threadripper 1900X 8-Core @ 3.80GHz (8 Cores / 16 Threads)Gigabyte X399 DESIGNARE EX-CF (F13a BIOS)AMD 17h64GBSamsung SSD 960 EVO 500GB + 8 x 2000GB Western Digital WD_BLACK SN770 2TB + 1000GB CT1000P5PSSD8NVIDIA Quadro P400NVIDIA GP107GL HD AudioDELL S2340T4 x Intel I350 + Intel 8265 / 8275Debian 115.10.0-19-amd64 (x86_64)GCC 10.2.1 20210110zfs1920x1080ext4OpenBenchmarking.orgKernel Details- Transparent Huge Pages: alwaysCompiler Details- --build=x86_64-linux-gnu --disable-vtable-verify --disable-werror --enable-bootstrap --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-gnu-unique-object --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --enable-libphobos-checking=release --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-link-mutex --enable-multiarch --enable-multilib --enable-nls --enable-objc-gc=auto --enable-offload-targets=nvptx-none=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-gcn/usr,hsa --enable-plugin --enable-shared --enable-threads=posix --host=x86_64-linux-gnu --program-prefix=x86_64-linux-gnu- --target=x86_64-linux-gnu --with-abi=m64 --with-arch-32=i686 --with-build-config=bootstrap-lto-lean --with-default-libstdcxx-abi=new --with-gcc-major-version-only --with-multilib-list=m32,m64,mx32 --with-target-system-zlib=auto --with-tune=generic --without-cuda-driver -v Processor Details- Scaling Governor: acpi-cpufreq schedutil (Boost: Enabled) - CPU Microcode: 0x8001137Disk Scheduler Details- ZFS zraid1 4xNVME Pool: NONEPython Details- Python 3.9.2Security Details- itlb_multihit: Not affected + l1tf: Not affected + mds: Not affected + meltdown: Not affected + mmio_stale_data: Not affected + retbleed: Mitigation of untrained return thunk; SMT vulnerable + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines IBPB: conditional STIBP: disabled RSB filling PBRSB-eIBRS: Not affected + srbds: Not affected + tsx_async_abort: Not affected Disk Details- ext4 mdadm raid5 4xNVME: NONE / relatime,rw,stripe=384 / raid5 nvme4n1p1[4] nvme3n1p1[2] nvme2n1p1[1] nvme1n1p1[0] Block Size: 4096

pts-disk-different-nvmesdbench: 12 Clientsdbench: 1 Clientsfs-mark: 5000 Files, 1MB Size, 4 Threadsfs-mark: 1000 Files, 1MB Sizefs-mark: 1000 Files, 1MB Size, No Sync/FSyncfs-mark: 4000 Files, 32 Sub Dirs, 1MB Sizesqlite: 128postmark: Disk Transaction Performancesqlite: 64fio: Seq Read - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Read - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Read - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Read - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Read - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Read - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Read - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Read - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Write - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Write - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Write - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Seq Write - Linux AIO - No - Yes - 2MB - Default Test Directoryfio: Rand Write - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Rand Write - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Write - Linux AIO - No - Yes - 4KB - Default Test Directoryfio: Seq Write - Linux AIO - No - Yes - 4KB - Default Test Directorycompilebench: Compilesqlite: 32sqlite: 1sqlite: 8compilebench: Read Compiled Treecompilebench: Initial CreateZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME2626.03410.1711632.7646.51359.3684.276.088328937.44832933312874956719421192034424540751225245810292065415001621713336701398.5723.1917.85211.2671212.73222.042484.85453.170537.9254.01733.3284.780.145506845.7202213338653066671199666667064769586501308829673241150004491478.0031.7938.81020.0052642.03411.07OpenBenchmarking.org

Dbench

12 Clients

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.012 ClientsZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME6001200180024003000SE +/- 5.07, N = 3SE +/- 21.52, N = 32626.032484.851. (CC) gcc options: -lpopt -O2

Dbench

1 Clients

OpenBenchmarking.orgMB/s, More Is BetterDbench 4.01 ClientsZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME100200300400500SE +/- 0.85, N = 3SE +/- 0.61, N = 3410.17453.171. (CC) gcc options: -lpopt -O2

FS-Mark

Test: 5000 Files, 1MB Size, 4 Threads

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 5000 Files, 1MB Size, 4 ThreadsZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME400800120016002000SE +/- 4.95, N = 3SE +/- 4.55, N = 121632.7537.9

FS-Mark

Test: 1000 Files, 1MB Size

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB SizeZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME140280420560700SE +/- 5.49, N = 15SE +/- 9.50, N = 15646.5254.0

FS-Mark

Test: 1000 Files, 1MB Size, No Sync/FSync

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 1000 Files, 1MB Size, No Sync/FSyncZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME400800120016002000SE +/- 12.01, N = 15SE +/- 19.84, N = 41359.31733.3

FS-Mark

Test: 4000 Files, 32 Sub Dirs, 1MB Size

OpenBenchmarking.orgFiles/s, More Is BetterFS-Mark 3.3Test: 4000 Files, 32 Sub Dirs, 1MB SizeZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME150300450600750SE +/- 7.90, N = 3SE +/- 2.79, N = 3684.2284.7

SQLite

Threads / Copies: 128

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.30.1Threads / Copies: 128ZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME20406080100SE +/- 0.10, N = 3SE +/- 0.28, N = 376.0980.151. (CC) gcc options: -O2 -lz -lm -ldl -lpthread

PostMark

Disk Transaction Performance

OpenBenchmarking.orgTPS, More Is BetterPostMark 1.51Disk Transaction PerformanceZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME11002200330044005500SE +/- 34.00, N = 3328950681. (CC) gcc options: -O3

SQLite

Threads / Copies: 64

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.30.1Threads / Copies: 64ZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME1020304050SE +/- 0.01, N = 3SE +/- 0.13, N = 337.4545.721. (CC) gcc options: -O2 -lz -lm -ldl -lpthread

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.29Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME70K140K210K280K350KSE +/- 1855.92, N = 3SE +/- 2728.45, N = 33293332213331. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.29Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME30060090012001500SE +/- 7.09, N = 3SE +/- 11.46, N = 312878651. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.29Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME70K140K210K280K350KSE +/- 185.59, N = 3SE +/- 1333.33, N = 3495673066671. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.29Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME30060090012001500SE +/- 0.88, N = 3SE +/- 5.17, N = 319411991. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.29Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME14002800420056007000SE +/- 29.49, N = 3SE +/- 14.99, N = 3211966661. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.29Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME14002800420056007000SE +/- 18.75, N = 3SE +/- 3.61, N = 3203467061. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.29Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Pool9001800270036004500SE +/- 58.64, N = 342451. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.29Type: Sequential Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Pool9001800270036004500SE +/- 37.36, N = 340751. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.29Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME30060090012001500SE +/- 8.99, N = 3SE +/- 1.76, N = 312254761. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.29Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME5001000150020002500SE +/- 17.98, N = 3SE +/- 3.53, N = 324589581. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.29Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME2004006008001000SE +/- 2.91, N = 3SE +/- 0.88, N = 310296501. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.29Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 2MB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME400800120016002000SE +/- 6.36, N = 3SE +/- 1.73, N = 3206513081. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.29Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME20K40K60K80K100KSE +/- 57.74, N = 3SE +/- 569.60, N = 341500829671. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.29Type: Random Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME70140210280350SE +/- 2.00, N = 31623241. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO Tester 3.29Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME40K80K120K160K200KSE +/- 666.67, N = 3SE +/- 577.35, N = 31713331150001. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Flexible IO Tester

Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.29Type: Sequential Write - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test DirectoryZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME140280420560700SE +/- 1.86, N = 3SE +/- 1.15, N = 36704491. (CC) gcc options: -rdynamic -lrt -lz -lpthread -lm -ldl -laio -std=gnu99 -ffast-math -include -O3 -fcommon -U_FORTIFY_SOURCE -march=native

Compile Bench

Test: Compile

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: CompileZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME30060090012001500SE +/- 6.06, N = 3SE +/- 11.73, N = 31398.571478.00

SQLite

Threads / Copies: 32

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.30.1Threads / Copies: 32ZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME714212835SE +/- 0.08, N = 3SE +/- 0.01, N = 323.1931.791. (CC) gcc options: -O2 -lz -lm -ldl -lpthread

SQLite

Threads / Copies: 1

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.30.1Threads / Copies: 1ZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME246810SE +/- 0.071, N = 7SE +/- 0.069, N = 107.8528.8101. (CC) gcc options: -O2 -lz -lm -ldl -lpthread

SQLite

Threads / Copies: 8

OpenBenchmarking.orgSeconds, Fewer Is BetterSQLite 3.30.1Threads / Copies: 8ZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME510152025SE +/- 0.08, N = 3SE +/- 0.05, N = 311.2720.011. (CC) gcc options: -O2 -lz -lm -ldl -lpthread

Compile Bench

Test: Read Compiled Tree

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Read Compiled TreeZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME6001200180024003000SE +/- 8.00, N = 3SE +/- 42.24, N = 31212.732642.03

Compile Bench

Test: Initial Create

OpenBenchmarking.orgMB/s, More Is BetterCompile Bench 0.6Test: Initial CreateZFS zraid1 4xNVME Poolext4 mdadm raid5 4xNVME90180270360450SE +/- 1.57, N = 3SE +/- 0.77, N = 3222.04411.07


Phoronix Test Suite v10.8.5