host-nfs-6-zfs-3xRaidz2-sas-3TB.20240616

2 x Intel Xeon E5-2697A v4 testing with a Supermicro PIO-648R-E1CR36L+-ST031 X10DRi-T4+ v1.01 (3.4 BIOS) and ASPEED on Debian GNU/Linux 12 via the Phoronix Test Suite.

Compare your own system(s) to this result file with the Phoronix Test Suite by running the command: phoronix-test-suite benchmark 2406186-MBI-HOSTNFS600
Jump To Table - Results

Statistics

Remove Outliers Before Calculating Averages

Graph Settings

Prefer Vertical Bar Graphs

Multi-Way Comparison

Condense Multi-Option Tests Into Single Result Graphs
Condense Test Profiles With Multiple Version Results Into Single Result Graphs

Table

Show Detailed System Result Table

Run Management

Result
Identifier
Performance Per
Dollar
Date
Run
  Test
  Duration
Newly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test
June 16
  7 Hours, 13 Minutes
Only show results matching title/arguments (delimit multiple options with a comma):
Do not show results matching title/arguments (delimit multiple options with a comma):


host-nfs-6-zfs-3xRaidz2-sas-3TB.20240616OpenBenchmarking.orgPhoronix Test Suite2 x Intel Xeon E5-2697A v4 @ 3.60GHz (32 Cores / 64 Threads)Supermicro PIO-648R-E1CR36L+-ST031 X10DRi-T4+ v1.01 (3.4 BIOS)Intel Xeon E7 v4/Xeon24 x 32 GB DDR4-1866MT/s Samsung M386A4G40DM0-CPB12 x 3001GB HUS723030ALS64 + 15 x 3001GB Seagate ST33000650SS + 9 x 3001GB Seagate ST3000NM0023 + 2048GB Lexar SSD NS100 + 32GB Cruzer BladeASPEEDDELL S2009W4 x Intel 10-Gigabit X540-AT2Debian GNU/Linux 126.8.4-3-pve (x86_64)GCC 12.2.0ext41600x900ProcessorMotherboardChipsetMemoryDiskGraphicsMonitorNetworkOSKernelCompilerFile-SystemScreen ResolutionHost-nfs-6-zfs-3xRaidz2-sas-3TB.20240616 BenchmarksSystem Logs- Transparent Huge Pages: madvise- MQ-DEADLINE / errors=remount-ro,relatime,rw / Block Size: 4096- Scaling Governor: intel_cpufreq performance - CPU Microcode: 0xb00003e- gather_data_sampling: Not affected + itlb_multihit: KVM: Mitigation of VMX disabled + l1tf: Mitigation of PTE Inversion; VMX: conditional cache flushes SMT vulnerable + mds: Mitigation of Clear buffers; SMT vulnerable + meltdown: Mitigation of PTI + mmio_stale_data: Mitigation of Clear buffers; SMT vulnerable + reg_file_data_sampling: Not affected + retbleed: Not affected + spec_rstack_overflow: Not affected + spec_store_bypass: Mitigation of SSB disabled via prctl + spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization + spectre_v2: Mitigation of Retpolines; IBPB: conditional; IBRS_FW; STIBP: conditional; RSB filling; PBRSB-eIBRS: Not affected; BHI: Not affected + srbds: Not affected + tsx_async_abort: Mitigation of Clear buffers; SMT vulnerable

host-nfs-6-zfs-3xRaidz2-sas-3TB.20240616fio: Seq Read - POSIX AIO - Yes - No - 4KB - /datafio: Rand Write - Linux AIO - Yes - No - 128MB - /datafio: Seq Read - Linux AIO - Yes - No - 4KB - /datafio: Rand Read - POSIX AIO - Yes - No - 128MB - /datafio: Rand Write - POSIX AIO - Yes - No - 16MB - /datafio: Seq Read - Linux AIO - Yes - No - 128MB - /datafio: Seq Write - Linux AIO - Yes - No - 128MB - /datafio: Rand Write - Sync - Yes - No - 8MB - /datafio: Rand Read - Sync - Yes - No - 128MB - /datafio: Rand Write - Sync - Yes - No - 4KB - /datafio: Rand Write - Sync - Yes - No - 4KB - /datafio: Rand Read - Sync - Yes - No - 64MB - /datafio: Rand Read - Sync - Yes - No - 16MB - /datafio: Rand Read - Sync - Yes - No - 8MB - /datafio: Seq Read - POSIX AIO - Yes - No - 16MB - /datafio: Rand Read - Sync - Yes - No - 512KB - /datafio: Seq Write - Linux AIO - Yes - No - 8MB - /datafio: Rand Write - Sync - Yes - No - 16MB - /datafio: Seq Write - Linux AIO - Yes - No - 16MB - /datafio: Rand Write - Sync - Yes - No - 64MB - /datafio: Rand Read - Sync - Yes - No - 4KB - /datafio: Rand Write - Sync - Yes - No - 128MB - /datafio: Rand Write - Linux AIO - Yes - No - 16MB - /datafio: Rand Write - Sync - Yes - No - 512KB - /datafio: Seq Write - Sync - Yes - No - 128MB - /datafio: Seq Read - Sync - Yes - No - 4KB - /datafio: Rand Write - POSIX AIO - Yes - No - 128MB - /datafio: Seq Read - Sync - Yes - No - 4KB - /datafio: Seq Read - Linux AIO - Yes - No - 8MB - /datafio: Seq Read - Sync - Yes - No - 8MB - /datafio: Seq Read - Linux AIO - Yes - No - 16MB - /datafio: Rand Read - Linux AIO - Yes - No - 4KB - /datafio: Seq Write - Linux AIO - Yes - No - 4KB - /datafio: Rand Read - Linux AIO - Yes - No - 4KB - /datafio: Seq Write - POSIX AIO - Yes - No - 4KB - /datafio: Rand Read - Linux AIO - Yes - No - 8MB - /datafio: Seq Read - POSIX AIO - Yes - No - 128MB - /datafio: Rand Read - POSIX AIO - Yes - No - 4KB - /datafio: Seq Write - POSIX AIO - Yes - No - 16MB - /datafio: Rand Read - POSIX AIO - Yes - No - 4KB - /datafio: Seq Write - POSIX AIO - Yes - No - 128MB - /datafio: Rand Read - POSIX AIO - Yes - No - 8MB - /datafio: Rand Read - Linux AIO - Yes - No - 512KB - /datafio: Seq Read - Sync - Yes - No - 16MB - /datafio: Rand Read - POSIX AIO - Yes - No - 512KB - /datafio: Seq Read - Sync - Yes - No - 64MB - /datafio: Rand Write - Linux AIO - Yes - No - 64MB - /datafio: Seq Write - Sync - Yes - No - 4KB - /datafio: Rand Write - POSIX AIO - Yes - No - 64MB - /datafio: Seq Write - Sync - Yes - No - 4KB - /datafio: Seq Write - Sync - Yes - No - 512KB - /datafio: Seq Write - Sync - Yes - No - 8MB - /datafio: Rand Write - Linux AIO - Yes - No - 512KB - /datafio: Rand Read - Linux AIO - Yes - No - 16MB - /datafio: Rand Write - POSIX AIO - Yes - No - 512KB - /datafio: Rand Read - Linux AIO - Yes - No - 64MB - /datafio: Seq Read - Linux AIO - Yes - No - 4KB - /datafio: Rand Read - POSIX AIO - Yes - No - 16MB - /datafio: Seq Read - POSIX AIO - Yes - No - 4KB - /datafio: Rand Read - POSIX AIO - Yes - No - 64MB - /datafio: Seq Read - POSIX AIO - Yes - No - 8MB - /datafio: Rand Write - Linux AIO - Yes - No - 4KB - /datafio: Seq Read - Linux AIO - Yes - No - 64MB - /datafio: Rand Write - Linux AIO - Yes - No - 4KB - /datafio: Seq Read - POSIX AIO - Yes - No - 64MB - /datafio: Rand Write - Linux AIO - Yes - No - 8MB - /datafio: Seq Write - Linux AIO - Yes - No - 4KB - /datafio: Rand Write - POSIX AIO - Yes - No - 4KB - /datafio: Seq Write - POSIX AIO - Yes - No - 4KB - /datafio: Rand Write - POSIX AIO - Yes - No - 4KB - /datafio: Seq Write - POSIX AIO - Yes - No - 8MB - /datafio: Rand Write - POSIX AIO - Yes - No - 8MB - /datafio: Seq Read - Linux AIO - Yes - No - 512KB - /datafio: Seq Read - Sync - Yes - No - 128MB - /datafio: Seq Read - POSIX AIO - Yes - No - 512KB - /datafio: Seq Read - Sync - Yes - No - 512KB - /datafio: Seq Write - Linux AIO - Yes - No - 64MB - /datafio: Seq Write - Sync - Yes - No - 16MB - /datafio: Seq Write - POSIX AIO - Yes - No - 64MB - /datafio: Seq Write - Sync - Yes - No - 64MB - /datafio: Seq Write - Linux AIO - Yes - No - 512KB - /datafio: Rand Read - Linux AIO - Yes - No - 128MB - /datafio: Seq Write - POSIX AIO - Yes - No - 512KB - /datafio: Rand Read - Sync - Yes - No - 4KB - /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test2692672655883206426172426258530902266404103367223525653047203430832583290924792196686332345259332412190114924442941332154270320782665276820016500023151845280259171767241723072184212422232076236671024021818003088312824402387264325152264002284105220172049363224092833176826891348464466451143332827283519152019197525842293289724002202240026162632268OpenBenchmarking.org

Flexible IO Tester

Fio is an advanced disk benchmark that depends upon the kernel's AIO access library. Learn more via the OpenBenchmarking.org test page.

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Sequential Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test60K120K180K240K300KSE +/- 3008.66, N = 15269267

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 29.96, N = 32655

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test2004006008001000SE +/- 9.04, N = 5883

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 70.29, N = 152064

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 50.67, N = 152617

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 24.87, N = 52426

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 27.45, N = 152585

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test7001400210028003500SE +/- 28.02, N = 33090

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 14.75, N = 32266

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test90180270360450SE +/- 3.39, N = 15404

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Random Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test20K40K60K80K100KSE +/- 864.41, N = 15103367

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 16.80, N = 32235

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 29.01, N = 42565

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test7001400210028003500SE +/- 36.59, N = 33047

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 22.64, N = 152034

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test7001400210028003500SE +/- 24.02, N = 33083

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 37.56, N = 152583

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 38.67, N = 152909

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 51.66, N = 122479

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 27.26, N = 152196

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Random Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test15K30K45K60K75KSE +/- 617.34, N = 368633

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 9.82, N = 32345

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 19.27, N = 152593

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test7001400210028003500SE +/- 33.06, N = 153241

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 38.93, N = 152190

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test2004006008001000SE +/- 7.44, N = 151149

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 93.27, N = 122444

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Sequential Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test60K120K180K240K300KSE +/- 1886.96, N = 15294133

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 23.69, N = 32154

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 19.43, N = 32703

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 20.35, N = 152078

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test60120180240300SE +/- 0.88, N = 3266

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test110220330440550SE +/- 3.59, N = 13527

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Random Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test15K30K45K60K75KSE +/- 200.00, N = 368200

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Sequential Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test40K80K120K160K200KSE +/- 1154.70, N = 3165000

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 22.91, N = 152315

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 89.63, N = 121845

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test60120180240300SE +/- 3.27, N = 15280

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 37.45, N = 152591

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Random Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test15K30K45K60K75KSE +/- 845.47, N = 1571767

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 128.85, N = 152417

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 24.73, N = 152307

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 22.60, N = 152184

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 22.33, N = 152124

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 24.44, N = 152223

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 9.67, N = 32076

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 24.84, N = 152366

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test150300450600750SE +/- 8.39, N = 15710

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 72.49, N = 152402

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Sequential Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test40K80K120K160K200KSE +/- 2158.48, N = 15181800

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test7001400210028003500SE +/- 59.06, N = 123088

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test7001400210028003500SE +/- 44.55, N = 33128

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 27.18, N = 152440

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 27.61, N = 42387

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 47.06, N = 152643

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 16.56, N = 32515

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Sequential Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test50K100K150K200K250KSE +/- 2249.44, N = 5226400

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 18.15, N = 32284

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test2004006008001000SE +/- 11.70, N = 151052

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 60.26, N = 152017

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 22.53, N = 52049

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test80160240320400SE +/- 4.10, N = 3363

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 32.11, N = 32240

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Random Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test20K40K60K80K100KSE +/- 1083.72, N = 392833

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 33.80, N = 121768

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 9.39, N = 32689

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Sequential Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test30K60K90K120K150KSE +/- 918.79, N = 13134846

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test100200300400500SE +/- 4.36, N = 3446

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test140280420560700SE +/- 4.63, N = 3645

OpenBenchmarking.orgIOPS, More Is BetterFlexible IO TesterType: Random Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test20K40K60K80K100KSE +/- 1201.85, N = 3114333

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 21.09, N = 32827

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 8MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 21.30, N = 32835

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 8.09, N = 31915

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 24.02, N = 152019

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test400800120016002000SE +/- 18.11, N = 151975

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 25.90, N = 152584

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 33.84, N = 152293

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 16MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 39.98, N = 32897

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 90.03, N = 152400

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 64MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 26.08, N = 152202

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test5001000150020002500SE +/- 25.78, N = 152400

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Linux AIO - Buffered: Yes - Direct: No - Block Size: 128MB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 30.38, N = 32616

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Sequential Write - IO Engine: POSIX AIO - Buffered: Yes - Direct: No - Block Size: 512KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test6001200180024003000SE +/- 31.10, N = 152632

OpenBenchmarking.orgMB/s, More Is BetterFlexible IO Tester 3.33Type: Random Read - IO Engine: Sync - Buffered: Yes - Direct: No - Block Size: 4KB - Disk Target: /dataNewly built, 3 VDAVs Of 12 SAS in RAIDZ2, Large Test60120180240300SE +/- 2.52, N = 3268

84 Results Shown

Flexible IO Tester:
  Seq Read - POSIX AIO - Yes - No - 4KB - /data
  Rand Write - Linux AIO - Yes - No - 128MB - /data
  Seq Read - Linux AIO - Yes - No - 4KB - /data
  Rand Read - POSIX AIO - Yes - No - 128MB - /data
  Rand Write - POSIX AIO - Yes - No - 16MB - /data
  Seq Read - Linux AIO - Yes - No - 128MB - /data
  Seq Write - Linux AIO - Yes - No - 128MB - /data
  Rand Write - Sync - Yes - No - 8MB - /data
  Rand Read - Sync - Yes - No - 128MB - /data
  Rand Write - Sync - Yes - No - 4KB - /data
  Rand Write - Sync - Yes - No - 4KB - /data
  Rand Read - Sync - Yes - No - 64MB - /data
  Rand Read - Sync - Yes - No - 16MB - /data
  Rand Read - Sync - Yes - No - 8MB - /data
  Seq Read - POSIX AIO - Yes - No - 16MB - /data
  Rand Read - Sync - Yes - No - 512KB - /data
  Seq Write - Linux AIO - Yes - No - 8MB - /data
  Rand Write - Sync - Yes - No - 16MB - /data
  Seq Write - Linux AIO - Yes - No - 16MB - /data
  Rand Write - Sync - Yes - No - 64MB - /data
  Rand Read - Sync - Yes - No - 4KB - /data
  Rand Write - Sync - Yes - No - 128MB - /data
  Rand Write - Linux AIO - Yes - No - 16MB - /data
  Rand Write - Sync - Yes - No - 512KB - /data
  Seq Write - Sync - Yes - No - 128MB - /data
  Seq Read - Sync - Yes - No - 4KB - /data
  Rand Write - POSIX AIO - Yes - No - 128MB - /data
  Seq Read - Sync - Yes - No - 4KB - /data
  Seq Read - Linux AIO - Yes - No - 8MB - /data
  Seq Read - Sync - Yes - No - 8MB - /data
  Seq Read - Linux AIO - Yes - No - 16MB - /data
  Rand Read - Linux AIO - Yes - No - 4KB - /data
  Seq Write - Linux AIO - Yes - No - 4KB - /data
  Rand Read - Linux AIO - Yes - No - 4KB - /data
  Seq Write - POSIX AIO - Yes - No - 4KB - /data
  Rand Read - Linux AIO - Yes - No - 8MB - /data
  Seq Read - POSIX AIO - Yes - No - 128MB - /data
  Rand Read - POSIX AIO - Yes - No - 4KB - /data
  Seq Write - POSIX AIO - Yes - No - 16MB - /data
  Rand Read - POSIX AIO - Yes - No - 4KB - /data
  Seq Write - POSIX AIO - Yes - No - 128MB - /data
  Rand Read - POSIX AIO - Yes - No - 8MB - /data
  Rand Read - Linux AIO - Yes - No - 512KB - /data
  Seq Read - Sync - Yes - No - 16MB - /data
  Rand Read - POSIX AIO - Yes - No - 512KB - /data
  Seq Read - Sync - Yes - No - 64MB - /data
  Rand Write - Linux AIO - Yes - No - 64MB - /data
  Seq Write - Sync - Yes - No - 4KB - /data
  Rand Write - POSIX AIO - Yes - No - 64MB - /data
  Seq Write - Sync - Yes - No - 4KB - /data
  Seq Write - Sync - Yes - No - 512KB - /data
  Seq Write - Sync - Yes - No - 8MB - /data
  Rand Write - Linux AIO - Yes - No - 512KB - /data
  Rand Read - Linux AIO - Yes - No - 16MB - /data
  Rand Write - POSIX AIO - Yes - No - 512KB - /data
  Rand Read - Linux AIO - Yes - No - 64MB - /data
  Seq Read - Linux AIO - Yes - No - 4KB - /data
  Rand Read - POSIX AIO - Yes - No - 16MB - /data
  Seq Read - POSIX AIO - Yes - No - 4KB - /data
  Rand Read - POSIX AIO - Yes - No - 64MB - /data
  Seq Read - POSIX AIO - Yes - No - 8MB - /data
  Rand Write - Linux AIO - Yes - No - 4KB - /data
  Seq Read - Linux AIO - Yes - No - 64MB - /data
  Rand Write - Linux AIO - Yes - No - 4KB - /data
  Seq Read - POSIX AIO - Yes - No - 64MB - /data
  Rand Write - Linux AIO - Yes - No - 8MB - /data
  Seq Write - Linux AIO - Yes - No - 4KB - /data
  Rand Write - POSIX AIO - Yes - No - 4KB - /data
  Seq Write - POSIX AIO - Yes - No - 4KB - /data
  Rand Write - POSIX AIO - Yes - No - 4KB - /data
  Seq Write - POSIX AIO - Yes - No - 8MB - /data
  Rand Write - POSIX AIO - Yes - No - 8MB - /data
  Seq Read - Linux AIO - Yes - No - 512KB - /data
  Seq Read - Sync - Yes - No - 128MB - /data
  Seq Read - POSIX AIO - Yes - No - 512KB - /data
  Seq Read - Sync - Yes - No - 512KB - /data
  Seq Write - Linux AIO - Yes - No - 64MB - /data
  Seq Write - Sync - Yes - No - 16MB - /data
  Seq Write - POSIX AIO - Yes - No - 64MB - /data
  Seq Write - Sync - Yes - No - 64MB - /data
  Seq Write - Linux AIO - Yes - No - 512KB - /data
  Rand Read - Linux AIO - Yes - No - 128MB - /data
  Seq Write - POSIX AIO - Yes - No - 512KB - /data
  Rand Read - Sync - Yes - No - 4KB - /data