Machine Learning Machine Learning

The machine learning test suite helps to benchmark a system for the popular pattern recognition and computational learning algorithms. Mainly different machine learning / deep learning benchmarks.

See how your system performs with this suite using the Phoronix Test Suite. It's as easy as running the phoronix-test-suite benchmark machine-learning command..

Tests In This Suite

  • AI Benchmark Alpha

  • Caffe

  •         Model: AlexNet - Acceleration: CPU - Iterations: 100
  •         Model: AlexNet - Acceleration: CPU - Iterations: 200
  •         Model: AlexNet - Acceleration: CPU - Iterations: 1000
  •         Model: GoogleNet - Acceleration: CPU - Iterations: 100
  •         Model: GoogleNet - Acceleration: CPU - Iterations: 200
  •         Model: GoogleNet - Acceleration: CPU - Iterations: 1000

  • DeepSpeech

  •         Acceleration: CPU

  • ECP-CANDLE

  •         Benchmark: P1B2
  •         Benchmark: P3B1
  •         Benchmark: P3B2

  • LeelaChessZero

  •         Backend: BLAS

  • Llama.cpp

  •         Model: llama-2-7b.Q4_0.gguf
  •         Model: llama-2-13b.Q4_0.gguf
  •         Model: llama-2-70b-chat.Q5_0.gguf

  • Llamafile

  •         Test: mistral-7b-instruct-v0.2.Q8_0 - Acceleration: CPU
  •         Test: llava-v1.5-7b-q4 - Acceleration: CPU
  •         Test: wizardcoder-python-34b-v1.0.Q6_K - Acceleration: CPU

  • Mlpack Benchmark

  •         Benchmark: scikit_svm
  •         Benchmark: scikit_linearridgeregression
  •         Benchmark: scikit_qda
  •         Benchmark: scikit_ica

  • Mobile Neural Network

  • NCNN

  •         Target: CPU
  •         Target: Vulkan GPU

  • Neural Magic DeepSparse

  •         Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Synchronous Single-Stream
  •         Model: NLP Text Classification, BERT base uncased SST2, Sparse INT8 - Scenario: Asynchronous Multi-Stream
  •         Model: NLP Text Classification, DistilBERT mnli - Scenario: Synchronous Single-Stream
  •         Model: NLP Text Classification, DistilBERT mnli - Scenario: Asynchronous Multi-Stream
  •         Model: CV Classification, ResNet-50 ImageNet - Scenario: Synchronous Single-Stream
  •         Model: CV Classification, ResNet-50 ImageNet - Scenario: Asynchronous Multi-Stream
  •         Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Synchronous Single-Stream
  •         Model: NLP Token Classification, BERT base uncased conll2003 - Scenario: Asynchronous Multi-Stream
  •         Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Synchronous Single-Stream
  •         Model: CV Detection, YOLOv5s COCO, Sparse INT8 - Scenario: Asynchronous Multi-Stream
  •         Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Synchronous Single-Stream
  •         Model: NLP Document Classification, oBERT base uncased on IMDB - Scenario: Asynchronous Multi-Stream
  •         Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Synchronous Single-Stream
  •         Model: CV Segmentation, 90% Pruned YOLACT Pruned - Scenario: Asynchronous Multi-Stream
  •         Model: ResNet-50, Baseline - Scenario: Synchronous Single-Stream
  •         Model: ResNet-50, Baseline - Scenario: Asynchronous Multi-Stream
  •         Model: ResNet-50, Sparse INT8 - Scenario: Synchronous Single-Stream
  •         Model: ResNet-50, Sparse INT8 - Scenario: Asynchronous Multi-Stream
  •         Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Synchronous Single-Stream
  •         Model: BERT-Large, NLP Question Answering, Sparse INT8 - Scenario: Asynchronous Multi-Stream
  •         Model: Llama2 Chat 7b Quantized - Scenario: Synchronous Single-Stream
  •         Model: Llama2 Chat 7b Quantized - Scenario: Asynchronous Multi-Stream

  • Numenta Anomaly Benchmark

  •         Detector: Bayesian Changepoint
  •         Detector: Windowed Gaussian
  •         Detector: Relative Entropy
  •         Detector: Earthgecko Skyline
  •         Detector: KNN CAD
  •         Detector: Contextual Anomaly Detector OSE

  • Numpy Benchmark

  • oneDNN

  •         Harness: Convolution Batch Shapes Auto - Engine: CPU
  •         Harness: Deconvolution Batch shapes_1d - Engine: CPU
  •         Harness: Deconvolution Batch shapes_3d - Engine: CPU
  •         Harness: IP Shapes 1D - Engine: CPU
  •         Harness: IP Shapes 3D - Engine: CPU
  •         Harness: Recurrent Neural Network Training - Engine: CPU
  •         Harness: Recurrent Neural Network Inference - Engine: CPU

  • ONNX Runtime

  •         Model: yolov4 - Device: CPU - Executor: Standard
  •         Model: yolov4 - Device: CPU - Executor: Parallel
  •         Model: fcn-resnet101-11 - Device: CPU - Executor: Standard
  •         Model: fcn-resnet101-11 - Device: CPU - Executor: Parallel
  •         Model: super-resolution-10 - Device: CPU - Executor: Standard
  •         Model: super-resolution-10 - Device: CPU - Executor: Parallel
  •         Model: bertsquad-12 - Device: CPU - Executor: Standard
  •         Model: bertsquad-12 - Device: CPU - Executor: Parallel
  •         Model: GPT-2 - Device: CPU - Executor: Standard
  •         Model: GPT-2 - Device: CPU - Executor: Parallel
  •         Model: ArcFace ResNet-100 - Device: CPU - Executor: Standard
  •         Model: ArcFace ResNet-100 - Device: CPU - Executor: Parallel
  •         Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Standard
  •         Model: ResNet50 v1-12-int8 - Device: CPU - Executor: Parallel
  •         Model: CaffeNet 12-int8 - Device: CPU - Executor: Standard
  •         Model: CaffeNet 12-int8 - Device: CPU - Executor: Parallel
  •         Model: Faster R-CNN R-50-FPN-int8 - Device: CPU - Executor: Standard
  •         Model: Faster R-CNN R-50-FPN-int8 - Device: CPU - Executor: Parallel
  •         Model: T5 Encoder - Device: CPU - Executor: Standard
  •         Model: T5 Encoder - Device: CPU - Executor: Parallel
  •         Model: ZFNet-512 - Device: CPU - Executor: Standard
  •         Model: ZFNet-512 - Device: CPU - Executor: Parallel
  •         Model: ResNet101_DUC_HDC-12 - Device: CPU - Executor: Standard
  •         Model: ResNet101_DUC_HDC-12 - Device: CPU - Executor: Parallel

  • OpenCV

  •         Test: DNN - Deep Neural Network

  • OpenVINO

  •         Model: Face Detection FP16 - Device: CPU
  •         Model: Face Detection FP16-INT8 - Device: CPU
  •         Model: Age Gender Recognition Retail 0013 FP16 - Device: CPU
  •         Model: Age Gender Recognition Retail 0013 FP16-INT8 - Device: CPU
  •         Model: Person Detection FP16 - Device: CPU
  •         Model: Person Detection FP32 - Device: CPU
  •         Model: Weld Porosity Detection FP16-INT8 - Device: CPU
  •         Model: Weld Porosity Detection FP16 - Device: CPU
  •         Model: Vehicle Detection FP16-INT8 - Device: CPU
  •         Model: Vehicle Detection FP16 - Device: CPU
  •         Model: Person Vehicle Bike Detection FP16 - Device: CPU
  •         Model: Machine Translation EN To DE FP16 - Device: CPU
  •         Model: Face Detection Retail FP16 - Device: CPU
  •         Model: Face Detection Retail FP16-INT8 - Device: CPU
  •         Model: Handwritten English Recognition FP16 - Device: CPU
  •         Model: Handwritten English Recognition FP16-INT8 - Device: CPU
  •         Model: Road Segmentation ADAS FP16 - Device: CPU
  •         Model: Road Segmentation ADAS FP16-INT8 - Device: CPU
  •         Model: Person Re-Identification Retail FP16 - Device: CPU
  •         Model: Noise Suppression Poconet-Like FP16 - Device: CPU

  • PlaidML

  •         FP16: No - Mode: Inference - Network: ResNet 50 - Device: CPU
  •         FP16: No - Mode: Inference - Network: VGG16 - Device: CPU

  • PyTorch

  •         Device: CPU - Batch Size: 1 - Model: ResNet-50
  •         Device: CPU - Batch Size: 1 - Model: ResNet-152
  •         Device: CPU - Batch Size: 1 - Model: Efficientnet_v2_l
  •         Device: CPU - Batch Size: 16 - Model: ResNet-50
  •         Device: CPU - Batch Size: 16 - Model: ResNet-152
  •         Device: CPU - Batch Size: 16 - Model: Efficientnet_v2_l
  •         Device: CPU - Batch Size: 32 - Model: ResNet-50
  •         Device: CPU - Batch Size: 32 - Model: ResNet-152
  •         Device: CPU - Batch Size: 32 - Model: Efficientnet_v2_l
  •         Device: CPU - Batch Size: 64 - Model: ResNet-50
  •         Device: CPU - Batch Size: 64 - Model: ResNet-152
  •         Device: CPU - Batch Size: 64 - Model: Efficientnet_v2_l
  •         Device: CPU - Batch Size: 256 - Model: ResNet-50
  •         Device: CPU - Batch Size: 256 - Model: ResNet-152
  •         Device: CPU - Batch Size: 256 - Model: Efficientnet_v2_l
  •         Device: CPU - Batch Size: 512 - Model: ResNet-50
  •         Device: CPU - Batch Size: 512 - Model: ResNet-152
  •         Device: CPU - Batch Size: 512 - Model: Efficientnet_v2_l
  •         Device: NVIDIA CUDA GPU - Batch Size: 1 - Model: ResNet-50
  •         Device: NVIDIA CUDA GPU - Batch Size: 1 - Model: ResNet-152
  •         Device: NVIDIA CUDA GPU - Batch Size: 1 - Model: Efficientnet_v2_l
  •         Device: NVIDIA CUDA GPU - Batch Size: 16 - Model: ResNet-50
  •         Device: NVIDIA CUDA GPU - Batch Size: 16 - Model: ResNet-152
  •         Device: NVIDIA CUDA GPU - Batch Size: 16 - Model: Efficientnet_v2_l
  •         Device: NVIDIA CUDA GPU - Batch Size: 32 - Model: ResNet-50
  •         Device: NVIDIA CUDA GPU - Batch Size: 32 - Model: ResNet-152
  •         Device: NVIDIA CUDA GPU - Batch Size: 32 - Model: Efficientnet_v2_l
  •         Device: NVIDIA CUDA GPU - Batch Size: 64 - Model: ResNet-50
  •         Device: NVIDIA CUDA GPU - Batch Size: 64 - Model: ResNet-152
  •         Device: NVIDIA CUDA GPU - Batch Size: 64 - Model: Efficientnet_v2_l
  •         Device: NVIDIA CUDA GPU - Batch Size: 256 - Model: ResNet-50
  •         Device: NVIDIA CUDA GPU - Batch Size: 256 - Model: ResNet-152
  •         Device: NVIDIA CUDA GPU - Batch Size: 256 - Model: Efficientnet_v2_l
  •         Device: NVIDIA CUDA GPU - Batch Size: 512 - Model: ResNet-50
  •         Device: NVIDIA CUDA GPU - Batch Size: 512 - Model: ResNet-152
  •         Device: NVIDIA CUDA GPU - Batch Size: 512 - Model: Efficientnet_v2_l

  • R Benchmark

  • RNNoise

  •         Input: 26 Minute Long Talking Sample

  • Scikit-Learn

  •         Benchmark: 20 Newsgroups / Logistic Regression
  •         Benchmark: Covertype Dataset Benchmark
  •         Benchmark: Feature Expansions
  •         Benchmark: GLM
  •         Benchmark: Glmnet
  •         Benchmark: Hist Gradient Boosting
  •         Benchmark: Hist Gradient Boosting Adult
  •         Benchmark: Hist Gradient Boosting Categorical Only
  •         Benchmark: Hist Gradient Boosting Higgs Boson
  •         Benchmark: Hist Gradient Boosting Threading
  •         Benchmark: Isolation Forest
  •         Benchmark: Isotonic / Perturbed Logarithm
  •         Benchmark: Isotonic / Logistic
  •         Benchmark: Isotonic / Pathological
  •         Benchmark: Kernel PCA Solvers / Time vs. N Components
  •         Benchmark: Kernel PCA Solvers / Time vs. N Samples
  •         Benchmark: Lasso
  •         Benchmark: LocalOutlierFactor
  •         Benchmark: SGDOneClassSVM
  •         Benchmark: Plot Fast KMeans
  •         Benchmark: Plot Hierarchical
  •         Benchmark: Plot Incremental PCA
  •         Benchmark: Plot Lasso Path
  •         Benchmark: Plot Neighbors
  •         Benchmark: Plot Non-Negative Matrix Factorization
  •         Benchmark: Plot OMP vs. LARS
  •         Benchmark: Plot Parallel Pairwise
  •         Benchmark: Plot Polynomial Kernel Approximation
  •         Benchmark: Plot Singular Value Decomposition
  •         Benchmark: Plot Ward
  •         Benchmark: Sparse Random Projections / 100 Iterations
  •         Benchmark: RCV1 Logreg Convergencet
  •         Benchmark: SAGA
  •         Benchmark: Sample Without Replacement
  •         Benchmark: SGD Regression
  •         Benchmark: Sparsify
  •         Benchmark: Text Vectorizers
  •         Benchmark: Tree
  •         Benchmark: MNIST Dataset
  •         Benchmark: TSNE MNIST Dataset

  • SHOC Scalable HeterOgeneous Computing

  •         Target: OpenCL - Benchmark: Bus Speed Download
  •         Target: OpenCL - Benchmark: Bus Speed Readback
  •         Target: OpenCL - Benchmark: Max SP Flops
  •         Target: OpenCL - Benchmark: Texture Read Bandwidth
  •         Target: OpenCL - Benchmark: FFT SP
  •         Target: OpenCL - Benchmark: GEMM SGEMM_N
  •         Target: OpenCL - Benchmark: MD5 Hash
  •         Target: OpenCL - Benchmark: Reduction
  •         Target: OpenCL - Benchmark: Triad
  •         Target: OpenCL - Benchmark: S3D

  • spaCy

  • TensorFlow

  •         Device: CPU - Batch Size: 1 - Model: VGG-16
  •         Device: CPU - Batch Size: 1 - Model: ResNet-50
  •         Device: CPU - Batch Size: 1 - Model: AlexNet
  •         Device: CPU - Batch Size: 1 - Model: GoogLeNet
  •         Device: CPU - Batch Size: 16 - Model: VGG-16
  •         Device: CPU - Batch Size: 16 - Model: ResNet-50
  •         Device: CPU - Batch Size: 16 - Model: AlexNet
  •         Device: CPU - Batch Size: 16 - Model: GoogLeNet
  •         Device: CPU - Batch Size: 32 - Model: VGG-16
  •         Device: CPU - Batch Size: 32 - Model: ResNet-50
  •         Device: CPU - Batch Size: 32 - Model: AlexNet
  •         Device: CPU - Batch Size: 32 - Model: GoogLeNet
  •         Device: CPU - Batch Size: 64 - Model: VGG-16
  •         Device: CPU - Batch Size: 64 - Model: ResNet-50
  •         Device: CPU - Batch Size: 64 - Model: AlexNet
  •         Device: CPU - Batch Size: 64 - Model: GoogLeNet
  •         Device: CPU - Batch Size: 256 - Model: VGG-16
  •         Device: CPU - Batch Size: 256 - Model: ResNet-50
  •         Device: CPU - Batch Size: 256 - Model: AlexNet
  •         Device: CPU - Batch Size: 256 - Model: GoogLeNet
  •         Device: CPU - Batch Size: 512 - Model: VGG-16
  •         Device: CPU - Batch Size: 512 - Model: ResNet-50
  •         Device: CPU - Batch Size: 512 - Model: AlexNet
  •         Device: CPU - Batch Size: 512 - Model: GoogLeNet
  •         Device: GPU - Batch Size: 1 - Model: VGG-16
  •         Device: GPU - Batch Size: 1 - Model: ResNet-50
  •         Device: GPU - Batch Size: 1 - Model: AlexNet
  •         Device: GPU - Batch Size: 1 - Model: GoogLeNet
  •         Device: GPU - Batch Size: 16 - Model: VGG-16
  •         Device: GPU - Batch Size: 16 - Model: ResNet-50
  •         Device: GPU - Batch Size: 16 - Model: AlexNet
  •         Device: GPU - Batch Size: 16 - Model: GoogLeNet
  •         Device: GPU - Batch Size: 32 - Model: VGG-16
  •         Device: GPU - Batch Size: 32 - Model: ResNet-50
  •         Device: GPU - Batch Size: 32 - Model: AlexNet
  •         Device: GPU - Batch Size: 32 - Model: GoogLeNet
  •         Device: GPU - Batch Size: 64 - Model: VGG-16
  •         Device: GPU - Batch Size: 64 - Model: ResNet-50
  •         Device: GPU - Batch Size: 64 - Model: AlexNet
  •         Device: GPU - Batch Size: 64 - Model: GoogLeNet
  •         Device: GPU - Batch Size: 256 - Model: VGG-16
  •         Device: GPU - Batch Size: 256 - Model: ResNet-50
  •         Device: GPU - Batch Size: 256 - Model: AlexNet
  •         Device: GPU - Batch Size: 256 - Model: GoogLeNet
  •         Device: GPU - Batch Size: 512 - Model: VGG-16
  •         Device: GPU - Batch Size: 512 - Model: ResNet-50
  •         Device: GPU - Batch Size: 512 - Model: AlexNet
  •         Device: GPU - Batch Size: 512 - Model: GoogLeNet

  • TensorFlow Lite

  •         Model: Mobilenet Float
  •         Model: Mobilenet Quant
  •         Model: NASNet Mobile
  •         Model: SqueezeNet
  •         Model: Inception ResNet V2
  •         Model: Inception V4

  • TNN

  •         Target: CPU - Model: DenseNet
  •         Target: CPU - Model: MobileNet v2
  •         Target: CPU - Model: SqueezeNet v1.1
  •         Target: CPU - Model: SqueezeNet v2

  • Whisper.cpp

  •         Model: ggml-base.en - Input: 2016 State of the Union
  •         Model: ggml-small.en - Input: 2016 State of the Union
  •         Model: ggml-medium.en - Input: 2016 State of the Union

  • Whisperfile

  •         Model Size: Tiny
  •         Model Size: Small
  •         Model Size: Medium

  • XNNPACK

Revision History Revision History

pts/machine-learning-1.3.10     Thu, 22 Aug 2024 07:04:08 GMT
Add Whisperfile to suite.

pts/machine-learning-1.3.9     Sun, 11 Aug 2024 14:18:46 GMT
Add xnnpack to machine learning test suite.

pts/machine-learning-1.3.8     Sat, 27 Jan 2024 19:46:48 GMT
Add large language models (llm) test suite.

pts/machine-learning-1.3.7     Fri, 17 Nov 2023 12:28:41 GMT
Add PyTorch to machine learning test suite.

pts/machine-learning-1.3.6     Sun, 06 Aug 2023 17:01:03 GMT
Add whisper-cpp to test suite.

pts/machine-learning-1.3.5     Thu, 13 Oct 2022 16:46:06 GMT
Add DeepSparse to ML test suite.

pts/machine-learning-1.3.4     Fri, 07 Oct 2022 19:12:01 GMT
Add pts/spacy (spaCy) to suite.

pts/machine-learning-1.3.3     Sun, 17 Jan 2021 07:57:25 GMT
Add ONNX to machine learning syite.

pts/machine-learning-1.3.2     Thu, 14 Jan 2021 14:01:03 GMT
Add ecp-candle to suite.

pts/machine-learning-1.3.1     Thu, 08 Oct 2020 09:23:48 GMT
Add OpenVINO to suite.

pts/machine-learning-1.3.0     Sun, 04 Oct 2020 13:02:02 GMT
Add OpenCV, TNN, Caffe, and other updates.

pts/machine-learning-1.2.10     Tue, 22 Sep 2020 18:11:26 GMT
Add OpenCV DNN test to machine-learning suite.

pts/machine-learning-1.2.9     Sat, 19 Sep 2020 12:54:40 GMT
Add NCNN to test suite.

pts/machine-learning-1.2.8     Thu, 17 Sep 2020 20:58:30 GMT
Add Alibaba Mobile Neural Network (mnn) test profile.

pts/machine-learning-1.2.7     Sun, 23 Aug 2020 14:17:27 GMT
Add tensorflow-lite test profile.

pts/machine-learning-1.2.6     Wed, 08 Jul 2020 14:28:35 GMT
Add ai-benchmark test profile to machine learning test suite.

pts/machine-learning-1.2.5     Wed, 17 Jun 2020 16:35:07 GMT
Use pts/onednn rather than pts/mkl-dnn due to rename.

pts/machine-learning-1.2.4     Thu, 28 May 2020 15:51:26 GMT
Add additional tests.

pts/machine-learning-1.2.3     Wed, 08 Apr 2020 16:21:57 GMT
Add tensorflow.

pts/machine-learning-1.2.2     Wed, 08 Apr 2020 14:00:00 GMT
Add numenta-nab and deepspeech to test suite.

pts/machine-learning-1.2.1     Mon, 24 Feb 2020 09:14:07 GMT
Update machine-learning test suite with batch mode for mlpack given its new options just added.

pts/machine-learning-1.2.0     Sun, 16 Feb 2020 19:07:13 GMT
Add more tests.

pts/machine-learning-1.1.0     Fri, 10 May 2019 15:47:23 GMT
Update tests.

pts/machine-learning-1.0.0     Mon, 01 Aug 2016 16:30:43 GMT
Initial commit