Inference Latency on Edge Devices
How much can inference speed be improved via quantization + operator fusion on Raspberry Pi?
Status
Publication
Personal Research Journal
Date
August 15, 2024
Authors
1

Abstract
This research investigates methods to improve deep learning inference speed on resource-constrained edge devices. We explore quantization techniques (FP32→INT8), operator fusion, and compare ARM CPU versus VideoCore GPU acceleration paths. Results show 2.4x speedups with quantized models while maintaining accuracy within acceptable thresholds.
- 1.Benchmarked FP32 vs INT8 performance
- 2.Measured thermal throttling under sustained inference
- 3.Compared ARM CPU vs VideoCore GPU paths
- ✓Quantized models improved speed by ~2.4x
- ✓Thermal ceiling reached faster on GPU
- ✓CPU path more stable but slower
- →Testing vendor-specific acceleration
- →Trying binary neural networks
Keywords
References
Jacob et al. (2018). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR.
Howard et al. (2019). Searching for MobileNetV3. ICCV.
Tan & Le (2021). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML.