Research Entry

Inference Latency on Edge Devices

How much can inference speed be improved via quantization + operator fusion on Raspberry Pi?

Status

ongoing

Publication

Personal Research Journal

Date

August 15, 2024

Authors

1

Inference Latency on Edge Devices

Abstract

This research investigates methods to improve deep learning inference speed on resource-constrained edge devices. We explore quantization techniques (FP32→INT8), operator fusion, and compare ARM CPU versus VideoCore GPU acceleration paths. Results show 2.4x speedups with quantized models while maintaining accuracy within acceptable thresholds.

Motivation
Real-time edge inference enables offline intelligent devices.
Experiments
  • 1.Benchmarked FP32 vs INT8 performance
  • 2.Measured thermal throttling under sustained inference
  • 3.Compared ARM CPU vs VideoCore GPU paths
Findings
  • ✓Quantized models improved speed by ~2.4x
  • ✓Thermal ceiling reached faster on GPU
  • ✓CPU path more stable but slower
Future Inquiries
  • →Testing vendor-specific acceleration
  • →Trying binary neural networks

Keywords

edge computing
performance optimization
model quantization
hardware acceleration
embedded systems

References

Jacob et al. (2018). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR.

Howard et al. (2019). Searching for MobileNetV3. ICCV.

Tan & Le (2021). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML.