Hello everyone!
I hope all has been well here in the Granite group!
Recently I've been pushing myself to get tangible metrics from my experiments running Granite-4.0-H-350M locally via Ollama. The quantization used is Q8_0.
I tested Granite 350m on 10 prompts, 10 times each with 2 dummy prompts each run. Metrics from Ollama logged, as well as thermal metrics for each run logged in JSON files in the GitHub repo!
Yes, it is a simple REPL style manual benchmark, but for the time being, it's suitable. Automated scripts to benchmark weren't working out too well due to the Pi's limitations. I do plan on addressing this with an automated benchmark in the future!
Given the CPU-only nature of the Raspberry Pi 5, thermal throttling issues are quite common when running larger models on a RPi 5. For reference, around 85c will activate throttling.
Summary of metrics:
| Valid Queries (excluding 20 dummies) |
100 |
| Overall Accuracy (Strict) |
47.0% |
| Overall Accuracy (w/ Partial Credit) |
50.0% |
Throughput:
Hardware Metrics:
| Avg CPU Temperature Rise |
+5.31°C |
| Std Dev (Temp Rise) |
1.50°C |
| Max Temperature Rise |
+9.4°C |
Question Categories & Accuracy
Category, Total, PASS/FAIL, Accuracy (Strict), Accuracy (Partial)
| Basic Intent Routing |
10 |
10 |
0 |
100%, |
100% |
| Agentic Routing – Tool Selection |
10 |
10 |
0 |
100% |
100% |
| Grounded Claim Status |
10 |
10 |
0 |
100% |
100% |
| Clean JSON Output |
9 |
9 |
0 |
100% |
100%
|
| Domain-Specific Classification |
5 |
4 |
1 |
80% |
80% |
| Harmful Request Detection |
10 |
1 |
9 |
10% |
10% |
| Priority + Category Labelling |
10 |
0 |
10 |
0% |
0% |
| Workflow Classification |
10 |
0 |
10 |
0% |
0% |
| Structured Validation |
10 |
0 |
10 |
0% |
0% |
| Language Detection + Issue Type |
10 |
0 |
10 |
0% |
0% |
High-Performing Categories (≥80% accuracy)
1. Basic Intent Routing (100%)
2. Agentic Routing – Tool Selection Only (100%)
3. Grounded Claim Status – Evidence Extraction (100%)
4. Clean JSON Output (100%)
5. Domain-Specific Classification (80%)
Low-Performing Categories (<15% accuracy)
6. Harmful Request Detection (10%)
7. Priority + Category Labelling (0%)
8. Workflow Classification (0%)
9. Structured Validation – Missing Fields (0%)
10. Language Detection + Issue Type (0%)
---
Full GitHub Repository with all eval logs: https://github.com/Jewelzufo/granite4-rpi5-benchmark
I plan on automating these benchmarks and testing this same model on many other domains for edge AI tasks. The results are pretty decent considering the model is running on the bare minimum. I'm impressed with Granite-4.0-H-350M!
Respectfully,
#LLM------------------------------
Julian Gonzalez
Generative AI Engineer
CreativeAct Technologies
Orlando, FL
------------------------------