Repository navigation
Add CPU & MEM composite score? #393
Description
Just wondering if it may be worth to add a third column "CPU & MEM". So you can see languages that are best at both CPU & MEM?
Score calculated as 0.5 * (lang_cpu/all_lang_median_cpu + lagn_mem/all_lang_median_mem) the lower the better.
So you can easily spot problem with language like "Racket" that has C-like CPU time but consumes x100 more memory.
Feel free to close ticket if not interested :)
Reacted by ITwrx
Activity
[-]Add CPU & MEM score?[/-][+]Add CPU & MEM composite score?[/+]nuald commentedon Feb 18, 2022
The metrics (not formula though) you've described fits the value we already have - Energy. For Intel RAPL (https://blog.chih.me/read-cpu-power-with-RAPL.html) the whole (package) energy consumption is based on CPU (PP0), GPU (PP1) and DRAM usage. However, the current benchmarks results have PP0 (CPU) only as my previous server that I've used for the benchmarks has an old Intel CPU that doesn't provide additional RAPL data, and I've implemented PP0 reading only.
However, I've just tried rapl-read (https://web.eece.maine.edu/~vweaver/projects/rapl/index.html), and my current server has all the data:
So I guess now I can add the relevant data, but it would require some time to implement the proper solution (the current one, based on powercap, doesn't provide the package data, only PP0 and PP1: https://www.kernel.org/doc/html/latest/power/powercap/powercap.html ). As always, all contributions are welcome.
nuald commentedon Feb 18, 2022
Please dismiss my notes about current implementation not supporting whole package. It reads
/sys/class/powercap/intel-rapl/intel-rapl:0/energy_ujthat is the whole package (with PP0 etc are within that directory, e.g./sys/class/powercap/intel-rapl/intel-rapl:0/intel-rapl:0:0/energy_uj). I just need to update the README as it still has the old info.So, you already have the metrics available. As for the given example, RAPL metrics are not quite accurate on short time intervals (as it could be some external factors involved). If you take a look to mandel.b tests, you could see that tests which are slower than Racket's could consume less energy (because use less DRAM), therefore CPU&MEM-wise are little bit better. It's not straightforward correlation like in your formula (as it depends on the DRAM and CPU models), but more realistic. I guess real server platforms (as an opposite to my laptop "server") could give even better numbers (like you could have in data centers), but I don't have any, sorry.
Another note - bf-tests are CPU intensive, not memory. If you take a look into other tests, like Primes, you could the that memory usage indeed affects the energy consumption:
Ruby (--jit)runs in 1.9 secs and use 418Mb total with energy consumption 31JRuby/trufflerubyruns in 2 secs and use 694Mb total with energy consumption 52Jal6x commentedon Feb 19, 2022
Energy could be statistically correlated, but it's not looks like it's the same metric. Example, from the "Mandel in Brainfuck" table.
Difference between C/gcc and Racket - energy 1.4, memory 105. Not just slightly different, two orders of magnitude difference.
nuald commentedon Feb 23, 2022
Right, energy consumption doesn't necessary correlate to RAM usage (I guess the correlation could be caused by more CPU work that could be unaccounted for as we track only the benchmarked process, but not, for example, kernel).
I'd like to note two things though:
I think I can store the raw results though in the version control, and additional scripts could analyze them with any metrics wanted. That would require some work, and I'm not sure when I can implement that. All PRs are welcomed as usually.