Regarding 64GiB RAM being too low, I suspect you might be right. Perhaps going up to 96GiB or 128GiB would be better? The total raw storage in the cluster will be 1,344TiB, but with Erasure Coding/redundancy, usable storage should fall just below the petabyte level. I’m keen to minimize buying any new hardware but if 64GiB per node just isn’t feasible, then I’m fine with spending to get to the bare minimum goal of saturating that 1 GbE link.
Speaking of which, regarding saturating the 1GbE link, that’s good to know. So long as I can do that, that’s all that really matters performance-wise.
Regarding “it should be an odd number to avoid split brain”, does the “it” in your statement mean the number of OSD nodes, the number of Manager nodes or the number of Monitor nodes?
In your experience, what have you found to be the bare minimum? a 4c/8t CPU at 3.5Ghz does indeed sound a bit undersized for 24 HDD-based OSDs, so I’d be curious to read what others are running.