I did some benchmarks between sparse files on GFS2 versus DRBD on top of LVM2 thin-provisioning. I thought it would be worth setting up DRBD mirrors on top of LVM2 volumes to gain some performance over a shared-disk file-system. But I couldn’t scale-up the machines to the point where I got to notice a difference. So far I’ve tested, and eventhough there’s still an additional layer of file-system, GFS2 scales as well, if not better than a thin-provioned LVM.
See Bonnie++ on steroids. As a result, drbdtop shows
DRBDTOP (kernel: 9.0.24; utils: 9.13.0; host: pro5s1)
┌◉ (LIVE UPDATING) Resource List
│ Name | Role | Disks | Peer Disks | Connections | Overall | Quorum
│ res-data | Primary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack1 | Secondary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack2 | Secondary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack3 | Secondary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack4 | Secondary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack5 | Primary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack6 | Primary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack7 | Primary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack8 | Primary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack9 | Primary | ✓ | ✓ | ✓ | ✓ | ✓
DRBDTOP (kernel: 9.0.24; utils: 9.13.0; host: pro5s2)
┌◉ (LIVE UPDATING) Resource List
│ Name | Role | Disks | Peer Disks | Connections | Overall | Quorum
│ res-data | Primary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack1 | Primary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack2 | Primary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack3 | Primary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack4 | Primary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack5 | Secondary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack6 | Secondary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack7 | Secondary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack8 | Secondary | ✓ | ✓ | ✓ | ✓ | ✓
│ res-slack9 | Secondary | ✓ | ✓ | ✓ | ✓ | ✓
Just a LUN either as a GFS2 sparse file vs thin-provisioned LV. Test was done one after another.
See raw results as text or html.
Same as run1 but here test was done concurrently (gfs2 + thindrbd at the same time), and against 9 mount points each.
See raw results as text or html.
Same as run2 but every bonnie++ test was processed 5 times with (-x 5).
See raw results as text or html.
As virtual disks are tremendously easier to setup and maintain using files instead of block-devices, it is clear to me that GFS2 is the way to go, just like what VMware is doing for ages with their VMFS. Unfortunately those prevent to benchmark their products in their EULA, not mentioning it is proprietary software anyway. So next time we will play with GFS2 vs. OCFS2.
Update Oct 2020: something I did not mention and maybe oversaw is that DLM over DRBD volumes is dangerous and one need to take very good care of HA issues to have it up and running securely (considering fencing). A stupid trick I did so far to avoid the fencing pain is to NOT have dual primaries nor dual-mounted GFS2, and define the optimal run-time with a dedicated active volume per node, while only enabling dual-primaries and dual-mounted GFS2 during hot-migrations. Besides, when ever HA would be necessary while a node crashes, those are NOT setup to automatically make the volumes active nor mount any file-systems upon reboot, so one can peacefully solve the split-brain situation manually.