Ceph works best if you have identical osd, quantity, type and capacity across the cluster, also works best on a 3+ node cluster.
I ran a mixed sata SSD/HDD 256gb/4tb cluster and it was always a bit pants. Now I have 7x1tb SSD per node (4nodes) and it works fantastic now.
Proxmox uses replica 3/2 failure at host level but you may find that EC works better for your mixed infra as you noticed you can’t meed the 3 host failure and so setting to osd failure level means data may be kept on a single host so would need to traverse the network to the other machine.
You may also need more than a single 10Gb nic too as you might start hitting bandwidth issues.
Ceph works best if you have identical osd, quantity, type and capacity across the cluster, also works best on a 3+ node cluster.
I ran a mixed sata SSD/HDD 256gb/4tb cluster and it was always a bit pants. Now I have 7x1tb SSD per node (4nodes) and it works fantastic now.
Proxmox uses replica 3/2 failure at host level but you may find that EC works better for your mixed infra as you noticed you can’t meed the 3 host failure and so setting to osd failure level means data may be kept on a single host so would need to traverse the network to the other machine.
You may also need more than a single 10Gb nic too as you might start hitting bandwidth issues.