Erasure support

[ciroiriarte]

Hi!, is erasure supported on the latest release of Nodeweaver?

[cdaffara]

Yes, erasure coding is one of the possible storage goals, along with HDD2 and SSD2 replication goals. Goals can be changed live, while the VM is running.

[penzoiders]

here’s a list of all Erasure Coding available goals:

7	TIER_EC21	TIER_EC21: $ec(2,1) {SSD SSD HDD}
8	TIER_EC22	TIER_EC22: $ec(2,2) {SSD SSD HDD HDD}

10	SSD_EC21	SSD_EC21: $ec(2,1) {SSD SSD SSD}
11	SSD_EC22	SSD_EC22: $ec(2,2) {SSD SSD SSD SSD}

14	HDD_EC21	HDD_EC21: $ec(2,1) {HDD HDD HDD}
15	HDD_EC22	HDD_EC22: $ec(2,2) {HDD HDD HDD HDD}

17	EC21	       EC21: $ec(2,1) {_ _ _}
18	EC22	       EC22: $ec(2,2) {_ _ _ _}

26	HDD_EC32	HDD_EC32: $ec(3,2) {HDD HDD HDD HDD HDD}
27	HDD_EC52	HDD_EC52: $ec(5,2) {HDD HDD HDD HDD HDD HDD HDD}

[ciroiriarte]

thanks for the quick answers!

[ciroiriarte]

Does it mean I cannot configure EC 5+2 or EC 4+2 with tiering?. Caching is handled differently?.

For example, with 2 x SSD drives and 4 x HDD, should I just use EC + caching and forget about tiering?

[penzoiders]

There is no caching (except from in-ram ARC cache from zfs volumes under the DFS), there’s only tiering.

Lower latency reads are preferred, writes get ACK only when all replicas are committed to nodes: if you opt for mixed media goals you’ll get writes as slow as the HDD reads as fast as SSD.

Following my 2 cents:

In real world scenarios we saw very few people uses EC goals, the preferred ones are SSD_2, HDD_2 and TIER_BASE (the faster, the cheaper and the I can’t afford full SSD just speed up my reads), someone use higher replica leverls {HDD/SSD}_3 for veery critical storage to survive multiple and simultaneous node failures… Anyway EC goals may be useful to increase resiliency w/o wasting storage if you have spare CPU cycles to trash away. EC replicas are an appreciated feature by the analysts but it’s something that the hardcore sysadmin don’t care about.

[penzoiders]

and to complete the reply: there are no user-defined goal levels, NodeWeaver provides a number of options (eventually a sensed feature request may be taken into account)

[ciroiriarte]

Thanks, what would be a sensible option to configure it, expecting EC & tiering with the following hardware available?:

2 x 0,9TB SSD
6 x 2TB HDD

like in “I was happy with Raid6 + storage tiering in the past” and “user probably won’t be saavy enough to customize per disk goals”

[penzoiders]

TL:DR --> no, for EC4+2/5+2 you need more hosts, no matter number of disks, and tiered EC4+2/5+2 are not generally available as of now to all users.

Long version:
replicas should be considered per NODE, not per DISK.
Will fallback on disks only if no other option is available (but that’s not the way to guarantee data integrity in an hyperconverged system designed as NodeWeaver)

For EC 5+2 you need 7 hosts, not 7 disks:
example: 5 nodes will hold a parity chunk, 2 nodes will hold the full chunk

For EC 5+2 on 4 hosts you won’t have enough resiliency:
example: on 3 nodes you’ll have 2 chunks (or 1 parity chunk + 1 chunk, or 2 parity chunks), and on 1 node you’ll have 1 chunk (1 chunk or 1 parity chunk)
the nodes with 2 chunks will place them on different chunkserver (one on HDD one on SSD ) but that doesn’t matter because, if that node dies, you’ll lose 2 replicas with the death of a single node and therefore 5+2 will do same of 2+2 in a 4 nodes scenario.
It’s just saving you a bit more if a single disks fails, but the price for overhead is not worthy.

That said custom tiering erasure coding are possible, but custom configurations are reserved to clusters with more than 5 nodes opting for 24/7 support and should be requested through ticket, otherwise will have to stick to proposed goal config.

no matter how many drives you have ssd or hdd, a single node exposes 4 types of chunkserver SSD, HDD, SSDCRYPT, HDDCRYPT (all disks fall in the relative media label and have a dynamic encrypted partition available for the encrypted goals).

in theory a tiered 5+2 EC goal may look like this (or similar to this depending on the media placement % balance)

TIER_EC52	TIER_EC52: $ec(5,2) {SSD SSD SSD HDD HDD HDD HDD}

to have a valid goal you still need to have 7 hosts, anything less will be accomplished only in “emergency mode” when strict goal media labels are not respected (this happens when a user fills too much a certain label SSD/HDD, NodeWeaver will automatically start balancing on the others), not by chance is called “emergency mode” : it’s something that should not be performed by default.

[ciroiriarte]

By the way, recalled a conversation where Carlo mentioned the use of EnhanceIO for cache, was that valid for older versions?

[ciroiriarte]

SSD SSD SSD HDD HDD HDD HDD

Shouldn’t this be:
HDD HDD HDD HDD HDD SSD SSD ?

So, for a small 2 node cluster, the only sensible option would be replica = 2?. I’m still puzzled by the no cache statement, as I recall having read about EnhanceIO in Nodeweaver.

[penzoiders]

that EC was an example, can be anyway. main fact is it’s multi host EC not multi DISK.

EnancheIO (present up to 10.3.2) has been removed in favor of media placement/tiering that has proved to be more flexible. EnancheIO was doing great until you fitted in the SSD space then there was a drop in performance when you passed the limit of your flash space usage. Was also difficut to manage with disk swapping.

Now in version 11.x every disk is indipendent, you can swap disk on the fly without doing anything but watching it replicating. All is automated as long as your controller supports HBA/passthrough mode and hot attach.

You can still use cache techniques on the VM level (some users define enancheIO on SSD_2 in front of EC or HDD goals)

[penzoiders]

and yes, with 2 nodes cluster numeric goals only make sense SSD_2, HDD_2, …

[ciroiriarte]

Thanks for the information, I couldn’t find any release notes or changelog with those comments.

I’m thinking about what should the end user see . Probably explicitly choose “SSD/Fast tier” and "Mechanical/Normal tier) according to it’s usecase. Performance-wise , the 10% SSD rule of thumb doesn’t make sense anymore, and probably neither going with 7.2K RPM NL-SAS or SATA for the mechanical disks…

[penzoiders]

10% SSD rule was ok for enancheio.

Now good rule is cold storage only (backups, archive, rarely accessed filesystems) on HDD, rest on SSD (System disks, VDI, dbms, erp…)

[cdaffara]

You can also use a caching mechanism inside of the VM (for example, EnhanceIO in Linux) by using an HDD2 virtual disk and an SSD2 one that is used for the caching tier; this way, its use is more tuned to the IO pattern of that single VM.
Some caching format may reappear in version 12, designed for specific verticals like VDI made from imported disk images, where you may have 2TB of virtual disk since it comes from a physical device, but makes no sense to have all of it as SSD.