Hi @jo_haemer. Great question!
I wouldn't think it has anything to do with the adularescence (sometimes called schiller). The schiller is more of an optical effect than anything else, though it can be exacerbated by the presence of other inclusions. Labradorite is pretty hard (6.5 I think, maybe less), so I think it is less about the hardness and much more about cleavage. It seems there is disagreement across multiple reliable sources about whether the cleavage is perfect in one direction or two, but I'm not sure that matters for our purposes.
Long story short - it's delicate and it cracks easily (I suspect) because of cleavage, not hardness or adularescence.
Perhaps someone who knows more than me can chime in!