I learn something new every day. Today I have learned about 2 more quirks with soft cloth.  Maybe someone already knows that but...

1. Did you know that even if you have disabled a soft cloth bake, and you do "Remove object animation" from a character, the very first frame of the soft cloth is still getting baked? This finally answers my question as to why skirt/dress is left behind sometimes when you transform a character at the beginning of the clip (while soft bake is off).

2. This one is even more weird. If you need to start an animation off with a pose far different from the default (lets say a sitting pose), and then later your character stands up, the soft cloth (a skirt in my case) is getting deformed for the rest of the clip without a chance to recover. I guess it matters how the physics gets initialized. But then you always need to start with a standing pose and cut final clip at the beginning if necessary (unless someone knows how to overcome this).
BTW, simulating with By Frame mode, if you wonder, does not change anything.

Here is a fresh project with a default character, non modified cloth and soft bake disabled. Shows both quirks.