That's pretty much it. There are some rafts, like the Aire that you mentioned that actually row very well, but in general it is difficult to tread water in them, let alone row them back upstream. Anchoring is a challenge. They actually tend to track fairly well, but get pushed around in turbulence.
Yes, way better in low/rocky water. Sliding over rocks is so much nicer than banging into them. Safer is in the hands of the rower though. I've seen rafts swamped on structure that I'm pretty sure a hard boat would have spun off of. The low sides can be a liability as well. I often ride in my driftboat without a lifevest (dumb I know), but you wouldn't catch me in a raft without one. One mis-step and you're over the side.
Lastly, storage and overall floor space will be lower than in your hard boat, so you need to plan out how much stuff to bring and where to store it.