You should check your eddy-viscosity field to see if it looks physical. On low-Re flows (some types of rotating cavities) we've had problems with discontinuities in the eddy-viscosity when using the Realizable k-epsilon - see my post below entitled "Advanced Turbulence Modeling in Fluent, Realizable k-epsilon Model" for further details.

1. Your explanations were very helpful.

2. What I still find confusing is the use of u* in the Fluent User's Guide as s.th. different from u-tau. It is because, as far as I know, u* has already been introduced as a synonym for u-tau (i.e. the same, e.g Tennekes, A first course in Turbulence, p.53). Am I right?

Tennekes & Lumley's definiton of u* is not the same as Fluent's (see Sung-Eun's post). Fluent's definition of u* does not involve the wall shear-stress - it is a pure turbulent velocity scale independent of wall-properties. Fluent calls T&L's u* for u_tau, the real name is "friction velocity". It's confusing, I agree.

I'm afraid that I don't have a straightforward concise answer to your question. Please bear with me if things get too wordy.

The Wolfstein's one-equation is a wall-distance based, "near-wall", low-Re model, inasmuch as it is based on Re_y as turbulence parameter where y is the distance to the wall. It's a great, practical turbulence model that I've frequently seen beats up most of the contemporary two-equation low-Re models, in terms of accuracy, let alone robustnerss. But it would be least ideal to represent the low-Re effects, say, in a pocket of transitional flows far from the wall, where Re_y may be big enough (the model then reduces to a high-Re model) , but the flow is in reality transitional. In those regions, Re_t (k*k/epsilon), which is a more relevant turbulent Reynolds number, is not necessarily large enough.

If Re_y is smaller than 200 in most of your domain including the region where the mixing happens vigorously, it's not an ideal situation for the two-layer model either because you're then using effectively one-equation model that doesn't account for the transport of one of the important turbulent quantities (epsilon) in the flow where non-equilibrium effects are presumably significant.

So, If I am to follow the common wisdom alluded to above, I'll tend toward one of the low-Re models FLUENT offers underhand. But I have no hard evidence to support the recommendation.

And what about the RNG-differential viscosity model which should form my point of view also address the problem?