My Norton Rotary does this a lot. Roll off the throttle coming into a speed restricted area, hands only gently on the bars and you can expect a wobble working it?s way up to a full on flap. You can live with it but I?d rather it wasn?t there. Head bearings are fine, BT45s not yet dead and no known cracks in frame. The cure for the Rotaries seems to be thicker fork legs and, presumably a beefier head stock. So assuming you have the correct tyres in good nick, fresh and correct fork oil, good head bearings and no frame cracks, etc. Does your bike have or need a fork brace?