What you are hearing is not the wastegate. You will never hear the wastegate. The wastegate is on the exhaust side of the turbo and routes exhaust gasses around, bypassing the turbine wheel to prevent over spooling. What you are hearing is because of the RPM drop, the engine itself is not taking as much air, so there is excess air in the intake system, and the noise is just the excess air backing up through the compressor wheel. The noise when it shifts and when you let off the throttle isn't really harmful. Surge, however, can cause serious turbo damage, from bending the compressor fins backwards to snapping the shaft. You will hear surge when you have the engine under load, and you will hear kind of a rapid "ch ch ch ch ch". It sounds like the noise you are already hearing, only does it constantly and rapidly.