Oil in the CAC, the boost tubes, and boots are a consequence of design - not a condition of it - and the oil does affect efficiency of the intercooler's heat transfer capacity. You do not need to replace your intercooler even if it has a significant quantity of oil throughout though unless you also have blown out the ends (not typical).
Positive Crankcase Ventilation (PCV) is a requirement both environmentally and performance-wise. Without ventilating crankcase gasses, engine seals would rupture and spill the entire contents of your oil system. Environmentally speaking, PCV provides relief for the gasses that blow-by piston rings to burn off in the combustion chamber where they escaped in a previous compression-combustion cycle.
An unintended consequence of PCV is that oil vapor is also picked up from the crankcase in addition to the combustion chamber blow-by gasses. This oil vapor adheres to every surface it comes into contact with beginning at the breather outlet inside the turbo compressor intake (it looks like a little hollow spear).
The oil vapor does react with silicone and breaks down its pliability resulting in splitting and either slipping the boots from the boost tubes or turbo compressor outlet or rupture frequenly on the lower connection between the intercooler and the inlet boost tube. As oil does not have as good of heat exchange as other fluids (air and coolant among others) it reduces the efficiency of the intercooler (it still works just not as well...it's not a significant atrophy though). Nevertheless, the elimination of this oil vapor will allow better heat exchange in the intercooler and avoid boot blowoff or boot ruptures.
The way to maintain PCV and eliminate the oil vapor is with inline filtration with a CCV reroute. Some would ventilate the crankcase vapors to atmosphere but a small differential in pressure will turn the system into a oil dispenser...only effective when attempting to elude capture by forces of evil intent (perhaps zombies chasing you down). Keeping the system in a closed-loop with a filter maintains the pressure but eliminates the tiny oil droplets in the vapor. Certainly the filter must be drained periodically as the oil accumulates; this is simple enough with the addition of a drain valve on the bottom of the filter body or through exchange of the filter media.
Both Stealth-Automotive.com and DieselPerformance.com carry CCV reroute kits with inline filtration to maintain proper PCV in a closed-loop system to comply with federal emissions standards (and avoid the potential for catastrophic oil loss with a pressure differential when ventilated to atmosphere). Use of inline filtration will immediately begin to allow oil within the intercooler and boost tubes to evaporate. If you wish to accelerate this process you can remove your intercooler and take it to a radiator shop for steam cleaning. Things will begin to clear up immediately though. I have the Stealth-Automotive.com kit and think it is very good.
It's a good modification but won't provide much performance increase if that is your primary consideration. A coolant filter and gauges are a more worthwhile investment as is proper maintenance of your coolant or flush and exchange to a better coolant (and then proper maintenance as well).
Hope this is helpful.