Actually the only thing that has to come off is the knob that controls the lumbar. You can either pull it off with your fingers (takes a lot of pressure) or you can take a long pair of needle nose pliers and get to it. To do so, push the seat cushion back a bit on top of the panel and you should be able to see down behind the panel. If you put the pliers around that (sround the shaft and pry against the peice of the knob that slides on to the shaft) and start to gently pry against the seat (bring the nose of the pliers toward the door) then it should pop right off.
The panel will then pop off once you release the tabs holding it on. You can reach these from going behind the panel on top and bottom (they can be a pain).
The power seat control switch will stay on. It unplugs from the back once you get the panel off. The recline handle will stay on the seat as well. The panels just slides out from around it.
Once the panel is off, you can then see the screws that hold the actual trim panel onto the seat. There are also 2 push pins holding it on, one on the front and one on the back. Just take your time and be patient and it comes off pretty easliy.
Sorry for the long post, I tried to be detailed as possible. If you need more info let me know. I can try to get some pics if you need. Good luck!!
BTW: The panel brackets on my driver seat are also broken from getting in and out of the truck. I even us my factory step bars.