Why does my foot hurt when I take my aircast walking boot off to rest or shower?

I have a stress fracture in my 5th metatarsal. I am wearing a walking boot most of the time without using any crutches (dr. did not say I needed them).I take the boot off to sleep, shower, and to drive. When I first take the boot off, my foot is really sore/painful. Is this normal? Any advice would be appreciated.