Any advice on how to deal with sleep deprivation most effectively?
I have pretty serious sleep deprivation on a regular basis and have developed some techniques for dealing with it. Unfortunately, none of them are very good for you. For one, I don't eat much. Keeps my stress hormones high, and the constant state of mild hunger keeps me more alert. A long the same lines I make sure not to dress warmly so as to be constantly slightly cold. It's best if I'm almost shivering. Once again it just helps keep me more alert and aware. I also try to shower mid-day instead of in the morning. Since showering gives me an alertness boost for a few hours. If I'm getting tired, I do some pushups, take a brisk walk around the block or up the stairs, or try to get in a friendly debate, which generally wakes me back up. Keeping busy in general is good. I also try to sit down as little as possible and stay in brightly lit areas. Going outside periodically helps as well. If I'm really serious about it I'll also drink more water than usual so I have to hit the restroom more often. The feeling of needing to pee and the frequent trips to the brightly lit bathroom environment help. They're little things but taken together they make a pretty big difference, for me at least.
Also, caffeine.
This isn't the advice you're looking for, but, many of the things you mention combine to make a recipe for insomnia. Now I don't know the context of your situation or how the problem started. But it looks like, intentionally or not, you have conditioned your body and mind such that when you feel fatigued or ready for sleep, you force yourself awake. Therefore when you actually try to go to sleep, your system responds by keeping you alert and awake. That's what you are training your mind and body to do.
In another post, you mentioned you had good sleep hygiene, but what you say here is in direct contradiction. Cold helps you go to sleep at bedtime. Skipping a meal before bed will help reset your circadian rhythm in one day - but every day? Caffeine carries it's own whole set of issues.
It bears mentioning that you can use the same classical conditioning to train yourself to sleep.