So, you find yourself On-Call

So, you find yourself on-call. Your life was just interrupted. You realize it is an issue that you ‘have to’ deal with. Are you ready? What are you missing to accomplish your task? In the heat of the moment, you may not have everything on hand that is needed, much less recall all the needed items. I will create a list of items (both physical and electronic), thoughts, ideas, etc. that you may need to successfully complete an on-call rotation. Every environment and company culture will be different, and some of these things may not apply. However, the purpose of this article is to get your creative juices flowing and list out the things you will need to have on-hand when you are on-call, so you are not caught unawares.

When it’s your turn to be on-call, you need to make sure that you have all the needed items handy to accomplish your task. Some of these items, which are ever present at your office, may not be present when you are away from the office. Connectivity can become an issue. Lists of information you reference, that exist on your network, may be inaccessible remotely. Your printed list of favorite phone numbers stuck to the wall by your phone, no longer present when you are away. What about mail access? Do you have it on your phone/blackberry? What about on the web? Have you practiced getting your mail from the multiple locations? Have you practiced gaining successful access to all needed servers remotely?

As I prepare to go on-call, I sit in the office and think of all the items I could use to help me out. I’ll list these out now and explain them as we go. Some of these items are physical items you need to carry with you. Others will be processes that you need to practice. Practice them at the office, but more importantly, you need to practice them remotely, to ensure you can successfully complete them.

Key fob
If you have any security processes you must go through to gain access to your network, make sure that you bring these with you. It won’t do to report to your boss that you left it at the office and cannot fix problem X until tomorrow.

VPN or other secure access
Make sure you practice this connectivity from home. On as many computers as you can. From as many locations as you can. Once I found myself unable to connect at home to the internet. I ended up hijacking the wireless signal of a neighbor, sitting on their steps and connecting. I’m glad that I had tested it on the laptop I was using, so that the scene of me sitting next door troubleshooting my database wasn’t any more stressful than it needed to be.

Laptop
If you have a company issued laptop, make sure you have the means to carry it with you to all your favorite haunts. I purchased a waterproof bag for mine so that I can take it on a boat. I have a small messenger bag that I use to quickly haul the laptop around in my car. This bag doesn’t have any of the excess items I normally carry with me to and from work. Having multiple means of transporting this laptop makes it easier for me to bring it along wherever I go.

I also configure another machine or 2 from home to be able to connect. My personal laptop or desktop are configured, in case my work laptop is not present. I practice often connecting with these machines to ensure success. Just last night I connected with my personal laptop, even though my work laptop was with me. I just needed to know I could do it.

List of Servers
What are all the servers you need to access? Do you have them all memorized in your head? If not, you may want to simply jot these down somewhere to help you remember them. I will often have a list of these servers in a spreadsheet with other pertinent information that is associated with them. But do I remember to print it out and carry with me?
Document Email Errors
We all have our favorite errors and solutions. Do you have these documented? Plans of actions can be created for each distinct type of error you may encounter. Having these documented in an organized way will help you determine the steps you need to take when you encounter these errors. I went thru my emails, and generalized them, grouped them, and stuck them into a spreadsheet with possible solutions. As I encounter new ones, I plan on adding them into this spreadsheet. But I need to make sure that I have these in an accessible location when off-site.

Document On-call Process
In a general sense, having a laundry list of steps that ‘should’ be taken when an error occurs will help you stay focused. Check the cpu, check the disk space, check i/o. Check whatever you normally check, document it and make a check list out of it. Follow it.

Cell Phone
Make sure the right people have your number, and that you have your phone with you. Use your landline while at home. Use the speaker phone feature to become hands free. Use an earpiece to become hands free. Just think of all the things you need to easily use your cell phone and communicate. Ever run out of battery on your phone? Keep a charging cord handy too.

Phone Numbers (boss, coworker, helpdesk, Networking)
Another list. Gather all the important people you may need to communicate with. Make sure you have all their numbers, and multiple ways to communicate with them, quickly and efficiently.

Email Addresses
Do you know who runs your helpdesk? Could you email the helpdesk alias? Or even an individual on the helpdesk? Do you have personal email addresses to people, in case their work email is not functioning? Any and all email addresses, in case you cannot reach your company directory. Make sure you get a list.

Remote Desktop Shortcuts
I always create multiple shortcuts of Remote Desktop links to my favorite servers. Then I do not have to remember or reference the server list I mentioned earlier. I can simply click on it, and connect, assuming my connectivity to the office has been established. Streamlining this process reduces my stress. Ensuring that its accessible from without the offices, while I am remote is critical.

Access to all relevant files at home and away
Often, I will find that a favorite file that I reference from work is not accessible from home. I need to have a copy of these local, so that I can reference it easily.

Wireless access – where can you connect from?
If you are lucky enough to have wireless access, have you tested it out? Can you connect from your bedroom, downstairs in the family room, how about in the car on the freeway, or near your favorite restaurant? Have you tried to connect from your favorite locations? It is important that you can connect, know how to connect, and don’t fumble with it when the time is at hand. Practice, practice, practice!

Webmail access
You may have a mail client on your work station. You may have it on your laptop. You may get mail on your blackberry/phone. But what if you are unable to use these devices? Can you get to your inbox from the web? Does your office email exist on a web based solution? Have you practiced connecting?

Escalation process
Document your escalation process. Ensure that you have a copy of it. Know it by heart.
Do whatever it takes to be able to successfully navigate it while away from your desk. There is nothing like being at dinner, getting that email, and not quite knowing what your next step is. If it’s documented, and known, and widely used by yourself and others, there is less room for error. Know who you need to escalate different issues too. When should you call your boss? When should you call a coworker? When should you escalate to the Helpdesk?

Learn from others
There is mostly likely someone at your office that knows more than you. Lean on this person to learn as much as you can about solving issues. If you are the person that knows it all, share it with others. Mentor them. Teach them. Someone will be the expert, and they can teach you what to do in certain situations. Even if it’s documented, it’s a good idea to shadow this expert and watch what they do. Take notes. Do a dry run with them. Do as many different errors/issues/etc. as you can; this will only shore up your ability to perform them alone, when the opportunity arises.

Phone
Make sure to keep your phone handy. Put it in a location where you can see it and hear it. It doesn’t need to be in your hand 24-7 while on-call, but it needs to be accessible. At home, I keep it next to my sofa while I watch TV. I put it on my nightstand at night. I keep a waterproof container on the boat so when we are boating and I’m on-call, its visible and present. Whatever you need to do to ensure that it’s there, but not attached to your person, this will help you to relax and enjoy the small amounts of free time you may have while on-call, but also allows you to respond quickly.

Sane Error Notifications
It’s a good idea to setup and properly configure your phone or whatever alerting system you may use to alert you while on-call. However, too much alerting will fray your nerves. Drill down the list of items to the bare minimum. I have specific email addresses that are used (4-5 of them) for specific errors and issues. I’ve setup ringtones on my phone to alert me when these come into my inbox on my phone. Only these. This way, I am notified audibly, and not when anything else comes into my mail box. This has helped me to be better attuned to errors, and not excuse them with the bulk of the other emails, like the boy who cried wolf.
Being on-call can be a huge stress inducing period. You find yourself going crazy. You may find yourself visiting jobsites, after you get fed up with enough of the craziness. However, if you look at how you are handling the on-call timeframe, and leverage some of these ideas, adding to this list with your own ideas and needs, you will find the time spent on-call more manageable. You may never look forward to it, but it may not be as painful as it has been in the past. This is my hope for all of you, and for me.