How to Handle Your First Security Breach: Survival Tips

December 23, 2013  |  Conrad Constantine

So you've finally accepted it's just a matter of time before you need to handle your first security breach; despite all the work you've put in to your monitoring and response program, the long hours chasing down those last unidentified systems on the network, the endless meetings with department stakeholders and the uncountable hours optimizing your SIEM's event correlation rules, you've come to terms with the reality of "when, not if".

Of course (unless you're fresh off the certification assembly line) you probably realized this in your first week on the job; Security Professionals are not well-renowned for the quality of their sleep, nor the health of their livers. This is a guide for everyone who is dreading the day when the excrement impacts the oscillator and all the "I Told you So's!" in the world, suddenly doesn't seem to count for much. I'm not going to tell you about dealing with a security breach in a technical sense or a legal sense; I'm going to talk about maintaining your mental health and career prospects during one.

Before anything else, let's recite a truth no matter what field you work in; during times of crisis you will see everyone's true colors brought forth - not least of which, your own.

The best laid schemes of mice and men
Go often awry,
And leave us nothing but grief and pain,
For promised joy!"

- Robert Burns

It comes as no surprise, that Murphy's Law will likely bring itself to the fore (true to its nature) at the very worst time. That one web proxy that isn't yet configured to log to the SIEM - that will be the one your attacker exfiltrates your data through. That virtual machine cluster that was provisioned a month ago, but is not yet in active use, is where they will stage the attack from. That four-hour quarterly maintenance window that went badly and you lost all the logs for - that is when the security breach will happen.

You are going to have your deductive skills tested to the limit; security breaches happen through the places you were not looking. If you are lucky, you will be able to infer what happened through the remaining audit artifacts on your network. You are going to have to make large leaps of deduction, and justify them to people desperate to hear (in any form) that "at least they didn't take everything! it's not as bad as we thought!"

Survival Tip #1: Be prepared for some quality time with your executive team. During a security breach, you will find a whole chain of people that previously were merely names on an Org Chart become imminently real. If your experience at the job has been constrained to sitting quietly at your desk doing 'your thing', you are going to have more exposure to the Executive Leadership of your Enterprise than you ever imagined. They are going to require fast and decisive answers from you,and it will drive you crazy, because you are an engineer and the unknowns always outweigh the knowns. Welcome to their world - you will be asked to make decisive, quick assessments of the information you have available and be held accountable for them afterwards.

Another thing that will happen is that you will perceive everything you've spent your time on amount to nothing, you will rant and rave to yourself (and all listeners) that this is just the proof you should quit and find another job; take solace in the realization that things may have been much worse without the work you rendered. It is not the fact that a security breach occurred, it is the scale of the breach that matters.

Remember that Business will endure, for better or worse. Realize that the truth of what you saw will never see the light of day - it will be spun into an acceptable story and you will be bound by law to keep the secrets of someone else's failure; you accepted that eventuality when you took a job in Information Security. The real trick is to survive the process with your sanity intact.

Survival Tip #2: Establish and document the timeline of events. Your first responsibility will be to create a complete and detailed timeline; the time for "I could have prevented this" has come and gone - your job is now to discover and document *how* this happened - but not your interpretation of *why* this happened; as much as you want to invoke all your "I Told You So!" instincts, this is not the time. A complete blow-by-blow timeline of how everything happened within your network is the primary information your command chain needs of you. This information is what is required for legal, PR, and the board members - it should be the primary deliverable that all other workflow is derived around. Most importantly, this is what will most effectively keep the management monkey off your back.

Survival Tip #3: Don't succumb to the endless requests for hourly updates - it can impact productivity. Expect to receive constant requests for updated status, but don't let updating too often get in the way of work - give analysts space to do actual analysis; at first you may want to do hourly status calls, but realize that a 15 minute conference call every hour is just robbing 25% of your productivity in doing actual forensics work and interrupting the flow of something that requires extensive lengths of focus to achieve. Do not be afraid to push back and give yourself time to report more accurate findings - make it clear that you can either deliver inaccurate information now, or accurate information in another hour from now. Ultimately, your job is to enable informed executive decisions at this point, so clearly set expectations that this is your goal.

Survival Tip #4: Keep calm, don't panic, and don't take things too personally. Things are going to get a little crazy, requests become orders and niceties fall to the wayside. In times of crisis, sanity becomes more important than pleasantries. A friend and co-worker during a security breach we worked on together, asked me once it was over if I thought he had permanently offended people on the team for his tactic of handing out orders to people in a very demanding, terse manner throughout the event. In fact, his no-nonsense approach of delegating requests to people that were most suited for the job, with his implicit understanding of what was within the realm of deliverable possibility, became a beacon of sanity as events unfolded. Studies have shown that people would rather work with unfriendly, competent people, than unfriendly, incompetent people. This effect becomes more pronounced during times of crisis; do not worry about offending people by not being nice to them, worry about not adding to the insanity. Inevitably, you are going to end up making some judgment calls that may be above your station and tasking people that you normally would have no authority over, on the understanding you'll answer for it later on if needed. As long as you make this clear at the time, any reasonable person should support you on this. Once again, times of crisis bring out people's true natures - it is going to be an inevitable by-product that you will know more about yourself and your co-workers after the event than you ever did before; for all of you yet to go through your first rodeo I wish you a positive outcome in this regard, but you should know that there are reasons why many people seek new employment after their present one undergoes a security breach.

Survival Tip #5: It's okay to ask for help and support, you've been through hell and back. As the long hours and sleepless nights count up, remember that there is an end; eventually you will have discovered all there is to discover; Executive Management will have all the information they require to do their job and life will return to the 'New Normal' once more. If public disclosure of your security breach is required, know that it is a double-edged sword - you may well experience great catharsis in knowing that the truth is finally out there, but you must come to terms beforehand that the PR spin engine will be operating at full pace and you will be under a mountain of non-disclosure. People working in Information Security generally tend to be self-reliant types and the idea of using support resources outside of your own network of friends and family may seem alien, even repellant to you, but if you work for a suitably sized organization they will have employee counselors, who can be bound to the same NDA's you are; don't underestimate the value of having someone you can legally discuss things with.

What does not kill me, makes me stronger."
- Friedrich Nietzsche, Twilight of the Idols

Handling a corporate security breach is likely to be one of the most intense moments of your security career; you wouldn't be faulted for wondering if it's time for a career change because of it. Remember however, that in the world of incident response, there are two types of people - those that have been through a major security breach, and those that haven't. Your employer will, in all likelihood, continue to remain in business and you will continue to remain employed. In this day and age, it is an accepted truth that all organizations will be breached at some point - what is important is how you handle it. Manage the stress, try not to say anything you can never take back and realize that you're going to come out of this with experience that you can't learn in any lab, nor any simulated exercise.

Because the next time it happens on your watch, you can go into it knowing, "This ain't your first rodeo".

Share this with others

Get price Free trial