Investigating breaches can be a bit overwhelming and very intimidating for teams that are not prepared. Your incident response (IR) plan should be written so that any of your team members can pick it up and understand going from daily incidents to investigating a major breach. I’ll write more on the IR plan on a later post. Between now and when you’re breached, you must arm your team members with knowledge. As with any profession, building a solid foundation is key. Therefore, formal training is necessary and every manager should ensure there is a training budget for each team member. I highly recommend training classes from the SANS Institute. A rule of thumb is to take one or two formal classes each year.
For a better understanding and appreciation investigating a breach two books come to mind:
These two books are true tales that take you into the work of computer espionage and computer crime. Stoll takes us from a 75-cent accounting error alert to hunting the intruder across international borders using great investigative techniques. Shimomura details the capture of an attacker that stole millions of dollars in credit card numbers and corporate trade secrets.
Logs, logs, logs…everywhere there are logs, and they are valuable. Team members should take time and review raw logs to get a better understanding of what exactly is in those logs. Log management tools and SIEMs do a good job at parsing logs but its always good to double check their work. But wait, remember logs are everywhere, so don’t forget to review logs on an endpoint that are not going into your SIEM. For Linux, most logs are located under /var/log:
- /var/log/messages – Contains global system messages
- /var/log/auth.log – Contains system authorization information
- /var/log/btmp – Contains information about failed login attempts
- /var/log/secure – Contains information related to authentication privileges.
- /var/log/wtmp or /var/log/utmp – Contains login records
I’m speaking of processes training, not process training. Team members should know system processes like the back of their hands. For example, in a Windows environment if ctfmon.exe is running from c:windows emp, is that bad?
Blogs are a great resource and two incident response blogs come to mind: JIIR and WindowsIR. In 2010, Corey Harrell started Journey into Incident Response (JIIR) to document his entry into this world we call incident response. He did a fantastic job so take the time and read through his years of blogging. Halen Carvey is the author of the WindowsIR blog and many books on the subject. Not only take the time and read through his blog but pick up his books for they are a great resource.
Speaking of technical books, every incident responder should have the following on their bookshelf:
- Incident Response & Computer Forensics, Third Edition by Jason Luttgens, Matthew Pepe, and Kevin Mandia
- The Art of Memory Forensics by Michael Hale Ligh
- Practical Malware Analysis by Michael Sikorski
- The Practice of Network Security Monitoring by Richard Bejtlich
- Hacking Exposed 7 by Stuart McClure
Youtube is your friend. There are a ton of videos on this subject but my favorite are:
- SANS DFIR Webcast -- Cache Me If You Can!
- Hunting Evil - Bart Hopper
- Lateral Movement - Harlan Carvey
- Intrusion Hunting for the Masses A Practical Guide - David Sharpe
- The Little JPEG that Could Hack Your Organization - Marcus Murray
- Plethora of videos from InfoSec conferences – an excellent resource from the tireless Adrian Crenshaw
One of the best methods to train is Red vs Blue Team and the newer phenomenon of Purple Teaming. By the way, on small teams, the Red and Blue team members might be the same person. It doesn’t matter if the Red team member has any training. Just start with the known knowns and grow from there!
“As we know, there are known knowns. There are things we know we know. We also know there are known unknowns. That is to say, we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know we don't know.”
- Donald Rumsfeld
A good first attack to try is the “Sticky Keys hack.” On Windows computer conduct the following:
- In the Command Prompt window, type copy c:windowssystem32sethc.exe c: and press Enter.
- Type copy /y c:windowssystem32cmd.exe c:windowssystem32sethc.exe and press Enter.
- Exit the Command Prompt and reboot the computer. At the login screen, tap Shift five times. The Command Prompt will pop up.
Now go to your detection tools and tweak where necessary. In addition to your own attacks, go through and re-create past incidents and/or penetration test. Where would you find the artifacts? How would you detect those today? Do you have rules written in your SIEM and/or endpoint detection tool? Take the time and hunt for cyber shenanigans. Think out of the box and think like an attacker.
I know I missed a ton of great resources so please look me up at the AlienVault booth (South Hall 1215) at RSA so I can add those resources to my list.
