Tag: production

I may not test often, but when I do it's in production

The thrill of testing in production 

if ($testing_in_production == true) {
RTFM();
}
else {
$Move=fast;
$Break=Shit;
}

 

I’ve been spending a lot of time hopefully making something better for a customer. They recently had an au

ditor c

ome in and tell them they were doing the most basic layers of security (i.e. Antivirus) all wrong and it needed to be redone. And the organization was given a deadline about a month away for 40 PCs and a dozen servers.

This is not a significant issue except the 13-hour timezone difference makes anything that gets messed up a little more precarious to go fix. My first

real sysadmin job allowed me the luxury of driving across town if I broke something.

In all cases I’m lucky that I have experience deploying the tools in a much larger environment. That environment was also under pressure. They had just been pwn’d and didn’t really know it until I stumbled across that. Really didn’t know what I was dealing with… at that time in 2008 I really had no clue what real information security was about. I learned quickly.

What I have also learned through many years of work, is that if you’re going to have to test in production, I recommend that you take a deep breath, slow down, and read the manual first. Knowing what the heck you are doing is only the first step. You really have to know *why* you are doing a thing. There’s no shortage of opportunity to move fast and break stuff, but with each instance there’s also an opportunity for learning and growth.

I may not test often, but when I do it's in production

With the amount of chaos in the world, inability for many OPSEC teams to focus on actually securing all the things, and the continual drive to still give customers what they want, there’s also no shortage of opportunities to learn-on-the-fly, be creative, and solve problems. Even in production.

Order of cuts and changes that occur to an IT project in a resource-constrained environment

My friend is giving a presentation in a few weeks regarding “How to run Information Technology Test and Evaluation in a resource-constrained environment.” We had some good fun coming up with a list of what an actual IT project looks when the resources suddenly disappear. These are in order of the things the Admins, Project Leads, and departments have to give up and live without as resources continue to diminish.

 

  1. A proper test environment – Because there’s nothing quite like the thrill of testing in production.I may not test often, but when I do it's in production
  2. High Availability – “It’s on a virtual machine… that’s highly available.” This is NOT what HA means.
  3. Administrator training – How hard can it be, the vendor has a great website, and the product has a GUI!
  4. Quality Documentation – There’s no time for actually documenting a project which is now being pushed through
  5. Backup/Junior administrator – We all know the issue that “what if (insert key technical staff person’s name here) was hit by a bus…” Well sometimes there’s nothing you can do but watch as the lowest paid admin packs his cube.
  6. Verification the backups are working – We just lost the backup administrator, they did a really good job, we’ll probably be able to restore if we need to.
  7. Basic security begins to slip – Changing the password for the service account is a lot more work than setting it to never expire. We are playing fast and lose now.
  8. Lead administrator – At some point the lead administrator leaves for another job. This happens because at some point they saw it coming long before he began talking about it.
  9. Project Crossover – Without the strong leadership within the project itself, it’s about this time another project on the ropes gets thrown in, as a way of cutting costs. Just because this project can sort of do the things of the other project, doesn’t mean it will work well. But then again it really only needs to work well enough.
  10. Change of Direction. “We’re going to the cloud!!” – Right about now the CEO or CIO have watched the most recent Gartner webinar and decided they must boldly approach the next greatest innovation that every other organization is doing. What could possibly go wrong?
  11. DEPLOY! – At this no one has the energy or tenacity to stop the work. Just because it no longer resembles the original project doesn’t mean it won’t be a great success.