What is the CISO Experience in a Red Team Exercise?

You’re about to have your first Red Team experience, or maybe your first one in the CISO seat of your organization. Maybe it’s just been a little while since your last one and you are curious how this one will go, what the Red Team will find, how your Blue Team will handle it, and what the longer tail takeaways post-engagement will be like.  

But before you begin, it’s important to consider: What am I not thinking about? Are we ready? How can I prepare for this?

What if I Have Specific Objectives for Red Teaming?

If you haven’t already, make sure you’ve discussed your objectives with your Red Team partners to ensure alignment with what you’re hoping to learn and focus on. This conversation will often center around matching Red Team objectives with the maturity of the security program and your Blue Team to get the most benefit from a Red Team exercise, because this definitely should not be a one-size-fits-all exercise. For example, at NetSPI, we tailor match the Tactics, Techniques, and Procedures (TTPs) we use to your currently known capabilities and gaps. Our goal is to help you grow your program in a meaningful and material way, even if resources are constrained and growth is gradual.

How Much Do I Tell My Team when Engaging Red Team Testing?

It’s most common for a Red Team exercise to be an extremely limited knowledge event. Who you provide advanced notice to is up to you. Our advice: less is more if you want to know how truly prepared your security program is.  

If you do these all the time, you may want to tell your team that a Red Team exercise will happen in the future but remain vague—no specific dates. This has a “Secret Shopper” effect, just like a retail clerk who is unsure if their customer is an actual customer, or a plant sent from corporate headquarters to evaluate the store. The foreknowledge that a secret shopper may arrive at any time can have a positive psychological effect, bringing out the best performance of the team. Likewise, your Blue Team may become naturally more vigilant simply because they know a Red Team may come anytime.

What if I have an MSSP or MDR Provider?

Since most MSSP or MDR provider relationships are focused solely on the ability to detect and respond to credible threats, it is best to NOT advise them in advance that the Red Team exercise is happening. However, post-exercise, it is critical that you properly read-in your provider so that they can collaborate with you on a path to improve detection and response coverage. NetSPI, specifically, loves to partner with MSSPs and MDR Providers, because they are your Blue Team on the front lines. Our objective isn’t to make your provider look bad; our objective is to prepare your organization for the eventuality of a real incident.

Should I Have Expectations on How Successful the Red Team Exercise Will Be? 

It’s probably best to set expectations that while your Blue Team will bring some friction to the Red Team, it will feel like the Red Team managed to get ahead and reach objectives too easily. This isn’t always the case, of course, and we love to have our best tradecraft get shut down by our customers!  

But since our Red Team constantly focuses on what works, what doesn’t, what security controls provide friction against which TTPs, etc., we are constantly improving. If our Red Team is successful, it doesn’t mean that the threat actors most likely to land in your environment will automatically have equal success.  

Threat groups tend to cluster around a smaller set of TTPs than our Red Team because they apply them at Internet scale across many organizations. If the techniques fail and a Blue Team contains them, they don’t care. There isn’t enough friction to change TTPs often if they still work on the next victim. Our goal is to be the best [simulated] threat actor we can be for you. This is a subtle, but important difference. 

Now all of that isn’t to say this is easy for our Red Team. By far the hardest part of our job is getting the initial access foothold into your organization. We don’t have magic 0-day exploits to walk right in. We have drudgery ahead of us: scouring your entire perimeter, learning about your business using Open-Source Intelligence (OSINT), social engineering our way in (if that’s in scope for your engagement) … essentially leaving no stone unturned.  

We prefer to do it this way, when possible, because once our Red Team lands inside your organization, it will “feel natural” to incident responders who eventually (hopefully) will see something unusual that they chase to its origin. But that said: do not over-index on this step. If your goal is to absolutely find a way from the outside into your organization, you probably should do an External Network Penetration Test instead.  

What you’re ultimately buying in a Red Team exercise is the detection and response cat-and-mouse game that helps you evaluate your readiness for a breach. You don’t get that benefit from us until we land inside your organization. Because neither you nor we have unlimited surplus budget, we will want to time box our efforts looking for the “natural” ingress point, and when we hit that point, we will want to switch to an “assumed breach” scenario where you seed us access. We can even do it this way from the start to save time and money.

What Happens After a Red Team Exercise? 

Besides the debrief meeting and handing you deliverables, what’s next for a CISO after a Red Team exercise? In most cases, there will be significant security engineering and process overhaul project work. Unlike a pentest, where a finding can be quite small and tactical, such as applying a patch, fixing permissions, changing a password, or updating a line of code, findings coming out of Red Team exercises are typically wide-reaching and systemic. Some may require projects that span more than a year to complete. It may be good for you to brief your CFO, CEO, and Board of Directors about the exercise in advance that you will likely come asking for a budget increase to cover control gaps. We can certainly help you with messaging there as well! Reach out anytime. 

What about Follow-Up Testing? 

While the Red Team may likely find and exploit vulnerabilities in your internal environment, they won’t exhaustively search for all related instances of that vulnerability. Red Teaming is a depth-first search: chaining vulnerabilities, detection gaps, process flaws, and misplaced human trust together to reach an objective.  

Penetration Testing, on the other hand, is a breadth-first search: locating all instances and permutations of all possible vulnerabilities. For example, if the Red Team finds a single instance of SQL injection on an internal web application, exploiting that to gain additional objectives or access, the best next step is to perform a top-to-bottom penetration test on that web application, to ensure nothing else was missed that the Red Team didn’t have time to find, or was trying to be too quiet to test. 

How Often Should I Plan for Red Team Testing?  

This is entirely up to you, of course, but here are some things for you to consider:  

• How much has changed with your controls since you completed the first Red Team exercise?
  If not much, don’t expect a wildly different experience in the Red Team’s ability to reach objectives—but the exercise can still be meaningful to give your Blue Team another chance to train and become more prepared for an actual event. You can also ask us to avoid certain things or modify the path towards objectives to vary from your prior experience. 
• How large and segmented is your business?
  If you have a lot of M&A, subsidiaries, disparate geographic locations, etc., you may benefit from intentionally scoping another Red Team exercise to land in another part of your organization sooner than later. These “satellite” organizations often provide less detection and response friction to adversaries looking for a path to pivot into the corporate mothership.
• What cadence are you trying to establish?
  It may be beneficial from a budgeting perspective to plan for a semi-annual or annual Red Team exercise to set a solid precedent with your CFO, CEO, and Board of Directors that this is a meaningful recurring part of your security program. When combined with the ideas above, the experiences each time will definitely vary. 

How Can I Tell if a Red Team Exercise is Successful? 

As the CISO, you will appreciate that a successful Red Team exercise has almost nothing to do with whether the Red Team reached an objective.  

The Red Team could reach an objective but highlight serious gaps in the process that you can quickly fix with existing controls or help make the business case for a security budget extension. Or they could be contained by your Blue Team without any new technical learnings, yet the confidence the Blue Team gains from containing the Red Team might be precisely what is needed for your security program. 

At the end of the day, “success” is largely a product of clearly defining the goals you have for the engagement and tying the results back to the identification and reduction of risk, improving your cybersecurity program, and protecting your organization. No two exercises are exactly alike! 

Whether you’re starting your first Red Team exercise, or you’re looking for an outside perspective on your overall security, NetSPI is here to help. Access our Red Team data sheet below to get started.