Transparently Secure

A security team who embraces openness about what it does and why, spreads understanding.

Rich Smith

People on security teams can often fall into the trap of playing secret agent. What is meant by this is that secrecy, segregation, and all sorts of other activities are undertaken in the name of security that oftentimes actually harm the goal of making an organization more secure because the security team wants to feel special or cool.

The principle of transparency in a security culture may at first seem counterintuitive; however, transparency is a principle that can take many forms, and it is really quite intuitive when you think about it in terms of spreading understanding and awareness about security to a larger group that is not security experts.

Examples of effective transparency can be as simple as explaining why a particular issue is of great concern to the security team, along with a clear and realistic explanation of the impact that could arise. Showing developers how unsanitized input may be misused in their application and the impact that this could have to the business is a far more transparent and valuable discussion of the concerns than pointing them to a coding best practices guide for the language in question and discussing the issue in the abstract. One drives empathy, understanding, and will help incrementally spread security knowledge. The other will more likely come across as an unhelpful edict.

Whenever there is an external security assessment (whether it is a penetration test or a continuous bug bounty) be sure to include the relevant development teams from scoping to completion, ultimately sharing the findings of any issues uncovered, and having the developers fully in the loop as mitigations and fixes are discussed and decided upon.

There is nothing more adversarial than a security team seeming to be working with an external party to pick holes in a system, and then have a series of changes and requirements appear from the process in a black-box fashion. The team whose project is being assessed rarely wants to have urgent requirements or shortcomings tossed over the wall to it, it wants to be engaged with to understand what the problems are that have been found, how they were found, and what the possible real-world impacts could be. Chapter 12 goes into this area in more detail and won’t be repeated here; however, the cultural aspects of how to have supportive conversations when security issues are found during assessment is important to consider before the pen testers have even looked at the first line of code or have thrown the first packet.

A great deal of the value of an assessment comes from being as transparent as possible with the teams whose work is subject to review. It is not only a chance to find serious security bugs and flaws, but is also a highly engaging and impactful opportunity for developers to learn from a team of highly focused security specialists. Simple, but unfortunately rare, approaches such as sharing a repository of security findings from internal or external assessments with the developers directly helps engage and share everything in the full gory detail. This can often also result in the creation of a positive feedback loop where developers end up contributing to a repository of issues themselves when they are able to see the kind of things the security folks care about. It may be surprising the number of security relevant bugs that are often lying around within the collective knowledge of a development team.

Another way in which the transparency principle can be embraced is by becoming more inclusive and inviting developers to be part of the security team and jump into the trenches. This may be part of the onboarding or bootcamp process that new developers go through when joining an organization, or it may be part of some continuing training or periodic cross-team exchange: the important thing is that people from outside of the security team get a chance to see what the day-to-day looks like. Having developers see the grisly details of a security team’s work will do more than anything else to drive empathy and understanding. Oftentimes it will result in the seeding of people across an organization who are not formally on the security team, but who remain closely in touch with them and continue to facilitate visibility to the development process while continuing to learn more about security. At Etsy these are called “security champions” and are a key component to the security team being able to stay available to as many other teams as possible.

A security team that can default to openness and only restrict as the exception will do a far better job at spreading knowledge about what it does, and most importantly, why it is doing it.

Don’t Play the Blame Game

Security failures will happen, only without blame will you be able to understand the true causes, learn from them, and improve.

Rich Smith

As much as it may be uncomfortable to admit, in a sufficiently complex system nothing that any security team does will ever be able to eliminate all possible causes of insecurity. Worse still, the same mistakes may happen again at some point in future.

Being blameless as one of the core tenets of your approach to security may seem like a slightly strange thing to have as one of a list of the three most important principles—surely part of keeping an organization secure means finding the problems that cause insecurity, and in turn making things more secure by finding what (or who) caused those problems, right?

Wrong!

Anyone working in a modern organization knows how complex things get, and how the complexity of the environment seems to continue to increase more than it simplifies. When technology and software are added to this mix, a point can be reached quite quickly where no single person is able to describe the intricacies of the system top to tail. Now, when that grand system is declared as something that needs to be secured, the difficulty of the task at hand can become so immense that shortcuts and corner-cutting exercises inevitably need to be found, as without them no forward progress could be made within the time frames required.

One such shortcut that has been used so pervasively that it regularly goes unquestioned is blame. Finding who is to blame is often the main focus of the activity that follows the discovery of an issue—but that misses the point. It is imperative to understand that building a security culture based around the belief that finding who to blame for a security issue will result in a more secure future is fundamentally flawed.

Many security issues may seemingly arise as a result of a decision or action an individual did, or didn’t, take. When such situations arise—and they will—a common next thought is how the person who seems at fault, the human who made the error, should be held accountable for his lapse in judgment. He caused a security event to occur, which may have caused the security team or possibly the wider organization to have a day from hell, and he should have the blame for doing so placed squarely at his feet.

Public shaming, restriction from touching that system again, demotion, or even being fired could be the order of the day. This, the blameful mindset will tell us, also has the added benefit of making everyone else pay attention and be sure they don’t make the same mistake lest they also suffer the same fate.

If we continue on this path of blaming, then we end up succumbing to a culture in which fear and deterrence is the way in which security improves, as people are just too scared of the consequences to make mistakes.

It would be fair to say that software engineering has much to learn from other engineering disciplines when thinking about blame, and in a wider sense, safety. A notable thinker in the space of safety engineering is Erik Hollnagel, who has applied his approaches across a variety of disciplines, including aviation, healthcare, and nuclear safety—and luckily for us, also in the space of software engineering safety. Hollnagel’s perspective is that the approach of identifying a root cause and then removing that error in an effort to fix things, and therefore make the system safer, works well for machines, but that the approach does not work well for people.

In fact, Hollnagel argues that the focus on finding the cause and then eliminating it has influenced the approach taken to risk assessment so completely that root causes are oftentimes retrospectively created by those investigating rather than being truly found. When human factors are considered in accidents, Hollnagel asserts that the inevitable variability of the performance of the human part of the system needs to be accounted for in the design of the system itself if it is to have a reduction in risk. A quote attributed to Hollnagel that summarizes his views of accidents involving humans well is:

We must strive to understand that accidents don’t happen because people gamble and lose.

Accidents happen because the person believes that:

…what is about to happen is not possible,

…or what is about to happen has no connection to what they are doing,

…or that the possibility of getting the intended outcome is well worth whatever risk there is.

Unless there is compelling evidence that someone in the organization is deliberately trying to cause a security event, the cause of the incident should not just be neatly explained away as human error and then quickly step into applying the appropriate blame. It is key to a progressive security culture to have a shared understanding that the people working in your organization are not trying to create a security event, but instead are trying to get their work done in a way that they believe would have a desirable outcome; however, something either unexpected or out of their control occurred, resulting in the incident. Rather than spending energy searching for the root cause as being who to blame, efforts should be taken to understand the reasons that underly why the person made the decisions she did, and what about the system within which she was working could be improved in order for her to detect things were about to go badly wrong.

If the focus is on assigning blame and delivering a level of retribution to the people perceived as the cause, then it follows that the people with the most knowledge about the event will not be forthcoming with a truthful perspective on everything they know, as they fear the consequences that may follow. If, however, the people closest to the security event are safe in the belief that they can be 100% honest in what they did, and why, and that that information will be used to improve the system itself, then you are far more likely to get insights to critical shortcomings of the system and potentially develop modifications to it to make the future use of it safer.

A mechanism that has been developed to help with this approach of finding the reasoning behind a decision, and then improving the system to ensure more informed decisions are made in future, is the blameless postmortem.

When considering blameless postmortems, it is important to go into the process understanding that the issues you are about to investigate may very well happen again. Preventing the future is not the goal—learning is, and to that end, learning needs to be the focus of the postmortem itself. Participants must consciously counter the basic human urge to find a single explanation and single fix for the incident. This is often helped by having a postmortem facilitator who gives a brief introduction reminding participants what they are all there for: a semi-structured group interview. Nothing more.

While this perspective sounds great on paper, how does it work in reality? From one of the author’s experiences at Etsy, security postmortems were some of the best attended meetings at the company and provided a regular opportunity to investigate root causes that resulted in significant improvements to the security of the systems. Postmortems empower engineers to do what they do best, to solve problems in innovative ways, as well as establish trust between engineers and a security team. The well attended meetings also provided a powerful opportunity to deliver security education and awareness to a large and diverse group of people from across the organization. Who doesn’t want to attend a meeting that’s digging into the details of high-impact security events and listen to experts from across the organization collaborate and problem-solve in realtime in front of them?

Finally, it is worth explicitly noting that postmortems can be a valuable way to create and share a knowledge base of issues across teams. By tackling root causes and reoccurring patterns holistically rather than on a case by case basis, solutions are devised scalably and systemically.

As we touched on in Chapter 13, the important takeaway is that while much of what has been discussed regarding blameless postmortems in the software engineering space has been written from an ops perspective, the lessons learned apply just as much to security, and there are many lessons that can be learned from the last near decade of DevOps development. For further reading on blameless postmortems, see the following:

• “What Etsy Does When Things Go Wrong: A 7-Step Guide”

• “Blameless PostMortems and a Just Culture”

• “Etsy’s Debriefing Facilitation Guide for Blameless Postmortems”

• “To Err Is Human: The ETTO Principle”

Scale Security, Empower the Edges

A practical approach to progressive security that embodies many of the principles described is actively including those outside of the security team in the security decision-making process itself, rather than excluding them and having security as something that happens to them. In many scenarios, the users themselves will have far more context about what may be going on that the security team will; including them in the security decision and resolution process helps ensure that responses to security events are proportional and prioritized.

Including everyone in security decision and resolution loops can be realized in a variety of ways and is something that should be fine-tuned for a particular organization’s needs. Examples of including everyone in the security decision-making process would be the following:

• Alert users over the internal chat system in the event of noncritical security events such as failed logins, logins from unfamiliar IPs, or traffic to an internal server they don’t normally access, and asking them to provide quick context on what’s going on. In many cases the explanation will be an innocuous one, and no further action will be required. In the event that the user does not know what has happened or does not respond within a time frame, the security team can step in and investigate further. A great discussion of this is “Distributed Security Alerting” by Slack’s Ryan Huber.

• Notify users at login time if their OS/browser/plug-ins are out of date, giving them the option to update immediately (ideally with one click, if possible), but also, and most importantly, allowing them to defer the update to a future point in time. The number of deferrals allowed before the only option is to update, as well as the versions of the software that can have updates deferred on, is down to the security team, but the important thing is to provide the user some shades of gray rather than just black and white. For example, if a user’s browser is only slightly out of date, it may be an acceptable trade-off to let him log in to get the action he needs to do complete and update later on, rather than blocking his access completely and forcing a browser restart at a incredibly inconvenient moment for him.

• Ask a user over email or chat if she has just installed some new software or has been manually editing her configuration as a change has been seen on her system such as a new kernel module being loaded, her shell history file got deleted, or a ramdisk was mounted, for example. While suspicious and possibly indicative of malicious activity, in an engineering environment there are many reasons a developer would be making lower-level changes to her setup.

• In internal phishing exercises, have the main metric being focused on as not the click-through rate on the phishing emails, but the report rate of users who let the security team know they received something strange. A commonly misplaced intention of phishing awareness programs is the aspiration to get the click-through rate down to zero. As the authors of this book can attest to only too well from across a multitude of organizations, while the goal of everyone in an organization doing the right thing and not clicking on the phish every time is laudible, it is wholly unrealistic. If the win condition is that all people have to do the right thing every time, then you’re probably playing the wrong game as you will rarely win. If your focus is on trying to ensure that one person needs to do the right thing, that being notifying the security team he received a phish, then that is a game you are far more likely to win.

These are just a few examples among many possibilities, but they all succeed at pulling the user and her specific context into the decision loop, which contributes to a security team that can better prioritize its work, and a user base that does not feel that security is a stick that is always being used to hit them. Additionally these kinds of interactions provide excellent points at which tiny snippets of security education can be delivered, and starts to build up a workforce that feels connected and responsible for the security decisions being made. Eventually this will result in the creation of positive feedback loops where users are actively engaging with a security team in the truest sense of see something, say something, or proactively giving security a heads-up that they are about to do something that may set off some alerts.

The Who Is Just as Important as the How

If this all makes sense in terms of what the mission of an effective security team should be, then the next step is actually executing on this mission. A great culture needs great people, and too often members of a security team are hired based on their technical skills alone, with little consideration for their soft skills. If it is believed that people are just as important as technology to having an effective security program, then we need to hire security practitioners with that in mind.

One reason why many security teams are actively avoided in their organization is that the personalities and attitudes of the members of the security team are actively working against the wider goal of making security approachable and understandable by the wider organization. Too often security practitioners can be patronizing or condescending in their interactions with the people they are there to support, some seemingly even taking delight in the security issues encountered by their coworkers.

While this is not a book about people management, it’s worth noting that the manager responsible for a security program has a special responsibility to be on the lookout for behavior and attitudes from the security team that are not conducive to the wider security goals, and to put a stop to it if and when it is found. The cardinal rule of thumb that should be front of mind when thinking about security team hiring is one that seems obvious, but historically seems to have been too often not followed:

Don’t hire assholes.

Closely followed by:

If you inadvertently do, or you inherit one, get rid of them ASAP.

A single abrasive member of a security team can do more to undermine the efforts made in the implementation of a progressive security program than almost anything else, as they are the face of that program that others in an organization have to interact with on a daily basis.

A great book that discuses this in much greater detail is The No Asshole Rule: Building a Civilized Workplace and Surviving One That Isn’t (Business Plus) by Robert I. Sutton. In the opinion of the authors, this should be required reading for every security manager, and should be kept close at hand and visible in case its valuable lessons start to be forgotten.

Securgonomics

Much of the discussion so far has been about the theoretical aspects of how a security culture can be established and nurtured; however, as culture is central to how people work with one another and act as a collective, the real-life aspects shouldn’t be ignored. Securgonomics is a completely made-up term created by one of this book’s authors (okay, it was Rich) as part of some presentation slideware: a portmanteau of security and ergonomics.

If the definition of ergonomics is:

er·go·nom·ics \ˌər-gə-ˈnä-miks\

noun

the study of people’s efficiency in their working environment

then the definition of securgonomics (if it were, in fact, a real word!) would be:

se.cure·go·nom·ics \si-ˈkyu̇r-gə-ˈnä-miks\

noun

the study of people’s efficiency of security interactions in their working environment

While a made-up term, it captures the very real observation that security teams can often fall into the trap of separating themselves from the people they are there to support. An Agile approach aims to break down silos because they inhibit easy communication and collaboration, and this needs to extend into the real-world space inhabited by security teams and others in an organization. The goal of all of this is to lower the barriers to interacting with the security team.

The use and design of the physical office space needs to help make the security team visible, easy to approach, and therefore easy to interact with. Too often a security team will lock itself behind doors and walls, which creates a very real us and them segregation. While it is true that security teams may be dealing with sensitive data as part of its daily role, it is also true that for much of the time its activities are not any more sensitive than the work being done by its coworkers.

Security teams can too often fall into the stereotypical trope of being secret squirrels working on things that couldn’t possibly be shared outside of the group. While occasionally this may be true, much of the time this perspective is falsely (and unnecessarily) created by the members of the security team themselves. A team that locks itself away from the rest of the organization will be at a significant disadvantage if it is trying to be a team of transparent enablers that help security become a responsibility that is genuinely shared by people across the organization.

Look at where your security team(s) are physically located, and ask yourself questions like these:

• Does this help or hinder others to be able to easily come over and interact with team members?

• Do people see the team as part of their daily routine, or do they have to go out of their way to speak to team members in person?

• Do new hires see the security team’s location on their first-day office tour?

• Is the security team near the teams that it works with most closely and frequently?

• Can others see when the security team is in the office? Can they see when team members are busy?

Where possible, having the security team in a location that has a lot of footfall enables easy interaction with them in a low-lift way. If others wander by the security team on the way to the coffee station or on their way in and out of the office, then micro-interactions will inevitably occur. Not requiring people to make a special journey to the security team will mean that more ad hoc conversations will happen more regularly. This has benefits to everyone, not the least of which is building up trusted relationships between the security team and everyone else.

Where the security folk sit is important; but even if they are in a prime spot where they are easily accessible to all, it doesn’t mean they should rest on their laurels: they need to also work to incentivize other people in the organization to interact with them. This doesn’t need to be overthought and can be as simple as having a sticker exchange, a lending library for security books, or some jars of candy. At Etsy, security candy became a phenomenon all of its own, whereby others across the organization took part in choosing which sweets would be in the next batch of candy to be ordered ,and where the current inventory could be queried through the company-wide chatbot, irccat.

There was even a hack-week project that set up an IP camera over the candy jars to allow remote viewing of what was currently up for grabs. While a fun and silly endeavor, the ROI seen from such a simple act of sharing candy with others in the office incentivized engagement and interaction with the security team in a way that vastly outweighed the monetary cost involved.

Having said all that about the physical location of the security team(s), the final piece of advice would be to not tether the team members to their desks. Having a security team whose method of work enables team members to be mobile around the office and to go and sit with the people they are working with where possible is equally important.

Set the expectation across the organization that a security engineer will go and sit alongside the developer with whom they are diagnosing an issue or explaining how to address a vulnerability. This simple act is a very visible embodiment of the principle of being a supportive and enabling team that wants to understand the other people in an organization, and that they are there to help others succeed on their missions in a secure way.

Dashboards

Lots of teams have dashboards displayed on big screens in their working environment to keep a high-level overview of trends and events that are occurring, and obviously security is no different. However, it is worth pointing out the cultural relevance that dashboards can hold and the dual purposes of the dashboards that may be on display. Having a better understanding of the different objectives that dashboards can help achieve will hopefully empower better decisions about which dashboards make the most sense for your environment.

We can roughly group workplace dashboards into four types:

1. Situational-awareness dashboards

2. Summary dashboards

3. Vanity dashboards

4. Intervention dashboards

Situational awareness dashboards are dashboards that display data that is of most use by a security team itself. The data displayed shows the current state of an environment or system, along with as much historic or contextual data as possible so as to allow team members to make a determination as to whether something may be wrong or not. The primary objective of a situational dashboard is for a security team to at a glance have a sense of whether everything is as expected for a given system. It is worth noting that situational dashboards only indicate that something is amiss and needs investigating, rather than telling you exactly what is taking place attack-wise. They provide easy ways of seeing symptoms. It is for the team then to jump in and determine the cause.

Examples of situational awareness dashboards are displays of data such as “successful versus unsuccessful login attempts over time” (Figure 15-1), “number of 2FA SMS, call and push codes sent over time,” and “number of detected XSS attempts over time” (Figure 15-2). The cultural benefits of situational awareness dashboards really center around them acting as a source of reassurance for people to see that the security of systems is being monitored and acted upon when needed. They can also be a source of engagement with the security team, especially if time-series graphs are being displayed and big changes are relevant. If people see graphs going crazy, they are often curious as to what caused the change and what is being done in response.

Figure 15-1. An example of a situational-awareness dashboard showing the number of successful logins to a web application versus the number of failed login attempts. Patterns between these two metrics can indicate different situations that warrant follow-up by the security team; for example, a rise in failed logins versus successful logins would be indicative of a password-guessing attack taking place. A drop in successful logins could indicate a problem with the authentication system, with legitimate users being denied access.

Figure 15-2. Another example of a situational awareness dashboard, this time showing a number of different types of attack/scanner detections. Spikes and extended shifts in behavior should prompt the security team to follow up to investigate the cause.

If a situational dashboard is dealing with user identifying information, such as a username, then care should be taken to pseudonymize that information to avoid creating a display that feels implicitly blameful or that becomes a source of embarrassment. An example where such obfuscation of data could well be prudent would be dashboards showing the recent VPN logins of users and the location from which the login originated. A pseudonymized identifier for the user is still useful for a security team to see events that team members may want to follow up on without displaying the real names associated with any such events. Then as part of the team’s follow-up, the pseudonymized identifier can be reversed back to a real user ID, which will be used to track down what may be happening.

Summary dashboards are really just a dashboard representation of periodic data rollups or executive summaries. Examples of summary dashboards would be displays of data like “total number of reported phishing emails in the last four weeks,” “dollars paid out in bug bounty this year,” or “external ports shown as open in the last portscan.” They present historic data summaries, often with minimal or no trend data, such as the week-over-week change.

Of the four dashboard categories, it can be argued that summary dashboards are the least useful for security teams, as they do not give information on the current state of security for a system or environment and so do not prompt responses or actions from a security team. They also have minimal value in terms of eliciting engagement from others. Summary dashboards don’t really provide much more information besides, Look! We’ve been doing things. Most of the information they show is better provided in rollup emails or executive summaries sent to those who need to know on a periodic basis. If space is at a premium, then look to your summary dashboards as the first to drop.

The mixture of the types of dashboards that make most sense for your organization is something that only you can determine; but collecting analytics on the interactions they prompt, anecdotal or otherwise, of the different mixtures of dashboards would be encouraged. If the dashboards displayed on big screens are also made available on the intranet for people to view in their browsers’ tracking hits can also be a useful proxy metric for the level of interest different dashboards create.

Vanity dashboards cover those dashboards that display data or visualizations that actually have minimal direct value to the security team’s day-to-day work. Instead, their purpose is to look cool and cause passersby to pause and watch, and hopefully, ask a question about what the cool stuff is.

Vanity dashboards often are the least data-dense of any of the dashboard categories, and value looking cool over actually showing data that can be acted upon. Common examples are animated spinning 3D globes showing the geographic sources of incoming attacks, or current top malware being detected/blocked in email. Many security vendors make such animated dashboards freely available on their websites, in the hope that animations that look like they are in Swordfish will convince you to buy their wares. Still, they can be a great source of vanity dashboards for minimal effort. If you would like to build your very own PewPew map (with sound!), there is an open source project called pewpew that will get you started and has lots of options for you to tweak and enhance (see Figure 15-3). Creating fun and ever-changing maps can be a lighthearted way to build a positive security culture for both the security team and the people that will get to view the results.

Figure 15-3. An example of a PewPew map, courtesy of the pewpew opensource project.

Despite the flashy appearance and low informational value of vanity dashboards, they can carry with them some of the highest cultural impact, as they play on the interest many people across an organization have about security (or hacking). They also create great opportunities for people to engage with the security team to talk about what all the laser shots coming from Eastern Europe to the data center actually mean. They are also the ones that people will often pause for a few seconds to watch, which presents an opportunity for security team members to chat with someone about that low-priority alert associated with their IP that came through earlier on.

Of the four types of dashboards, vanity dashboards are those that you should look to switch up periodically to keep them fresh and to continue to illicit engagement from the organization.

Intervention dashboards are possibly the simplest of all the dashboards, as well as being the ones that are (hopefully) seen least often. The sole purpose of intervention dashboards is to alert a security team to a critical situation requiring immediate attention; they are triggered by some pre-configured event and should interrupt the display any of the other types of dashboards. Examples of intervention dashboards would be events such as a critical credential has been detected as being leaked to GitHub, or that an internal honeypot is being interacted with.

Intervention dashboards have a couple of specific benefits beyond other types of alerts and notifications that a security team will likely already be using. Firstly they are a public collective alert. When an intervention dashboard is triggered, the entire security team is made aware at the same time and can prioritize the response to the situation over any other activity taking place. Secondly, and arguably most importantly, intervention dashboards allow others within an organization to see that a serious event is in flight and that the security team is working on understanding and/or responding to the situation. While this second point may self-obvious or self-evident, the value of people outside of a security team being aware that something security related is underway not only facilitates greater empathy for the work of a security team, but also results in future opportunities to engage and educate when people enquire about what is happening.

Some rules of thumb for intervention dashboards are that they should be reserved for the most critical security events that are being monitored, and the frequency of them being displayed should be low and result in an all-hands-on-deck response from the security team. The subject of any intervention dashboard should also be readily actionable, ideally having a dedicated runbook associated with them that drives the initial response and information gathering activity. The data density of intervention dashboards should be low, with a clear description of the triggering event, a timestamp, and possibly a reference to the runbook or initial actions that should come first in a response. The look and feel of the dashboard screen that is triggered is pretty much a free-for-all as long as it is immediately noticeable and stands out from the other dashboards in the rotation; for example, a plain red screen with flashing white Arial text, or a gif of an angry honey badger could both serve this purpose equally well (angry honey badgers are way cooler though).

For any of the dashboards to be available and useful, the correct monitoring and data collection needs to be in place. Approaches to security relevant monitoring are covered in Chapter 8, Threat Assessments and Understanding Attacks.