PCI DSS: Rules-Based

The Payment Card Industry Data Security Standard (PCI DSS) is a cross-industry standard of practice that many teams will end up having to deal with, as it applies to any system that deals with credit or debit card payments, either directly, or through a third-party service.

Rather than vague legal statements that you must demonstrate appropriate due care, PCI DSS lays out specific, mostly concrete requirements and obligations: you must do this, you must not do that, and this is how you must prove it. It sets reasonably clear expectations about what risks need to be managed and how, about what data needs to be protected and how, what testing or reviews you need to do, and when you need to do them.

PCI DSS is described in a number of guides, supplementary documents, notices, checklists, and FAQs; but the core of the regulation is relatively small. While it isn’t fun (regulations are never fun), it’s not that difficult to follow. There’s even a Quick Reference Guide, only about 40 pages long, that does a decent job of outlining what you need to do.

There are 12 sections in a PCI-compliant security program. While people can—and do—argue about the effectiveness or completeness of the specific controls, PCI DSS is a reasonably good model for a security program, whether or not you deal with credit card information. It anticipates many of the risks and security issues that you need to deal with in any online system that holds important data, and lays out a structured approach to manage them. Just substitute “cardholder data” with whatever sensitive or private data that you need to manage:

Secure network design

Use of firewalls and network segmentation. Ensure that configurations are periodically reviewed and that all configuration changes are tested. Map all connections and data flows.

Configuration hardening

Identifying all of the systems involved, and making sure that they are configured safely to “industry-accepted definitions” (eg., CIS), specifically making sure to change vendor-supplied default credentials.

Protect sensitive data (cardholder data) at rest

Limit/restrict storage. Use one-way hashing, strong encryption, tokenization, masking, or truncation. Specific restrictions on what data can/cannot be stored.

Secure communications

Using strong encryption and security protocols when transmitting sensitive data over open networks.

Protecting systems against malware

Use of antivirus systems.

Develop and maintain secure systems and applications

We’ll look at this section in more detail below.

Restrict access to cardholder data

Access control restricted based on business need-to-know.

User identification and authentication guidelines

Including when multifactor authentication is required.

Physical security

Restricting and tracking physical access to data centers and sensitive areas, and to backup media.

Track and monitor all access to network and cardholder data

Auditing, logging. Specific requirements for auditing of activities, access to restricted data, and retention of audit trails.

Regular security testing and scanning

Scanning for wireless access points, internal and external network vulnerability scanning, penetration testing, intrustion detection/prevention, and detective change control.

Paperwork

Governance, risk assessment, and information security policies to prove that management and everyone else understands the requirements and how they are being met, in addition to insisting that operational procedures and policies for each of the previous requirements are “documented, in use, and known to all affected parties.”

This includes regular security and compliance awareness training for developers and operations.

In order to be compliant, your organization must meet all of these requirements, and undergo regular assessments.

Requirement 6 is the section that applies most directly to software development. It addresses requirements, design, coding, testing, reviews, and implementation.

6.1 vulnerability management

Identify vulnerabilities, determine priority based on risk, and deal with them. We looked at vulnerability management in more detail in Chapter 6, Agile Vulnerability Management.

6.2 patching

Make sure that patches to third-party software are up to date. Critical patches must be installed within one month.

6.3 secure coding practices to ensure that software is built “based on industry best practices”

This includes specific requirements such as ensuring that all test accounts and credentials are removed before applications are put into production, reviewing code changes (manually or using automated tools) for security vulnerabilities, and making sure that code review results are reviewed and approved by management before the code changes are released.

6.4 change control

Including clear separation of duties between production and development/test, evidence of security testing, impact analysis and backout planning, and documented change approval by authorized parties.

6.5 train the development team in secure coding

At least annually, and provide them with secure coding guidelines.

6.6 protect web applications

Requires an application vulnerability assessment review at least once per year, or before rolling out major changes. Alternatively, you can protect your system by implementing a runtime defense solution that blocks attacks (such as a Web Application Firewall).

6.7 paperwork

To prove all of this.

That’s “all there is” to PCI compliance from a development perspective, beyond correctly implementing controls over credit cardholder data in Requirement 3 and access restrictions to cardholder data in Requirement 7. The devil, of course, is in the details.

Reg SCI: Outcome-Based

Contrast this with Reg SCI (Systems Compliance and Integrity), which is one of a number of regulations that stock exchanges and other financial services organizations in the US need to meet. It is concerned with ensuring the capacity, integrity, resiliency, availability, and security of financial systems.

Reg SCI is 743 pages of legalese, much of it describing the process followed to draft the regulation. The core of the regulation is captured in about 20 pages starting somewhere after page 700. It requires that organizations develop and maintain “written policies and procedures reasonably designed” to ensure that systems are developed and operated safely and that they meet all necessary legal requirements.

These policies and procedures must cover system design and development, testing, capacity planning and testing, continuity planning and testing, change management, network design and management, system operations, monitoring, incident response, physical security, and information security, as well as outsourcing any of these responsibilities. Reg SCI also lays out obligations for reporting to the SEC, and for annual audits.

Policies and procedures may be considered to be “reasonably designed” if they comply with “information technology practices that are widely available to information professionals in the financial sector and issued by a recognized organization” such as the US government.

The list of recognized best practices includes, for almost every area, NIST SP 800-53r4, which is a 462-page document describing 218 specific risk management, security, and privacy controls and how to apply them for US federal information systems (based on the risk profile of the system). Controls that apply to software development are described under a number of different control areas, such as System Acquisition (mostly), Configuration Management, Risk Assessment, System and Information Integrity, Awareness and Training, Privacy, Security Assessment and Authorization, and so on. Each of these controls reference other controls or other NIST documents, and other publications and standards.

The overall effect is, well, overwhelming. Determining what you need to do, what you should do, what you can afford to do, and how to prove it, is left up to each organization—and to its auditors.

NIST SP 800-53r4 doesn’t say you can’t build and deliver systems using Agile and Lean methods, but it is a shining example of everything that Agile and Lean is against: legal policies and shelf-loads of paper that are designed to make a bureaucrat happy, but that don’t translate to specific, measurable requirements that can be satisfied.

The SEC makes it clear that the list of standards is offered as guidance. Your controls and programs would be assessed against these standards. If you don’t choose to follow them, the burden is on you to prove that your system of controls is “reasonably designed.”

Compliance Stories and Compliance in Stories

Mandatory compliance testing, reviews, and other items in compliance checklists need to be considered in the team’s Definition of Done, for stories, sprints, and releases:

• What reviews need to be done and who needs to do them

• What testing needs to be done

• What documentation needs to be updated

• What evidence needs to be provided that all of this was done

In Chapter 5 we looked at how to write security stories to help in implementing security requirements or security controls. In the same way, you may want to write compliance stories to describe explicit steps that need to be done and proven, separate from compliance criteria that might be part of specific user stories.

Compliance stories can act as reminders or schedule placeholders for controls that need to be put into place up front, including compulsory training for team members, and for assessments such as audits and pen tests.

Use tools like osquery and InSpec to write automated online compliance checks, and to provide traceability back to specific regulations or rule areas or governance controls.

More Code, Less Paperwork

Agile and Lean teams look for ways to minimize waste by keeping documentation to a minimum: working code over documentation.

But when it comes to compliance, some paperwork is unavoidable.

So what is the minimum amount of documentation that you need to satisfy your compliance and governance obligations? And what can you leverage from artifacts that you are already creating?

You will need documented security and risk management policies and guidelines, directional stuff establishing management accountability and clearly communicating responsibilities. And legal protection, such as confidentiality and nondisclosure agreements. All of this will need to be handed out to all staff, signed off, filed, and regularly reviewed and updated.

You will also need to keep a record of requirements so that they can be audited, and to trace changes: you won’t be able to just write stories on index cards or sticky notes pasted on a wall, and then throw them out as they are implemented.

But most compliance procedures and detailed checklists can be, and should be, taken out of documents and pushed into team workflows for developers and operations, and, wherever possible, into code. These should be enforced through rules in your automated build and deployment pipelines, in automated tests and checks, and by taking advantage of the audit trails that are automatically created.

To do this you will need to bring management, compliance, internal audit, the PMO, and the security team together with development and operations.

Compliance rules and control workflows need to be defined up front by all of these stakeholders working together—and any changes to the rules or workflows should be formally approved by management and documented, for example, in a change advisory board (CAB) meeting. Developers and operations need to walk through procedures and checklists with compliance and security and the PMO, map out key controls, and agree on simple ways to automate them. Management needs to understand how operational risks, security risks, and other risks will be controlled and managed through these automated workflows, tests, and pipelines.

Let’s look at how to solve some specific compliance concerns, in code.

Traceability and Assurance in Continuous Delivery

In Chapter 11, we explained how an automated build pipeline works, and how to use continuous integration and continuous delivery tools and workflows to test and deploy changes quickly and safely.

Now you get to use all of this for compliance, tracing each change from when it was requested to when it was delivered, to show that changes are handled properly and consistently. These are the steps that an Agile team could take to prove traceability and assurance in a regulatory environment:

1. As we’ve seen, instead of working from detailed requirements specifications, Agile teams like to write up requirements in simple, concrete user stories on index cards or sticky notes that are thrown away once the story is implemented. But to satisfy compliance, you’ll need to keep a record of each feature and change, who asked for it, who approved it, and the acceptance criteria (i.e., conditions of satisfaction) that were agreed to.

   While you could try to do this using a spreadsheet, for example, teams today can take advantage of Agile project management tools or story trackers like Rally, Version One, or Jira to provide insight into what the team is doing, and what it has already done from an auditing perspective.

2. The team works with compliance to agree on its Definition of Done, including the evidence needed to prove compliance for stories, sprints, and releases.

3. Everything—application code, configuration recipes and templates, tests, policies, and documentation (everything, that is, except for secrets)—is committed to version control, with a tie back to the specific requirement or change request or bug report (using a story ID, ticket number, or some other unique identifying tag that can be referenced as a comment on check-in), so that you have a detailed history of everything associated with each change.

4. Commit filters automatically scan for secrets and unsafe calls before code can be merged in to the mainline.

5. Changes to code and configuration are reviewed—before commit if possible—and the results of reviews are visible to the team, using Git pull requests or a collaborative code review tool like Gerrit or Review Board.

   Reviewers follow checklists to ensure that all code meets the team’s standards and guidelines, and to watch out for unsafe coding practices. Management periodically audits to make sure that reviews are done consistently, and that engineers aren’t rubber-stamping each other’s work.

6. Every change (to application code and to configuration) is tested through continuous integration or continuous delivery: TDD, static code analysis, and other scanning, and automated acceptance testing as described in Chapter 11. Test coverage is automatically measured. Any serious problems cause the build to fail.

   Because tests are checked into a source repo, you can review the tests and match them up against the acceptance criteria for each story, to see if the requirements were implemented correctly.

7. Code (including code dependencies) and infrastructure are regularly scanned for vulnerabilities as part of your automated pipelines, and vulnerabilities found are recorded and pushed into the team’s backlog to be fixed.

8. Changes are deployed automatically to acceptance test, then staging, and, if all tests and checks pass, to production, so that you can show when a change was promoted to each environment, and how this was done.

9. Systems are regularly scanned for unauthorized changes using detective change controls (like Tripwire or OSSEC).

This is a beautiful thing for operations and support. You can tell when changes are made or vulnerabilities are patched, and, if something goes wrong, you can trace exactly what was changed so that you can fix it quickly. It’s a beautiful thing for governance, because you can follow each change and ensure that the changes were made consistently and responsibly. And it’s a beautiful thing for compliance, because you can prove all of this to your auditors.

Managing Changes in Continuous Delivery

Change control is fundamental to compliance regulations, and to governance frameworks like ITIL (the IT Infrastructure Library) and COBIT. This comprises:

• Making sure that all changes are authorized—SOX, insider threats, fraud, Reg SCI

• Minimizing the operational risk of change—making sure that changes are understood, tested, and safe

• Ensuring that changes are auditable, which builds on traceability

Most governance frameworks deal with change management in a bureaucratic, paperwork-heavy way, with forward planning, and formal approvals at change advisory board (CAB) meetings where a committee meets to assess the risks and impact of a change, determine preparedness, and agree to scheduling.

This is in essential conflict with Agile and Lean development, and especially with DevOps and continuous deployment which are predicated on frequent, iterative change, including running A/B experiments in production to get feedback. This could mean making changes several times a week, or several times a day—or at organizations like Amazon, several times per second.

How the heck can you have change control when you are rolling out changes every few seconds?

By taking the risk out of change up front, running every change through the same battery of tests and checks that we described earlier. And by optimizing for small, incremental changes.

While ITIL change management is designed to deal with infrequent, high-risk “big bang” changes, most changes by Agile and DevOps teams are small and low-risk, and can flow under the bar. They can be treated as standard or routine changes that have been preapproved by management, and that don’t require a heavyweight change review meeting.

Many larger changes can also be made this way, using dark launching. This is a practice made famous at Flickr and Facebook, where changes to code are hidden behind a feature flag: a switch that will only be turned on after getting approval. In the meantime, the team can continue to make and test changes in incremental steps, releasing the code to production without impacting operations. This can, in some cases, involve running the new code in simulation mode to collect data on usage and performance, or deploying it to a small community of users for beta testing, until everyone is confident that the feature is ready.

Before turning on the feature, the team members could hold an operational readiness review, or “pre-mortem” review meeting to go over their preparedness, explore failure scenarios and their ability to respond to them, and ensure that everyone is informed of the change in advance.

The risk of change can also be minimized by automating the steps involved, ensuring that changes are tested and deployed consistently and repeatably. All changes—to code and to configuration—should be rolled out using your automated build and delivery pipelines, the same pipelines that you use for testing, to take advantage of the built-in controls, to make sure that the steps have been rehearsed and proven, and to provide full traceability and visibility. Everyone knows what changes were made, when, by who, how they were tested, and how and when they were deployed.

By the time that you’re ready to deploy to production, you’ve already run the change through development, to acceptance testing, and staging, all using the same steps.

In this model, changes become constant, routine, and predictable.

Dealing with Separation of Duties

One concern that needs to be addressed in change management, especially in DevOps environments where engineers can push changes automatically into production using continuous deployment, is separation of duties.

Separation of duties ensures that no single person can have full control from start to end of a change, and that changes cannot be made without testing and approval. This is intended to prevent malicious insider attacks and fraud, and to prevent honest insiders from taking shortcuts and bypassing the checks and balances that are designed to protect the system and the organization from security risks and operational risks.

Separation of duties is spelled out as a practice in governance frameworks like ITIL and COBIT, and it’s expected in other frameworks and regulations like ISO/IEC 27001, SOC 1 and SOC 2, SOX 404, PCI DSS, NIST 800-53, Reg SCI, and others. It ties into change control, as well as data privacy (by limiting the number of people who have access to production data).

Auditors are used to looking for evidence of separation of duties, such as network segmentation between development and production systems, and matrices clearly showing that developers, testers, operations, and support analysts are assigned to different roles, with access restrictions to the different systems and commands.

The most obvious implementation of this principle is the Waterfall/CMMI model which requires a series of documented handoffs between roles:

• Business analysts define requirements, and get approval from the business owner.

• Designers take the requirements and create specifications.

• Developers write code to implement the specifications, and then hand the code off for testing.

• Independent testers verify that the code meets the specifications.

• Operations packages the code into a release, and waits for a change manager to complete an impact analysis and risk assessment, and schedule the change to be deployed.

• All these steps are documented and tracked for management review and sign off.

Auditors like this a lot. Look at the clear, documented handoffs and reviews and approvals, the double checks and opportunities to catch mistakes and malfeasance.

But look at all the unnecessary delays and overhead costs, and the many chances for misunderstandings and miscommunication. This is why almost nobody builds and delivers systems this way any more.

The DevOps Audit Toolkit makes the argument that you don’t need all of these handoffs to prevent fraud and insider maliciousness, and to ensure that all changes are properly authorized, tested, and tracked. You can even empower developers to push changes directly to production, as long as they do the following:

• The team agrees and ensures that all changes meet the Definition of Done, which includes that all changes are tested against the acceptance criteria defined for each story, and successfully reviewed by the Product Owner, who represents the interests of the business and management

• Peer reviews (or pair programming) ensure that no engineer (dev or ops) can make a change without at least one other person in the organization being aware of it and understanding it. You could even insist that the team assign reviewers randomly to prevent collusion.

• Every change—to code or configuration—is made through automated build and deployment pipelines, which ensure that they are tested and tracked.

• Developers can be given read-only access to production system logs and monitors so that they can help in troubleshooting. But any fixes or patches need to be made through the automated build pipeline (fixing forward) or by automatically rolling changes back (again, through the same automated pipeline).

• All changes made through the pipeline are done in a transparent way, logged and published to dashboards, chat rooms, and so on.

• Production access logs are regularly reviewed by management or compliance.

• Access credentials are reviewed regularly: including access to different environments, access to repos, and access to the pipelines and configuration management tools.

• Automated detective change control tools (such as Tripwire, OSSEC, AIDE, and UpGuard) are used to alert on unauthorized changes to the build environment and to production systems. If you are deploying changes a few times per month or even several times per week, this is straightforward. If you are making changes multiple times a day, you need to be careful to filter out approved, automated changes to show the exceptions. It’s also important that these alerts are sent to someone outside of the engineering team, to security or compliance or management for review, to ensure that there is no conflict of interest.

For an informed and balanced auditor’s perspective on separation of duties in DevOps environments, see Douglas Barbin’s article, “Auditing DevOps – Developers with Access to Production”.

Keeping Auditors Happy

To make auditors happy, you need to provide them with evidence that you’ve met specific compliance requirements, or evidence that proves compliance with defined outcomes.

Just as beauty is in the eye of the beholder, compliance is in the opinion of the auditor. Auditors might not understand the approach that you are taking at first, especially auditors who are accustomed to reviewing detailed policies and procedures, and making people fill out checklists and spreadsheets.

You will need to explain how your pipelines work, walk them through your controls, and through the code and repos and tests and audit logs, and show how it all ties together. But a good auditor should be able to appreciate what you are doing, and recognize how it is good for you, as well as for them.

If you follow the approach outlined here, using your automated build pipelines as a control plane for compliance, you can prove that you are regularly scanning and reviewing code and infrastructure for vulnerabilities, and you can track when vulnerabilities were found and fixed.

You can provide a complete audit trail for every change, from when the change was requested and why, who authorized the change, who made the change and what that person changed, who reviewed the change and what they found in the review, how and when the change was tested and what the results were, to when it was deployed:

• You can prove that changes were reviewed and tested, and prove that changes were all made in a standardized, repeatable way.

• You can show that compliance policies and checks were enforced in reviews, in testing, in scanning, and in release control.

• You can show that you’ve enforced separation of duties between development and operations.

• And, if you are following the controls consistently, you can prove that all of this was done for every change.

In the same way that frequently exercising build and deployment steps reduces operational risks, exercising compliance on every change, following the same standardized process and automated steps each time, minimizes the risks of compliance violations. You — and your auditors — can be confident that all changes are made the same way, that all code is run through the same tests and checks, and that everything is tracked the same way, from start to finish.

Auditors can verify that your control program is consistent, complete, repeatable, and auditable. We can see the smile on your auditor’s face already.

Dealing with Auditors When They Aren’t Happy

Your auditors won’t always be happy, especially if they are coming in to investigate after a security breach.

Although many regulations lay out reporting requirements in the event of a breach, it’s not always clear what your priorities should be, especially when you are in the middle of dealing with a security incident. There are a lot of factors to understand and consider.

How serious is the breach? Different agencies have different bars for what you need to report, and when. What if you fall under several different compliance and legal jurisdictions?

Who do you need to report to first? Legal counsel? Your Board? A forensics specialist? Law enforcement? Your partners and customers? Your insurance agent? The regulators? Government agencies? Who do you contact? What information do you need to provide, and how quickly?

You should have all of this worked out in advance as part of your incident response playbooks.

Once you’ve dealt with disclosure, you need to prepare for the follow-up analysis and audit(s) to understand what went wrong, how badly it went wrong, who will be held responsible or accountable, and how much it will cost to make up for what went wrong. If you experience a serious breach, and you work in a regulated environment, your organization will by definition be found noncompliant—after all, if you were 100% compliant, how could anything have possibly gone wrong? An investigation just has to prove this, and with the power of perfect hindsight, it will. After a breach or a serious compliance failure, everyone will be trying to find the smoking gun, and they will keep looking for mistakes and gaps and scapegoats until they have enough smoke to fill out their report.

Repeatable, automated workflows with built-in audit trails, and evidence checked into version control, will at least make this less painful and less expensive for you and for the investigators, and help to show that you were doing some (hopefully most) things right.

Continuous Compliance and Breaches

In compliance environments, we talk about certification and attestation as if they are the end goal of the compliance process. In some respects, however, they are just the beginning. The aim of compliance schemes is to ensure that the organization is continuously compliant. Each of the requirements (at least in prescriptive, rules-based regulations) are designed to be recurring and present in all day-to-day activities. This is why most certifications or attestation reviews are done across a period of time, such as a year, to verify that your controls were consistently applied.

This moves compliance from being a moment in time, auditable requirement to a continuous compliance need. By using the automation suggestions in this chapter, your team and organization can verify its continued compliance and identify issues early. While this may not prevent a compliance breach from happening, it will mean that if the auditors are sent to your door, you will have evidence of your processes and approaches up to and at the time of the breach, not just on your last attestation or certification audit.

Certification Doesn’t Mean That You Are Secure

Certification or a successful attestation report does not mean that your organization is compliant. It means that you’ve satisfied a series of conditions that a regulator would reasonably expect, using a recognized and validated approach. A regulator could still find you in breach of compliance for some reason, although you’ve reduced your risk significantly.

Certification does not mean that you are secure. Several organizations that held certifications or passed all of their audits still suffered from high-profile breaches or serious operational failures. Just because you’ve met a certification qualification doesn’t mean that you can stop looking, learning, and improving.