Keeping card data out of your error reports

Where card data gets in

Payment code handles a short list of very sensitive values: the card number (the primary account number, or PAN), the security code (CVV or CVC), the expiry date, bank account and routing numbers, IBANs, and the API keys that move money. Any of them can end up in an error report, a log line or a breadcrumb:

• Exception messages: a validation error that quotes the number it rejected, or a processor's error re-raised with the request that caused it
• Context you attach: the whole payment object, or the params hash, added to make debugging easier
• Request data: a form or JSON body that posts card fields to your own server
• Breadcrumbs and logs: a console.log of a payment method in the browser, an HTTP client logging request bodies, a job queued with card details as its arguments
• URLs: a token or an account number in a query string or a webhook path
• Secrets: "Invalid API key provided: sk_live_..." in a message, or a configuration dump at boot

The most effective fix sits upstream of all of them: if card numbers never reach your servers, no error report from your servers can contain one.

PCI DSS scope, carefully

PCI DSS applies to the systems that store, process or transmit cardholder data (the PAN, and the cardholder's name, expiry date and service code when they're kept with it) and to the systems connected to them. Sensitive authentication data, which includes the full security code and the data on the magnetic stripe or chip, may not be stored after authorization at all, even encrypted.

So a card number in an error report isn't only a leak to tidy up. A copy of cardholder data now sits in your error tracker, in its notifications and anywhere the report was pasted, and those can be pulled into your scope. A security code in one is data that should never have been stored anywhere.

What's in scope for you depends on your own environment, and it's settled with your assessor, not by any tool. ForgeOps makes no PCI DSS claim of its own, and no ForgeOps setting takes a system out of scope by itself. What its settings can do is make it much less likely that card data reaches it at all, and the rest of this guide is how.

Keep the PAN off your servers

Most payment processors offer hosted payment fields or a hosted checkout: the card form is served by the processor, often inside an iframe, and your server receives a token instead of the number. Your code charges the token, and a failure is about a token, a decline code and an amount, none of which is cardholder data.

Write your own messages the same way. Name a payment by its token or your own id, and a card by its last four digits, the usual way to identify one without the number. Never echo back what the customer typed:

Ruby

# Before: the number is the message, and the message is in every copy of the report.
raise PaymentError, "Card #{params[:card_number]} declined: #{response.body}"

# After: what you need to debug, and none of it is cardholder data.
raise PaymentError, "Charge declined (#{response.decline_code}) for payment method " \
  "#{payment_method.id}, card ending #{payment_method.last4}"

What ForgeOps scrubbing catches

Every event ForgeOps receives is scrubbed twice: by the SDK before it leaves your process, and again by ForgeOps on arrival, before the event is queued or stored. Scrubbing covers the message, the stack trace, context, tags, breadcrumbs, and the masked SQL statement if you've opted in to sending one. For payment data it uses these signals:

• Field names. A value is replaced with [FILTERED] when its key, lowercased and with everything but letters and digits removed, contains a name from a fixed list, including creditcard, cardnumber, cardnum, cvv, cvv2, cvc, iban, accountnumber, routingnumber, bankaccount, password, secret, token, apikey and authorization. So card_number, cardNumber, Credit-Card, billing_iban, accountNumber, routing_number and bank_account_id all match, along with everything nested under them.
• Card numbers in text, however they're grouped. A run of 13 to 19 digits, written bare or split by single spaces or dashes (4242424242424242, 4242 4242 4242 4242, or American Express's 3782 822463 10005), is replaced when it starts with a card network's prefix and passes the Luhn check, the check digit every card number carries. A card followed by its security code loses only the card number, and several cards in a row are each found. A number written in four groups of four is replaced whether or not its check digit works out.
• IBANs in text, written compact or in groups of four (DE89370400440532013000, or GB82 WEST 1234 5698 7654 32), when the IBAN's own check digits pass the mod 97 check.
• Secrets in text: Stripe secret and restricted keys (sk_live_, sk_test_, rk_live_ or rk_test_ followed by at least ten letters or digits), AWS access key ids, GitHub tokens, JWTs and Bearer tokens, as well as email addresses and formatted SSNs.

Ruby: the card number, IBAN and Stripe key rules and the field names, runnable on their own

# The rules ForgeOps scrubbing uses for card numbers, IBANs and Stripe keys, and its field names.
SENSITIVE_KEYS = %w[
  password passwd pwd
  secret apisecret clientsecret secretkey
  token accesstoken refreshtoken apikey apitoken authorization authtoken bearer sessiontoken csrftoken
  creditcard cardnumber cardnum cvv cvv2 cvc
  iban accountnumber routingnumber bankaccount
  ssn socialsecuritynumber socialsecurity
  privatekey
].freeze

IBAN = /(?<![A-Za-z0-9_])[A-Z]{2}[0-9]{2}(?:[A-Z0-9]{11,30}|(?: [A-Z0-9]{4}){2,7}(?: [A-Z0-9]{1,4})?)(?![A-Za-z0-9_])/
FORMATTED_CARD = /\b\d{4}[ -]\d{4}[ -]\d{4}[ -]\d{1,4}\b/
STRIPE_KEY = /\b[sr]k_(?:live|test)_[A-Za-z0-9]{10,}\b/
DIGIT_RUN = /(?<![A-Za-z0-9_])\d(?:[ -]?\d)*(?![A-Za-z0-9_])/
CARD_PREFIXES = /\A(?:4|5[1-5]|2(?:22[1-9]|2[3-9]\d|[3-6]\d\d|7[01]\d|720)|3[47]|6011|64[4-9]|65|35(?:2[89]|[3-8]\d)|36|30[0-5]|62)/

def luhn?(digits)
  sum = digits.reverse.each_char.with_index.sum do |char, index|
    value = char.to_i
    next value if index.even?

    doubled = value * 2
    doubled > 9 ? doubled - 9 : doubled
  end
  (sum % 10).zero?
end

def card_number?(digits)
  digits.length.between?(13, 19) && digits.match?(CARD_PREFIXES) && luhn?(digits)
end

# Move the first four characters to the end, turn each letter into two digits (A is 10), and a
# valid IBAN leaves a remainder of 1 when divided by 97.
def iban?(candidate)
  compact = candidate.delete(" ")
  return false unless compact.length.between?(15, 34)

  (compact[4..] + compact[0, 4]).gsub(/[A-Z]/) { |letter| (letter.ord - 55).to_s }.to_i % 97 == 1
end

# A run of digits joined by single spaces or dashes, in groups. From each group, the longest
# stretch of whole groups that is a card number is replaced, so a card followed by its security
# code, or two cards in a row, are each found.
def scrub_card_numbers_in(run)
  groups = run.split(/[ -]/)
  separators = run.scan(/[ -]/)
  out = +""
  i = 0
  while i < groups.length
    last = (groups.length - 1).downto(i).find { |j| card_number?(groups[i..j].join) }
    if last
      out << "[CREDIT CARD FILTERED]" << separators[last].to_s
      i = last + 1
    else
      out << groups[i] << separators[i].to_s
      i += 1
    end
  end
  out
end

def scrub_text(text)
  text = text.gsub(IBAN) { |candidate| iban?(candidate) ? "[IBAN FILTERED]" : candidate }
  text = text.gsub(FORMATTED_CARD, "[CREDIT CARD FILTERED]").gsub(STRIPE_KEY, "[STRIPE KEY FILTERED]")
  text.gsub(DIGIT_RUN) { |run| scrub_card_numbers_in(run) }
end

def sensitive_key?(key)
  normalized = key.to_s.downcase.gsub(/[^a-z0-9]/, "")
  SENSITIVE_KEYS.any? { |sensitive| normalized.include?(sensitive) }
end

[
  "Card 4242424242424242 was declined",
  "Card 4242 4242 4242 4242 was declined",
  "Amex 3782 822463 10005 failed",
  "Card and CVV 4242424242424242 123",
  "Retried 4242424242424242 4000056655665556",
  "Couldn't find Invoice with id=88214451",
  "Order 1234567890123456789 not found",
  "Retry scheduled for 1729700000000",
  "Card 4242424242424241 was declined",
  "Card 4242.4242.4242.4242 was declined",
  "Payout failed for DE89370400440532013000",
  "IBAN GB82 WEST 1234 5698 7654 32 rejected",
  "Invalid API key provided: sk_live_51Hx9aQ2eZvKYlo2C"
].each { |message| puts scrub_text(message) }

%w[card_number cvc stripe_secret billing_iban accountNumber routing_number bank_account_id last4 account_name].each do |key|
  puts "#{key}: #{sensitive_key?(key) ? 'redacted' : 'kept'}"
end

Each of the six card numbers in the first five messages becomes [CREDIT CARD FILTERED], with the 123 after one of them left as it was. The next five are sent as they are: the invoice id is too short to be a card number, the 19 digit order number and the timestamp start with 1, which no card network uses, 4242424242424241 fails the Luhn check, and a card number written with dots isn't recognized. Both IBANs become [IBAN FILTERED] and the Stripe key [STRIPE KEY FILTERED]. Of the field names, everything is redacted except last4 and account_name.

The prefix and the check digit are a trade-off. Error messages are full of long numeric ids and timestamps ("Couldn't find Invoice with id=88214451"), and they're what you read to debug, so a rule that redacted every long number would make reports worse at their one job. Requiring a network's prefix and a valid check digit keeps nearly all of them readable, since an id has to start like a card number and also pass a check that only one random number in ten passes. Now and then a long id does both and is redacted.

Some things aren't recognized in text at all. Bank account and routing numbers are plain digits with no check that tells them from an id, so they're caught under a field name such as account_number, never in a sentence. A card number written with other separators, such as dots or slashes, or with more than one space between its groups, is sent as it is, and so is a security code on its own. Treat scrubbing as a net for mistakes, not as the way card data is kept out.

Add your own field names

The built-in list already covers iban, account_number, routing_number and bank_account. For fields it doesn't name, add them under Additional sensitive fields in the project's Privacy settings, one per line. ForgeOps then redacts any context, tag or breadcrumb value whose key contains one of them, when the event arrives and before it's queued or stored. The SDK's own pass uses only the built-in list, so these values still leave your app; the setting keeps them from being stored.

Matching is by substring, the same as for the built-in names, so choose names that can't hide inside innocent ones. sort_code and tax_id are safe choices, while pan would also redact company and expand. Each name needs at least three letters or digits.

Ruby

begin
  Payouts.send_to_bank!(payout)
rescue Payouts::Rejected => e
  ForgeOpsTracker.capture_exception(e, context: {
    payout_id: payout.id,                  # kept
    decline_code: e.code,                  # kept
    card_number: payout.card_number,       # a built-in name: [FILTERED]
    billing_iban: payout.iban,             # a built-in name: [FILTERED]
    account_number: payout.account_number, # a built-in name: [FILTERED]
    sort_code: payout.sort_code            # [FILTERED] once sort_code is an additional field
  })
  raise
end

Better still is not to attach those values at all. A payout id lets you find the account when you need it, with your own access controls in the way.

Breadcrumbs and the browser

Breadcrumbs are scrubbed like the rest of the report, their message and their data. In the browser they record console output, navigations, clicks and the method and URL of each fetch or XHR request, never request or response bodies. The risk is console output: a console.log of a payment method or a form's values while someone debugged the checkout puts them in the trail, in a shape the patterns may not recognize. Remove those calls, or turn breadcrumbs off in the bundle that serves your payment pages:

TypeScript, in the browser

import * as forgeOpsTracker from "@forge-ops/tracker-web";

forgeOpsTracker.init({
  dsn: "https://<api_key>@getforgeops.net/api/v1/events",
  breadcrumbs: false, // default true: console output, navigations, clicks, requests
});

Server SDKs record breadcrumbs too, such as the queries, controller actions and outbound calls in a Rails request, and turn them off with track_breadcrumbs (trackBreadcrumbs in Node).

The same rule for your own logs

Your logs aren't scrubbed by anyone but you, and the rule ForgeOps uses for card numbers works there too: 13 to 19 digits, bare or split by single spaces or dashes, with a card network's prefix and a valid Luhn check digit. Here it is as a Python logging filter that keeps the last four digits:

Python, standard library only

import logging
import re
import sys

# Digits joined by single spaces or dashes, not touching a letter, digit or underscore.
RUN = re.compile(r"(?<![A-Za-z0-9_])[0-9](?:[ -]?[0-9])*(?![A-Za-z0-9_])")
# The card networks' leading digits: Visa, Mastercard, Amex, Discover, JCB, Diners, UnionPay.
NETWORK = re.compile(r"4|5[1-5]|2(?:22[1-9]|2[3-9][0-9]|[3-6][0-9]{2}|7[01][0-9]|720)|3[47]|6011|64[4-9]|65|35(?:2[89]|[3-8][0-9])|36|30[0-5]|62")


def luhn_valid(digits):
    total = 0
    for index, char in enumerate(reversed(digits)):
        digit = int(char)
        if index % 2 == 1:
            digit *= 2
            if digit > 9:
                digit -= 9
        total += digit
    return total % 10 == 0


def is_card_number(digits):
    return 13 <= len(digits) <= 19 and NETWORK.match(digits) is not None and luhn_valid(digits)


def redact_run(match):
    parts = re.split(r"([ -])", match.group(0))
    groups, separators = parts[0::2], parts[1::2] + [""]
    out, i = [], 0
    while i < len(groups):
        # The longest stretch of whole groups from here that is a card number, so a card
        # followed by more digits, or two cards in a row, are each found.
        last = next((j for j in range(len(groups) - 1, i - 1, -1) if is_card_number("".join(groups[i:j + 1]))), None)
        if last is None:
            out.append(groups[i] + separators[i])
            i += 1
        else:
            digits = "".join(groups[i:last + 1])
            out.append("[CARD ...{}]".format(digits[-4:]) + separators[last])
            i = last + 1
    return "".join(out)


def redact_card_numbers(text):
    return RUN.sub(redact_run, text)


class CardNumberFilter(logging.Filter):
    def filter(self, record):
        record.msg = redact_card_numbers(record.getMessage())
        record.args = ()
        return True


handler = logging.StreamHandler(sys.stdout)
handler.addFilter(CardNumberFilter())
logging.basicConfig(level=logging.INFO, handlers=[handler], format="%(levelname)s %(message)s")
log = logging.getLogger("payments")

log.warning("Refund failed for cards %s", "4242424242424242 4000056655665556")
log.warning("Charge failed for card %s", "3782 822463 10005")
log.info("Invoice %s paid", "1234567890123")

It logs [CARD ...4242] [CARD ...5556] and [CARD ...0005] for the cards and leaves the invoice number alone, since no card network's numbers start with 1. Now and then a long id that starts like a card and passes the check will be redacted too, which in a log is a fair price. A security code has no check digit to recognize it by, so don't log one at all. The same function is useful anywhere you build a message from values you don't control, such as before re-raising a processor's error.

API keys and secrets

A leaked secret key is a different problem from a leaked card number: it can move money until it's rotated. ForgeOps scrubbing replaces Stripe secret and restricted keys, AWS access key ids, GitHub tokens, JWTs and Bearer tokens wherever they appear in text, and any value under a key containing secret, token, apikey, authorization or password. Other providers' key formats aren't recognized in free text, so one that turns up in a message is sent as it is. A webhook signing secret is caught under a key like webhook_secret, but not in a sentence.

If a live key ever appears in an error report, rotate it. Deleting the report doesn't remove the copies in notifications, chat and email.

When nothing should leave: strict privacy

For a payments service where no message, path or context can be trusted, turn on strict privacy. The SDK then sends only structured facts about each error: its type, stack trace, release, environment and route, and the user's id. Messages, breadcrumbs, raw paths, SQL statements and source lines are never sent, and context and tags only under the field names you allow, with plain values. Turn it on in the SDK and under Privacy in the project's settings, which applies the same rules again on arrival.

JavaScript, Node

import * as forgeOpsTracker from "@forge-ops/tracker";
import { processor } from "./processor.js";

forgeOpsTracker.init({
  dsn: process.env.FORGE_OPS_DSN,
  strictPrivacy: true,
  privacyAllowedFields: ["error_code", "decline_code"],
});

export async function capturePayment(payment) {
  try {
    await processor.capture(payment.intentId, payment.amountCents);
  } catch (err) {
    forgeOpsTracker.captureException(err, {
      error_code: "CAPTURE_FAILED", // allowed: sent
      decline_code: err.code,       // allowed: sent, when it's a plain value
      card_last4: payment.last4,    // not allowed: never sent
    });
    // err.message is never sent either: a fingerprint of it groups the error.
    throw err;
  }
}

Errors still group, through a one-way fingerprint of the message computed in your process, and deploy correlation, likely cause and incident detection work as before. What you give up is the message text, the breadcrumbs and session replays.

Session replay on checkout pages

If you record sessions, the checkout is the page to think about. The ForgeOps browser recorder masks the value of every text input and textarea with one fixed placeholder, whatever was typed and however long it was, every character of text on the page with a bullet, text-bearing attributes such as placeholder and aria-label, and which boxes are ticked and which option is chosen. Every address it records keeps only its scheme, host, port and path, so a token in a query string isn't recorded. It doesn't look inside iframes, so a processor's hosted card fields aren't recorded at all.

Don't add data-forgeops-unmask to a payment form or anything around it. An unmasked element shows its fields' value attributes, and some frameworks, React among them, keep that attribute in step with what's typed. Session replay is off unless you turn it on, it's on the Enterprise plan, and nothing is recorded under strict privacy.