Perl error tracking

Know when your Perl app breaks, before your users do

ForgeOps connects the errors your Perl app reports to the deploys, traces, and changes around them, so when production breaks, the incident already says who's affected, what changed, and where to look.

A PSGI middleware or a Dancer2 plugin reports anything that escapes a route, no further wiring needed. Running on ForgeOps' own dedicated infrastructure, so nothing about a Perl app's errors or its users' data gets routed through a third-party error-tracking vendor. See an incident investigated →

Install

cpanm ForgeOps::Tracker

use ForgeOps::Tracker;
ForgeOps::Tracker::init(dsn => 'https://<api_key>@getforgeops.net/api/v1/events');

A PSGI middleware or a Dancer2 plugin reports anything that escapes a route, no further wiring needed.

Every client reports the same shape

Wherever it comes from, an event arrives with an exception class, a message, and a backtrace, plus whatever environment/release/server context that client can gather on its own. ForgeOps fingerprints and groups on the first three, so a Perl app's issues list works exactly the same way every other language's does: report the same bug a thousand times and it's still one row, not a thousand.

No agent, no sidecar: it's a plain HTTPS POST to ForgeOps' own ingestion API, so nothing about a Perl app's traffic gets routed through a third-party pipeline first.


Works with your framework


The Perl client hooks in at the framework level, not by wrapping individual calls, so nothing about how you already write Perl code has to change.


PSGI / Plack

enable '+ForgeOps::Tracker::Integrations::PSGI' wraps any PSGI-speaking app, framework or not.

Dancer2

A plugin hooks Dancer2's own on_route_exception, before Dancer2 renders its own error page.

Plain scripts

Report explicitly at the catch site: ForgeOps::Tracker::report($@, context) after an eval block. Whatever's still queued is sent when the script ends (waiting at most 5 seconds); call ForgeOps::Tracker::flush($timeout) before POSIX::_exit or exec.

What it looks like

The issue list

Every Perl issue reported into a project shows up here: title, status, assignee, event count, and when it was last seen, the same columns as any other language's project.

The issue list for a project reporting from a Perl app

The issue detail

Every occurrence keeps its own timestamp, environment, release, and server, plus the backtrace exactly as Perl reported it, expandable per occurrence rather than flattened into one generic stack.

A Perl issue's page in ForgeOps: its title, the exception class and the code it came from beneath it, then its verdict with its failure count, customers affected, and when it was first seen, above the triage buttons

Alerting

A new Perl issue, a regression, or a spike past a threshold you set can reach Slack, Microsoft Teams, email, PagerDuty, Opsgenie, or a generic webhook, configured per project, per trigger.

Notification rules configured per trigger and channel, e.g. a new issue to Slack, a regression to email

Heartbeats

For the failure mode error tracking can't see on its own: a Perl cron job or recurring task that's supposed to run but silently stopped. Each heartbeat gets its own ping URL and its own grace period before it alerts.

A heartbeat's ping URL and status, showing its expected interval, grace period, and last check-in

Performance monitoring

Two more integrations, one per framework, time every request too: the Dancer2 plugin reports the matched route's own pattern (e.g. "GET /users/:id", so a distinct user id doesn't explode into its own separate transaction); plain PSGI has no route-matching concept of its own, so that middleware reports the raw request path instead, and a Dancer2 request that never matches any route (a 404) isn't recorded at all, since Dancer2's own after_request hook never fires for that case. Counted in-process and flushed as a small periodic aggregate on a background thread. Requires a plan that includes performance monitoring; on a plan that doesn't, the periodic reports are accepted but not recorded (the response says why), so a working setup never looks broken. Each report also carries a small latency histogram (version 0.7.0 and later), so ForgeOps shows an approximate p50/p95/p99 per transaction, not just an average, accurate to the width of the latency bucket a duration falls into.

The Performance page, showing each transaction's request count and p50, p95 and p99 latency, plus slow queries
# PSGI / Plack
enable '+ForgeOps::Tracker::Integrations::PSGIPerformance';

# Dancer2
use ForgeOps::Tracker::Integrations::Dancer2Performance;

ForgeOps::Tracker::init(
    dsn                        => '...',
    track_performance          => 0,  # opt out entirely
    performance_flush_interval => 30, # seconds; default 60
);

Distributed tracing

With the PSGI or Dancer2 performance middleware enabled, each request becomes a trace whose root span is the request itself; add the rest by hand. Only a slow request's trace is ever sent: whether it crossed the threshold (one second by default, configurable) is decided entirely inside your process, so a fast request costs nothing over the wire. Requires a plan that includes distributed tracing.

A trace's waterfall for one slow request, with its controller, service, database, cache and HTTP spans
my $order = ForgeOps::Tracker::span('charge card', sub { $gateway->charge($id) },
    kind => 'service', data => { order_id => $id });
ForgeOps::Tracker::span('fetch rates', sub { $http->get($url) }, kind => 'http');

# Something you timed yourself (kind is one of controller/service/database/redis/http/job/other;
# $started_at is Time::HiRes::time):
ForgeOps::Tracker::record_span('SELECT orders', 'database', $started_at, $duration_ms);

Errors now carry the request they happened in: its trace id, and under Dancer2 its endpoint (POST /checkout) too. Each occurrence links to its exact trace, and a request that fails is always traced, however fast. With PSGIPerformance or Dancer2Performance, a request that arrives with the standard traceparent header continues the caller's trace on its own; pass it on by wrapping an outbound call in http_span, which records it as an http span and hands you the headers to add, with any HTTP client. Once the projects are linked in ForgeOps an error shows the errors another project raised in the same request. Needs ForgeOps::Tracker 0.9.0.

One trace waterfall running from a mobile app's checkout tap into the API it called, with the API's controller, database, and payment calls nested underneath, each project labeled
my $response = ForgeOps::Tracker::http_span(POST => $url, sub {
    my ($headers) = @_; # { traceparent => '00-...' }
    $http->post($url, { headers => { %$headers, 'Content-Type' => 'application/json' }, content => $body });
});

ForgeOps::Tracker::init(
    dsn                       => '...',
    trace_propagation_targets => ['api.example.com'], # only these hosts; undef (the default) means all
);

SQL in a request

When the request an issue failed in was traced, the issue opens with its slowest query: how long it took, its share of the request's time, how many queries the request ran, and a likely N+1 warning when the same statement ran five or more times. DBI isn't instrumented automatically, so wrap a query in a database span and pass its SQL as statement; bind values are never read. The statement is masked before it leaves your process, every string and number replaced with a question mark, and the same SQL shows under that span's bar in the trace's waterfall. Needs ForgeOps::Tracker 0.11.0, on a plan that includes distributed tracing.

An issue's Slowest query in this request card: a 2.7 second order search that took 86% of a 3.2 second request, a likely N+1 warning for a line_items query run 24 times, and the query plan calling out a sequential scan on orders
my $sql = "SELECT id FROM orders WHERE customer_id = ? AND status = 'open'";

my $ids = ForgeOps::Tracker::span('Load open orders',
    sub { $dbh->selectcol_arrayref($sql, {}, $customer_id) },
    kind => 'database', statement => $sql, db_system => 'postgresql');
# Sent as "SELECT id FROM orders WHERE customer_id = ? AND status = ?"

What changed

A feature flag flip or a config edit can break things with no deploy at all. Record one with record_change and it shows under What changed on any issue that starts in the next two hours, and on the project's Changes page beside your deploys, labeled potentially relevant, never as the cause. init() also queues the Perl version once per process, so a Perl upgrade shows up with nothing to call; module versions are left out, since Perl has no single reliable record of which ones an app runs. Environment variable names (never values) are opt-in. Needs ForgeOps::Tracker 0.10.0, on a plan that includes change tracking.

An issue's verdict listing what changed just before it started, labeled potentially relevant: the gateway_retry_v2 flag turned on for 100% of checkouts 3 minutes before, and a stripe upgrade and a new environment variable detected at startup after the deploy 7 minutes before
ForgeOps::Tracker::init(
    dsn                 => $ENV{FORGE_OPS_DSN},
    detect_changes      => 1,   # the default; 0 sends no startup snapshot
    track_env_var_names => 1,   # default 0; names only, never values
);

ForgeOps::Tracker::record_change(
    kind    => 'feature_flag',
    title   => 'gateway_retry_v2 turned on for 100% of checkouts',
    details => { flag => 'gateway_retry_v2', from => '10%', to => '100%' },
    actor   => 'priya@example.com',
);

Custom metrics and infrastructure monitoring

Track a business event you name yourself (a signup, a payment) or a reading from one of your own hosts, with one explicit call each; nothing is automatic. The value defaults to 1, so a bare call is a counter; pass a real amount for anything else (it can be negative, for a refund). A captured value is stored as its own row, so counts and sums you compute later are exact. Calls are buffered and flushed as one batch in the background, so they are safe to make inside a request, and flush_metrics sends whatever is buffered right now. The hostname of an infrastructure reading defaults to the configured server name. Requires a plan that includes custom metrics and infrastructure monitoring.

A dashboard with custom-metric widgets (orders placed, checkout conversion, orders over time) and infrastructure widgets (average readings by host and by metric)
ForgeOps::Tracker::capture_metric('signup');          # value defaults to 1: a bare counter
ForgeOps::Tracker::capture_metric('payment', 49);     # a real magnitude; it may be negative (a refund)

ForgeOps::Tracker::capture_infrastructure_metric('cpu', 0.42);                    # hostname defaults to server_name
ForgeOps::Tracker::capture_infrastructure_metric('disk', 0.81, hostname => 'db-1');
ForgeOps::Tracker::flush_metrics();                                                # optional: send right now

Database errors

When an error comes from a database call, the issue shows which stored procedure, table or view its SQL touched, so you know where to start looking. DBI puts the statement in the error text when the handle has ShowErrorStatement on (DBIx::Class turns it on for you), and this client reads it from there, never the bound values. The names are sent by default and are identifiers, never values. The SQL statement itself is opt-in, with every string and number replaced by a question mark before it leaves your app, and a project setting can stop ForgeOps storing the statement at all.

An issue occurrence's Database section, naming the stored function and the view the failing query touched, with the statement's values replaced by question marks
my $dbh = DBI->connect($dsn, $user, $pass, { RaiseError => 1, ShowErrorStatement => 1 });

# Opt in to also sending the masked statement (default 0).
ForgeOps::Tracker::init(dsn => '...', capture_sql_statement => 1);

Questions

Do I need to change how I already handle errors?

Almost never. Every client hooks the framework's own exception handling directly, the same way the install snippet above shows, so your own logs, error pages, and rescue blocks keep working exactly as they did before, and ForgeOps just also finds out about it. The one common exception is a background-job runner that doesn't forward through that same mechanism on its own; that needs one small hook added instead, not a change to anything already there.

Where does the data actually go?

Straight to ForgeOps' own ingestion API over plain HTTPS. No agent, no sidecar, and no third-party ingestion service in between, so nothing about your app's traffic or its users' data gets routed through anyone else's pipeline first.

What happens if the same bug fires a thousand times?

It's fingerprinted from its exception class, message, and backtrace, so a thousand reports of the same bug still show up as one issue with an event count of 1,000, not a thousand separate rows to dig through. And a runaway loop can't use up your monthly event quota: once one issue repeats faster than your plan's hourly limit (50 an hour on Free), further repeats are still counted, and still count toward spike alerts, but they aren't stored or charged to your quota.

Is anything scrubbed before it's stored?

Yes. Emails, credit card numbers, and known API key/token formats are redacted out of every event before it's even written, on every plan, with room for per-project custom field names on top of the defaults.

Try it with your own Perl app

Free plan included, no credit card required. New organizations start with 14 days of Business.

Get started free