By now, you’ve possibly heard about what people are calling The Hackening (opens in new tab). Due to major advances in generative AI, cyberattacks are skyrocketing. Our entire world is interconnected and we’re facing a massive problem where zero-day exploits are being found daily, with the recent PHP flaw being particularly vicious (opens in new tab). These are flaws being exposed by people looking for them. I’m finding them by accident.
For the publicly reported exploits:
- A hook in Dancer2 that refuses your request might still allow the request to run (opens in new tab).
- In some configurations, Dancer2 allowed attackers to read any file on your system (opens in new tab).
- In Mojolicious, improper data sanitization could allow XSS (opens in new tab).
Unfortunately, if you check the GitHub Advisory Database for my contributions (opens in new tab), none of them appear. I believe that’s because Perl isn’t an officially supported ecosystem on GitHub, so my page is blank.
Those are the reported exploits I’ve found. Two other teams (one Perl, one not) have pushed back on my findings. And for private repos I’ve been examining, I’ve been finding even more. But this has all been a (un)happy accident.
When working with legacy code, after you clean up your steering files, there’s a good chance you need a significantly improved test suite. I generally find a strong correlation between a solid test suite and solid code, but when I trained teams on AI and told them to build out test suites, they didn’t. It seems like a lot of work and most teams, unlike me, don’t have 25 years of experience writing automated tests.
My training wasn’t providing as much value as I wanted. So I fixed it.
PAAD (opens in new tab) allows you to write production-quality code
with AI, by using engineering-led AI, instead of AI-led engineering. It
now ships a
/test-roadmap (opens in new tab)
skill that is dead simple to use: just keep spamming /test-roadmap in fresh
sessions until it’s done.
Amongst other things, it uses failure injection and mutation testing to avoid test theater (opens in new tab). It builds e2e tests for critical flows, unit tests, and integration tests to validate that those units can talk to each other. Generally, I find that test suites wind up with roughly 85% statement coverage when the roadmap is finished.
Because Anthropic (opens in new tab) doesn’t want you using Claude
for bad stuff, when you use their most powerful models on cybersecurity, they
often bump you down to a less powerful model. And I keep getting bumped down to
Opus 4.8 when running /test-roadmap because my “no test theater” tests
keep finding zero-day exploits in the code, and then writing tests to validate
that those exploits are real. I didn’t intend this. I’m not a security
researcher, but having more rigorous tests naturally leads to this end result. I
have also changed public GitHub repositories to private when I kept finding
these holes.
The problem is so severe that PAAD v2 now writes security issues to a gitignored directory. You don’t want to overlook those files and push them to a public repo.
If I’m finding these by accident, bad guys are finding them on purpose, and they’re not contacting the maintainers. They’re using these exploits. Or selling them. And it’s probably just going to get easier as AI gets more powerful.
That aside, if you are using AI to help you write code for legacy systems,
fix your steering and then build your test
suite with /test-roadmap. You’ll be amazed (and perhaps horrified) at what it
turns up.
Side note: As of this writing, PAAD is now being cloned about 80 times a day,
and recently a large Brazilian company let me know they’re using
/test-roadmap to build out test suites for many of their products. Sooner or
later, I need to find companies willing to publicly endorse PAAD. They only
tell me privately.