PaperCut security post-incident report from April 2023

In April of this year we had a significant security incident in PaperCut MF/NG that affected our customers. In this post today, I want to be courageous, live our company values, and transparently share what happened, what we’ve learned, and what our plans are to continuously improve our security posture for the future.

I personally learned a lot during this time, and I hope that the transparent approach also helps others get value from our experience too, such as other software vendors in our industry, our partners and resellers, and our customers alike.

Before we break into the details, as CEO and Co-founder, and as a Software Engineer who actually contributed to the code that caused the problem, I want to acknowledge the impact this issue has had. I wish to thank our customers and partners for the efforts they put in to patch vulnerable servers, and apologize for the impact. I understand that our products are mission critical to your organizations, and we don’t take that responsibility lightly. The code leading to the vulnerability in question was added almost 15 years ago. If I didn’t author this code, I definitely reviewed it as there were only a few people in the company in those days. The buck stops with me, and I’m personally committed to making sure we all learn from this.

What happened?

In January 2023, two vulnerabilities were identified in PaperCut Software’s MF/NG applications, brought to our attention by security researchers from Trend Micro.

We released a software patch on 8th March to address these issues, however, a month later by 18th April, the first attack was observed on a system that hadn’t been updated.

Though 98% of our customer base had either already implemented the patch or had firewall protection, around 2,000 organizations were at high risk. These unprotected systems, accessible via the internet, became prime targets for ransomware attacks (some of which were staged by Nation-state-backed hacker groups).

In response, PaperCut Software initiated an urgent outreach program to secure these public vulnerable systems.

Timeline

All dates and times are reported in AEST (Melbourne, Australia), unless otherwise noted

A TL;DR summary timeline is summarized below. A full timeline, with annotated learnings can be found at the bottom of the post.

Microsoft is attributing the recently reported attacks exploiting the CVE-2023-27350 and CVE-2023-27351 vulnerabilities in print management software PaperCut to deliver Clop ransomware to the threat actor tracked as Lace Tempest (overlaps with FIN11 and TA505).

— Microsoft Threat Intelligence (@MsftSecIntel) April 26, 2023

More actors are exploiting unpatched CVE-2023-27350 in print management software Papercut since we last reported on Lace Tempest. Microsoft has now observed Iranian state-sponsored threat actors Mint Sandstorm (PHOSPHORUS) & Mango Sandstorm (MERCURY) exploiting CVE-2023-27350.

— Microsoft Threat Intelligence (@MsftSecIntel) May 5, 2023

What we learned from this incident and what we’ll do next

From our post-incident reflections, we’ve derived the following insights and action steps. They were collected from retrospectives conducted across multiple teams and were enriched by insights from our security consultants, partners, and customers. Our learnings have been categorized under Product, Organizational, and Data and Communications domains.

Product change learnings

1. Application notification and dashboard alerts are often ignored

The security update was issued via our typical release process which includes an in-app check-for-updates notification, release notes, RSS feed, in-app dashboard news, and partner portals. Because it was a security focused release, subscribed customers were also notified via the security mailing list. We learned:

• Many customers did not notice the severity level - announcements were not generally visually different.
• Some customers rarely log into the admin console, and hence requests to update or check-for-updates were missed.
• In some workplaces, administrators had changed and the new admin was not aware of notification sources and locations.
• Many messages were ignored in the noise of less urgent notifications.
• Our customer-to-subscribe ratio on the security mailing list was low

Our action plan

• Many software vendors are now starting to make security specific notifications inside the app that are highly visible and require action. A good example is Apple’s recent changes around “Rapid Security Response (RSR) updates”. These notifications are highly visible and require action. Our plan is to introduce this new style of alert in our applications.   (NOTE:  This has been actioned in the  23.0 release)
• We’re actively promoting the security mailing list, and uptake has been significant. We plan to increase awareness further by introducing a new process in PaperCut MF/NG’s administrator console. For example, when a new admin is added to PaperCut MF/NG, they’ll be encouraged to ensure their organization contact information is current, and have their attention drawn to the security mailing list (also see Organization Changes Learnings below).

2. Our approach to penetration testing needs to step-change in line with the security landscape

It’s clear that PaperCut, and printing in general, is now “above the radar” in the hacker and malicious actor domain. This heightened visibility isn’t a transient phase—it’s the new reality. Recognizing this, we must intensify our penetration testing efforts and deepen our engagement with the InfoSec community.

Our action plan

We’ve incorporated several new top-tier pen testers into our white box penetration testing initiative, where testers have access to the source code under a confidential agreement. This not only increases the frequency of our testing but also brings varied expertise.

Furthermore, we’ve created a specialized internal security team responsible for coordinating with prominent InfoSec partners and vendors to assist more research on PaperCut MF/NG. Our objective is to intensify the focus on PaperCut’s security and expedite the journey from issue reporting to resolution. 

In tandem with this, we set up a group of our most experienced engineers who regularly need to look at ways of enhancing the security of the product. This has resulted in a series of security-centric releases (version 22.1 is an example) to facilitate this transition. 

All of this complements our ongoing efforts to enhance development practices in terms of tools, training, and procedures - a journey we embarked upon with our ISO27001 adoption and enhancement.

3. Familiarity with our suggested server security hardening measures was not widespread

Although a significant number of our customers remained protected due to their security strategies—even if they hadn’t applied the latest patches—we identified a segment that wasn’t acquainted with these measures. We provide Knowledge Base articles and dedicated sections in our user manual addressing this area. However, leveraging these resources necessitates proactive efforts to locate and adhere to them.

Our action plan

Similar to the in-app notification and alerts change, we plan to make a more visible link to security hardening practices within the product. Where possible, we will actively highlight potential actions that may apply to the environment. We will re-evaluate security defaults, for example, a public addressable server should restrict access to selected areas based on approved network address locations such as where printers are located. (Note:  This has been actioned in the  23.0 release)

We will also review application default sessions, and where appropriate consider more secure or “locked down” defaults. Keeping environments secure is not something we can do alone, it’s a joint responsibility with our partners and customers and we need to work closely with all parties to ensure that we are all staying on top of security.

Organization change learnings

1. Strengthening our communication and proactive stance with infosec vendors is crucial to ensure they grasp the nuances of our product, our customer demographics, and our partner ecosystem

In the wake of the incident, our review and retros were not confined to just PaperCut. We also included Trend Micro, who played a pivotal role in identifying the core vulnerability, in our process. This exploration revealed some gaps in understanding how our product is implemented and used. Notably, it’s frequently managed by the customer, with upgrades subject to processes like change controls, planned downtime periods, or updates timed around school vacation periods, among other nuances.

We learned there is actually a strong desire for more cooperation, and this is exemplified by this quote from a leader in Trend Micros’s ZDI team:

We thank you for the opportunity to collaborate with you on improving your processes and we are “fighting” for the same cause. I appreciate your coming forward with this request because it’s not too often that vendors will reach out for guidance like this. I can only think of a handful out of the hundreds we deal with each year. We are oftentimes seen as the “enemy” but in reality, we just want to ensure our Trend Micro customers are protected regardless of whether or not we are able to release protections for any particular vulnerability we acquire or discover.​​​

Our action plan

To bolster our security, we commit to amplifying our interactions with the InfoSec community and security resources. Our approach will be both anticipatory and responsive, especially during incident evaluations. Our initiatives will encompass the following:

• Enlightening the community about the intricacies of our software’s deployment and usage
• Pushing for a grace period before any complete disclosure of vulnerabilities or proof of concepts
• Involving InfoSec experts in our post-incident analysis, ensuring comprehensive learnings for every issue.

2. Improved CVE reporting and coordination is crucial

Traditionally, we’ve depended on the InfoSec community for synchronizing updates to the CVE database, typically via the CNA registered security researcher. We recognized that a significant number of our customers rely on CVE notification services or deploy software scanning tools that utilize CVEs for insights. During this incident, there were noticeable lags in updating and publishing the relevant CVE, leading to customers using this process being alerted belatedly.

Our action plan

To protect our customers, we want to control CVEs so the publish date and information provided in them match our own advisories and releases. We’ve taken the action of joining MITRE as a CNA. By doing so, we’ll be directly responsible for overseeing this crucial process.

Data and communication learnings

1. Our customer contact information is often not current

During our active call out period we attempted to contact identified high-risk customers with publicly addressable unpatched PaperCut MF/NG servers. During this phase we identified that some customer contact information was stale or outdated. This was a hindrance and we often needed to resort to web search and LinkedIn to get current information. Most of our on-record contact information was collected at the time of purchase, or at best the customer’s latest support ticket. With many administrators and IT people changing roles in the last few years, a notable percentage of the contact details were incorrect.

NOTE: I should also acknowledge I’m proud of our team’s effort during this phase. Many team members went to great lengths to ensure the right person was contacted, and customers were patched quickly. A few stories:

• A Canadian college’s overnight helpdesk person was surprised to receive a call from Melbourne. He usually deals with late night calls from stressed students trying to submit assignments on deadlines. He took it upon himself to help track down the new admin responsible for the server and made sure they were briefed on arrival in the morning.

• Our team had issues navigating the switchboard of a Thai Government organization and was unable to leave a message. Their IT department had a fax number, and we got their attention quickly that way resulting in the server being upgraded soon after.

• DMs via LinkedIn got us quickly in touch with a new admin, again resulting in quick server actions.

However we need to acknowledge that incorrect contact information was inefficient and impacted the time it took us to contact some of our high-risk customers.

Our action plan

We will add the ability to review and update customer contact information from within the application, via the About Page or equivalent. When a new administrator account is created, we’ll ensure this admin is encouraged to review this. We’ll also look at the feasibility of promoting the security mailing list to new admins.

2. We have inconsistent emergency response processes in our channel network of partners and resellers

We need parallel paths of communication into our customers. Many of our resellers are trusted partners of our customers and were able to get important messages to our customers quickly. However, in some cases the relationships didn’t exist the way we would have liked and so these messages didn’t get through.

Our partners can be extremely valuable to support our customers in cases like this. 
EXAMPLE: A channel partner called the security contact at a major university. The response was dismissive indicating they would get to it “in the next few weeks”. Clearly this isn’t good enough and so the partner called the head of IT and said “if it’s not upgraded this morning I’ll get in the car and do it myself”.

In other cases we have channel partners where security is regarded as “a new topic”  in our print domain. PrintNightmare put security on the radar in print but it was seen as a one-off event instead of a need to drive a systematic security posture improvement. In this incident, there were some cases where security messages or support were either not effectively conveyed to the customers by our partners, or when conveyed, they were not passed on to the appropriate individuals within the customer organizations.  This was a key learning for both us and our partners.

Our action plan

• We will add security as a dedicated component in our partner and reseller training and certification program
• We will define formal cybersecurity DEFCON levels, and expected responses for both PaperCut and our partners
• We will work with a sample of partners to include them in our future table-top war game exercises and response drills
• In addition to the product changes listed above to ensure contact information is current, we’ll encourage our partners to assist here too.

All the learnings, plans and actions listed above are owned by leadership, and I, as the CEO and co-founder will be overseeing them. While the vulnerability discussed in this review did not impact our next generation cloud products (PaperCut Hive and PaperCut Pocket), the learnings will be applied to all products and processes across PaperCut Software.

Closing thoughts

We’re well aware that there is never a convenient moment to respond to a security incident. A heartfelt thanks goes out to all our customers, partners, and resellers who worked diligently and swiftly during this challenging period to promptly update systems. The efforts and resilience displayed by our PaperCut support team, alongside our partners, filled me with immense pride. 

This incident, while testing, will undeniably make PaperCut stronger. We’re committed to deriving crucial lessons from this experience and, in our spirit of openness, will continue to share our learnings for the collective benefit of all. To every customer who stood by us: thank you. Your trust and support drive us forward.

Full timeline

This version of the timeline has more detail than the summary above. We have also annotated it with mixing in the learnings with a mindset of “If we had applied the learnings over the incident again, where would things be different”:

Microsoft is attributing the recently reported attacks exploiting the CVE-2023-27350 and CVE-2023-27351 vulnerabilities in print management software PaperCut to deliver Clop ransomware to the threat actor tracked as Lace Tempest (overlaps with FIN11 and TA505).

— Microsoft Threat Intelligence (@MsftSecIntel) April 26, 2023

More actors are exploiting unpatched CVE-2023-27350 in print management software Papercut since we last reported on Lace Tempest. Microsoft has now observed Iranian state-sponsored threat actors Mint Sandstorm (PHOSPHORUS) & Mango Sandstorm (MERCURY) exploiting CVE-2023-27350.

— Microsoft Threat Intelligence (@MsftSecIntel) May 5, 2023