Troubleshooting and optimizing Server Performance

“Help! Our PaperCut Application server is acting unpredictably, and we see lots of intermittent issues especially during peak usage times; how should we go about troubleshooting this problem?”

Symptoms of a performance problem

If you’ve already tried “turning it off and on again” and you still see some of the symptoms in the list below, then this might be the right troubleshooting article for you.

• The PaperCut server’s web interface is slow to load.
• Logging into MFDs may be slow or time out. These devices may display an error similar to “unable to connect to server” or “connection timed out” when users try to log in, especially in the middle of the day.
• The server might show consistently high CPU or Memory usage, plateauing at 100%.
• The common thread is that the problem often cannot be reproduced consistently, and is usually reported during the busiest times of the day for printing.
• You may see the error Warning: Print Provider “printsrv01|192.0.1.5”: Print job “Document-1.pdf” printed by “user1” is offline and has been temporarily hidden from the Print Release screen. as described in the article “Why was this job ‘temporarily hidden’?” after you have ruled out the other causes.

Slow Printing or Disappearing Print Jobs can also be indicators of a performance issue but also have many other causes.

Check the provisioning and basic configuration

You may have already gone through the suggestions in our article on Keeping a healthy system, and you’re still having trouble. So what’s the next thing to check?

Every server OS has a built-in utility for checking which apps are using the greatest amount of CPU or Memory. Here’s how to find how much CPU and memory is getting used by PaperCut on each type of server:

• Windows - in the native OS Resource Monitor or Task Manager applications, look for pc-app.exe.
• macOS - in the native OS’ Activity Monitor application, look for the process named Java running as the user papercut.
• Linux - open the Terminal command window and run the top command, look for the process named pc-app.

Make sure the PaperCut NG/MF server has adequate CPU/Memory allocation

Check our Server Sizing Guide to make sure that the server is scaled with the appropriate resources. Remember that Site Servers and Secondary PaperCut servers need adequate resources as well to function as part of the printing ecosystem.

One extreme sign of a performance issue is to look in the service.log file found on any PaperCut Application server for messages like JVM Process has not received any CPU time for X seconds. This indicates that the PaperCut application, which runs as a Java Virtual Machine (JVM), is not getting enough resources from the operating system or virtual host. The service.log file can be found in the \[app-path]\PaperCut NG/MF\server\log.

INFO   | jvm 1    | 2020/04/14 13:11:31 | JVM Process has not received any CPU time for 23 seconds.  Extending timeouts.
INFO   | jvm 1    | 2020/04/15 05:08:17 | JVM Process has not received any CPU time for 21 seconds.  Extending timeouts.
INFO   | jvm 1    | 2020/04/15 20:56:36 | JVM Process has not received any CPU time for 19 seconds.  Extending timeouts.

Make sure the server PaperCut NG/MF server doesn’t have too much memory allocation

It is possible to have too much of a good thing. And, surprisingly, RAM is definitely one of those things for PaperCut. PaperCut NG and PaperCut MF are Java applications. Java has a garbage collection routine that will use a percentage of the memory allocated to the application as a marker for when to trigger cleanup. When too much memory is available to the Application Server, full garbage collections will happen less frequently but take longer to complete, creating a drain on resources. Follow our Server Sizing Guide to give your PaperCut server the right amount of memory, and don’t go overboard. By default the PaperCut server will use one quarter (¼) of system memory. If you want to fine-tune this value, follow our guide on how to adjust the amount of memory available to PaperCut NG/MF.

Reserve the resource allocation for VMs

This point is extremely important and often overlooked. If the PaperCut server is running on a virtual machine, make sure the memory is reserved and not dynamically allocated. We have seen that the Java Virtual Machine that PaperCut runs on top of does not like it when memory is part of a pool that can be dynamically allocated to other servers.

For a VM running in a VMware ESXi host, this setting is called “Reserve all guest memory” and you want it enabled. For a VM running in a Microsoft Hyper-V host, this setting is called “Dynamic Memory” and you want it disabled.

See ”Does PaperCut work on a Virtual Machine (VM)?” for more information.

Additional Configuration to reduce load

Check how many print jobs are being held

PaperCut NG/MF has to track each print job held on the server, and more print jobs held means more server resources used. You might start to run into a problem if you have configured PaperCut NG/MF to hold print jobs for longer than the default of 2–4 hours.

To see how many print jobs are being held, log into the PaperCut server as administrator and look for the number of “Hold/release jobs” on the dashboard. It will vary depending on how well-provisioned your server is, but generally, more than 1500 held jobs is cause for concern. Consider changing the Job Timeout period to a lower value to reduce this number. This change will not take effect immediately, because it will only apply to any new print jobs that reach the server so it may be a few hours before you start to see the benefits.

Too many clients running the Direct Print Monitor

The PaperCut Direct Print Monitor may be installed on user’s workstations to track printing to local or USB printers. However when there are several hundred or thousands of clients running the Direct Print Monitor this may put a strain on the PaperCut server.

This scenario is discussed in much greater detail in our article Environments with large numbers of Direct Print Monitors.

Check on the database, offload to another machine

For a small boost, you can look into optimizing your database

Optimization may help a bit, however, internal databases produce a fair amount of load on the application server regardless. Even if you are not experiencing issues with the performance of the database itself, it may be worth moving the database off to another machine to distribute the load a bit more - particularly if you already have a database server in your environment.

See our article on upgrading to an external database

If NG/MF is already configured to use an external database…

Having an external database is a good idea in large environments, however we’ve seen situations where this can turn into a bottleneck if not properly configured.

A log line like this one, found in the server.log file suggests that the database stopped responding to PaperCut.

2023–04–27 08:17:30,699 ERROR SqlExceptionHelper - An attempt by a client to checkout a Connection has timed out. (Bin-RPC invoke ‘extDeviceAPI.getDeviceConfig’) [http-4626947]

If you are seeing errors like this one where the connection to the database is timing out, ask the following questions.

• Is the database server properly resourced and is it near enough to the PaperCut Application server to ensure a good connection?
• Review the PaperCut instructions for your specific database type to ensure you are using the current recommended driver and connection string.
• Run a 3rd party I/O test on your database server to help identify if there is a latency issue when writing to the disk array.
• To narrow down whether this is a problem with the specific database you are using, consider switching database types to see if the problem goes away.

Importantly, involve your database administrator in the discussion and ask them to examine the configuration.

Turn off debug when not needed

We sometimes see that Debug Mode is left enabled on busy servers. While debug mode is important to catch and diagnose an issue, it shouldn’t be left on indefinitely because the additional read/write operations can cause a measurable increase in load at large sites (on the scale of tens of thousands of clients). If you are troubleshooting a performance issue and leave debug mode enabled, this can compound the problem while clients have to wait for the server to update the log file. Heed this advice:

• Standard logging in most cases will include the information we need to search for key messages.
• Leave debug logging disabled on the server, unless it’s required to diagnose an active problem.
• If planning a restart during high daytime load, disable debug logging before the restart.
• When troubleshooting a performance issue, check to make sure that the different types of debug modes have not been left on by accident, including Application Server Debug Mode, SSL Debug, and SMTP Debug.

Check on your health API calls

Health API calls (such as the requests made by PRTG or Zabbix) can put a large amount of load on the server. Consider reducing the frequency of check-ins coming from your monitoring system to free up resource usage a bit.

Move auxiliary services to other machines

There are many optional features that can be run on the PaperCut server. The good news here is that you move many of these services to other machines. These reference other PaperCut components - however the same philosophy can be applied to apps from other vendors on the same hardware as PaperCut. As an IT best practice, we strongly recommend having a dedicated server in your environment for PaperCut NG/MF.

The following are simple and easy to move off of the application server to free up a bit of capacity:

• User Printing: Move the sharing, authentication, and actual print rendering off your Primary Application server with a PaperCut Secondary server
• Mobility Print: You can turn any Secondary Server into a PaperCut Mobility Print server by installing both services.
• Web Print: See our documentation on Setting up a web print sandbox!

Print Deploy

We have a separate article on how to reduce the impact of Print Deploy on your application server. Take a look through it to further optimize your resource usage!

Start a support ticket

Are things still not working great after following the above steps? Enable Debug mode temporarily to download the Diagnostic Bundle from your PaperCut server and head over to our Support Portal for further assistance. We can analyze these logs to see if anything stands out to us.