Application Server Failover

1. Install PaperCut NG/MF , and then stop the Application Server .
2. Upsize to an external database . Each Application Server must use the same database instance.
3. Start the Application Server and confirm that PaperCut NG/MF is fully functioning.
4. Stop all of the PaperCut services (PaperCut Application Server service, PaperCut Print Provider service, etc.).
5. On the Application Server, disable the printing and tracking services. (Stop CUPS or disable the Spooler, and disable the PaperCut Print Provider service )
6. Go to [app-path]/server.
7. Copy the following folder and files to the configured network storage:
   • /data folder
   • server.properties file
   • security.properties file
   • for perpetual customers (no subscription) - the application.license file
   • 

8. Open the server.properties file in a text editor.
9. Make the following changes to the file:
   • Remove the “#” symbol to uncomment the following lines:
     • high-availability-guard.enabled = Y
     • high-availability-guard.fail-over-internal-secs = 30
     • high-availability-guard.startup-timeout-secs = 300
     • See “Configuration keys for App Server Failover” if you need more details.
   • Change the high-availability-guard.enabled config option to “Y” if it’s not already.
10. Save the file.
11. Rename the existing local /data folder in the \[app-path\]/server/ directory to data.bak.
12. Make a folder-level symlink for the /data directory.
    • On Windows: mklink /D "C:\Program Files\PaperCut MF\server\data" //network-storage/papercut/data
    • On Linux: Edit the /etc/fstab to mount the /data folder on network drive. Remember to include a system.automount parameter in the options. Then, run ln -s /usr/local/papercut/server/data /mnt/network-share/data
13. Make a file-level symlink for each of the remaining copied files:
    • application.license
    • security.properties
    • server.properties

14. Start all of the PaperCut services (PaperCut Application server service, PaperCut Print Deploy Server service). Confirm that the now-active Application Server is still functioning as expected.
15. Open the Admin web interface and navigate to Options > Advanced > Server Address.
16. Set the Server Address to the DNS name or VIP of the NLB.
17. Open the advanced config editor and find the key system.network-address. Configure this key to the IP or the hostname of the load balancer
18. To make full use of the NLB and ensure that all PaperCut components work during a failover event, review the following articles:
    • Changing the Server Name or IP for devices
    • Changing the Server Name or IP for PaperCut features
19. Set up the SSL certificate for your servers. (See “Additional details on load balancer configuration” below.)

If using a Virtual Machine

1. Stop the PaperCut Services on your pre-configured Application Server.
2. Clone the Application Server host to make a second, identical app server. Your hypervisor may require you to shut down the virtual machine to clone.
3. Ensure that the PaperCut services are still stopped on the new clone of the Application Server.
4. In the newly created Application Server clone, navigate to the [app-path]/server folder.
5. Delete the file called ‘server.uuid’.
6. Start the PaperCut services.
7. Repeat for each server behind the Network Load Balancer

If using physical hardware, or if cloning is not possible

1. Install PaperCut and then stop all of the PaperCut services. (PaperCut Application Server service, PaperCut Print Provider service, etc.).
2. In the [app-path]/server/ directory, rename the existing local /data folder to data.bak.
3. Make a folder-level symlink for the data directory detailed within steps 12-13 in the Configure the Active App Server section.
4. Make a file-level symlink for each of the remaining copied files:
   • application.license
   • security.properties
   • server.properties
5. Make sure the PaperCut Application Server service is running under a user account that has Read/Write access to the shared network folder you set up earlier.
6. Start all of the PaperCut services (PaperCut Application Server service, PaperCut Print Provider service, etc.).
7. Repeat for each server behind the Network Load Balancer.

When finished, confirm when attempting to access the Admin web interface on your new server that the below message is displayed:

1. Log in to PaperCut NG/MF as an admin and go to Options > Advanced.
2. In the System Health Monitoring section, copy the URL path (the info after :9191) in the GET query parameter field:
3. In the load balancer settings, paste the above endpoint as the monitoring URL.
4. If the load balancer allows a monitoring interval, we recommend setting it to 30 seconds. A lower number will unnecessarily increase the load on the Application Server.

| Expected Responses | |— | — | | HTTP 200 OK | The server is active and ready to handle PaperCut traffic | | HTTP 503 | The server is in passive mode and is not handling PaperCut traffic |

5. Configure all external components (secondary servers, user clients, MFDs, etc.) to point to the virtual service IP address or hostname that has been set up on the Network Load Balancer.

SSL Configurations

If you want to configure SSL for this server, it’s best practice to use a common SSL certificate for the Applicaiton Server and the Network Load Balancer’s VIP. A single SSL certificate will remove the chance for SSL certificate date mismatching and should simplify the maintenance for you in the future.

If you choose to enforce SSL traffic on your load balancer, you’ll need to configure the PaperCut Print Provider to support HTTPS. This feature is available in version 23 of PaperCut NG/MF and above. HTTPS traffic will not be configured in the print provider by default.

Using a Layer 4 (Transport Layer) Load Balancer

Layer 4 / L4 / Transport layer NLBs will direct traffic at a very low-level on the network. They involve less compute power to work, but are a bit old-fashioned and can involve a fair amount of manual configuration. Layer 4 load balancing was what was initially used for the PaperCut App Server Failover. As such, this configuration is the most tested and widely documented.

Network adapters and loopbacks

Layer 4 will require that a network adapter is configured for the Virtual IP Address (VIP) of the NLB. This network adapter should be set up as a loopback so that the address can be held by the server without being used on the accessible network. Additionally, this setup will often require that you do not allow for ARP responses on the configured loopback adapters.

Example steps for Windows Device Manager

1. In a Run prompt, type ‘hdwwiz’ to open the “Add hardware” wizard.
2. Select “Install the hardware that I manually select from a list (Advanced) and click next.
3. Find the “network adapters” section of the list.
4. In the manufacturer list on the left side, find the Microsoft entry.
5. Select Microsoft KM-TEST Loopback Adapter and complete the install.
6. In the Network Connections manager, find the newly added network adapter.
7. Select Properties and navigate to the TCP/IPv4 menu.
8. Populate the IPv4 address settings with the VIP address of the NLB. Optionally, you can name the adapter something more intuitive than the default name that is assigned
9. In the TCP/IPv4 properties menu, click the Advanced button.
10. Uncheck Automatic metric. Set the Interface Metric field to 254.
11. Configure “Weak Host” parameters for each of the Network adapters using the following command format: netsh interface ipv4 set interface "[interface name]" weakhostreceive=enabled

Example steps for Linux

1. Open terminal.
2. Run the command ip addr add [virtual ip address] dev lo

Preserve client IP address

To make sure that the server is capable of full functionality, configure the network load balancer to preserve the client IP address. This allows the logs and access policies to work correctly for the IP of the client, instead of filtering against the IP of the NLB. Some vendors will use the terminology “Preserve Client IP”, “Use Source IP”, or it can be written in the inverse like “Disable SNAT”.

Direct server return

We recommend a direct-server-return configuration to lower the potential bottlenecks associated with handling all ingress and egress traffic with the load balancer. Depending on the load balancer, this could be known as Direct Server Return (DSR), n-path routing, Direct Return (DR), Asymmetric Routing, or SwitchBack among others.

This allows for the PaperCut Application Server to reply directly to the client that is sending the requests without the involvement of the NLB. In addition to the overhead savings with DSR, you can also avoid some configurations like static routes or default gateways on the application servers.

If your environment does not support direct return, or you don’t want to configure direct server return, you can configure the load balancer as a static route destination for any of the clients IP addresses that you want to interact with the PaperCut app server.

Using a Layer 7 (Application Layer) load balancer

Layer 7 / L7 / Application Layer NLBs will handle traffic at a much higher level than Layer 4. Because of the higher-level operation, L7 NLBs are relatively intensive on computing resources. However, many modern load balancing solutions will have no problem handling a substantial amount of PaperCut traffic before the L4 / L7 distinction will come into play.Working with HTTP instead of TCP allows PaperCut to leverage X-Forwarded-For headers for logging and responding to requests.

SSL Traffic and termination

Most Layer 7 solutions will not require an encrypted connection between the NLB and the backend server that’s hosting PaperCut, so it’s fine to continue using the PaperCut self-signed certificate, and allowing HTTP unencrypted traffic to the server. Otherwise, if you do want to encrypt traffic between the load balancer and the PaperCut Application Server, you should assign a common SSL certificate for both the NLB endpoint and the Application Server.

####App Server configuration for L7 Support Application load balancers will need to be configured to trust and reference the X-Forwarded-For header that is added by the NLB. This configuration will be set with the “trusted proxy servers” configuration in the advanced section of the options menu.

If you do not set the above configuration, you might start to see Cross Site Request Forgery errors. If you see these errors, you might need to follow our steps for CSRF Error resolution here.

PaperCut will need a shared network file system to store common configuration files that need to be referenced by the Application Servers. This will need to be mounted using the OS-native command line. The PaperCut Application Server will continue to reference its default file paths (for example, C:\Program Files\PaperCut MF\server\server.properties) - however, the actual location of these files should be on the network storage.

It’s best to use a protocol that you are familiar with and that fits with your Operating System’s supported standards. For example, Windows servers could likely leverage an SMB share for the data and configuration files, while Linux servers might benefit from an NFS share.

The database is used as a means of communicating which node is the active server in the PaperCut Application Server pool. As such, you should ensure that the database is operating quick enough so to not accidentally trigger the Application Server’s failover. The default configuration is generally fine, but keep this consideration in mind when setting up bandwidth and allocations for the database.

If you need to configure a highly available database to meet your organization’s resiliency needs, the Application Server will support this. You don’t need to take additional configurations into account for the PaperCut server - you can implement whatever high availability configuration that is available for your RDBMS.

These configurations are stored within the server.properties file, which is stored on the network storage you configure.

Where do site servers fit into my failover environment

Site servers are useful for tiding over multifunction devices temporarily. Things like hold/release, find-me, filters and restrictions (conversions and policy), account selection, basic accounting and authentication will continue to work in the event of an outage. If you need, site servers should be configured locally and in-office within the network so that the location could feasibly go offline and continue to function. If there is not a place for hardware on the network, it is generally best to leverage Papercut Application Server Failover without the use of site servers to satisfy your high availability needs.

Extra considerations with site servers

Site servers will apply a load on both the database and the application server. Make sure that the database server and application servers have ample bandwidth to sustain both the syncing tasks of the Site server, and the heartbeat and data transactions of the application server.

Configuring Print Deploy to work with App Server Failover

As of the November 2022 Print Deploy server version (1.7.2214), the Print Deploy server natively supports automatic cutover for App Server Failover environments. If you plan on leveraging Print Deploy with your App Server Failover configuration, you should perform the following steps on each Application Server:

1. Stop the PaperCut Print Deploy Server service.
2. Open the [app-path]/providers/print-deploy/[os]/data/config/client.conf.toml file in a text editor with elevated permissions.
3. Set the following config key to true - if the key does not exist, you can add it as it is displayed below to the file:
   • HighAvailabilityCheckEnabled=true
4. Open the [app-path]/providers/print-deploy/[os]/default-pc-print-deploy.conf file in a text editor with elevated permissions.
5. Set the StartupRandomDelaySecs value to 0 in the ”ScheduledTasks": section of the config file.
6. Set the StartupRandomDelaySecs value to 0 in the ”StartupTasks": section of the config file.
7. Save the file.
8. On the active application server, navigate to [app-path]/providers/print-deploy/[os]/data
9. Copy the /repository folder to the network share in a manner similar to the application server data folder above. Rename the remaining copy to repository.bak.
10. Do the same for the /config and /translations folders
11. Copy the individual files tls.cer and tls.pem to the shared storage and make a file-level symlink for those.
12. Change the “accessible host” configuration to the VIP or hostname for the network load balancer.
13. Configure the Network load balancer to use the same health check as the application server to route to the Print Deploy ports. To do this, you can either:
    • Expand the used ports for your existing application
    • Create a separate VIP/endpoint for print deploy, and re-create the health check for the application server.

Optional configurations

• If you have enabled auto-update already (this setting is off by default), open pc-print-deploy.conf in the /[app-path]/providers/print-deploy/[os]/ directory
  • Set the StartupRandomDelaySecs value to 0 in the "ScheduledTasks": section of the config file.
  • Set the StartupRandomDelaySecs value to 0 in the "StartupTasks": section of the config file<
  • Save the file.
• In the /[app-path]/providers/print-deploy/[os]/data/config/ directory, open the client.conf.toml file and edit the HighAvailabilityAppServerProbeIntervalInSecs
  • Use this value to specify how often the PaperCut Print Deploy server should query the PaperCut NG/MF Application Server health API URL. The PaperCut Print Deploy server determines the status of the PaperCut NG/MF Application server using the HTTP code of the response.The default value is every 15 seconds.

Automatic updates

Automatic updates can work with Application Server Failover enabled, however, you need to confirm that the above optional steps are applied to each of the Print Deploy servers. These configurations will prevent the Print Deploy server version from being different for a notable duration of time enabling failover to occur without issue.

Print Deploy health and function checks

While the Application Server health check should be used for the routing direction in the NLB, you can run health checks on the Print Deploy endpoint to confirm that the server service is running correctly. If you do an http check to https://[application-server]:9192/deploy/ping, it should return an HTTP 200 if the Print Deploy server is responsive. You could use this check in a health monitoring application to track the uptime of PaperCut Print Deploy.

All potential web print servers must use the same web-print-hot-folder. The web-print-hot-folder is already included within the /data folder of the PaperCut NG/MF Application Server [app-path]\server\data\web-print-hot-folder and should be copied already from the steps in the Configure the Application Server(s) section, step 8 above.

If you are using sandbox mode , make sure all your web print servers have access to the shared directory in the data folder.

For custom reports to work across all potential servers, the [app-path]/PaperCut [NG or MF]/server/reports on each Application Server needs to be moved to the shared storage.

On each applicaiton server perform the following:

1. Stop the PaperCut Application Server service.
2. Navigate to the [app-path]/PaperCut [NG or MF]/server/reports.
3. Rename the folder reports.bak.
4. Copy the folder to the network drive.
5. Create a folder-level symlink for the default reports directory to the network storage like you have in previous steps.
6. Restart the PaperCut Application Server service.

Once complete, custom reports can be created and maintained in the same way, but with the network storage as the target path.

If there’s a failover incident, the user clients need to seamlessly switch over to the new active server. For that to happen, user clients must point to the network load balancer’s IP address. The user client’s configuration is propagated when it’s installed from the server.
In both Application Servers, update the User Client config to point to the network load balancer’s IP address.

1. Open [app-path]/PaperCut [NG or MF]/client/[os]/config.properties.
2. Update the server-name and server-address and server-port to reflect the endpoints of the NLB.
3. Save the file.

Follow the normal upgrade path with some small adjustments:

1. Stop the PaperCut Application Server service on all servers.

2. Upgrade all servers.

3. Start the PaperCut Application Server services one at a time.

You can subscribe to application log updates so that if there’s a failover event, you’re automatically notified.

1. In the Application Server Options > Notifications tab, under System Notifications, select Error level events and enter the email address of each recipient.

   

2. Check the SMTP Server Options are set up so that the server can send out email notifications. If an Application Server fails, you’ll see this message in the Application Log:

   

3. We also recommend setting up monitoring and notifications on the Network Load Balancer, so you are aware of any of the servers in the pool become unavailable.