I just saw that Atlassian, the provider of the essential community tools like Confluence wiki and JIRA ticket system, updated their wiki on the importance of monitoring the “lifeblood of your organization”.

They even outline the important monitoring tasks you need, and stress that it will help when dealing with their own world class support.

Monitoring involves a number of essential tasks, including those listed below:

• Monitoring log files.
• Checking for HTTP-availability and performance (e.g. by getting the same page every five minutes and displaying the time on a graph).
• Looking at many different parameters such as load, connections, IO, database-trends, and so …

The built-in HQ database is PostgreSQL. Recently, users have been discovering PostgreSQL has a certain limitation: it will not execute more than 2 billion transactions between vacuums. In rare cases, an HQ built-in database can get into this state.

If this happens, the database will stop accepting connections and HQ, which needs a data store, will obviously cease to operate properly. The immediate symptom will be that users will not be able to log in to HQ and the message displayed on the screen will be The backend datasource is unavailable.

That error is not enough to say for sure that the problem is PostgreSQL avoiding wraparound failure by not accepting connections. A quick look at the hqdb.log can confirm. The telltale log entries look like this:

FATAL: database is not accepting commands to avoid wraparound data loss in database "postgres"
HINT: Stop the postmaster and …