Background:
We are using Microsoft Office Project Server 2007 in a small farm environment (one web/app server, Windows Server 2008 R2, and a second database server, SQL Server 2008 R2 Enterprise … both are VMs, database server is a dedicated instance on a shared server).
Symptoms:
Sometimes when we publish our master project, one or more of the inserted sub-projects gets stuck in the Project Server queue in “Getting Queued” status (both the Project Publish Notifications and Project Publish jobs). Project Server appears to automatically resubmit those jobs, but they remain in “Waiting to be Processed” status until the “Getting Queued” jobs are manually cancelled (without cancelling other jobs in the correlation) … then the resubmitted jobs process and complete successfully.
We have reviewed potentially related posts and they do not appear to be the same problem(s).
Notes:
- Most of the sub-project publish jobs and their successors complete successfully.
- This seems more likely to happen when we have larger numbers of inserted sub-projects … more than 30. (We’re working on a business process workaround to reduce the number of inserted sub-projects, but we would still like to get to the root cause on this.)
- This was not happening in our previous single-server environment (not sure how many sub-projects we had at a time, though).
- There does not seem to be any pattern of which project(s) this happens to.
- Our DBA does not see any error messages on the database server.
Questions:
It appears to me that we have two possible problems:
- What is causing the initial job to fail while being enqueued?
- Why do the initial jobs remain stuck in the queue, blocking the re-submitted jobs from processing? (If the resubmittals would process automatically, I’m not sure we would need to solve the first problem …)
Research – Initial Job Failure: The ULS log includes messages about:
- PSI: WinProj.QueuePublish [QUEUE] Thread was being aborted
- PSI: WinProj.QueuePublish [QUEUE] SendMessage failed
- PSI: WinProj.QueuePublish Enqueue of publish failed for project: {project uid}
- PSI: WinProj.QueuePublish System.Threading.ThreadAbortException: Thread was being aborted. at …
- PSI: WinProj.QueuePublish [QUEUE] [SENDER] ProjectQ: CancelMessageGroup(): GroupUID: {group uid}
- PSI: WinProj.QueuePublish [QUEUE] ExecuteNonQuery requires an open and available Connection. The connection’s current state is closed.
- PSError: Success (0) Queue SQL connection is being retried. Error: System.InvalidOperationException: ExecuteNonQuery requires an open and available connection …
- PSError: ProjectPublishEnqueueFailure (23007) Publish queue job could not be put in the queue. Project GUID is: {project uid}
Has anyone else encountered this? How have you resolved it?
Thanks so much for your help!