Queue troubleshooting for Project Server 2007

I found the batch job and not starting topics useful and wanted them local.

Overview:
A queue is a waiting line that becomes a necessity when the number of service requests becomes greater than the optimum serving capacity. Nearly all critical operations in the Office Project Server 2007 system go through the Office Project Server 2007 Queuing System. These include:

• Project Save
• Project Publish
• Timesheet Save
• Timesheet Submit
• Project Backup/Restore
• Report Data Service
• Cube Building Service
• Server Side Scheduling (and node-consistency processing)

Project Queuing System provides the following advantages:

1. Reliability
2. Scalability
3. Manageability

The Project Queuing System can hang on occasion for many reasons, this requires some administration know how in order to resolve these stuck jobs.
Viewing the queue:

• Logon to PWA as a user with Administrator rights
• Select Server Settings from the Quick Launch of the PWA Home Page
• From the Server Settings page select Manage Queue
• Set the filter options as desired and refresh the Job Grid

By default the Job History is set to only show jobs from the current date and to show all Job Completion States other than “Success”
Actions to take if the queue stops processing jobs:
There are several reasons why the queue can get stuck. The easiest way to know whether the queue is stuck is to look in the Manage Queue page in PWA. If there are several jobs sitting in the “Waiting to be processes” state, then the queue is most likely stuck.
The thing to do first is to find the one job that is in queue position 1 and that is holding up all the other jobs from processing. Follow these steps to help find the culprit job:

• In the “Manage Queue” page in PWA>Server Settings expand the “Job History” and set the “From Date” back to as early as you think the stuck job might be found. A week back would probably be sufficient if the queue is regularly monitored
• In the “Job Completion Status” remove all jobs statuses and only add back in the following: Getting Queued, Processing
• Hit the “Refresh” button in the Jobs Grid

Clearing a job that is stuck in the “Getting Queued” state:
If a job state is ‘Getting Queued’ it can be force cancelled directly from the manage queue page.

• Under Advanced Options, check the box next to ‘Cancel jobs getting enqueued’
• Uncheck the option ’Cancel subsequent jobs in the correlation’ if you want the subsequent jobs (i.e. jobs in queue position 2 – n) to still process
• In the Job Grid, select the checkbox next to the job stuck in the state of ‘Getting Queued’
• Click ‘Cancel Jobs’
• Refresh the screen to validate that the job was cancelled
• If it does not cancel, attempt this process again
• All other jobs should start processing immediately

Clearing a job that is stuck in the “Processing” state:
If a job state is stuck in ‘Processing’ it can only be cleared by restarting the queue services. Please note, restarting the queue services requires administration rights to the server.
Steps:

• From your computer, select Start and Run
• In dialog box, type MSTSC, enter
  • Type in the name of the project server enter
  • Enter Administrator User Name & Password
• Once on the server go to Start and Run again and type SERVICES.MSC
• In the services window find the job “Microsoft Office Project Server Queue Service”
• Right click on this job and click restart
• If the job restarts successfully then go back to the “Manage Queue” page in PWA and verify that the jobs begin processing again
• Jobs should start processing right away. Hit refresh several times to validate jobs are processing

Note: If jobs are getting stuck in the “Processing” state on a regular basis you should consider creating a batch job that simulates the steps above. The job described in the next section performs the same “Restart” action as described above. It also describes how to schedule this simple batch job to restart the queue services using the “AT” command in dos. This means that once this is set up, you will be less likely to have to perform this task yourself as it will fix itself as often as you have the job scheduled (example: every 1 hour or every 6 hours). If the issue is not occurring, restarting the queue service takes a second or so and it should not cause any harm to restart on a periodical basis.

Viewing, Editing or Deleting the Scheduled Queue Restart Batch Jobs:
Steps:

1. Place a text file with the .bat extension in the following location on the project server: C:QueueRestartBatchJobsrestart_queue_services.bat
   The text in this batch job should be as follows:
   net stop "Microsoft Office Project Server Queue Service"
   net start "Microsoft Office Project Server Queue Service"
2. To schedule this batch job to run as a job:
   • Go to a command prompt on the server and type
     AT 20:30 /EVERY:M,T,W,Th,F,S,Su C:QueueRestartBatchJobsrestart_queue_services.bat
   • This example sets the restart_queue_serverices.bat batch file located in the C:QueueRestartBatchJobs folder to run every day at 20:30 which represents 8:30 PM server time
3. To view all currently scheduled jobs:
   • Go to a command prompt on the server and type: At
   • Click Enter
4. To delete a job:
   • Go to a command prompt on the server and type: At n /delete <