Subject: CCL: PBS

From: "pderosa-Latech" <pderosa:at:latech.edu>
Date: Tue, 2 Nov 2004 16:31:58 -0600

Thank you Marcel, Torben, Reinaldo, and Reuti for your prompt answer to my e-mail and a list of suggestions you provided me with.

I think I found a solution which is weird but it worked (at least it has been working for a few hours now).  I checked the scheduler logs and noticed that there was a message I overlooked before 
"Not enough memory available"

This message makes no sense since I have 2GB on each node (1gb per processor) and I was requesting 100MB, in addition I have set the server variable resources_max.mem=1GB.  I tried different values for this variable but the solution was to unset it altogether.  (qmgr -c 'unset server resources_max.mem)

Now, if any of you wants to enlighten me about why is this I will appreciate, but at this point it is just curiosity.

I got a couple some tips that may be useful to some of you dealing with PBS so I am going to share.

Thanks again

Pedro

********************************************************************************************************
Do you have pbs_sched running ?
And did you start the daemons in the right order ?
E.g. first ctl then sched or so ..
[see manual]

********************************************************************************************************
We have also experienced another oddity with this system on a perhaps 
related matter. We observed that the load as displayed by "top" was 
very high (>3) when we ran single cpu frequency jobs with Jaguar using 
kernel 2.4.20-8smp, that came with the RH 9 distribution (I have not 
tested Gaussian, so I can't say for sure that this particular quirk 
isn't program specific). However, when we upgraded the kernel to 
2.4.20-31.9smp the load dropped to the expected value of ca. one and 
the walltime dropped drastically, whereas cpu- and systime were almost 
unchanged. The odd thing is that as far as I have understood, there is 
nothing in this kernel upgrade that can explain this observation - for 
example the I/O subsystem was not improved significantly. However, the 
observations are nonetheless reproducible, so if you are not already 
running the latest kernel revision I would recommend that you upgrade!

********************************************************************************************************
I also use (open)PBS in our linux cluster under 
RedHat 7.2. You should put in your pbs_server.conf 
file the line

set server node_pack = False

This should prevent job's node packing.
In any case, there's is a mailing list related 
to openPBS. It's outdated as the soft is no longer 
distributed but it still has useful information. 
Have a look at http://www.openpbs.org and then to 
the UserArea (you must register first, it's free - 
don't worry).

********************************************************************************************************
what PBS version are you running? If it's OpenPBS, you may think about 
changing to SGE:

http://gridengine.sunsource.net

In addition to our own cluster with SGE, we have access to a cluster at 
another university, which has only problems with OpenPBS because of some 
never running jobs. the only solution there is to kill and resubmit the 
jobs. Maybe it's similar to your problem.