bugCOOL - Bugs: bug #28400, 'No module named subprocess' in...

 
 
Show feedback again

You are not allowed to post comments on this tracker with your current authentication level.

bug #28400: 'No module named subprocess' in QMTEST for osx in COOL_HEAD

Submitted by:  Andrea Valassi <valassi>
Submitted on:  2007-07-30 12:48  
 
Category: COOL (Config)Severity: 3 - Normal
Priority: 5 - NormalStatus: Fixed
Privacy: PublicAssigned to: Andrea Valassi <valassi>
Open/Closed: ClosedRelease: HEAD

(Jump to the original submission Jump to the original submission)

2007-08-07 16:26, comment #18:

Ciao Marco, thanks for committing the changes:
http://cool.cvs.cern.ch/cgi-bin/coo...
This has come after tag COOL_2_2_1-pre2.
Andrea

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-08-07 12:31, comment #17:

My idea is to have a quick and simple way to disable (temporarily) all the time-outs.
One use case is to be able to measure new sensible time-out values.

I have a script that allows to modify the time-outs from a full qmtest report, by increasing the time-outs that needs to be increased, and not touching those already good enough.
Unfortunately, to produce such a report I have to disable the time-out setting it to -1. The env. variable allows to ignore the value without touching the files.

Cheers
Marco

Marco Clemencic <clemenci>
Project Member
2007-08-07 12:27, comment #16:

Ah maybe I see what you mean: actually you can provide an env variable to disable the timeout ONLY for the manual tests, while the timeout would ONLY stay for the nightlies. This sounds ok to me too!

But in any case you need a larger timeout for osx ppc in the nightlies...

Andrea

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-08-07 12:22, comment #15:

Ciao Marco, whatever you think most appropriate for the nightlies, keeping into account PPC is slow... I would imagine that if you can easily implement an env variable to disable the timeout (only on some platforms?), maybe you can also do that for increasing the timeout (eg apply a factor 3-4 to PPC versus linux)?
Andrea

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-08-07 12:20, comment #14:

Hi Andrea,

just thought about it: what about an option (an environment variable) to disable the time-out?
I can also implement platform-specific time-outs, but I would be much more complicated.

Cheers
Marco

Marco Clemencic <clemenci>
Project Member
2007-08-07 12:10, comment #13:

Hi again,

as we just discussed I increased the timeouts to 1h for both sqlite and mysql. This is much more than the 24/27 minutes observed on bash tests, but is probably too much because the 24/27 minutes include coolDropDB, which is not included in the timeout. Will rerun and can then reduce the timeout later.

http://cool.cvs.cern.ch/cgi-bin/coo...
http://cool.cvs.cern.ch/cgi-bin/coo...

Andrea

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-08-07 11:54, comment #12:

Ciao Marco, thanks for having a look.

On PPC, it seems that tests are really very slow. Th etimeout is definitely not enough. See for instance:
http://cool.cvs.cern.ch/cgi-bin/coo...

=== Running 'test_RelationalCool_RelationalFolder' (mysql) ===
=== dbId: mysql://pclhcb55.cern.ch;schema=AVALASSI;dbname=COOLMPD_ ===
Thu Aug 2 17:51:50 CEST 2007
...
OK (82)
[OVAL] Cppunit-result =0
Thu Aug 2 18:17:38 CEST 2007

=== Running 'test_RelationalCool_RelationalFolder' (sqlite) ===
=== dbId: sqlite://;schema=/tmp/sqliteTest-avalassi-COOLMPD_.db;dbname=COOLMPD_ ===
Thu Aug 2 18:27:23 CEST 2007
...
OK (82)
[OVAL] Cppunit-result =0
Thu Aug 2 18:51:18 CEST 2007

It takes 26 minutes on mysql and 24 minutes on sqlite. The present timeouts only allow for 6 and 10 minutes, respectively. I would change the tiomoeuts (either at the global level, or only for this platform if possible). I would put one hour for both, just like for oracle... just in case. What do you think?

Thanks
Andrea

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-08-06 18:32, comment #11:

Hi Andrea,

I looked at your output. Actually I've seen it once in the nightlies too.

It seems to indicate that the QMTest time-out does not work on OSX.

You have the same error in:

  • relationalcool.mysql.relationalfolder
  • relationalcool.sqlite.relationalfolder

and for both the error arrives exactly after the time-out perios has elapsed.

I do not believe that the two tests took exactly the same time as the time allowed, and the kill just missed the target. It is most probable that it is a bug in QMTest. (Needs further investigation)

It would be interesting to understand if the time-out period is too small, or if the tests got stuck.

Cheers
Marco

Marco Clemencic <clemenci>
Project Member
2007-08-06 18:21, comment #10:

Confirm that private qmtest works:
http://cool.cvs.cern.ch/cgi-bin/coo...

Results are as expected except for one mysql failure, to be investigated, but probably not related to qmtest.

Bug closed.
Andrea

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-08-06 16:27, comment #9:

In the end I created my own QMTEST installation for Mac PPC, which was extremely quick.

Download the code from
http://www.codesourcery.com/qmtest/...
into /afs/cern.ch/sw/lcg/app/releases/COOL/internal/QMtest

On macsft001:
cd ~/myLCG/COOL_HEAD/src/config/cmt
source CMT_env.csh
source setup.csh
cd /afs/cern.ch/sw/lcg/app/releases/COOL/internal/QMtest
tar -xvzf qm-2.3.tar
cd qm-2.3
python setup.py build
python setup.py check
mkdir -p /afs/cern.ch/sw/lcg/app/releases/COOL/internal/QMtest/2.3.0_python242/osx104_ppc_gcc401/
python setup.py install --prefix=/afs/cern.ch/sw/lcg/app/releases/COOL/internal/QMtest/2.3.0_python242/osx104_ppc_gcc401/
ls
/afs/cern.ch/sw/lcg/app/releases/COOL/internal/QMtest/2.3.0_python242/osx104_ppc_gcc401/

Check OK:
ls /afs/cern.ch/sw/lcg/external/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib/
==> python2.3/
ls /afs/cern.ch/sw/lcg/app/releases/COOL/internal/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib/
==> python2.4/

The first tests fail with 'unexpected standard output'.
Thanks to Marco, he suggested that the SPI install may contain one
non-default line in command.py. Indeed:

diff -x'*pyc' -r \
/afs/cern.ch/sw/lcg/external/QMtest/2.3.0_python242/slc4_ia32_gcc34/lib \
/afs/cern.ch/sw/lcg/app/releases/COOL/internal/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib
returns one difference in command.py:
< # causes.append("standard output")
---

> causes.append("standard output")

Hence I change it to be like the one of SPI.

I tried one simple test
qmtest run coolkernel
and this is now successful.

I also changed the config to use my own version:
http://cool.cvs.cern.ch/cgi-bin/coo...

I will keep this open until I commit an ok logfile.

Andrea

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-07-30 13:47, comment #8:

Hi Stefan,

it does not happen in the nightlies.
I cannot check for PPC because I cannot find any test output for the last week. (than can be due to the failure in running qmtest)

Cheers
Marco

Marco Clemencic <clemenci>
Project Member
2007-07-30 13:42, comment #7:

FYI, in the last AF it was decided to drop osx104_ppc_gcc401 as an official LCG platform.

http://lcgapp.cern.ch/project/mgmt/...

Do you know whether this also happens for osx/intel?

Cheers

Stefan

Stefan Roiser <roiser>
Project Member
2007-07-30 13:40, comment #6:

I just opened SPI bug #28403 related to this.

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-07-30 13:32, comment #5:

Yes actually subprocess was only added after 220:
http://cool.cvs.cern.ch/cgi-bin/coo...

My tests using pure cool220 work ok because they do not need this module.

So this is indeed an LCGSPI bug in the python installation. Stefan, can you please have a look?

Thanks
Andrea

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-07-30 13:29, comment #4:

Ciao Marco, thanks.

Yes indeed the python23/24 mix looks strange.

I do not understand how come I manage to run from COOL220 however! This only seems to be a HEAD problem. By any chance, is it that you added this 'subprocess' in the qmtest config only after 220 was released?

Thanks
Andrea

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-07-30 13:22, comment #3:

Hi Andrea,

the problem seems to be something in the installation of QMTest for osx ppc.

The missing module "subprocess" is a standard Python module introduced in python 2.4.
The strange thing is that is seems that QMTest is configured to use python 2.3 on ppc:

/afs/.cern.ch/sw/lcg/external/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib/python2.3/site-packages

while it uses python 2.4 on the other platforms:

/afs/cern.ch/sw/lcg/external/QMtest/2.3.0_python242/slc4_ia32_gcc34/lib/python2.4/site-packages

Another funny thing is that the root directory of QMTest is "/afs/.cern.ch" instead of "/afs/cern.ch", but it may be that QMTest has been rebuilt in place and the pyc files point to ".cern.ch".

Anther thing that I find strange is that the python path set internally by QMTest is (on ppc):

['/afs/cern.ch/sw/lcg/external/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib/python2.3/site-packages', '/afs/cern.ch/user/m/marcocle/scratch0/workspace-europa/COOL/osx104_ppc_gcc401_dbg/lib', '/afs/cern.ch/sw/lcg/external/Grid/LFC/1.6.5/osx104_ppc_gcc401/lib/python', '/afs/cern.ch/user/m/marcocle/scratch0/workspace-europa/COOL/osx104_ppc_gcc401_dbg/python', '/afs/cern.ch/sw/lcg/external/root/5.14.00g/osx104_ppc_gcc401_dbg/root/lib', '/afs/cern.ch/sw/lcg/app/releases/CORAL/CORAL_1_9_0/osx104_ppc_gcc401_dbg/python', '/afs/cern.ch/sw/lcg/app/releases/CORAL/CORAL_1_9_0/osx104_ppc_gcc401_dbg/lib', '/afs/cern.ch/sw/lcg/app/spi/tools/0.6.9/python', '/sw/lib/python23.zip', '/sw/lib/python2.3', '/sw/lib/python2.3/plat-darwin', '/sw/lib/python2.3/plat-mac', '/sw/lib/python2.3/plat-mac/lib-scriptpackages', '/sw/lib/python2.3/lib-tk', '/sw/lib/python2.3/lib-dynload', '/sw/lib/python2.3/site-packages']

while it should look like:

['/afs/cern.ch/sw/lcg/external/QMtest/2.3.0_python242/slc4_ia32_gcc34/lib/python2.4/site-packages', '/afs/cern.ch/user/m/marcocle/scratch0/workspace-europa/COOL/slc4_ia32_gcc34_dbg/lib', '/afs/cern.ch/sw/lcg/external/Grid/LFC/1.6.5/slc4_ia32_gcc34/lib/python', '/afs/cern.ch/user/m/marcocle/scratch0/workspace-europa/COOL/slc4_ia32_gcc34_dbg/python', '/afs/cern.ch/sw/lcg/external/root/5.14.00g/slc4_ia32_gcc34_dbg/root/lib', '/afs/cern.ch/sw/lcg/app/releases/CORAL/CORAL_1_9_0/slc4_ia32_gcc34_dbg/python', '/afs/cern.ch/sw/lcg/app/releases/CORAL/CORAL_1_9_0/slc4_ia32_gcc34_dbg/lib', '/afs/cern.ch/sw/lcg/app/spi/tools/0.6.9/python', '/afs/cern.ch/user/m/marcocle/lib/python', '/afs/cern.ch/sw/lcg/external/Python/2.4.2/slc4_ia32_gcc346/lib/python24.zip', '/afs/cern.ch/sw/lcg/external/Python/2.4.2/slc4_ia32_gcc346/lib/python2.4', '/afs/cern.ch/sw/lcg/external/Python/2.4.2/slc4_ia32_gcc346/lib/python2.4/plat-linux2', '/afs/cern.ch/sw/lcg/external/Python/2.4.2/slc4_ia32_gcc346/lib/python2.4/lib-tk', '/afs/cern.ch/sw/lcg/external/Python/2.4.2/slc4_ia32_gcc346/lib/python2.4/lib-dynload', '/afs/cern.ch/sw/lcg/external/Python/2.4.2/slc4_ia32_gcc346/lib/python2.4/site-packages']

I can investigate a bit more. In the mean time I added Stefan in CC.

Cheers
Marco

Marco Clemencic <clemenci>
Project Member
2007-07-30 12:58, comment #2:

Just in case this was a CMTINSTALLAREA problems, I set it before the setup:

ssh macphsft01
klog
tcsh
cd /afs/cern.ch/user/a/avalassi/myLCG/COOL_HEAD/src/config/cmt/
setenv CMTINSTALLAREA /afs/cern.ch/user/a/avalassi/myLCG/COOL_HEAD/src
setenv CMTCONFIG osx104_ppc_gcc401_dbg
source CMT_env.csh
source setup.csh
cd ../qmtest
qmtest run -o /tmp/COOLHEAD-$CMTCONFIG.qmr coolkernel

However I see no difference: the qmr file binary differs, but the summary is the same
http://cool.cvs.cern.ch/cgi-bin/coo...

Not clear what the problem is...
NOte also that osx install is atrnage, it seems to mix python 23 and 24... but for COOL220 it works, so this is not the issue.

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-07-30 12:50, comment #1:

Fir reference, heres the full output:
http://cool.cvs.cern.ch/cgi-bin/coo...

Note that QMTEST_CLASS_PATH was defined in the test above (because of the way I chose to cd, without using ~/myLCG).

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.
2007-07-30 12:48, original submission:

The COOL220 config for osx seems ok. The following produces real results:

ssh macphsft01
klog
tcsh
cd /afs/cern.ch/sw/lcg/app/releases/COOL/COOL_2_2_0/src/config/cmt
setenv CMTCONFIG osx104_ppc_gcc401_dbg
source CMT_env.csh
source setup.csh
cd ../qmtest
qmtest run -o /tmp/COOL220-$CMTCONFIG.qmr cool

However, I do not manage to produce decent results from the HEAD:

ssh macphsft01
klog
tcsh
cd /afs/cern.ch/user/a/avalassi/myLCG/COOL_HEAD/src/config/cmt/
setenv CMTCONFIG osx104_ppc_gcc401_dbg
source CMT_env.csh
source setup.csh
cd ../qmtest
qmtest run -o /tmp/COOLHEAD-$CMTCONFIG.qmr coolkernel

This fails:

coolkernel.folderspec : UNTESTED
Could not load test.

qmtest.exception:

xml_database.TestFileError: A problem occurred while loading the XML test file "coolkernel.folderspec":
Exception 'exceptions.ImportError' : 'No module named subprocess'

File "/afs/.cern.ch/sw/lcg/external/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib/python2.3/site-packages/qm/test/base.py", line 223, in get_extension_class_from_directory
path + sys.path)

File "/afs/.cern.ch/sw/lcg/external/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib/python2.3/site-packages/qm/common.py", line 366, in load_class
module = load_module(module_name, search_path, load_path)

File "/afs/.cern.ch/sw/lcg/external/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib/python2.3/site-packages/qm/common.py", line 312, in load_module
module = imp.load_module(name, file, file_name, description)

File "/afs/cern.ch/user/a/avalassi/myLCG/COOL_HEAD/src/../src/config/qmtest/COOLTests.py", line 40, in ?
from subprocess import Popen, PIPE, STDOUT

The "COOLTests.StandardTest" extension class could not be loaded..

qmtest.traceback:

File "/afs/.cern.ch/sw/lcg/external/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib/python2.3/site-packages/qm/test/execution_engine.py", line 796, in __GetTestDescriptor
return self.__database.GetTest(test_id)

File "/afs/.cern.ch/sw/lcg/external/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib/python2.3/site-packages/qm/test/file_database.py", line 58, in GetTest
return self._GetTestFromPath(test_id, os.path.normpath(path))

File "/afs/.cern.ch/sw/lcg/external/QMtest/2.3.0_python242/osx104_ppc_gcc401/lib/python2.3/site-packages/qm/test/classes/xml_database.py", line 69, in _GetTestFromPath
raise TestFileError, message

What is wrong here?

Andrea Valassi <valassi>
Project AdministratorIn charge of this item.

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by roiser (Posted a comment)
  • -unavailable- added by clemenci (Posted a comment)
  • -unavailable- added by clemenci
  • -unavailable- added by valassi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independantly of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    2007-08-06 18:21valassiOpen/ClosedOpen=>Closed
    2007-08-06 16:27valassiStatusNone=>Fixed
      Assigned toNone=>valassi
    2007-07-30 13:32valassiSummaryFix QMTEST config for osx in COOL_HEAD=>'No module named subprocess' in QMTEST for osx in COOL_HEAD
    2007-07-30 13:22clemenciCarbon-Copy-=>Added roiser
    Show feedback again

    Back to the top


    Powered by Savane SVN (toward 3.1)