Mercurial > hg > truffle
annotate mx/sanitycheck.py @ 7267:a4b84ba6dc2e
Introduction of the Truffle API for efficient implementation of dynamic languages on top of the Graal VM. New projects com.oracle.truffle.api for the API definition and com.oracle.truffle.api.test for API tests and documentation.
author | Thomas Wuerthinger <thomas.wuerthinger@oracle.com> |
---|---|
date | Tue, 18 Dec 2012 15:33:55 +0100 |
parents | e3ada110d3cf |
children | dd903cdfe708 |
rev | line source |
---|---|
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
1 # ---------------------------------------------------------------------------------------------------- |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
2 # |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
3 # Copyright (c) 2007, 2012, Oracle and/or its affiliates. All rights reserved. |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
4 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
5 # |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
6 # This code is free software; you can redistribute it and/or modify it |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
7 # under the terms of the GNU General Public License version 2 only, as |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
8 # published by the Free Software Foundation. |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
9 # |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
10 # This code is distributed in the hope that it will be useful, but WITHOUT |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
11 # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
12 # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
13 # version 2 for more details (a copy is included in the LICENSE file that |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
14 # accompanied this code). |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
15 # |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
16 # You should have received a copy of the GNU General Public License version |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
17 # 2 along with this work; if not, write to the Free Software Foundation, |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
18 # Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
19 # |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
20 # Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
21 # or visit www.oracle.com if you need additional information or have any |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
22 # questions. |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
23 # |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
24 # ---------------------------------------------------------------------------------------------------- |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
25 |
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
26 from outputparser import OutputParser, Matcher |
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
27 import re |
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
28 import mx |
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
29 import os |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
30 from os.path import isfile, join, exists |
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
31 |
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
32 dacapoSanityWarmup = { |
4286
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
33 'avrora': [0, 0, 3, 6, 13], |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
34 'batik': [0, 0, 5, 5, 20], |
4286
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
35 'eclipse': [2, 4, 5, 10, 16], |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
36 'fop': [4, 8, 10, 20, 30], |
4286
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
37 'h2': [0, 0, 5, 5, 8], |
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
38 'jython': [0, 0, 5, 10, 13], |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
39 'luindex': [0, 0, 5, 10, 10], |
4286
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
40 'lusearch': [0, 4, 5, 5, 8], |
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
41 'pmd': [0, 0, 5, 10, 13], |
4482
7903b6c28f9c
Add hsdis command to mx
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4480
diff
changeset
|
42 'sunflow': [0, 2, 5, 10, 15], |
4286
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
43 'tomcat': [0, 0, 5, 10, 15], |
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
44 'tradebeans': [0, 0, 5, 10, 13], |
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
45 'tradesoap': [2, 4, 5, 10, 15], |
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
46 'xalan': [0, 0, 5, 10, 18], |
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
47 } |
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
48 |
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
49 dacapoScalaSanityWarmup = { |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
50 'actors': [0, 0, 2, 8, 10], |
6720
b62581e29024
disabled scaladacapo apparat benchmark
Lukas Stadler <lukas.stadler@jku.at>
parents:
6319
diff
changeset
|
51 # (lstadler) apparat was disabled due to a deadlock which I think is the benchmarks fault. |
b62581e29024
disabled scaladacapo apparat benchmark
Lukas Stadler <lukas.stadler@jku.at>
parents:
6319
diff
changeset
|
52 # 'apparat': [0, 0, 1, 2, 3], |
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
53 'factorie': [0, 0, 2, 5, 5], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
54 'kiama': [0, 0, 3, 13, 15], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
55 'scalac': [0, 0, 5, 15, 20], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
56 'scaladoc': [0, 0, 5, 15, 15], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
57 'scalap': [0, 0, 5, 15, 20], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
58 'scalariform':[0, 0, 6, 15, 20], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
59 'scalatest': [0, 0, 2, 10, 12], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
60 'scalaxb': [0, 0, 5, 15, 25], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
61 'specs': [0, 0, 3, 13, 18], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
62 'tmt': [0, 0, 3, 10, 12] |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
63 } |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
64 |
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
65 dacapoGateBuildLevels = { |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
66 'avrora': ['product', 'fastdebug', 'debug'], |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
67 'batik': ['product', 'fastdebug', 'debug'], |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
68 'eclipse': ['product'], |
4284
1339f1817d28
gate : do not run tradesoap in fastdebug mode, run fop in fastdebug mode only
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4282
diff
changeset
|
69 'fop': [ 'fastdebug', 'debug'], |
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
70 'h2': ['product', 'fastdebug', 'debug'], |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
71 'jython': ['product', 'fastdebug', 'debug'], |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
72 'luindex': ['product', 'fastdebug', 'debug'], |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
73 'lusearch': ['product'], |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
74 'pmd': ['product', 'fastdebug', 'debug'], |
6319
0a0d0dfd83bc
Gate : do not run tradesoap in fastdebug mode, do not run sunflow in product mode
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
6296
diff
changeset
|
75 'sunflow': [ 'fastdebug', 'debug'], |
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
76 'tomcat': ['product', 'fastdebug', 'debug'], |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
77 'tradebeans': ['product', 'fastdebug', 'debug'], |
6319
0a0d0dfd83bc
Gate : do not run tradesoap in fastdebug mode, do not run sunflow in product mode
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
6296
diff
changeset
|
78 'tradesoap': ['product'], |
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
79 'xalan': ['product', 'fastdebug', 'debug'], |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
80 } |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
81 |
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
82 dacapoScalaGateBuildLevels = { |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
83 'actors': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
84 'apparat': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
85 'factorie': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
86 'kiama': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
87 'scalac': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
88 'scaladoc': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
89 'scalap': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
90 'scalariform':['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
91 'scalatest': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
92 'scalaxb': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
93 'specs': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
94 'tmt': ['product', 'fastdebug', 'debug'], |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
95 } |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
96 |
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
97 class SanityCheckLevel: |
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
98 Fast, Gate, Normal, Extensive, Benchmark = range(5) |
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
99 |
5877
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
100 def getSPECjbb2005(benchArgs = []): |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
101 |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
102 specjbb2005 = mx.get_env('SPECJBB2005') |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
103 if specjbb2005 is None or not exists(join(specjbb2005, 'jbb.jar')): |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
104 mx.abort('Please set the SPECJBB2005 environment variable to a SPECjbb2005 directory') |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
105 |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
106 score = re.compile(r"^Valid run, Score is (?P<score>[0-9]+)$") |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
107 error = re.compile(r"VALIDATION ERROR") |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
108 success = re.compile(r"^Valid run, Score is [0-9]+$") |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
109 matcher = Matcher(score, {'const:group' : "const:SPECjbb2005", 'const:name' : 'const:score', 'const:score' : 'score'}) |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
110 classpath = ['jbb.jar', 'check.jar'] |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
111 return Test("SPECjbb2005", ['spec.jbb.JBBmain', '-propfile', 'SPECjbb.props'] + benchArgs, [success], [error], [matcher], vmOpts=['-Xms3g', '-XX:+UseSerialGC', '-cp', os.pathsep.join(classpath)], defaultCwd=specjbb2005) |
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
112 |
4334
0893aef10ed4
fix for SPECjvm2008 bench
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4332
diff
changeset
|
113 def getSPECjvm2008(benchArgs = [], skipKitValidation=False, warmupTime=None, iterationTime=None): |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
114 |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
115 specjvm2008 = mx.get_env('SPECJVM2008') |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
116 if specjvm2008 is None or not exists(join(specjvm2008, 'SPECjvm2008.jar')): |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
117 mx.abort('Please set the SPECJVM2008 environment variable to a SPECjvm2008 directory') |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
118 |
6296
89ed4e8c45c4
Update SPECjvm2008 bench regex
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5877
diff
changeset
|
119 score = re.compile(r"^(Score on|Noncompliant) (?P<benchmark>[a-zA-Z0-9\._]+)( result)?: (?P<score>[0-9]+((,|\.)[0-9]+)?)( SPECjvm2008 Base)? ops/m$") |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
120 error = re.compile(r"^Errors in benchmark: ") |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
121 # The ' ops/m' at the end of the success string is important : it's how you can tell valid and invalid runs apart |
6296
89ed4e8c45c4
Update SPECjvm2008 bench regex
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5877
diff
changeset
|
122 success = re.compile(r"^(Noncompliant c|C)omposite result: [0-9]+((,|\.)[0-9]+)?( SPECjvm2008 (Base|Peak))? ops/m$") |
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
123 matcher = Matcher(score, {'const:group' : "const:SPECjvm2008", 'const:name' : 'benchmark', 'const:score' : 'score'}, startNewLine=True) |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
124 |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
125 opts = [] |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
126 if warmupTime is not None: |
4230
b780ecb920c9
Add SPECjvm2008 to benchmarks, use dacapos defined in sanitycheck for the gate
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4228
diff
changeset
|
127 opts += ['-wt', str(warmupTime)] |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
128 if iterationTime is not None: |
4230
b780ecb920c9
Add SPECjvm2008 to benchmarks, use dacapos defined in sanitycheck for the gate
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4228
diff
changeset
|
129 opts += ['-it', str(iterationTime)] |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
130 if skipKitValidation: |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
131 opts += ['-ikv'] |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
132 |
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
133 return Test("SPECjvm2008", ['-jar', 'SPECjvm2008.jar'] + opts + benchArgs, [success], [error], [matcher], vmOpts=['-Xms3g', '-XX:+UseSerialGC'], defaultCwd=specjvm2008) |
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
134 |
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
135 def getDacapos(level=SanityCheckLevel.Normal, gateBuildLevel=None, dacapoArgs=[]): |
4217
f3271682fe5a
Various small fix around benchmarks in gate scripts
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4215
diff
changeset
|
136 checks = [] |
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
137 |
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
138 for (bench, ns) in dacapoSanityWarmup.items(): |
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
139 if ns[level] > 0: |
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
140 if gateBuildLevel is None or gateBuildLevel in dacapoGateBuildLevels[bench]: |
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
141 checks.append(getDacapo(bench, ns[level], dacapoArgs)) |
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
142 |
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
143 return checks |
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
144 |
4217
f3271682fe5a
Various small fix around benchmarks in gate scripts
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4215
diff
changeset
|
145 def getDacapo(name, n, dacapoArgs=[]): |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
146 dacapo = mx.get_env('DACAPO_CP') |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
147 if dacapo is None: |
4244
a69889e5a8a5
Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents:
4240
diff
changeset
|
148 l = mx.library('DACAPO', False) |
a69889e5a8a5
Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents:
4240
diff
changeset
|
149 if l is not None: |
a69889e5a8a5
Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents:
4240
diff
changeset
|
150 dacapo = l.get_path(True) |
a69889e5a8a5
Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents:
4240
diff
changeset
|
151 else: |
a69889e5a8a5
Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents:
4240
diff
changeset
|
152 mx.abort('DaCapo 9.12 jar file must be specified with DACAPO_CP environment variable or as DACAPO library') |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
153 |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
154 if not isfile(dacapo) or not dacapo.endswith('.jar'): |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
155 mx.abort('Specified DaCapo jar file does not exist or is not a jar file: ' + dacapo) |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
156 |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
157 dacapoSuccess = re.compile(r"^===== DaCapo 9\.12 ([a-zA-Z0-9_]+) PASSED in ([0-9]+) msec =====$") |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
158 dacapoFail = re.compile(r"^===== DaCapo 9\.12 ([a-zA-Z0-9_]+) FAILED (warmup|) =====$") |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
159 dacapoTime = re.compile(r"===== DaCapo 9\.12 (?P<benchmark>[a-zA-Z0-9_]+) PASSED in (?P<time>[0-9]+) msec =====") |
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
160 dacapoTime1 = re.compile(r"===== DaCapo 9\.12 (?P<benchmark>[a-zA-Z0-9_]+) completed warmup 1 in (?P<time>[0-9]+) msec =====") |
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
161 |
5196
beb13d0326ba
make DaCapo benchmark matchers produce multiple result lines
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5195
diff
changeset
|
162 dacapoMatcher = Matcher(dacapoTime, {'const:group' : "const:DaCapo", 'const:name' : 'benchmark', 'const:score' : 'time'}, startNewLine=True) |
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
163 dacapoMatcher1 = Matcher(dacapoTime1, {'const:group' : "const:DaCapo-1stRun", 'const:name' : 'benchmark', 'const:score' : 'time'}) |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
164 |
6990
e3ada110d3cf
Small cleanups after NPG
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
6720
diff
changeset
|
165 return Test("DaCapo-" + name, ['-jar', dacapo, name, '-n', str(n), ] + dacapoArgs, [dacapoSuccess], [dacapoFail], [dacapoMatcher, dacapoMatcher1], ['-Xms2g', '-XX:+UseSerialGC']) |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
166 |
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
167 def getScalaDacapos(level=SanityCheckLevel.Normal, gateBuildLevel=None, dacapoArgs=[]): |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
168 checks = [] |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
169 |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
170 for (bench, ns) in dacapoScalaSanityWarmup.items(): |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
171 if ns[level] > 0: |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
172 if gateBuildLevel is None or gateBuildLevel in dacapoScalaGateBuildLevels[bench]: |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
173 checks.append(getScalaDacapo(bench, ns[level], dacapoArgs)) |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
174 |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
175 return checks |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
176 |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
177 def getScalaDacapo(name, n, dacapoArgs=[]): |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
178 dacapo = mx.get_env('DACAPO_SCALA_CP') |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
179 if dacapo is None: |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
180 l = mx.library('DACAPO_SCALA', False) |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
181 if l is not None: |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
182 dacapo = l.get_path(True) |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
183 else: |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
184 mx.abort('Scala DaCapo 0.1.0 jar file must be specified with DACAPO_SCALA_CP environment variable or as DACAPO_SCALA library') |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
185 |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
186 if not isfile(dacapo) or not dacapo.endswith('.jar'): |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
187 mx.abort('Specified Scala DaCapo jar file does not exist or is not a jar file: ' + dacapo) |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
188 |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
189 dacapoSuccess = re.compile(r"^===== DaCapo 0\.1\.0(-SNAPSHOT)? ([a-zA-Z0-9_]+) PASSED in ([0-9]+) msec =====$") |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
190 dacapoFail = re.compile(r"^===== DaCapo 0\.1\.0(-SNAPSHOT)? ([a-zA-Z0-9_]+) FAILED (warmup|) =====$") |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
191 dacapoTime = re.compile(r"===== DaCapo 0\.1\.0(-SNAPSHOT)? (?P<benchmark>[a-zA-Z0-9_]+) PASSED in (?P<time>[0-9]+) msec =====") |
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
192 |
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
193 dacapoMatcher = Matcher(dacapoTime, {'const:group' : "const:Scala-DaCapo", 'const:name' : 'benchmark', 'const:score' : 'time'}) |
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
194 |
6990
e3ada110d3cf
Small cleanups after NPG
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
6720
diff
changeset
|
195 return Test("Scala-DaCapo-" + name, ['-jar', dacapo, name, '-n', str(n), ] + dacapoArgs, [dacapoSuccess], [dacapoFail], [dacapoMatcher], ['-Xms2g', '-XX:+UseSerialGC']) |
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
196 |
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
197 def getBootstraps(): |
4282
063ea022532c
mx commands : improve bench command, fix in the outparser so that no empty 'line' is retruned if there was no match, fix bootstrap's regex
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4244
diff
changeset
|
198 time = re.compile(r"Bootstrapping Graal\.+ in (?P<time>[0-9]+) ms") |
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
199 scoreMatcher = Matcher(time, {'const:group' : 'const:Bootstrap', 'const:name' : 'const:BootstrapTime', 'const:score' : 'time'}) |
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
200 scoreMatcherBig = Matcher(time, {'const:group' : 'const:Bootstrap-bigHeap', 'const:name' : 'const:BootstrapTime', 'const:score' : 'time'}) |
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
201 |
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
202 tests = [] |
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
203 tests.append(Test("Bootstrap", ['-version'], successREs=[time], scoreMatchers=[scoreMatcher], ingoreVms=['client', 'server'])) |
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
204 tests.append(Test("Bootstrap-bigHeap", ['-version'], successREs=[time], scoreMatchers=[scoreMatcherBig], vmOpts=['-Xms2g'], ingoreVms=['client', 'server'])) |
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
205 return tests |
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
206 |
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
207 """ |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
208 Encapsulates a single program that is a sanity test and/or a benchmark. |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
209 """ |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
210 class Test: |
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
211 def __init__(self, name, cmd, successREs=[], failureREs=[], scoreMatchers=[], vmOpts=[], defaultCwd=None, ingoreVms=[]): |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
212 self.name = name |
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
213 self.successREs = successREs |
4416
be787de79394
Fail bench/test on "Exception occured in scope: "
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4335
diff
changeset
|
214 self.failureREs = failureREs + [re.compile(r"Exception occured in scope: ")] |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
215 self.scoreMatchers = scoreMatchers |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
216 self.vmOpts = vmOpts |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
217 self.cmd = cmd |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
218 self.defaultCwd = defaultCwd |
5143
f5511facb897
Add posibility to ignore some tests/benchmarks with some vms : use it to disable bootstrap benchmarks for client/server
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5045
diff
changeset
|
219 self.ingoreVms = ingoreVms; |
f5511facb897
Add posibility to ignore some tests/benchmarks with some vms : use it to disable bootstrap benchmarks for client/server
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5045
diff
changeset
|
220 |
4238
e0d09e05aa9b
Made the gate time each DaCapo test individually.
Doug Simon <doug.simon@oracle.com>
parents:
4230
diff
changeset
|
221 |
e0d09e05aa9b
Made the gate time each DaCapo test individually.
Doug Simon <doug.simon@oracle.com>
parents:
4230
diff
changeset
|
222 def __str__(self): |
e0d09e05aa9b
Made the gate time each DaCapo test individually.
Doug Simon <doug.simon@oracle.com>
parents:
4230
diff
changeset
|
223 return self.name |
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
224 |
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
225 def test(self, vm, cwd=None, opts=[], vmbuild=None): |
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
226 """ |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
227 Run this program as a sanity test. |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
228 """ |
5143
f5511facb897
Add posibility to ignore some tests/benchmarks with some vms : use it to disable bootstrap benchmarks for client/server
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5045
diff
changeset
|
229 if (vm in self.ingoreVms): |
f5511facb897
Add posibility to ignore some tests/benchmarks with some vms : use it to disable bootstrap benchmarks for client/server
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5045
diff
changeset
|
230 return True; |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
231 if cwd is None: |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
232 cwd = self.defaultCwd |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
233 parser = OutputParser(nonZeroIsFatal = False) |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
234 jvmError = re.compile(r"(?P<jvmerror>([A-Z]:|/).*[/\\]hs_err_pid[0-9]+\.log)") |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
235 parser.addMatcher(Matcher(jvmError, {'const:jvmError' : 'jvmerror'})) |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
236 |
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
237 for successRE in self.successREs: |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
238 parser.addMatcher(Matcher(successRE, {'const:passed' : 'const:1'})) |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
239 for failureRE in self.failureREs: |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
240 parser.addMatcher(Matcher(failureRE, {'const:failed' : 'const:1'})) |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
241 |
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
242 result = parser.parse(vm, self.vmOpts + opts + self.cmd, cwd, vmbuild) |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
243 |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
244 parsedLines = result['parsed'] |
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
245 if len(parsedLines) == 0: |
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
246 return False |
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
247 |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
248 assert len(parsedLines) == 1, 'Test matchers should not return more than one line' |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
249 |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
250 parsed = parsedLines[0] |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
251 |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
252 if parsed.has_key('jvmError'): |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
253 mx.log('/!\\JVM Error : dumping error log...') |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
254 f = open(parsed['jvmError'], 'rb'); |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
255 for line in iter(f.readline, ''): |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
256 mx.log(line.rstrip()) |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
257 f.close() |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
258 os.unlink(parsed['jvmError']) |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
259 return False |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
260 |
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
261 if parsed.has_key('failed') and parsed['failed'] is '1': |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
262 return False |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
263 |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
264 return result['retcode'] is 0 and parsed.has_key('passed') and parsed['passed'] is '1' |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
265 |
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
266 def bench(self, vm, cwd=None, opts=[], vmbuild=None): |
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
267 """ |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
268 Run this program as a benchmark. |
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
269 """ |
5143
f5511facb897
Add posibility to ignore some tests/benchmarks with some vms : use it to disable bootstrap benchmarks for client/server
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5045
diff
changeset
|
270 if (vm in self.ingoreVms): |
f5511facb897
Add posibility to ignore some tests/benchmarks with some vms : use it to disable bootstrap benchmarks for client/server
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5045
diff
changeset
|
271 return {}; |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
272 if cwd is None: |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
273 cwd = self.defaultCwd |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
274 parser = OutputParser(nonZeroIsFatal = False) |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
275 |
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
276 for successRE in self.successREs: |
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
277 parser.addMatcher(Matcher(successRE, {'const:passed' : 'const:1'})) |
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
278 for failureRE in self.failureREs: |
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
279 parser.addMatcher(Matcher(failureRE, {'const:failed' : 'const:1'})) |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
280 for scoreMatcher in self.scoreMatchers: |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
281 parser.addMatcher(scoreMatcher) |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
282 |
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
283 result = parser.parse(vm, self.vmOpts + opts + self.cmd, cwd, vmbuild) |
4217
f3271682fe5a
Various small fix around benchmarks in gate scripts
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4215
diff
changeset
|
284 if result['retcode'] is not 0: |
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
285 mx.abort("Benchmark failed (non-zero retcode)") |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
286 |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
287 parsed = result['parsed'] |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
288 |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
289 ret = {} |
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
290 |
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
291 passed = False; |
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
292 |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
293 for line in parsed: |
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
294 assert (line.has_key('name') and line.has_key('score') and line.has_key('group')) or line.has_key('passed') or line.has_key('failed') |
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
295 if line.has_key('failed') and line['failed'] is '1': |
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
296 mx.abort("Benchmark failed") |
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
297 if line.has_key('passed') and line['passed'] is '1': |
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
298 passed = True |
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
299 if line.has_key('name') and line.has_key('score') and line.has_key('group'): |
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
300 if not ret.has_key(line['group']): |
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
301 ret[line['group']] = {}; |
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
302 ret[line['group']][line['name']] = line['score'] |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
303 |
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
304 if not passed: |
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
305 mx.abort("Benchmark failed (not passed)") |
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
306 |
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
307 return ret |