Mercurial > hg > truffle
annotate mx/sanitycheck.py @ 7564:c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
| author | Doug Simon <doug.simon@oracle.com> |
|---|---|
| date | Wed, 30 Jan 2013 11:33:31 +0100 |
| parents | 3aab15f42934 |
| children | 9a521597686b a8bc60aeacb8 |
| rev | line source |
|---|---|
|
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
1 # ---------------------------------------------------------------------------------------------------- |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
2 # |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
3 # Copyright (c) 2007, 2012, Oracle and/or its affiliates. All rights reserved. |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
4 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
5 # |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
6 # This code is free software; you can redistribute it and/or modify it |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
7 # under the terms of the GNU General Public License version 2 only, as |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
8 # published by the Free Software Foundation. |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
9 # |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
10 # This code is distributed in the hope that it will be useful, but WITHOUT |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
11 # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
12 # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
13 # version 2 for more details (a copy is included in the LICENSE file that |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
14 # accompanied this code). |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
15 # |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
16 # You should have received a copy of the GNU General Public License version |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
17 # 2 along with this work; if not, write to the Free Software Foundation, |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
18 # Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
19 # |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
20 # Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
21 # or visit www.oracle.com if you need additional information or have any |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
22 # questions. |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
23 # |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
24 # ---------------------------------------------------------------------------------------------------- |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
25 |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
26 from outputparser import OutputParser, ValuesMatcher |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
27 import re, mx, commands, os, sys, StringIO, subprocess |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
28 from os.path import isfile, join, exists |
|
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
29 |
|
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
30 dacapoSanityWarmup = { |
|
4286
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
31 'avrora': [0, 0, 3, 6, 13], |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
32 'batik': [0, 0, 5, 5, 20], |
|
4286
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
33 'eclipse': [2, 4, 5, 10, 16], |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
34 'fop': [4, 8, 10, 20, 30], |
|
4286
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
35 'h2': [0, 0, 5, 5, 8], |
|
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
36 'jython': [0, 0, 5, 10, 13], |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
37 'luindex': [0, 0, 5, 10, 10], |
|
4286
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
38 'lusearch': [0, 4, 5, 5, 8], |
|
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
39 'pmd': [0, 0, 5, 10, 13], |
|
4482
7903b6c28f9c
Add hsdis command to mx
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4480
diff
changeset
|
40 'sunflow': [0, 2, 5, 10, 15], |
|
4286
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
41 'tomcat': [0, 0, 5, 10, 15], |
|
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
42 'tradebeans': [0, 0, 5, 10, 13], |
|
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
43 'tradesoap': [2, 4, 5, 10, 15], |
|
76190108a9f2
Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4284
diff
changeset
|
44 'xalan': [0, 0, 5, 10, 18], |
|
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
45 } |
|
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
46 |
|
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
47 dacapoScalaSanityWarmup = { |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
48 'actors': [0, 0, 2, 8, 10], |
|
6720
b62581e29024
disabled scaladacapo apparat benchmark
Lukas Stadler <lukas.stadler@jku.at>
parents:
6319
diff
changeset
|
49 # (lstadler) apparat was disabled due to a deadlock which I think is the benchmarks fault. |
|
b62581e29024
disabled scaladacapo apparat benchmark
Lukas Stadler <lukas.stadler@jku.at>
parents:
6319
diff
changeset
|
50 # 'apparat': [0, 0, 1, 2, 3], |
|
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
51 'factorie': [0, 0, 2, 5, 5], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
52 'kiama': [0, 0, 3, 13, 15], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
53 'scalac': [0, 0, 5, 15, 20], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
54 'scaladoc': [0, 0, 5, 15, 15], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
55 'scalap': [0, 0, 5, 15, 20], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
56 'scalariform':[0, 0, 6, 15, 20], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
57 'scalatest': [0, 0, 2, 10, 12], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
58 'scalaxb': [0, 0, 5, 15, 25], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
59 'specs': [0, 0, 3, 13, 18], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
60 'tmt': [0, 0, 3, 10, 12] |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
61 } |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
62 |
|
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
63 dacapoGateBuildLevels = { |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
64 'avrora': ['product', 'fastdebug', 'debug'], |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
65 'batik': ['product', 'fastdebug', 'debug'], |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
66 'eclipse': ['product'], |
|
4284
1339f1817d28
gate : do not run tradesoap in fastdebug mode, run fop in fastdebug mode only
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4282
diff
changeset
|
67 'fop': [ 'fastdebug', 'debug'], |
|
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
68 'h2': ['product', 'fastdebug', 'debug'], |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
69 'jython': ['product', 'fastdebug', 'debug'], |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
70 'luindex': ['product', 'fastdebug', 'debug'], |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
71 'lusearch': ['product'], |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
72 'pmd': ['product', 'fastdebug', 'debug'], |
|
6319
0a0d0dfd83bc
Gate : do not run tradesoap in fastdebug mode, do not run sunflow in product mode
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
6296
diff
changeset
|
73 'sunflow': [ 'fastdebug', 'debug'], |
|
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
74 'tomcat': ['product', 'fastdebug', 'debug'], |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
75 'tradebeans': ['product', 'fastdebug', 'debug'], |
|
6319
0a0d0dfd83bc
Gate : do not run tradesoap in fastdebug mode, do not run sunflow in product mode
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
6296
diff
changeset
|
76 'tradesoap': ['product'], |
|
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
77 'xalan': ['product', 'fastdebug', 'debug'], |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
78 } |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
79 |
|
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
80 dacapoScalaGateBuildLevels = { |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
81 'actors': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
82 'apparat': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
83 'factorie': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
84 'kiama': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
85 'scalac': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
86 'scaladoc': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
87 'scalap': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
88 'scalariform':['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
89 'scalatest': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
90 'scalaxb': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
91 'specs': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
92 'tmt': ['product', 'fastdebug', 'debug'], |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
93 } |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
94 |
|
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
95 class SanityCheckLevel: |
|
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
96 Fast, Gate, Normal, Extensive, Benchmark = range(5) |
|
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
97 |
|
5877
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
98 def getSPECjbb2005(benchArgs = []): |
|
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
99 |
|
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
100 specjbb2005 = mx.get_env('SPECJBB2005') |
|
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
101 if specjbb2005 is None or not exists(join(specjbb2005, 'jbb.jar')): |
|
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
102 mx.abort('Please set the SPECJBB2005 environment variable to a SPECjbb2005 directory') |
|
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
103 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
104 score = re.compile(r"^Valid run, Score is (?P<score>[0-9]+)$", re.MULTILINE) |
|
5877
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
105 error = re.compile(r"VALIDATION ERROR") |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
106 success = re.compile(r"^Valid run, Score is [0-9]+$", re.MULTILINE) |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
107 matcher = ValuesMatcher(score, {'group' : 'SPECjbb2005', 'name' : 'score', 'score' : '<score>'}) |
|
5877
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
108 classpath = ['jbb.jar', 'check.jar'] |
|
7367
cbcee2e1ce51
Make sure that compressed oops are not used in benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
7345
diff
changeset
|
109 return Test("SPECjbb2005", ['spec.jbb.JBBmain', '-propfile', 'SPECjbb.props'] + benchArgs, [success], [error], [matcher], vmOpts=['-Xms3g', '-XX:+UseSerialGC', '-XX:-UseCompressedOops', '-cp', os.pathsep.join(classpath)], defaultCwd=specjbb2005) |
|
5877
0e54d9bb922d
Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5196
diff
changeset
|
110 |
|
7345
dd903cdfe708
Add possibility for -ict option to SPECjvm2008 benchmark execution.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents:
6990
diff
changeset
|
111 def getSPECjvm2008(benchArgs = [], skipCheck=False, skipKitValidation=False, warmupTime=None, iterationTime=None): |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
112 |
|
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
113 specjvm2008 = mx.get_env('SPECJVM2008') |
|
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
114 if specjvm2008 is None or not exists(join(specjvm2008, 'SPECjvm2008.jar')): |
|
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
115 mx.abort('Please set the SPECJVM2008 environment variable to a SPECjvm2008 directory') |
|
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
116 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
117 score = re.compile(r"^(Score on|Noncompliant) (?P<benchmark>[a-zA-Z0-9\._]+)( result)?: (?P<score>[0-9]+((,|\.)[0-9]+)?)( SPECjvm2008 Base)? ops/m$", re.MULTILINE) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
118 error = re.compile(r"^Errors in benchmark: ", re.MULTILINE) |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
119 # The ' ops/m' at the end of the success string is important : it's how you can tell valid and invalid runs apart |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
120 success = re.compile(r"^(Noncompliant c|C)omposite result: [0-9]+((,|\.)[0-9]+)?( SPECjvm2008 (Base|Peak))? ops/m$", re.MULTILINE) |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
121 matcher = ValuesMatcher(score, {'group' : 'SPECjvm2008', 'name' : '<benchmark>', 'score' : '<score>'}) |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
122 |
|
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
123 opts = [] |
|
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
124 if warmupTime is not None: |
|
4230
b780ecb920c9
Add SPECjvm2008 to benchmarks, use dacapos defined in sanitycheck for the gate
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4228
diff
changeset
|
125 opts += ['-wt', str(warmupTime)] |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
126 if iterationTime is not None: |
|
4230
b780ecb920c9
Add SPECjvm2008 to benchmarks, use dacapos defined in sanitycheck for the gate
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4228
diff
changeset
|
127 opts += ['-it', str(iterationTime)] |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
128 if skipKitValidation: |
|
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
129 opts += ['-ikv'] |
|
7345
dd903cdfe708
Add possibility for -ict option to SPECjvm2008 benchmark execution.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents:
6990
diff
changeset
|
130 if skipCheck: |
|
dd903cdfe708
Add possibility for -ict option to SPECjvm2008 benchmark execution.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents:
6990
diff
changeset
|
131 opts += ['-ict'] |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
132 |
|
7367
cbcee2e1ce51
Make sure that compressed oops are not used in benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
7345
diff
changeset
|
133 return Test("SPECjvm2008", ['-jar', 'SPECjvm2008.jar'] + opts + benchArgs, [success], [error], [matcher], vmOpts=['-Xms3g', '-XX:+UseSerialGC', '-XX:-UseCompressedOops'], defaultCwd=specjvm2008) |
|
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
134 |
|
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
135 def getDacapos(level=SanityCheckLevel.Normal, gateBuildLevel=None, dacapoArgs=[]): |
|
4217
f3271682fe5a
Various small fix around benchmarks in gate scripts
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4215
diff
changeset
|
136 checks = [] |
|
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
137 |
|
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
138 for (bench, ns) in dacapoSanityWarmup.items(): |
|
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
139 if ns[level] > 0: |
|
4240
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
140 if gateBuildLevel is None or gateBuildLevel in dacapoGateBuildLevels[bench]: |
|
394424b7df1a
Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents:
4238
diff
changeset
|
141 checks.append(getDacapo(bench, ns[level], dacapoArgs)) |
|
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
142 |
|
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
143 return checks |
|
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
144 |
|
4217
f3271682fe5a
Various small fix around benchmarks in gate scripts
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4215
diff
changeset
|
145 def getDacapo(name, n, dacapoArgs=[]): |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
146 dacapo = mx.get_env('DACAPO_CP') |
|
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
147 if dacapo is None: |
|
4244
a69889e5a8a5
Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents:
4240
diff
changeset
|
148 l = mx.library('DACAPO', False) |
|
a69889e5a8a5
Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents:
4240
diff
changeset
|
149 if l is not None: |
|
a69889e5a8a5
Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents:
4240
diff
changeset
|
150 dacapo = l.get_path(True) |
|
a69889e5a8a5
Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents:
4240
diff
changeset
|
151 else: |
|
a69889e5a8a5
Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents:
4240
diff
changeset
|
152 mx.abort('DaCapo 9.12 jar file must be specified with DACAPO_CP environment variable or as DACAPO library') |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
153 |
|
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
154 if not isfile(dacapo) or not dacapo.endswith('.jar'): |
|
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
155 mx.abort('Specified DaCapo jar file does not exist or is not a jar file: ' + dacapo) |
|
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
156 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
157 dacapoSuccess = re.compile(r"^===== DaCapo 9\.12 ([a-zA-Z0-9_]+) PASSED in ([0-9]+) msec =====$", re.MULTILINE) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
158 dacapoFail = re.compile(r"^===== DaCapo 9\.12 ([a-zA-Z0-9_]+) FAILED (warmup|) =====$", re.MULTILINE) |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
159 dacapoTime = re.compile(r"===== DaCapo 9\.12 (?P<benchmark>[a-zA-Z0-9_]+) PASSED in (?P<time>[0-9]+) msec =====") |
|
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
160 dacapoTime1 = re.compile(r"===== DaCapo 9\.12 (?P<benchmark>[a-zA-Z0-9_]+) completed warmup 1 in (?P<time>[0-9]+) msec =====") |
|
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
161 |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
162 dacapoMatcher = ValuesMatcher(dacapoTime, {'group' : 'DaCapo', 'name' : '<benchmark>', 'score' : '<time>'}) |
|
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
163 dacapoMatcher1 = ValuesMatcher(dacapoTime1, {'group' : 'DaCapo-1stRun', 'name' : '<benchmark>', 'score' : '<time>'}) |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
164 |
|
7367
cbcee2e1ce51
Make sure that compressed oops are not used in benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
7345
diff
changeset
|
165 return Test("DaCapo-" + name, ['-jar', dacapo, name, '-n', str(n), ] + dacapoArgs, [dacapoSuccess], [dacapoFail], [dacapoMatcher, dacapoMatcher1], ['-Xms2g', '-XX:+UseSerialGC', '-XX:-UseCompressedOops']) |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
166 |
|
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
167 def getScalaDacapos(level=SanityCheckLevel.Normal, gateBuildLevel=None, dacapoArgs=[]): |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
168 checks = [] |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
169 |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
170 for (bench, ns) in dacapoScalaSanityWarmup.items(): |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
171 if ns[level] > 0: |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
172 if gateBuildLevel is None or gateBuildLevel in dacapoScalaGateBuildLevels[bench]: |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
173 checks.append(getScalaDacapo(bench, ns[level], dacapoArgs)) |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
174 |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
175 return checks |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
176 |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
177 def getScalaDacapo(name, n, dacapoArgs=[]): |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
178 dacapo = mx.get_env('DACAPO_SCALA_CP') |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
179 if dacapo is None: |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
180 l = mx.library('DACAPO_SCALA', False) |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
181 if l is not None: |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
182 dacapo = l.get_path(True) |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
183 else: |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
184 mx.abort('Scala DaCapo 0.1.0 jar file must be specified with DACAPO_SCALA_CP environment variable or as DACAPO_SCALA library') |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
185 |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
186 if not isfile(dacapo) or not dacapo.endswith('.jar'): |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
187 mx.abort('Specified Scala DaCapo jar file does not exist or is not a jar file: ' + dacapo) |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
188 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
189 dacapoSuccess = re.compile(r"^===== DaCapo 0\.1\.0(-SNAPSHOT)? ([a-zA-Z0-9_]+) PASSED in ([0-9]+) msec =====$", re.MULTILINE) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
190 dacapoFail = re.compile(r"^===== DaCapo 0\.1\.0(-SNAPSHOT)? ([a-zA-Z0-9_]+) FAILED (warmup|) =====$", re.MULTILINE) |
|
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
191 dacapoTime = re.compile(r"===== DaCapo 0\.1\.0(-SNAPSHOT)? (?P<benchmark>[a-zA-Z0-9_]+) PASSED in (?P<time>[0-9]+) msec =====") |
|
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
192 |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
193 dacapoMatcher = ValuesMatcher(dacapoTime, {'group' : "Scala-DaCapo", 'name' : '<benchmark>', 'score' : '<time>'}) |
|
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
194 |
|
7367
cbcee2e1ce51
Make sure that compressed oops are not used in benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
7345
diff
changeset
|
195 return Test("Scala-DaCapo-" + name, ['-jar', dacapo, name, '-n', str(n), ] + dacapoArgs, [dacapoSuccess], [dacapoFail], [dacapoMatcher], ['-Xms2g', '-XX:+UseSerialGC', '-XX:-UseCompressedOops']) |
|
4480
7d6490436b57
Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4416
diff
changeset
|
196 |
|
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
197 def getBootstraps(): |
|
4282
063ea022532c
mx commands : improve bench command, fix in the outparser so that no empty 'line' is retruned if there was no match, fix bootstrap's regex
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4244
diff
changeset
|
198 time = re.compile(r"Bootstrapping Graal\.+ in (?P<time>[0-9]+) ms") |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
199 scoreMatcher = ValuesMatcher(time, {'group' : 'Bootstrap', 'name' : 'BootstrapTime', 'score' : '<time>'}) |
|
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
200 scoreMatcherBig = ValuesMatcher(time, {'group' : 'Bootstrap-bigHeap', 'name' : 'BootstrapTime', 'score' : '<time>'}) |
|
5195
8125c1f6472b
Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
5148
diff
changeset
|
201 |
|
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
202 tests = [] |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
203 tests.append(Test("Bootstrap", ['-version'], successREs=[time], scoreMatchers=[scoreMatcher], ignoredVMs=['client', 'server'], benchmarkCompilationRate=False)) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
204 tests.append(Test("Bootstrap-bigHeap", ['-version'], successREs=[time], scoreMatchers=[scoreMatcherBig], vmOpts=['-Xms2g'], ignoredVMs=['client', 'server'], benchmarkCompilationRate=False)) |
|
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
205 return tests |
|
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
206 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
207 class Tee: |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
208 def __init__(self): |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
209 self.output = StringIO.StringIO() |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
210 def eat(self, line): |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
211 self.output.write(line) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
212 sys.stdout.write(line) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
213 |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
214 _debugBenchParser = False |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
215 |
|
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
216 """ |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
217 Encapsulates a single program that is a sanity test and/or a benchmark. |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
218 """ |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
219 class Test: |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
220 def __init__(self, name, cmd, successREs=[], failureREs=[], scoreMatchers=[], vmOpts=[], defaultCwd=None, ignoredVMs=[], benchmarkCompilationRate=True): |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
221 |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
222 self.name = name |
|
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
223 self.successREs = successREs |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
224 self.failureREs = failureREs + [re.compile(r"Exception occurred in scope: ")] |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
225 self.scoreMatchers = scoreMatchers |
|
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
226 self.vmOpts = vmOpts |
|
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
227 self.cmd = cmd |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
228 self.defaultCwd = defaultCwd |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
229 self.ignoredVMs = ignoredVMs |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
230 self.benchmarkCompilationRate = benchmarkCompilationRate |
|
4238
e0d09e05aa9b
Made the gate time each DaCapo test individually.
Doug Simon <doug.simon@oracle.com>
parents:
4230
diff
changeset
|
231 |
|
e0d09e05aa9b
Made the gate time each DaCapo test individually.
Doug Simon <doug.simon@oracle.com>
parents:
4230
diff
changeset
|
232 def __str__(self): |
|
e0d09e05aa9b
Made the gate time each DaCapo test individually.
Doug Simon <doug.simon@oracle.com>
parents:
4230
diff
changeset
|
233 return self.name |
|
4150
c78bace5086a
start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff
changeset
|
234 |
|
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
235 def test(self, vm, cwd=None, opts=[], vmbuild=None): |
|
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
236 """ |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
237 Run this program as a sanity test. |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
238 """ |
| 7562 | 239 if (vm in self.ignoredVMs): |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
240 return True |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
241 if cwd is None: |
|
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
242 cwd = self.defaultCwd |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
243 parser = OutputParser() |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
244 jvmError = re.compile(r"(?P<jvmerror>([A-Z]:|/).*[/\\]hs_err_pid[0-9]+\.log)") |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
245 parser.addMatcher(ValuesMatcher(jvmError, {'jvmError' : '<jvmerror>'})) |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
246 |
|
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
247 for successRE in self.successREs: |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
248 parser.addMatcher(ValuesMatcher(successRE, {'passed' : '1'})) |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
249 for failureRE in self.failureREs: |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
250 parser.addMatcher(ValuesMatcher(failureRE, {'failed' : '1'})) |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
251 |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
252 tee = Tee() |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
253 retcode = commands.vm(self.vmOpts + opts + self.cmd, vm, nonZeroIsFatal=False, out=tee.eat, err=subprocess.STDOUT, cwd=cwd, vmbuild=vmbuild) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
254 output = tee.output.getvalue() |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
255 valueMaps = parser.parse(output) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
256 |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
257 if len(valueMaps) == 0: |
|
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
258 return False |
|
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
259 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
260 assert len(valueMaps) == 1, 'Test matchers should not return more than one record' |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
261 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
262 record = valueMaps[0] |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
263 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
264 jvmErrorFile = record.get('jvmError') |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
265 if jvmErrorFile: |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
266 mx.log('/!\\JVM Error : dumping error log...') |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
267 with open(jvmErrorFile, 'rb') as fp: |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
268 mx.log(fp.read()) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
269 os.unlink(jvmErrorFile) |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
270 return False |
|
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
271 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
272 if record.get('failed') == '1': |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
273 return False |
|
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
274 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
275 return retcode == 0 and record.get('passed') == '1' |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
276 |
|
4219
47f7d91d34cf
Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4217
diff
changeset
|
277 def bench(self, vm, cwd=None, opts=[], vmbuild=None): |
|
4225
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
278 """ |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
279 Run this program as a benchmark. |
|
339cf8d4904d
Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents:
4219
diff
changeset
|
280 """ |
| 7562 | 281 if (vm in self.ignoredVMs): |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
282 return {} |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
283 if cwd is None: |
|
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
284 cwd = self.defaultCwd |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
285 parser = OutputParser() |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
286 |
|
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
287 for successRE in self.successREs: |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
288 parser.addMatcher(ValuesMatcher(successRE, {'passed' : '1'})) |
|
4228
e872562f95f8
add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4225
diff
changeset
|
289 for failureRE in self.failureREs: |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
290 parser.addMatcher(ValuesMatcher(failureRE, {'failed' : '1'})) |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
291 for scoreMatcher in self.scoreMatchers: |
|
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
292 parser.addMatcher(scoreMatcher) |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
293 |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
294 if self.benchmarkCompilationRate: |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
295 opts.append('-Dgraal.benchmark.compilation=true') |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
296 bps = re.compile(r"ParsedBytecodesPerSecond@final: (?P<rate>[0-9]+)") |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
297 ibps = re.compile(r"InlinedBytecodesPerSecond@final: (?P<rate>[0-9]+)") |
|
7564
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
298 parser.addMatcher(ValuesMatcher(bps, {'group' : 'ParsedBytecodesPerSecond', 'name' : self.name, 'score' : '<rate>'})) |
|
c420a487b10f
changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents:
7563
diff
changeset
|
299 parser.addMatcher(ValuesMatcher(ibps, {'group' : 'InlinedBytecodesPerSecond', 'name' : self.name, 'score' : '<rate>'})) |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
300 |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
301 outputfile = self.name + '.output' |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
302 if _debugBenchParser and exists(outputfile): |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
303 with open(outputfile) as fp: |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
304 output = fp.read() |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
305 mx.log(output) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
306 else: |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
307 tee = Tee() |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
308 if commands.vm(self.vmOpts + opts + self.cmd, vm, nonZeroIsFatal=False, out=tee.eat, err=subprocess.STDOUT, cwd=cwd, vmbuild=vmbuild) != 0: |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
309 mx.abort("Benchmark failed (non-zero retcode)") |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
310 output = tee.output.getvalue() |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
311 if _debugBenchParser: |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
312 with open(outputfile, 'wb') as fp: |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
313 fp.write(output) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
314 |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
315 ret = {} |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
316 passed = False |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
317 for valueMap in parser.parse(output): |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
318 assert (valueMap.has_key('name') and valueMap.has_key('score') and valueMap.has_key('group')) or valueMap.has_key('passed') or valueMap.has_key('failed'), valueMap |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
319 if valueMap.get('failed') == '1': |
|
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
320 mx.abort("Benchmark failed") |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
321 if valueMap.get('passed') == '1': |
|
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
322 passed = True |
|
7563
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
323 groupName = valueMap.get('group') |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
324 if groupName: |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
325 group = ret.setdefault(groupName, {}) |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
326 name = valueMap.get('name') |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
327 score = valueMap.get('score') |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
328 if name and score: |
|
3aab15f42934
moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents:
7562
diff
changeset
|
329 group[name] = score |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
330 |
|
4332
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
331 if not passed: |
|
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
332 mx.abort("Benchmark failed (not passed)") |
|
9dbcdd6f1464
If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents:
4331
diff
changeset
|
333 |
|
4215
a2caa019ba3a
Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
4150
diff
changeset
|
334 return ret |