Mercurial > hg > truffle

annotate mx/sanitycheck.py @ 8147:9786ac8fff61

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Use different vm args for SPECjbb2013
author Gilles Duboscq <duboscq@ssw.jku.at>
date Thu, 07 Mar 2013 11:58:34 +0100
parents 01aeaf194641
children 22429580c7a8
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4225
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
1 # ----------------------------------------------------------------------------------------------------
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
2 #
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
3 # Copyright (c) 2007, 2012, Oracle and/or its affiliates. All rights reserved.
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
4 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
5 #
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
6 # This code is free software; you can redistribute it and/or modify it
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
7 # under the terms of the GNU General Public License version 2 only, as
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
8 # published by the Free Software Foundation.
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
9 #
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
10 # This code is distributed in the hope that it will be useful, but WITHOUT
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
11 # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
12 # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
13 # version 2 for more details (a copy is included in the LICENSE file that
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
14 # accompanied this code).
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
15 #
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
16 # You should have received a copy of the GNU General Public License version
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
17 # 2 along with this work; if not, write to the Free Software Foundation,
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
18 # Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
19 #
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
20 # Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
21 # or visit www.oracle.com if you need additional information or have any
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
22 # questions.
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
23 #
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
24 # ----------------------------------------------------------------------------------------------------
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
25
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
26 from outputparser import OutputParser, ValuesMatcher
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
27 import re, mx, commands, os, sys, StringIO, subprocess
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
28 from os.path import isfile, join, exists
4150
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
29
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
30 dacapoSanityWarmup = {
4286
76190108a9f2 Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4284
diff changeset
31 'avrora': [0, 0, 3, 6, 13],
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
32 'batik': [0, 0, 5, 5, 20],
4286
76190108a9f2 Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4284
diff changeset
33 'eclipse': [2, 4, 5, 10, 16],
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
34 'fop': [4, 8, 10, 20, 30],
4286
76190108a9f2 Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4284
diff changeset
35 'h2': [0, 0, 5, 5, 8],
76190108a9f2 Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4284
diff changeset
36 'jython': [0, 0, 5, 10, 13],
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
37 'luindex': [0, 0, 5, 10, 10],
4286
76190108a9f2 Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4284
diff changeset
38 'lusearch': [0, 4, 5, 5, 8],
76190108a9f2 Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4284
diff changeset
39 'pmd': [0, 0, 5, 10, 13],
4482
7903b6c28f9c Add hsdis command to mx
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4480
diff changeset
40 'sunflow': [0, 2, 5, 10, 15],
4286
76190108a9f2 Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4284
diff changeset
41 'tomcat': [0, 0, 5, 10, 15],
76190108a9f2 Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4284
diff changeset
42 'tradebeans': [0, 0, 5, 10, 13],
76190108a9f2 Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4284
diff changeset
43 'tradesoap': [2, 4, 5, 10, 15],
76190108a9f2 Tweak some benchmark settings
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4284
diff changeset
44 'xalan': [0, 0, 5, 10, 18],
4150
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
45 }
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
46
4480
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
47 dacapoScalaSanityWarmup = {
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
48 'actors': [0, 0, 2, 8, 10],
6720
b62581e29024 disabled scaladacapo apparat benchmark
Lukas Stadler <lukas.stadler@jku.at>
parents: 6319
diff changeset
49 # (lstadler) apparat was disabled due to a deadlock which I think is the benchmarks fault.
b62581e29024 disabled scaladacapo apparat benchmark
Lukas Stadler <lukas.stadler@jku.at>
parents: 6319
diff changeset
50 # 'apparat': [0, 0, 1, 2, 3],
4480
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
51 'factorie': [0, 0, 2, 5, 5],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
52 'kiama': [0, 0, 3, 13, 15],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
53 'scalac': [0, 0, 5, 15, 20],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
54 'scaladoc': [0, 0, 5, 15, 15],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
55 'scalap': [0, 0, 5, 15, 20],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
56 'scalariform':[0, 0, 6, 15, 20],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
57 'scalatest': [0, 0, 2, 10, 12],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
58 'scalaxb': [0, 0, 5, 15, 25],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
59 'specs': [0, 0, 3, 13, 18],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
60 'tmt': [0, 0, 3, 10, 12]
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
61 }
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
62
4240
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
63 dacapoGateBuildLevels = {
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
64 'avrora': ['product', 'fastdebug', 'debug'],
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
65 'batik': ['product', 'fastdebug', 'debug'],
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
66 'eclipse': ['product'],
4284
1339f1817d28 gate : do not run tradesoap in fastdebug mode, run fop in fastdebug mode only
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4282
diff changeset
67 'fop': [ 'fastdebug', 'debug'],
4240
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
68 'h2': ['product', 'fastdebug', 'debug'],
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
69 'jython': ['product', 'fastdebug', 'debug'],
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
70 'luindex': ['product', 'fastdebug', 'debug'],
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
71 'lusearch': ['product'],
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
72 'pmd': ['product', 'fastdebug', 'debug'],
6319
0a0d0dfd83bc Gate : do not run tradesoap in fastdebug mode, do not run sunflow in product mode
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 6296
diff changeset
73 'sunflow': [ 'fastdebug', 'debug'],
4240
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
74 'tomcat': ['product', 'fastdebug', 'debug'],
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
75 'tradebeans': ['product', 'fastdebug', 'debug'],
6319
0a0d0dfd83bc Gate : do not run tradesoap in fastdebug mode, do not run sunflow in product mode
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 6296
diff changeset
76 'tradesoap': ['product'],
4240
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
77 'xalan': ['product', 'fastdebug', 'debug'],
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
78 }
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
79
4480
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
80 dacapoScalaGateBuildLevels = {
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
81 'actors': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
82 'apparat': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
83 'factorie': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
84 'kiama': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
85 'scalac': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
86 'scaladoc': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
87 'scalap': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
88 'scalariform':['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
89 'scalatest': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
90 'scalaxb': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
91 'specs': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
92 'tmt': ['product', 'fastdebug', 'debug'],
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
93 }
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
94
4150
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
95 class SanityCheckLevel:
4219
47f7d91d34cf Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4217
diff changeset
96 Fast, Gate, Normal, Extensive, Benchmark = range(5)
47f7d91d34cf Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4217
diff changeset
97
5877
0e54d9bb922d Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5196
diff changeset
98 def getSPECjbb2005(benchArgs = []):
0e54d9bb922d Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5196
diff changeset
99
0e54d9bb922d Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5196
diff changeset
100 specjbb2005 = mx.get_env('SPECJBB2005')
0e54d9bb922d Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5196
diff changeset
101 if specjbb2005 is None or not exists(join(specjbb2005, 'jbb.jar')):
0e54d9bb922d Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5196
diff changeset
102 mx.abort('Please set the SPECJBB2005 environment variable to a SPECjbb2005 directory')
0e54d9bb922d Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5196
diff changeset
103
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
104 score = re.compile(r"^Valid run, Score is (?P<score>[0-9]+)$", re.MULTILINE)
5877
0e54d9bb922d Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5196
diff changeset
105 error = re.compile(r"VALIDATION ERROR")
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
106 success = re.compile(r"^Valid run, Score is [0-9]+$", re.MULTILINE)
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
107 matcher = ValuesMatcher(score, {'group' : 'SPECjbb2005', 'name' : 'score', 'score' : '<score>'})
5877
0e54d9bb922d Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5196
diff changeset
108 classpath = ['jbb.jar', 'check.jar']
7367
cbcee2e1ce51 Make sure that compressed oops are not used in benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7345
diff changeset
109 return Test("SPECjbb2005", ['spec.jbb.JBBmain', '-propfile', 'SPECjbb.props'] + benchArgs, [success], [error], [matcher], vmOpts=['-Xms3g', '-XX:+UseSerialGC', '-XX:-UseCompressedOops', '-cp', os.pathsep.join(classpath)], defaultCwd=specjbb2005)
7565
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
110
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
111 def getSPECjbb2013(benchArgs = []):
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
112
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
113 specjbb2013 = mx.get_env('SPECJBB2013')
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
114 if specjbb2013 is None or not exists(join(specjbb2013, 'specjbb2013.jar')):
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
115 mx.abort('Please set the SPECJBB2013 environment variable to a SPECjbb2013 directory')
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
116
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
117 jops = re.compile(r"^RUN RESULT: hbIR \(max attempted\) = [0-9]+, hbIR \(settled\) = [0-9]+, max-jOPS = (?P<max>[0-9]+), critical-jOPS = (?P<critical>[0-9]+)$", re.MULTILINE)
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
118 #error?
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
119 success = re.compile(r"org.spec.jbb.controller: Run finished", re.MULTILINE)
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
120 matcherMax = ValuesMatcher(jops, {'group' : 'SPECjbb2013', 'name' : 'max', 'score' : '<max>'})
9a521597686b Add SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7564
diff changeset
121 matcherCritical = ValuesMatcher(jops, {'group' : 'SPECjbb2013', 'name' : 'critical', 'score' : '<critical>'})
8147
9786ac8fff61 Use different vm args for SPECjbb2013
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7656
diff changeset
122 return Test("SPECjbb2013", ['-jar', 'specjbb2013.jar', '-m', 'composite'] + benchArgs, [success], [], [matcherCritical, matcherMax], vmOpts=['-Xmx6g', '-Xms6g', '-Xmn3g', '-XX:+UseParallelOldGC', '-XX:-UseAdaptiveSizePolicy', '-XX:-UseBiasedLocking', '-XX:-UseCompressedOops'], defaultCwd=specjbb2013)
5877
0e54d9bb922d Add SPECjbb2005 to the benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5196
diff changeset
123
7345
dd903cdfe708 Add possibility for -ict option to SPECjvm2008 benchmark execution.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 6990
diff changeset
124 def getSPECjvm2008(benchArgs = [], skipCheck=False, skipKitValidation=False, warmupTime=None, iterationTime=None):
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
125
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
126 specjvm2008 = mx.get_env('SPECJVM2008')
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
127 if specjvm2008 is None or not exists(join(specjvm2008, 'SPECjvm2008.jar')):
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
128 mx.abort('Please set the SPECJVM2008 environment variable to a SPECjvm2008 directory')
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
129
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
130 score = re.compile(r"^(Score on|Noncompliant) (?P<benchmark>[a-zA-Z0-9\._]+)( result)?: (?P<score>[0-9]+((,|\.)[0-9]+)?)( SPECjvm2008 Base)? ops/m$", re.MULTILINE)
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
131 error = re.compile(r"^Errors in benchmark: ", re.MULTILINE)
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
132 # The ' ops/m' at the end of the success string is important : it's how you can tell valid and invalid runs apart
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
133 success = re.compile(r"^(Noncompliant c|C)omposite result: [0-9]+((,|\.)[0-9]+)?( SPECjvm2008 (Base|Peak))? ops/m$", re.MULTILINE)
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
134 matcher = ValuesMatcher(score, {'group' : 'SPECjvm2008', 'name' : '<benchmark>', 'score' : '<score>'})
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
135
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
136 opts = []
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
137 if warmupTime is not None:
4230
b780ecb920c9 Add SPECjvm2008 to benchmarks, use dacapos defined in sanitycheck for the gate
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4228
diff changeset
138 opts += ['-wt', str(warmupTime)]
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
139 if iterationTime is not None:
4230
b780ecb920c9 Add SPECjvm2008 to benchmarks, use dacapos defined in sanitycheck for the gate
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4228
diff changeset
140 opts += ['-it', str(iterationTime)]
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
141 if skipKitValidation:
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
142 opts += ['-ikv']
7345
dd903cdfe708 Add possibility for -ict option to SPECjvm2008 benchmark execution.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 6990
diff changeset
143 if skipCheck:
dd903cdfe708 Add possibility for -ict option to SPECjvm2008 benchmark execution.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 6990
diff changeset
144 opts += ['-ict']
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
145
7367
cbcee2e1ce51 Make sure that compressed oops are not used in benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7345
diff changeset
146 return Test("SPECjvm2008", ['-jar', 'SPECjvm2008.jar'] + opts + benchArgs, [success], [error], [matcher], vmOpts=['-Xms3g', '-XX:+UseSerialGC', '-XX:-UseCompressedOops'], defaultCwd=specjvm2008)
4150
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
147
4240
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
148 def getDacapos(level=SanityCheckLevel.Normal, gateBuildLevel=None, dacapoArgs=[]):
4217
f3271682fe5a Various small fix around benchmarks in gate scripts
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4215
diff changeset
149 checks = []
4150
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
150
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
151 for (bench, ns) in dacapoSanityWarmup.items():
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
152 if ns[level] > 0:
4240
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
153 if gateBuildLevel is None or gateBuildLevel in dacapoGateBuildLevels[bench]:
394424b7df1a Disabled certain DaCapo tests for fastdebug testing in the gate.
Doug Simon <doug.simon@oracle.com>
parents: 4238
diff changeset
154 checks.append(getDacapo(bench, ns[level], dacapoArgs))
4150
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
155
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
156 return checks
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
157
4217
f3271682fe5a Various small fix around benchmarks in gate scripts
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4215
diff changeset
158 def getDacapo(name, n, dacapoArgs=[]):
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
159 dacapo = mx.get_env('DACAPO_CP')
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
160 if dacapo is None:
4244
a69889e5a8a5 Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents: 4240
diff changeset
161 l = mx.library('DACAPO', False)
a69889e5a8a5 Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents: 4240
diff changeset
162 if l is not None:
a69889e5a8a5 Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents: 4240
diff changeset
163 dacapo = l.get_path(True)
a69889e5a8a5 Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents: 4240
diff changeset
164 else:
a69889e5a8a5 Made the DaCapo library be downloaded only on demand.
Doug Simon <doug.simon@oracle.com>
parents: 4240
diff changeset
165 mx.abort('DaCapo 9.12 jar file must be specified with DACAPO_CP environment variable or as DACAPO library')
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
166
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
167 if not isfile(dacapo) or not dacapo.endswith('.jar'):
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
168 mx.abort('Specified DaCapo jar file does not exist or is not a jar file: ' + dacapo)
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
169
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
170 dacapoSuccess = re.compile(r"^===== DaCapo 9\.12 ([a-zA-Z0-9_]+) PASSED in ([0-9]+) msec =====$", re.MULTILINE)
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
171 dacapoFail = re.compile(r"^===== DaCapo 9\.12 ([a-zA-Z0-9_]+) FAILED (warmup|) =====$", re.MULTILINE)
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
172 dacapoTime = re.compile(r"===== DaCapo 9\.12 (?P<benchmark>[a-zA-Z0-9_]+) PASSED in (?P<time>[0-9]+) msec =====")
5195
8125c1f6472b Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5148
diff changeset
173 dacapoTime1 = re.compile(r"===== DaCapo 9\.12 (?P<benchmark>[a-zA-Z0-9_]+) completed warmup 1 in (?P<time>[0-9]+) msec =====")
4150
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
174
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
175 dacapoMatcher = ValuesMatcher(dacapoTime, {'group' : 'DaCapo', 'name' : '<benchmark>', 'score' : '<time>'})
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
176 dacapoMatcher1 = ValuesMatcher(dacapoTime1, {'group' : 'DaCapo-1stRun', 'name' : '<benchmark>', 'score' : '<time>'})
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
177
7367
cbcee2e1ce51 Make sure that compressed oops are not used in benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7345
diff changeset
178 return Test("DaCapo-" + name, ['-jar', dacapo, name, '-n', str(n), ] + dacapoArgs, [dacapoSuccess], [dacapoFail], [dacapoMatcher, dacapoMatcher1], ['-Xms2g', '-XX:+UseSerialGC', '-XX:-UseCompressedOops'])
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
179
4480
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
180 def getScalaDacapos(level=SanityCheckLevel.Normal, gateBuildLevel=None, dacapoArgs=[]):
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
181 checks = []
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
182
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
183 for (bench, ns) in dacapoScalaSanityWarmup.items():
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
184 if ns[level] > 0:
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
185 if gateBuildLevel is None or gateBuildLevel in dacapoScalaGateBuildLevels[bench]:
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
186 checks.append(getScalaDacapo(bench, ns[level], dacapoArgs))
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
187
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
188 return checks
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
189
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
190 def getScalaDacapo(name, n, dacapoArgs=[]):
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
191 dacapo = mx.get_env('DACAPO_SCALA_CP')
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
192 if dacapo is None:
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
193 l = mx.library('DACAPO_SCALA', False)
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
194 if l is not None:
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
195 dacapo = l.get_path(True)
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
196 else:
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
197 mx.abort('Scala DaCapo 0.1.0 jar file must be specified with DACAPO_SCALA_CP environment variable or as DACAPO_SCALA library')
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
198
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
199 if not isfile(dacapo) or not dacapo.endswith('.jar'):
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
200 mx.abort('Specified Scala DaCapo jar file does not exist or is not a jar file: ' + dacapo)
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
201
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
202 dacapoSuccess = re.compile(r"^===== DaCapo 0\.1\.0(-SNAPSHOT)? ([a-zA-Z0-9_]+) PASSED in ([0-9]+) msec =====$", re.MULTILINE)
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
203 dacapoFail = re.compile(r"^===== DaCapo 0\.1\.0(-SNAPSHOT)? ([a-zA-Z0-9_]+) FAILED (warmup|) =====$", re.MULTILINE)
4480
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
204 dacapoTime = re.compile(r"===== DaCapo 0\.1\.0(-SNAPSHOT)? (?P<benchmark>[a-zA-Z0-9_]+) PASSED in (?P<time>[0-9]+) msec =====")
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
205
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
206 dacapoMatcher = ValuesMatcher(dacapoTime, {'group' : "Scala-DaCapo", 'name' : '<benchmark>', 'score' : '<time>'})
4480
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
207
7367
cbcee2e1ce51 Make sure that compressed oops are not used in benchmarks
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 7345
diff changeset
208 return Test("Scala-DaCapo-" + name, ['-jar', dacapo, name, '-n', str(n), ] + dacapoArgs, [dacapoSuccess], [dacapoFail], [dacapoMatcher], ['-Xms2g', '-XX:+UseSerialGC', '-XX:-UseCompressedOops'])
4480
7d6490436b57 Add Scala DaCapo benchs
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4416
diff changeset
209
4219
47f7d91d34cf Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4217
diff changeset
210 def getBootstraps():
4282
063ea022532c mx commands : improve bench command, fix in the outparser so that no empty 'line' is retruned if there was no match, fix bootstrap's regex
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4244
diff changeset
211 time = re.compile(r"Bootstrapping Graal\.+ in (?P<time>[0-9]+) ms")
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
212 scoreMatcher = ValuesMatcher(time, {'group' : 'Bootstrap', 'name' : 'BootstrapTime', 'score' : '<time>'})
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
213 scoreMatcherBig = ValuesMatcher(time, {'group' : 'Bootstrap-bigHeap', 'name' : 'BootstrapTime', 'score' : '<time>'})
5195
8125c1f6472b Add DaCapo 1st run as a benchmark group
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 5148
diff changeset
214
4219
47f7d91d34cf Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4217
diff changeset
215 tests = []
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
216 tests.append(Test("Bootstrap", ['-version'], successREs=[time], scoreMatchers=[scoreMatcher], ignoredVMs=['client', 'server'], benchmarkCompilationRate=False))
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
217 tests.append(Test("Bootstrap-bigHeap", ['-version'], successREs=[time], scoreMatchers=[scoreMatcherBig], vmOpts=['-Xms2g'], ignoredVMs=['client', 'server'], benchmarkCompilationRate=False))
4219
47f7d91d34cf Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4217
diff changeset
218 return tests
47f7d91d34cf Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4217
diff changeset
219
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
220 class Tee:
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
221 def __init__(self):
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
222 self.output = StringIO.StringIO()
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
223 def eat(self, line):
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
224 self.output.write(line)
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
225 sys.stdout.write(line)
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
226
4225
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
227 """
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
228 Encapsulates a single program that is a sanity test and/or a benchmark.
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
229 """
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
230 class Test:
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
231 def __init__(self, name, cmd, successREs=[], failureREs=[], scoreMatchers=[], vmOpts=[], defaultCwd=None, ignoredVMs=[], benchmarkCompilationRate=True):
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
232
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
233 self.name = name
4225
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
234 self.successREs = successREs
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
235 self.failureREs = failureREs + [re.compile(r"Exception occurred in scope: ")]
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
236 self.scoreMatchers = scoreMatchers
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
237 self.vmOpts = vmOpts
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
238 self.cmd = cmd
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
239 self.defaultCwd = defaultCwd
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
240 self.ignoredVMs = ignoredVMs
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
241 self.benchmarkCompilationRate = benchmarkCompilationRate
7654
641a4c6ac1ce fixed repetition of compilation metric option on command line when running multiple benchmarks (JIRA issue GRAAL-66)
Doug Simon <doug.simon@oracle.com>
parents: 7571
diff changeset
242 if benchmarkCompilationRate:
641a4c6ac1ce fixed repetition of compilation metric option on command line when running multiple benchmarks (JIRA issue GRAAL-66)
Doug Simon <doug.simon@oracle.com>
parents: 7571
diff changeset
243 self.vmOpts = self.vmOpts + ['-XX:+CITime']
4238
e0d09e05aa9b Made the gate time each DaCapo test individually.
Doug Simon <doug.simon@oracle.com>
parents: 4230
diff changeset
244
e0d09e05aa9b Made the gate time each DaCapo test individually.
Doug Simon <doug.simon@oracle.com>
parents: 4230
diff changeset
245 def __str__(self):
e0d09e05aa9b Made the gate time each DaCapo test individually.
Doug Simon <doug.simon@oracle.com>
parents: 4230
diff changeset
246 return self.name
4150
c78bace5086a start work on integrating old hooks into mx, work on sanity checks
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents:
diff changeset
247
4219
47f7d91d34cf Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4217
diff changeset
248 def test(self, vm, cwd=None, opts=[], vmbuild=None):
4225
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
249 """
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
250 Run this program as a sanity test.
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
251 """
7562
1c09bcebd61f spelling fix
Doug Simon <doug.simon@oracle.com>
parents: 7367
diff changeset
252 if (vm in self.ignoredVMs):
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
253 return True
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
254 if cwd is None:
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
255 cwd = self.defaultCwd
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
256 parser = OutputParser()
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
257 jvmError = re.compile(r"(?P<jvmerror>([A-Z]:|/).*[/\\]hs_err_pid[0-9]+\.log)")
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
258 parser.addMatcher(ValuesMatcher(jvmError, {'jvmError' : '<jvmerror>'}))
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
259
4225
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
260 for successRE in self.successREs:
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
261 parser.addMatcher(ValuesMatcher(successRE, {'passed' : '1'}))
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
262 for failureRE in self.failureREs:
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
263 parser.addMatcher(ValuesMatcher(failureRE, {'failed' : '1'}))
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
264
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
265 tee = Tee()
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
266 retcode = commands.vm(self.vmOpts + opts + self.cmd, vm, nonZeroIsFatal=False, out=tee.eat, err=subprocess.STDOUT, cwd=cwd, vmbuild=vmbuild)
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
267 output = tee.output.getvalue()
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
268 valueMaps = parser.parse(output)
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
269
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
270 if len(valueMaps) == 0:
4332
9dbcdd6f1464 If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4331
diff changeset
271 return False
9dbcdd6f1464 If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4331
diff changeset
272
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
273 assert len(valueMaps) == 1, 'Test matchers should not return more than one record'
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
274
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
275 record = valueMaps[0]
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
276
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
277 jvmErrorFile = record.get('jvmError')
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
278 if jvmErrorFile:
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
279 mx.log('/!\\JVM Error : dumping error log...')
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
280 with open(jvmErrorFile, 'rb') as fp:
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
281 mx.log(fp.read())
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
282 os.unlink(jvmErrorFile)
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
283 return False
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
284
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
285 if record.get('failed') == '1':
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
286 return False
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
287
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
288 return retcode == 0 and record.get('passed') == '1'
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
289
4219
47f7d91d34cf Fix javac build issues with classpath
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4217
diff changeset
290 def bench(self, vm, cwd=None, opts=[], vmbuild=None):
4225
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
291 """
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
292 Run this program as a benchmark.
339cf8d4904d Made mx.run work properly when stderr is redirected to stdout.
Doug Simon <doug.simon@oracle.com>
parents: 4219
diff changeset
293 """
7562
1c09bcebd61f spelling fix
Doug Simon <doug.simon@oracle.com>
parents: 7367
diff changeset
294 if (vm in self.ignoredVMs):
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
295 return {}
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
296 if cwd is None:
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
297 cwd = self.defaultCwd
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
298 parser = OutputParser()
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
299
4332
9dbcdd6f1464 If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4331
diff changeset
300 for successRE in self.successREs:
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
301 parser.addMatcher(ValuesMatcher(successRE, {'passed' : '1'}))
4228
e872562f95f8 add a simple specjvm2008 command, fail a benchmark if one of the failureRE matches
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4225
diff changeset
302 for failureRE in self.failureREs:
7564
c420a487b10f changed convention for specifying constants versus named groups in a template for a value to extract from the output of a benchmark execution
Doug Simon <doug.simon@oracle.com>
parents: 7563
diff changeset
303 parser.addMatcher(ValuesMatcher(failureRE, {'failed' : '1'}))
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
304 for scoreMatcher in self.scoreMatchers:
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
305 parser.addMatcher(scoreMatcher)
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
306
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
307 if self.benchmarkCompilationRate:
7570
2025455e7d80 added collection of compilation speed metric for C1 and C2 to bench command
Doug Simon <doug.simon@oracle.com>
parents: 7569
diff changeset
308 if vm == 'graal':
2025455e7d80 added collection of compilation speed metric for C1 and C2 to bench command
Doug Simon <doug.simon@oracle.com>
parents: 7569
diff changeset
309 bps = re.compile(r"ParsedBytecodesPerSecond@final: (?P<rate>[0-9]+)")
2025455e7d80 added collection of compilation speed metric for C1 and C2 to bench command
Doug Simon <doug.simon@oracle.com>
parents: 7569
diff changeset
310 ibps = re.compile(r"InlinedBytecodesPerSecond@final: (?P<rate>[0-9]+)")
2025455e7d80 added collection of compilation speed metric for C1 and C2 to bench command
Doug Simon <doug.simon@oracle.com>
parents: 7569
diff changeset
311 parser.addMatcher(ValuesMatcher(bps, {'group' : 'ParsedBytecodesPerSecond', 'name' : self.name, 'score' : '<rate>'}))
2025455e7d80 added collection of compilation speed metric for C1 and C2 to bench command
Doug Simon <doug.simon@oracle.com>
parents: 7569
diff changeset
312 parser.addMatcher(ValuesMatcher(ibps, {'group' : 'InlinedBytecodesPerSecond', 'name' : self.name, 'score' : '<rate>'}))
2025455e7d80 added collection of compilation speed metric for C1 and C2 to bench command
Doug Simon <doug.simon@oracle.com>
parents: 7569
diff changeset
313 else:
2025455e7d80 added collection of compilation speed metric for C1 and C2 to bench command
Doug Simon <doug.simon@oracle.com>
parents: 7569
diff changeset
314 ibps = re.compile(r"(?P<compiler>[\w]+) compilation speed: +(?P<rate>[0-9]+) bytes/s {standard")
2025455e7d80 added collection of compilation speed metric for C1 and C2 to bench command
Doug Simon <doug.simon@oracle.com>
parents: 7569
diff changeset
315 parser.addMatcher(ValuesMatcher(ibps, {'group' : 'InlinedBytecodesPerSecond', 'name' : '<compiler>:' + self.name, 'score' : '<rate>'}))
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
316
7567
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
317 startDelim = 'START: ' + self.name
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
318 endDelim = 'END: ' + self.name
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
319
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
320 outputfile = os.environ.get('BENCH_OUTPUT', None)
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
321 if outputfile:
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
322 # Used only to debug output parsing
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
323 with open(outputfile) as fp:
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
324 output = fp.read()
7567
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
325 start = output.find(startDelim)
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
326 end = output.find(endDelim, start)
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
327 if start == -1 and end == -1:
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
328 return {}
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
329 output = output[start + len(startDelim + os.linesep): end]
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
330 mx.log(startDelim)
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
331 mx.log(output)
7567
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
332 mx.log(endDelim)
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
333 else:
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
334 tee = Tee()
7567
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
335 mx.log(startDelim)
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
336 if commands.vm(self.vmOpts + opts + self.cmd, vm, nonZeroIsFatal=False, out=tee.eat, err=subprocess.STDOUT, cwd=cwd, vmbuild=vmbuild) != 0:
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
337 mx.abort("Benchmark failed (non-zero retcode)")
7567
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
338 mx.log(endDelim)
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
339 output = tee.output.getvalue()
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
340
7567
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
341 groups = {}
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
342 passed = False
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
343 for valueMap in parser.parse(output):
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
344 assert (valueMap.has_key('name') and valueMap.has_key('score') and valueMap.has_key('group')) or valueMap.has_key('passed') or valueMap.has_key('failed'), valueMap
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
345 if valueMap.get('failed') == '1':
4332
9dbcdd6f1464 If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4331
diff changeset
346 mx.abort("Benchmark failed")
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
347 if valueMap.get('passed') == '1':
4332
9dbcdd6f1464 If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4331
diff changeset
348 passed = True
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
349 groupName = valueMap.get('group')
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
350 if groupName:
7567
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
351 group = groups.setdefault(groupName, {})
7563
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
352 name = valueMap.get('name')
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
353 score = valueMap.get('score')
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
354 if name and score:
3aab15f42934 moved execution of a benchmark out of OutputParser
Doug Simon <doug.simon@oracle.com>
parents: 7562
diff changeset
355 group[name] = score
4215
a2caa019ba3a Fix mx : commands' scripts mx_init hook should be called before parsing command line arguments.
Gilles Duboscq <gilles.m.duboscq@gmail.com>
parents: 4150
diff changeset
356
4332
9dbcdd6f1464 If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4331
diff changeset
357 if not passed:
9dbcdd6f1464 If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4331
diff changeset
358 mx.abort("Benchmark failed (not passed)")
9dbcdd6f1464 If a benchmark fails it should fail hard, not silently
Gilles Duboscq <duboscq@ssw.jku.at>
parents: 4331
diff changeset
359
7567
a8bc60aeacb8 fix bug in parsing of SPECjvm2008 output
Doug Simon <doug.simon@oracle.com>
parents: 7564
diff changeset
360 return groups