Mercurial > hg > truffle

annotate graal/com.oracle.graal.hotspot/src/com/oracle/graal/hotspot/replacements/WriteBarrierSnippets.java @ 11235:cc5473cf32bb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fix snippet counter naming
author Christos Kotselidis <christos.kotselidis@oracle.com>
date Wed, 07 Aug 2013 14:07:06 +0200
parents 73eb4d8a4015
children ef6915cf1e59
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
1 /*
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
2 * Copyright (c) 2012, 2013, Oracle and/or its affiliates. All rights reserved.
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
3 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
4 *
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
5 * This code is free software; you can redistribute it and/or modify it
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
6 * under the terms of the GNU General Public License version 2 only, as
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
7 * published by the Free Software Foundation.
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
8 *
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
9 * This code is distributed in the hope that it will be useful, but WITHOUT
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
10 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
11 * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
12 * version 2 for more details (a copy is included in the LICENSE file that
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
13 * accompanied this code).
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
14 *
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
15 * You should have received a copy of the GNU General Public License version
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
16 * 2 along with this work; if not, write to the Free Software Foundation,
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
17 * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
18 *
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
19 * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
20 * or visit www.oracle.com if you need additional information or have any
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
21 * questions.
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
22 */
8508
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
23 package com.oracle.graal.hotspot.replacements;
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
24
9513
659bb6cf930c rename: HotSpotSnippetUtils -> HotSpotReplacementsUtil
Doug Simon <doug.simon@oracle.com>
parents: 9015
diff changeset
25 import static com.oracle.graal.hotspot.replacements.HotSpotReplacementsUtil.*;
10787
388fbd0dd4a4 moved BranchNodeProbabilities to graal.nodes project
Doug Simon <doug.simon@oracle.com>
parents: 10701
diff changeset
26 import static com.oracle.graal.nodes.extended.BranchProbabilityNode.*;
10481
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
27 import static com.oracle.graal.phases.GraalOptions.*;
8508
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
28 import static com.oracle.graal.replacements.SnippetTemplate.*;
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
29
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
30 import com.oracle.graal.api.code.*;
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
31 import com.oracle.graal.api.meta.*;
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
32 import com.oracle.graal.graph.Node.ConstantNodeParameter;
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
33 import com.oracle.graal.graph.Node.NodeIntrinsic;
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
34 import com.oracle.graal.hotspot.nodes.*;
8905
c502db57e687 Rename ArrayWriteBarrier to SerialWriteBarrier and move it to graal.nodes
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8904
diff changeset
35 import com.oracle.graal.nodes.*;
10828
b05e1ff3aac0 Rename WriteBarrierType to BarrierType
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10817
diff changeset
36 import com.oracle.graal.nodes.HeapAccess.BarrierType;
10517
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
37 import com.oracle.graal.nodes.extended.*;
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
38 import com.oracle.graal.nodes.spi.*;
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
39 import com.oracle.graal.phases.*;
8508
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
40 import com.oracle.graal.replacements.*;
8903
876a449f17ab Merge FieldWriteBarrier into ArrayWriteBarrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8895
diff changeset
41 import com.oracle.graal.replacements.Snippet.ConstantParameter;
8508
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
42 import com.oracle.graal.replacements.SnippetTemplate.AbstractTemplates;
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
43 import com.oracle.graal.replacements.SnippetTemplate.Arguments;
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
44 import com.oracle.graal.replacements.SnippetTemplate.SnippetInfo;
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
45 import com.oracle.graal.replacements.nodes.*;
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
46 import com.oracle.graal.word.*;
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
47
8508
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
48 public class WriteBarrierSnippets implements Snippets {
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
49
10481
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
50 private static final SnippetCounter.Group countersWriteBarriers = SnippetCounters.getValue() ? new SnippetCounter.Group("WriteBarriers") : null;
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
51 private static final SnippetCounter serialFieldWriteBarrierCounter = new SnippetCounter(countersWriteBarriers, "serialFieldWriteBarrier", "Number of Serial Field Write Barriers");
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
52 private static final SnippetCounter serialArrayWriteBarrierCounter = new SnippetCounter(countersWriteBarriers, "serialArrayWriteBarrier", "Number of Serial Array Write Barriers");
10955
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
53 private static final SnippetCounter g1AttemptedPostWriteBarrierCounter = new SnippetCounter(countersWriteBarriers, "g1AttemptedPostWriteBarrierCounter",
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
54 "Number of attempted G1 Post Write Barriers");
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
55 private static final SnippetCounter g1AttemptedPreWriteBarrierCounter = new SnippetCounter(countersWriteBarriers, "g1AttemptedPreWriteBarrierCounter", "Number of G1 attempted Pre Write Barriers");
11235
cc5473cf32bb Fix snippet counter naming
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 11234
diff changeset
56 private static final SnippetCounter g1AttemptedRefFieldBarrierCounter = new SnippetCounter(countersWriteBarriers, "g1AttemptedRefFieldBarrierCounter",
10955
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
57 "Number of G1 attempted Ref Field Read Barriers");
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
58 private static final SnippetCounter g1EffectivePostWriteBarrierCounter = new SnippetCounter(countersWriteBarriers, "g1EffectivePostWriteBarrierCounter",
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
59 "Number of effective G1 Post Write Barriers");
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
60 private static final SnippetCounter g1EffectivePreWriteBarrierCounter = new SnippetCounter(countersWriteBarriers, "g1EffectivePreWriteBarrierCounter", "Number of G1 effective Pre Write Barriers");
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
61 private static final SnippetCounter g1EffectiveRefFieldBarrierCounter = new SnippetCounter(countersWriteBarriers, "g1EffectiveRefFieldBarrierCounter",
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
62 "Number of G1 effective Ref Field Read Barriers");
10481
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
63
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
64 @Snippet
10945
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
65 public static void serialWriteBarrier(Object object, Object location, @ConstantParameter boolean usePrecise, @ConstantParameter boolean alwaysNull) {
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
66 // No barriers are added if we are always storing a null.
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
67 if (alwaysNull) {
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
68 return;
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
69 }
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
70 Object fixedObject = FixedValueAnchorNode.getObject(object);
8903
876a449f17ab Merge FieldWriteBarrier into ArrayWriteBarrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8895
diff changeset
71 Pointer oop;
876a449f17ab Merge FieldWriteBarrier into ArrayWriteBarrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8895
diff changeset
72 if (usePrecise) {
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
73 oop = Word.fromArray(fixedObject, location);
10481
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
74 serialArrayWriteBarrierCounter.inc();
8903
876a449f17ab Merge FieldWriteBarrier into ArrayWriteBarrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8895
diff changeset
75 } else {
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
76 oop = Word.fromObject(fixedObject);
10481
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
77 serialFieldWriteBarrierCounter.inc();
8903
876a449f17ab Merge FieldWriteBarrier into ArrayWriteBarrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8895
diff changeset
78 }
8518
4a79f7c0d1a4 -Fix assertion errors
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8517
diff changeset
79 Word base = (Word) oop.unsignedShiftRight(cardTableShift());
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
80 long startAddress = cardTableStart();
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
81 int displacement = 0;
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
82 if (((int) startAddress) == startAddress) {
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
83 displacement = (int) startAddress;
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
84 } else {
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
85 base = base.add(Word.unsigned(cardTableStart()));
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
86 }
8520
450abf358987 -Bug fix in card marking
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8518
diff changeset
87 base.writeByte(displacement, (byte) 0);
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
88 }
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
89
8963
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
90 @Snippet
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
91 public static void serialArrayRangeWriteBarrier(Object object, int startIndex, int length) {
10942
56a86b60c9d3 Early exit of SerialArrayRange barrier if vector length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10941
diff changeset
92 if (length == 0) {
56a86b60c9d3 Early exit of SerialArrayRange barrier if vector length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10941
diff changeset
93 return;
56a86b60c9d3 Early exit of SerialArrayRange barrier if vector length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10941
diff changeset
94 }
8989
40e7444d62ee Variable renaming
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8985
diff changeset
95 Object dest = FixedValueAnchorNode.getObject(object);
8963
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
96 int cardShift = cardTableShift();
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
97 long cardStart = cardTableStart();
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
98 final int scale = arrayIndexScale(Kind.Object);
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
99 int header = arrayBaseOffset(Kind.Object);
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
100 long dstAddr = GetObjectAddressNode.get(dest);
8989
40e7444d62ee Variable renaming
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8985
diff changeset
101 long start = (dstAddr + header + (long) startIndex * scale) >>> cardShift;
40e7444d62ee Variable renaming
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8985
diff changeset
102 long end = (dstAddr + header + ((long) startIndex + length - 1) * scale) >>> cardShift;
8963
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
103 long count = end - start + 1;
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
104 while (count-- > 0) {
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
105 DirectStoreNode.store((start + cardStart) + count, false, Kind.Boolean);
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
106 }
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
107 }
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
108
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
109 @Snippet
10623
13c6440ac750 Add nullcheck at G1 Pre Barrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10622
diff changeset
110 public static void g1PreWriteBarrier(Object object, Object expectedObject, Object location, @ConstantParameter boolean doLoad, @ConstantParameter boolean nullCheck,
13c6440ac750 Add nullcheck at G1 Pre Barrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10622
diff changeset
111 @ConstantParameter boolean trace) {
13c6440ac750 Add nullcheck at G1 Pre Barrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10622
diff changeset
112 if (nullCheck && object == null) {
13c6440ac750 Add nullcheck at G1 Pre Barrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10622
diff changeset
113 DeoptimizeNode.deopt(DeoptimizationAction.InvalidateReprofile, DeoptimizationReason.NullCheckException);
13c6440ac750 Add nullcheck at G1 Pre Barrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10622
diff changeset
114 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
115 Word thread = thread();
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
116 Object fixedObject = FixedValueAnchorNode.getObject(object);
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
117 verifyOop(fixedObject);
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
118 Object fixedExpectedObject = FixedValueAnchorNode.getObject(expectedObject);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
119 Word field = (Word) Word.fromArray(fixedObject, location);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
120 Word previousOop = (Word) Word.fromObject(fixedExpectedObject);
10520
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
121 byte markingValue = thread.readByte(g1SATBQueueMarkingOffset());
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
122 Word bufferAddress = thread.readWord(g1SATBQueueBufferOffset());
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
123 Word indexAddress = thread.add(g1SATBQueueIndexOffset());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
124 Word indexValue = indexAddress.readWord(0);
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
125 int gcCycle = 0;
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
126 if (trace) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
127 gcCycle = (int) Word.unsigned(HotSpotReplacementsUtil.gcTotalCollectionsAddress()).readLong(0);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
128 log(trace, "[%d] G1-Pre Thread %p Object %p\n", gcCycle, thread.rawValue(), Word.fromObject(fixedObject).rawValue());
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
129 log(trace, "[%d] G1-Pre Thread %p Expected Object %p\n", gcCycle, thread.rawValue(), Word.fromObject(fixedExpectedObject).rawValue());
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
130 log(trace, "[%d] G1-Pre Thread %p Field %p\n", gcCycle, thread.rawValue(), field.rawValue());
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
131 log(trace, "[%d] G1-Pre Thread %p Marking %d\n", gcCycle, thread.rawValue(), markingValue);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
132 log(trace, "[%d] G1-Pre Thread %p DoLoad %d\n", gcCycle, thread.rawValue(), doLoad ? 1L : 0L);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
133 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
134 // If the concurrent marker is enabled, the barrier is issued.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
135 if (probability(NOT_LIKELY_PROBABILITY, markingValue != (byte) 0)) {
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
136 // If the previous value has to be loaded (before the write), the load is issued.
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
137 // The load is always issued except the cases of CAS and referent field.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
138 if (probability(LIKELY_PROBABILITY, doLoad)) {
10828
b05e1ff3aac0 Rename WriteBarrierType to BarrierType
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10817
diff changeset
139 previousOop = (Word) Word.fromObject(field.readObject(0, BarrierType.NONE, true));
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
140 if (trace) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
141 log(trace, "[%d] G1-Pre Thread %p Previous Object %p\n ", gcCycle, thread.rawValue(), previousOop.rawValue());
10631
2e82291febf4 Oop verification after printing oop
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10630
diff changeset
142 verifyOop(previousOop.toObject());
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
143 }
10955
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
144 g1AttemptedPreWriteBarrierCounter.inc();
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
145 } else {
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
146 g1AttemptedRefFieldBarrierCounter.inc();
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
147 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
148 // If the previous value is null the barrier should not be issued.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
149 if (probability(FREQUENT_PROBABILITY, previousOop.notEqual(0))) {
10955
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
150 if (doLoad) {
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
151 g1EffectivePreWriteBarrierCounter.inc();
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
152 } else {
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
153 g1EffectiveRefFieldBarrierCounter.inc();
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
154 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
155 // If the thread-local SATB buffer is full issue a native call which will
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
156 // initialize a new one and add the entry.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
157 if (probability(FREQUENT_PROBABILITY, indexValue.notEqual(0))) {
10520
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
158 Word nextIndex = indexValue.subtract(wordSize());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
159 Word logAddress = bufferAddress.add(nextIndex);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
160 // Log the object to be marked as well as update the SATB's buffer next index.
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
161 logAddress.writeWord(0, previousOop);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
162 indexAddress.writeWord(0, nextIndex);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
163 } else {
10517
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
164 g1PreBarrierStub(G1WBPRECALL, previousOop.toObject());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
165 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
166 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
167 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
168 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
169
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
170 @Snippet
10945
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
171 public static void g1PostWriteBarrier(Object object, Object value, Object location, @ConstantParameter boolean usePrecise, @ConstantParameter boolean alwaysNull, @ConstantParameter boolean trace) {
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
172 // No barriers are added if we are always storing a null.
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
173 if (alwaysNull) {
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
174 return;
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
175 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
176 Word thread = thread();
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
177 Object fixedObject = FixedValueAnchorNode.getObject(object);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
178 Object fixedValue = FixedValueAnchorNode.getObject(value);
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
179 verifyOop(fixedObject);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
180 verifyOop(fixedValue);
10953
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
181 validateObject(fixedObject, fixedValue);
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
182 Word oop;
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
183 if (usePrecise) {
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
184 oop = (Word) Word.fromArray(fixedObject, location);
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
185 } else {
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
186 oop = (Word) Word.fromObject(fixedObject);
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
187 }
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
188 int gcCycle = 0;
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
189 if (trace) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
190 gcCycle = (int) Word.unsigned(HotSpotReplacementsUtil.gcTotalCollectionsAddress()).readLong(0);
10630
ca3865947682 Debug log cleanup
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10623
diff changeset
191 log(trace, "[%d] G1-Post Thread: %p Object: %p\n", gcCycle, thread.rawValue(), Word.fromObject(fixedObject).rawValue());
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
192 log(trace, "[%d] G1-Post Thread: %p Field: %p\n", gcCycle, thread.rawValue(), oop.rawValue());
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
193 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
194 Word writtenValue = (Word) Word.fromObject(fixedValue);
10520
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
195 Word bufferAddress = thread.readWord(g1CardQueueBufferOffset());
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
196 Word indexAddress = thread.add(g1CardQueueIndexOffset());
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
197 Word indexValue = thread.readWord(g1CardQueueIndexOffset());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
198 // The result of the xor reveals whether the installed pointer crosses heap regions.
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
199 // In case it does the write barrier has to be issued.
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
200 Word xorResult = (oop.xor(writtenValue)).unsignedShiftRight(logOfHeapRegionGrainBytes());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
201
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
202 // Calculate the address of the card to be enqueued to the
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
203 // thread local card queue.
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
204 Word cardBase = oop.unsignedShiftRight(cardTableShift());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
205 long startAddress = cardTableStart();
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
206 int displacement = 0;
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
207 if (((int) startAddress) == startAddress) {
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
208 displacement = (int) startAddress;
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
209 } else {
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
210 cardBase = cardBase.add(Word.unsigned(cardTableStart()));
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
211 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
212 Word cardAddress = cardBase.add(displacement);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
213
10955
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
214 g1AttemptedPostWriteBarrierCounter.inc();
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
215 if (probability(LIKELY_PROBABILITY, xorResult.notEqual(0))) {
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
216 // If the written value is not null continue with the barrier addition.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
217 if (probability(FREQUENT_PROBABILITY, writtenValue.notEqual(0))) {
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
218 byte cardByte = cardAddress.readByte(0);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
219 // If the card is already dirty, (hence already enqueued) skip the insertion.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
220 if (probability(LIKELY_PROBABILITY, cardByte != (byte) 0)) {
10955
769ad79b8e0c Add G1 Barrier Counters
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10954
diff changeset
221 g1EffectivePostWriteBarrierCounter.inc();
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
222 log(trace, "[%d] G1-Post Thread: %p Card: %p \n", gcCycle, thread.rawValue(), Word.unsigned(cardByte).rawValue());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
223 cardAddress.writeByte(0, (byte) 0);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
224 // If the thread local card queue is full, issue a native call which will
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
225 // initialize a new one and add the card entry.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
226 if (probability(FREQUENT_PROBABILITY, indexValue.notEqual(0))) {
10520
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
227 Word nextIndex = indexValue.subtract(wordSize());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
228 Word logAddress = bufferAddress.add(nextIndex);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
229 // Log the object to be scanned as well as update
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
230 // the card queue's next index.
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
231 logAddress.writeWord(0, cardAddress);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
232 indexAddress.writeWord(0, nextIndex);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
233 } else {
10517
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
234 g1PostBarrierStub(G1WBPOSTCALL, cardAddress);
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
235 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
236 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
237 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
238 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
239 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
240
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
241 @Snippet
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
242 public static void g1ArrayRangePreWriteBarrier(Object object, int startIndex, int length) {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
243 Word thread = thread();
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
244 byte markingValue = thread.readByte(g1SATBQueueMarkingOffset());
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
245 // If the concurrent marker is not enabled or the vector length is zero, return.
10951
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
246 if (markingValue == (byte) 0 || length == 0) {
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
247 return;
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
248 }
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
249 Object dest = FixedValueAnchorNode.getObject(object);
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
250 Word bufferAddress = thread.readWord(g1SATBQueueBufferOffset());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
251 Word indexAddress = thread.add(g1SATBQueueIndexOffset());
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
252 long dstAddr = GetObjectAddressNode.get(dest);
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
253 long indexValue = indexAddress.readWord(0).rawValue();
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
254 final int scale = arrayIndexScale(Kind.Object);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
255 int header = arrayBaseOffset(Kind.Object);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
256
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
257 for (int i = startIndex; i < length; i++) {
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
258 long address = dstAddr + header + (i * scale);
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
259 Pointer oop = Word.fromObject(Word.unsigned(address).readObject(0, BarrierType.NONE, true));
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
260 verifyOop(oop.toObject());
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
261 if (oop.notEqual(0)) {
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
262 if (indexValue != 0) {
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
263 indexValue = indexValue - wordSize();
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
264 Word logAddress = bufferAddress.add(Word.unsigned(indexValue));
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
265 // Log the object to be marked as well as update the SATB's buffer next index.
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
266 logAddress.writeWord(0, oop);
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
267 indexAddress.writeWord(0, Word.unsigned(indexValue));
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
268 } else {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
269 g1PreBarrierStub(G1WBPRECALL, oop.toObject());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
270 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
271 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
272 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
273 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
274
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
275 @Snippet
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
276 public static void g1ArrayRangePostWriteBarrier(Object object, int startIndex, int length) {
10951
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
277 if (length == 0) {
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
278 return;
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
279 }
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
280 Object dest = FixedValueAnchorNode.getObject(object);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
281 Word thread = thread();
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
282 Word bufferAddress = thread.readWord(g1CardQueueBufferOffset());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
283 Word indexAddress = thread.add(g1CardQueueIndexOffset());
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
284 long indexValue = thread.readWord(g1CardQueueIndexOffset()).rawValue();
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
285
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
286 int cardShift = cardTableShift();
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
287 long cardStart = cardTableStart();
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
288 final int scale = arrayIndexScale(Kind.Object);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
289 int header = arrayBaseOffset(Kind.Object);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
290 long dstAddr = GetObjectAddressNode.get(dest);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
291 long start = (dstAddr + header + (long) startIndex * scale) >>> cardShift;
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
292 long end = (dstAddr + header + ((long) startIndex + length - 1) * scale) >>> cardShift;
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
293 long count = end - start + 1;
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
294
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
295 while (count-- > 0) {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
296 Word cardAddress = Word.unsigned((start + cardStart) + count);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
297 byte cardByte = cardAddress.readByte(0);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
298 // If the card is already dirty, (hence already enqueued) skip the insertion.
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
299 if (cardByte != (byte) 0) {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
300 cardAddress.writeByte(0, (byte) 0);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
301 // If the thread local card queue is full, issue a native call which will
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
302 // initialize a new one and add the card entry.
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
303 if (indexValue != 0) {
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
304 indexValue = indexValue - wordSize();
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
305 Word logAddress = bufferAddress.add(Word.unsigned(indexValue));
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
306 // Log the object to be scanned as well as update
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
307 // the card queue's next index.
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
308 logAddress.writeWord(0, cardAddress);
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
309 indexAddress.writeWord(0, Word.unsigned(indexValue));
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
310 } else {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
311 g1PostBarrierStub(G1WBPOSTCALL, cardAddress);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
312 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
313 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
314 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
315 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
316
10517
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
317 public static final ForeignCallDescriptor G1WBPRECALL = new ForeignCallDescriptor("write_barrier_pre", void.class, Object.class);
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
318
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
319 @NodeIntrinsic(ForeignCallNode.class)
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
320 private static native void g1PreBarrierStub(@ConstantNodeParameter ForeignCallDescriptor descriptor, Object object);
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
321
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
322 public static final ForeignCallDescriptor G1WBPOSTCALL = new ForeignCallDescriptor("write_barrier_post", void.class, Word.class);
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
323
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
324 @NodeIntrinsic(ForeignCallNode.class)
11234
73eb4d8a4015 Change scope of logging methods
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 11221
diff changeset
325 public static native void g1PostBarrierStub(@ConstantNodeParameter ForeignCallDescriptor descriptor, Word card);
10517
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
326
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
327 public static class Templates extends AbstractTemplates {
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
328
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
329 private final SnippetInfo serialWriteBarrier = snippet(WriteBarrierSnippets.class, "serialWriteBarrier");
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
330 private final SnippetInfo serialArrayRangeWriteBarrier = snippet(WriteBarrierSnippets.class, "serialArrayRangeWriteBarrier");
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
331 private final SnippetInfo g1PreWriteBarrier = snippet(WriteBarrierSnippets.class, "g1PreWriteBarrier");
10829
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
332 private final SnippetInfo g1ReferentReadBarrier = snippet(WriteBarrierSnippets.class, "g1PreWriteBarrier");
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
333 private final SnippetInfo g1PostWriteBarrier = snippet(WriteBarrierSnippets.class, "g1PostWriteBarrier");
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
334 private final SnippetInfo g1ArrayRangePreWriteBarrier = snippet(WriteBarrierSnippets.class, "g1ArrayRangePreWriteBarrier");
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
335 private final SnippetInfo g1ArrayRangePostWriteBarrier = snippet(WriteBarrierSnippets.class, "g1ArrayRangePostWriteBarrier");
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
336
8627
75db7afee829 implemented lazy installation of replacements (GRAAL-137)
Doug Simon <doug.simon@oracle.com>
parents: 8571
diff changeset
337 public Templates(CodeCacheProvider runtime, Replacements replacements, TargetDescription target) {
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
338 super(runtime, replacements, target);
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
339 }
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
340
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
341 public void lower(SerialWriteBarrier writeBarrier, @SuppressWarnings("unused") LoweringTool tool) {
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
342 Arguments args = new Arguments(serialWriteBarrier);
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
343 args.add("object", writeBarrier.getObject());
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
344 args.add("location", writeBarrier.getLocation());
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
345 args.addConst("usePrecise", writeBarrier.usePrecise());
10945
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
346 args.addConst("alwaysNull", writeBarrier.getValue().objectStamp().alwaysNull());
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
347 template(args).instantiate(runtime, writeBarrier, DEFAULT_REPLACER, args);
8571
0bacc5564d18 Revert WB refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8531
diff changeset
348 }
8963
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
349
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
350 public void lower(SerialArrayRangeWriteBarrier arrayRangeWriteBarrier, @SuppressWarnings("unused") LoweringTool tool) {
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
351 Arguments args = new Arguments(serialArrayRangeWriteBarrier);
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
352 args.add("object", arrayRangeWriteBarrier.getObject());
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
353 args.add("startIndex", arrayRangeWriteBarrier.getStartIndex());
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
354 args.add("length", arrayRangeWriteBarrier.getLength());
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
355 template(args).instantiate(runtime, arrayRangeWriteBarrier, DEFAULT_REPLACER, args);
8963
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
356 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
357
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
358 public void lower(G1PreWriteBarrier writeBarrierPre, @SuppressWarnings("unused") LoweringTool tool) {
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
359 Arguments args = new Arguments(g1PreWriteBarrier);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
360 args.add("object", writeBarrierPre.getObject());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
361 args.add("expectedObject", writeBarrierPre.getExpectedObject());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
362 args.add("location", writeBarrierPre.getLocation());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
363 args.addConst("doLoad", writeBarrierPre.doLoad());
10669
f11cd00d4115 Make G1PreWriteBarrier a DeoptimizingNode.
Roland Schatz <roland.schatz@oracle.com>
parents: 10631
diff changeset
364 args.addConst("nullCheck", writeBarrierPre.getNullCheck());
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
365 args.addConst("trace", traceBarrier());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
366 template(args).instantiate(runtime, writeBarrierPre, DEFAULT_REPLACER, args);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
367 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
368
10829
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
369 public void lower(G1ReferentFieldReadBarrier readBarrier, @SuppressWarnings("unused") LoweringTool tool) {
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
370 Arguments args = new Arguments(g1ReferentReadBarrier);
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
371 args.add("object", readBarrier.getObject());
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
372 args.add("expectedObject", readBarrier.getExpectedObject());
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
373 args.add("location", readBarrier.getLocation());
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
374 args.addConst("doLoad", readBarrier.doLoad());
10941
b9718191858c G1ReferentFieldReadBarrier is not a deoptimizing node
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10829
diff changeset
375 args.addConst("nullCheck", false);
10829
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
376 args.addConst("trace", traceBarrier());
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
377 template(args).instantiate(runtime, readBarrier, DEFAULT_REPLACER, args);
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
378 }
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
379
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
380 public void lower(G1PostWriteBarrier writeBarrierPost, @SuppressWarnings("unused") LoweringTool tool) {
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
381 Arguments args = new Arguments(g1PostWriteBarrier);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
382 args.add("object", writeBarrierPost.getObject());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
383 args.add("value", writeBarrierPost.getValue());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
384 args.add("location", writeBarrierPost.getLocation());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
385 args.addConst("usePrecise", writeBarrierPost.usePrecise());
10945
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
386 args.addConst("alwaysNull", writeBarrierPost.getValue().objectStamp().alwaysNull());
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
387 args.addConst("trace", traceBarrier());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
388 template(args).instantiate(runtime, writeBarrierPost, DEFAULT_REPLACER, args);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
389 }
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
390
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
391 public void lower(G1ArrayRangePreWriteBarrier arrayRangeWriteBarrier, @SuppressWarnings("unused") LoweringTool tool) {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
392 Arguments args = new Arguments(g1ArrayRangePreWriteBarrier);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
393 args.add("object", arrayRangeWriteBarrier.getObject());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
394 args.add("startIndex", arrayRangeWriteBarrier.getStartIndex());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
395 args.add("length", arrayRangeWriteBarrier.getLength());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
396 template(args).instantiate(runtime, arrayRangeWriteBarrier, DEFAULT_REPLACER, args);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
397 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
398
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
399 public void lower(G1ArrayRangePostWriteBarrier arrayRangeWriteBarrier, @SuppressWarnings("unused") LoweringTool tool) {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
400 Arguments args = new Arguments(g1ArrayRangePostWriteBarrier);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
401 args.add("object", arrayRangeWriteBarrier.getObject());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
402 args.add("startIndex", arrayRangeWriteBarrier.getStartIndex());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
403 args.add("length", arrayRangeWriteBarrier.getLength());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
404 template(args).instantiate(runtime, arrayRangeWriteBarrier, DEFAULT_REPLACER, args);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
405 }
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
406 }
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
407
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
408 /**
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
409 * Log method of debugging purposes.
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
410 */
11234
73eb4d8a4015 Change scope of logging methods
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 11221
diff changeset
411 public static void log(boolean enabled, String format, long value) {
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
412 if (enabled) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
413 Log.printf(format, value);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
414 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
415 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
416
11234
73eb4d8a4015 Change scope of logging methods
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 11221
diff changeset
417 public static void log(boolean enabled, String format, long value1, long value2) {
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
418 if (enabled) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
419 Log.printf(format, value1, value2);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
420 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
421 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
422
11234
73eb4d8a4015 Change scope of logging methods
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 11221
diff changeset
423 public static void log(boolean enabled, String format, long value1, long value2, long value3) {
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
424 if (enabled) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
425 Log.printf(format, value1, value2, value3);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
426 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
427 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
428
11234
73eb4d8a4015 Change scope of logging methods
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 11221
diff changeset
429 public static boolean traceBarrier() {
10701
f1cd428fca9f Remove caching of gcStartCycles
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10669
diff changeset
430 return GraalOptions.GCDebugStartCycle.getValue() > 0 && ((int) Word.unsigned(HotSpotReplacementsUtil.gcTotalCollectionsAddress()).readLong(0) > GraalOptions.GCDebugStartCycle.getValue());
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
431 }
10953
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
432
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
433 /**
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
434 * Validation helper method which performs sanity checks on write operations. The addresses of
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
435 * both the object and the value being written are checked in order to determine if they reside
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
436 * in a valid heap region. If an object is stale, an invalid access is performed in order to
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
437 * prematurely crash the VM and debug the stack trace of the faulty method.
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
438 */
11234
73eb4d8a4015 Change scope of logging methods
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 11221
diff changeset
439 public static void validateObject(Object parent, Object child) {
11221
8d4bd13c6983 Field renaming
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10955
diff changeset
440 if (verifyOops() && child != null && !validateOop(VALIDATE_OBJECT, parent, child)) {
10953
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
441 log(true, "Verification ERROR, Parent: %p Child: %p\n", Word.fromObject(parent).rawValue(), Word.fromObject(child).rawValue());
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
442 DirectObjectStoreNode.storeWord(null, 0, 0, Word.zero());
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
443 }
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
444 }
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
445
11221
8d4bd13c6983 Field renaming
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10955
diff changeset
446 public static final ForeignCallDescriptor VALIDATE_OBJECT = new ForeignCallDescriptor("validate_object", boolean.class, Word.class, Word.class);
10953
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
447
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
448 @NodeIntrinsic(ForeignCallNode.class)
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
449 private static native boolean validateOop(@ConstantNodeParameter ForeignCallDescriptor descriptor, Object parent, Object object);
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
450
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
451 }