Mercurial > hg > graal-compiler

annotate graal/com.oracle.graal.hotspot/src/com/oracle/graal/hotspot/replacements/WriteBarrierSnippets.java @ 10954:d348dd117239

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Make heap verification conditional
author Christos Kotselidis <christos.kotselidis@oracle.com>
date Fri, 02 Aug 2013 20:07:43 +0200
parents 97e282186b5b
children 769ad79b8e0c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
1 /*
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
2 * Copyright (c) 2012, 2013, Oracle and/or its affiliates. All rights reserved.
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
3 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
4 *
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
5 * This code is free software; you can redistribute it and/or modify it
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
6 * under the terms of the GNU General Public License version 2 only, as
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
7 * published by the Free Software Foundation.
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
8 *
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
9 * This code is distributed in the hope that it will be useful, but WITHOUT
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
10 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
11 * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
12 * version 2 for more details (a copy is included in the LICENSE file that
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
13 * accompanied this code).
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
14 *
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
15 * You should have received a copy of the GNU General Public License version
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
16 * 2 along with this work; if not, write to the Free Software Foundation,
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
17 * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
18 *
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
19 * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
20 * or visit www.oracle.com if you need additional information or have any
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
21 * questions.
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
22 */
8508
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
23 package com.oracle.graal.hotspot.replacements;
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
24
9513
659bb6cf930c rename: HotSpotSnippetUtils -> HotSpotReplacementsUtil
Doug Simon <doug.simon@oracle.com>
parents: 9015
diff changeset
25 import static com.oracle.graal.hotspot.replacements.HotSpotReplacementsUtil.*;
10787
388fbd0dd4a4 moved BranchNodeProbabilities to graal.nodes project
Doug Simon <doug.simon@oracle.com>
parents: 10701
diff changeset
26 import static com.oracle.graal.nodes.extended.BranchProbabilityNode.*;
10481
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
27 import static com.oracle.graal.phases.GraalOptions.*;
8508
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
28 import static com.oracle.graal.replacements.SnippetTemplate.*;
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
29
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
30 import com.oracle.graal.api.code.*;
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
31 import com.oracle.graal.api.meta.*;
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
32 import com.oracle.graal.graph.Node.ConstantNodeParameter;
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
33 import com.oracle.graal.graph.Node.NodeIntrinsic;
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
34 import com.oracle.graal.hotspot.nodes.*;
8905
c502db57e687 Rename ArrayWriteBarrier to SerialWriteBarrier and move it to graal.nodes
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8904
diff changeset
35 import com.oracle.graal.nodes.*;
10828
b05e1ff3aac0 Rename WriteBarrierType to BarrierType
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10817
diff changeset
36 import com.oracle.graal.nodes.HeapAccess.BarrierType;
10517
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
37 import com.oracle.graal.nodes.extended.*;
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
38 import com.oracle.graal.nodes.spi.*;
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
39 import com.oracle.graal.phases.*;
8508
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
40 import com.oracle.graal.replacements.*;
8903
876a449f17ab Merge FieldWriteBarrier into ArrayWriteBarrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8895
diff changeset
41 import com.oracle.graal.replacements.Snippet.ConstantParameter;
8508
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
42 import com.oracle.graal.replacements.SnippetTemplate.AbstractTemplates;
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
43 import com.oracle.graal.replacements.SnippetTemplate.Arguments;
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
44 import com.oracle.graal.replacements.SnippetTemplate.SnippetInfo;
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
45 import com.oracle.graal.replacements.nodes.*;
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
46 import com.oracle.graal.word.*;
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
47
8508
ef97193256d0 -Merge with trunk
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8507
diff changeset
48 public class WriteBarrierSnippets implements Snippets {
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
49
10481
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
50 private static final SnippetCounter.Group countersWriteBarriers = SnippetCounters.getValue() ? new SnippetCounter.Group("WriteBarriers") : null;
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
51 private static final SnippetCounter serialFieldWriteBarrierCounter = new SnippetCounter(countersWriteBarriers, "serialFieldWriteBarrier", "Number of Serial Field Write Barriers");
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
52 private static final SnippetCounter serialArrayWriteBarrierCounter = new SnippetCounter(countersWriteBarriers, "serialArrayWriteBarrier", "Number of Serial Array Write Barriers");
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
53
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
54 @Snippet
10945
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
55 public static void serialWriteBarrier(Object object, Object location, @ConstantParameter boolean usePrecise, @ConstantParameter boolean alwaysNull) {
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
56 // No barriers are added if we are always storing a null.
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
57 if (alwaysNull) {
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
58 return;
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
59 }
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
60 Object fixedObject = FixedValueAnchorNode.getObject(object);
8903
876a449f17ab Merge FieldWriteBarrier into ArrayWriteBarrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8895
diff changeset
61 Pointer oop;
876a449f17ab Merge FieldWriteBarrier into ArrayWriteBarrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8895
diff changeset
62 if (usePrecise) {
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
63 oop = Word.fromArray(fixedObject, location);
10481
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
64 serialArrayWriteBarrierCounter.inc();
8903
876a449f17ab Merge FieldWriteBarrier into ArrayWriteBarrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8895
diff changeset
65 } else {
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
66 oop = Word.fromObject(fixedObject);
10481
29e9a5d18c70 Clean up.
Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
parents: 9513
diff changeset
67 serialFieldWriteBarrierCounter.inc();
8903
876a449f17ab Merge FieldWriteBarrier into ArrayWriteBarrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8895
diff changeset
68 }
8518
4a79f7c0d1a4 -Fix assertion errors
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8517
diff changeset
69 Word base = (Word) oop.unsignedShiftRight(cardTableShift());
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
70 long startAddress = cardTableStart();
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
71 int displacement = 0;
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
72 if (((int) startAddress) == startAddress) {
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
73 displacement = (int) startAddress;
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
74 } else {
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
75 base = base.add(Word.unsigned(cardTableStart()));
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
76 }
8520
450abf358987 -Bug fix in card marking
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8518
diff changeset
77 base.writeByte(displacement, (byte) 0);
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
78 }
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
79
8963
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
80 @Snippet
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
81 public static void serialArrayRangeWriteBarrier(Object object, int startIndex, int length) {
10942
56a86b60c9d3 Early exit of SerialArrayRange barrier if vector length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10941
diff changeset
82 if (length == 0) {
56a86b60c9d3 Early exit of SerialArrayRange barrier if vector length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10941
diff changeset
83 return;
56a86b60c9d3 Early exit of SerialArrayRange barrier if vector length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10941
diff changeset
84 }
8989
40e7444d62ee Variable renaming
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8985
diff changeset
85 Object dest = FixedValueAnchorNode.getObject(object);
8963
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
86 int cardShift = cardTableShift();
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
87 long cardStart = cardTableStart();
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
88 final int scale = arrayIndexScale(Kind.Object);
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
89 int header = arrayBaseOffset(Kind.Object);
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
90 long dstAddr = GetObjectAddressNode.get(dest);
8989
40e7444d62ee Variable renaming
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8985
diff changeset
91 long start = (dstAddr + header + (long) startIndex * scale) >>> cardShift;
40e7444d62ee Variable renaming
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8985
diff changeset
92 long end = (dstAddr + header + ((long) startIndex + length - 1) * scale) >>> cardShift;
8963
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
93 long count = end - start + 1;
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
94 while (count-- > 0) {
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
95 DirectStoreNode.store((start + cardStart) + count, false, Kind.Boolean);
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
96 }
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
97 }
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
98
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
99 @Snippet
10623
13c6440ac750 Add nullcheck at G1 Pre Barrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10622
diff changeset
100 public static void g1PreWriteBarrier(Object object, Object expectedObject, Object location, @ConstantParameter boolean doLoad, @ConstantParameter boolean nullCheck,
13c6440ac750 Add nullcheck at G1 Pre Barrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10622
diff changeset
101 @ConstantParameter boolean trace) {
13c6440ac750 Add nullcheck at G1 Pre Barrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10622
diff changeset
102 if (nullCheck && object == null) {
13c6440ac750 Add nullcheck at G1 Pre Barrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10622
diff changeset
103 DeoptimizeNode.deopt(DeoptimizationAction.InvalidateReprofile, DeoptimizationReason.NullCheckException);
13c6440ac750 Add nullcheck at G1 Pre Barrier
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10622
diff changeset
104 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
105 Word thread = thread();
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
106 Object fixedObject = FixedValueAnchorNode.getObject(object);
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
107 verifyOop(fixedObject);
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
108 Object fixedExpectedObject = FixedValueAnchorNode.getObject(expectedObject);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
109 Word field = (Word) Word.fromArray(fixedObject, location);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
110 Word previousOop = (Word) Word.fromObject(fixedExpectedObject);
10520
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
111 byte markingValue = thread.readByte(g1SATBQueueMarkingOffset());
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
112 Word bufferAddress = thread.readWord(g1SATBQueueBufferOffset());
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
113 Word indexAddress = thread.add(g1SATBQueueIndexOffset());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
114 Word indexValue = indexAddress.readWord(0);
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
115 int gcCycle = 0;
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
116 if (trace) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
117 gcCycle = (int) Word.unsigned(HotSpotReplacementsUtil.gcTotalCollectionsAddress()).readLong(0);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
118 log(trace, "[%d] G1-Pre Thread %p Object %p\n", gcCycle, thread.rawValue(), Word.fromObject(fixedObject).rawValue());
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
119 log(trace, "[%d] G1-Pre Thread %p Expected Object %p\n", gcCycle, thread.rawValue(), Word.fromObject(fixedExpectedObject).rawValue());
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
120 log(trace, "[%d] G1-Pre Thread %p Field %p\n", gcCycle, thread.rawValue(), field.rawValue());
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
121 log(trace, "[%d] G1-Pre Thread %p Marking %d\n", gcCycle, thread.rawValue(), markingValue);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
122 log(trace, "[%d] G1-Pre Thread %p DoLoad %d\n", gcCycle, thread.rawValue(), doLoad ? 1L : 0L);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
123 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
124 // If the concurrent marker is enabled, the barrier is issued.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
125 if (probability(NOT_LIKELY_PROBABILITY, markingValue != (byte) 0)) {
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
126 // If the previous value has to be loaded (before the write), the load is issued.
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
127 // The load is always issued except the cases of CAS and referent field.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
128 if (probability(LIKELY_PROBABILITY, doLoad)) {
10828
b05e1ff3aac0 Rename WriteBarrierType to BarrierType
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10817
diff changeset
129 previousOop = (Word) Word.fromObject(field.readObject(0, BarrierType.NONE, true));
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
130 if (trace) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
131 log(trace, "[%d] G1-Pre Thread %p Previous Object %p\n ", gcCycle, thread.rawValue(), previousOop.rawValue());
10631
2e82291febf4 Oop verification after printing oop
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10630
diff changeset
132 verifyOop(previousOop.toObject());
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
133 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
134 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
135 // If the previous value is null the barrier should not be issued.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
136 if (probability(FREQUENT_PROBABILITY, previousOop.notEqual(0))) {
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
137 // If the thread-local SATB buffer is full issue a native call which will
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
138 // initialize a new one and add the entry.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
139 if (probability(FREQUENT_PROBABILITY, indexValue.notEqual(0))) {
10520
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
140 Word nextIndex = indexValue.subtract(wordSize());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
141 Word logAddress = bufferAddress.add(nextIndex);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
142 // Log the object to be marked as well as update the SATB's buffer next index.
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
143 logAddress.writeWord(0, previousOop);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
144 indexAddress.writeWord(0, nextIndex);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
145 } else {
10517
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
146 g1PreBarrierStub(G1WBPRECALL, previousOop.toObject());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
147 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
148 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
149 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
150 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
151
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
152 @Snippet
10945
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
153 public static void g1PostWriteBarrier(Object object, Object value, Object location, @ConstantParameter boolean usePrecise, @ConstantParameter boolean alwaysNull, @ConstantParameter boolean trace) {
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
154 // No barriers are added if we are always storing a null.
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
155 if (alwaysNull) {
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
156 return;
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
157 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
158 Word thread = thread();
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
159 Object fixedObject = FixedValueAnchorNode.getObject(object);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
160 Object fixedValue = FixedValueAnchorNode.getObject(value);
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
161 verifyOop(fixedObject);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
162 verifyOop(fixedValue);
10953
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
163 validateObject(fixedObject, fixedValue);
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
164 Word oop;
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
165 if (usePrecise) {
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
166 oop = (Word) Word.fromArray(fixedObject, location);
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
167 } else {
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
168 oop = (Word) Word.fromObject(fixedObject);
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
169 }
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
170 int gcCycle = 0;
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
171 if (trace) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
172 gcCycle = (int) Word.unsigned(HotSpotReplacementsUtil.gcTotalCollectionsAddress()).readLong(0);
10630
ca3865947682 Debug log cleanup
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10623
diff changeset
173 log(trace, "[%d] G1-Post Thread: %p Object: %p\n", gcCycle, thread.rawValue(), Word.fromObject(fixedObject).rawValue());
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
174 log(trace, "[%d] G1-Post Thread: %p Field: %p\n", gcCycle, thread.rawValue(), oop.rawValue());
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
175 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
176 Word writtenValue = (Word) Word.fromObject(fixedValue);
10520
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
177 Word bufferAddress = thread.readWord(g1CardQueueBufferOffset());
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
178 Word indexAddress = thread.add(g1CardQueueIndexOffset());
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
179 Word indexValue = thread.readWord(g1CardQueueIndexOffset());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
180 // The result of the xor reveals whether the installed pointer crosses heap regions.
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
181 // In case it does the write barrier has to be issued.
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
182 Word xorResult = (oop.xor(writtenValue)).unsignedShiftRight(logOfHeapRegionGrainBytes());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
183
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
184 // Calculate the address of the card to be enqueued to the
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
185 // thread local card queue.
10950
d2187f88cdd2 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10945
diff changeset
186 Word cardBase = oop.unsignedShiftRight(cardTableShift());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
187 long startAddress = cardTableStart();
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
188 int displacement = 0;
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
189 if (((int) startAddress) == startAddress) {
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
190 displacement = (int) startAddress;
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
191 } else {
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
192 cardBase = cardBase.add(Word.unsigned(cardTableStart()));
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
193 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
194 Word cardAddress = cardBase.add(displacement);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
195
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
196 if (probability(LIKELY_PROBABILITY, xorResult.notEqual(0))) {
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
197 // If the written value is not null continue with the barrier addition.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
198 if (probability(FREQUENT_PROBABILITY, writtenValue.notEqual(0))) {
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
199 byte cardByte = cardAddress.readByte(0);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
200 // If the card is already dirty, (hence already enqueued) skip the insertion.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
201 if (probability(LIKELY_PROBABILITY, cardByte != (byte) 0)) {
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
202 log(trace, "[%d] G1-Post Thread: %p Card: %p \n", gcCycle, thread.rawValue(), Word.unsigned(cardByte).rawValue());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
203 cardAddress.writeByte(0, (byte) 0);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
204 // If the thread local card queue is full, issue a native call which will
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
205 // initialize a new one and add the card entry.
10622
844be189150c Add probabilities in WB snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10620
diff changeset
206 if (probability(FREQUENT_PROBABILITY, indexValue.notEqual(0))) {
10520
590f8f159309 Static imports in write barrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10519
diff changeset
207 Word nextIndex = indexValue.subtract(wordSize());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
208 Word logAddress = bufferAddress.add(nextIndex);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
209 // Log the object to be scanned as well as update
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
210 // the card queue's next index.
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
211 logAddress.writeWord(0, cardAddress);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
212 indexAddress.writeWord(0, nextIndex);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
213 } else {
10517
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
214 g1PostBarrierStub(G1WBPOSTCALL, cardAddress);
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
215 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
216 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
217 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
218 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
219 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
220
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
221 @Snippet
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
222 public static void g1ArrayRangePreWriteBarrier(Object object, int startIndex, int length) {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
223 Word thread = thread();
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
224 byte markingValue = thread.readByte(g1SATBQueueMarkingOffset());
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
225 // If the concurrent marker is not enabled or the vector length is zero, return.
10951
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
226 if (markingValue == (byte) 0 || length == 0) {
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
227 return;
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
228 }
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
229 Object dest = FixedValueAnchorNode.getObject(object);
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
230 Word bufferAddress = thread.readWord(g1SATBQueueBufferOffset());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
231 Word indexAddress = thread.add(g1SATBQueueIndexOffset());
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
232 long dstAddr = GetObjectAddressNode.get(dest);
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
233 long indexValue = indexAddress.readWord(0).rawValue();
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
234 final int scale = arrayIndexScale(Kind.Object);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
235 int header = arrayBaseOffset(Kind.Object);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
236
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
237 for (int i = startIndex; i < length; i++) {
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
238 long address = dstAddr + header + (i * scale);
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
239 Pointer oop = Word.fromObject(Word.unsigned(address).readObject(0, BarrierType.NONE, true));
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
240 verifyOop(oop.toObject());
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
241 if (oop.notEqual(0)) {
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
242 if (indexValue != 0) {
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
243 indexValue = indexValue - wordSize();
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
244 Word logAddress = bufferAddress.add(Word.unsigned(indexValue));
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
245 // Log the object to be marked as well as update the SATB's buffer next index.
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
246 logAddress.writeWord(0, oop);
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
247 indexAddress.writeWord(0, Word.unsigned(indexValue));
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
248 } else {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
249 g1PreBarrierStub(G1WBPRECALL, oop.toObject());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
250 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
251 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
252 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
253 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
254
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
255 @Snippet
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
256 public static void g1ArrayRangePostWriteBarrier(Object object, int startIndex, int length) {
10951
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
257 if (length == 0) {
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
258 return;
e2ac8bf20433 Early exits from G1 Array Range barriers if length==0
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10950
diff changeset
259 }
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
260 Object dest = FixedValueAnchorNode.getObject(object);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
261 Word thread = thread();
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
262 Word bufferAddress = thread.readWord(g1CardQueueBufferOffset());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
263 Word indexAddress = thread.add(g1CardQueueIndexOffset());
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
264 long indexValue = thread.readWord(g1CardQueueIndexOffset()).rawValue();
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
265
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
266 int cardShift = cardTableShift();
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
267 long cardStart = cardTableStart();
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
268 final int scale = arrayIndexScale(Kind.Object);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
269 int header = arrayBaseOffset(Kind.Object);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
270 long dstAddr = GetObjectAddressNode.get(dest);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
271 long start = (dstAddr + header + (long) startIndex * scale) >>> cardShift;
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
272 long end = (dstAddr + header + ((long) startIndex + length - 1) * scale) >>> cardShift;
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
273 long count = end - start + 1;
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
274
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
275 while (count-- > 0) {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
276 Word cardAddress = Word.unsigned((start + cardStart) + count);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
277 byte cardByte = cardAddress.readByte(0);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
278 // If the card is already dirty, (hence already enqueued) skip the insertion.
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
279 if (cardByte != (byte) 0) {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
280 cardAddress.writeByte(0, (byte) 0);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
281 // If the thread local card queue is full, issue a native call which will
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
282 // initialize a new one and add the card entry.
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
283 if (indexValue != 0) {
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
284 indexValue = indexValue - wordSize();
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
285 Word logAddress = bufferAddress.add(Word.unsigned(indexValue));
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
286 // Log the object to be scanned as well as update
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
287 // the card queue's next index.
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
288 logAddress.writeWord(0, cardAddress);
10952
b43bc053ce8f Fix bug in CardQueue/SATB buffer indexes' calculation in ArrayRangeBarrier snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10951
diff changeset
289 indexAddress.writeWord(0, Word.unsigned(indexValue));
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
290 } else {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
291 g1PostBarrierStub(G1WBPOSTCALL, cardAddress);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
292 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
293 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
294 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
295 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
296
10517
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
297 public static final ForeignCallDescriptor G1WBPRECALL = new ForeignCallDescriptor("write_barrier_pre", void.class, Object.class);
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
298
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
299 @NodeIntrinsic(ForeignCallNode.class)
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
300 private static native void g1PreBarrierStub(@ConstantNodeParameter ForeignCallDescriptor descriptor, Object object);
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
301
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
302 public static final ForeignCallDescriptor G1WBPOSTCALL = new ForeignCallDescriptor("write_barrier_post", void.class, Word.class);
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
303
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
304 @NodeIntrinsic(ForeignCallNode.class)
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
305 private static native void g1PostBarrierStub(@ConstantNodeParameter ForeignCallDescriptor descriptor, Word card);
c2c9afee00b5 Replace G1 stub call nodes with intrinsics
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10505
diff changeset
306
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
307 public static class Templates extends AbstractTemplates {
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
308
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
309 private final SnippetInfo serialWriteBarrier = snippet(WriteBarrierSnippets.class, "serialWriteBarrier");
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
310 private final SnippetInfo serialArrayRangeWriteBarrier = snippet(WriteBarrierSnippets.class, "serialArrayRangeWriteBarrier");
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
311 private final SnippetInfo g1PreWriteBarrier = snippet(WriteBarrierSnippets.class, "g1PreWriteBarrier");
10829
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
312
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
313 private final SnippetInfo g1ReferentReadBarrier = snippet(WriteBarrierSnippets.class, "g1PreWriteBarrier");
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
314 private final SnippetInfo g1PostWriteBarrier = snippet(WriteBarrierSnippets.class, "g1PostWriteBarrier");
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
315 private final SnippetInfo g1ArrayRangePreWriteBarrier = snippet(WriteBarrierSnippets.class, "g1ArrayRangePreWriteBarrier");
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
316 private final SnippetInfo g1ArrayRangePostWriteBarrier = snippet(WriteBarrierSnippets.class, "g1ArrayRangePostWriteBarrier");
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
317
8627
75db7afee829 implemented lazy installation of replacements (GRAAL-137)
Doug Simon <doug.simon@oracle.com>
parents: 8571
diff changeset
318 public Templates(CodeCacheProvider runtime, Replacements replacements, TargetDescription target) {
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
319 super(runtime, replacements, target);
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
320 }
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
321
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
322 public void lower(SerialWriteBarrier writeBarrier, @SuppressWarnings("unused") LoweringTool tool) {
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
323 Arguments args = new Arguments(serialWriteBarrier);
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
324 args.add("object", writeBarrier.getObject());
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
325 args.add("location", writeBarrier.getLocation());
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
326 args.addConst("usePrecise", writeBarrier.usePrecise());
10945
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
327 args.addConst("alwaysNull", writeBarrier.getValue().objectStamp().alwaysNull());
10944
3a9f3ec8efc6 Minor refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10942
diff changeset
328 template(args).instantiate(runtime, writeBarrier, DEFAULT_REPLACER, args);
8571
0bacc5564d18 Revert WB refactoring
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8531
diff changeset
329 }
8963
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
330
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
331 public void lower(SerialArrayRangeWriteBarrier arrayRangeWriteBarrier, @SuppressWarnings("unused") LoweringTool tool) {
9015
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
332 Arguments args = new Arguments(serialArrayRangeWriteBarrier);
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
333 args.add("object", arrayRangeWriteBarrier.getObject());
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
334 args.add("startIndex", arrayRangeWriteBarrier.getStartIndex());
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
335 args.add("length", arrayRangeWriteBarrier.getLength());
92d2bedb5dfc Change of Snippet template and instantiation process
Christian Wimmer <christian.wimmer@oracle.com>
parents: 8989
diff changeset
336 template(args).instantiate(runtime, arrayRangeWriteBarrier, DEFAULT_REPLACER, args);
8963
63eae4723b18 Creation of snippets for ArrayCopy write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 8905
diff changeset
337 }
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
338
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
339 public void lower(G1PreWriteBarrier writeBarrierPre, @SuppressWarnings("unused") LoweringTool tool) {
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
340 Arguments args = new Arguments(g1PreWriteBarrier);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
341 args.add("object", writeBarrierPre.getObject());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
342 args.add("expectedObject", writeBarrierPre.getExpectedObject());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
343 args.add("location", writeBarrierPre.getLocation());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
344 args.addConst("doLoad", writeBarrierPre.doLoad());
10669
f11cd00d4115 Make G1PreWriteBarrier a DeoptimizingNode.
Roland Schatz <roland.schatz@oracle.com>
parents: 10631
diff changeset
345 args.addConst("nullCheck", writeBarrierPre.getNullCheck());
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
346 args.addConst("trace", traceBarrier());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
347 template(args).instantiate(runtime, writeBarrierPre, DEFAULT_REPLACER, args);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
348 }
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
349
10829
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
350 public void lower(G1ReferentFieldReadBarrier readBarrier, @SuppressWarnings("unused") LoweringTool tool) {
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
351 Arguments args = new Arguments(g1ReferentReadBarrier);
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
352 args.add("object", readBarrier.getObject());
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
353 args.add("expectedObject", readBarrier.getExpectedObject());
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
354 args.add("location", readBarrier.getLocation());
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
355 args.addConst("doLoad", readBarrier.doLoad());
10941
b9718191858c G1ReferentFieldReadBarrier is not a deoptimizing node
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10829
diff changeset
356 args.addConst("nullCheck", false);
10829
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
357 args.addConst("trace", traceBarrier());
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
358 template(args).instantiate(runtime, readBarrier, DEFAULT_REPLACER, args);
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
359 }
761002c55728 Add new node for referent field read barrier (G1 GC)
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10828
diff changeset
360
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
361 public void lower(G1PostWriteBarrier writeBarrierPost, @SuppressWarnings("unused") LoweringTool tool) {
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
362 Arguments args = new Arguments(g1PostWriteBarrier);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
363 args.add("object", writeBarrierPost.getObject());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
364 args.add("value", writeBarrierPost.getValue());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
365 args.add("location", writeBarrierPost.getLocation());
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
366 args.addConst("usePrecise", writeBarrierPost.usePrecise());
10945
a03d31fa01d9 Writes with always null stamps omit only post barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10944
diff changeset
367 args.addConst("alwaysNull", writeBarrierPost.getValue().objectStamp().alwaysNull());
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
368 args.addConst("trace", traceBarrier());
10494
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
369 template(args).instantiate(runtime, writeBarrierPost, DEFAULT_REPLACER, args);
6f331db530f6 Add G1 Barrier Snippets
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 9513
diff changeset
370 }
10552
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
371
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
372 public void lower(G1ArrayRangePreWriteBarrier arrayRangeWriteBarrier, @SuppressWarnings("unused") LoweringTool tool) {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
373 Arguments args = new Arguments(g1ArrayRangePreWriteBarrier);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
374 args.add("object", arrayRangeWriteBarrier.getObject());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
375 args.add("startIndex", arrayRangeWriteBarrier.getStartIndex());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
376 args.add("length", arrayRangeWriteBarrier.getLength());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
377 template(args).instantiate(runtime, arrayRangeWriteBarrier, DEFAULT_REPLACER, args);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
378 }
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
379
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
380 public void lower(G1ArrayRangePostWriteBarrier arrayRangeWriteBarrier, @SuppressWarnings("unused") LoweringTool tool) {
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
381 Arguments args = new Arguments(g1ArrayRangePostWriteBarrier);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
382 args.add("object", arrayRangeWriteBarrier.getObject());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
383 args.add("startIndex", arrayRangeWriteBarrier.getStartIndex());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
384 args.add("length", arrayRangeWriteBarrier.getLength());
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
385 template(args).instantiate(runtime, arrayRangeWriteBarrier, DEFAULT_REPLACER, args);
ca8efaf18f27 Add ArrayRange Snippets for G1
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10547
diff changeset
386 }
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
387 }
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
388
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
389 /**
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
390 * Log method of debugging purposes.
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
391 */
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
392 private static void log(boolean enabled, String format, long value) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
393 if (enabled) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
394 Log.printf(format, value);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
395 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
396 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
397
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
398 private static void log(boolean enabled, String format, long value1, long value2) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
399 if (enabled) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
400 Log.printf(format, value1, value2);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
401 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
402 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
403
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
404 private static void log(boolean enabled, String format, long value1, long value2, long value3) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
405 if (enabled) {
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
406 Log.printf(format, value1, value2, value3);
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
407 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
408 }
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
409
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
410 private static boolean traceBarrier() {
10701
f1cd428fca9f Remove caching of gcStartCycles
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10669
diff changeset
411 return GraalOptions.GCDebugStartCycle.getValue() > 0 && ((int) Word.unsigned(HotSpotReplacementsUtil.gcTotalCollectionsAddress()).readLong(0) > GraalOptions.GCDebugStartCycle.getValue());
10620
260e75dc4b6e Add tracing info to G1 Write Barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10552
diff changeset
412 }
10953
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
413
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
414 /**
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
415 * Validation helper method which performs sanity checks on write operations. The addresses of
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
416 * both the object and the value being written are checked in order to determine if they reside
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
417 * in a valid heap region. If an object is stale, an invalid access is performed in order to
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
418 * prematurely crash the VM and debug the stack trace of the faulty method.
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
419 */
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
420 private static void validateObject(Object parent, Object child) {
10954
d348dd117239 Make heap verification conditional
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10953
diff changeset
421 if (verifyOops() && child != null && !validateOop(VALIDATEOBJECT, parent, child)) {
10953
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
422 log(true, "Verification ERROR, Parent: %p Child: %p\n", Word.fromObject(parent).rawValue(), Word.fromObject(child).rawValue());
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
423 DirectObjectStoreNode.storeWord(null, 0, 0, Word.zero());
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
424 }
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
425 }
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
426
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
427 public static final ForeignCallDescriptor VALIDATEOBJECT = new ForeignCallDescriptor("validate_object", boolean.class, Word.class, Word.class);
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
428
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
429 @NodeIntrinsic(ForeignCallNode.class)
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
430 private static native boolean validateOop(@ConstantNodeParameter ForeignCallDescriptor descriptor, Object parent, Object object);
97e282186b5b Add heap sanity checker with premature hard crash for debugging write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents: 10952
diff changeset
431
8488
225c984588ee -Snippetization of CMS write barriers
Christos Kotselidis <christos.kotselidis@oracle.com>
parents:
diff changeset
432 }