graal-compiler: src/share/vm/gc_implementation/g1/concurrentMark.inline.hpp annotate

annotate src/share/vm/gc_implementation/g1/concurrentMark.inline.hpp @ 8733:9def4075da6d

8008079: G1: Add nextObject routine to CMBitMapRO and replace nextWord Summary: Update the task local finger to the start of the next object when marking aborts, in order to avoid the redundant scanning of all 0's when the marking task restarts, if otherwise updating to the next word. In addition, reuse the routine nextObject() in routine iterate(). Reviewed-by: johnc, ysr Contributed-by: tamao <tao.mao@oracle.com>

author	tamao
date	Tue, 05 Mar 2013 15:36:56 -0800
parents	8a5ea0a9ccc4
children	d55c004e1d4d

rev	line source
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	1 /*
8733 9def4075da6d 8008079: G1: Add nextObject routine to CMBitMapRO and replace nextWord tamao parents: 6862 diff changeset	2 * Copyright (c) 2001, 2013, Oracle and/or its affiliates. All rights reserved.
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	3 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	4 *
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	5 * This code is free software; you can redistribute it and/or modify it
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	6 * under the terms of the GNU General Public License version 2 only, as
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	7 * published by the Free Software Foundation.
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	8 *
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	9 * This code is distributed in the hope that it will be useful, but WITHOUT
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	10 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	11 * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	12 * version 2 for more details (a copy is included in the LICENSE file that
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	13 * accompanied this code).
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	14 *
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	15 * You should have received a copy of the GNU General Public License version
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	16 * 2 along with this work; if not, write to the Free Software Foundation,
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	17 * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	18 *
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	19 * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	20 * or visit www.oracle.com if you need additional information or have any
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	21 * questions.
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	22 *
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	23 */
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	24
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	25 #ifndef SHARE_VM_GC_IMPLEMENTATION_G1_CONCURRENTMARK_INLINE_HPP
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	26 #define SHARE_VM_GC_IMPLEMENTATION_G1_CONCURRENTMARK_INLINE_HPP
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	27
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	28 #include "gc_implementation/g1/concurrentMark.hpp"
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	29 #include "gc_implementation/g1/g1CollectedHeap.inline.hpp"
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	30
6812 988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	31 // Utility routine to set an exclusive range of cards on the given
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	32 // card liveness bitmap
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	33 inline void ConcurrentMark::set_card_bitmap_range(BitMap* card_bm,
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	34 BitMap::idx_t start_idx,
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	35 BitMap::idx_t end_idx,
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	36 bool is_par) {
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	37
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	38 // Set the exclusive bit range [start_idx, end_idx).
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	39 assert((end_idx - start_idx) > 0, "at least one card");
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	40 assert(end_idx <= card_bm->size(), "sanity");
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	41
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	42 // Silently clip the end index
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	43 end_idx = MIN2(end_idx, card_bm->size());
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	44
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	45 // For small ranges use a simple loop; otherwise use set_range or
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	46 // use par_at_put_range (if parallel). The range is made up of the
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	47 // cards that are spanned by an object/mem region so 8 cards will
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	48 // allow up to object sizes up to 4K to be handled using the loop.
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	49 if ((end_idx - start_idx) <= 8) {
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	50 for (BitMap::idx_t i = start_idx; i < end_idx; i += 1) {
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	51 if (is_par) {
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	52 card_bm->par_set_bit(i);
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	53 } else {
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	54 card_bm->set_bit(i);
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	55 }
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	56 }
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	57 } else {
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	58 // Note BitMap::par_at_put_range() and BitMap::set_range() are exclusive.
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	59 if (is_par) {
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	60 card_bm->par_at_put_range(start_idx, end_idx, true);
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	61 } else {
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	62 card_bm->set_range(start_idx, end_idx);
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	63 }
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	64 }
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	65 }
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	66
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	67 // Returns the index in the liveness accounting card bitmap
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	68 // for the given address
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	69 inline BitMap::idx_t ConcurrentMark::card_bitmap_index_for(HeapWord* addr) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	70 // Below, the term "card num" means the result of shifting an address
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	71 // by the card shift -- address 0 corresponds to card number 0. One
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	72 // must subtract the card num of the bottom of the heap to obtain a
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	73 // card table index.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	74 intptr_t card_num = intptr_t(uintptr_t(addr) >> CardTableModRefBS::card_shift);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	75 return card_num - heap_bottom_card_num();
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	76 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	77
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	78 // Counts the given memory region in the given task/worker
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	79 // counting data structures.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	80 inline void ConcurrentMark::count_region(MemRegion mr, HeapRegion* hr,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	81 size_t* marked_bytes_array,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	82 BitMap* task_card_bm) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	83 G1CollectedHeap* g1h = _g1h;
6812 988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	84 CardTableModRefBS* ct_bs = (CardTableModRefBS*) (g1h->barrier_set());
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	85
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	86 HeapWord* start = mr.start();
6812 988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	87 HeapWord* end = mr.end();
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	88 size_t region_size_bytes = mr.byte_size();
6010 720b6a76dd9d 7157073: G1: type change size_t -> uint for region counts / indexes tonyp parents: 4837 diff changeset	89 uint index = hr->hrs_index();
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	90
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	91 assert(!hr->continuesHumongous(), "should not be HC region");
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	92 assert(hr == g1h->heap_region_containing(start), "sanity");
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	93 assert(hr == g1h->heap_region_containing(mr.last()), "sanity");
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	94 assert(marked_bytes_array != NULL, "pre-condition");
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	95 assert(task_card_bm != NULL, "pre-condition");
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	96
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	97 // Add to the task local marked bytes for this region.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	98 marked_bytes_array[index] += region_size_bytes;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	99
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	100 BitMap::idx_t start_idx = card_bitmap_index_for(start);
6812 988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	101 BitMap::idx_t end_idx = card_bitmap_index_for(end);
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	102
6812 988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	103 // Note: if we're looking at the last region in heap - end
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	104 // could be actually just beyond the end of the heap; end_idx
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	105 // will then correspond to a (non-existent) card that is also
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	106 // just beyond the heap.
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	107 if (g1h->is_in_g1_reserved(end) && !ct_bs->is_card_aligned(end)) {
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	108 // end of region is not card aligned - incremement to cover
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	109 // all the cards spanned by the region.
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	110 end_idx += 1;
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	111 }
6812 988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	112 // The card bitmap is task/worker specific => no need to use
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	113 // the 'par' BitMap routines.
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	114 // Set bits in the exclusive bit range [start_idx, end_idx).
988bf00cc564 7200261: G1: Liveness counting inconsistencies during marking verification johnc parents: 6010 diff changeset	115 set_card_bitmap_range(task_card_bm, start_idx, end_idx, false /* is_par */);
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	116 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	117
4837 eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	118 // Counts the given memory region in the task/worker counting
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	119 // data structures for the given worker id.
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	120 inline void ConcurrentMark::count_region(MemRegion mr,
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	121 HeapRegion* hr,
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	122 uint worker_id) {
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	123 size_t* marked_bytes_array = count_marked_bytes_array_for(worker_id);
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	124 BitMap* task_card_bm = count_card_bitmap_for(worker_id);
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	125 count_region(mr, hr, marked_bytes_array, task_card_bm);
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	126 }
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	127
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	128 // Counts the given memory region, which may be a single object, in the
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	129 // task/worker counting data structures for the given worker id.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	130 inline void ConcurrentMark::count_region(MemRegion mr, uint worker_id) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	131 HeapWord* addr = mr.start();
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	132 HeapRegion* hr = _g1h->heap_region_containing_raw(addr);
4837 eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	133 count_region(mr, hr, worker_id);
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	134 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	135
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	136 // Counts the given object in the given task/worker counting data structures.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	137 inline void ConcurrentMark::count_object(oop obj,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	138 HeapRegion* hr,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	139 size_t* marked_bytes_array,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	140 BitMap* task_card_bm) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	141 MemRegion mr((HeapWord*)obj, obj->size());
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	142 count_region(mr, hr, marked_bytes_array, task_card_bm);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	143 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	144
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	145 // Counts the given object in the task/worker counting data
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	146 // structures for the given worker id.
4837 eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	147 inline void ConcurrentMark::count_object(oop obj,
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	148 HeapRegion* hr,
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	149 uint worker_id) {
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	150 size_t* marked_bytes_array = count_marked_bytes_array_for(worker_id);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	151 BitMap* task_card_bm = count_card_bitmap_for(worker_id);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	152 HeapWord* addr = (HeapWord*) obj;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	153 count_object(obj, hr, marked_bytes_array, task_card_bm);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	154 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	155
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	156 // Attempts to mark the given object and, if successful, counts
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	157 // the object in the given task/worker counting structures.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	158 inline bool ConcurrentMark::par_mark_and_count(oop obj,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	159 HeapRegion* hr,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	160 size_t* marked_bytes_array,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	161 BitMap* task_card_bm) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	162 HeapWord* addr = (HeapWord*)obj;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	163 if (_nextMarkBitMap->parMark(addr)) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	164 // Update the task specific count data for the object.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	165 count_object(obj, hr, marked_bytes_array, task_card_bm);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	166 return true;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	167 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	168 return false;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	169 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	170
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	171 // Attempts to mark the given object and, if successful, counts
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	172 // the object in the task/worker counting structures for the
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	173 // given worker id.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	174 inline bool ConcurrentMark::par_mark_and_count(oop obj,
4837 eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	175 size_t word_size,
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	176 HeapRegion* hr,
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	177 uint worker_id) {
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	178 HeapWord* addr = (HeapWord*)obj;
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	179 if (_nextMarkBitMap->parMark(addr)) {
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	180 MemRegion mr(addr, word_size);
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	181 count_region(mr, hr, worker_id);
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	182 return true;
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	183 }
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	184 return false;
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	185 }
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	186
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	187 // Attempts to mark the given object and, if successful, counts
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	188 // the object in the task/worker counting structures for the
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	189 // given worker id.
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	190 inline bool ConcurrentMark::par_mark_and_count(oop obj,
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	191 HeapRegion* hr,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	192 uint worker_id) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	193 HeapWord* addr = (HeapWord*)obj;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	194 if (_nextMarkBitMap->parMark(addr)) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	195 // Update the task specific count data for the object.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	196 count_object(obj, hr, worker_id);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	197 return true;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	198 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	199 return false;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	200 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	201
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	202 // As above - but we don't know the heap region containing the
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	203 // object and so have to supply it.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	204 inline bool ConcurrentMark::par_mark_and_count(oop obj, uint worker_id) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	205 HeapWord* addr = (HeapWord*)obj;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	206 HeapRegion* hr = _g1h->heap_region_containing_raw(addr);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	207 return par_mark_and_count(obj, hr, worker_id);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	208 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	209
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	210 // Similar to the above routine but we already know the size, in words, of
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	211 // the object that we wish to mark/count
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	212 inline bool ConcurrentMark::par_mark_and_count(oop obj,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	213 size_t word_size,
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	214 uint worker_id) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	215 HeapWord* addr = (HeapWord*)obj;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	216 if (_nextMarkBitMap->parMark(addr)) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	217 // Update the task specific count data for the object.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	218 MemRegion mr(addr, word_size);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	219 count_region(mr, worker_id);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	220 return true;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	221 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	222 return false;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	223 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	224
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	225 // Unconditionally mark the given object, and unconditinally count
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	226 // the object in the counting structures for worker id 0.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	227 // Should not be called from parallel code.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	228 inline bool ConcurrentMark::mark_and_count(oop obj, HeapRegion* hr) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	229 HeapWord* addr = (HeapWord*)obj;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	230 _nextMarkBitMap->mark(addr);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	231 // Update the task specific count data for the object.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	232 count_object(obj, hr, 0 /* worker_id */);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	233 return true;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	234 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	235
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	236 // As above - but we don't have the heap region containing the
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	237 // object, so we have to supply it.
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	238 inline bool ConcurrentMark::mark_and_count(oop obj) {
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	239 HeapWord* addr = (HeapWord*)obj;
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	240 HeapRegion* hr = _g1h->heap_region_containing_raw(addr);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	241 return mark_and_count(obj, hr);
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	242 }
d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	243
4827 2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	244 inline bool CMBitMapRO::iterate(BitMapClosure* cl, MemRegion mr) {
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	245 HeapWord* start_addr = MAX2(startWord(), mr.start());
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	246 HeapWord* end_addr = MIN2(endWord(), mr.end());
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	247
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	248 if (end_addr > start_addr) {
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	249 // Right-open interval [start-offset, end-offset).
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	250 BitMap::idx_t start_offset = heapWordToOffset(start_addr);
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	251 BitMap::idx_t end_offset = heapWordToOffset(end_addr);
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	252
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	253 start_offset = _bm.get_next_one_offset(start_offset, end_offset);
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	254 while (start_offset < end_offset) {
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	255 if (!cl->do_bit(start_offset)) {
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	256 return false;
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	257 }
8733 9def4075da6d 8008079: G1: Add nextObject routine to CMBitMapRO and replace nextWord tamao parents: 6862 diff changeset	258 HeapWord* next_addr = MIN2(nextObject(offsetToHeapWord(start_offset)), end_addr);
4827 2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	259 BitMap::idx_t next_offset = heapWordToOffset(next_addr);
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	260 start_offset = _bm.get_next_one_offset(next_offset, end_offset);
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	261 }
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	262 }
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	263 return true;
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	264 }
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	265
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	266 inline bool CMBitMapRO::iterate(BitMapClosure* cl) {
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	267 MemRegion mr(startWord(), sizeInWords());
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	268 return iterate(cl, mr);
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	269 }
2e966d967c5c 7121547: G1: High number mispredicted branches while iterating over the marking bitmap johnc parents: 4787 diff changeset	270
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	271 inline void CMTask::push(oop obj) {
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	272 HeapWord* objAddr = (HeapWord*) obj;
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	273 assert(_g1h->is_in_g1_reserved(objAddr), "invariant");
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	274 assert(!_g1h->is_on_master_free_list(
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	275 _g1h->heap_region_containing((HeapWord*) objAddr)), "invariant");
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	276 assert(!_g1h->is_obj_ill(obj), "invariant");
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	277 assert(_nextMarkBitMap->isMarked(objAddr), "invariant");
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	278
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	279 if (_cm->verbose_high()) {
6862 8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	280 gclog_or_tty->print_cr("[%u] pushing "PTR_FORMAT, _worker_id, (void*) obj);
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	281 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	282
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	283 if (!_task_queue->push(obj)) {
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	284 // The local task queue looks full. We need to push some entries
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	285 // to the global stack.
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	286
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	287 if (_cm->verbose_medium()) {
6862 8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	288 gclog_or_tty->print_cr("[%u] task queue overflow, "
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	289 "moving entries to the global stack",
6862 8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	290 _worker_id);
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	291 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	292 move_entries_to_global_stack();
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	293
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	294 // this should succeed since, even if we overflow the global
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	295 // stack, we should have definitely removed some entries from the
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	296 // local queue. So, there must be space on it.
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	297 bool success = _task_queue->push(obj);
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	298 assert(success, "invariant");
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	299 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	300
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	301 statsOnly( int tmp_size = _task_queue->size();
3776 23d434c6290d 7055073: G1: code cleanup in the concurrentMark.* files tonyp parents: 3771 diff changeset	302 if (tmp_size > _local_max_size) {
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	303 _local_max_size = tmp_size;
3776 23d434c6290d 7055073: G1: code cleanup in the concurrentMark.* files tonyp parents: 3771 diff changeset	304 }
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	305 ++_local_pushes );
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	306 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	307
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	308 // This determines whether the method below will check both the local
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	309 // and global fingers when determining whether to push on the stack a
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	310 // gray object (value 1) or whether it will only check the global one
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	311 // (value 0). The tradeoffs are that the former will be a bit more
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	312 // accurate and possibly push less on the stack, but it might also be
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	313 // a little bit slower.
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	314
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	315 #define _CHECK_BOTH_FINGERS_ 1
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	316
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	317 inline void CMTask::deal_with_reference(oop obj) {
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	318 if (_cm->verbose_high()) {
6862 8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	319 gclog_or_tty->print_cr("[%u] we're dealing with reference = "PTR_FORMAT,
8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	320 _worker_id, (void*) obj);
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	321 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	322
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	323 ++_refs_reached;
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	324
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	325 HeapWord* objAddr = (HeapWord*) obj;
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	326 assert(obj->is_oop_or_null(true /* ignore mark word */), "Error");
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	327 if (_g1h->is_in_g1_reserved(objAddr)) {
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	328 assert(obj != NULL, "null check is implicit");
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	329 if (!_nextMarkBitMap->isMarked(objAddr)) {
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	330 // Only get the containing region if the object is not marked on the
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	331 // bitmap (otherwise, it's a waste of time since we won't do
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	332 // anything with it).
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	333 HeapRegion* hr = _g1h->heap_region_containing_raw(obj);
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	334 if (!hr->obj_allocated_since_next_marking(obj)) {
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	335 if (_cm->verbose_high()) {
6862 8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	336 gclog_or_tty->print_cr("[%u] "PTR_FORMAT" is not considered marked",
8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	337 _worker_id, (void*) obj);
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	338 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	339
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	340 // we need to mark it first
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	341 if (_cm->par_mark_and_count(obj, hr, _marked_bytes_array, _card_bm)) {
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	342 // No OrderAccess:store_load() is needed. It is implicit in the
4836 d30fa85f9994 6484965: G1: piggy-back liveness accounting phase on marking johnc parents: 4827 diff changeset	343 // CAS done in CMBitMap::parMark() call in the routine above.
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	344 HeapWord* global_finger = _cm->finger();
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	345
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	346 #if _CHECK_BOTH_FINGERS_
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	347 // we will check both the local and global fingers
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	348
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	349 if (_finger != NULL && objAddr < _finger) {
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	350 if (_cm->verbose_high()) {
6862 8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	351 gclog_or_tty->print_cr("[%u] below the local finger ("PTR_FORMAT"), "
8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	352 "pushing it", _worker_id, _finger);
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	353 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	354 push(obj);
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	355 } else if (_curr_region != NULL && objAddr < _region_limit) {
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	356 // do nothing
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	357 } else if (objAddr < global_finger) {
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	358 // Notice that the global finger might be moving forward
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	359 // concurrently. This is not a problem. In the worst case, we
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	360 // mark the object while it is above the global finger and, by
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	361 // the time we read the global finger, it has moved forward
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	362 // passed this object. In this case, the object will probably
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	363 // be visited when a task is scanning the region and will also
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	364 // be pushed on the stack. So, some duplicate work, but no
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	365 // correctness problems.
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	366
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	367 if (_cm->verbose_high()) {
6862 8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	368 gclog_or_tty->print_cr("[%u] below the global finger "
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	369 "("PTR_FORMAT"), pushing it",
6862 8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	370 _worker_id, global_finger);
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	371 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	372 push(obj);
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	373 } else {
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	374 // do nothing
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	375 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	376 #else // _CHECK_BOTH_FINGERS_
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	377 // we will only check the global finger
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	378
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	379 if (objAddr < global_finger) {
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	380 // see long comment above
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	381
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	382 if (_cm->verbose_high()) {
6862 8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	383 gclog_or_tty->print_cr("[%u] below the global finger "
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	384 "("PTR_FORMAT"), pushing it",
6862 8a5ea0a9ccc4 7127708: G1: change task num types from int to uint in concurrent mark johnc parents: 6812 diff changeset	385 _worker_id, global_finger);
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	386 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	387 push(obj);
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	388 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	389 #endif // _CHECK_BOTH_FINGERS_
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	390 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	391 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	392 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	393 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	394 }
842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	395
4787 2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	396 inline void ConcurrentMark::markPrev(oop p) {
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	397 assert(!_prevMarkBitMap->isMarked((HeapWord*) p), "sanity");
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	398 // Note we are overriding the read-only view of the prev map here, via
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	399 // the cast.
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	400 ((CMBitMap)_prevMarkBitMap)->mark((HeapWord) p);
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	401 }
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	402
4837 eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	403 inline void ConcurrentMark::grayRoot(oop obj, size_t word_size,
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	404 uint worker_id, HeapRegion* hr) {
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	405 assert(obj != NULL, "pre-condition");
4787 2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	406 HeapWord* addr = (HeapWord*) obj;
4837 eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	407 if (hr == NULL) {
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	408 hr = _g1h->heap_region_containing_raw(addr);
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	409 } else {
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	410 assert(hr->is_in(addr), "pre-condition");
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	411 }
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	412 assert(hr != NULL, "sanity");
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	413 // Given that we're looking for a region that contains an object
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	414 // header it's impossible to get back a HC region.
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	415 assert(!hr->continuesHumongous(), "sanity");
4787 2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	416
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	417 // We cannot assert that word_size == obj->size() given that obj
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	418 // might not be in a consistent state (another thread might be in
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	419 // the process of copying it). So the best thing we can do is to
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	420 // assert that word_size is under an upper bound which is its
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	421 // containing region's capacity.
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	422 assert(word_size * HeapWordSize <= hr->capacity(),
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	423 err_msg("size: "SIZE_FORMAT" capacity: "SIZE_FORMAT" "HR_FORMAT,
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	424 word_size * HeapWordSize, hr->capacity(),
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	425 HR_FORMAT_PARAMS(hr)));
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	426
4837 eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	427 if (addr < hr->next_top_at_mark_start()) {
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	428 if (!_nextMarkBitMap->isMarked(addr)) {
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	429 par_mark_and_count(obj, word_size, hr, worker_id);
eff609af17d7 7127706: G1: re-enable survivors during the initial-mark pause tonyp parents: 4836 diff changeset	430 }
4787 2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	431 }
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	432 }
2ace1c4ee8da 6888336: G1: avoid explicitly marking and pushing objects in survivor spaces tonyp parents: 3776 diff changeset	433
3771 842b840e67db 7046558: G1: concurrent marking optimizations tonyp parents: diff changeset	434 #endif // SHARE_VM_GC_IMPLEMENTATION_G1_CONCURRENTMARK_INLINE_HPP

Mercurial > hg > graal-compiler

annotate src/share/vm/gc_implementation/g1/concurrentMark.inline.hpp @ 8733:9def4075da6d