Mercurial > hg > graal-compiler
annotate src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp @ 452:00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
Summary: When we encounter marking stack overflow during precleaning of Reference lists, we were using the overflow list mechanism, which can cause problems on account of mutating the mark word of the header because of conflicts with mutator accesses and updates of that field. Instead we should use the usual mechanism for overflow handling in concurrent phases, namely dirtying of the card on which the overflowed object lies. Since precleaning effectively does a form of discovered list processing, albeit with discovery enabled, we needed to adjust some code to be correct in the face of interleaved processing and discovery.
Reviewed-by: apetrusenko, jcoomes
| author | ysr |
|---|---|
| date | Thu, 20 Nov 2008 12:27:41 -0800 |
| parents | 5d254928c888 |
| children | c96030fff130 |
| rev | line source |
|---|---|
| 0 | 1 /* |
| 196 | 2 * Copyright 2001-2008 Sun Microsystems, Inc. All Rights Reserved. |
| 0 | 3 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. |
| 4 * | |
| 5 * This code is free software; you can redistribute it and/or modify it | |
| 6 * under the terms of the GNU General Public License version 2 only, as | |
| 7 * published by the Free Software Foundation. | |
| 8 * | |
| 9 * This code is distributed in the hope that it will be useful, but WITHOUT | |
| 10 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | |
| 11 * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License | |
| 12 * version 2 for more details (a copy is included in the LICENSE file that | |
| 13 * accompanied this code). | |
| 14 * | |
| 15 * You should have received a copy of the GNU General Public License version | |
| 16 * 2 along with this work; if not, write to the Free Software Foundation, | |
| 17 * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. | |
| 18 * | |
| 19 * Please contact Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, | |
| 20 * CA 95054 USA or visit www.sun.com if you need additional information or | |
| 21 * have any questions. | |
| 22 * | |
| 23 */ | |
| 24 | |
| 25 # include "incls/_precompiled.incl" | |
| 26 # include "incls/_concurrentMarkSweepGeneration.cpp.incl" | |
| 27 | |
| 28 // statics | |
| 29 CMSCollector* ConcurrentMarkSweepGeneration::_collector = NULL; | |
| 30 bool CMSCollector::_full_gc_requested = false; | |
| 31 | |
| 32 ////////////////////////////////////////////////////////////////// | |
| 33 // In support of CMS/VM thread synchronization | |
| 34 ////////////////////////////////////////////////////////////////// | |
| 35 // We split use of the CGC_lock into 2 "levels". | |
| 36 // The low-level locking is of the usual CGC_lock monitor. We introduce | |
| 37 // a higher level "token" (hereafter "CMS token") built on top of the | |
| 38 // low level monitor (hereafter "CGC lock"). | |
| 39 // The token-passing protocol gives priority to the VM thread. The | |
| 40 // CMS-lock doesn't provide any fairness guarantees, but clients | |
| 41 // should ensure that it is only held for very short, bounded | |
| 42 // durations. | |
| 43 // | |
| 44 // When either of the CMS thread or the VM thread is involved in | |
| 45 // collection operations during which it does not want the other | |
| 46 // thread to interfere, it obtains the CMS token. | |
| 47 // | |
| 48 // If either thread tries to get the token while the other has | |
| 49 // it, that thread waits. However, if the VM thread and CMS thread | |
| 50 // both want the token, then the VM thread gets priority while the | |
| 51 // CMS thread waits. This ensures, for instance, that the "concurrent" | |
| 52 // phases of the CMS thread's work do not block out the VM thread | |
| 53 // for long periods of time as the CMS thread continues to hog | |
| 54 // the token. (See bug 4616232). | |
| 55 // | |
| 56 // The baton-passing functions are, however, controlled by the | |
| 57 // flags _foregroundGCShouldWait and _foregroundGCIsActive, | |
| 58 // and here the low-level CMS lock, not the high level token, | |
| 59 // ensures mutual exclusion. | |
| 60 // | |
| 61 // Two important conditions that we have to satisfy: | |
| 62 // 1. if a thread does a low-level wait on the CMS lock, then it | |
| 63 // relinquishes the CMS token if it were holding that token | |
| 64 // when it acquired the low-level CMS lock. | |
| 65 // 2. any low-level notifications on the low-level lock | |
| 66 // should only be sent when a thread has relinquished the token. | |
| 67 // | |
| 68 // In the absence of either property, we'd have potential deadlock. | |
| 69 // | |
| 70 // We protect each of the CMS (concurrent and sequential) phases | |
| 71 // with the CMS _token_, not the CMS _lock_. | |
| 72 // | |
| 73 // The only code protected by CMS lock is the token acquisition code | |
| 74 // itself, see ConcurrentMarkSweepThread::[de]synchronize(), and the | |
| 75 // baton-passing code. | |
| 76 // | |
| 77 // Unfortunately, i couldn't come up with a good abstraction to factor and | |
| 78 // hide the naked CGC_lock manipulation in the baton-passing code | |
| 79 // further below. That's something we should try to do. Also, the proof | |
| 80 // of correctness of this 2-level locking scheme is far from obvious, | |
| 81 // and potentially quite slippery. We have an uneasy supsicion, for instance, | |
| 82 // that there may be a theoretical possibility of delay/starvation in the | |
| 83 // low-level lock/wait/notify scheme used for the baton-passing because of | |
| 84 // potential intereference with the priority scheme embodied in the | |
| 85 // CMS-token-passing protocol. See related comments at a CGC_lock->wait() | |
| 86 // invocation further below and marked with "XXX 20011219YSR". | |
| 87 // Indeed, as we note elsewhere, this may become yet more slippery | |
| 88 // in the presence of multiple CMS and/or multiple VM threads. XXX | |
| 89 | |
| 90 class CMSTokenSync: public StackObj { | |
| 91 private: | |
| 92 bool _is_cms_thread; | |
| 93 public: | |
| 94 CMSTokenSync(bool is_cms_thread): | |
| 95 _is_cms_thread(is_cms_thread) { | |
| 96 assert(is_cms_thread == Thread::current()->is_ConcurrentGC_thread(), | |
| 97 "Incorrect argument to constructor"); | |
| 98 ConcurrentMarkSweepThread::synchronize(_is_cms_thread); | |
| 99 } | |
| 100 | |
| 101 ~CMSTokenSync() { | |
| 102 assert(_is_cms_thread ? | |
| 103 ConcurrentMarkSweepThread::cms_thread_has_cms_token() : | |
| 104 ConcurrentMarkSweepThread::vm_thread_has_cms_token(), | |
| 105 "Incorrect state"); | |
| 106 ConcurrentMarkSweepThread::desynchronize(_is_cms_thread); | |
| 107 } | |
| 108 }; | |
| 109 | |
| 110 // Convenience class that does a CMSTokenSync, and then acquires | |
| 111 // upto three locks. | |
| 112 class CMSTokenSyncWithLocks: public CMSTokenSync { | |
| 113 private: | |
| 114 // Note: locks are acquired in textual declaration order | |
| 115 // and released in the opposite order | |
| 116 MutexLockerEx _locker1, _locker2, _locker3; | |
| 117 public: | |
| 118 CMSTokenSyncWithLocks(bool is_cms_thread, Mutex* mutex1, | |
| 119 Mutex* mutex2 = NULL, Mutex* mutex3 = NULL): | |
| 120 CMSTokenSync(is_cms_thread), | |
| 121 _locker1(mutex1, Mutex::_no_safepoint_check_flag), | |
| 122 _locker2(mutex2, Mutex::_no_safepoint_check_flag), | |
| 123 _locker3(mutex3, Mutex::_no_safepoint_check_flag) | |
| 124 { } | |
| 125 }; | |
| 126 | |
| 127 | |
| 128 // Wrapper class to temporarily disable icms during a foreground cms collection. | |
| 129 class ICMSDisabler: public StackObj { | |
| 130 public: | |
| 131 // The ctor disables icms and wakes up the thread so it notices the change; | |
| 132 // the dtor re-enables icms. Note that the CMSCollector methods will check | |
| 133 // CMSIncrementalMode. | |
| 134 ICMSDisabler() { CMSCollector::disable_icms(); CMSCollector::start_icms(); } | |
| 135 ~ICMSDisabler() { CMSCollector::enable_icms(); } | |
| 136 }; | |
| 137 | |
| 138 ////////////////////////////////////////////////////////////////// | |
| 139 // Concurrent Mark-Sweep Generation ///////////////////////////// | |
| 140 ////////////////////////////////////////////////////////////////// | |
| 141 | |
| 142 NOT_PRODUCT(CompactibleFreeListSpace* debug_cms_space;) | |
| 143 | |
| 144 // This struct contains per-thread things necessary to support parallel | |
| 145 // young-gen collection. | |
| 146 class CMSParGCThreadState: public CHeapObj { | |
| 147 public: | |
| 148 CFLS_LAB lab; | |
| 149 PromotionInfo promo; | |
| 150 | |
| 151 // Constructor. | |
| 152 CMSParGCThreadState(CompactibleFreeListSpace* cfls) : lab(cfls) { | |
| 153 promo.setSpace(cfls); | |
| 154 } | |
| 155 }; | |
| 156 | |
| 157 ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration( | |
| 158 ReservedSpace rs, size_t initial_byte_size, int level, | |
| 159 CardTableRS* ct, bool use_adaptive_freelists, | |
| 160 FreeBlockDictionary::DictionaryChoice dictionaryChoice) : | |
| 161 CardGeneration(rs, initial_byte_size, level, ct), | |
| 162 _dilatation_factor(((double)MinChunkSize)/((double)(oopDesc::header_size()))), | |
| 163 _debug_collection_type(Concurrent_collection_type) | |
| 164 { | |
| 165 HeapWord* bottom = (HeapWord*) _virtual_space.low(); | |
| 166 HeapWord* end = (HeapWord*) _virtual_space.high(); | |
| 167 | |
| 168 _direct_allocated_words = 0; | |
| 169 NOT_PRODUCT( | |
| 170 _numObjectsPromoted = 0; | |
| 171 _numWordsPromoted = 0; | |
| 172 _numObjectsAllocated = 0; | |
| 173 _numWordsAllocated = 0; | |
| 174 ) | |
| 175 | |
| 176 _cmsSpace = new CompactibleFreeListSpace(_bts, MemRegion(bottom, end), | |
| 177 use_adaptive_freelists, | |
| 178 dictionaryChoice); | |
| 179 NOT_PRODUCT(debug_cms_space = _cmsSpace;) | |
| 180 if (_cmsSpace == NULL) { | |
| 181 vm_exit_during_initialization( | |
| 182 "CompactibleFreeListSpace allocation failure"); | |
| 183 } | |
| 184 _cmsSpace->_gen = this; | |
| 185 | |
| 186 _gc_stats = new CMSGCStats(); | |
| 187 | |
| 188 // Verify the assumption that FreeChunk::_prev and OopDesc::_klass | |
| 189 // offsets match. The ability to tell free chunks from objects | |
| 190 // depends on this property. | |
| 191 debug_only( | |
| 192 FreeChunk* junk = NULL; | |
| 187 | 193 assert(UseCompressedOops || |
| 194 junk->prev_addr() == (void*)(oop(junk)->klass_addr()), | |
| 0 | 195 "Offset of FreeChunk::_prev within FreeChunk must match" |
| 196 " that of OopDesc::_klass within OopDesc"); | |
| 197 ) | |
| 198 if (ParallelGCThreads > 0) { | |
| 199 typedef CMSParGCThreadState* CMSParGCThreadStatePtr; | |
| 200 _par_gc_thread_states = | |
| 201 NEW_C_HEAP_ARRAY(CMSParGCThreadStatePtr, ParallelGCThreads); | |
| 202 if (_par_gc_thread_states == NULL) { | |
| 203 vm_exit_during_initialization("Could not allocate par gc structs"); | |
| 204 } | |
| 205 for (uint i = 0; i < ParallelGCThreads; i++) { | |
| 206 _par_gc_thread_states[i] = new CMSParGCThreadState(cmsSpace()); | |
| 207 if (_par_gc_thread_states[i] == NULL) { | |
| 208 vm_exit_during_initialization("Could not allocate par gc structs"); | |
| 209 } | |
| 210 } | |
| 211 } else { | |
| 212 _par_gc_thread_states = NULL; | |
| 213 } | |
| 214 _incremental_collection_failed = false; | |
| 215 // The "dilatation_factor" is the expansion that can occur on | |
| 216 // account of the fact that the minimum object size in the CMS | |
| 217 // generation may be larger than that in, say, a contiguous young | |
| 218 // generation. | |
| 219 // Ideally, in the calculation below, we'd compute the dilatation | |
| 220 // factor as: MinChunkSize/(promoting_gen's min object size) | |
| 221 // Since we do not have such a general query interface for the | |
| 222 // promoting generation, we'll instead just use the mimimum | |
| 223 // object size (which today is a header's worth of space); | |
| 224 // note that all arithmetic is in units of HeapWords. | |
| 225 assert(MinChunkSize >= oopDesc::header_size(), "just checking"); | |
| 226 assert(_dilatation_factor >= 1.0, "from previous assert"); | |
| 227 } | |
| 228 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
229 |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
230 // The field "_initiating_occupancy" represents the occupancy percentage |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
231 // at which we trigger a new collection cycle. Unless explicitly specified |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
232 // via CMSInitiating[Perm]OccupancyFraction (argument "io" below), it |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
233 // is calculated by: |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
234 // |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
235 // Let "f" be MinHeapFreeRatio in |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
236 // |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
237 // _intiating_occupancy = 100-f + |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
238 // f * (CMSTrigger[Perm]Ratio/100) |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
239 // where CMSTrigger[Perm]Ratio is the argument "tr" below. |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
240 // |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
241 // That is, if we assume the heap is at its desired maximum occupancy at the |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
242 // end of a collection, we let CMSTrigger[Perm]Ratio of the (purported) free |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
243 // space be allocated before initiating a new collection cycle. |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
244 // |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
245 void ConcurrentMarkSweepGeneration::init_initiating_occupancy(intx io, intx tr) { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
246 assert(io <= 100 && tr >= 0 && tr <= 100, "Check the arguments"); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
247 if (io >= 0) { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
248 _initiating_occupancy = (double)io / 100.0; |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
249 } else { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
250 _initiating_occupancy = ((100 - MinHeapFreeRatio) + |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
251 (double)(tr * MinHeapFreeRatio) / 100.0) |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
252 / 100.0; |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
253 } |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
254 } |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
255 |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
256 |
| 0 | 257 void ConcurrentMarkSweepGeneration::ref_processor_init() { |
| 258 assert(collector() != NULL, "no collector"); | |
| 259 collector()->ref_processor_init(); | |
| 260 } | |
| 261 | |
| 262 void CMSCollector::ref_processor_init() { | |
| 263 if (_ref_processor == NULL) { | |
| 264 // Allocate and initialize a reference processor | |
| 265 _ref_processor = ReferenceProcessor::create_ref_processor( | |
| 266 _span, // span | |
| 267 _cmsGen->refs_discovery_is_atomic(), // atomic_discovery | |
| 268 _cmsGen->refs_discovery_is_mt(), // mt_discovery | |
| 269 &_is_alive_closure, | |
| 270 ParallelGCThreads, | |
| 271 ParallelRefProcEnabled); | |
| 272 // Initialize the _ref_processor field of CMSGen | |
| 273 _cmsGen->set_ref_processor(_ref_processor); | |
| 274 | |
| 275 // Allocate a dummy ref processor for perm gen. | |
| 276 ReferenceProcessor* rp2 = new ReferenceProcessor(); | |
| 277 if (rp2 == NULL) { | |
| 278 vm_exit_during_initialization("Could not allocate ReferenceProcessor object"); | |
| 279 } | |
| 280 _permGen->set_ref_processor(rp2); | |
| 281 } | |
| 282 } | |
| 283 | |
| 284 CMSAdaptiveSizePolicy* CMSCollector::size_policy() { | |
| 285 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 286 assert(gch->kind() == CollectedHeap::GenCollectedHeap, | |
| 287 "Wrong type of heap"); | |
| 288 CMSAdaptiveSizePolicy* sp = (CMSAdaptiveSizePolicy*) | |
| 289 gch->gen_policy()->size_policy(); | |
| 290 assert(sp->is_gc_cms_adaptive_size_policy(), | |
| 291 "Wrong type of size policy"); | |
| 292 return sp; | |
| 293 } | |
| 294 | |
| 295 CMSGCAdaptivePolicyCounters* CMSCollector::gc_adaptive_policy_counters() { | |
| 296 CMSGCAdaptivePolicyCounters* results = | |
| 297 (CMSGCAdaptivePolicyCounters*) collector_policy()->counters(); | |
| 298 assert( | |
| 299 results->kind() == GCPolicyCounters::CMSGCAdaptivePolicyCountersKind, | |
| 300 "Wrong gc policy counter kind"); | |
| 301 return results; | |
| 302 } | |
| 303 | |
| 304 | |
| 305 void ConcurrentMarkSweepGeneration::initialize_performance_counters() { | |
| 306 | |
| 307 const char* gen_name = "old"; | |
| 308 | |
| 309 // Generation Counters - generation 1, 1 subspace | |
| 310 _gen_counters = new GenerationCounters(gen_name, 1, 1, &_virtual_space); | |
| 311 | |
| 312 _space_counters = new GSpaceCounters(gen_name, 0, | |
| 313 _virtual_space.reserved_size(), | |
| 314 this, _gen_counters); | |
| 315 } | |
| 316 | |
| 317 CMSStats::CMSStats(ConcurrentMarkSweepGeneration* cms_gen, unsigned int alpha): | |
| 318 _cms_gen(cms_gen) | |
| 319 { | |
| 320 assert(alpha <= 100, "bad value"); | |
| 321 _saved_alpha = alpha; | |
| 322 | |
| 323 // Initialize the alphas to the bootstrap value of 100. | |
| 324 _gc0_alpha = _cms_alpha = 100; | |
| 325 | |
| 326 _cms_begin_time.update(); | |
| 327 _cms_end_time.update(); | |
| 328 | |
| 329 _gc0_duration = 0.0; | |
| 330 _gc0_period = 0.0; | |
| 331 _gc0_promoted = 0; | |
| 332 | |
| 333 _cms_duration = 0.0; | |
| 334 _cms_period = 0.0; | |
| 335 _cms_allocated = 0; | |
| 336 | |
| 337 _cms_used_at_gc0_begin = 0; | |
| 338 _cms_used_at_gc0_end = 0; | |
| 339 _allow_duty_cycle_reduction = false; | |
| 340 _valid_bits = 0; | |
| 341 _icms_duty_cycle = CMSIncrementalDutyCycle; | |
| 342 } | |
| 343 | |
| 344 // If promotion failure handling is on use | |
| 345 // the padded average size of the promotion for each | |
| 346 // young generation collection. | |
| 347 double CMSStats::time_until_cms_gen_full() const { | |
| 348 size_t cms_free = _cms_gen->cmsSpace()->free(); | |
| 349 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 350 size_t expected_promotion = gch->get_gen(0)->capacity(); | |
| 351 if (HandlePromotionFailure) { | |
| 352 expected_promotion = MIN2( | |
| 353 (size_t) _cms_gen->gc_stats()->avg_promoted()->padded_average(), | |
| 354 expected_promotion); | |
| 355 } | |
| 356 if (cms_free > expected_promotion) { | |
| 357 // Start a cms collection if there isn't enough space to promote | |
| 358 // for the next minor collection. Use the padded average as | |
| 359 // a safety factor. | |
| 360 cms_free -= expected_promotion; | |
| 361 | |
| 362 // Adjust by the safety factor. | |
| 363 double cms_free_dbl = (double)cms_free; | |
| 364 cms_free_dbl = cms_free_dbl * (100.0 - CMSIncrementalSafetyFactor) / 100.0; | |
| 365 | |
| 366 if (PrintGCDetails && Verbose) { | |
| 367 gclog_or_tty->print_cr("CMSStats::time_until_cms_gen_full: cms_free " | |
| 368 SIZE_FORMAT " expected_promotion " SIZE_FORMAT, | |
| 369 cms_free, expected_promotion); | |
| 370 gclog_or_tty->print_cr(" cms_free_dbl %f cms_consumption_rate %f", | |
| 371 cms_free_dbl, cms_consumption_rate() + 1.0); | |
| 372 } | |
| 373 // Add 1 in case the consumption rate goes to zero. | |
| 374 return cms_free_dbl / (cms_consumption_rate() + 1.0); | |
| 375 } | |
| 376 return 0.0; | |
| 377 } | |
| 378 | |
| 379 // Compare the duration of the cms collection to the | |
| 380 // time remaining before the cms generation is empty. | |
| 381 // Note that the time from the start of the cms collection | |
| 382 // to the start of the cms sweep (less than the total | |
| 383 // duration of the cms collection) can be used. This | |
| 384 // has been tried and some applications experienced | |
| 385 // promotion failures early in execution. This was | |
| 386 // possibly because the averages were not accurate | |
| 387 // enough at the beginning. | |
| 388 double CMSStats::time_until_cms_start() const { | |
| 389 // We add "gc0_period" to the "work" calculation | |
| 390 // below because this query is done (mostly) at the | |
| 391 // end of a scavenge, so we need to conservatively | |
| 392 // account for that much possible delay | |
| 393 // in the query so as to avoid concurrent mode failures | |
| 394 // due to starting the collection just a wee bit too | |
| 395 // late. | |
| 396 double work = cms_duration() + gc0_period(); | |
| 397 double deadline = time_until_cms_gen_full(); | |
| 398 if (work > deadline) { | |
| 399 if (Verbose && PrintGCDetails) { | |
| 400 gclog_or_tty->print( | |
| 401 " CMSCollector: collect because of anticipated promotion " | |
| 402 "before full %3.7f + %3.7f > %3.7f ", cms_duration(), | |
| 403 gc0_period(), time_until_cms_gen_full()); | |
| 404 } | |
| 405 return 0.0; | |
| 406 } | |
| 407 return work - deadline; | |
| 408 } | |
| 409 | |
| 410 // Return a duty cycle based on old_duty_cycle and new_duty_cycle, limiting the | |
| 411 // amount of change to prevent wild oscillation. | |
| 412 unsigned int CMSStats::icms_damped_duty_cycle(unsigned int old_duty_cycle, | |
| 413 unsigned int new_duty_cycle) { | |
| 414 assert(old_duty_cycle <= 100, "bad input value"); | |
| 415 assert(new_duty_cycle <= 100, "bad input value"); | |
| 416 | |
| 417 // Note: use subtraction with caution since it may underflow (values are | |
| 418 // unsigned). Addition is safe since we're in the range 0-100. | |
| 419 unsigned int damped_duty_cycle = new_duty_cycle; | |
| 420 if (new_duty_cycle < old_duty_cycle) { | |
| 421 const unsigned int largest_delta = MAX2(old_duty_cycle / 4, 5U); | |
| 422 if (new_duty_cycle + largest_delta < old_duty_cycle) { | |
| 423 damped_duty_cycle = old_duty_cycle - largest_delta; | |
| 424 } | |
| 425 } else if (new_duty_cycle > old_duty_cycle) { | |
| 426 const unsigned int largest_delta = MAX2(old_duty_cycle / 4, 15U); | |
| 427 if (new_duty_cycle > old_duty_cycle + largest_delta) { | |
| 428 damped_duty_cycle = MIN2(old_duty_cycle + largest_delta, 100U); | |
| 429 } | |
| 430 } | |
| 431 assert(damped_duty_cycle <= 100, "invalid duty cycle computed"); | |
| 432 | |
| 433 if (CMSTraceIncrementalPacing) { | |
| 434 gclog_or_tty->print(" [icms_damped_duty_cycle(%d,%d) = %d] ", | |
| 435 old_duty_cycle, new_duty_cycle, damped_duty_cycle); | |
| 436 } | |
| 437 return damped_duty_cycle; | |
| 438 } | |
| 439 | |
| 440 unsigned int CMSStats::icms_update_duty_cycle_impl() { | |
| 441 assert(CMSIncrementalPacing && valid(), | |
| 442 "should be handled in icms_update_duty_cycle()"); | |
| 443 | |
| 444 double cms_time_so_far = cms_timer().seconds(); | |
| 445 double scaled_duration = cms_duration_per_mb() * _cms_used_at_gc0_end / M; | |
| 446 double scaled_duration_remaining = fabsd(scaled_duration - cms_time_so_far); | |
| 447 | |
| 448 // Avoid division by 0. | |
| 449 double time_until_full = MAX2(time_until_cms_gen_full(), 0.01); | |
| 450 double duty_cycle_dbl = 100.0 * scaled_duration_remaining / time_until_full; | |
| 451 | |
| 452 unsigned int new_duty_cycle = MIN2((unsigned int)duty_cycle_dbl, 100U); | |
| 453 if (new_duty_cycle > _icms_duty_cycle) { | |
| 454 // Avoid very small duty cycles (1 or 2); 0 is allowed. | |
| 455 if (new_duty_cycle > 2) { | |
| 456 _icms_duty_cycle = icms_damped_duty_cycle(_icms_duty_cycle, | |
| 457 new_duty_cycle); | |
| 458 } | |
| 459 } else if (_allow_duty_cycle_reduction) { | |
| 460 // The duty cycle is reduced only once per cms cycle (see record_cms_end()). | |
| 461 new_duty_cycle = icms_damped_duty_cycle(_icms_duty_cycle, new_duty_cycle); | |
| 462 // Respect the minimum duty cycle. | |
| 463 unsigned int min_duty_cycle = (unsigned int)CMSIncrementalDutyCycleMin; | |
| 464 _icms_duty_cycle = MAX2(new_duty_cycle, min_duty_cycle); | |
| 465 } | |
| 466 | |
| 467 if (PrintGCDetails || CMSTraceIncrementalPacing) { | |
| 468 gclog_or_tty->print(" icms_dc=%d ", _icms_duty_cycle); | |
| 469 } | |
| 470 | |
| 471 _allow_duty_cycle_reduction = false; | |
| 472 return _icms_duty_cycle; | |
| 473 } | |
| 474 | |
| 475 #ifndef PRODUCT | |
| 476 void CMSStats::print_on(outputStream *st) const { | |
| 477 st->print(" gc0_alpha=%d,cms_alpha=%d", _gc0_alpha, _cms_alpha); | |
| 478 st->print(",gc0_dur=%g,gc0_per=%g,gc0_promo=" SIZE_FORMAT, | |
| 479 gc0_duration(), gc0_period(), gc0_promoted()); | |
| 480 st->print(",cms_dur=%g,cms_dur_per_mb=%g,cms_per=%g,cms_alloc=" SIZE_FORMAT, | |
| 481 cms_duration(), cms_duration_per_mb(), | |
| 482 cms_period(), cms_allocated()); | |
| 483 st->print(",cms_since_beg=%g,cms_since_end=%g", | |
| 484 cms_time_since_begin(), cms_time_since_end()); | |
| 485 st->print(",cms_used_beg=" SIZE_FORMAT ",cms_used_end=" SIZE_FORMAT, | |
| 486 _cms_used_at_gc0_begin, _cms_used_at_gc0_end); | |
| 487 if (CMSIncrementalMode) { | |
| 488 st->print(",dc=%d", icms_duty_cycle()); | |
| 489 } | |
| 490 | |
| 491 if (valid()) { | |
| 492 st->print(",promo_rate=%g,cms_alloc_rate=%g", | |
| 493 promotion_rate(), cms_allocation_rate()); | |
| 494 st->print(",cms_consumption_rate=%g,time_until_full=%g", | |
| 495 cms_consumption_rate(), time_until_cms_gen_full()); | |
| 496 } | |
| 497 st->print(" "); | |
| 498 } | |
| 499 #endif // #ifndef PRODUCT | |
| 500 | |
| 501 CMSCollector::CollectorState CMSCollector::_collectorState = | |
| 502 CMSCollector::Idling; | |
| 503 bool CMSCollector::_foregroundGCIsActive = false; | |
| 504 bool CMSCollector::_foregroundGCShouldWait = false; | |
| 505 | |
| 506 CMSCollector::CMSCollector(ConcurrentMarkSweepGeneration* cmsGen, | |
| 507 ConcurrentMarkSweepGeneration* permGen, | |
| 508 CardTableRS* ct, | |
| 509 ConcurrentMarkSweepPolicy* cp): | |
| 510 _cmsGen(cmsGen), | |
| 511 _permGen(permGen), | |
| 512 _ct(ct), | |
| 513 _ref_processor(NULL), // will be set later | |
| 514 _conc_workers(NULL), // may be set later | |
| 515 _abort_preclean(false), | |
| 516 _start_sampling(false), | |
| 517 _between_prologue_and_epilogue(false), | |
| 518 _markBitMap(0, Mutex::leaf + 1, "CMS_markBitMap_lock"), | |
| 519 _perm_gen_verify_bit_map(0, -1 /* no mutex */, "No_lock"), | |
| 520 _modUnionTable((CardTableModRefBS::card_shift - LogHeapWordSize), | |
| 521 -1 /* lock-free */, "No_lock" /* dummy */), | |
| 522 _modUnionClosure(&_modUnionTable), | |
| 523 _modUnionClosurePar(&_modUnionTable), | |
|
143
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
524 // Adjust my span to cover old (cms) gen and perm gen |
|
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
525 _span(cmsGen->reserved()._union(permGen->reserved())), |
|
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
526 // Construct the is_alive_closure with _span & markBitMap |
|
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
527 _is_alive_closure(_span, &_markBitMap), |
| 0 | 528 _restart_addr(NULL), |
| 529 _overflow_list(NULL), | |
| 530 _preserved_oop_stack(NULL), | |
| 531 _preserved_mark_stack(NULL), | |
| 532 _stats(cmsGen), | |
| 533 _eden_chunk_array(NULL), // may be set in ctor body | |
| 534 _eden_chunk_capacity(0), // -- ditto -- | |
| 535 _eden_chunk_index(0), // -- ditto -- | |
| 536 _survivor_plab_array(NULL), // -- ditto -- | |
| 537 _survivor_chunk_array(NULL), // -- ditto -- | |
| 538 _survivor_chunk_capacity(0), // -- ditto -- | |
| 539 _survivor_chunk_index(0), // -- ditto -- | |
| 540 _ser_pmc_preclean_ovflw(0), | |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
541 _ser_kac_preclean_ovflw(0), |
| 0 | 542 _ser_pmc_remark_ovflw(0), |
| 543 _par_pmc_remark_ovflw(0), | |
| 544 _ser_kac_ovflw(0), | |
| 545 _par_kac_ovflw(0), | |
| 546 #ifndef PRODUCT | |
| 547 _num_par_pushes(0), | |
| 548 #endif | |
| 549 _collection_count_start(0), | |
| 550 _verifying(false), | |
| 551 _icms_start_limit(NULL), | |
| 552 _icms_stop_limit(NULL), | |
| 553 _verification_mark_bm(0, Mutex::leaf + 1, "CMS_verification_mark_bm_lock"), | |
| 554 _completed_initialization(false), | |
| 555 _collector_policy(cp), | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
556 _should_unload_classes(false), |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
557 _concurrent_cycles_since_last_unload(0), |
| 0 | 558 _sweep_estimate(CMS_SweepWeight, CMS_SweepPadding) |
| 559 { | |
| 560 if (ExplicitGCInvokesConcurrentAndUnloadsClasses) { | |
| 561 ExplicitGCInvokesConcurrent = true; | |
| 562 } | |
| 563 // Now expand the span and allocate the collection support structures | |
| 564 // (MUT, marking bit map etc.) to cover both generations subject to | |
| 565 // collection. | |
| 566 | |
| 567 // First check that _permGen is adjacent to _cmsGen and above it. | |
| 568 assert( _cmsGen->reserved().word_size() > 0 | |
| 569 && _permGen->reserved().word_size() > 0, | |
| 570 "generations should not be of zero size"); | |
| 571 assert(_cmsGen->reserved().intersection(_permGen->reserved()).is_empty(), | |
| 572 "_cmsGen and _permGen should not overlap"); | |
| 573 assert(_cmsGen->reserved().end() == _permGen->reserved().start(), | |
| 574 "_cmsGen->end() different from _permGen->start()"); | |
| 575 | |
| 576 // For use by dirty card to oop closures. | |
| 577 _cmsGen->cmsSpace()->set_collector(this); | |
| 578 _permGen->cmsSpace()->set_collector(this); | |
| 579 | |
| 580 // Allocate MUT and marking bit map | |
| 581 { | |
| 582 MutexLockerEx x(_markBitMap.lock(), Mutex::_no_safepoint_check_flag); | |
| 583 if (!_markBitMap.allocate(_span)) { | |
| 584 warning("Failed to allocate CMS Bit Map"); | |
| 585 return; | |
| 586 } | |
| 587 assert(_markBitMap.covers(_span), "_markBitMap inconsistency?"); | |
| 588 } | |
| 589 { | |
| 590 _modUnionTable.allocate(_span); | |
| 591 assert(_modUnionTable.covers(_span), "_modUnionTable inconsistency?"); | |
| 592 } | |
| 593 | |
| 594 if (!_markStack.allocate(CMSMarkStackSize)) { | |
| 595 warning("Failed to allocate CMS Marking Stack"); | |
| 596 return; | |
| 597 } | |
| 598 if (!_revisitStack.allocate(CMSRevisitStackSize)) { | |
| 599 warning("Failed to allocate CMS Revisit Stack"); | |
| 600 return; | |
| 601 } | |
| 602 | |
| 603 // Support for multi-threaded concurrent phases | |
| 604 if (ParallelGCThreads > 0 && CMSConcurrentMTEnabled) { | |
| 605 if (FLAG_IS_DEFAULT(ParallelCMSThreads)) { | |
| 606 // just for now | |
| 607 FLAG_SET_DEFAULT(ParallelCMSThreads, (ParallelGCThreads + 3)/4); | |
| 608 } | |
| 609 if (ParallelCMSThreads > 1) { | |
| 610 _conc_workers = new YieldingFlexibleWorkGang("Parallel CMS Threads", | |
| 611 ParallelCMSThreads, true); | |
| 612 if (_conc_workers == NULL) { | |
| 613 warning("GC/CMS: _conc_workers allocation failure: " | |
| 614 "forcing -CMSConcurrentMTEnabled"); | |
| 615 CMSConcurrentMTEnabled = false; | |
| 616 } | |
| 617 } else { | |
| 618 CMSConcurrentMTEnabled = false; | |
| 619 } | |
| 620 } | |
| 621 if (!CMSConcurrentMTEnabled) { | |
| 622 ParallelCMSThreads = 0; | |
| 623 } else { | |
| 624 // Turn off CMSCleanOnEnter optimization temporarily for | |
| 625 // the MT case where it's not fixed yet; see 6178663. | |
| 626 CMSCleanOnEnter = false; | |
| 627 } | |
| 628 assert((_conc_workers != NULL) == (ParallelCMSThreads > 1), | |
| 629 "Inconsistency"); | |
| 630 | |
| 631 // Parallel task queues; these are shared for the | |
| 632 // concurrent and stop-world phases of CMS, but | |
| 633 // are not shared with parallel scavenge (ParNew). | |
| 634 { | |
| 635 uint i; | |
| 636 uint num_queues = (uint) MAX2(ParallelGCThreads, ParallelCMSThreads); | |
| 637 | |
| 638 if ((CMSParallelRemarkEnabled || CMSConcurrentMTEnabled | |
| 639 || ParallelRefProcEnabled) | |
| 640 && num_queues > 0) { | |
| 641 _task_queues = new OopTaskQueueSet(num_queues); | |
| 642 if (_task_queues == NULL) { | |
| 643 warning("task_queues allocation failure."); | |
| 644 return; | |
| 645 } | |
| 646 _hash_seed = NEW_C_HEAP_ARRAY(int, num_queues); | |
| 647 if (_hash_seed == NULL) { | |
| 648 warning("_hash_seed array allocation failure"); | |
| 649 return; | |
| 650 } | |
| 651 | |
| 652 // XXX use a global constant instead of 64! | |
| 653 typedef struct OopTaskQueuePadded { | |
| 654 OopTaskQueue work_queue; | |
| 655 char pad[64 - sizeof(OopTaskQueue)]; // prevent false sharing | |
| 656 } OopTaskQueuePadded; | |
| 657 | |
| 658 for (i = 0; i < num_queues; i++) { | |
| 659 OopTaskQueuePadded *q_padded = new OopTaskQueuePadded(); | |
| 660 if (q_padded == NULL) { | |
| 661 warning("work_queue allocation failure."); | |
| 662 return; | |
| 663 } | |
| 664 _task_queues->register_queue(i, &q_padded->work_queue); | |
| 665 } | |
| 666 for (i = 0; i < num_queues; i++) { | |
| 667 _task_queues->queue(i)->initialize(); | |
| 668 _hash_seed[i] = 17; // copied from ParNew | |
| 669 } | |
| 670 } | |
| 671 } | |
| 672 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
673 _cmsGen ->init_initiating_occupancy(CMSInitiatingOccupancyFraction, CMSTriggerRatio); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
674 _permGen->init_initiating_occupancy(CMSInitiatingPermOccupancyFraction, CMSTriggerPermRatio); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
675 |
| 0 | 676 // Clip CMSBootstrapOccupancy between 0 and 100. |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
677 _bootstrap_occupancy = ((double)MIN2((uintx)100, MAX2((uintx)0, CMSBootstrapOccupancy))) |
| 0 | 678 /(double)100; |
| 679 | |
| 680 _full_gcs_since_conc_gc = 0; | |
| 681 | |
| 682 // Now tell CMS generations the identity of their collector | |
| 683 ConcurrentMarkSweepGeneration::set_collector(this); | |
| 684 | |
| 685 // Create & start a CMS thread for this CMS collector | |
| 686 _cmsThread = ConcurrentMarkSweepThread::start(this); | |
| 687 assert(cmsThread() != NULL, "CMS Thread should have been created"); | |
| 688 assert(cmsThread()->collector() == this, | |
| 689 "CMS Thread should refer to this gen"); | |
| 690 assert(CGC_lock != NULL, "Where's the CGC_lock?"); | |
| 691 | |
| 692 // Support for parallelizing young gen rescan | |
| 693 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 694 _young_gen = gch->prev_gen(_cmsGen); | |
| 695 if (gch->supports_inline_contig_alloc()) { | |
| 696 _top_addr = gch->top_addr(); | |
| 697 _end_addr = gch->end_addr(); | |
| 698 assert(_young_gen != NULL, "no _young_gen"); | |
| 699 _eden_chunk_index = 0; | |
| 700 _eden_chunk_capacity = (_young_gen->max_capacity()+CMSSamplingGrain)/CMSSamplingGrain; | |
| 701 _eden_chunk_array = NEW_C_HEAP_ARRAY(HeapWord*, _eden_chunk_capacity); | |
| 702 if (_eden_chunk_array == NULL) { | |
| 703 _eden_chunk_capacity = 0; | |
| 704 warning("GC/CMS: _eden_chunk_array allocation failure"); | |
| 705 } | |
| 706 } | |
| 707 assert(_eden_chunk_array != NULL || _eden_chunk_capacity == 0, "Error"); | |
| 708 | |
| 709 // Support for parallelizing survivor space rescan | |
| 710 if (CMSParallelRemarkEnabled && CMSParallelSurvivorRemarkEnabled) { | |
| 711 size_t max_plab_samples = MaxNewSize/((SurvivorRatio+2)*MinTLABSize); | |
| 712 _survivor_plab_array = NEW_C_HEAP_ARRAY(ChunkArray, ParallelGCThreads); | |
| 713 _survivor_chunk_array = NEW_C_HEAP_ARRAY(HeapWord*, 2*max_plab_samples); | |
| 714 _cursor = NEW_C_HEAP_ARRAY(size_t, ParallelGCThreads); | |
| 715 if (_survivor_plab_array == NULL || _survivor_chunk_array == NULL | |
| 716 || _cursor == NULL) { | |
| 717 warning("Failed to allocate survivor plab/chunk array"); | |
| 718 if (_survivor_plab_array != NULL) { | |
| 719 FREE_C_HEAP_ARRAY(ChunkArray, _survivor_plab_array); | |
| 720 _survivor_plab_array = NULL; | |
| 721 } | |
| 722 if (_survivor_chunk_array != NULL) { | |
| 723 FREE_C_HEAP_ARRAY(HeapWord*, _survivor_chunk_array); | |
| 724 _survivor_chunk_array = NULL; | |
| 725 } | |
| 726 if (_cursor != NULL) { | |
| 727 FREE_C_HEAP_ARRAY(size_t, _cursor); | |
| 728 _cursor = NULL; | |
| 729 } | |
| 730 } else { | |
| 731 _survivor_chunk_capacity = 2*max_plab_samples; | |
| 732 for (uint i = 0; i < ParallelGCThreads; i++) { | |
| 733 HeapWord** vec = NEW_C_HEAP_ARRAY(HeapWord*, max_plab_samples); | |
| 734 if (vec == NULL) { | |
| 735 warning("Failed to allocate survivor plab array"); | |
| 736 for (int j = i; j > 0; j--) { | |
| 737 FREE_C_HEAP_ARRAY(HeapWord*, _survivor_plab_array[j-1].array()); | |
| 738 } | |
| 739 FREE_C_HEAP_ARRAY(ChunkArray, _survivor_plab_array); | |
| 740 FREE_C_HEAP_ARRAY(HeapWord*, _survivor_chunk_array); | |
| 741 _survivor_plab_array = NULL; | |
| 742 _survivor_chunk_array = NULL; | |
| 743 _survivor_chunk_capacity = 0; | |
| 744 break; | |
| 745 } else { | |
| 746 ChunkArray* cur = | |
| 747 ::new (&_survivor_plab_array[i]) ChunkArray(vec, | |
| 748 max_plab_samples); | |
| 749 assert(cur->end() == 0, "Should be 0"); | |
| 750 assert(cur->array() == vec, "Should be vec"); | |
| 751 assert(cur->capacity() == max_plab_samples, "Error"); | |
| 752 } | |
| 753 } | |
| 754 } | |
| 755 } | |
| 756 assert( ( _survivor_plab_array != NULL | |
| 757 && _survivor_chunk_array != NULL) | |
| 758 || ( _survivor_chunk_capacity == 0 | |
| 759 && _survivor_chunk_index == 0), | |
| 760 "Error"); | |
| 761 | |
| 762 // Choose what strong roots should be scanned depending on verification options | |
| 763 // and perm gen collection mode. | |
| 764 if (!CMSClassUnloadingEnabled) { | |
| 765 // If class unloading is disabled we want to include all classes into the root set. | |
| 766 add_root_scanning_option(SharedHeap::SO_AllClasses); | |
| 767 } else { | |
| 768 add_root_scanning_option(SharedHeap::SO_SystemClasses); | |
| 769 } | |
| 770 | |
| 771 NOT_PRODUCT(_overflow_counter = CMSMarkStackOverflowInterval;) | |
| 772 _gc_counters = new CollectorCounters("CMS", 1); | |
| 773 _completed_initialization = true; | |
| 774 _sweep_timer.start(); // start of time | |
| 775 } | |
| 776 | |
| 777 const char* ConcurrentMarkSweepGeneration::name() const { | |
| 778 return "concurrent mark-sweep generation"; | |
| 779 } | |
| 780 void ConcurrentMarkSweepGeneration::update_counters() { | |
| 781 if (UsePerfData) { | |
| 782 _space_counters->update_all(); | |
| 783 _gen_counters->update_all(); | |
| 784 } | |
| 785 } | |
| 786 | |
| 787 // this is an optimized version of update_counters(). it takes the | |
| 788 // used value as a parameter rather than computing it. | |
| 789 // | |
| 790 void ConcurrentMarkSweepGeneration::update_counters(size_t used) { | |
| 791 if (UsePerfData) { | |
| 792 _space_counters->update_used(used); | |
| 793 _space_counters->update_capacity(); | |
| 794 _gen_counters->update_all(); | |
| 795 } | |
| 796 } | |
| 797 | |
| 798 void ConcurrentMarkSweepGeneration::print() const { | |
| 799 Generation::print(); | |
| 800 cmsSpace()->print(); | |
| 801 } | |
| 802 | |
| 803 #ifndef PRODUCT | |
| 804 void ConcurrentMarkSweepGeneration::print_statistics() { | |
| 805 cmsSpace()->printFLCensus(0); | |
| 806 } | |
| 807 #endif | |
| 808 | |
| 809 void ConcurrentMarkSweepGeneration::printOccupancy(const char *s) { | |
| 810 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 811 if (PrintGCDetails) { | |
| 812 if (Verbose) { | |
| 813 gclog_or_tty->print(" [%d %s-%s: "SIZE_FORMAT"("SIZE_FORMAT")]", | |
| 814 level(), short_name(), s, used(), capacity()); | |
| 815 } else { | |
| 816 gclog_or_tty->print(" [%d %s-%s: "SIZE_FORMAT"K("SIZE_FORMAT"K)]", | |
| 817 level(), short_name(), s, used() / K, capacity() / K); | |
| 818 } | |
| 819 } | |
| 820 if (Verbose) { | |
| 821 gclog_or_tty->print(" "SIZE_FORMAT"("SIZE_FORMAT")", | |
| 822 gch->used(), gch->capacity()); | |
| 823 } else { | |
| 824 gclog_or_tty->print(" "SIZE_FORMAT"K("SIZE_FORMAT"K)", | |
| 825 gch->used() / K, gch->capacity() / K); | |
| 826 } | |
| 827 } | |
| 828 | |
| 829 size_t | |
| 830 ConcurrentMarkSweepGeneration::contiguous_available() const { | |
| 831 // dld proposes an improvement in precision here. If the committed | |
| 832 // part of the space ends in a free block we should add that to | |
| 833 // uncommitted size in the calculation below. Will make this | |
| 834 // change later, staying with the approximation below for the | |
| 835 // time being. -- ysr. | |
| 836 return MAX2(_virtual_space.uncommitted_size(), unsafe_max_alloc_nogc()); | |
| 837 } | |
| 838 | |
| 839 size_t | |
| 840 ConcurrentMarkSweepGeneration::unsafe_max_alloc_nogc() const { | |
| 841 return _cmsSpace->max_alloc_in_words() * HeapWordSize; | |
| 842 } | |
| 843 | |
| 844 size_t ConcurrentMarkSweepGeneration::max_available() const { | |
| 845 return free() + _virtual_space.uncommitted_size(); | |
| 846 } | |
| 847 | |
| 848 bool ConcurrentMarkSweepGeneration::promotion_attempt_is_safe( | |
| 849 size_t max_promotion_in_bytes, | |
| 850 bool younger_handles_promotion_failure) const { | |
| 851 | |
| 852 // This is the most conservative test. Full promotion is | |
| 853 // guaranteed if this is used. The multiplicative factor is to | |
| 854 // account for the worst case "dilatation". | |
| 855 double adjusted_max_promo_bytes = _dilatation_factor * max_promotion_in_bytes; | |
| 856 if (adjusted_max_promo_bytes > (double)max_uintx) { // larger than size_t | |
| 857 adjusted_max_promo_bytes = (double)max_uintx; | |
| 858 } | |
| 859 bool result = (max_contiguous_available() >= (size_t)adjusted_max_promo_bytes); | |
| 860 | |
| 861 if (younger_handles_promotion_failure && !result) { | |
| 862 // Full promotion is not guaranteed because fragmentation | |
| 863 // of the cms generation can prevent the full promotion. | |
| 864 result = (max_available() >= (size_t)adjusted_max_promo_bytes); | |
| 865 | |
| 866 if (!result) { | |
| 867 // With promotion failure handling the test for the ability | |
| 868 // to support the promotion does not have to be guaranteed. | |
| 869 // Use an average of the amount promoted. | |
| 870 result = max_available() >= (size_t) | |
| 871 gc_stats()->avg_promoted()->padded_average(); | |
| 872 if (PrintGC && Verbose && result) { | |
| 873 gclog_or_tty->print_cr( | |
| 874 "\nConcurrentMarkSweepGeneration::promotion_attempt_is_safe" | |
| 875 " max_available: " SIZE_FORMAT | |
| 876 " avg_promoted: " SIZE_FORMAT, | |
| 877 max_available(), (size_t) | |
| 878 gc_stats()->avg_promoted()->padded_average()); | |
| 879 } | |
| 880 } else { | |
| 881 if (PrintGC && Verbose) { | |
| 882 gclog_or_tty->print_cr( | |
| 883 "\nConcurrentMarkSweepGeneration::promotion_attempt_is_safe" | |
| 884 " max_available: " SIZE_FORMAT | |
| 885 " adj_max_promo_bytes: " SIZE_FORMAT, | |
| 886 max_available(), (size_t)adjusted_max_promo_bytes); | |
| 887 } | |
| 888 } | |
| 889 } else { | |
| 890 if (PrintGC && Verbose) { | |
| 891 gclog_or_tty->print_cr( | |
| 892 "\nConcurrentMarkSweepGeneration::promotion_attempt_is_safe" | |
| 893 " contiguous_available: " SIZE_FORMAT | |
| 894 " adj_max_promo_bytes: " SIZE_FORMAT, | |
| 895 max_contiguous_available(), (size_t)adjusted_max_promo_bytes); | |
| 896 } | |
| 897 } | |
| 898 return result; | |
| 899 } | |
| 900 | |
| 901 CompactibleSpace* | |
| 902 ConcurrentMarkSweepGeneration::first_compaction_space() const { | |
| 903 return _cmsSpace; | |
| 904 } | |
| 905 | |
| 906 void ConcurrentMarkSweepGeneration::reset_after_compaction() { | |
| 907 // Clear the promotion information. These pointers can be adjusted | |
| 908 // along with all the other pointers into the heap but | |
| 909 // compaction is expected to be a rare event with | |
| 910 // a heap using cms so don't do it without seeing the need. | |
| 911 if (ParallelGCThreads > 0) { | |
| 912 for (uint i = 0; i < ParallelGCThreads; i++) { | |
| 913 _par_gc_thread_states[i]->promo.reset(); | |
| 914 } | |
| 915 } | |
| 916 } | |
| 917 | |
| 918 void ConcurrentMarkSweepGeneration::space_iterate(SpaceClosure* blk, bool usedOnly) { | |
| 919 blk->do_space(_cmsSpace); | |
| 920 } | |
| 921 | |
| 922 void ConcurrentMarkSweepGeneration::compute_new_size() { | |
| 923 assert_locked_or_safepoint(Heap_lock); | |
| 924 | |
| 925 // If incremental collection failed, we just want to expand | |
| 926 // to the limit. | |
| 927 if (incremental_collection_failed()) { | |
| 928 clear_incremental_collection_failed(); | |
| 929 grow_to_reserved(); | |
| 930 return; | |
| 931 } | |
| 932 | |
| 933 size_t expand_bytes = 0; | |
| 934 double free_percentage = ((double) free()) / capacity(); | |
| 935 double desired_free_percentage = (double) MinHeapFreeRatio / 100; | |
| 936 double maximum_free_percentage = (double) MaxHeapFreeRatio / 100; | |
| 937 | |
| 938 // compute expansion delta needed for reaching desired free percentage | |
| 939 if (free_percentage < desired_free_percentage) { | |
| 940 size_t desired_capacity = (size_t)(used() / ((double) 1 - desired_free_percentage)); | |
| 941 assert(desired_capacity >= capacity(), "invalid expansion size"); | |
| 942 expand_bytes = MAX2(desired_capacity - capacity(), MinHeapDeltaBytes); | |
| 943 } | |
| 944 if (expand_bytes > 0) { | |
| 945 if (PrintGCDetails && Verbose) { | |
| 946 size_t desired_capacity = (size_t)(used() / ((double) 1 - desired_free_percentage)); | |
| 947 gclog_or_tty->print_cr("\nFrom compute_new_size: "); | |
| 948 gclog_or_tty->print_cr(" Free fraction %f", free_percentage); | |
| 949 gclog_or_tty->print_cr(" Desired free fraction %f", | |
| 950 desired_free_percentage); | |
| 951 gclog_or_tty->print_cr(" Maximum free fraction %f", | |
| 952 maximum_free_percentage); | |
| 953 gclog_or_tty->print_cr(" Capactiy "SIZE_FORMAT, capacity()/1000); | |
| 954 gclog_or_tty->print_cr(" Desired capacity "SIZE_FORMAT, | |
| 955 desired_capacity/1000); | |
| 956 int prev_level = level() - 1; | |
| 957 if (prev_level >= 0) { | |
| 958 size_t prev_size = 0; | |
| 959 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 960 Generation* prev_gen = gch->_gens[prev_level]; | |
| 961 prev_size = prev_gen->capacity(); | |
| 962 gclog_or_tty->print_cr(" Younger gen size "SIZE_FORMAT, | |
| 963 prev_size/1000); | |
| 964 } | |
| 965 gclog_or_tty->print_cr(" unsafe_max_alloc_nogc "SIZE_FORMAT, | |
| 966 unsafe_max_alloc_nogc()/1000); | |
| 967 gclog_or_tty->print_cr(" contiguous available "SIZE_FORMAT, | |
| 968 contiguous_available()/1000); | |
| 969 gclog_or_tty->print_cr(" Expand by "SIZE_FORMAT" (bytes)", | |
| 970 expand_bytes); | |
| 971 } | |
| 972 // safe if expansion fails | |
| 973 expand(expand_bytes, 0, CMSExpansionCause::_satisfy_free_ratio); | |
| 974 if (PrintGCDetails && Verbose) { | |
| 975 gclog_or_tty->print_cr(" Expanded free fraction %f", | |
| 976 ((double) free()) / capacity()); | |
| 977 } | |
| 978 } | |
| 979 } | |
| 980 | |
| 981 Mutex* ConcurrentMarkSweepGeneration::freelistLock() const { | |
| 982 return cmsSpace()->freelistLock(); | |
| 983 } | |
| 984 | |
| 985 HeapWord* ConcurrentMarkSweepGeneration::allocate(size_t size, | |
| 986 bool tlab) { | |
| 987 CMSSynchronousYieldRequest yr; | |
| 988 MutexLockerEx x(freelistLock(), | |
| 989 Mutex::_no_safepoint_check_flag); | |
| 990 return have_lock_and_allocate(size, tlab); | |
| 991 } | |
| 992 | |
| 993 HeapWord* ConcurrentMarkSweepGeneration::have_lock_and_allocate(size_t size, | |
| 994 bool tlab) { | |
| 995 assert_lock_strong(freelistLock()); | |
| 996 size_t adjustedSize = CompactibleFreeListSpace::adjustObjectSize(size); | |
| 997 HeapWord* res = cmsSpace()->allocate(adjustedSize); | |
| 998 // Allocate the object live (grey) if the background collector has | |
| 999 // started marking. This is necessary because the marker may | |
| 1000 // have passed this address and consequently this object will | |
| 1001 // not otherwise be greyed and would be incorrectly swept up. | |
| 1002 // Note that if this object contains references, the writing | |
| 1003 // of those references will dirty the card containing this object | |
| 1004 // allowing the object to be blackened (and its references scanned) | |
| 1005 // either during a preclean phase or at the final checkpoint. | |
| 1006 if (res != NULL) { | |
| 1007 collector()->direct_allocated(res, adjustedSize); | |
| 1008 _direct_allocated_words += adjustedSize; | |
| 1009 // allocation counters | |
| 1010 NOT_PRODUCT( | |
| 1011 _numObjectsAllocated++; | |
| 1012 _numWordsAllocated += (int)adjustedSize; | |
| 1013 ) | |
| 1014 } | |
| 1015 return res; | |
| 1016 } | |
| 1017 | |
| 1018 // In the case of direct allocation by mutators in a generation that | |
| 1019 // is being concurrently collected, the object must be allocated | |
| 1020 // live (grey) if the background collector has started marking. | |
| 1021 // This is necessary because the marker may | |
| 1022 // have passed this address and consequently this object will | |
| 1023 // not otherwise be greyed and would be incorrectly swept up. | |
| 1024 // Note that if this object contains references, the writing | |
| 1025 // of those references will dirty the card containing this object | |
| 1026 // allowing the object to be blackened (and its references scanned) | |
| 1027 // either during a preclean phase or at the final checkpoint. | |
| 1028 void CMSCollector::direct_allocated(HeapWord* start, size_t size) { | |
| 1029 assert(_markBitMap.covers(start, size), "Out of bounds"); | |
| 1030 if (_collectorState >= Marking) { | |
| 1031 MutexLockerEx y(_markBitMap.lock(), | |
| 1032 Mutex::_no_safepoint_check_flag); | |
| 1033 // [see comments preceding SweepClosure::do_blk() below for details] | |
| 1034 // 1. need to mark the object as live so it isn't collected | |
| 1035 // 2. need to mark the 2nd bit to indicate the object may be uninitialized | |
| 1036 // 3. need to mark the end of the object so sweeper can skip over it | |
| 1037 // if it's uninitialized when the sweeper reaches it. | |
| 1038 _markBitMap.mark(start); // object is live | |
| 1039 _markBitMap.mark(start + 1); // object is potentially uninitialized? | |
| 1040 _markBitMap.mark(start + size - 1); | |
| 1041 // mark end of object | |
| 1042 } | |
| 1043 // check that oop looks uninitialized | |
| 187 | 1044 assert(oop(start)->klass_or_null() == NULL, "_klass should be NULL"); |
| 0 | 1045 } |
| 1046 | |
| 1047 void CMSCollector::promoted(bool par, HeapWord* start, | |
| 1048 bool is_obj_array, size_t obj_size) { | |
| 1049 assert(_markBitMap.covers(start), "Out of bounds"); | |
| 1050 // See comment in direct_allocated() about when objects should | |
| 1051 // be allocated live. | |
| 1052 if (_collectorState >= Marking) { | |
| 1053 // we already hold the marking bit map lock, taken in | |
| 1054 // the prologue | |
| 1055 if (par) { | |
| 1056 _markBitMap.par_mark(start); | |
| 1057 } else { | |
| 1058 _markBitMap.mark(start); | |
| 1059 } | |
| 1060 // We don't need to mark the object as uninitialized (as | |
| 1061 // in direct_allocated above) because this is being done with the | |
| 1062 // world stopped and the object will be initialized by the | |
| 1063 // time the sweeper gets to look at it. | |
| 1064 assert(SafepointSynchronize::is_at_safepoint(), | |
| 1065 "expect promotion only at safepoints"); | |
| 1066 | |
| 1067 if (_collectorState < Sweeping) { | |
| 1068 // Mark the appropriate cards in the modUnionTable, so that | |
| 1069 // this object gets scanned before the sweep. If this is | |
| 1070 // not done, CMS generation references in the object might | |
| 1071 // not get marked. | |
| 1072 // For the case of arrays, which are otherwise precisely | |
| 1073 // marked, we need to dirty the entire array, not just its head. | |
| 1074 if (is_obj_array) { | |
| 1075 // The [par_]mark_range() method expects mr.end() below to | |
| 1076 // be aligned to the granularity of a bit's representation | |
| 1077 // in the heap. In the case of the MUT below, that's a | |
| 1078 // card size. | |
| 1079 MemRegion mr(start, | |
| 1080 (HeapWord*)round_to((intptr_t)(start + obj_size), | |
| 1081 CardTableModRefBS::card_size /* bytes */)); | |
| 1082 if (par) { | |
| 1083 _modUnionTable.par_mark_range(mr); | |
| 1084 } else { | |
| 1085 _modUnionTable.mark_range(mr); | |
| 1086 } | |
| 1087 } else { // not an obj array; we can just mark the head | |
| 1088 if (par) { | |
| 1089 _modUnionTable.par_mark(start); | |
| 1090 } else { | |
| 1091 _modUnionTable.mark(start); | |
| 1092 } | |
| 1093 } | |
| 1094 } | |
| 1095 } | |
| 1096 } | |
| 1097 | |
| 1098 static inline size_t percent_of_space(Space* space, HeapWord* addr) | |
| 1099 { | |
| 1100 size_t delta = pointer_delta(addr, space->bottom()); | |
| 1101 return (size_t)(delta * 100.0 / (space->capacity() / HeapWordSize)); | |
| 1102 } | |
| 1103 | |
| 1104 void CMSCollector::icms_update_allocation_limits() | |
| 1105 { | |
| 1106 Generation* gen0 = GenCollectedHeap::heap()->get_gen(0); | |
| 1107 EdenSpace* eden = gen0->as_DefNewGeneration()->eden(); | |
| 1108 | |
| 1109 const unsigned int duty_cycle = stats().icms_update_duty_cycle(); | |
| 1110 if (CMSTraceIncrementalPacing) { | |
| 1111 stats().print(); | |
| 1112 } | |
| 1113 | |
| 1114 assert(duty_cycle <= 100, "invalid duty cycle"); | |
| 1115 if (duty_cycle != 0) { | |
| 1116 // The duty_cycle is a percentage between 0 and 100; convert to words and | |
| 1117 // then compute the offset from the endpoints of the space. | |
| 1118 size_t free_words = eden->free() / HeapWordSize; | |
| 1119 double free_words_dbl = (double)free_words; | |
| 1120 size_t duty_cycle_words = (size_t)(free_words_dbl * duty_cycle / 100.0); | |
| 1121 size_t offset_words = (free_words - duty_cycle_words) / 2; | |
| 1122 | |
| 1123 _icms_start_limit = eden->top() + offset_words; | |
| 1124 _icms_stop_limit = eden->end() - offset_words; | |
| 1125 | |
| 1126 // The limits may be adjusted (shifted to the right) by | |
| 1127 // CMSIncrementalOffset, to allow the application more mutator time after a | |
| 1128 // young gen gc (when all mutators were stopped) and before CMS starts and | |
| 1129 // takes away one or more cpus. | |
| 1130 if (CMSIncrementalOffset != 0) { | |
| 1131 double adjustment_dbl = free_words_dbl * CMSIncrementalOffset / 100.0; | |
| 1132 size_t adjustment = (size_t)adjustment_dbl; | |
| 1133 HeapWord* tmp_stop = _icms_stop_limit + adjustment; | |
| 1134 if (tmp_stop > _icms_stop_limit && tmp_stop < eden->end()) { | |
| 1135 _icms_start_limit += adjustment; | |
| 1136 _icms_stop_limit = tmp_stop; | |
| 1137 } | |
| 1138 } | |
| 1139 } | |
| 1140 if (duty_cycle == 0 || (_icms_start_limit == _icms_stop_limit)) { | |
| 1141 _icms_start_limit = _icms_stop_limit = eden->end(); | |
| 1142 } | |
| 1143 | |
| 1144 // Install the new start limit. | |
| 1145 eden->set_soft_end(_icms_start_limit); | |
| 1146 | |
| 1147 if (CMSTraceIncrementalMode) { | |
| 1148 gclog_or_tty->print(" icms alloc limits: " | |
| 1149 PTR_FORMAT "," PTR_FORMAT | |
| 1150 " (" SIZE_FORMAT "%%," SIZE_FORMAT "%%) ", | |
| 1151 _icms_start_limit, _icms_stop_limit, | |
| 1152 percent_of_space(eden, _icms_start_limit), | |
| 1153 percent_of_space(eden, _icms_stop_limit)); | |
| 1154 if (Verbose) { | |
| 1155 gclog_or_tty->print("eden: "); | |
| 1156 eden->print_on(gclog_or_tty); | |
| 1157 } | |
| 1158 } | |
| 1159 } | |
| 1160 | |
| 1161 // Any changes here should try to maintain the invariant | |
| 1162 // that if this method is called with _icms_start_limit | |
| 1163 // and _icms_stop_limit both NULL, then it should return NULL | |
| 1164 // and not notify the icms thread. | |
| 1165 HeapWord* | |
| 1166 CMSCollector::allocation_limit_reached(Space* space, HeapWord* top, | |
| 1167 size_t word_size) | |
| 1168 { | |
| 1169 // A start_limit equal to end() means the duty cycle is 0, so treat that as a | |
| 1170 // nop. | |
| 1171 if (CMSIncrementalMode && _icms_start_limit != space->end()) { | |
| 1172 if (top <= _icms_start_limit) { | |
| 1173 if (CMSTraceIncrementalMode) { | |
| 1174 space->print_on(gclog_or_tty); | |
| 1175 gclog_or_tty->stamp(); | |
| 1176 gclog_or_tty->print_cr(" start limit top=" PTR_FORMAT | |
| 1177 ", new limit=" PTR_FORMAT | |
| 1178 " (" SIZE_FORMAT "%%)", | |
| 1179 top, _icms_stop_limit, | |
| 1180 percent_of_space(space, _icms_stop_limit)); | |
| 1181 } | |
| 1182 ConcurrentMarkSweepThread::start_icms(); | |
| 1183 assert(top < _icms_stop_limit, "Tautology"); | |
| 1184 if (word_size < pointer_delta(_icms_stop_limit, top)) { | |
| 1185 return _icms_stop_limit; | |
| 1186 } | |
| 1187 | |
| 1188 // The allocation will cross both the _start and _stop limits, so do the | |
| 1189 // stop notification also and return end(). | |
| 1190 if (CMSTraceIncrementalMode) { | |
| 1191 space->print_on(gclog_or_tty); | |
| 1192 gclog_or_tty->stamp(); | |
| 1193 gclog_or_tty->print_cr(" +stop limit top=" PTR_FORMAT | |
| 1194 ", new limit=" PTR_FORMAT | |
| 1195 " (" SIZE_FORMAT "%%)", | |
| 1196 top, space->end(), | |
| 1197 percent_of_space(space, space->end())); | |
| 1198 } | |
| 1199 ConcurrentMarkSweepThread::stop_icms(); | |
| 1200 return space->end(); | |
| 1201 } | |
| 1202 | |
| 1203 if (top <= _icms_stop_limit) { | |
| 1204 if (CMSTraceIncrementalMode) { | |
| 1205 space->print_on(gclog_or_tty); | |
| 1206 gclog_or_tty->stamp(); | |
| 1207 gclog_or_tty->print_cr(" stop limit top=" PTR_FORMAT | |
| 1208 ", new limit=" PTR_FORMAT | |
| 1209 " (" SIZE_FORMAT "%%)", | |
| 1210 top, space->end(), | |
| 1211 percent_of_space(space, space->end())); | |
| 1212 } | |
| 1213 ConcurrentMarkSweepThread::stop_icms(); | |
| 1214 return space->end(); | |
| 1215 } | |
| 1216 | |
| 1217 if (CMSTraceIncrementalMode) { | |
| 1218 space->print_on(gclog_or_tty); | |
| 1219 gclog_or_tty->stamp(); | |
| 1220 gclog_or_tty->print_cr(" end limit top=" PTR_FORMAT | |
| 1221 ", new limit=" PTR_FORMAT, | |
| 1222 top, NULL); | |
| 1223 } | |
| 1224 } | |
| 1225 | |
| 1226 return NULL; | |
| 1227 } | |
| 1228 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
1229 oop ConcurrentMarkSweepGeneration::promote(oop obj, size_t obj_size) { |
| 0 | 1230 assert(obj_size == (size_t)obj->size(), "bad obj_size passed in"); |
| 1231 // allocate, copy and if necessary update promoinfo -- | |
| 1232 // delegate to underlying space. | |
| 1233 assert_lock_strong(freelistLock()); | |
| 1234 | |
| 1235 #ifndef PRODUCT | |
| 1236 if (Universe::heap()->promotion_should_fail()) { | |
| 1237 return NULL; | |
| 1238 } | |
| 1239 #endif // #ifndef PRODUCT | |
| 1240 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
1241 oop res = _cmsSpace->promote(obj, obj_size); |
| 0 | 1242 if (res == NULL) { |
| 1243 // expand and retry | |
| 1244 size_t s = _cmsSpace->expansionSpaceRequired(obj_size); // HeapWords | |
| 1245 expand(s*HeapWordSize, MinHeapDeltaBytes, | |
| 1246 CMSExpansionCause::_satisfy_promotion); | |
| 1247 // Since there's currently no next generation, we don't try to promote | |
| 1248 // into a more senior generation. | |
| 1249 assert(next_gen() == NULL, "assumption, based upon which no attempt " | |
| 1250 "is made to pass on a possibly failing " | |
| 1251 "promotion to next generation"); | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
1252 res = _cmsSpace->promote(obj, obj_size); |
| 0 | 1253 } |
| 1254 if (res != NULL) { | |
| 1255 // See comment in allocate() about when objects should | |
| 1256 // be allocated live. | |
| 1257 assert(obj->is_oop(), "Will dereference klass pointer below"); | |
| 1258 collector()->promoted(false, // Not parallel | |
| 1259 (HeapWord*)res, obj->is_objArray(), obj_size); | |
| 1260 // promotion counters | |
| 1261 NOT_PRODUCT( | |
| 1262 _numObjectsPromoted++; | |
| 1263 _numWordsPromoted += | |
| 1264 (int)(CompactibleFreeListSpace::adjustObjectSize(obj->size())); | |
| 1265 ) | |
| 1266 } | |
| 1267 return res; | |
| 1268 } | |
| 1269 | |
| 1270 | |
| 1271 HeapWord* | |
| 1272 ConcurrentMarkSweepGeneration::allocation_limit_reached(Space* space, | |
| 1273 HeapWord* top, | |
| 1274 size_t word_sz) | |
| 1275 { | |
| 1276 return collector()->allocation_limit_reached(space, top, word_sz); | |
| 1277 } | |
| 1278 | |
| 1279 // Things to support parallel young-gen collection. | |
| 1280 oop | |
| 1281 ConcurrentMarkSweepGeneration::par_promote(int thread_num, | |
| 1282 oop old, markOop m, | |
| 1283 size_t word_sz) { | |
| 1284 #ifndef PRODUCT | |
| 1285 if (Universe::heap()->promotion_should_fail()) { | |
| 1286 return NULL; | |
| 1287 } | |
| 1288 #endif // #ifndef PRODUCT | |
| 1289 | |
| 1290 CMSParGCThreadState* ps = _par_gc_thread_states[thread_num]; | |
| 1291 PromotionInfo* promoInfo = &ps->promo; | |
| 1292 // if we are tracking promotions, then first ensure space for | |
| 1293 // promotion (including spooling space for saving header if necessary). | |
| 1294 // then allocate and copy, then track promoted info if needed. | |
| 1295 // When tracking (see PromotionInfo::track()), the mark word may | |
| 1296 // be displaced and in this case restoration of the mark word | |
| 1297 // occurs in the (oop_since_save_marks_)iterate phase. | |
| 1298 if (promoInfo->tracking() && !promoInfo->ensure_spooling_space()) { | |
| 1299 // Out of space for allocating spooling buffers; | |
| 1300 // try expanding and allocating spooling buffers. | |
| 1301 if (!expand_and_ensure_spooling_space(promoInfo)) { | |
| 1302 return NULL; | |
| 1303 } | |
| 1304 } | |
| 1305 assert(promoInfo->has_spooling_space(), "Control point invariant"); | |
| 1306 HeapWord* obj_ptr = ps->lab.alloc(word_sz); | |
| 1307 if (obj_ptr == NULL) { | |
| 1308 obj_ptr = expand_and_par_lab_allocate(ps, word_sz); | |
| 1309 if (obj_ptr == NULL) { | |
| 1310 return NULL; | |
| 1311 } | |
| 1312 } | |
| 1313 oop obj = oop(obj_ptr); | |
| 187 | 1314 assert(obj->klass_or_null() == NULL, "Object should be uninitialized here."); |
| 0 | 1315 // Otherwise, copy the object. Here we must be careful to insert the |
| 1316 // klass pointer last, since this marks the block as an allocated object. | |
| 187 | 1317 // Except with compressed oops it's the mark word. |
| 0 | 1318 HeapWord* old_ptr = (HeapWord*)old; |
| 1319 if (word_sz > (size_t)oopDesc::header_size()) { | |
| 1320 Copy::aligned_disjoint_words(old_ptr + oopDesc::header_size(), | |
| 1321 obj_ptr + oopDesc::header_size(), | |
| 1322 word_sz - oopDesc::header_size()); | |
| 1323 } | |
| 187 | 1324 |
| 1325 if (UseCompressedOops) { | |
| 1326 // Copy gap missed by (aligned) header size calculation above | |
| 1327 obj->set_klass_gap(old->klass_gap()); | |
| 1328 } | |
| 1329 | |
| 0 | 1330 // Restore the mark word copied above. |
| 1331 obj->set_mark(m); | |
| 187 | 1332 |
| 0 | 1333 // Now we can track the promoted object, if necessary. We take care |
| 1334 // To delay the transition from uninitialized to full object | |
| 1335 // (i.e., insertion of klass pointer) until after, so that it | |
| 1336 // atomically becomes a promoted object. | |
| 1337 if (promoInfo->tracking()) { | |
| 1338 promoInfo->track((PromotedObject*)obj, old->klass()); | |
| 1339 } | |
| 187 | 1340 |
| 1341 // Finally, install the klass pointer (this should be volatile). | |
| 0 | 1342 obj->set_klass(old->klass()); |
| 1343 | |
| 1344 assert(old->is_oop(), "Will dereference klass ptr below"); | |
| 1345 collector()->promoted(true, // parallel | |
| 1346 obj_ptr, old->is_objArray(), word_sz); | |
| 1347 | |
| 1348 NOT_PRODUCT( | |
| 1349 Atomic::inc(&_numObjectsPromoted); | |
| 1350 Atomic::add((jint)CompactibleFreeListSpace::adjustObjectSize(obj->size()), | |
| 1351 &_numWordsPromoted); | |
| 1352 ) | |
| 1353 | |
| 1354 return obj; | |
| 1355 } | |
| 1356 | |
| 1357 void | |
| 1358 ConcurrentMarkSweepGeneration:: | |
| 1359 par_promote_alloc_undo(int thread_num, | |
| 1360 HeapWord* obj, size_t word_sz) { | |
| 1361 // CMS does not support promotion undo. | |
| 1362 ShouldNotReachHere(); | |
| 1363 } | |
| 1364 | |
| 1365 void | |
| 1366 ConcurrentMarkSweepGeneration:: | |
| 1367 par_promote_alloc_done(int thread_num) { | |
| 1368 CMSParGCThreadState* ps = _par_gc_thread_states[thread_num]; | |
| 1369 ps->lab.retire(); | |
| 1370 #if CFLS_LAB_REFILL_STATS | |
| 1371 if (thread_num == 0) { | |
| 1372 _cmsSpace->print_par_alloc_stats(); | |
| 1373 } | |
| 1374 #endif | |
| 1375 } | |
| 1376 | |
| 1377 void | |
| 1378 ConcurrentMarkSweepGeneration:: | |
| 1379 par_oop_since_save_marks_iterate_done(int thread_num) { | |
| 1380 CMSParGCThreadState* ps = _par_gc_thread_states[thread_num]; | |
| 1381 ParScanWithoutBarrierClosure* dummy_cl = NULL; | |
| 1382 ps->promo.promoted_oops_iterate_nv(dummy_cl); | |
| 1383 } | |
| 1384 | |
| 1385 // XXXPERM | |
| 1386 bool ConcurrentMarkSweepGeneration::should_collect(bool full, | |
| 1387 size_t size, | |
| 1388 bool tlab) | |
| 1389 { | |
| 1390 // We allow a STW collection only if a full | |
| 1391 // collection was requested. | |
| 1392 return full || should_allocate(size, tlab); // FIX ME !!! | |
| 1393 // This and promotion failure handling are connected at the | |
| 1394 // hip and should be fixed by untying them. | |
| 1395 } | |
| 1396 | |
| 1397 bool CMSCollector::shouldConcurrentCollect() { | |
| 1398 if (_full_gc_requested) { | |
| 1399 assert(ExplicitGCInvokesConcurrent, "Unexpected state"); | |
| 1400 if (Verbose && PrintGCDetails) { | |
| 1401 gclog_or_tty->print_cr("CMSCollector: collect because of explicit " | |
| 1402 " gc request"); | |
| 1403 } | |
| 1404 return true; | |
| 1405 } | |
| 1406 | |
| 1407 // For debugging purposes, change the type of collection. | |
| 1408 // If the rotation is not on the concurrent collection | |
| 1409 // type, don't start a concurrent collection. | |
| 1410 NOT_PRODUCT( | |
| 1411 if (RotateCMSCollectionTypes && | |
| 1412 (_cmsGen->debug_collection_type() != | |
| 1413 ConcurrentMarkSweepGeneration::Concurrent_collection_type)) { | |
| 1414 assert(_cmsGen->debug_collection_type() != | |
| 1415 ConcurrentMarkSweepGeneration::Unknown_collection_type, | |
| 1416 "Bad cms collection type"); | |
| 1417 return false; | |
| 1418 } | |
| 1419 ) | |
| 1420 | |
| 1421 FreelistLocker x(this); | |
| 1422 // ------------------------------------------------------------------ | |
| 1423 // Print out lots of information which affects the initiation of | |
| 1424 // a collection. | |
| 1425 if (PrintCMSInitiationStatistics && stats().valid()) { | |
| 1426 gclog_or_tty->print("CMSCollector shouldConcurrentCollect: "); | |
| 1427 gclog_or_tty->stamp(); | |
| 1428 gclog_or_tty->print_cr(""); | |
| 1429 stats().print_on(gclog_or_tty); | |
| 1430 gclog_or_tty->print_cr("time_until_cms_gen_full %3.7f", | |
| 1431 stats().time_until_cms_gen_full()); | |
| 1432 gclog_or_tty->print_cr("free="SIZE_FORMAT, _cmsGen->free()); | |
| 1433 gclog_or_tty->print_cr("contiguous_available="SIZE_FORMAT, | |
| 1434 _cmsGen->contiguous_available()); | |
| 1435 gclog_or_tty->print_cr("promotion_rate=%g", stats().promotion_rate()); | |
| 1436 gclog_or_tty->print_cr("cms_allocation_rate=%g", stats().cms_allocation_rate()); | |
| 1437 gclog_or_tty->print_cr("occupancy=%3.7f", _cmsGen->occupancy()); | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1438 gclog_or_tty->print_cr("initiatingOccupancy=%3.7f", _cmsGen->initiating_occupancy()); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1439 gclog_or_tty->print_cr("initiatingPermOccupancy=%3.7f", _permGen->initiating_occupancy()); |
| 0 | 1440 } |
| 1441 // ------------------------------------------------------------------ | |
| 1442 | |
| 1443 // If the estimated time to complete a cms collection (cms_duration()) | |
| 1444 // is less than the estimated time remaining until the cms generation | |
| 1445 // is full, start a collection. | |
| 1446 if (!UseCMSInitiatingOccupancyOnly) { | |
| 1447 if (stats().valid()) { | |
| 1448 if (stats().time_until_cms_start() == 0.0) { | |
| 1449 return true; | |
| 1450 } | |
| 1451 } else { | |
| 1452 // We want to conservatively collect somewhat early in order | |
| 1453 // to try and "bootstrap" our CMS/promotion statistics; | |
| 1454 // this branch will not fire after the first successful CMS | |
| 1455 // collection because the stats should then be valid. | |
| 1456 if (_cmsGen->occupancy() >= _bootstrap_occupancy) { | |
| 1457 if (Verbose && PrintGCDetails) { | |
| 1458 gclog_or_tty->print_cr( | |
| 1459 " CMSCollector: collect for bootstrapping statistics:" | |
| 1460 " occupancy = %f, boot occupancy = %f", _cmsGen->occupancy(), | |
| 1461 _bootstrap_occupancy); | |
| 1462 } | |
| 1463 return true; | |
| 1464 } | |
| 1465 } | |
| 1466 } | |
| 1467 | |
| 1468 // Otherwise, we start a collection cycle if either the perm gen or | |
| 1469 // old gen want a collection cycle started. Each may use | |
| 1470 // an appropriate criterion for making this decision. | |
| 1471 // XXX We need to make sure that the gen expansion | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1472 // criterion dovetails well with this. XXX NEED TO FIX THIS |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1473 if (_cmsGen->should_concurrent_collect()) { |
| 0 | 1474 if (Verbose && PrintGCDetails) { |
| 1475 gclog_or_tty->print_cr("CMS old gen initiated"); | |
| 1476 } | |
| 1477 return true; | |
| 1478 } | |
| 1479 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1480 // We start a collection if we believe an incremental collection may fail; |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1481 // this is not likely to be productive in practice because it's probably too |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1482 // late anyway. |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1483 GenCollectedHeap* gch = GenCollectedHeap::heap(); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1484 assert(gch->collector_policy()->is_two_generation_policy(), |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1485 "You may want to check the correctness of the following"); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1486 if (gch->incremental_collection_will_fail()) { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1487 if (PrintGCDetails && Verbose) { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1488 gclog_or_tty->print("CMSCollector: collect because incremental collection will fail "); |
| 0 | 1489 } |
| 1490 return true; | |
| 1491 } | |
| 1492 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1493 if (CMSClassUnloadingEnabled && _permGen->should_concurrent_collect()) { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1494 bool res = update_should_unload_classes(); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1495 if (res) { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1496 if (Verbose && PrintGCDetails) { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1497 gclog_or_tty->print_cr("CMS perm gen initiated"); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1498 } |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1499 return true; |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1500 } |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1501 } |
| 0 | 1502 return false; |
| 1503 } | |
| 1504 | |
| 1505 // Clear _expansion_cause fields of constituent generations | |
| 1506 void CMSCollector::clear_expansion_cause() { | |
| 1507 _cmsGen->clear_expansion_cause(); | |
| 1508 _permGen->clear_expansion_cause(); | |
| 1509 } | |
| 1510 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1511 // We should be conservative in starting a collection cycle. To |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1512 // start too eagerly runs the risk of collecting too often in the |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1513 // extreme. To collect too rarely falls back on full collections, |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1514 // which works, even if not optimum in terms of concurrent work. |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1515 // As a work around for too eagerly collecting, use the flag |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1516 // UseCMSInitiatingOccupancyOnly. This also has the advantage of |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1517 // giving the user an easily understandable way of controlling the |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1518 // collections. |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1519 // We want to start a new collection cycle if any of the following |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1520 // conditions hold: |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1521 // . our current occupancy exceeds the configured initiating occupancy |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1522 // for this generation, or |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1523 // . we recently needed to expand this space and have not, since that |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1524 // expansion, done a collection of this generation, or |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1525 // . the underlying space believes that it may be a good idea to initiate |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1526 // a concurrent collection (this may be based on criteria such as the |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1527 // following: the space uses linear allocation and linear allocation is |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1528 // going to fail, or there is believed to be excessive fragmentation in |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1529 // the generation, etc... or ... |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1530 // [.(currently done by CMSCollector::shouldConcurrentCollect() only for |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1531 // the case of the old generation, not the perm generation; see CR 6543076): |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1532 // we may be approaching a point at which allocation requests may fail because |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1533 // we will be out of sufficient free space given allocation rate estimates.] |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1534 bool ConcurrentMarkSweepGeneration::should_concurrent_collect() const { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1535 |
| 0 | 1536 assert_lock_strong(freelistLock()); |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1537 if (occupancy() > initiating_occupancy()) { |
| 0 | 1538 if (PrintGCDetails && Verbose) { |
| 1539 gclog_or_tty->print(" %s: collect because of occupancy %f / %f ", | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1540 short_name(), occupancy(), initiating_occupancy()); |
| 0 | 1541 } |
| 1542 return true; | |
| 1543 } | |
| 1544 if (UseCMSInitiatingOccupancyOnly) { | |
| 1545 return false; | |
| 1546 } | |
| 1547 if (expansion_cause() == CMSExpansionCause::_satisfy_allocation) { | |
| 1548 if (PrintGCDetails && Verbose) { | |
| 1549 gclog_or_tty->print(" %s: collect because expanded for allocation ", | |
| 1550 short_name()); | |
| 1551 } | |
| 1552 return true; | |
| 1553 } | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1554 if (_cmsSpace->should_concurrent_collect()) { |
| 0 | 1555 if (PrintGCDetails && Verbose) { |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
1556 gclog_or_tty->print(" %s: collect because cmsSpace says so ", |
| 0 | 1557 short_name()); |
| 1558 } | |
| 1559 return true; | |
| 1560 } | |
| 1561 return false; | |
| 1562 } | |
| 1563 | |
| 1564 void ConcurrentMarkSweepGeneration::collect(bool full, | |
| 1565 bool clear_all_soft_refs, | |
| 1566 size_t size, | |
| 1567 bool tlab) | |
| 1568 { | |
| 1569 collector()->collect(full, clear_all_soft_refs, size, tlab); | |
| 1570 } | |
| 1571 | |
| 1572 void CMSCollector::collect(bool full, | |
| 1573 bool clear_all_soft_refs, | |
| 1574 size_t size, | |
| 1575 bool tlab) | |
| 1576 { | |
| 1577 if (!UseCMSCollectionPassing && _collectorState > Idling) { | |
| 1578 // For debugging purposes skip the collection if the state | |
| 1579 // is not currently idle | |
| 1580 if (TraceCMSState) { | |
| 1581 gclog_or_tty->print_cr("Thread " INTPTR_FORMAT " skipped full:%d CMS state %d", | |
| 1582 Thread::current(), full, _collectorState); | |
| 1583 } | |
| 1584 return; | |
| 1585 } | |
| 1586 | |
| 1587 // The following "if" branch is present for defensive reasons. | |
| 1588 // In the current uses of this interface, it can be replaced with: | |
| 1589 // assert(!GC_locker.is_active(), "Can't be called otherwise"); | |
| 1590 // But I am not placing that assert here to allow future | |
| 1591 // generality in invoking this interface. | |
| 1592 if (GC_locker::is_active()) { | |
| 1593 // A consistency test for GC_locker | |
| 1594 assert(GC_locker::needs_gc(), "Should have been set already"); | |
| 1595 // Skip this foreground collection, instead | |
| 1596 // expanding the heap if necessary. | |
| 1597 // Need the free list locks for the call to free() in compute_new_size() | |
| 1598 compute_new_size(); | |
| 1599 return; | |
| 1600 } | |
| 1601 acquire_control_and_collect(full, clear_all_soft_refs); | |
| 1602 _full_gcs_since_conc_gc++; | |
| 1603 | |
| 1604 } | |
| 1605 | |
| 1606 void CMSCollector::request_full_gc(unsigned int full_gc_count) { | |
| 1607 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 1608 unsigned int gc_count = gch->total_full_collections(); | |
| 1609 if (gc_count == full_gc_count) { | |
| 1610 MutexLockerEx y(CGC_lock, Mutex::_no_safepoint_check_flag); | |
| 1611 _full_gc_requested = true; | |
| 1612 CGC_lock->notify(); // nudge CMS thread | |
| 1613 } | |
| 1614 } | |
| 1615 | |
| 1616 | |
| 1617 // The foreground and background collectors need to coordinate in order | |
| 1618 // to make sure that they do not mutually interfere with CMS collections. | |
| 1619 // When a background collection is active, | |
| 1620 // the foreground collector may need to take over (preempt) and | |
| 1621 // synchronously complete an ongoing collection. Depending on the | |
| 1622 // frequency of the background collections and the heap usage | |
| 1623 // of the application, this preemption can be seldom or frequent. | |
| 1624 // There are only certain | |
| 1625 // points in the background collection that the "collection-baton" | |
| 1626 // can be passed to the foreground collector. | |
| 1627 // | |
| 1628 // The foreground collector will wait for the baton before | |
| 1629 // starting any part of the collection. The foreground collector | |
| 1630 // will only wait at one location. | |
| 1631 // | |
| 1632 // The background collector will yield the baton before starting a new | |
| 1633 // phase of the collection (e.g., before initial marking, marking from roots, | |
| 1634 // precleaning, final re-mark, sweep etc.) This is normally done at the head | |
| 1635 // of the loop which switches the phases. The background collector does some | |
| 1636 // of the phases (initial mark, final re-mark) with the world stopped. | |
| 1637 // Because of locking involved in stopping the world, | |
| 1638 // the foreground collector should not block waiting for the background | |
| 1639 // collector when it is doing a stop-the-world phase. The background | |
| 1640 // collector will yield the baton at an additional point just before | |
| 1641 // it enters a stop-the-world phase. Once the world is stopped, the | |
| 1642 // background collector checks the phase of the collection. If the | |
| 1643 // phase has not changed, it proceeds with the collection. If the | |
| 1644 // phase has changed, it skips that phase of the collection. See | |
| 1645 // the comments on the use of the Heap_lock in collect_in_background(). | |
| 1646 // | |
| 1647 // Variable used in baton passing. | |
| 1648 // _foregroundGCIsActive - Set to true by the foreground collector when | |
| 1649 // it wants the baton. The foreground clears it when it has finished | |
| 1650 // the collection. | |
| 1651 // _foregroundGCShouldWait - Set to true by the background collector | |
| 1652 // when it is running. The foreground collector waits while | |
| 1653 // _foregroundGCShouldWait is true. | |
| 1654 // CGC_lock - monitor used to protect access to the above variables | |
| 1655 // and to notify the foreground and background collectors. | |
| 1656 // _collectorState - current state of the CMS collection. | |
| 1657 // | |
| 1658 // The foreground collector | |
| 1659 // acquires the CGC_lock | |
| 1660 // sets _foregroundGCIsActive | |
| 1661 // waits on the CGC_lock for _foregroundGCShouldWait to be false | |
| 1662 // various locks acquired in preparation for the collection | |
| 1663 // are released so as not to block the background collector | |
| 1664 // that is in the midst of a collection | |
| 1665 // proceeds with the collection | |
| 1666 // clears _foregroundGCIsActive | |
| 1667 // returns | |
| 1668 // | |
| 1669 // The background collector in a loop iterating on the phases of the | |
| 1670 // collection | |
| 1671 // acquires the CGC_lock | |
| 1672 // sets _foregroundGCShouldWait | |
| 1673 // if _foregroundGCIsActive is set | |
| 1674 // clears _foregroundGCShouldWait, notifies _CGC_lock | |
| 1675 // waits on _CGC_lock for _foregroundGCIsActive to become false | |
| 1676 // and exits the loop. | |
| 1677 // otherwise | |
| 1678 // proceed with that phase of the collection | |
| 1679 // if the phase is a stop-the-world phase, | |
| 1680 // yield the baton once more just before enqueueing | |
| 1681 // the stop-world CMS operation (executed by the VM thread). | |
| 1682 // returns after all phases of the collection are done | |
| 1683 // | |
| 1684 | |
| 1685 void CMSCollector::acquire_control_and_collect(bool full, | |
| 1686 bool clear_all_soft_refs) { | |
| 1687 assert(SafepointSynchronize::is_at_safepoint(), "should be at safepoint"); | |
| 1688 assert(!Thread::current()->is_ConcurrentGC_thread(), | |
| 1689 "shouldn't try to acquire control from self!"); | |
| 1690 | |
| 1691 // Start the protocol for acquiring control of the | |
| 1692 // collection from the background collector (aka CMS thread). | |
| 1693 assert(ConcurrentMarkSweepThread::vm_thread_has_cms_token(), | |
| 1694 "VM thread should have CMS token"); | |
| 1695 // Remember the possibly interrupted state of an ongoing | |
| 1696 // concurrent collection | |
| 1697 CollectorState first_state = _collectorState; | |
| 1698 | |
| 1699 // Signal to a possibly ongoing concurrent collection that | |
| 1700 // we want to do a foreground collection. | |
| 1701 _foregroundGCIsActive = true; | |
| 1702 | |
| 1703 // Disable incremental mode during a foreground collection. | |
| 1704 ICMSDisabler icms_disabler; | |
| 1705 | |
| 1706 // release locks and wait for a notify from the background collector | |
| 1707 // releasing the locks in only necessary for phases which | |
| 1708 // do yields to improve the granularity of the collection. | |
| 1709 assert_lock_strong(bitMapLock()); | |
| 1710 // We need to lock the Free list lock for the space that we are | |
| 1711 // currently collecting. | |
| 1712 assert(haveFreelistLocks(), "Must be holding free list locks"); | |
| 1713 bitMapLock()->unlock(); | |
| 1714 releaseFreelistLocks(); | |
| 1715 { | |
| 1716 MutexLockerEx x(CGC_lock, Mutex::_no_safepoint_check_flag); | |
| 1717 if (_foregroundGCShouldWait) { | |
| 1718 // We are going to be waiting for action for the CMS thread; | |
| 1719 // it had better not be gone (for instance at shutdown)! | |
| 1720 assert(ConcurrentMarkSweepThread::cmst() != NULL, | |
| 1721 "CMS thread must be running"); | |
| 1722 // Wait here until the background collector gives us the go-ahead | |
| 1723 ConcurrentMarkSweepThread::clear_CMS_flag( | |
| 1724 ConcurrentMarkSweepThread::CMS_vm_has_token); // release token | |
| 1725 // Get a possibly blocked CMS thread going: | |
| 1726 // Note that we set _foregroundGCIsActive true above, | |
| 1727 // without protection of the CGC_lock. | |
| 1728 CGC_lock->notify(); | |
| 1729 assert(!ConcurrentMarkSweepThread::vm_thread_wants_cms_token(), | |
| 1730 "Possible deadlock"); | |
| 1731 while (_foregroundGCShouldWait) { | |
| 1732 // wait for notification | |
| 1733 CGC_lock->wait(Mutex::_no_safepoint_check_flag); | |
| 1734 // Possibility of delay/starvation here, since CMS token does | |
| 1735 // not know to give priority to VM thread? Actually, i think | |
| 1736 // there wouldn't be any delay/starvation, but the proof of | |
| 1737 // that "fact" (?) appears non-trivial. XXX 20011219YSR | |
| 1738 } | |
| 1739 ConcurrentMarkSweepThread::set_CMS_flag( | |
| 1740 ConcurrentMarkSweepThread::CMS_vm_has_token); | |
| 1741 } | |
| 1742 } | |
| 1743 // The CMS_token is already held. Get back the other locks. | |
| 1744 assert(ConcurrentMarkSweepThread::vm_thread_has_cms_token(), | |
| 1745 "VM thread should have CMS token"); | |
| 1746 getFreelistLocks(); | |
| 1747 bitMapLock()->lock_without_safepoint_check(); | |
| 1748 if (TraceCMSState) { | |
| 1749 gclog_or_tty->print_cr("CMS foreground collector has asked for control " | |
| 1750 INTPTR_FORMAT " with first state %d", Thread::current(), first_state); | |
| 1751 gclog_or_tty->print_cr(" gets control with state %d", _collectorState); | |
| 1752 } | |
| 1753 | |
| 1754 // Check if we need to do a compaction, or if not, whether | |
| 1755 // we need to start the mark-sweep from scratch. | |
| 1756 bool should_compact = false; | |
| 1757 bool should_start_over = false; | |
| 1758 decide_foreground_collection_type(clear_all_soft_refs, | |
| 1759 &should_compact, &should_start_over); | |
| 1760 | |
| 1761 NOT_PRODUCT( | |
| 1762 if (RotateCMSCollectionTypes) { | |
| 1763 if (_cmsGen->debug_collection_type() == | |
| 1764 ConcurrentMarkSweepGeneration::MSC_foreground_collection_type) { | |
| 1765 should_compact = true; | |
| 1766 } else if (_cmsGen->debug_collection_type() == | |
| 1767 ConcurrentMarkSweepGeneration::MS_foreground_collection_type) { | |
| 1768 should_compact = false; | |
| 1769 } | |
| 1770 } | |
| 1771 ) | |
| 1772 | |
| 1773 if (PrintGCDetails && first_state > Idling) { | |
| 1774 GCCause::Cause cause = GenCollectedHeap::heap()->gc_cause(); | |
| 1775 if (GCCause::is_user_requested_gc(cause) || | |
| 1776 GCCause::is_serviceability_requested_gc(cause)) { | |
| 1777 gclog_or_tty->print(" (concurrent mode interrupted)"); | |
| 1778 } else { | |
| 1779 gclog_or_tty->print(" (concurrent mode failure)"); | |
| 1780 } | |
| 1781 } | |
| 1782 | |
| 1783 if (should_compact) { | |
| 1784 // If the collection is being acquired from the background | |
| 1785 // collector, there may be references on the discovered | |
| 1786 // references lists that have NULL referents (being those | |
| 1787 // that were concurrently cleared by a mutator) or | |
| 1788 // that are no longer active (having been enqueued concurrently | |
| 1789 // by the mutator). | |
| 1790 // Scrub the list of those references because Mark-Sweep-Compact | |
| 1791 // code assumes referents are not NULL and that all discovered | |
| 1792 // Reference objects are active. | |
| 1793 ref_processor()->clean_up_discovered_references(); | |
| 1794 | |
| 1795 do_compaction_work(clear_all_soft_refs); | |
| 1796 | |
| 1797 // Has the GC time limit been exceeded? | |
| 1798 check_gc_time_limit(); | |
| 1799 | |
| 1800 } else { | |
| 1801 do_mark_sweep_work(clear_all_soft_refs, first_state, | |
| 1802 should_start_over); | |
| 1803 } | |
| 1804 // Reset the expansion cause, now that we just completed | |
| 1805 // a collection cycle. | |
| 1806 clear_expansion_cause(); | |
| 1807 _foregroundGCIsActive = false; | |
| 1808 return; | |
| 1809 } | |
| 1810 | |
| 1811 void CMSCollector::check_gc_time_limit() { | |
| 1812 | |
| 1813 // Ignore explicit GC's. Exiting here does not set the flag and | |
| 1814 // does not reset the count. Updating of the averages for system | |
| 1815 // GC's is still controlled by UseAdaptiveSizePolicyWithSystemGC. | |
| 1816 GCCause::Cause gc_cause = GenCollectedHeap::heap()->gc_cause(); | |
| 1817 if (GCCause::is_user_requested_gc(gc_cause) || | |
| 1818 GCCause::is_serviceability_requested_gc(gc_cause)) { | |
| 1819 return; | |
| 1820 } | |
| 1821 | |
| 1822 // Calculate the fraction of the CMS generation was freed during | |
| 1823 // the last collection. | |
| 1824 // Only consider the STW compacting cost for now. | |
| 1825 // | |
| 1826 // Note that the gc time limit test only works for the collections | |
| 1827 // of the young gen + tenured gen and not for collections of the | |
| 1828 // permanent gen. That is because the calculation of the space | |
| 1829 // freed by the collection is the free space in the young gen + | |
| 1830 // tenured gen. | |
| 1831 | |
| 1832 double fraction_free = | |
| 1833 ((double)_cmsGen->free())/((double)_cmsGen->max_capacity()); | |
| 1834 if ((100.0 * size_policy()->compacting_gc_cost()) > | |
| 1835 ((double) GCTimeLimit) && | |
| 1836 ((fraction_free * 100) < GCHeapFreeLimit)) { | |
| 1837 size_policy()->inc_gc_time_limit_count(); | |
| 1838 if (UseGCOverheadLimit && | |
| 1839 (size_policy()->gc_time_limit_count() > | |
| 1840 AdaptiveSizePolicyGCTimeLimitThreshold)) { | |
| 1841 size_policy()->set_gc_time_limit_exceeded(true); | |
| 1842 // Avoid consecutive OOM due to the gc time limit by resetting | |
| 1843 // the counter. | |
| 1844 size_policy()->reset_gc_time_limit_count(); | |
| 1845 if (PrintGCDetails) { | |
| 1846 gclog_or_tty->print_cr(" GC is exceeding overhead limit " | |
| 1847 "of %d%%", GCTimeLimit); | |
| 1848 } | |
| 1849 } else { | |
| 1850 if (PrintGCDetails) { | |
| 1851 gclog_or_tty->print_cr(" GC would exceed overhead limit " | |
| 1852 "of %d%%", GCTimeLimit); | |
| 1853 } | |
| 1854 } | |
| 1855 } else { | |
| 1856 size_policy()->reset_gc_time_limit_count(); | |
| 1857 } | |
| 1858 } | |
| 1859 | |
| 1860 // Resize the perm generation and the tenured generation | |
| 1861 // after obtaining the free list locks for the | |
| 1862 // two generations. | |
| 1863 void CMSCollector::compute_new_size() { | |
| 1864 assert_locked_or_safepoint(Heap_lock); | |
| 1865 FreelistLocker z(this); | |
| 1866 _permGen->compute_new_size(); | |
| 1867 _cmsGen->compute_new_size(); | |
| 1868 } | |
| 1869 | |
| 1870 // A work method used by foreground collection to determine | |
| 1871 // what type of collection (compacting or not, continuing or fresh) | |
| 1872 // it should do. | |
| 1873 // NOTE: the intent is to make UseCMSCompactAtFullCollection | |
| 1874 // and CMSCompactWhenClearAllSoftRefs the default in the future | |
| 1875 // and do away with the flags after a suitable period. | |
| 1876 void CMSCollector::decide_foreground_collection_type( | |
| 1877 bool clear_all_soft_refs, bool* should_compact, | |
| 1878 bool* should_start_over) { | |
| 1879 // Normally, we'll compact only if the UseCMSCompactAtFullCollection | |
| 1880 // flag is set, and we have either requested a System.gc() or | |
| 1881 // the number of full gc's since the last concurrent cycle | |
| 1882 // has exceeded the threshold set by CMSFullGCsBeforeCompaction, | |
| 1883 // or if an incremental collection has failed | |
| 1884 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 1885 assert(gch->collector_policy()->is_two_generation_policy(), | |
| 1886 "You may want to check the correctness of the following"); | |
| 1887 // Inform cms gen if this was due to partial collection failing. | |
| 1888 // The CMS gen may use this fact to determine its expansion policy. | |
| 1889 if (gch->incremental_collection_will_fail()) { | |
| 1890 assert(!_cmsGen->incremental_collection_failed(), | |
| 1891 "Should have been noticed, reacted to and cleared"); | |
| 1892 _cmsGen->set_incremental_collection_failed(); | |
| 1893 } | |
| 1894 *should_compact = | |
| 1895 UseCMSCompactAtFullCollection && | |
| 1896 ((_full_gcs_since_conc_gc >= CMSFullGCsBeforeCompaction) || | |
| 1897 GCCause::is_user_requested_gc(gch->gc_cause()) || | |
| 1898 gch->incremental_collection_will_fail()); | |
| 1899 *should_start_over = false; | |
| 1900 if (clear_all_soft_refs && !*should_compact) { | |
| 1901 // We are about to do a last ditch collection attempt | |
| 1902 // so it would normally make sense to do a compaction | |
| 1903 // to reclaim as much space as possible. | |
| 1904 if (CMSCompactWhenClearAllSoftRefs) { | |
| 1905 // Default: The rationale is that in this case either | |
| 1906 // we are past the final marking phase, in which case | |
| 1907 // we'd have to start over, or so little has been done | |
| 1908 // that there's little point in saving that work. Compaction | |
| 1909 // appears to be the sensible choice in either case. | |
| 1910 *should_compact = true; | |
| 1911 } else { | |
| 1912 // We have been asked to clear all soft refs, but not to | |
| 1913 // compact. Make sure that we aren't past the final checkpoint | |
| 1914 // phase, for that is where we process soft refs. If we are already | |
| 1915 // past that phase, we'll need to redo the refs discovery phase and | |
| 1916 // if necessary clear soft refs that weren't previously | |
| 1917 // cleared. We do so by remembering the phase in which | |
| 1918 // we came in, and if we are past the refs processing | |
| 1919 // phase, we'll choose to just redo the mark-sweep | |
| 1920 // collection from scratch. | |
| 1921 if (_collectorState > FinalMarking) { | |
| 1922 // We are past the refs processing phase; | |
| 1923 // start over and do a fresh synchronous CMS cycle | |
| 1924 _collectorState = Resetting; // skip to reset to start new cycle | |
| 1925 reset(false /* == !asynch */); | |
| 1926 *should_start_over = true; | |
| 1927 } // else we can continue a possibly ongoing current cycle | |
| 1928 } | |
| 1929 } | |
| 1930 } | |
| 1931 | |
| 1932 // A work method used by the foreground collector to do | |
| 1933 // a mark-sweep-compact. | |
| 1934 void CMSCollector::do_compaction_work(bool clear_all_soft_refs) { | |
| 1935 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 1936 TraceTime t("CMS:MSC ", PrintGCDetails && Verbose, true, gclog_or_tty); | |
| 1937 if (PrintGC && Verbose && !(GCCause::is_user_requested_gc(gch->gc_cause()))) { | |
| 1938 gclog_or_tty->print_cr("Compact ConcurrentMarkSweepGeneration after %d " | |
| 1939 "collections passed to foreground collector", _full_gcs_since_conc_gc); | |
| 1940 } | |
| 1941 | |
| 1942 // Sample collection interval time and reset for collection pause. | |
| 1943 if (UseAdaptiveSizePolicy) { | |
| 1944 size_policy()->msc_collection_begin(); | |
| 1945 } | |
| 1946 | |
| 1947 // Temporarily widen the span of the weak reference processing to | |
| 1948 // the entire heap. | |
| 1949 MemRegion new_span(GenCollectedHeap::heap()->reserved_region()); | |
| 1950 ReferenceProcessorSpanMutator x(ref_processor(), new_span); | |
| 1951 | |
| 1952 // Temporarily, clear the "is_alive_non_header" field of the | |
| 1953 // reference processor. | |
| 1954 ReferenceProcessorIsAliveMutator y(ref_processor(), NULL); | |
| 1955 | |
| 1956 // Temporarily make reference _processing_ single threaded (non-MT). | |
| 1957 ReferenceProcessorMTProcMutator z(ref_processor(), false); | |
| 1958 | |
| 1959 // Temporarily make refs discovery atomic | |
| 1960 ReferenceProcessorAtomicMutator w(ref_processor(), true); | |
| 1961 | |
| 1962 ref_processor()->set_enqueuing_is_done(false); | |
| 1963 ref_processor()->enable_discovery(); | |
| 1964 // If an asynchronous collection finishes, the _modUnionTable is | |
| 1965 // all clear. If we are assuming the collection from an asynchronous | |
| 1966 // collection, clear the _modUnionTable. | |
| 1967 assert(_collectorState != Idling || _modUnionTable.isAllClear(), | |
| 1968 "_modUnionTable should be clear if the baton was not passed"); | |
| 1969 _modUnionTable.clear_all(); | |
| 1970 | |
| 1971 // We must adjust the allocation statistics being maintained | |
| 1972 // in the free list space. We do so by reading and clearing | |
| 1973 // the sweep timer and updating the block flux rate estimates below. | |
| 1974 assert(_sweep_timer.is_active(), "We should never see the timer inactive"); | |
| 1975 _sweep_timer.stop(); | |
| 1976 // Note that we do not use this sample to update the _sweep_estimate. | |
| 1977 _cmsGen->cmsSpace()->beginSweepFLCensus((float)(_sweep_timer.seconds()), | |
| 1978 _sweep_estimate.padded_average()); | |
| 1979 | |
| 1980 GenMarkSweep::invoke_at_safepoint(_cmsGen->level(), | |
| 1981 ref_processor(), clear_all_soft_refs); | |
| 1982 #ifdef ASSERT | |
| 1983 CompactibleFreeListSpace* cms_space = _cmsGen->cmsSpace(); | |
| 1984 size_t free_size = cms_space->free(); | |
| 1985 assert(free_size == | |
| 1986 pointer_delta(cms_space->end(), cms_space->compaction_top()) | |
| 1987 * HeapWordSize, | |
| 1988 "All the free space should be compacted into one chunk at top"); | |
| 1989 assert(cms_space->dictionary()->totalChunkSize( | |
| 1990 debug_only(cms_space->freelistLock())) == 0 || | |
| 1991 cms_space->totalSizeInIndexedFreeLists() == 0, | |
| 1992 "All the free space should be in a single chunk"); | |
| 1993 size_t num = cms_space->totalCount(); | |
| 1994 assert((free_size == 0 && num == 0) || | |
| 1995 (free_size > 0 && (num == 1 || num == 2)), | |
| 1996 "There should be at most 2 free chunks after compaction"); | |
| 1997 #endif // ASSERT | |
| 1998 _collectorState = Resetting; | |
| 1999 assert(_restart_addr == NULL, | |
| 2000 "Should have been NULL'd before baton was passed"); | |
| 2001 reset(false /* == !asynch */); | |
| 2002 _cmsGen->reset_after_compaction(); | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
2003 _concurrent_cycles_since_last_unload = 0; |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
2004 |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
2005 if (verifying() && !should_unload_classes()) { |
| 0 | 2006 perm_gen_verify_bit_map()->clear_all(); |
| 2007 } | |
| 2008 | |
| 2009 // Clear any data recorded in the PLAB chunk arrays. | |
| 2010 if (_survivor_plab_array != NULL) { | |
| 2011 reset_survivor_plab_arrays(); | |
| 2012 } | |
| 2013 | |
| 2014 // Adjust the per-size allocation stats for the next epoch. | |
| 2015 _cmsGen->cmsSpace()->endSweepFLCensus(sweepCount() /* fake */); | |
| 2016 // Restart the "sweep timer" for next epoch. | |
| 2017 _sweep_timer.reset(); | |
| 2018 _sweep_timer.start(); | |
| 2019 | |
| 2020 // Sample collection pause time and reset for collection interval. | |
| 2021 if (UseAdaptiveSizePolicy) { | |
| 2022 size_policy()->msc_collection_end(gch->gc_cause()); | |
| 2023 } | |
| 2024 | |
| 2025 // For a mark-sweep-compact, compute_new_size() will be called | |
| 2026 // in the heap's do_collection() method. | |
| 2027 } | |
| 2028 | |
| 2029 // A work method used by the foreground collector to do | |
| 2030 // a mark-sweep, after taking over from a possibly on-going | |
| 2031 // concurrent mark-sweep collection. | |
| 2032 void CMSCollector::do_mark_sweep_work(bool clear_all_soft_refs, | |
| 2033 CollectorState first_state, bool should_start_over) { | |
| 2034 if (PrintGC && Verbose) { | |
| 2035 gclog_or_tty->print_cr("Pass concurrent collection to foreground " | |
| 2036 "collector with count %d", | |
| 2037 _full_gcs_since_conc_gc); | |
| 2038 } | |
| 2039 switch (_collectorState) { | |
| 2040 case Idling: | |
| 2041 if (first_state == Idling || should_start_over) { | |
| 2042 // The background GC was not active, or should | |
| 2043 // restarted from scratch; start the cycle. | |
| 2044 _collectorState = InitialMarking; | |
| 2045 } | |
| 2046 // If first_state was not Idling, then a background GC | |
| 2047 // was in progress and has now finished. No need to do it | |
| 2048 // again. Leave the state as Idling. | |
| 2049 break; | |
| 2050 case Precleaning: | |
| 2051 // In the foreground case don't do the precleaning since | |
| 2052 // it is not done concurrently and there is extra work | |
| 2053 // required. | |
| 2054 _collectorState = FinalMarking; | |
| 2055 } | |
| 2056 if (PrintGCDetails && | |
| 2057 (_collectorState > Idling || | |
| 2058 !GCCause::is_user_requested_gc(GenCollectedHeap::heap()->gc_cause()))) { | |
| 2059 gclog_or_tty->print(" (concurrent mode failure)"); | |
| 2060 } | |
| 2061 collect_in_foreground(clear_all_soft_refs); | |
| 2062 | |
| 2063 // For a mark-sweep, compute_new_size() will be called | |
| 2064 // in the heap's do_collection() method. | |
| 2065 } | |
| 2066 | |
| 2067 | |
| 2068 void CMSCollector::getFreelistLocks() const { | |
| 2069 // Get locks for all free lists in all generations that this | |
| 2070 // collector is responsible for | |
| 2071 _cmsGen->freelistLock()->lock_without_safepoint_check(); | |
| 2072 _permGen->freelistLock()->lock_without_safepoint_check(); | |
| 2073 } | |
| 2074 | |
| 2075 void CMSCollector::releaseFreelistLocks() const { | |
| 2076 // Release locks for all free lists in all generations that this | |
| 2077 // collector is responsible for | |
| 2078 _cmsGen->freelistLock()->unlock(); | |
| 2079 _permGen->freelistLock()->unlock(); | |
| 2080 } | |
| 2081 | |
| 2082 bool CMSCollector::haveFreelistLocks() const { | |
| 2083 // Check locks for all free lists in all generations that this | |
| 2084 // collector is responsible for | |
| 2085 assert_lock_strong(_cmsGen->freelistLock()); | |
| 2086 assert_lock_strong(_permGen->freelistLock()); | |
| 2087 PRODUCT_ONLY(ShouldNotReachHere()); | |
| 2088 return true; | |
| 2089 } | |
| 2090 | |
| 2091 // A utility class that is used by the CMS collector to | |
| 2092 // temporarily "release" the foreground collector from its | |
| 2093 // usual obligation to wait for the background collector to | |
| 2094 // complete an ongoing phase before proceeding. | |
| 2095 class ReleaseForegroundGC: public StackObj { | |
| 2096 private: | |
| 2097 CMSCollector* _c; | |
| 2098 public: | |
| 2099 ReleaseForegroundGC(CMSCollector* c) : _c(c) { | |
| 2100 assert(_c->_foregroundGCShouldWait, "Else should not need to call"); | |
| 2101 MutexLockerEx x(CGC_lock, Mutex::_no_safepoint_check_flag); | |
| 2102 // allow a potentially blocked foreground collector to proceed | |
| 2103 _c->_foregroundGCShouldWait = false; | |
| 2104 if (_c->_foregroundGCIsActive) { | |
| 2105 CGC_lock->notify(); | |
| 2106 } | |
| 2107 assert(!ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 2108 "Possible deadlock"); | |
| 2109 } | |
| 2110 | |
| 2111 ~ReleaseForegroundGC() { | |
| 2112 assert(!_c->_foregroundGCShouldWait, "Usage protocol violation?"); | |
| 2113 MutexLockerEx x(CGC_lock, Mutex::_no_safepoint_check_flag); | |
| 2114 _c->_foregroundGCShouldWait = true; | |
| 2115 } | |
| 2116 }; | |
| 2117 | |
| 2118 // There are separate collect_in_background and collect_in_foreground because of | |
| 2119 // the different locking requirements of the background collector and the | |
| 2120 // foreground collector. There was originally an attempt to share | |
| 2121 // one "collect" method between the background collector and the foreground | |
| 2122 // collector but the if-then-else required made it cleaner to have | |
| 2123 // separate methods. | |
| 2124 void CMSCollector::collect_in_background(bool clear_all_soft_refs) { | |
| 2125 assert(Thread::current()->is_ConcurrentGC_thread(), | |
| 2126 "A CMS asynchronous collection is only allowed on a CMS thread."); | |
| 2127 | |
| 2128 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 2129 { | |
| 2130 bool safepoint_check = Mutex::_no_safepoint_check_flag; | |
| 2131 MutexLockerEx hl(Heap_lock, safepoint_check); | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
2132 FreelistLocker fll(this); |
| 0 | 2133 MutexLockerEx x(CGC_lock, safepoint_check); |
| 2134 if (_foregroundGCIsActive || !UseAsyncConcMarkSweepGC) { | |
| 2135 // The foreground collector is active or we're | |
| 2136 // not using asynchronous collections. Skip this | |
| 2137 // background collection. | |
| 2138 assert(!_foregroundGCShouldWait, "Should be clear"); | |
| 2139 return; | |
| 2140 } else { | |
| 2141 assert(_collectorState == Idling, "Should be idling before start."); | |
| 2142 _collectorState = InitialMarking; | |
| 2143 // Reset the expansion cause, now that we are about to begin | |
| 2144 // a new cycle. | |
| 2145 clear_expansion_cause(); | |
| 2146 } | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
2147 // Decide if we want to enable class unloading as part of the |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
2148 // ensuing concurrent GC cycle. |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
2149 update_should_unload_classes(); |
| 0 | 2150 _full_gc_requested = false; // acks all outstanding full gc requests |
| 2151 // Signal that we are about to start a collection | |
| 2152 gch->increment_total_full_collections(); // ... starting a collection cycle | |
| 2153 _collection_count_start = gch->total_full_collections(); | |
| 2154 } | |
| 2155 | |
| 2156 // Used for PrintGC | |
| 2157 size_t prev_used; | |
| 2158 if (PrintGC && Verbose) { | |
| 2159 prev_used = _cmsGen->used(); // XXXPERM | |
| 2160 } | |
| 2161 | |
| 2162 // The change of the collection state is normally done at this level; | |
| 2163 // the exceptions are phases that are executed while the world is | |
| 2164 // stopped. For those phases the change of state is done while the | |
| 2165 // world is stopped. For baton passing purposes this allows the | |
| 2166 // background collector to finish the phase and change state atomically. | |
| 2167 // The foreground collector cannot wait on a phase that is done | |
| 2168 // while the world is stopped because the foreground collector already | |
| 2169 // has the world stopped and would deadlock. | |
| 2170 while (_collectorState != Idling) { | |
| 2171 if (TraceCMSState) { | |
| 2172 gclog_or_tty->print_cr("Thread " INTPTR_FORMAT " in CMS state %d", | |
| 2173 Thread::current(), _collectorState); | |
| 2174 } | |
| 2175 // The foreground collector | |
| 2176 // holds the Heap_lock throughout its collection. | |
| 2177 // holds the CMS token (but not the lock) | |
| 2178 // except while it is waiting for the background collector to yield. | |
| 2179 // | |
| 2180 // The foreground collector should be blocked (not for long) | |
| 2181 // if the background collector is about to start a phase | |
| 2182 // executed with world stopped. If the background | |
| 2183 // collector has already started such a phase, the | |
| 2184 // foreground collector is blocked waiting for the | |
| 2185 // Heap_lock. The stop-world phases (InitialMarking and FinalMarking) | |
| 2186 // are executed in the VM thread. | |
| 2187 // | |
| 2188 // The locking order is | |
| 2189 // PendingListLock (PLL) -- if applicable (FinalMarking) | |
| 2190 // Heap_lock (both this & PLL locked in VM_CMS_Operation::prologue()) | |
| 2191 // CMS token (claimed in | |
| 2192 // stop_world_and_do() --> | |
| 2193 // safepoint_synchronize() --> | |
| 2194 // CMSThread::synchronize()) | |
| 2195 | |
| 2196 { | |
| 2197 // Check if the FG collector wants us to yield. | |
| 2198 CMSTokenSync x(true); // is cms thread | |
| 2199 if (waitForForegroundGC()) { | |
| 2200 // We yielded to a foreground GC, nothing more to be | |
| 2201 // done this round. | |
| 2202 assert(_foregroundGCShouldWait == false, "We set it to false in " | |
| 2203 "waitForForegroundGC()"); | |
| 2204 if (TraceCMSState) { | |
| 2205 gclog_or_tty->print_cr("CMS Thread " INTPTR_FORMAT | |
| 2206 " exiting collection CMS state %d", | |
| 2207 Thread::current(), _collectorState); | |
| 2208 } | |
| 2209 return; | |
| 2210 } else { | |
| 2211 // The background collector can run but check to see if the | |
| 2212 // foreground collector has done a collection while the | |
| 2213 // background collector was waiting to get the CGC_lock | |
| 2214 // above. If yes, break so that _foregroundGCShouldWait | |
| 2215 // is cleared before returning. | |
| 2216 if (_collectorState == Idling) { | |
| 2217 break; | |
| 2218 } | |
| 2219 } | |
| 2220 } | |
| 2221 | |
| 2222 assert(_foregroundGCShouldWait, "Foreground collector, if active, " | |
| 2223 "should be waiting"); | |
| 2224 | |
| 2225 switch (_collectorState) { | |
| 2226 case InitialMarking: | |
| 2227 { | |
| 2228 ReleaseForegroundGC x(this); | |
| 2229 stats().record_cms_begin(); | |
| 2230 | |
| 2231 VM_CMS_Initial_Mark initial_mark_op(this); | |
| 2232 VMThread::execute(&initial_mark_op); | |
| 2233 } | |
| 2234 // The collector state may be any legal state at this point | |
| 2235 // since the background collector may have yielded to the | |
| 2236 // foreground collector. | |
| 2237 break; | |
| 2238 case Marking: | |
| 2239 // initial marking in checkpointRootsInitialWork has been completed | |
| 2240 if (markFromRoots(true)) { // we were successful | |
| 2241 assert(_collectorState == Precleaning, "Collector state should " | |
| 2242 "have changed"); | |
| 2243 } else { | |
| 2244 assert(_foregroundGCIsActive, "Internal state inconsistency"); | |
| 2245 } | |
| 2246 break; | |
| 2247 case Precleaning: | |
| 2248 if (UseAdaptiveSizePolicy) { | |
| 2249 size_policy()->concurrent_precleaning_begin(); | |
| 2250 } | |
| 2251 // marking from roots in markFromRoots has been completed | |
| 2252 preclean(); | |
| 2253 if (UseAdaptiveSizePolicy) { | |
| 2254 size_policy()->concurrent_precleaning_end(); | |
| 2255 } | |
| 2256 assert(_collectorState == AbortablePreclean || | |
| 2257 _collectorState == FinalMarking, | |
| 2258 "Collector state should have changed"); | |
| 2259 break; | |
| 2260 case AbortablePreclean: | |
| 2261 if (UseAdaptiveSizePolicy) { | |
| 2262 size_policy()->concurrent_phases_resume(); | |
| 2263 } | |
| 2264 abortable_preclean(); | |
| 2265 if (UseAdaptiveSizePolicy) { | |
| 2266 size_policy()->concurrent_precleaning_end(); | |
| 2267 } | |
| 2268 assert(_collectorState == FinalMarking, "Collector state should " | |
| 2269 "have changed"); | |
| 2270 break; | |
| 2271 case FinalMarking: | |
| 2272 { | |
| 2273 ReleaseForegroundGC x(this); | |
| 2274 | |
| 2275 VM_CMS_Final_Remark final_remark_op(this); | |
| 2276 VMThread::execute(&final_remark_op); | |
| 2277 } | |
| 2278 assert(_foregroundGCShouldWait, "block post-condition"); | |
| 2279 break; | |
| 2280 case Sweeping: | |
| 2281 if (UseAdaptiveSizePolicy) { | |
| 2282 size_policy()->concurrent_sweeping_begin(); | |
| 2283 } | |
| 2284 // final marking in checkpointRootsFinal has been completed | |
| 2285 sweep(true); | |
| 2286 assert(_collectorState == Resizing, "Collector state change " | |
| 2287 "to Resizing must be done under the free_list_lock"); | |
| 2288 _full_gcs_since_conc_gc = 0; | |
| 2289 | |
| 2290 // Stop the timers for adaptive size policy for the concurrent phases | |
| 2291 if (UseAdaptiveSizePolicy) { | |
| 2292 size_policy()->concurrent_sweeping_end(); | |
| 2293 size_policy()->concurrent_phases_end(gch->gc_cause(), | |
| 2294 gch->prev_gen(_cmsGen)->capacity(), | |
| 2295 _cmsGen->free()); | |
| 2296 } | |
| 2297 | |
| 2298 case Resizing: { | |
| 2299 // Sweeping has been completed... | |
| 2300 // At this point the background collection has completed. | |
| 2301 // Don't move the call to compute_new_size() down | |
| 2302 // into code that might be executed if the background | |
| 2303 // collection was preempted. | |
| 2304 { | |
| 2305 ReleaseForegroundGC x(this); // unblock FG collection | |
| 2306 MutexLockerEx y(Heap_lock, Mutex::_no_safepoint_check_flag); | |
| 2307 CMSTokenSync z(true); // not strictly needed. | |
| 2308 if (_collectorState == Resizing) { | |
| 2309 compute_new_size(); | |
| 2310 _collectorState = Resetting; | |
| 2311 } else { | |
| 2312 assert(_collectorState == Idling, "The state should only change" | |
| 2313 " because the foreground collector has finished the collection"); | |
| 2314 } | |
| 2315 } | |
| 2316 break; | |
| 2317 } | |
| 2318 case Resetting: | |
| 2319 // CMS heap resizing has been completed | |
| 2320 reset(true); | |
| 2321 assert(_collectorState == Idling, "Collector state should " | |
| 2322 "have changed"); | |
| 2323 stats().record_cms_end(); | |
| 2324 // Don't move the concurrent_phases_end() and compute_new_size() | |
| 2325 // calls to here because a preempted background collection | |
| 2326 // has it's state set to "Resetting". | |
| 2327 break; | |
| 2328 case Idling: | |
| 2329 default: | |
| 2330 ShouldNotReachHere(); | |
| 2331 break; | |
| 2332 } | |
| 2333 if (TraceCMSState) { | |
| 2334 gclog_or_tty->print_cr(" Thread " INTPTR_FORMAT " done - next CMS state %d", | |
| 2335 Thread::current(), _collectorState); | |
| 2336 } | |
| 2337 assert(_foregroundGCShouldWait, "block post-condition"); | |
| 2338 } | |
| 2339 | |
| 2340 // Should this be in gc_epilogue? | |
| 2341 collector_policy()->counters()->update_counters(); | |
| 2342 | |
| 2343 { | |
| 2344 // Clear _foregroundGCShouldWait and, in the event that the | |
| 2345 // foreground collector is waiting, notify it, before | |
| 2346 // returning. | |
| 2347 MutexLockerEx x(CGC_lock, Mutex::_no_safepoint_check_flag); | |
| 2348 _foregroundGCShouldWait = false; | |
| 2349 if (_foregroundGCIsActive) { | |
| 2350 CGC_lock->notify(); | |
| 2351 } | |
| 2352 assert(!ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 2353 "Possible deadlock"); | |
| 2354 } | |
| 2355 if (TraceCMSState) { | |
| 2356 gclog_or_tty->print_cr("CMS Thread " INTPTR_FORMAT | |
| 2357 " exiting collection CMS state %d", | |
| 2358 Thread::current(), _collectorState); | |
| 2359 } | |
| 2360 if (PrintGC && Verbose) { | |
| 2361 _cmsGen->print_heap_change(prev_used); | |
| 2362 } | |
| 2363 } | |
| 2364 | |
| 2365 void CMSCollector::collect_in_foreground(bool clear_all_soft_refs) { | |
| 2366 assert(_foregroundGCIsActive && !_foregroundGCShouldWait, | |
| 2367 "Foreground collector should be waiting, not executing"); | |
| 2368 assert(Thread::current()->is_VM_thread(), "A foreground collection" | |
| 2369 "may only be done by the VM Thread with the world stopped"); | |
| 2370 assert(ConcurrentMarkSweepThread::vm_thread_has_cms_token(), | |
| 2371 "VM thread should have CMS token"); | |
| 2372 | |
| 2373 NOT_PRODUCT(TraceTime t("CMS:MS (foreground) ", PrintGCDetails && Verbose, | |
| 2374 true, gclog_or_tty);) | |
| 2375 if (UseAdaptiveSizePolicy) { | |
| 2376 size_policy()->ms_collection_begin(); | |
| 2377 } | |
| 2378 COMPILER2_PRESENT(DerivedPointerTableDeactivate dpt_deact); | |
| 2379 | |
| 2380 HandleMark hm; // Discard invalid handles created during verification | |
| 2381 | |
| 2382 if (VerifyBeforeGC && | |
| 2383 GenCollectedHeap::heap()->total_collections() >= VerifyGCStartAt) { | |
| 2384 Universe::verify(true); | |
| 2385 } | |
| 2386 | |
| 2387 bool init_mark_was_synchronous = false; // until proven otherwise | |
| 2388 while (_collectorState != Idling) { | |
| 2389 if (TraceCMSState) { | |
| 2390 gclog_or_tty->print_cr("Thread " INTPTR_FORMAT " in CMS state %d", | |
| 2391 Thread::current(), _collectorState); | |
| 2392 } | |
| 2393 switch (_collectorState) { | |
| 2394 case InitialMarking: | |
| 2395 init_mark_was_synchronous = true; // fact to be exploited in re-mark | |
| 2396 checkpointRootsInitial(false); | |
| 2397 assert(_collectorState == Marking, "Collector state should have changed" | |
| 2398 " within checkpointRootsInitial()"); | |
| 2399 break; | |
| 2400 case Marking: | |
| 2401 // initial marking in checkpointRootsInitialWork has been completed | |
| 2402 if (VerifyDuringGC && | |
| 2403 GenCollectedHeap::heap()->total_collections() >= VerifyGCStartAt) { | |
| 2404 gclog_or_tty->print("Verify before initial mark: "); | |
| 2405 Universe::verify(true); | |
| 2406 } | |
| 2407 { | |
| 2408 bool res = markFromRoots(false); | |
| 2409 assert(res && _collectorState == FinalMarking, "Collector state should " | |
| 2410 "have changed"); | |
| 2411 break; | |
| 2412 } | |
| 2413 case FinalMarking: | |
| 2414 if (VerifyDuringGC && | |
| 2415 GenCollectedHeap::heap()->total_collections() >= VerifyGCStartAt) { | |
| 2416 gclog_or_tty->print("Verify before re-mark: "); | |
| 2417 Universe::verify(true); | |
| 2418 } | |
| 2419 checkpointRootsFinal(false, clear_all_soft_refs, | |
| 2420 init_mark_was_synchronous); | |
| 2421 assert(_collectorState == Sweeping, "Collector state should not " | |
| 2422 "have changed within checkpointRootsFinal()"); | |
| 2423 break; | |
| 2424 case Sweeping: | |
| 2425 // final marking in checkpointRootsFinal has been completed | |
| 2426 if (VerifyDuringGC && | |
| 2427 GenCollectedHeap::heap()->total_collections() >= VerifyGCStartAt) { | |
| 2428 gclog_or_tty->print("Verify before sweep: "); | |
| 2429 Universe::verify(true); | |
| 2430 } | |
| 2431 sweep(false); | |
| 2432 assert(_collectorState == Resizing, "Incorrect state"); | |
| 2433 break; | |
| 2434 case Resizing: { | |
| 2435 // Sweeping has been completed; the actual resize in this case | |
| 2436 // is done separately; nothing to be done in this state. | |
| 2437 _collectorState = Resetting; | |
| 2438 break; | |
| 2439 } | |
| 2440 case Resetting: | |
| 2441 // The heap has been resized. | |
| 2442 if (VerifyDuringGC && | |
| 2443 GenCollectedHeap::heap()->total_collections() >= VerifyGCStartAt) { | |
| 2444 gclog_or_tty->print("Verify before reset: "); | |
| 2445 Universe::verify(true); | |
| 2446 } | |
| 2447 reset(false); | |
| 2448 assert(_collectorState == Idling, "Collector state should " | |
| 2449 "have changed"); | |
| 2450 break; | |
| 2451 case Precleaning: | |
| 2452 case AbortablePreclean: | |
| 2453 // Elide the preclean phase | |
| 2454 _collectorState = FinalMarking; | |
| 2455 break; | |
| 2456 default: | |
| 2457 ShouldNotReachHere(); | |
| 2458 } | |
| 2459 if (TraceCMSState) { | |
| 2460 gclog_or_tty->print_cr(" Thread " INTPTR_FORMAT " done - next CMS state %d", | |
| 2461 Thread::current(), _collectorState); | |
| 2462 } | |
| 2463 } | |
| 2464 | |
| 2465 if (UseAdaptiveSizePolicy) { | |
| 2466 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 2467 size_policy()->ms_collection_end(gch->gc_cause()); | |
| 2468 } | |
| 2469 | |
| 2470 if (VerifyAfterGC && | |
| 2471 GenCollectedHeap::heap()->total_collections() >= VerifyGCStartAt) { | |
| 2472 Universe::verify(true); | |
| 2473 } | |
| 2474 if (TraceCMSState) { | |
| 2475 gclog_or_tty->print_cr("CMS Thread " INTPTR_FORMAT | |
| 2476 " exiting collection CMS state %d", | |
| 2477 Thread::current(), _collectorState); | |
| 2478 } | |
| 2479 } | |
| 2480 | |
| 2481 bool CMSCollector::waitForForegroundGC() { | |
| 2482 bool res = false; | |
| 2483 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 2484 "CMS thread should have CMS token"); | |
| 2485 // Block the foreground collector until the | |
| 2486 // background collectors decides whether to | |
| 2487 // yield. | |
| 2488 MutexLockerEx x(CGC_lock, Mutex::_no_safepoint_check_flag); | |
| 2489 _foregroundGCShouldWait = true; | |
| 2490 if (_foregroundGCIsActive) { | |
| 2491 // The background collector yields to the | |
| 2492 // foreground collector and returns a value | |
| 2493 // indicating that it has yielded. The foreground | |
| 2494 // collector can proceed. | |
| 2495 res = true; | |
| 2496 _foregroundGCShouldWait = false; | |
| 2497 ConcurrentMarkSweepThread::clear_CMS_flag( | |
| 2498 ConcurrentMarkSweepThread::CMS_cms_has_token); | |
| 2499 ConcurrentMarkSweepThread::set_CMS_flag( | |
| 2500 ConcurrentMarkSweepThread::CMS_cms_wants_token); | |
| 2501 // Get a possibly blocked foreground thread going | |
| 2502 CGC_lock->notify(); | |
| 2503 if (TraceCMSState) { | |
| 2504 gclog_or_tty->print_cr("CMS Thread " INTPTR_FORMAT " waiting at CMS state %d", | |
| 2505 Thread::current(), _collectorState); | |
| 2506 } | |
| 2507 while (_foregroundGCIsActive) { | |
| 2508 CGC_lock->wait(Mutex::_no_safepoint_check_flag); | |
| 2509 } | |
| 2510 ConcurrentMarkSweepThread::set_CMS_flag( | |
| 2511 ConcurrentMarkSweepThread::CMS_cms_has_token); | |
| 2512 ConcurrentMarkSweepThread::clear_CMS_flag( | |
| 2513 ConcurrentMarkSweepThread::CMS_cms_wants_token); | |
| 2514 } | |
| 2515 if (TraceCMSState) { | |
| 2516 gclog_or_tty->print_cr("CMS Thread " INTPTR_FORMAT " continuing at CMS state %d", | |
| 2517 Thread::current(), _collectorState); | |
| 2518 } | |
| 2519 return res; | |
| 2520 } | |
| 2521 | |
| 2522 // Because of the need to lock the free lists and other structures in | |
| 2523 // the collector, common to all the generations that the collector is | |
| 2524 // collecting, we need the gc_prologues of individual CMS generations | |
| 2525 // delegate to their collector. It may have been simpler had the | |
| 2526 // current infrastructure allowed one to call a prologue on a | |
| 2527 // collector. In the absence of that we have the generation's | |
| 2528 // prologue delegate to the collector, which delegates back | |
| 2529 // some "local" work to a worker method in the individual generations | |
| 2530 // that it's responsible for collecting, while itself doing any | |
| 2531 // work common to all generations it's responsible for. A similar | |
| 2532 // comment applies to the gc_epilogue()'s. | |
| 2533 // The role of the varaible _between_prologue_and_epilogue is to | |
| 2534 // enforce the invocation protocol. | |
| 2535 void CMSCollector::gc_prologue(bool full) { | |
| 2536 // Call gc_prologue_work() for each CMSGen and PermGen that | |
| 2537 // we are responsible for. | |
| 2538 | |
| 2539 // The following locking discipline assumes that we are only called | |
| 2540 // when the world is stopped. | |
| 2541 assert(SafepointSynchronize::is_at_safepoint(), "world is stopped assumption"); | |
| 2542 | |
| 2543 // The CMSCollector prologue must call the gc_prologues for the | |
| 2544 // "generations" (including PermGen if any) that it's responsible | |
| 2545 // for. | |
| 2546 | |
| 2547 assert( Thread::current()->is_VM_thread() | |
| 2548 || ( CMSScavengeBeforeRemark | |
| 2549 && Thread::current()->is_ConcurrentGC_thread()), | |
| 2550 "Incorrect thread type for prologue execution"); | |
| 2551 | |
| 2552 if (_between_prologue_and_epilogue) { | |
| 2553 // We have already been invoked; this is a gc_prologue delegation | |
| 2554 // from yet another CMS generation that we are responsible for, just | |
| 2555 // ignore it since all relevant work has already been done. | |
| 2556 return; | |
| 2557 } | |
| 2558 | |
| 2559 // set a bit saying prologue has been called; cleared in epilogue | |
| 2560 _between_prologue_and_epilogue = true; | |
| 2561 // Claim locks for common data structures, then call gc_prologue_work() | |
| 2562 // for each CMSGen and PermGen that we are responsible for. | |
| 2563 | |
| 2564 getFreelistLocks(); // gets free list locks on constituent spaces | |
| 2565 bitMapLock()->lock_without_safepoint_check(); | |
| 2566 | |
| 2567 // Should call gc_prologue_work() for all cms gens we are responsible for | |
| 2568 bool registerClosure = _collectorState >= Marking | |
| 2569 && _collectorState < Sweeping; | |
| 2570 ModUnionClosure* muc = ParallelGCThreads > 0 ? &_modUnionClosurePar | |
| 2571 : &_modUnionClosure; | |
| 2572 _cmsGen->gc_prologue_work(full, registerClosure, muc); | |
| 2573 _permGen->gc_prologue_work(full, registerClosure, muc); | |
| 2574 | |
| 2575 if (!full) { | |
| 2576 stats().record_gc0_begin(); | |
| 2577 } | |
| 2578 } | |
| 2579 | |
| 2580 void ConcurrentMarkSweepGeneration::gc_prologue(bool full) { | |
| 2581 // Delegate to CMScollector which knows how to coordinate between | |
| 2582 // this and any other CMS generations that it is responsible for | |
| 2583 // collecting. | |
| 2584 collector()->gc_prologue(full); | |
| 2585 } | |
| 2586 | |
| 2587 // This is a "private" interface for use by this generation's CMSCollector. | |
| 2588 // Not to be called directly by any other entity (for instance, | |
| 2589 // GenCollectedHeap, which calls the "public" gc_prologue method above). | |
| 2590 void ConcurrentMarkSweepGeneration::gc_prologue_work(bool full, | |
| 2591 bool registerClosure, ModUnionClosure* modUnionClosure) { | |
| 2592 assert(!incremental_collection_failed(), "Shouldn't be set yet"); | |
| 2593 assert(cmsSpace()->preconsumptionDirtyCardClosure() == NULL, | |
| 2594 "Should be NULL"); | |
| 2595 if (registerClosure) { | |
| 2596 cmsSpace()->setPreconsumptionDirtyCardClosure(modUnionClosure); | |
| 2597 } | |
| 2598 cmsSpace()->gc_prologue(); | |
| 2599 // Clear stat counters | |
| 2600 NOT_PRODUCT( | |
| 2601 assert(_numObjectsPromoted == 0, "check"); | |
| 2602 assert(_numWordsPromoted == 0, "check"); | |
| 2603 if (Verbose && PrintGC) { | |
| 2604 gclog_or_tty->print("Allocated "SIZE_FORMAT" objects, " | |
| 2605 SIZE_FORMAT" bytes concurrently", | |
| 2606 _numObjectsAllocated, _numWordsAllocated*sizeof(HeapWord)); | |
| 2607 } | |
| 2608 _numObjectsAllocated = 0; | |
| 2609 _numWordsAllocated = 0; | |
| 2610 ) | |
| 2611 } | |
| 2612 | |
| 2613 void CMSCollector::gc_epilogue(bool full) { | |
| 2614 // The following locking discipline assumes that we are only called | |
| 2615 // when the world is stopped. | |
| 2616 assert(SafepointSynchronize::is_at_safepoint(), | |
| 2617 "world is stopped assumption"); | |
| 2618 | |
| 2619 // Currently the CMS epilogue (see CompactibleFreeListSpace) merely checks | |
| 2620 // if linear allocation blocks need to be appropriately marked to allow the | |
| 2621 // the blocks to be parsable. We also check here whether we need to nudge the | |
| 2622 // CMS collector thread to start a new cycle (if it's not already active). | |
| 2623 assert( Thread::current()->is_VM_thread() | |
| 2624 || ( CMSScavengeBeforeRemark | |
| 2625 && Thread::current()->is_ConcurrentGC_thread()), | |
| 2626 "Incorrect thread type for epilogue execution"); | |
| 2627 | |
| 2628 if (!_between_prologue_and_epilogue) { | |
| 2629 // We have already been invoked; this is a gc_epilogue delegation | |
| 2630 // from yet another CMS generation that we are responsible for, just | |
| 2631 // ignore it since all relevant work has already been done. | |
| 2632 return; | |
| 2633 } | |
| 2634 assert(haveFreelistLocks(), "must have freelist locks"); | |
| 2635 assert_lock_strong(bitMapLock()); | |
| 2636 | |
| 2637 _cmsGen->gc_epilogue_work(full); | |
| 2638 _permGen->gc_epilogue_work(full); | |
| 2639 | |
| 2640 if (_collectorState == AbortablePreclean || _collectorState == Precleaning) { | |
| 2641 // in case sampling was not already enabled, enable it | |
| 2642 _start_sampling = true; | |
| 2643 } | |
| 2644 // reset _eden_chunk_array so sampling starts afresh | |
| 2645 _eden_chunk_index = 0; | |
| 2646 | |
| 2647 size_t cms_used = _cmsGen->cmsSpace()->used(); | |
| 2648 size_t perm_used = _permGen->cmsSpace()->used(); | |
| 2649 | |
| 2650 // update performance counters - this uses a special version of | |
| 2651 // update_counters() that allows the utilization to be passed as a | |
| 2652 // parameter, avoiding multiple calls to used(). | |
| 2653 // | |
| 2654 _cmsGen->update_counters(cms_used); | |
| 2655 _permGen->update_counters(perm_used); | |
| 2656 | |
| 2657 if (CMSIncrementalMode) { | |
| 2658 icms_update_allocation_limits(); | |
| 2659 } | |
| 2660 | |
| 2661 bitMapLock()->unlock(); | |
| 2662 releaseFreelistLocks(); | |
| 2663 | |
| 2664 _between_prologue_and_epilogue = false; // ready for next cycle | |
| 2665 } | |
| 2666 | |
| 2667 void ConcurrentMarkSweepGeneration::gc_epilogue(bool full) { | |
| 2668 collector()->gc_epilogue(full); | |
| 2669 | |
| 2670 // Also reset promotion tracking in par gc thread states. | |
| 2671 if (ParallelGCThreads > 0) { | |
| 2672 for (uint i = 0; i < ParallelGCThreads; i++) { | |
| 2673 _par_gc_thread_states[i]->promo.stopTrackingPromotions(); | |
| 2674 } | |
| 2675 } | |
| 2676 } | |
| 2677 | |
| 2678 void ConcurrentMarkSweepGeneration::gc_epilogue_work(bool full) { | |
| 2679 assert(!incremental_collection_failed(), "Should have been cleared"); | |
| 2680 cmsSpace()->setPreconsumptionDirtyCardClosure(NULL); | |
| 2681 cmsSpace()->gc_epilogue(); | |
| 2682 // Print stat counters | |
| 2683 NOT_PRODUCT( | |
| 2684 assert(_numObjectsAllocated == 0, "check"); | |
| 2685 assert(_numWordsAllocated == 0, "check"); | |
| 2686 if (Verbose && PrintGC) { | |
| 2687 gclog_or_tty->print("Promoted "SIZE_FORMAT" objects, " | |
| 2688 SIZE_FORMAT" bytes", | |
| 2689 _numObjectsPromoted, _numWordsPromoted*sizeof(HeapWord)); | |
| 2690 } | |
| 2691 _numObjectsPromoted = 0; | |
| 2692 _numWordsPromoted = 0; | |
| 2693 ) | |
| 2694 | |
| 2695 if (PrintGC && Verbose) { | |
| 2696 // Call down the chain in contiguous_available needs the freelistLock | |
| 2697 // so print this out before releasing the freeListLock. | |
| 2698 gclog_or_tty->print(" Contiguous available "SIZE_FORMAT" bytes ", | |
| 2699 contiguous_available()); | |
| 2700 } | |
| 2701 } | |
| 2702 | |
| 2703 #ifndef PRODUCT | |
| 2704 bool CMSCollector::have_cms_token() { | |
| 2705 Thread* thr = Thread::current(); | |
| 2706 if (thr->is_VM_thread()) { | |
| 2707 return ConcurrentMarkSweepThread::vm_thread_has_cms_token(); | |
| 2708 } else if (thr->is_ConcurrentGC_thread()) { | |
| 2709 return ConcurrentMarkSweepThread::cms_thread_has_cms_token(); | |
| 2710 } else if (thr->is_GC_task_thread()) { | |
| 2711 return ConcurrentMarkSweepThread::vm_thread_has_cms_token() && | |
| 2712 ParGCRareEvent_lock->owned_by_self(); | |
| 2713 } | |
| 2714 return false; | |
| 2715 } | |
| 2716 #endif | |
| 2717 | |
| 2718 // Check reachability of the given heap address in CMS generation, | |
| 2719 // treating all other generations as roots. | |
| 2720 bool CMSCollector::is_cms_reachable(HeapWord* addr) { | |
| 2721 // We could "guarantee" below, rather than assert, but i'll | |
| 2722 // leave these as "asserts" so that an adventurous debugger | |
| 2723 // could try this in the product build provided some subset of | |
| 2724 // the conditions were met, provided they were intersted in the | |
| 2725 // results and knew that the computation below wouldn't interfere | |
| 2726 // with other concurrent computations mutating the structures | |
| 2727 // being read or written. | |
| 2728 assert(SafepointSynchronize::is_at_safepoint(), | |
| 2729 "Else mutations in object graph will make answer suspect"); | |
| 2730 assert(have_cms_token(), "Should hold cms token"); | |
| 2731 assert(haveFreelistLocks(), "must hold free list locks"); | |
| 2732 assert_lock_strong(bitMapLock()); | |
| 2733 | |
| 2734 // Clear the marking bit map array before starting, but, just | |
| 2735 // for kicks, first report if the given address is already marked | |
| 2736 gclog_or_tty->print_cr("Start: Address 0x%x is%s marked", addr, | |
| 2737 _markBitMap.isMarked(addr) ? "" : " not"); | |
| 2738 | |
| 2739 if (verify_after_remark()) { | |
| 2740 MutexLockerEx x(verification_mark_bm()->lock(), Mutex::_no_safepoint_check_flag); | |
| 2741 bool result = verification_mark_bm()->isMarked(addr); | |
| 2742 gclog_or_tty->print_cr("TransitiveMark: Address 0x%x %s marked", addr, | |
| 2743 result ? "IS" : "is NOT"); | |
| 2744 return result; | |
| 2745 } else { | |
| 2746 gclog_or_tty->print_cr("Could not compute result"); | |
| 2747 return false; | |
| 2748 } | |
| 2749 } | |
| 2750 | |
| 2751 //////////////////////////////////////////////////////// | |
| 2752 // CMS Verification Support | |
| 2753 //////////////////////////////////////////////////////// | |
| 2754 // Following the remark phase, the following invariant | |
| 2755 // should hold -- each object in the CMS heap which is | |
| 2756 // marked in markBitMap() should be marked in the verification_mark_bm(). | |
| 2757 | |
| 2758 class VerifyMarkedClosure: public BitMapClosure { | |
| 2759 CMSBitMap* _marks; | |
| 2760 bool _failed; | |
| 2761 | |
| 2762 public: | |
| 2763 VerifyMarkedClosure(CMSBitMap* bm): _marks(bm), _failed(false) {} | |
| 2764 | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
2765 bool do_bit(size_t offset) { |
| 0 | 2766 HeapWord* addr = _marks->offsetToHeapWord(offset); |
| 2767 if (!_marks->isMarked(addr)) { | |
| 2768 oop(addr)->print(); | |
| 2769 gclog_or_tty->print_cr(" ("INTPTR_FORMAT" should have been marked)", addr); | |
| 2770 _failed = true; | |
| 2771 } | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
2772 return true; |
| 0 | 2773 } |
| 2774 | |
| 2775 bool failed() { return _failed; } | |
| 2776 }; | |
| 2777 | |
| 2778 bool CMSCollector::verify_after_remark() { | |
| 2779 gclog_or_tty->print(" [Verifying CMS Marking... "); | |
| 2780 MutexLockerEx ml(verification_mark_bm()->lock(), Mutex::_no_safepoint_check_flag); | |
| 2781 static bool init = false; | |
| 2782 | |
| 2783 assert(SafepointSynchronize::is_at_safepoint(), | |
| 2784 "Else mutations in object graph will make answer suspect"); | |
| 2785 assert(have_cms_token(), | |
| 2786 "Else there may be mutual interference in use of " | |
| 2787 " verification data structures"); | |
| 2788 assert(_collectorState > Marking && _collectorState <= Sweeping, | |
| 2789 "Else marking info checked here may be obsolete"); | |
| 2790 assert(haveFreelistLocks(), "must hold free list locks"); | |
| 2791 assert_lock_strong(bitMapLock()); | |
| 2792 | |
| 2793 | |
| 2794 // Allocate marking bit map if not already allocated | |
| 2795 if (!init) { // first time | |
| 2796 if (!verification_mark_bm()->allocate(_span)) { | |
| 2797 return false; | |
| 2798 } | |
| 2799 init = true; | |
| 2800 } | |
| 2801 | |
| 2802 assert(verification_mark_stack()->isEmpty(), "Should be empty"); | |
| 2803 | |
| 2804 // Turn off refs discovery -- so we will be tracing through refs. | |
| 2805 // This is as intended, because by this time | |
| 2806 // GC must already have cleared any refs that need to be cleared, | |
| 2807 // and traced those that need to be marked; moreover, | |
| 2808 // the marking done here is not going to intefere in any | |
| 2809 // way with the marking information used by GC. | |
| 2810 NoRefDiscovery no_discovery(ref_processor()); | |
| 2811 | |
| 2812 COMPILER2_PRESENT(DerivedPointerTableDeactivate dpt_deact;) | |
| 2813 | |
| 2814 // Clear any marks from a previous round | |
| 2815 verification_mark_bm()->clear_all(); | |
| 2816 assert(verification_mark_stack()->isEmpty(), "markStack should be empty"); | |
| 2817 assert(overflow_list_is_empty(), "overflow list should be empty"); | |
| 2818 | |
| 2819 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 2820 gch->ensure_parsability(false); // fill TLABs, but no need to retire them | |
| 2821 // Update the saved marks which may affect the root scans. | |
| 2822 gch->save_marks(); | |
| 2823 | |
| 2824 if (CMSRemarkVerifyVariant == 1) { | |
| 2825 // In this first variant of verification, we complete | |
| 2826 // all marking, then check if the new marks-verctor is | |
| 2827 // a subset of the CMS marks-vector. | |
| 2828 verify_after_remark_work_1(); | |
| 2829 } else if (CMSRemarkVerifyVariant == 2) { | |
| 2830 // In this second variant of verification, we flag an error | |
| 2831 // (i.e. an object reachable in the new marks-vector not reachable | |
| 2832 // in the CMS marks-vector) immediately, also indicating the | |
| 2833 // identify of an object (A) that references the unmarked object (B) -- | |
| 2834 // presumably, a mutation to A failed to be picked up by preclean/remark? | |
| 2835 verify_after_remark_work_2(); | |
| 2836 } else { | |
| 2837 warning("Unrecognized value %d for CMSRemarkVerifyVariant", | |
| 2838 CMSRemarkVerifyVariant); | |
| 2839 } | |
| 2840 gclog_or_tty->print(" done] "); | |
| 2841 return true; | |
| 2842 } | |
| 2843 | |
| 2844 void CMSCollector::verify_after_remark_work_1() { | |
| 2845 ResourceMark rm; | |
| 2846 HandleMark hm; | |
| 2847 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 2848 | |
| 2849 // Mark from roots one level into CMS | |
| 2850 MarkRefsIntoClosure notOlder(_span, verification_mark_bm(), true /* nmethods */); | |
| 2851 gch->rem_set()->prepare_for_younger_refs_iterate(false); // Not parallel. | |
| 2852 | |
| 2853 gch->gen_process_strong_roots(_cmsGen->level(), | |
| 2854 true, // younger gens are roots | |
| 2855 true, // collecting perm gen | |
| 2856 SharedHeap::ScanningOption(roots_scanning_options()), | |
| 2857 NULL, ¬Older); | |
| 2858 | |
| 2859 // Now mark from the roots | |
| 2860 assert(_revisitStack.isEmpty(), "Should be empty"); | |
| 2861 MarkFromRootsClosure markFromRootsClosure(this, _span, | |
| 2862 verification_mark_bm(), verification_mark_stack(), &_revisitStack, | |
| 2863 false /* don't yield */, true /* verifying */); | |
| 2864 assert(_restart_addr == NULL, "Expected pre-condition"); | |
| 2865 verification_mark_bm()->iterate(&markFromRootsClosure); | |
| 2866 while (_restart_addr != NULL) { | |
| 2867 // Deal with stack overflow: by restarting at the indicated | |
| 2868 // address. | |
| 2869 HeapWord* ra = _restart_addr; | |
| 2870 markFromRootsClosure.reset(ra); | |
| 2871 _restart_addr = NULL; | |
| 2872 verification_mark_bm()->iterate(&markFromRootsClosure, ra, _span.end()); | |
| 2873 } | |
| 2874 assert(verification_mark_stack()->isEmpty(), "Should have been drained"); | |
| 2875 verify_work_stacks_empty(); | |
| 2876 // Should reset the revisit stack above, since no class tree | |
| 2877 // surgery is forthcoming. | |
| 2878 _revisitStack.reset(); // throwing away all contents | |
| 2879 | |
| 2880 // Marking completed -- now verify that each bit marked in | |
| 2881 // verification_mark_bm() is also marked in markBitMap(); flag all | |
| 2882 // errors by printing corresponding objects. | |
| 2883 VerifyMarkedClosure vcl(markBitMap()); | |
| 2884 verification_mark_bm()->iterate(&vcl); | |
| 2885 if (vcl.failed()) { | |
| 2886 gclog_or_tty->print("Verification failed"); | |
| 2887 Universe::heap()->print(); | |
| 2888 fatal(" ... aborting"); | |
| 2889 } | |
| 2890 } | |
| 2891 | |
| 2892 void CMSCollector::verify_after_remark_work_2() { | |
| 2893 ResourceMark rm; | |
| 2894 HandleMark hm; | |
| 2895 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 2896 | |
| 2897 // Mark from roots one level into CMS | |
| 2898 MarkRefsIntoVerifyClosure notOlder(_span, verification_mark_bm(), | |
| 2899 markBitMap(), true /* nmethods */); | |
| 2900 gch->rem_set()->prepare_for_younger_refs_iterate(false); // Not parallel. | |
| 2901 gch->gen_process_strong_roots(_cmsGen->level(), | |
| 2902 true, // younger gens are roots | |
| 2903 true, // collecting perm gen | |
| 2904 SharedHeap::ScanningOption(roots_scanning_options()), | |
| 2905 NULL, ¬Older); | |
| 2906 | |
| 2907 // Now mark from the roots | |
| 2908 assert(_revisitStack.isEmpty(), "Should be empty"); | |
| 2909 MarkFromRootsVerifyClosure markFromRootsClosure(this, _span, | |
| 2910 verification_mark_bm(), markBitMap(), verification_mark_stack()); | |
| 2911 assert(_restart_addr == NULL, "Expected pre-condition"); | |
| 2912 verification_mark_bm()->iterate(&markFromRootsClosure); | |
| 2913 while (_restart_addr != NULL) { | |
| 2914 // Deal with stack overflow: by restarting at the indicated | |
| 2915 // address. | |
| 2916 HeapWord* ra = _restart_addr; | |
| 2917 markFromRootsClosure.reset(ra); | |
| 2918 _restart_addr = NULL; | |
| 2919 verification_mark_bm()->iterate(&markFromRootsClosure, ra, _span.end()); | |
| 2920 } | |
| 2921 assert(verification_mark_stack()->isEmpty(), "Should have been drained"); | |
| 2922 verify_work_stacks_empty(); | |
| 2923 // Should reset the revisit stack above, since no class tree | |
| 2924 // surgery is forthcoming. | |
| 2925 _revisitStack.reset(); // throwing away all contents | |
| 2926 | |
| 2927 // Marking completed -- now verify that each bit marked in | |
| 2928 // verification_mark_bm() is also marked in markBitMap(); flag all | |
| 2929 // errors by printing corresponding objects. | |
| 2930 VerifyMarkedClosure vcl(markBitMap()); | |
| 2931 verification_mark_bm()->iterate(&vcl); | |
| 2932 assert(!vcl.failed(), "Else verification above should not have succeeded"); | |
| 2933 } | |
| 2934 | |
| 2935 void ConcurrentMarkSweepGeneration::save_marks() { | |
| 2936 // delegate to CMS space | |
| 2937 cmsSpace()->save_marks(); | |
| 2938 for (uint i = 0; i < ParallelGCThreads; i++) { | |
| 2939 _par_gc_thread_states[i]->promo.startTrackingPromotions(); | |
| 2940 } | |
| 2941 } | |
| 2942 | |
| 2943 bool ConcurrentMarkSweepGeneration::no_allocs_since_save_marks() { | |
| 2944 return cmsSpace()->no_allocs_since_save_marks(); | |
| 2945 } | |
| 2946 | |
| 2947 #define CMS_SINCE_SAVE_MARKS_DEFN(OopClosureType, nv_suffix) \ | |
| 2948 \ | |
| 2949 void ConcurrentMarkSweepGeneration:: \ | |
| 2950 oop_since_save_marks_iterate##nv_suffix(OopClosureType* cl) { \ | |
| 2951 cl->set_generation(this); \ | |
| 2952 cmsSpace()->oop_since_save_marks_iterate##nv_suffix(cl); \ | |
| 2953 cl->reset_generation(); \ | |
| 2954 save_marks(); \ | |
| 2955 } | |
| 2956 | |
| 2957 ALL_SINCE_SAVE_MARKS_CLOSURES(CMS_SINCE_SAVE_MARKS_DEFN) | |
| 2958 | |
| 2959 void | |
| 2960 ConcurrentMarkSweepGeneration::object_iterate_since_last_GC(ObjectClosure* blk) | |
| 2961 { | |
| 2962 // Not currently implemented; need to do the following. -- ysr. | |
| 2963 // dld -- I think that is used for some sort of allocation profiler. So it | |
| 2964 // really means the objects allocated by the mutator since the last | |
| 2965 // GC. We could potentially implement this cheaply by recording only | |
| 2966 // the direct allocations in a side data structure. | |
| 2967 // | |
| 2968 // I think we probably ought not to be required to support these | |
| 2969 // iterations at any arbitrary point; I think there ought to be some | |
| 2970 // call to enable/disable allocation profiling in a generation/space, | |
| 2971 // and the iterator ought to return the objects allocated in the | |
| 2972 // gen/space since the enable call, or the last iterator call (which | |
| 2973 // will probably be at a GC.) That way, for gens like CM&S that would | |
| 2974 // require some extra data structure to support this, we only pay the | |
| 2975 // cost when it's in use... | |
| 2976 cmsSpace()->object_iterate_since_last_GC(blk); | |
| 2977 } | |
| 2978 | |
| 2979 void | |
| 2980 ConcurrentMarkSweepGeneration::younger_refs_iterate(OopsInGenClosure* cl) { | |
| 2981 cl->set_generation(this); | |
| 2982 younger_refs_in_space_iterate(_cmsSpace, cl); | |
| 2983 cl->reset_generation(); | |
| 2984 } | |
| 2985 | |
| 2986 void | |
| 2987 ConcurrentMarkSweepGeneration::oop_iterate(MemRegion mr, OopClosure* cl) { | |
| 2988 if (freelistLock()->owned_by_self()) { | |
| 2989 Generation::oop_iterate(mr, cl); | |
| 2990 } else { | |
| 2991 MutexLockerEx x(freelistLock(), Mutex::_no_safepoint_check_flag); | |
| 2992 Generation::oop_iterate(mr, cl); | |
| 2993 } | |
| 2994 } | |
| 2995 | |
| 2996 void | |
| 2997 ConcurrentMarkSweepGeneration::oop_iterate(OopClosure* cl) { | |
| 2998 if (freelistLock()->owned_by_self()) { | |
| 2999 Generation::oop_iterate(cl); | |
| 3000 } else { | |
| 3001 MutexLockerEx x(freelistLock(), Mutex::_no_safepoint_check_flag); | |
| 3002 Generation::oop_iterate(cl); | |
| 3003 } | |
| 3004 } | |
| 3005 | |
| 3006 void | |
| 3007 ConcurrentMarkSweepGeneration::object_iterate(ObjectClosure* cl) { | |
| 3008 if (freelistLock()->owned_by_self()) { | |
| 3009 Generation::object_iterate(cl); | |
| 3010 } else { | |
| 3011 MutexLockerEx x(freelistLock(), Mutex::_no_safepoint_check_flag); | |
| 3012 Generation::object_iterate(cl); | |
| 3013 } | |
| 3014 } | |
| 3015 | |
| 3016 void | |
| 3017 ConcurrentMarkSweepGeneration::pre_adjust_pointers() { | |
| 3018 } | |
| 3019 | |
| 3020 void | |
| 3021 ConcurrentMarkSweepGeneration::post_compact() { | |
| 3022 } | |
| 3023 | |
| 3024 void | |
| 3025 ConcurrentMarkSweepGeneration::prepare_for_verify() { | |
| 3026 // Fix the linear allocation blocks to look like free blocks. | |
| 3027 | |
| 3028 // Locks are normally acquired/released in gc_prologue/gc_epilogue, but those | |
| 3029 // are not called when the heap is verified during universe initialization and | |
| 3030 // at vm shutdown. | |
| 3031 if (freelistLock()->owned_by_self()) { | |
| 3032 cmsSpace()->prepare_for_verify(); | |
| 3033 } else { | |
| 3034 MutexLockerEx fll(freelistLock(), Mutex::_no_safepoint_check_flag); | |
| 3035 cmsSpace()->prepare_for_verify(); | |
| 3036 } | |
| 3037 } | |
| 3038 | |
| 3039 void | |
| 3040 ConcurrentMarkSweepGeneration::verify(bool allow_dirty /* ignored */) { | |
| 3041 // Locks are normally acquired/released in gc_prologue/gc_epilogue, but those | |
| 3042 // are not called when the heap is verified during universe initialization and | |
| 3043 // at vm shutdown. | |
| 3044 if (freelistLock()->owned_by_self()) { | |
| 3045 cmsSpace()->verify(false /* ignored */); | |
| 3046 } else { | |
| 3047 MutexLockerEx fll(freelistLock(), Mutex::_no_safepoint_check_flag); | |
| 3048 cmsSpace()->verify(false /* ignored */); | |
| 3049 } | |
| 3050 } | |
| 3051 | |
| 3052 void CMSCollector::verify(bool allow_dirty /* ignored */) { | |
| 3053 _cmsGen->verify(allow_dirty); | |
| 3054 _permGen->verify(allow_dirty); | |
| 3055 } | |
| 3056 | |
| 3057 #ifndef PRODUCT | |
| 3058 bool CMSCollector::overflow_list_is_empty() const { | |
| 3059 assert(_num_par_pushes >= 0, "Inconsistency"); | |
| 3060 if (_overflow_list == NULL) { | |
| 3061 assert(_num_par_pushes == 0, "Inconsistency"); | |
| 3062 } | |
| 3063 return _overflow_list == NULL; | |
| 3064 } | |
| 3065 | |
| 3066 // The methods verify_work_stacks_empty() and verify_overflow_empty() | |
| 3067 // merely consolidate assertion checks that appear to occur together frequently. | |
| 3068 void CMSCollector::verify_work_stacks_empty() const { | |
| 3069 assert(_markStack.isEmpty(), "Marking stack should be empty"); | |
| 3070 assert(overflow_list_is_empty(), "Overflow list should be empty"); | |
| 3071 } | |
| 3072 | |
| 3073 void CMSCollector::verify_overflow_empty() const { | |
| 3074 assert(overflow_list_is_empty(), "Overflow list should be empty"); | |
| 3075 assert(no_preserved_marks(), "No preserved marks"); | |
| 3076 } | |
| 3077 #endif // PRODUCT | |
| 3078 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3079 // Decide if we want to enable class unloading as part of the |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3080 // ensuing concurrent GC cycle. We will collect the perm gen and |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3081 // unload classes if it's the case that: |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3082 // (1) an explicit gc request has been made and the flag |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3083 // ExplicitGCInvokesConcurrentAndUnloadsClasses is set, OR |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3084 // (2) (a) class unloading is enabled at the command line, and |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3085 // (b) (i) perm gen threshold has been crossed, or |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3086 // (ii) old gen is getting really full, or |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3087 // (iii) the previous N CMS collections did not collect the |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3088 // perm gen |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3089 // NOTE: Provided there is no change in the state of the heap between |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3090 // calls to this method, it should have idempotent results. Moreover, |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3091 // its results should be monotonically increasing (i.e. going from 0 to 1, |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3092 // but not 1 to 0) between successive calls between which the heap was |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3093 // not collected. For the implementation below, it must thus rely on |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3094 // the property that concurrent_cycles_since_last_unload() |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3095 // will not decrease unless a collection cycle happened and that |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3096 // _permGen->should_concurrent_collect() and _cmsGen->is_too_full() are |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3097 // themselves also monotonic in that sense. See check_monotonicity() |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3098 // below. |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3099 bool CMSCollector::update_should_unload_classes() { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3100 _should_unload_classes = false; |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3101 // Condition 1 above |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3102 if (_full_gc_requested && ExplicitGCInvokesConcurrentAndUnloadsClasses) { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3103 _should_unload_classes = true; |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3104 } else if (CMSClassUnloadingEnabled) { // Condition 2.a above |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3105 // Disjuncts 2.b.(i,ii,iii) above |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3106 _should_unload_classes = (concurrent_cycles_since_last_unload() >= |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3107 CMSClassUnloadingMaxInterval) |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3108 || _permGen->should_concurrent_collect() |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3109 || _cmsGen->is_too_full(); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3110 } |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3111 return _should_unload_classes; |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3112 } |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3113 |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3114 bool ConcurrentMarkSweepGeneration::is_too_full() const { |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3115 bool res = should_concurrent_collect(); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3116 res = res && (occupancy() > (double)CMSIsTooFullPercentage/100.0); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3117 return res; |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3118 } |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3119 |
| 0 | 3120 void CMSCollector::setup_cms_unloading_and_verification_state() { |
| 3121 const bool should_verify = VerifyBeforeGC || VerifyAfterGC || VerifyDuringGC | |
| 3122 || VerifyBeforeExit; | |
| 3123 const int rso = SharedHeap::SO_Symbols | SharedHeap::SO_Strings | |
| 3124 | SharedHeap::SO_CodeCache; | |
| 3125 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3126 if (should_unload_classes()) { // Should unload classes this cycle |
| 0 | 3127 remove_root_scanning_option(rso); // Shrink the root set appropriately |
| 3128 set_verifying(should_verify); // Set verification state for this cycle | |
| 3129 return; // Nothing else needs to be done at this time | |
| 3130 } | |
| 3131 | |
| 3132 // Not unloading classes this cycle | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3133 assert(!should_unload_classes(), "Inconsitency!"); |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
3134 if ((!verifying() || unloaded_classes_last_cycle()) && should_verify) { |
| 0 | 3135 // We were not verifying, or we _were_ unloading classes in the last cycle, |
| 3136 // AND some verification options are enabled this cycle; in this case, | |
| 3137 // we must make sure that the deadness map is allocated if not already so, | |
| 3138 // and cleared (if already allocated previously -- | |
| 3139 // CMSBitMap::sizeInBits() is used to determine if it's allocated). | |
| 3140 if (perm_gen_verify_bit_map()->sizeInBits() == 0) { | |
| 3141 if (!perm_gen_verify_bit_map()->allocate(_permGen->reserved())) { | |
| 3142 warning("Failed to allocate permanent generation verification CMS Bit Map;\n" | |
| 3143 "permanent generation verification disabled"); | |
| 3144 return; // Note that we leave verification disabled, so we'll retry this | |
| 3145 // allocation next cycle. We _could_ remember this failure | |
| 3146 // and skip further attempts and permanently disable verification | |
| 3147 // attempts if that is considered more desirable. | |
| 3148 } | |
| 3149 assert(perm_gen_verify_bit_map()->covers(_permGen->reserved()), | |
| 3150 "_perm_gen_ver_bit_map inconsistency?"); | |
| 3151 } else { | |
| 3152 perm_gen_verify_bit_map()->clear_all(); | |
| 3153 } | |
| 3154 // Include symbols, strings and code cache elements to prevent their resurrection. | |
| 3155 add_root_scanning_option(rso); | |
| 3156 set_verifying(true); | |
| 3157 } else if (verifying() && !should_verify) { | |
| 3158 // We were verifying, but some verification flags got disabled. | |
| 3159 set_verifying(false); | |
| 3160 // Exclude symbols, strings and code cache elements from root scanning to | |
| 3161 // reduce IM and RM pauses. | |
| 3162 remove_root_scanning_option(rso); | |
| 3163 } | |
| 3164 } | |
| 3165 | |
| 3166 | |
| 3167 #ifndef PRODUCT | |
| 3168 HeapWord* CMSCollector::block_start(const void* p) const { | |
| 3169 const HeapWord* addr = (HeapWord*)p; | |
| 3170 if (_span.contains(p)) { | |
| 3171 if (_cmsGen->cmsSpace()->is_in_reserved(addr)) { | |
| 3172 return _cmsGen->cmsSpace()->block_start(p); | |
| 3173 } else { | |
| 3174 assert(_permGen->cmsSpace()->is_in_reserved(addr), | |
| 3175 "Inconsistent _span?"); | |
| 3176 return _permGen->cmsSpace()->block_start(p); | |
| 3177 } | |
| 3178 } | |
| 3179 return NULL; | |
| 3180 } | |
| 3181 #endif | |
| 3182 | |
| 3183 HeapWord* | |
| 3184 ConcurrentMarkSweepGeneration::expand_and_allocate(size_t word_size, | |
| 3185 bool tlab, | |
| 3186 bool parallel) { | |
| 3187 assert(!tlab, "Can't deal with TLAB allocation"); | |
| 3188 MutexLockerEx x(freelistLock(), Mutex::_no_safepoint_check_flag); | |
| 3189 expand(word_size*HeapWordSize, MinHeapDeltaBytes, | |
| 3190 CMSExpansionCause::_satisfy_allocation); | |
| 3191 if (GCExpandToAllocateDelayMillis > 0) { | |
| 3192 os::sleep(Thread::current(), GCExpandToAllocateDelayMillis, false); | |
| 3193 } | |
|
9
173195ff483a
6642634: Test nsk/regression/b6186200 crashed with SIGSEGV
ysr
parents:
7
diff
changeset
|
3194 return have_lock_and_allocate(word_size, tlab); |
| 0 | 3195 } |
| 3196 | |
| 3197 // YSR: All of this generation expansion/shrinking stuff is an exact copy of | |
| 3198 // OneContigSpaceCardGeneration, which makes me wonder if we should move this | |
| 3199 // to CardGeneration and share it... | |
|
271
818a18cd69a8
6730514: assertion failure in mangling code when expanding by 0 bytes
jmasa
parents:
196
diff
changeset
|
3200 bool ConcurrentMarkSweepGeneration::expand(size_t bytes, size_t expand_bytes) { |
|
818a18cd69a8
6730514: assertion failure in mangling code when expanding by 0 bytes
jmasa
parents:
196
diff
changeset
|
3201 return CardGeneration::expand(bytes, expand_bytes); |
|
818a18cd69a8
6730514: assertion failure in mangling code when expanding by 0 bytes
jmasa
parents:
196
diff
changeset
|
3202 } |
|
818a18cd69a8
6730514: assertion failure in mangling code when expanding by 0 bytes
jmasa
parents:
196
diff
changeset
|
3203 |
| 0 | 3204 void ConcurrentMarkSweepGeneration::expand(size_t bytes, size_t expand_bytes, |
| 3205 CMSExpansionCause::Cause cause) | |
| 3206 { | |
|
271
818a18cd69a8
6730514: assertion failure in mangling code when expanding by 0 bytes
jmasa
parents:
196
diff
changeset
|
3207 |
|
818a18cd69a8
6730514: assertion failure in mangling code when expanding by 0 bytes
jmasa
parents:
196
diff
changeset
|
3208 bool success = expand(bytes, expand_bytes); |
|
818a18cd69a8
6730514: assertion failure in mangling code when expanding by 0 bytes
jmasa
parents:
196
diff
changeset
|
3209 |
| 0 | 3210 // remember why we expanded; this information is used |
| 3211 // by shouldConcurrentCollect() when making decisions on whether to start | |
| 3212 // a new CMS cycle. | |
| 3213 if (success) { | |
| 3214 set_expansion_cause(cause); | |
| 3215 if (PrintGCDetails && Verbose) { | |
| 3216 gclog_or_tty->print_cr("Expanded CMS gen for %s", | |
| 3217 CMSExpansionCause::to_string(cause)); | |
| 3218 } | |
| 3219 } | |
| 3220 } | |
| 3221 | |
| 3222 HeapWord* ConcurrentMarkSweepGeneration::expand_and_par_lab_allocate(CMSParGCThreadState* ps, size_t word_sz) { | |
| 3223 HeapWord* res = NULL; | |
| 3224 MutexLocker x(ParGCRareEvent_lock); | |
| 3225 while (true) { | |
| 3226 // Expansion by some other thread might make alloc OK now: | |
| 3227 res = ps->lab.alloc(word_sz); | |
| 3228 if (res != NULL) return res; | |
| 3229 // If there's not enough expansion space available, give up. | |
| 3230 if (_virtual_space.uncommitted_size() < (word_sz * HeapWordSize)) { | |
| 3231 return NULL; | |
| 3232 } | |
| 3233 // Otherwise, we try expansion. | |
| 3234 expand(word_sz*HeapWordSize, MinHeapDeltaBytes, | |
| 3235 CMSExpansionCause::_allocate_par_lab); | |
| 3236 // Now go around the loop and try alloc again; | |
| 3237 // A competing par_promote might beat us to the expansion space, | |
| 3238 // so we may go around the loop again if promotion fails agaion. | |
| 3239 if (GCExpandToAllocateDelayMillis > 0) { | |
| 3240 os::sleep(Thread::current(), GCExpandToAllocateDelayMillis, false); | |
| 3241 } | |
| 3242 } | |
| 3243 } | |
| 3244 | |
| 3245 | |
| 3246 bool ConcurrentMarkSweepGeneration::expand_and_ensure_spooling_space( | |
| 3247 PromotionInfo* promo) { | |
| 3248 MutexLocker x(ParGCRareEvent_lock); | |
| 3249 size_t refill_size_bytes = promo->refillSize() * HeapWordSize; | |
| 3250 while (true) { | |
| 3251 // Expansion by some other thread might make alloc OK now: | |
| 3252 if (promo->ensure_spooling_space()) { | |
| 3253 assert(promo->has_spooling_space(), | |
| 3254 "Post-condition of successful ensure_spooling_space()"); | |
| 3255 return true; | |
| 3256 } | |
| 3257 // If there's not enough expansion space available, give up. | |
| 3258 if (_virtual_space.uncommitted_size() < refill_size_bytes) { | |
| 3259 return false; | |
| 3260 } | |
| 3261 // Otherwise, we try expansion. | |
| 3262 expand(refill_size_bytes, MinHeapDeltaBytes, | |
| 3263 CMSExpansionCause::_allocate_par_spooling_space); | |
| 3264 // Now go around the loop and try alloc again; | |
| 3265 // A competing allocation might beat us to the expansion space, | |
| 3266 // so we may go around the loop again if allocation fails again. | |
| 3267 if (GCExpandToAllocateDelayMillis > 0) { | |
| 3268 os::sleep(Thread::current(), GCExpandToAllocateDelayMillis, false); | |
| 3269 } | |
| 3270 } | |
| 3271 } | |
| 3272 | |
| 3273 | |
| 3274 | |
| 3275 void ConcurrentMarkSweepGeneration::shrink(size_t bytes) { | |
| 3276 assert_locked_or_safepoint(Heap_lock); | |
| 3277 size_t size = ReservedSpace::page_align_size_down(bytes); | |
| 3278 if (size > 0) { | |
| 3279 shrink_by(size); | |
| 3280 } | |
| 3281 } | |
| 3282 | |
| 3283 bool ConcurrentMarkSweepGeneration::grow_by(size_t bytes) { | |
| 3284 assert_locked_or_safepoint(Heap_lock); | |
| 3285 bool result = _virtual_space.expand_by(bytes); | |
| 3286 if (result) { | |
| 3287 HeapWord* old_end = _cmsSpace->end(); | |
| 3288 size_t new_word_size = | |
| 3289 heap_word_size(_virtual_space.committed_size()); | |
| 3290 MemRegion mr(_cmsSpace->bottom(), new_word_size); | |
| 3291 _bts->resize(new_word_size); // resize the block offset shared array | |
| 3292 Universe::heap()->barrier_set()->resize_covered_region(mr); | |
| 3293 // Hmmmm... why doesn't CFLS::set_end verify locking? | |
| 3294 // This is quite ugly; FIX ME XXX | |
| 3295 _cmsSpace->assert_locked(); | |
| 3296 _cmsSpace->set_end((HeapWord*)_virtual_space.high()); | |
| 3297 | |
| 3298 // update the space and generation capacity counters | |
| 3299 if (UsePerfData) { | |
| 3300 _space_counters->update_capacity(); | |
| 3301 _gen_counters->update_all(); | |
| 3302 } | |
| 3303 | |
| 3304 if (Verbose && PrintGC) { | |
| 3305 size_t new_mem_size = _virtual_space.committed_size(); | |
| 3306 size_t old_mem_size = new_mem_size - bytes; | |
| 3307 gclog_or_tty->print_cr("Expanding %s from %ldK by %ldK to %ldK", | |
| 3308 name(), old_mem_size/K, bytes/K, new_mem_size/K); | |
| 3309 } | |
| 3310 } | |
| 3311 return result; | |
| 3312 } | |
| 3313 | |
| 3314 bool ConcurrentMarkSweepGeneration::grow_to_reserved() { | |
| 3315 assert_locked_or_safepoint(Heap_lock); | |
| 3316 bool success = true; | |
| 3317 const size_t remaining_bytes = _virtual_space.uncommitted_size(); | |
| 3318 if (remaining_bytes > 0) { | |
| 3319 success = grow_by(remaining_bytes); | |
| 3320 DEBUG_ONLY(if (!success) warning("grow to reserved failed");) | |
| 3321 } | |
| 3322 return success; | |
| 3323 } | |
| 3324 | |
| 3325 void ConcurrentMarkSweepGeneration::shrink_by(size_t bytes) { | |
| 3326 assert_locked_or_safepoint(Heap_lock); | |
| 3327 assert_lock_strong(freelistLock()); | |
| 3328 // XXX Fix when compaction is implemented. | |
| 3329 warning("Shrinking of CMS not yet implemented"); | |
| 3330 return; | |
| 3331 } | |
| 3332 | |
| 3333 | |
| 3334 // Simple ctor/dtor wrapper for accounting & timer chores around concurrent | |
| 3335 // phases. | |
| 3336 class CMSPhaseAccounting: public StackObj { | |
| 3337 public: | |
| 3338 CMSPhaseAccounting(CMSCollector *collector, | |
| 3339 const char *phase, | |
| 3340 bool print_cr = true); | |
| 3341 ~CMSPhaseAccounting(); | |
| 3342 | |
| 3343 private: | |
| 3344 CMSCollector *_collector; | |
| 3345 const char *_phase; | |
| 3346 elapsedTimer _wallclock; | |
| 3347 bool _print_cr; | |
| 3348 | |
| 3349 public: | |
| 3350 // Not MT-safe; so do not pass around these StackObj's | |
| 3351 // where they may be accessed by other threads. | |
| 3352 jlong wallclock_millis() { | |
| 3353 assert(_wallclock.is_active(), "Wall clock should not stop"); | |
| 3354 _wallclock.stop(); // to record time | |
| 3355 jlong ret = _wallclock.milliseconds(); | |
| 3356 _wallclock.start(); // restart | |
| 3357 return ret; | |
| 3358 } | |
| 3359 }; | |
| 3360 | |
| 3361 CMSPhaseAccounting::CMSPhaseAccounting(CMSCollector *collector, | |
| 3362 const char *phase, | |
| 3363 bool print_cr) : | |
| 3364 _collector(collector), _phase(phase), _print_cr(print_cr) { | |
| 3365 | |
| 3366 if (PrintCMSStatistics != 0) { | |
| 3367 _collector->resetYields(); | |
| 3368 } | |
| 3369 if (PrintGCDetails && PrintGCTimeStamps) { | |
| 3370 gclog_or_tty->date_stamp(PrintGCDateStamps); | |
| 3371 gclog_or_tty->stamp(); | |
| 3372 gclog_or_tty->print_cr(": [%s-concurrent-%s-start]", | |
| 3373 _collector->cmsGen()->short_name(), _phase); | |
| 3374 } | |
| 3375 _collector->resetTimer(); | |
| 3376 _wallclock.start(); | |
| 3377 _collector->startTimer(); | |
| 3378 } | |
| 3379 | |
| 3380 CMSPhaseAccounting::~CMSPhaseAccounting() { | |
| 3381 assert(_wallclock.is_active(), "Wall clock should not have stopped"); | |
| 3382 _collector->stopTimer(); | |
| 3383 _wallclock.stop(); | |
| 3384 if (PrintGCDetails) { | |
| 3385 gclog_or_tty->date_stamp(PrintGCDateStamps); | |
| 3386 if (PrintGCTimeStamps) { | |
| 3387 gclog_or_tty->stamp(); | |
| 3388 gclog_or_tty->print(": "); | |
| 3389 } | |
| 3390 gclog_or_tty->print("[%s-concurrent-%s: %3.3f/%3.3f secs]", | |
| 3391 _collector->cmsGen()->short_name(), | |
| 3392 _phase, _collector->timerValue(), _wallclock.seconds()); | |
| 3393 if (_print_cr) { | |
| 3394 gclog_or_tty->print_cr(""); | |
| 3395 } | |
| 3396 if (PrintCMSStatistics != 0) { | |
| 3397 gclog_or_tty->print_cr(" (CMS-concurrent-%s yielded %d times)", _phase, | |
| 3398 _collector->yields()); | |
| 3399 } | |
| 3400 } | |
| 3401 } | |
| 3402 | |
| 3403 // CMS work | |
| 3404 | |
| 3405 // Checkpoint the roots into this generation from outside | |
| 3406 // this generation. [Note this initial checkpoint need only | |
| 3407 // be approximate -- we'll do a catch up phase subsequently.] | |
| 3408 void CMSCollector::checkpointRootsInitial(bool asynch) { | |
| 3409 assert(_collectorState == InitialMarking, "Wrong collector state"); | |
| 3410 check_correct_thread_executing(); | |
| 3411 ReferenceProcessor* rp = ref_processor(); | |
| 3412 SpecializationStats::clear(); | |
| 3413 assert(_restart_addr == NULL, "Control point invariant"); | |
| 3414 if (asynch) { | |
| 3415 // acquire locks for subsequent manipulations | |
| 3416 MutexLockerEx x(bitMapLock(), | |
| 3417 Mutex::_no_safepoint_check_flag); | |
| 3418 checkpointRootsInitialWork(asynch); | |
| 3419 rp->verify_no_references_recorded(); | |
| 3420 rp->enable_discovery(); // enable ("weak") refs discovery | |
| 3421 _collectorState = Marking; | |
| 3422 } else { | |
| 3423 // (Weak) Refs discovery: this is controlled from genCollectedHeap::do_collection | |
| 3424 // which recognizes if we are a CMS generation, and doesn't try to turn on | |
| 3425 // discovery; verify that they aren't meddling. | |
| 3426 assert(!rp->discovery_is_atomic(), | |
| 3427 "incorrect setting of discovery predicate"); | |
| 3428 assert(!rp->discovery_enabled(), "genCollectedHeap shouldn't control " | |
| 3429 "ref discovery for this generation kind"); | |
| 3430 // already have locks | |
| 3431 checkpointRootsInitialWork(asynch); | |
| 3432 rp->enable_discovery(); // now enable ("weak") refs discovery | |
| 3433 _collectorState = Marking; | |
| 3434 } | |
| 3435 SpecializationStats::print(); | |
| 3436 } | |
| 3437 | |
| 3438 void CMSCollector::checkpointRootsInitialWork(bool asynch) { | |
| 3439 assert(SafepointSynchronize::is_at_safepoint(), "world should be stopped"); | |
| 3440 assert(_collectorState == InitialMarking, "just checking"); | |
| 3441 | |
| 3442 // If there has not been a GC[n-1] since last GC[n] cycle completed, | |
| 3443 // precede our marking with a collection of all | |
| 3444 // younger generations to keep floating garbage to a minimum. | |
| 3445 // XXX: we won't do this for now -- it's an optimization to be done later. | |
| 3446 | |
| 3447 // already have locks | |
| 3448 assert_lock_strong(bitMapLock()); | |
| 3449 assert(_markBitMap.isAllClear(), "was reset at end of previous cycle"); | |
| 3450 | |
| 3451 // Setup the verification and class unloading state for this | |
| 3452 // CMS collection cycle. | |
| 3453 setup_cms_unloading_and_verification_state(); | |
| 3454 | |
| 3455 NOT_PRODUCT(TraceTime t("\ncheckpointRootsInitialWork", | |
| 3456 PrintGCDetails && Verbose, true, gclog_or_tty);) | |
| 3457 if (UseAdaptiveSizePolicy) { | |
| 3458 size_policy()->checkpoint_roots_initial_begin(); | |
| 3459 } | |
| 3460 | |
| 3461 // Reset all the PLAB chunk arrays if necessary. | |
| 3462 if (_survivor_plab_array != NULL && !CMSPLABRecordAlways) { | |
| 3463 reset_survivor_plab_arrays(); | |
| 3464 } | |
| 3465 | |
| 3466 ResourceMark rm; | |
| 3467 HandleMark hm; | |
| 3468 | |
| 3469 FalseClosure falseClosure; | |
| 3470 // In the case of a synchronous collection, we will elide the | |
| 3471 // remark step, so it's important to catch all the nmethod oops | |
| 3472 // in this step; hence the last argument to the constrcutor below. | |
| 3473 MarkRefsIntoClosure notOlder(_span, &_markBitMap, !asynch /* nmethods */); | |
| 3474 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 3475 | |
| 3476 verify_work_stacks_empty(); | |
| 3477 verify_overflow_empty(); | |
| 3478 | |
| 3479 gch->ensure_parsability(false); // fill TLABs, but no need to retire them | |
| 3480 // Update the saved marks which may affect the root scans. | |
| 3481 gch->save_marks(); | |
| 3482 | |
| 3483 // weak reference processing has not started yet. | |
| 3484 ref_processor()->set_enqueuing_is_done(false); | |
| 3485 | |
| 3486 { | |
| 3487 COMPILER2_PRESENT(DerivedPointerTableDeactivate dpt_deact;) | |
| 3488 gch->rem_set()->prepare_for_younger_refs_iterate(false); // Not parallel. | |
| 3489 gch->gen_process_strong_roots(_cmsGen->level(), | |
| 3490 true, // younger gens are roots | |
| 3491 true, // collecting perm gen | |
| 3492 SharedHeap::ScanningOption(roots_scanning_options()), | |
| 3493 NULL, ¬Older); | |
| 3494 } | |
| 3495 | |
| 3496 // Clear mod-union table; it will be dirtied in the prologue of | |
| 3497 // CMS generation per each younger generation collection. | |
| 3498 | |
| 3499 assert(_modUnionTable.isAllClear(), | |
| 3500 "Was cleared in most recent final checkpoint phase" | |
| 3501 " or no bits are set in the gc_prologue before the start of the next " | |
| 3502 "subsequent marking phase."); | |
| 3503 | |
| 3504 // Temporarily disabled, since pre/post-consumption closures don't | |
| 3505 // care about precleaned cards | |
| 3506 #if 0 | |
| 3507 { | |
| 3508 MemRegion mr = MemRegion((HeapWord*)_virtual_space.low(), | |
| 3509 (HeapWord*)_virtual_space.high()); | |
| 3510 _ct->ct_bs()->preclean_dirty_cards(mr); | |
| 3511 } | |
| 3512 #endif | |
| 3513 | |
| 3514 // Save the end of the used_region of the constituent generations | |
| 3515 // to be used to limit the extent of sweep in each generation. | |
| 3516 save_sweep_limits(); | |
| 3517 if (UseAdaptiveSizePolicy) { | |
| 3518 size_policy()->checkpoint_roots_initial_end(gch->gc_cause()); | |
| 3519 } | |
| 3520 verify_overflow_empty(); | |
| 3521 } | |
| 3522 | |
| 3523 bool CMSCollector::markFromRoots(bool asynch) { | |
| 3524 // we might be tempted to assert that: | |
| 3525 // assert(asynch == !SafepointSynchronize::is_at_safepoint(), | |
| 3526 // "inconsistent argument?"); | |
| 3527 // However that wouldn't be right, because it's possible that | |
| 3528 // a safepoint is indeed in progress as a younger generation | |
| 3529 // stop-the-world GC happens even as we mark in this generation. | |
| 3530 assert(_collectorState == Marking, "inconsistent state?"); | |
| 3531 check_correct_thread_executing(); | |
| 3532 verify_overflow_empty(); | |
| 3533 | |
| 3534 bool res; | |
| 3535 if (asynch) { | |
| 3536 | |
| 3537 // Start the timers for adaptive size policy for the concurrent phases | |
| 3538 // Do it here so that the foreground MS can use the concurrent | |
| 3539 // timer since a foreground MS might has the sweep done concurrently | |
| 3540 // or STW. | |
| 3541 if (UseAdaptiveSizePolicy) { | |
| 3542 size_policy()->concurrent_marking_begin(); | |
| 3543 } | |
| 3544 | |
| 3545 // Weak ref discovery note: We may be discovering weak | |
| 3546 // refs in this generation concurrent (but interleaved) with | |
| 3547 // weak ref discovery by a younger generation collector. | |
| 3548 | |
| 3549 CMSTokenSyncWithLocks ts(true, bitMapLock()); | |
| 3550 TraceCPUTime tcpu(PrintGCDetails, true, gclog_or_tty); | |
| 3551 CMSPhaseAccounting pa(this, "mark", !PrintGCDetails); | |
| 3552 res = markFromRootsWork(asynch); | |
| 3553 if (res) { | |
| 3554 _collectorState = Precleaning; | |
| 3555 } else { // We failed and a foreground collection wants to take over | |
| 3556 assert(_foregroundGCIsActive, "internal state inconsistency"); | |
| 3557 assert(_restart_addr == NULL, "foreground will restart from scratch"); | |
| 3558 if (PrintGCDetails) { | |
| 3559 gclog_or_tty->print_cr("bailing out to foreground collection"); | |
| 3560 } | |
| 3561 } | |
| 3562 if (UseAdaptiveSizePolicy) { | |
| 3563 size_policy()->concurrent_marking_end(); | |
| 3564 } | |
| 3565 } else { | |
| 3566 assert(SafepointSynchronize::is_at_safepoint(), | |
| 3567 "inconsistent with asynch == false"); | |
| 3568 if (UseAdaptiveSizePolicy) { | |
| 3569 size_policy()->ms_collection_marking_begin(); | |
| 3570 } | |
| 3571 // already have locks | |
| 3572 res = markFromRootsWork(asynch); | |
| 3573 _collectorState = FinalMarking; | |
| 3574 if (UseAdaptiveSizePolicy) { | |
| 3575 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 3576 size_policy()->ms_collection_marking_end(gch->gc_cause()); | |
| 3577 } | |
| 3578 } | |
| 3579 verify_overflow_empty(); | |
| 3580 return res; | |
| 3581 } | |
| 3582 | |
| 3583 bool CMSCollector::markFromRootsWork(bool asynch) { | |
| 3584 // iterate over marked bits in bit map, doing a full scan and mark | |
| 3585 // from these roots using the following algorithm: | |
| 3586 // . if oop is to the right of the current scan pointer, | |
| 3587 // mark corresponding bit (we'll process it later) | |
| 3588 // . else (oop is to left of current scan pointer) | |
| 3589 // push oop on marking stack | |
| 3590 // . drain the marking stack | |
| 3591 | |
| 3592 // Note that when we do a marking step we need to hold the | |
| 3593 // bit map lock -- recall that direct allocation (by mutators) | |
| 3594 // and promotion (by younger generation collectors) is also | |
| 3595 // marking the bit map. [the so-called allocate live policy.] | |
| 3596 // Because the implementation of bit map marking is not | |
| 3597 // robust wrt simultaneous marking of bits in the same word, | |
| 3598 // we need to make sure that there is no such interference | |
| 3599 // between concurrent such updates. | |
| 3600 | |
| 3601 // already have locks | |
| 3602 assert_lock_strong(bitMapLock()); | |
| 3603 | |
| 3604 // Clear the revisit stack, just in case there are any | |
| 3605 // obsolete contents from a short-circuited previous CMS cycle. | |
| 3606 _revisitStack.reset(); | |
| 3607 verify_work_stacks_empty(); | |
| 3608 verify_overflow_empty(); | |
| 3609 assert(_revisitStack.isEmpty(), "tabula rasa"); | |
| 3610 | |
| 3611 bool result = false; | |
| 3612 if (CMSConcurrentMTEnabled && ParallelCMSThreads > 0) { | |
| 3613 result = do_marking_mt(asynch); | |
| 3614 } else { | |
| 3615 result = do_marking_st(asynch); | |
| 3616 } | |
| 3617 return result; | |
| 3618 } | |
| 3619 | |
| 3620 // Forward decl | |
| 3621 class CMSConcMarkingTask; | |
| 3622 | |
| 3623 class CMSConcMarkingTerminator: public ParallelTaskTerminator { | |
| 3624 CMSCollector* _collector; | |
| 3625 CMSConcMarkingTask* _task; | |
| 3626 bool _yield; | |
| 3627 protected: | |
| 3628 virtual void yield(); | |
| 3629 public: | |
| 3630 // "n_threads" is the number of threads to be terminated. | |
| 3631 // "queue_set" is a set of work queues of other threads. | |
| 3632 // "collector" is the CMS collector associated with this task terminator. | |
| 3633 // "yield" indicates whether we need the gang as a whole to yield. | |
| 3634 CMSConcMarkingTerminator(int n_threads, TaskQueueSetSuper* queue_set, | |
| 3635 CMSCollector* collector, bool yield) : | |
| 3636 ParallelTaskTerminator(n_threads, queue_set), | |
| 3637 _collector(collector), | |
| 3638 _yield(yield) { } | |
| 3639 | |
| 3640 void set_task(CMSConcMarkingTask* task) { | |
| 3641 _task = task; | |
| 3642 } | |
| 3643 }; | |
| 3644 | |
| 3645 // MT Concurrent Marking Task | |
| 3646 class CMSConcMarkingTask: public YieldingFlexibleGangTask { | |
| 3647 CMSCollector* _collector; | |
| 3648 YieldingFlexibleWorkGang* _workers; // the whole gang | |
| 3649 int _n_workers; // requested/desired # workers | |
| 3650 bool _asynch; | |
| 3651 bool _result; | |
| 3652 CompactibleFreeListSpace* _cms_space; | |
| 3653 CompactibleFreeListSpace* _perm_space; | |
| 3654 HeapWord* _global_finger; | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3655 HeapWord* _restart_addr; |
| 0 | 3656 |
| 3657 // Exposed here for yielding support | |
| 3658 Mutex* const _bit_map_lock; | |
| 3659 | |
| 3660 // The per thread work queues, available here for stealing | |
| 3661 OopTaskQueueSet* _task_queues; | |
| 3662 CMSConcMarkingTerminator _term; | |
| 3663 | |
| 3664 public: | |
| 3665 CMSConcMarkingTask(CMSCollector* collector, | |
| 3666 CompactibleFreeListSpace* cms_space, | |
| 3667 CompactibleFreeListSpace* perm_space, | |
| 3668 bool asynch, int n_workers, | |
| 3669 YieldingFlexibleWorkGang* workers, | |
| 3670 OopTaskQueueSet* task_queues): | |
| 3671 YieldingFlexibleGangTask("Concurrent marking done multi-threaded"), | |
| 3672 _collector(collector), | |
| 3673 _cms_space(cms_space), | |
| 3674 _perm_space(perm_space), | |
| 3675 _asynch(asynch), _n_workers(n_workers), _result(true), | |
| 3676 _workers(workers), _task_queues(task_queues), | |
| 3677 _term(n_workers, task_queues, _collector, asynch), | |
| 3678 _bit_map_lock(collector->bitMapLock()) | |
| 3679 { | |
| 3680 assert(n_workers <= workers->total_workers(), | |
| 3681 "Else termination won't work correctly today"); // XXX FIX ME! | |
| 3682 _requested_size = n_workers; | |
| 3683 _term.set_task(this); | |
| 3684 assert(_cms_space->bottom() < _perm_space->bottom(), | |
| 3685 "Finger incorrectly initialized below"); | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3686 _restart_addr = _global_finger = _cms_space->bottom(); |
| 0 | 3687 } |
| 3688 | |
| 3689 | |
| 3690 OopTaskQueueSet* task_queues() { return _task_queues; } | |
| 3691 | |
| 3692 OopTaskQueue* work_queue(int i) { return task_queues()->queue(i); } | |
| 3693 | |
| 3694 HeapWord** global_finger_addr() { return &_global_finger; } | |
| 3695 | |
| 3696 CMSConcMarkingTerminator* terminator() { return &_term; } | |
| 3697 | |
| 3698 void work(int i); | |
| 3699 | |
| 3700 virtual void coordinator_yield(); // stuff done by coordinator | |
| 3701 bool result() { return _result; } | |
| 3702 | |
| 3703 void reset(HeapWord* ra) { | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3704 assert(_global_finger >= _cms_space->end(), "Postcondition of ::work(i)"); |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3705 assert(_global_finger >= _perm_space->end(), "Postcondition of ::work(i)"); |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3706 assert(ra < _perm_space->end(), "ra too large"); |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3707 _restart_addr = _global_finger = ra; |
| 0 | 3708 _term.reset_for_reuse(); |
| 3709 } | |
| 3710 | |
| 3711 static bool get_work_from_overflow_stack(CMSMarkStack* ovflw_stk, | |
| 3712 OopTaskQueue* work_q); | |
| 3713 | |
| 3714 private: | |
| 3715 void do_scan_and_mark(int i, CompactibleFreeListSpace* sp); | |
| 3716 void do_work_steal(int i); | |
| 3717 void bump_global_finger(HeapWord* f); | |
| 3718 }; | |
| 3719 | |
| 3720 void CMSConcMarkingTerminator::yield() { | |
| 3721 if (ConcurrentMarkSweepThread::should_yield() && | |
| 3722 !_collector->foregroundGCIsActive() && | |
| 3723 _yield) { | |
| 3724 _task->yield(); | |
| 3725 } else { | |
| 3726 ParallelTaskTerminator::yield(); | |
| 3727 } | |
| 3728 } | |
| 3729 | |
| 3730 //////////////////////////////////////////////////////////////// | |
| 3731 // Concurrent Marking Algorithm Sketch | |
| 3732 //////////////////////////////////////////////////////////////// | |
| 3733 // Until all tasks exhausted (both spaces): | |
| 3734 // -- claim next available chunk | |
| 3735 // -- bump global finger via CAS | |
| 3736 // -- find first object that starts in this chunk | |
| 3737 // and start scanning bitmap from that position | |
| 3738 // -- scan marked objects for oops | |
| 3739 // -- CAS-mark target, and if successful: | |
| 3740 // . if target oop is above global finger (volatile read) | |
| 3741 // nothing to do | |
| 3742 // . if target oop is in chunk and above local finger | |
| 3743 // then nothing to do | |
| 3744 // . else push on work-queue | |
| 3745 // -- Deal with possible overflow issues: | |
| 3746 // . local work-queue overflow causes stuff to be pushed on | |
| 3747 // global (common) overflow queue | |
| 3748 // . always first empty local work queue | |
| 3749 // . then get a batch of oops from global work queue if any | |
| 3750 // . then do work stealing | |
| 3751 // -- When all tasks claimed (both spaces) | |
| 3752 // and local work queue empty, | |
| 3753 // then in a loop do: | |
| 3754 // . check global overflow stack; steal a batch of oops and trace | |
| 3755 // . try to steal from other threads oif GOS is empty | |
| 3756 // . if neither is available, offer termination | |
| 3757 // -- Terminate and return result | |
| 3758 // | |
| 3759 void CMSConcMarkingTask::work(int i) { | |
| 3760 elapsedTimer _timer; | |
| 3761 ResourceMark rm; | |
| 3762 HandleMark hm; | |
| 3763 | |
| 3764 DEBUG_ONLY(_collector->verify_overflow_empty();) | |
| 3765 | |
| 3766 // Before we begin work, our work queue should be empty | |
| 3767 assert(work_queue(i)->size() == 0, "Expected to be empty"); | |
| 3768 // Scan the bitmap covering _cms_space, tracing through grey objects. | |
| 3769 _timer.start(); | |
| 3770 do_scan_and_mark(i, _cms_space); | |
| 3771 _timer.stop(); | |
| 3772 if (PrintCMSStatistics != 0) { | |
| 3773 gclog_or_tty->print_cr("Finished cms space scanning in %dth thread: %3.3f sec", | |
| 3774 i, _timer.seconds()); // XXX: need xxx/xxx type of notation, two timers | |
| 3775 } | |
| 3776 | |
| 3777 // ... do the same for the _perm_space | |
| 3778 _timer.reset(); | |
| 3779 _timer.start(); | |
| 3780 do_scan_and_mark(i, _perm_space); | |
| 3781 _timer.stop(); | |
| 3782 if (PrintCMSStatistics != 0) { | |
| 3783 gclog_or_tty->print_cr("Finished perm space scanning in %dth thread: %3.3f sec", | |
| 3784 i, _timer.seconds()); // XXX: need xxx/xxx type of notation, two timers | |
| 3785 } | |
| 3786 | |
| 3787 // ... do work stealing | |
| 3788 _timer.reset(); | |
| 3789 _timer.start(); | |
| 3790 do_work_steal(i); | |
| 3791 _timer.stop(); | |
| 3792 if (PrintCMSStatistics != 0) { | |
| 3793 gclog_or_tty->print_cr("Finished work stealing in %dth thread: %3.3f sec", | |
| 3794 i, _timer.seconds()); // XXX: need xxx/xxx type of notation, two timers | |
| 3795 } | |
| 3796 assert(_collector->_markStack.isEmpty(), "Should have been emptied"); | |
| 3797 assert(work_queue(i)->size() == 0, "Should have been emptied"); | |
| 3798 // Note that under the current task protocol, the | |
| 3799 // following assertion is true even of the spaces | |
| 3800 // expanded since the completion of the concurrent | |
| 3801 // marking. XXX This will likely change under a strict | |
| 3802 // ABORT semantics. | |
| 3803 assert(_global_finger > _cms_space->end() && | |
| 3804 _global_finger >= _perm_space->end(), | |
| 3805 "All tasks have been completed"); | |
| 3806 DEBUG_ONLY(_collector->verify_overflow_empty();) | |
| 3807 } | |
| 3808 | |
| 3809 void CMSConcMarkingTask::bump_global_finger(HeapWord* f) { | |
| 3810 HeapWord* read = _global_finger; | |
| 3811 HeapWord* cur = read; | |
| 3812 while (f > read) { | |
| 3813 cur = read; | |
| 3814 read = (HeapWord*) Atomic::cmpxchg_ptr(f, &_global_finger, cur); | |
| 3815 if (cur == read) { | |
| 3816 // our cas succeeded | |
| 3817 assert(_global_finger >= f, "protocol consistency"); | |
| 3818 break; | |
| 3819 } | |
| 3820 } | |
| 3821 } | |
| 3822 | |
| 3823 // This is really inefficient, and should be redone by | |
| 3824 // using (not yet available) block-read and -write interfaces to the | |
| 3825 // stack and the work_queue. XXX FIX ME !!! | |
| 3826 bool CMSConcMarkingTask::get_work_from_overflow_stack(CMSMarkStack* ovflw_stk, | |
| 3827 OopTaskQueue* work_q) { | |
| 3828 // Fast lock-free check | |
| 3829 if (ovflw_stk->length() == 0) { | |
| 3830 return false; | |
| 3831 } | |
| 3832 assert(work_q->size() == 0, "Shouldn't steal"); | |
| 3833 MutexLockerEx ml(ovflw_stk->par_lock(), | |
| 3834 Mutex::_no_safepoint_check_flag); | |
| 3835 // Grab up to 1/4 the size of the work queue | |
| 3836 size_t num = MIN2((size_t)work_q->max_elems()/4, | |
| 3837 (size_t)ParGCDesiredObjsFromOverflowList); | |
| 3838 num = MIN2(num, ovflw_stk->length()); | |
| 3839 for (int i = (int) num; i > 0; i--) { | |
| 3840 oop cur = ovflw_stk->pop(); | |
| 3841 assert(cur != NULL, "Counted wrong?"); | |
| 3842 work_q->push(cur); | |
| 3843 } | |
| 3844 return num > 0; | |
| 3845 } | |
| 3846 | |
| 3847 void CMSConcMarkingTask::do_scan_and_mark(int i, CompactibleFreeListSpace* sp) { | |
| 3848 SequentialSubTasksDone* pst = sp->conc_par_seq_tasks(); | |
| 3849 int n_tasks = pst->n_tasks(); | |
| 3850 // We allow that there may be no tasks to do here because | |
| 3851 // we are restarting after a stack overflow. | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3852 assert(pst->valid() || n_tasks == 0, "Uninitialized use?"); |
| 0 | 3853 int nth_task = 0; |
| 3854 | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3855 HeapWord* aligned_start = sp->bottom(); |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3856 if (sp->used_region().contains(_restart_addr)) { |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3857 // Align down to a card boundary for the start of 0th task |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3858 // for this space. |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3859 aligned_start = |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3860 (HeapWord*)align_size_down((uintptr_t)_restart_addr, |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3861 CardTableModRefBS::card_size); |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3862 } |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3863 |
| 0 | 3864 size_t chunk_size = sp->marking_task_size(); |
| 3865 while (!pst->is_task_claimed(/* reference */ nth_task)) { | |
| 3866 // Having claimed the nth task in this space, | |
| 3867 // compute the chunk that it corresponds to: | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3868 MemRegion span = MemRegion(aligned_start + nth_task*chunk_size, |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3869 aligned_start + (nth_task+1)*chunk_size); |
| 0 | 3870 // Try and bump the global finger via a CAS; |
| 3871 // note that we need to do the global finger bump | |
| 3872 // _before_ taking the intersection below, because | |
| 3873 // the task corresponding to that region will be | |
| 3874 // deemed done even if the used_region() expands | |
| 3875 // because of allocation -- as it almost certainly will | |
| 3876 // during start-up while the threads yield in the | |
| 3877 // closure below. | |
| 3878 HeapWord* finger = span.end(); | |
| 3879 bump_global_finger(finger); // atomically | |
| 3880 // There are null tasks here corresponding to chunks | |
| 3881 // beyond the "top" address of the space. | |
| 3882 span = span.intersection(sp->used_region()); | |
| 3883 if (!span.is_empty()) { // Non-null task | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3884 HeapWord* prev_obj; |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3885 assert(!span.contains(_restart_addr) || nth_task == 0, |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3886 "Inconsistency"); |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3887 if (nth_task == 0) { |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3888 // For the 0th task, we'll not need to compute a block_start. |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3889 if (span.contains(_restart_addr)) { |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3890 // In the case of a restart because of stack overflow, |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3891 // we might additionally skip a chunk prefix. |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3892 prev_obj = _restart_addr; |
| 0 | 3893 } else { |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3894 prev_obj = span.start(); |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3895 } |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3896 } else { |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3897 // We want to skip the first object because |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3898 // the protocol is to scan any object in its entirety |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3899 // that _starts_ in this span; a fortiori, any |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3900 // object starting in an earlier span is scanned |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3901 // as part of an earlier claimed task. |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3902 // Below we use the "careful" version of block_start |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3903 // so we do not try to navigate uninitialized objects. |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3904 prev_obj = sp->block_start_careful(span.start()); |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3905 // Below we use a variant of block_size that uses the |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3906 // Printezis bits to avoid waiting for allocated |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3907 // objects to become initialized/parsable. |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3908 while (prev_obj < span.start()) { |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3909 size_t sz = sp->block_size_no_stall(prev_obj, _collector); |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3910 if (sz > 0) { |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3911 prev_obj += sz; |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3912 } else { |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3913 // In this case we may end up doing a bit of redundant |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3914 // scanning, but that appears unavoidable, short of |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3915 // locking the free list locks; see bug 6324141. |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3916 break; |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3917 } |
| 0 | 3918 } |
| 3919 } | |
| 3920 if (prev_obj < span.end()) { | |
| 3921 MemRegion my_span = MemRegion(prev_obj, span.end()); | |
| 3922 // Do the marking work within a non-empty span -- | |
| 3923 // the last argument to the constructor indicates whether the | |
| 3924 // iteration should be incremental with periodic yields. | |
| 3925 Par_MarkFromRootsClosure cl(this, _collector, my_span, | |
| 3926 &_collector->_markBitMap, | |
| 3927 work_queue(i), | |
| 3928 &_collector->_markStack, | |
| 3929 &_collector->_revisitStack, | |
| 3930 _asynch); | |
| 3931 _collector->_markBitMap.iterate(&cl, my_span.start(), my_span.end()); | |
| 3932 } // else nothing to do for this task | |
| 3933 } // else nothing to do for this task | |
| 3934 } | |
| 3935 // We'd be tempted to assert here that since there are no | |
| 3936 // more tasks left to claim in this space, the global_finger | |
| 3937 // must exceed space->top() and a fortiori space->end(). However, | |
| 3938 // that would not quite be correct because the bumping of | |
| 3939 // global_finger occurs strictly after the claiming of a task, | |
| 3940 // so by the time we reach here the global finger may not yet | |
| 3941 // have been bumped up by the thread that claimed the last | |
| 3942 // task. | |
| 3943 pst->all_tasks_completed(); | |
| 3944 } | |
| 3945 | |
| 3946 class Par_ConcMarkingClosure: public OopClosure { | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
3947 private: |
| 0 | 3948 CMSCollector* _collector; |
| 3949 MemRegion _span; | |
| 3950 CMSBitMap* _bit_map; | |
| 3951 CMSMarkStack* _overflow_stack; | |
| 3952 CMSMarkStack* _revisit_stack; // XXXXXX Check proper use | |
| 3953 OopTaskQueue* _work_queue; | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
3954 protected: |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
3955 DO_OOP_WORK_DEFN |
| 0 | 3956 public: |
| 3957 Par_ConcMarkingClosure(CMSCollector* collector, OopTaskQueue* work_queue, | |
| 3958 CMSBitMap* bit_map, CMSMarkStack* overflow_stack): | |
| 3959 _collector(collector), | |
| 3960 _span(_collector->_span), | |
| 3961 _work_queue(work_queue), | |
| 3962 _bit_map(bit_map), | |
| 3963 _overflow_stack(overflow_stack) { } // need to initialize revisit stack etc. | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
3964 virtual void do_oop(oop* p); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
3965 virtual void do_oop(narrowOop* p); |
| 0 | 3966 void trim_queue(size_t max); |
| 3967 void handle_stack_overflow(HeapWord* lost); | |
| 3968 }; | |
| 3969 | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3970 // Grey object scanning during work stealing phase -- |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3971 // the salient assumption here is that any references |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3972 // that are in these stolen objects being scanned must |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3973 // already have been initialized (else they would not have |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3974 // been published), so we do not need to check for |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3975 // uninitialized objects before pushing here. |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
3976 void Par_ConcMarkingClosure::do_oop(oop obj) { |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
3977 assert(obj->is_oop_or_null(true), "expected an oop or NULL"); |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
3978 HeapWord* addr = (HeapWord*)obj; |
| 0 | 3979 // Check if oop points into the CMS generation |
| 3980 // and is not marked | |
| 3981 if (_span.contains(addr) && !_bit_map->isMarked(addr)) { | |
| 3982 // a white object ... | |
| 3983 // If we manage to "claim" the object, by being the | |
| 3984 // first thread to mark it, then we push it on our | |
| 3985 // marking stack | |
| 3986 if (_bit_map->par_mark(addr)) { // ... now grey | |
| 3987 // push on work queue (grey set) | |
| 3988 bool simulate_overflow = false; | |
| 3989 NOT_PRODUCT( | |
| 3990 if (CMSMarkStackOverflowALot && | |
| 3991 _collector->simulate_overflow()) { | |
| 3992 // simulate a stack overflow | |
| 3993 simulate_overflow = true; | |
| 3994 } | |
| 3995 ) | |
| 3996 if (simulate_overflow || | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
3997 !(_work_queue->push(obj) || _overflow_stack->par_push(obj))) { |
| 0 | 3998 // stack overflow |
| 3999 if (PrintCMSStatistics != 0) { | |
| 4000 gclog_or_tty->print_cr("CMS marking stack overflow (benign) at " | |
| 4001 SIZE_FORMAT, _overflow_stack->capacity()); | |
| 4002 } | |
| 4003 // We cannot assert that the overflow stack is full because | |
| 4004 // it may have been emptied since. | |
| 4005 assert(simulate_overflow || | |
| 4006 _work_queue->size() == _work_queue->max_elems(), | |
| 4007 "Else push should have succeeded"); | |
| 4008 handle_stack_overflow(addr); | |
| 4009 } | |
| 4010 } // Else, some other thread got there first | |
| 4011 } | |
| 4012 } | |
| 4013 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
4014 void Par_ConcMarkingClosure::do_oop(oop* p) { Par_ConcMarkingClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
4015 void Par_ConcMarkingClosure::do_oop(narrowOop* p) { Par_ConcMarkingClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
4016 |
| 0 | 4017 void Par_ConcMarkingClosure::trim_queue(size_t max) { |
| 4018 while (_work_queue->size() > max) { | |
| 4019 oop new_oop; | |
| 4020 if (_work_queue->pop_local(new_oop)) { | |
| 4021 assert(new_oop->is_oop(), "Should be an oop"); | |
| 4022 assert(_bit_map->isMarked((HeapWord*)new_oop), "Grey object"); | |
| 4023 assert(_span.contains((HeapWord*)new_oop), "Not in span"); | |
| 4024 assert(new_oop->is_parsable(), "Should be parsable"); | |
| 4025 new_oop->oop_iterate(this); // do_oop() above | |
| 4026 } | |
| 4027 } | |
| 4028 } | |
| 4029 | |
| 4030 // Upon stack overflow, we discard (part of) the stack, | |
| 4031 // remembering the least address amongst those discarded | |
| 4032 // in CMSCollector's _restart_address. | |
| 4033 void Par_ConcMarkingClosure::handle_stack_overflow(HeapWord* lost) { | |
| 4034 // We need to do this under a mutex to prevent other | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
4035 // workers from interfering with the work done below. |
| 0 | 4036 MutexLockerEx ml(_overflow_stack->par_lock(), |
| 4037 Mutex::_no_safepoint_check_flag); | |
| 4038 // Remember the least grey address discarded | |
| 4039 HeapWord* ra = (HeapWord*)_overflow_stack->least_value(lost); | |
| 4040 _collector->lower_restart_addr(ra); | |
| 4041 _overflow_stack->reset(); // discard stack contents | |
| 4042 _overflow_stack->expand(); // expand the stack if possible | |
| 4043 } | |
| 4044 | |
| 4045 | |
| 4046 void CMSConcMarkingTask::do_work_steal(int i) { | |
| 4047 OopTaskQueue* work_q = work_queue(i); | |
| 4048 oop obj_to_scan; | |
| 4049 CMSBitMap* bm = &(_collector->_markBitMap); | |
| 4050 CMSMarkStack* ovflw = &(_collector->_markStack); | |
| 4051 int* seed = _collector->hash_seed(i); | |
| 4052 Par_ConcMarkingClosure cl(_collector, work_q, bm, ovflw); | |
| 4053 while (true) { | |
| 4054 cl.trim_queue(0); | |
| 4055 assert(work_q->size() == 0, "Should have been emptied above"); | |
| 4056 if (get_work_from_overflow_stack(ovflw, work_q)) { | |
| 4057 // Can't assert below because the work obtained from the | |
| 4058 // overflow stack may already have been stolen from us. | |
| 4059 // assert(work_q->size() > 0, "Work from overflow stack"); | |
| 4060 continue; | |
| 4061 } else if (task_queues()->steal(i, seed, /* reference */ obj_to_scan)) { | |
| 4062 assert(obj_to_scan->is_oop(), "Should be an oop"); | |
| 4063 assert(bm->isMarked((HeapWord*)obj_to_scan), "Grey object"); | |
| 4064 obj_to_scan->oop_iterate(&cl); | |
| 4065 } else if (terminator()->offer_termination()) { | |
| 4066 assert(work_q->size() == 0, "Impossible!"); | |
| 4067 break; | |
| 4068 } | |
| 4069 } | |
| 4070 } | |
| 4071 | |
| 4072 // This is run by the CMS (coordinator) thread. | |
| 4073 void CMSConcMarkingTask::coordinator_yield() { | |
| 4074 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 4075 "CMS thread should hold CMS token"); | |
| 4076 | |
| 4077 // First give up the locks, then yield, then re-lock | |
| 4078 // We should probably use a constructor/destructor idiom to | |
| 4079 // do this unlock/lock or modify the MutexUnlocker class to | |
| 4080 // serve our purpose. XXX | |
| 4081 assert_lock_strong(_bit_map_lock); | |
| 4082 _bit_map_lock->unlock(); | |
| 4083 ConcurrentMarkSweepThread::desynchronize(true); | |
| 4084 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 4085 _collector->stopTimer(); | |
| 4086 if (PrintCMSStatistics != 0) { | |
| 4087 _collector->incrementYields(); | |
| 4088 } | |
| 4089 _collector->icms_wait(); | |
| 4090 | |
| 4091 // It is possible for whichever thread initiated the yield request | |
| 4092 // not to get a chance to wake up and take the bitmap lock between | |
| 4093 // this thread releasing it and reacquiring it. So, while the | |
| 4094 // should_yield() flag is on, let's sleep for a bit to give the | |
| 4095 // other thread a chance to wake up. The limit imposed on the number | |
| 4096 // of iterations is defensive, to avoid any unforseen circumstances | |
| 4097 // putting us into an infinite loop. Since it's always been this | |
| 4098 // (coordinator_yield()) method that was observed to cause the | |
| 4099 // problem, we are using a parameter (CMSCoordinatorYieldSleepCount) | |
| 4100 // which is by default non-zero. For the other seven methods that | |
| 4101 // also perform the yield operation, as are using a different | |
| 4102 // parameter (CMSYieldSleepCount) which is by default zero. This way we | |
| 4103 // can enable the sleeping for those methods too, if necessary. | |
| 4104 // See 6442774. | |
| 4105 // | |
| 4106 // We really need to reconsider the synchronization between the GC | |
| 4107 // thread and the yield-requesting threads in the future and we | |
| 4108 // should really use wait/notify, which is the recommended | |
| 4109 // way of doing this type of interaction. Additionally, we should | |
| 4110 // consolidate the eight methods that do the yield operation and they | |
| 4111 // are almost identical into one for better maintenability and | |
| 4112 // readability. See 6445193. | |
| 4113 // | |
| 4114 // Tony 2006.06.29 | |
| 4115 for (unsigned i = 0; i < CMSCoordinatorYieldSleepCount && | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
4116 ConcurrentMarkSweepThread::should_yield() && |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
4117 !CMSCollector::foregroundGCIsActive(); ++i) { |
| 0 | 4118 os::sleep(Thread::current(), 1, false); |
| 4119 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 4120 } | |
| 4121 | |
| 4122 ConcurrentMarkSweepThread::synchronize(true); | |
| 4123 _bit_map_lock->lock_without_safepoint_check(); | |
| 4124 _collector->startTimer(); | |
| 4125 } | |
| 4126 | |
| 4127 bool CMSCollector::do_marking_mt(bool asynch) { | |
| 4128 assert(ParallelCMSThreads > 0 && conc_workers() != NULL, "precondition"); | |
| 4129 // In the future this would be determined ergonomically, based | |
| 4130 // on #cpu's, # active mutator threads (and load), and mutation rate. | |
| 4131 int num_workers = ParallelCMSThreads; | |
| 4132 | |
| 4133 CompactibleFreeListSpace* cms_space = _cmsGen->cmsSpace(); | |
| 4134 CompactibleFreeListSpace* perm_space = _permGen->cmsSpace(); | |
| 4135 | |
| 4136 CMSConcMarkingTask tsk(this, cms_space, perm_space, | |
| 4137 asynch, num_workers /* number requested XXX */, | |
| 4138 conc_workers(), task_queues()); | |
| 4139 | |
| 4140 // Since the actual number of workers we get may be different | |
| 4141 // from the number we requested above, do we need to do anything different | |
| 4142 // below? In particular, may be we need to subclass the SequantialSubTasksDone | |
| 4143 // class?? XXX | |
| 4144 cms_space ->initialize_sequential_subtasks_for_marking(num_workers); | |
| 4145 perm_space->initialize_sequential_subtasks_for_marking(num_workers); | |
| 4146 | |
| 4147 // Refs discovery is already non-atomic. | |
| 4148 assert(!ref_processor()->discovery_is_atomic(), "Should be non-atomic"); | |
| 4149 // Mutate the Refs discovery so it is MT during the | |
| 4150 // multi-threaded marking phase. | |
| 4151 ReferenceProcessorMTMutator mt(ref_processor(), num_workers > 1); | |
| 4152 | |
| 4153 conc_workers()->start_task(&tsk); | |
| 4154 while (tsk.yielded()) { | |
| 4155 tsk.coordinator_yield(); | |
| 4156 conc_workers()->continue_task(&tsk); | |
| 4157 } | |
| 4158 // If the task was aborted, _restart_addr will be non-NULL | |
| 4159 assert(tsk.completed() || _restart_addr != NULL, "Inconsistency"); | |
| 4160 while (_restart_addr != NULL) { | |
| 4161 // XXX For now we do not make use of ABORTED state and have not | |
| 4162 // yet implemented the right abort semantics (even in the original | |
| 4163 // single-threaded CMS case). That needs some more investigation | |
| 4164 // and is deferred for now; see CR# TBF. 07252005YSR. XXX | |
| 4165 assert(!CMSAbortSemantics || tsk.aborted(), "Inconsistency"); | |
| 4166 // If _restart_addr is non-NULL, a marking stack overflow | |
| 4167 // occured; we need to do a fresh marking iteration from the | |
| 4168 // indicated restart address. | |
| 4169 if (_foregroundGCIsActive && asynch) { | |
| 4170 // We may be running into repeated stack overflows, having | |
| 4171 // reached the limit of the stack size, while making very | |
| 4172 // slow forward progress. It may be best to bail out and | |
| 4173 // let the foreground collector do its job. | |
| 4174 // Clear _restart_addr, so that foreground GC | |
| 4175 // works from scratch. This avoids the headache of | |
| 4176 // a "rescan" which would otherwise be needed because | |
| 4177 // of the dirty mod union table & card table. | |
| 4178 _restart_addr = NULL; | |
| 4179 return false; | |
| 4180 } | |
| 4181 // Adjust the task to restart from _restart_addr | |
| 4182 tsk.reset(_restart_addr); | |
| 4183 cms_space ->initialize_sequential_subtasks_for_marking(num_workers, | |
| 4184 _restart_addr); | |
| 4185 perm_space->initialize_sequential_subtasks_for_marking(num_workers, | |
| 4186 _restart_addr); | |
| 4187 _restart_addr = NULL; | |
| 4188 // Get the workers going again | |
| 4189 conc_workers()->start_task(&tsk); | |
| 4190 while (tsk.yielded()) { | |
| 4191 tsk.coordinator_yield(); | |
| 4192 conc_workers()->continue_task(&tsk); | |
| 4193 } | |
| 4194 } | |
| 4195 assert(tsk.completed(), "Inconsistency"); | |
| 4196 assert(tsk.result() == true, "Inconsistency"); | |
| 4197 return true; | |
| 4198 } | |
| 4199 | |
| 4200 bool CMSCollector::do_marking_st(bool asynch) { | |
| 4201 ResourceMark rm; | |
| 4202 HandleMark hm; | |
| 4203 | |
| 4204 MarkFromRootsClosure markFromRootsClosure(this, _span, &_markBitMap, | |
| 4205 &_markStack, &_revisitStack, CMSYield && asynch); | |
| 4206 // the last argument to iterate indicates whether the iteration | |
| 4207 // should be incremental with periodic yields. | |
| 4208 _markBitMap.iterate(&markFromRootsClosure); | |
| 4209 // If _restart_addr is non-NULL, a marking stack overflow | |
| 4210 // occured; we need to do a fresh iteration from the | |
| 4211 // indicated restart address. | |
| 4212 while (_restart_addr != NULL) { | |
| 4213 if (_foregroundGCIsActive && asynch) { | |
| 4214 // We may be running into repeated stack overflows, having | |
| 4215 // reached the limit of the stack size, while making very | |
| 4216 // slow forward progress. It may be best to bail out and | |
| 4217 // let the foreground collector do its job. | |
| 4218 // Clear _restart_addr, so that foreground GC | |
| 4219 // works from scratch. This avoids the headache of | |
| 4220 // a "rescan" which would otherwise be needed because | |
| 4221 // of the dirty mod union table & card table. | |
| 4222 _restart_addr = NULL; | |
| 4223 return false; // indicating failure to complete marking | |
| 4224 } | |
| 4225 // Deal with stack overflow: | |
| 4226 // we restart marking from _restart_addr | |
| 4227 HeapWord* ra = _restart_addr; | |
| 4228 markFromRootsClosure.reset(ra); | |
| 4229 _restart_addr = NULL; | |
| 4230 _markBitMap.iterate(&markFromRootsClosure, ra, _span.end()); | |
| 4231 } | |
| 4232 return true; | |
| 4233 } | |
| 4234 | |
| 4235 void CMSCollector::preclean() { | |
| 4236 check_correct_thread_executing(); | |
| 4237 assert(Thread::current()->is_ConcurrentGC_thread(), "Wrong thread"); | |
| 4238 verify_work_stacks_empty(); | |
| 4239 verify_overflow_empty(); | |
| 4240 _abort_preclean = false; | |
| 4241 if (CMSPrecleaningEnabled) { | |
| 4242 _eden_chunk_index = 0; | |
| 4243 size_t used = get_eden_used(); | |
| 4244 size_t capacity = get_eden_capacity(); | |
| 4245 // Don't start sampling unless we will get sufficiently | |
| 4246 // many samples. | |
| 4247 if (used < (capacity/(CMSScheduleRemarkSamplingRatio * 100) | |
| 4248 * CMSScheduleRemarkEdenPenetration)) { | |
| 4249 _start_sampling = true; | |
| 4250 } else { | |
| 4251 _start_sampling = false; | |
| 4252 } | |
| 4253 TraceCPUTime tcpu(PrintGCDetails, true, gclog_or_tty); | |
| 4254 CMSPhaseAccounting pa(this, "preclean", !PrintGCDetails); | |
| 4255 preclean_work(CMSPrecleanRefLists1, CMSPrecleanSurvivors1); | |
| 4256 } | |
| 4257 CMSTokenSync x(true); // is cms thread | |
| 4258 if (CMSPrecleaningEnabled) { | |
| 4259 sample_eden(); | |
| 4260 _collectorState = AbortablePreclean; | |
| 4261 } else { | |
| 4262 _collectorState = FinalMarking; | |
| 4263 } | |
| 4264 verify_work_stacks_empty(); | |
| 4265 verify_overflow_empty(); | |
| 4266 } | |
| 4267 | |
| 4268 // Try and schedule the remark such that young gen | |
| 4269 // occupancy is CMSScheduleRemarkEdenPenetration %. | |
| 4270 void CMSCollector::abortable_preclean() { | |
| 4271 check_correct_thread_executing(); | |
| 4272 assert(CMSPrecleaningEnabled, "Inconsistent control state"); | |
| 4273 assert(_collectorState == AbortablePreclean, "Inconsistent control state"); | |
| 4274 | |
| 4275 // If Eden's current occupancy is below this threshold, | |
| 4276 // immediately schedule the remark; else preclean | |
| 4277 // past the next scavenge in an effort to | |
| 4278 // schedule the pause as described avove. By choosing | |
| 4279 // CMSScheduleRemarkEdenSizeThreshold >= max eden size | |
| 4280 // we will never do an actual abortable preclean cycle. | |
| 4281 if (get_eden_used() > CMSScheduleRemarkEdenSizeThreshold) { | |
| 4282 TraceCPUTime tcpu(PrintGCDetails, true, gclog_or_tty); | |
| 4283 CMSPhaseAccounting pa(this, "abortable-preclean", !PrintGCDetails); | |
| 4284 // We need more smarts in the abortable preclean | |
| 4285 // loop below to deal with cases where allocation | |
| 4286 // in young gen is very very slow, and our precleaning | |
| 4287 // is running a losing race against a horde of | |
| 4288 // mutators intent on flooding us with CMS updates | |
| 4289 // (dirty cards). | |
| 4290 // One, admittedly dumb, strategy is to give up | |
| 4291 // after a certain number of abortable precleaning loops | |
| 4292 // or after a certain maximum time. We want to make | |
| 4293 // this smarter in the next iteration. | |
| 4294 // XXX FIX ME!!! YSR | |
| 4295 size_t loops = 0, workdone = 0, cumworkdone = 0, waited = 0; | |
| 4296 while (!(should_abort_preclean() || | |
| 4297 ConcurrentMarkSweepThread::should_terminate())) { | |
| 4298 workdone = preclean_work(CMSPrecleanRefLists2, CMSPrecleanSurvivors2); | |
| 4299 cumworkdone += workdone; | |
| 4300 loops++; | |
| 4301 // Voluntarily terminate abortable preclean phase if we have | |
| 4302 // been at it for too long. | |
| 4303 if ((CMSMaxAbortablePrecleanLoops != 0) && | |
| 4304 loops >= CMSMaxAbortablePrecleanLoops) { | |
| 4305 if (PrintGCDetails) { | |
| 4306 gclog_or_tty->print(" CMS: abort preclean due to loops "); | |
| 4307 } | |
| 4308 break; | |
| 4309 } | |
| 4310 if (pa.wallclock_millis() > CMSMaxAbortablePrecleanTime) { | |
| 4311 if (PrintGCDetails) { | |
| 4312 gclog_or_tty->print(" CMS: abort preclean due to time "); | |
| 4313 } | |
| 4314 break; | |
| 4315 } | |
| 4316 // If we are doing little work each iteration, we should | |
| 4317 // take a short break. | |
| 4318 if (workdone < CMSAbortablePrecleanMinWorkPerIteration) { | |
| 4319 // Sleep for some time, waiting for work to accumulate | |
| 4320 stopTimer(); | |
| 4321 cmsThread()->wait_on_cms_lock(CMSAbortablePrecleanWaitMillis); | |
| 4322 startTimer(); | |
| 4323 waited++; | |
| 4324 } | |
| 4325 } | |
| 4326 if (PrintCMSStatistics > 0) { | |
| 4327 gclog_or_tty->print(" [%d iterations, %d waits, %d cards)] ", | |
| 4328 loops, waited, cumworkdone); | |
| 4329 } | |
| 4330 } | |
| 4331 CMSTokenSync x(true); // is cms thread | |
| 4332 if (_collectorState != Idling) { | |
| 4333 assert(_collectorState == AbortablePreclean, | |
| 4334 "Spontaneous state transition?"); | |
| 4335 _collectorState = FinalMarking; | |
| 4336 } // Else, a foreground collection completed this CMS cycle. | |
| 4337 return; | |
| 4338 } | |
| 4339 | |
| 4340 // Respond to an Eden sampling opportunity | |
| 4341 void CMSCollector::sample_eden() { | |
| 4342 // Make sure a young gc cannot sneak in between our | |
| 4343 // reading and recording of a sample. | |
| 4344 assert(Thread::current()->is_ConcurrentGC_thread(), | |
| 4345 "Only the cms thread may collect Eden samples"); | |
| 4346 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 4347 "Should collect samples while holding CMS token"); | |
| 4348 if (!_start_sampling) { | |
| 4349 return; | |
| 4350 } | |
| 4351 if (_eden_chunk_array) { | |
| 4352 if (_eden_chunk_index < _eden_chunk_capacity) { | |
| 4353 _eden_chunk_array[_eden_chunk_index] = *_top_addr; // take sample | |
| 4354 assert(_eden_chunk_array[_eden_chunk_index] <= *_end_addr, | |
| 4355 "Unexpected state of Eden"); | |
| 4356 // We'd like to check that what we just sampled is an oop-start address; | |
| 4357 // however, we cannot do that here since the object may not yet have been | |
| 4358 // initialized. So we'll instead do the check when we _use_ this sample | |
| 4359 // later. | |
| 4360 if (_eden_chunk_index == 0 || | |
| 4361 (pointer_delta(_eden_chunk_array[_eden_chunk_index], | |
| 4362 _eden_chunk_array[_eden_chunk_index-1]) | |
| 4363 >= CMSSamplingGrain)) { | |
| 4364 _eden_chunk_index++; // commit sample | |
| 4365 } | |
| 4366 } | |
| 4367 } | |
| 4368 if ((_collectorState == AbortablePreclean) && !_abort_preclean) { | |
| 4369 size_t used = get_eden_used(); | |
| 4370 size_t capacity = get_eden_capacity(); | |
| 4371 assert(used <= capacity, "Unexpected state of Eden"); | |
| 4372 if (used > (capacity/100 * CMSScheduleRemarkEdenPenetration)) { | |
| 4373 _abort_preclean = true; | |
| 4374 } | |
| 4375 } | |
| 4376 } | |
| 4377 | |
| 4378 | |
| 4379 size_t CMSCollector::preclean_work(bool clean_refs, bool clean_survivor) { | |
| 4380 assert(_collectorState == Precleaning || | |
| 4381 _collectorState == AbortablePreclean, "incorrect state"); | |
| 4382 ResourceMark rm; | |
| 4383 HandleMark hm; | |
| 4384 // Do one pass of scrubbing the discovered reference lists | |
| 4385 // to remove any reference objects with strongly-reachable | |
| 4386 // referents. | |
| 4387 if (clean_refs) { | |
| 4388 ReferenceProcessor* rp = ref_processor(); | |
| 4389 CMSPrecleanRefsYieldClosure yield_cl(this); | |
| 4390 assert(rp->span().equals(_span), "Spans should be equal"); | |
| 4391 CMSKeepAliveClosure keep_alive(this, _span, &_markBitMap, | |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
4392 &_markStack, true /* preclean */); |
| 0 | 4393 CMSDrainMarkingStackClosure complete_trace(this, |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
4394 _span, &_markBitMap, &_markStack, |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
4395 &keep_alive, true /* preclean */); |
| 0 | 4396 |
| 4397 // We don't want this step to interfere with a young | |
| 4398 // collection because we don't want to take CPU | |
| 4399 // or memory bandwidth away from the young GC threads | |
| 4400 // (which may be as many as there are CPUs). | |
| 4401 // Note that we don't need to protect ourselves from | |
| 4402 // interference with mutators because they can't | |
| 4403 // manipulate the discovered reference lists nor affect | |
| 4404 // the computed reachability of the referents, the | |
| 4405 // only properties manipulated by the precleaning | |
| 4406 // of these reference lists. | |
| 4407 stopTimer(); | |
| 4408 CMSTokenSyncWithLocks x(true /* is cms thread */, | |
| 4409 bitMapLock()); | |
| 4410 startTimer(); | |
| 4411 sample_eden(); | |
| 4412 // The following will yield to allow foreground | |
| 4413 // collection to proceed promptly. XXX YSR: | |
| 4414 // The code in this method may need further | |
| 4415 // tweaking for better performance and some restructuring | |
| 4416 // for cleaner interfaces. | |
| 4417 rp->preclean_discovered_references( | |
| 4418 rp->is_alive_non_header(), &keep_alive, &complete_trace, | |
| 4419 &yield_cl); | |
| 4420 } | |
| 4421 | |
| 4422 if (clean_survivor) { // preclean the active survivor space(s) | |
| 4423 assert(_young_gen->kind() == Generation::DefNew || | |
| 4424 _young_gen->kind() == Generation::ParNew || | |
| 4425 _young_gen->kind() == Generation::ASParNew, | |
| 4426 "incorrect type for cast"); | |
| 4427 DefNewGeneration* dng = (DefNewGeneration*)_young_gen; | |
| 4428 PushAndMarkClosure pam_cl(this, _span, ref_processor(), | |
| 4429 &_markBitMap, &_modUnionTable, | |
| 4430 &_markStack, &_revisitStack, | |
| 4431 true /* precleaning phase */); | |
| 4432 stopTimer(); | |
| 4433 CMSTokenSyncWithLocks ts(true /* is cms thread */, | |
| 4434 bitMapLock()); | |
| 4435 startTimer(); | |
| 4436 unsigned int before_count = | |
| 4437 GenCollectedHeap::heap()->total_collections(); | |
| 4438 SurvivorSpacePrecleanClosure | |
| 4439 sss_cl(this, _span, &_markBitMap, &_markStack, | |
| 4440 &pam_cl, before_count, CMSYield); | |
| 4441 dng->from()->object_iterate_careful(&sss_cl); | |
| 4442 dng->to()->object_iterate_careful(&sss_cl); | |
| 4443 } | |
| 4444 MarkRefsIntoAndScanClosure | |
| 4445 mrias_cl(_span, ref_processor(), &_markBitMap, &_modUnionTable, | |
| 4446 &_markStack, &_revisitStack, this, CMSYield, | |
| 4447 true /* precleaning phase */); | |
| 4448 // CAUTION: The following closure has persistent state that may need to | |
| 4449 // be reset upon a decrease in the sequence of addresses it | |
| 4450 // processes. | |
| 4451 ScanMarkedObjectsAgainCarefullyClosure | |
| 4452 smoac_cl(this, _span, | |
| 4453 &_markBitMap, &_markStack, &_revisitStack, &mrias_cl, CMSYield); | |
| 4454 | |
| 4455 // Preclean dirty cards in ModUnionTable and CardTable using | |
| 4456 // appropriate convergence criterion; | |
| 4457 // repeat CMSPrecleanIter times unless we find that | |
| 4458 // we are losing. | |
| 4459 assert(CMSPrecleanIter < 10, "CMSPrecleanIter is too large"); | |
| 4460 assert(CMSPrecleanNumerator < CMSPrecleanDenominator, | |
| 4461 "Bad convergence multiplier"); | |
| 4462 assert(CMSPrecleanThreshold >= 100, | |
| 4463 "Unreasonably low CMSPrecleanThreshold"); | |
| 4464 | |
| 4465 size_t numIter, cumNumCards, lastNumCards, curNumCards; | |
| 4466 for (numIter = 0, cumNumCards = lastNumCards = curNumCards = 0; | |
| 4467 numIter < CMSPrecleanIter; | |
| 4468 numIter++, lastNumCards = curNumCards, cumNumCards += curNumCards) { | |
| 4469 curNumCards = preclean_mod_union_table(_cmsGen, &smoac_cl); | |
| 4470 if (CMSPermGenPrecleaningEnabled) { | |
| 4471 curNumCards += preclean_mod_union_table(_permGen, &smoac_cl); | |
| 4472 } | |
| 4473 if (Verbose && PrintGCDetails) { | |
| 4474 gclog_or_tty->print(" (modUnionTable: %d cards)", curNumCards); | |
| 4475 } | |
| 4476 // Either there are very few dirty cards, so re-mark | |
| 4477 // pause will be small anyway, or our pre-cleaning isn't | |
| 4478 // that much faster than the rate at which cards are being | |
| 4479 // dirtied, so we might as well stop and re-mark since | |
| 4480 // precleaning won't improve our re-mark time by much. | |
| 4481 if (curNumCards <= CMSPrecleanThreshold || | |
| 4482 (numIter > 0 && | |
| 4483 (curNumCards * CMSPrecleanDenominator > | |
| 4484 lastNumCards * CMSPrecleanNumerator))) { | |
| 4485 numIter++; | |
| 4486 cumNumCards += curNumCards; | |
| 4487 break; | |
| 4488 } | |
| 4489 } | |
| 4490 curNumCards = preclean_card_table(_cmsGen, &smoac_cl); | |
| 4491 if (CMSPermGenPrecleaningEnabled) { | |
| 4492 curNumCards += preclean_card_table(_permGen, &smoac_cl); | |
| 4493 } | |
| 4494 cumNumCards += curNumCards; | |
| 4495 if (PrintGCDetails && PrintCMSStatistics != 0) { | |
| 4496 gclog_or_tty->print_cr(" (cardTable: %d cards, re-scanned %d cards, %d iterations)", | |
| 4497 curNumCards, cumNumCards, numIter); | |
| 4498 } | |
| 4499 return cumNumCards; // as a measure of useful work done | |
| 4500 } | |
| 4501 | |
| 4502 // PRECLEANING NOTES: | |
| 4503 // Precleaning involves: | |
| 4504 // . reading the bits of the modUnionTable and clearing the set bits. | |
| 4505 // . For the cards corresponding to the set bits, we scan the | |
| 4506 // objects on those cards. This means we need the free_list_lock | |
| 4507 // so that we can safely iterate over the CMS space when scanning | |
| 4508 // for oops. | |
| 4509 // . When we scan the objects, we'll be both reading and setting | |
| 4510 // marks in the marking bit map, so we'll need the marking bit map. | |
| 4511 // . For protecting _collector_state transitions, we take the CGC_lock. | |
| 4512 // Note that any races in the reading of of card table entries by the | |
| 4513 // CMS thread on the one hand and the clearing of those entries by the | |
| 4514 // VM thread or the setting of those entries by the mutator threads on the | |
| 4515 // other are quite benign. However, for efficiency it makes sense to keep | |
| 4516 // the VM thread from racing with the CMS thread while the latter is | |
| 4517 // dirty card info to the modUnionTable. We therefore also use the | |
| 4518 // CGC_lock to protect the reading of the card table and the mod union | |
| 4519 // table by the CM thread. | |
| 4520 // . We run concurrently with mutator updates, so scanning | |
| 4521 // needs to be done carefully -- we should not try to scan | |
| 4522 // potentially uninitialized objects. | |
| 4523 // | |
| 4524 // Locking strategy: While holding the CGC_lock, we scan over and | |
| 4525 // reset a maximal dirty range of the mod union / card tables, then lock | |
| 4526 // the free_list_lock and bitmap lock to do a full marking, then | |
| 4527 // release these locks; and repeat the cycle. This allows for a | |
| 4528 // certain amount of fairness in the sharing of these locks between | |
| 4529 // the CMS collector on the one hand, and the VM thread and the | |
| 4530 // mutators on the other. | |
| 4531 | |
| 4532 // NOTE: preclean_mod_union_table() and preclean_card_table() | |
| 4533 // further below are largely identical; if you need to modify | |
| 4534 // one of these methods, please check the other method too. | |
| 4535 | |
| 4536 size_t CMSCollector::preclean_mod_union_table( | |
| 4537 ConcurrentMarkSweepGeneration* gen, | |
| 4538 ScanMarkedObjectsAgainCarefullyClosure* cl) { | |
| 4539 verify_work_stacks_empty(); | |
| 4540 verify_overflow_empty(); | |
| 4541 | |
| 4542 // strategy: starting with the first card, accumulate contiguous | |
| 4543 // ranges of dirty cards; clear these cards, then scan the region | |
| 4544 // covered by these cards. | |
| 4545 | |
| 4546 // Since all of the MUT is committed ahead, we can just use | |
| 4547 // that, in case the generations expand while we are precleaning. | |
| 4548 // It might also be fine to just use the committed part of the | |
| 4549 // generation, but we might potentially miss cards when the | |
| 4550 // generation is rapidly expanding while we are in the midst | |
| 4551 // of precleaning. | |
| 4552 HeapWord* startAddr = gen->reserved().start(); | |
| 4553 HeapWord* endAddr = gen->reserved().end(); | |
| 4554 | |
| 4555 cl->setFreelistLock(gen->freelistLock()); // needed for yielding | |
| 4556 | |
| 4557 size_t numDirtyCards, cumNumDirtyCards; | |
| 4558 HeapWord *nextAddr, *lastAddr; | |
| 4559 for (cumNumDirtyCards = numDirtyCards = 0, | |
| 4560 nextAddr = lastAddr = startAddr; | |
| 4561 nextAddr < endAddr; | |
| 4562 nextAddr = lastAddr, cumNumDirtyCards += numDirtyCards) { | |
| 4563 | |
| 4564 ResourceMark rm; | |
| 4565 HandleMark hm; | |
| 4566 | |
| 4567 MemRegion dirtyRegion; | |
| 4568 { | |
| 4569 stopTimer(); | |
| 4570 CMSTokenSync ts(true); | |
| 4571 startTimer(); | |
| 4572 sample_eden(); | |
| 4573 // Get dirty region starting at nextOffset (inclusive), | |
| 4574 // simultaneously clearing it. | |
| 4575 dirtyRegion = | |
| 4576 _modUnionTable.getAndClearMarkedRegion(nextAddr, endAddr); | |
| 4577 assert(dirtyRegion.start() >= nextAddr, | |
| 4578 "returned region inconsistent?"); | |
| 4579 } | |
| 4580 // Remember where the next search should begin. | |
| 4581 // The returned region (if non-empty) is a right open interval, | |
| 4582 // so lastOffset is obtained from the right end of that | |
| 4583 // interval. | |
| 4584 lastAddr = dirtyRegion.end(); | |
| 4585 // Should do something more transparent and less hacky XXX | |
| 4586 numDirtyCards = | |
| 4587 _modUnionTable.heapWordDiffToOffsetDiff(dirtyRegion.word_size()); | |
| 4588 | |
| 4589 // We'll scan the cards in the dirty region (with periodic | |
| 4590 // yields for foreground GC as needed). | |
| 4591 if (!dirtyRegion.is_empty()) { | |
| 4592 assert(numDirtyCards > 0, "consistency check"); | |
| 4593 HeapWord* stop_point = NULL; | |
| 4594 { | |
| 4595 stopTimer(); | |
| 4596 CMSTokenSyncWithLocks ts(true, gen->freelistLock(), | |
| 4597 bitMapLock()); | |
| 4598 startTimer(); | |
| 4599 verify_work_stacks_empty(); | |
| 4600 verify_overflow_empty(); | |
| 4601 sample_eden(); | |
| 4602 stop_point = | |
| 4603 gen->cmsSpace()->object_iterate_careful_m(dirtyRegion, cl); | |
| 4604 } | |
| 4605 if (stop_point != NULL) { | |
| 4606 // The careful iteration stopped early either because it found an | |
| 4607 // uninitialized object, or because we were in the midst of an | |
| 4608 // "abortable preclean", which should now be aborted. Redirty | |
| 4609 // the bits corresponding to the partially-scanned or unscanned | |
| 4610 // cards. We'll either restart at the next block boundary or | |
| 4611 // abort the preclean. | |
| 4612 assert((CMSPermGenPrecleaningEnabled && (gen == _permGen)) || | |
| 4613 (_collectorState == AbortablePreclean && should_abort_preclean()), | |
| 4614 "Unparsable objects should only be in perm gen."); | |
| 4615 | |
| 4616 stopTimer(); | |
| 4617 CMSTokenSyncWithLocks ts(true, bitMapLock()); | |
| 4618 startTimer(); | |
| 4619 _modUnionTable.mark_range(MemRegion(stop_point, dirtyRegion.end())); | |
| 4620 if (should_abort_preclean()) { | |
| 4621 break; // out of preclean loop | |
| 4622 } else { | |
| 4623 // Compute the next address at which preclean should pick up; | |
| 4624 // might need bitMapLock in order to read P-bits. | |
| 4625 lastAddr = next_card_start_after_block(stop_point); | |
| 4626 } | |
| 4627 } | |
| 4628 } else { | |
| 4629 assert(lastAddr == endAddr, "consistency check"); | |
| 4630 assert(numDirtyCards == 0, "consistency check"); | |
| 4631 break; | |
| 4632 } | |
| 4633 } | |
| 4634 verify_work_stacks_empty(); | |
| 4635 verify_overflow_empty(); | |
| 4636 return cumNumDirtyCards; | |
| 4637 } | |
| 4638 | |
| 4639 // NOTE: preclean_mod_union_table() above and preclean_card_table() | |
| 4640 // below are largely identical; if you need to modify | |
| 4641 // one of these methods, please check the other method too. | |
| 4642 | |
| 4643 size_t CMSCollector::preclean_card_table(ConcurrentMarkSweepGeneration* gen, | |
| 4644 ScanMarkedObjectsAgainCarefullyClosure* cl) { | |
| 4645 // strategy: it's similar to precleamModUnionTable above, in that | |
| 4646 // we accumulate contiguous ranges of dirty cards, mark these cards | |
| 4647 // precleaned, then scan the region covered by these cards. | |
| 4648 HeapWord* endAddr = (HeapWord*)(gen->_virtual_space.high()); | |
| 4649 HeapWord* startAddr = (HeapWord*)(gen->_virtual_space.low()); | |
| 4650 | |
| 4651 cl->setFreelistLock(gen->freelistLock()); // needed for yielding | |
| 4652 | |
| 4653 size_t numDirtyCards, cumNumDirtyCards; | |
| 4654 HeapWord *lastAddr, *nextAddr; | |
| 4655 | |
| 4656 for (cumNumDirtyCards = numDirtyCards = 0, | |
| 4657 nextAddr = lastAddr = startAddr; | |
| 4658 nextAddr < endAddr; | |
| 4659 nextAddr = lastAddr, cumNumDirtyCards += numDirtyCards) { | |
| 4660 | |
| 4661 ResourceMark rm; | |
| 4662 HandleMark hm; | |
| 4663 | |
| 4664 MemRegion dirtyRegion; | |
| 4665 { | |
| 4666 // See comments in "Precleaning notes" above on why we | |
| 4667 // do this locking. XXX Could the locking overheads be | |
| 4668 // too high when dirty cards are sparse? [I don't think so.] | |
| 4669 stopTimer(); | |
| 4670 CMSTokenSync x(true); // is cms thread | |
| 4671 startTimer(); | |
| 4672 sample_eden(); | |
| 4673 // Get and clear dirty region from card table | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
4674 dirtyRegion = _ct->ct_bs()->dirty_card_range_after_reset( |
|
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
4675 MemRegion(nextAddr, endAddr), |
|
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
4676 true, |
|
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
4677 CardTableModRefBS::precleaned_card_val()); |
|
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
4678 |
| 0 | 4679 assert(dirtyRegion.start() >= nextAddr, |
| 4680 "returned region inconsistent?"); | |
| 4681 } | |
| 4682 lastAddr = dirtyRegion.end(); | |
| 4683 numDirtyCards = | |
| 4684 dirtyRegion.word_size()/CardTableModRefBS::card_size_in_words; | |
| 4685 | |
| 4686 if (!dirtyRegion.is_empty()) { | |
| 4687 stopTimer(); | |
| 4688 CMSTokenSyncWithLocks ts(true, gen->freelistLock(), bitMapLock()); | |
| 4689 startTimer(); | |
| 4690 sample_eden(); | |
| 4691 verify_work_stacks_empty(); | |
| 4692 verify_overflow_empty(); | |
| 4693 HeapWord* stop_point = | |
| 4694 gen->cmsSpace()->object_iterate_careful_m(dirtyRegion, cl); | |
| 4695 if (stop_point != NULL) { | |
| 4696 // The careful iteration stopped early because it found an | |
| 4697 // uninitialized object. Redirty the bits corresponding to the | |
| 4698 // partially-scanned or unscanned cards, and start again at the | |
| 4699 // next block boundary. | |
| 4700 assert(CMSPermGenPrecleaningEnabled || | |
| 4701 (_collectorState == AbortablePreclean && should_abort_preclean()), | |
| 4702 "Unparsable objects should only be in perm gen."); | |
| 4703 _ct->ct_bs()->invalidate(MemRegion(stop_point, dirtyRegion.end())); | |
| 4704 if (should_abort_preclean()) { | |
| 4705 break; // out of preclean loop | |
| 4706 } else { | |
| 4707 // Compute the next address at which preclean should pick up. | |
| 4708 lastAddr = next_card_start_after_block(stop_point); | |
| 4709 } | |
| 4710 } | |
| 4711 } else { | |
| 4712 break; | |
| 4713 } | |
| 4714 } | |
| 4715 verify_work_stacks_empty(); | |
| 4716 verify_overflow_empty(); | |
| 4717 return cumNumDirtyCards; | |
| 4718 } | |
| 4719 | |
| 4720 void CMSCollector::checkpointRootsFinal(bool asynch, | |
| 4721 bool clear_all_soft_refs, bool init_mark_was_synchronous) { | |
| 4722 assert(_collectorState == FinalMarking, "incorrect state transition?"); | |
| 4723 check_correct_thread_executing(); | |
| 4724 // world is stopped at this checkpoint | |
| 4725 assert(SafepointSynchronize::is_at_safepoint(), | |
| 4726 "world should be stopped"); | |
| 4727 verify_work_stacks_empty(); | |
| 4728 verify_overflow_empty(); | |
| 4729 | |
| 4730 SpecializationStats::clear(); | |
| 4731 if (PrintGCDetails) { | |
| 4732 gclog_or_tty->print("[YG occupancy: "SIZE_FORMAT" K ("SIZE_FORMAT" K)]", | |
| 4733 _young_gen->used() / K, | |
| 4734 _young_gen->capacity() / K); | |
| 4735 } | |
| 4736 if (asynch) { | |
| 4737 if (CMSScavengeBeforeRemark) { | |
| 4738 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 4739 // Temporarily set flag to false, GCH->do_collection will | |
| 4740 // expect it to be false and set to true | |
| 4741 FlagSetting fl(gch->_is_gc_active, false); | |
| 4742 NOT_PRODUCT(TraceTime t("Scavenge-Before-Remark", | |
| 4743 PrintGCDetails && Verbose, true, gclog_or_tty);) | |
| 4744 int level = _cmsGen->level() - 1; | |
| 4745 if (level >= 0) { | |
| 4746 gch->do_collection(true, // full (i.e. force, see below) | |
| 4747 false, // !clear_all_soft_refs | |
| 4748 0, // size | |
| 4749 false, // is_tlab | |
| 4750 level // max_level | |
| 4751 ); | |
| 4752 } | |
| 4753 } | |
| 4754 FreelistLocker x(this); | |
| 4755 MutexLockerEx y(bitMapLock(), | |
| 4756 Mutex::_no_safepoint_check_flag); | |
| 4757 assert(!init_mark_was_synchronous, "but that's impossible!"); | |
| 4758 checkpointRootsFinalWork(asynch, clear_all_soft_refs, false); | |
| 4759 } else { | |
| 4760 // already have all the locks | |
| 4761 checkpointRootsFinalWork(asynch, clear_all_soft_refs, | |
| 4762 init_mark_was_synchronous); | |
| 4763 } | |
| 4764 verify_work_stacks_empty(); | |
| 4765 verify_overflow_empty(); | |
| 4766 SpecializationStats::print(); | |
| 4767 } | |
| 4768 | |
| 4769 void CMSCollector::checkpointRootsFinalWork(bool asynch, | |
| 4770 bool clear_all_soft_refs, bool init_mark_was_synchronous) { | |
| 4771 | |
| 4772 NOT_PRODUCT(TraceTime tr("checkpointRootsFinalWork", PrintGCDetails, false, gclog_or_tty);) | |
| 4773 | |
| 4774 assert(haveFreelistLocks(), "must have free list locks"); | |
| 4775 assert_lock_strong(bitMapLock()); | |
| 4776 | |
| 4777 if (UseAdaptiveSizePolicy) { | |
| 4778 size_policy()->checkpoint_roots_final_begin(); | |
| 4779 } | |
| 4780 | |
| 4781 ResourceMark rm; | |
| 4782 HandleMark hm; | |
| 4783 | |
| 4784 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 4785 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
4786 if (should_unload_classes()) { |
| 0 | 4787 CodeCache::gc_prologue(); |
| 4788 } | |
| 4789 assert(haveFreelistLocks(), "must have free list locks"); | |
| 4790 assert_lock_strong(bitMapLock()); | |
| 4791 | |
| 4792 if (!init_mark_was_synchronous) { | |
| 4793 // We might assume that we need not fill TLAB's when | |
| 4794 // CMSScavengeBeforeRemark is set, because we may have just done | |
| 4795 // a scavenge which would have filled all TLAB's -- and besides | |
| 4796 // Eden would be empty. This however may not always be the case -- | |
| 4797 // for instance although we asked for a scavenge, it may not have | |
| 4798 // happened because of a JNI critical section. We probably need | |
| 4799 // a policy for deciding whether we can in that case wait until | |
| 4800 // the critical section releases and then do the remark following | |
| 4801 // the scavenge, and skip it here. In the absence of that policy, | |
| 4802 // or of an indication of whether the scavenge did indeed occur, | |
| 4803 // we cannot rely on TLAB's having been filled and must do | |
| 4804 // so here just in case a scavenge did not happen. | |
| 4805 gch->ensure_parsability(false); // fill TLAB's, but no need to retire them | |
| 4806 // Update the saved marks which may affect the root scans. | |
| 4807 gch->save_marks(); | |
| 4808 | |
| 4809 { | |
| 4810 COMPILER2_PRESENT(DerivedPointerTableDeactivate dpt_deact;) | |
| 4811 | |
| 4812 // Note on the role of the mod union table: | |
| 4813 // Since the marker in "markFromRoots" marks concurrently with | |
| 4814 // mutators, it is possible for some reachable objects not to have been | |
| 4815 // scanned. For instance, an only reference to an object A was | |
| 4816 // placed in object B after the marker scanned B. Unless B is rescanned, | |
| 4817 // A would be collected. Such updates to references in marked objects | |
| 4818 // are detected via the mod union table which is the set of all cards | |
| 4819 // dirtied since the first checkpoint in this GC cycle and prior to | |
| 4820 // the most recent young generation GC, minus those cleaned up by the | |
| 4821 // concurrent precleaning. | |
| 4822 if (CMSParallelRemarkEnabled && ParallelGCThreads > 0) { | |
| 4823 TraceTime t("Rescan (parallel) ", PrintGCDetails, false, gclog_or_tty); | |
| 4824 do_remark_parallel(); | |
| 4825 } else { | |
| 4826 TraceTime t("Rescan (non-parallel) ", PrintGCDetails, false, | |
| 4827 gclog_or_tty); | |
| 4828 do_remark_non_parallel(); | |
| 4829 } | |
| 4830 } | |
| 4831 } else { | |
| 4832 assert(!asynch, "Can't have init_mark_was_synchronous in asynch mode"); | |
| 4833 // The initial mark was stop-world, so there's no rescanning to | |
| 4834 // do; go straight on to the next step below. | |
| 4835 } | |
| 4836 verify_work_stacks_empty(); | |
| 4837 verify_overflow_empty(); | |
| 4838 | |
| 4839 { | |
| 4840 NOT_PRODUCT(TraceTime ts("refProcessingWork", PrintGCDetails, false, gclog_or_tty);) | |
| 4841 refProcessingWork(asynch, clear_all_soft_refs); | |
| 4842 } | |
| 4843 verify_work_stacks_empty(); | |
| 4844 verify_overflow_empty(); | |
| 4845 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
4846 if (should_unload_classes()) { |
| 0 | 4847 CodeCache::gc_epilogue(); |
| 4848 } | |
| 4849 | |
| 4850 // If we encountered any (marking stack / work queue) overflow | |
| 4851 // events during the current CMS cycle, take appropriate | |
| 4852 // remedial measures, where possible, so as to try and avoid | |
| 4853 // recurrence of that condition. | |
| 4854 assert(_markStack.isEmpty(), "No grey objects"); | |
| 4855 size_t ser_ovflw = _ser_pmc_remark_ovflw + _ser_pmc_preclean_ovflw + | |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
4856 _ser_kac_ovflw + _ser_kac_preclean_ovflw; |
| 0 | 4857 if (ser_ovflw > 0) { |
| 4858 if (PrintCMSStatistics != 0) { | |
| 4859 gclog_or_tty->print_cr("Marking stack overflow (benign) " | |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
4860 "(pmc_pc="SIZE_FORMAT", pmc_rm="SIZE_FORMAT", kac="SIZE_FORMAT |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
4861 ", kac_preclean="SIZE_FORMAT")", |
| 0 | 4862 _ser_pmc_preclean_ovflw, _ser_pmc_remark_ovflw, |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
4863 _ser_kac_ovflw, _ser_kac_preclean_ovflw); |
| 0 | 4864 } |
| 4865 _markStack.expand(); | |
| 4866 _ser_pmc_remark_ovflw = 0; | |
| 4867 _ser_pmc_preclean_ovflw = 0; | |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
4868 _ser_kac_preclean_ovflw = 0; |
| 0 | 4869 _ser_kac_ovflw = 0; |
| 4870 } | |
| 4871 if (_par_pmc_remark_ovflw > 0 || _par_kac_ovflw > 0) { | |
| 4872 if (PrintCMSStatistics != 0) { | |
| 4873 gclog_or_tty->print_cr("Work queue overflow (benign) " | |
| 4874 "(pmc_rm="SIZE_FORMAT", kac="SIZE_FORMAT")", | |
| 4875 _par_pmc_remark_ovflw, _par_kac_ovflw); | |
| 4876 } | |
| 4877 _par_pmc_remark_ovflw = 0; | |
| 4878 _par_kac_ovflw = 0; | |
| 4879 } | |
| 4880 if (PrintCMSStatistics != 0) { | |
| 4881 if (_markStack._hit_limit > 0) { | |
| 4882 gclog_or_tty->print_cr(" (benign) Hit max stack size limit ("SIZE_FORMAT")", | |
| 4883 _markStack._hit_limit); | |
| 4884 } | |
| 4885 if (_markStack._failed_double > 0) { | |
| 4886 gclog_or_tty->print_cr(" (benign) Failed stack doubling ("SIZE_FORMAT")," | |
| 4887 " current capacity "SIZE_FORMAT, | |
| 4888 _markStack._failed_double, | |
| 4889 _markStack.capacity()); | |
| 4890 } | |
| 4891 } | |
| 4892 _markStack._hit_limit = 0; | |
| 4893 _markStack._failed_double = 0; | |
| 4894 | |
| 4895 if ((VerifyAfterGC || VerifyDuringGC) && | |
| 4896 GenCollectedHeap::heap()->total_collections() >= VerifyGCStartAt) { | |
| 4897 verify_after_remark(); | |
| 4898 } | |
| 4899 | |
| 4900 // Change under the freelistLocks. | |
| 4901 _collectorState = Sweeping; | |
| 4902 // Call isAllClear() under bitMapLock | |
| 4903 assert(_modUnionTable.isAllClear(), "Should be clear by end of the" | |
| 4904 " final marking"); | |
| 4905 if (UseAdaptiveSizePolicy) { | |
| 4906 size_policy()->checkpoint_roots_final_end(gch->gc_cause()); | |
| 4907 } | |
| 4908 } | |
| 4909 | |
| 4910 // Parallel remark task | |
| 4911 class CMSParRemarkTask: public AbstractGangTask { | |
| 4912 CMSCollector* _collector; | |
| 4913 WorkGang* _workers; | |
| 4914 int _n_workers; | |
| 4915 CompactibleFreeListSpace* _cms_space; | |
| 4916 CompactibleFreeListSpace* _perm_space; | |
| 4917 | |
| 4918 // The per-thread work queues, available here for stealing. | |
| 4919 OopTaskQueueSet* _task_queues; | |
| 4920 ParallelTaskTerminator _term; | |
| 4921 | |
| 4922 public: | |
| 4923 CMSParRemarkTask(CMSCollector* collector, | |
| 4924 CompactibleFreeListSpace* cms_space, | |
| 4925 CompactibleFreeListSpace* perm_space, | |
| 4926 int n_workers, WorkGang* workers, | |
| 4927 OopTaskQueueSet* task_queues): | |
| 4928 AbstractGangTask("Rescan roots and grey objects in parallel"), | |
| 4929 _collector(collector), | |
| 4930 _cms_space(cms_space), _perm_space(perm_space), | |
| 4931 _n_workers(n_workers), | |
| 4932 _workers(workers), | |
| 4933 _task_queues(task_queues), | |
| 4934 _term(workers->total_workers(), task_queues) { } | |
| 4935 | |
| 4936 OopTaskQueueSet* task_queues() { return _task_queues; } | |
| 4937 | |
| 4938 OopTaskQueue* work_queue(int i) { return task_queues()->queue(i); } | |
| 4939 | |
| 4940 ParallelTaskTerminator* terminator() { return &_term; } | |
| 4941 | |
| 4942 void work(int i); | |
| 4943 | |
| 4944 private: | |
| 4945 // Work method in support of parallel rescan ... of young gen spaces | |
| 4946 void do_young_space_rescan(int i, Par_MarkRefsIntoAndScanClosure* cl, | |
| 4947 ContiguousSpace* space, | |
| 4948 HeapWord** chunk_array, size_t chunk_top); | |
| 4949 | |
| 4950 // ... of dirty cards in old space | |
| 4951 void do_dirty_card_rescan_tasks(CompactibleFreeListSpace* sp, int i, | |
| 4952 Par_MarkRefsIntoAndScanClosure* cl); | |
| 4953 | |
| 4954 // ... work stealing for the above | |
| 4955 void do_work_steal(int i, Par_MarkRefsIntoAndScanClosure* cl, int* seed); | |
| 4956 }; | |
| 4957 | |
| 4958 void CMSParRemarkTask::work(int i) { | |
| 4959 elapsedTimer _timer; | |
| 4960 ResourceMark rm; | |
| 4961 HandleMark hm; | |
| 4962 | |
| 4963 // ---------- rescan from roots -------------- | |
| 4964 _timer.start(); | |
| 4965 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 4966 Par_MarkRefsIntoAndScanClosure par_mrias_cl(_collector, | |
| 4967 _collector->_span, _collector->ref_processor(), | |
| 4968 &(_collector->_markBitMap), | |
| 4969 work_queue(i), &(_collector->_revisitStack)); | |
| 4970 | |
| 4971 // Rescan young gen roots first since these are likely | |
| 4972 // coarsely partitioned and may, on that account, constitute | |
| 4973 // the critical path; thus, it's best to start off that | |
| 4974 // work first. | |
| 4975 // ---------- young gen roots -------------- | |
| 4976 { | |
| 4977 DefNewGeneration* dng = _collector->_young_gen->as_DefNewGeneration(); | |
| 4978 EdenSpace* eden_space = dng->eden(); | |
| 4979 ContiguousSpace* from_space = dng->from(); | |
| 4980 ContiguousSpace* to_space = dng->to(); | |
| 4981 | |
| 4982 HeapWord** eca = _collector->_eden_chunk_array; | |
| 4983 size_t ect = _collector->_eden_chunk_index; | |
| 4984 HeapWord** sca = _collector->_survivor_chunk_array; | |
| 4985 size_t sct = _collector->_survivor_chunk_index; | |
| 4986 | |
| 4987 assert(ect <= _collector->_eden_chunk_capacity, "out of bounds"); | |
| 4988 assert(sct <= _collector->_survivor_chunk_capacity, "out of bounds"); | |
| 4989 | |
| 4990 do_young_space_rescan(i, &par_mrias_cl, to_space, NULL, 0); | |
| 4991 do_young_space_rescan(i, &par_mrias_cl, from_space, sca, sct); | |
| 4992 do_young_space_rescan(i, &par_mrias_cl, eden_space, eca, ect); | |
| 4993 | |
| 4994 _timer.stop(); | |
| 4995 if (PrintCMSStatistics != 0) { | |
| 4996 gclog_or_tty->print_cr( | |
| 4997 "Finished young gen rescan work in %dth thread: %3.3f sec", | |
| 4998 i, _timer.seconds()); | |
| 4999 } | |
| 5000 } | |
| 5001 | |
| 5002 // ---------- remaining roots -------------- | |
| 5003 _timer.reset(); | |
| 5004 _timer.start(); | |
| 5005 gch->gen_process_strong_roots(_collector->_cmsGen->level(), | |
| 5006 false, // yg was scanned above | |
| 5007 true, // collecting perm gen | |
| 5008 SharedHeap::ScanningOption(_collector->CMSCollector::roots_scanning_options()), | |
| 5009 NULL, &par_mrias_cl); | |
| 5010 _timer.stop(); | |
| 5011 if (PrintCMSStatistics != 0) { | |
| 5012 gclog_or_tty->print_cr( | |
| 5013 "Finished remaining root rescan work in %dth thread: %3.3f sec", | |
| 5014 i, _timer.seconds()); | |
| 5015 } | |
| 5016 | |
| 5017 // ---------- rescan dirty cards ------------ | |
| 5018 _timer.reset(); | |
| 5019 _timer.start(); | |
| 5020 | |
| 5021 // Do the rescan tasks for each of the two spaces | |
| 5022 // (cms_space and perm_space) in turn. | |
| 5023 do_dirty_card_rescan_tasks(_cms_space, i, &par_mrias_cl); | |
| 5024 do_dirty_card_rescan_tasks(_perm_space, i, &par_mrias_cl); | |
| 5025 _timer.stop(); | |
| 5026 if (PrintCMSStatistics != 0) { | |
| 5027 gclog_or_tty->print_cr( | |
| 5028 "Finished dirty card rescan work in %dth thread: %3.3f sec", | |
| 5029 i, _timer.seconds()); | |
| 5030 } | |
| 5031 | |
| 5032 // ---------- steal work from other threads ... | |
| 5033 // ---------- ... and drain overflow list. | |
| 5034 _timer.reset(); | |
| 5035 _timer.start(); | |
| 5036 do_work_steal(i, &par_mrias_cl, _collector->hash_seed(i)); | |
| 5037 _timer.stop(); | |
| 5038 if (PrintCMSStatistics != 0) { | |
| 5039 gclog_or_tty->print_cr( | |
| 5040 "Finished work stealing in %dth thread: %3.3f sec", | |
| 5041 i, _timer.seconds()); | |
| 5042 } | |
| 5043 } | |
| 5044 | |
| 5045 void | |
| 5046 CMSParRemarkTask::do_young_space_rescan(int i, | |
| 5047 Par_MarkRefsIntoAndScanClosure* cl, ContiguousSpace* space, | |
| 5048 HeapWord** chunk_array, size_t chunk_top) { | |
| 5049 // Until all tasks completed: | |
| 5050 // . claim an unclaimed task | |
| 5051 // . compute region boundaries corresponding to task claimed | |
| 5052 // using chunk_array | |
| 5053 // . par_oop_iterate(cl) over that region | |
| 5054 | |
| 5055 ResourceMark rm; | |
| 5056 HandleMark hm; | |
| 5057 | |
| 5058 SequentialSubTasksDone* pst = space->par_seq_tasks(); | |
| 5059 assert(pst->valid(), "Uninitialized use?"); | |
| 5060 | |
| 5061 int nth_task = 0; | |
| 5062 int n_tasks = pst->n_tasks(); | |
| 5063 | |
| 5064 HeapWord *start, *end; | |
| 5065 while (!pst->is_task_claimed(/* reference */ nth_task)) { | |
| 5066 // We claimed task # nth_task; compute its boundaries. | |
| 5067 if (chunk_top == 0) { // no samples were taken | |
| 5068 assert(nth_task == 0 && n_tasks == 1, "Can have only 1 EdenSpace task"); | |
| 5069 start = space->bottom(); | |
| 5070 end = space->top(); | |
| 5071 } else if (nth_task == 0) { | |
| 5072 start = space->bottom(); | |
| 5073 end = chunk_array[nth_task]; | |
| 5074 } else if (nth_task < (jint)chunk_top) { | |
| 5075 assert(nth_task >= 1, "Control point invariant"); | |
| 5076 start = chunk_array[nth_task - 1]; | |
| 5077 end = chunk_array[nth_task]; | |
| 5078 } else { | |
| 5079 assert(nth_task == (jint)chunk_top, "Control point invariant"); | |
| 5080 start = chunk_array[chunk_top - 1]; | |
| 5081 end = space->top(); | |
| 5082 } | |
| 5083 MemRegion mr(start, end); | |
| 5084 // Verify that mr is in space | |
| 5085 assert(mr.is_empty() || space->used_region().contains(mr), | |
| 5086 "Should be in space"); | |
| 5087 // Verify that "start" is an object boundary | |
| 5088 assert(mr.is_empty() || oop(mr.start())->is_oop(), | |
| 5089 "Should be an oop"); | |
| 5090 space->par_oop_iterate(mr, cl); | |
| 5091 } | |
| 5092 pst->all_tasks_completed(); | |
| 5093 } | |
| 5094 | |
| 5095 void | |
| 5096 CMSParRemarkTask::do_dirty_card_rescan_tasks( | |
| 5097 CompactibleFreeListSpace* sp, int i, | |
| 5098 Par_MarkRefsIntoAndScanClosure* cl) { | |
| 5099 // Until all tasks completed: | |
| 5100 // . claim an unclaimed task | |
| 5101 // . compute region boundaries corresponding to task claimed | |
| 5102 // . transfer dirty bits ct->mut for that region | |
| 5103 // . apply rescanclosure to dirty mut bits for that region | |
| 5104 | |
| 5105 ResourceMark rm; | |
| 5106 HandleMark hm; | |
| 5107 | |
| 5108 OopTaskQueue* work_q = work_queue(i); | |
| 5109 ModUnionClosure modUnionClosure(&(_collector->_modUnionTable)); | |
| 5110 // CAUTION! CAUTION! CAUTION! CAUTION! CAUTION! CAUTION! CAUTION! | |
| 5111 // CAUTION: This closure has state that persists across calls to | |
| 5112 // the work method dirty_range_iterate_clear() in that it has | |
| 5113 // imbedded in it a (subtype of) UpwardsObjectClosure. The | |
| 5114 // use of that state in the imbedded UpwardsObjectClosure instance | |
| 5115 // assumes that the cards are always iterated (even if in parallel | |
| 5116 // by several threads) in monotonically increasing order per each | |
| 5117 // thread. This is true of the implementation below which picks | |
| 5118 // card ranges (chunks) in monotonically increasing order globally | |
| 5119 // and, a-fortiori, in monotonically increasing order per thread | |
| 5120 // (the latter order being a subsequence of the former). | |
| 5121 // If the work code below is ever reorganized into a more chaotic | |
| 5122 // work-partitioning form than the current "sequential tasks" | |
| 5123 // paradigm, the use of that persistent state will have to be | |
| 5124 // revisited and modified appropriately. See also related | |
| 5125 // bug 4756801 work on which should examine this code to make | |
| 5126 // sure that the changes there do not run counter to the | |
| 5127 // assumptions made here and necessary for correctness and | |
| 5128 // efficiency. Note also that this code might yield inefficient | |
| 5129 // behaviour in the case of very large objects that span one or | |
| 5130 // more work chunks. Such objects would potentially be scanned | |
| 5131 // several times redundantly. Work on 4756801 should try and | |
| 5132 // address that performance anomaly if at all possible. XXX | |
| 5133 MemRegion full_span = _collector->_span; | |
| 5134 CMSBitMap* bm = &(_collector->_markBitMap); // shared | |
| 5135 CMSMarkStack* rs = &(_collector->_revisitStack); // shared | |
| 5136 MarkFromDirtyCardsClosure | |
| 5137 greyRescanClosure(_collector, full_span, // entire span of interest | |
| 5138 sp, bm, work_q, rs, cl); | |
| 5139 | |
| 5140 SequentialSubTasksDone* pst = sp->conc_par_seq_tasks(); | |
| 5141 assert(pst->valid(), "Uninitialized use?"); | |
| 5142 int nth_task = 0; | |
| 5143 const int alignment = CardTableModRefBS::card_size * BitsPerWord; | |
| 5144 MemRegion span = sp->used_region(); | |
| 5145 HeapWord* start_addr = span.start(); | |
| 5146 HeapWord* end_addr = (HeapWord*)round_to((intptr_t)span.end(), | |
| 5147 alignment); | |
| 5148 const size_t chunk_size = sp->rescan_task_size(); // in HeapWord units | |
| 5149 assert((HeapWord*)round_to((intptr_t)start_addr, alignment) == | |
| 5150 start_addr, "Check alignment"); | |
| 5151 assert((size_t)round_to((intptr_t)chunk_size, alignment) == | |
| 5152 chunk_size, "Check alignment"); | |
| 5153 | |
| 5154 while (!pst->is_task_claimed(/* reference */ nth_task)) { | |
| 5155 // Having claimed the nth_task, compute corresponding mem-region, | |
| 5156 // which is a-fortiori aligned correctly (i.e. at a MUT bopundary). | |
| 5157 // The alignment restriction ensures that we do not need any | |
| 5158 // synchronization with other gang-workers while setting or | |
| 5159 // clearing bits in thus chunk of the MUT. | |
| 5160 MemRegion this_span = MemRegion(start_addr + nth_task*chunk_size, | |
| 5161 start_addr + (nth_task+1)*chunk_size); | |
| 5162 // The last chunk's end might be way beyond end of the | |
| 5163 // used region. In that case pull back appropriately. | |
| 5164 if (this_span.end() > end_addr) { | |
| 5165 this_span.set_end(end_addr); | |
| 5166 assert(!this_span.is_empty(), "Program logic (calculation of n_tasks)"); | |
| 5167 } | |
| 5168 // Iterate over the dirty cards covering this chunk, marking them | |
| 5169 // precleaned, and setting the corresponding bits in the mod union | |
| 5170 // table. Since we have been careful to partition at Card and MUT-word | |
| 5171 // boundaries no synchronization is needed between parallel threads. | |
| 5172 _collector->_ct->ct_bs()->dirty_card_iterate(this_span, | |
| 5173 &modUnionClosure); | |
| 5174 | |
| 5175 // Having transferred these marks into the modUnionTable, | |
| 5176 // rescan the marked objects on the dirty cards in the modUnionTable. | |
| 5177 // Even if this is at a synchronous collection, the initial marking | |
| 5178 // may have been done during an asynchronous collection so there | |
| 5179 // may be dirty bits in the mod-union table. | |
| 5180 _collector->_modUnionTable.dirty_range_iterate_clear( | |
| 5181 this_span, &greyRescanClosure); | |
| 5182 _collector->_modUnionTable.verifyNoOneBitsInRange( | |
| 5183 this_span.start(), | |
| 5184 this_span.end()); | |
| 5185 } | |
| 5186 pst->all_tasks_completed(); // declare that i am done | |
| 5187 } | |
| 5188 | |
| 5189 // . see if we can share work_queues with ParNew? XXX | |
| 5190 void | |
| 5191 CMSParRemarkTask::do_work_steal(int i, Par_MarkRefsIntoAndScanClosure* cl, | |
| 5192 int* seed) { | |
| 5193 OopTaskQueue* work_q = work_queue(i); | |
| 5194 NOT_PRODUCT(int num_steals = 0;) | |
| 5195 oop obj_to_scan; | |
| 5196 CMSBitMap* bm = &(_collector->_markBitMap); | |
| 5197 size_t num_from_overflow_list = | |
| 5198 MIN2((size_t)work_q->max_elems()/4, | |
| 5199 (size_t)ParGCDesiredObjsFromOverflowList); | |
| 5200 | |
| 5201 while (true) { | |
| 5202 // Completely finish any left over work from (an) earlier round(s) | |
| 5203 cl->trim_queue(0); | |
| 5204 // Now check if there's any work in the overflow list | |
| 5205 if (_collector->par_take_from_overflow_list(num_from_overflow_list, | |
| 5206 work_q)) { | |
| 5207 // found something in global overflow list; | |
| 5208 // not yet ready to go stealing work from others. | |
| 5209 // We'd like to assert(work_q->size() != 0, ...) | |
| 5210 // because we just took work from the overflow list, | |
| 5211 // but of course we can't since all of that could have | |
| 5212 // been already stolen from us. | |
| 5213 // "He giveth and He taketh away." | |
| 5214 continue; | |
| 5215 } | |
| 5216 // Verify that we have no work before we resort to stealing | |
| 5217 assert(work_q->size() == 0, "Have work, shouldn't steal"); | |
| 5218 // Try to steal from other queues that have work | |
| 5219 if (task_queues()->steal(i, seed, /* reference */ obj_to_scan)) { | |
| 5220 NOT_PRODUCT(num_steals++;) | |
| 5221 assert(obj_to_scan->is_oop(), "Oops, not an oop!"); | |
| 5222 assert(bm->isMarked((HeapWord*)obj_to_scan), "Stole an unmarked oop?"); | |
| 5223 // Do scanning work | |
| 5224 obj_to_scan->oop_iterate(cl); | |
| 5225 // Loop around, finish this work, and try to steal some more | |
| 5226 } else if (terminator()->offer_termination()) { | |
| 5227 break; // nirvana from the infinite cycle | |
| 5228 } | |
| 5229 } | |
| 5230 NOT_PRODUCT( | |
| 5231 if (PrintCMSStatistics != 0) { | |
| 5232 gclog_or_tty->print("\n\t(%d: stole %d oops)", i, num_steals); | |
| 5233 } | |
| 5234 ) | |
| 5235 assert(work_q->size() == 0 && _collector->overflow_list_is_empty(), | |
| 5236 "Else our work is not yet done"); | |
| 5237 } | |
| 5238 | |
| 5239 // Return a thread-local PLAB recording array, as appropriate. | |
| 5240 void* CMSCollector::get_data_recorder(int thr_num) { | |
| 5241 if (_survivor_plab_array != NULL && | |
| 5242 (CMSPLABRecordAlways || | |
| 5243 (_collectorState > Marking && _collectorState < FinalMarking))) { | |
| 5244 assert(thr_num < (int)ParallelGCThreads, "thr_num is out of bounds"); | |
| 5245 ChunkArray* ca = &_survivor_plab_array[thr_num]; | |
| 5246 ca->reset(); // clear it so that fresh data is recorded | |
| 5247 return (void*) ca; | |
| 5248 } else { | |
| 5249 return NULL; | |
| 5250 } | |
| 5251 } | |
| 5252 | |
| 5253 // Reset all the thread-local PLAB recording arrays | |
| 5254 void CMSCollector::reset_survivor_plab_arrays() { | |
| 5255 for (uint i = 0; i < ParallelGCThreads; i++) { | |
| 5256 _survivor_plab_array[i].reset(); | |
| 5257 } | |
| 5258 } | |
| 5259 | |
| 5260 // Merge the per-thread plab arrays into the global survivor chunk | |
| 5261 // array which will provide the partitioning of the survivor space | |
| 5262 // for CMS rescan. | |
| 5263 void CMSCollector::merge_survivor_plab_arrays(ContiguousSpace* surv) { | |
| 5264 assert(_survivor_plab_array != NULL, "Error"); | |
| 5265 assert(_survivor_chunk_array != NULL, "Error"); | |
| 5266 assert(_collectorState == FinalMarking, "Error"); | |
| 5267 for (uint j = 0; j < ParallelGCThreads; j++) { | |
| 5268 _cursor[j] = 0; | |
| 5269 } | |
| 5270 HeapWord* top = surv->top(); | |
| 5271 size_t i; | |
| 5272 for (i = 0; i < _survivor_chunk_capacity; i++) { // all sca entries | |
| 5273 HeapWord* min_val = top; // Higher than any PLAB address | |
| 5274 uint min_tid = 0; // position of min_val this round | |
| 5275 for (uint j = 0; j < ParallelGCThreads; j++) { | |
| 5276 ChunkArray* cur_sca = &_survivor_plab_array[j]; | |
| 5277 if (_cursor[j] == cur_sca->end()) { | |
| 5278 continue; | |
| 5279 } | |
| 5280 assert(_cursor[j] < cur_sca->end(), "ctl pt invariant"); | |
| 5281 HeapWord* cur_val = cur_sca->nth(_cursor[j]); | |
| 5282 assert(surv->used_region().contains(cur_val), "Out of bounds value"); | |
| 5283 if (cur_val < min_val) { | |
| 5284 min_tid = j; | |
| 5285 min_val = cur_val; | |
| 5286 } else { | |
| 5287 assert(cur_val < top, "All recorded addresses should be less"); | |
| 5288 } | |
| 5289 } | |
| 5290 // At this point min_val and min_tid are respectively | |
| 5291 // the least address in _survivor_plab_array[j]->nth(_cursor[j]) | |
| 5292 // and the thread (j) that witnesses that address. | |
| 5293 // We record this address in the _survivor_chunk_array[i] | |
| 5294 // and increment _cursor[min_tid] prior to the next round i. | |
| 5295 if (min_val == top) { | |
| 5296 break; | |
| 5297 } | |
| 5298 _survivor_chunk_array[i] = min_val; | |
| 5299 _cursor[min_tid]++; | |
| 5300 } | |
| 5301 // We are all done; record the size of the _survivor_chunk_array | |
| 5302 _survivor_chunk_index = i; // exclusive: [0, i) | |
| 5303 if (PrintCMSStatistics > 0) { | |
| 5304 gclog_or_tty->print(" (Survivor:" SIZE_FORMAT "chunks) ", i); | |
| 5305 } | |
| 5306 // Verify that we used up all the recorded entries | |
| 5307 #ifdef ASSERT | |
| 5308 size_t total = 0; | |
| 5309 for (uint j = 0; j < ParallelGCThreads; j++) { | |
| 5310 assert(_cursor[j] == _survivor_plab_array[j].end(), "Ctl pt invariant"); | |
| 5311 total += _cursor[j]; | |
| 5312 } | |
| 5313 assert(total == _survivor_chunk_index, "Ctl Pt Invariant"); | |
| 5314 // Check that the merged array is in sorted order | |
| 5315 if (total > 0) { | |
| 5316 for (size_t i = 0; i < total - 1; i++) { | |
| 5317 if (PrintCMSStatistics > 0) { | |
| 5318 gclog_or_tty->print(" (chunk" SIZE_FORMAT ":" INTPTR_FORMAT ") ", | |
| 5319 i, _survivor_chunk_array[i]); | |
| 5320 } | |
| 5321 assert(_survivor_chunk_array[i] < _survivor_chunk_array[i+1], | |
| 5322 "Not sorted"); | |
| 5323 } | |
| 5324 } | |
| 5325 #endif // ASSERT | |
| 5326 } | |
| 5327 | |
| 5328 // Set up the space's par_seq_tasks structure for work claiming | |
| 5329 // for parallel rescan of young gen. | |
| 5330 // See ParRescanTask where this is currently used. | |
| 5331 void | |
| 5332 CMSCollector:: | |
| 5333 initialize_sequential_subtasks_for_young_gen_rescan(int n_threads) { | |
| 5334 assert(n_threads > 0, "Unexpected n_threads argument"); | |
| 5335 DefNewGeneration* dng = (DefNewGeneration*)_young_gen; | |
| 5336 | |
| 5337 // Eden space | |
| 5338 { | |
| 5339 SequentialSubTasksDone* pst = dng->eden()->par_seq_tasks(); | |
| 5340 assert(!pst->valid(), "Clobbering existing data?"); | |
| 5341 // Each valid entry in [0, _eden_chunk_index) represents a task. | |
| 5342 size_t n_tasks = _eden_chunk_index + 1; | |
| 5343 assert(n_tasks == 1 || _eden_chunk_array != NULL, "Error"); | |
| 5344 pst->set_par_threads(n_threads); | |
| 5345 pst->set_n_tasks((int)n_tasks); | |
| 5346 } | |
| 5347 | |
| 5348 // Merge the survivor plab arrays into _survivor_chunk_array | |
| 5349 if (_survivor_plab_array != NULL) { | |
| 5350 merge_survivor_plab_arrays(dng->from()); | |
| 5351 } else { | |
| 5352 assert(_survivor_chunk_index == 0, "Error"); | |
| 5353 } | |
| 5354 | |
| 5355 // To space | |
| 5356 { | |
| 5357 SequentialSubTasksDone* pst = dng->to()->par_seq_tasks(); | |
| 5358 assert(!pst->valid(), "Clobbering existing data?"); | |
| 5359 pst->set_par_threads(n_threads); | |
| 5360 pst->set_n_tasks(1); | |
| 5361 assert(pst->valid(), "Error"); | |
| 5362 } | |
| 5363 | |
| 5364 // From space | |
| 5365 { | |
| 5366 SequentialSubTasksDone* pst = dng->from()->par_seq_tasks(); | |
| 5367 assert(!pst->valid(), "Clobbering existing data?"); | |
| 5368 size_t n_tasks = _survivor_chunk_index + 1; | |
| 5369 assert(n_tasks == 1 || _survivor_chunk_array != NULL, "Error"); | |
| 5370 pst->set_par_threads(n_threads); | |
| 5371 pst->set_n_tasks((int)n_tasks); | |
| 5372 assert(pst->valid(), "Error"); | |
| 5373 } | |
| 5374 } | |
| 5375 | |
| 5376 // Parallel version of remark | |
| 5377 void CMSCollector::do_remark_parallel() { | |
| 5378 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 5379 WorkGang* workers = gch->workers(); | |
| 5380 assert(workers != NULL, "Need parallel worker threads."); | |
| 5381 int n_workers = workers->total_workers(); | |
| 5382 CompactibleFreeListSpace* cms_space = _cmsGen->cmsSpace(); | |
| 5383 CompactibleFreeListSpace* perm_space = _permGen->cmsSpace(); | |
| 5384 | |
| 5385 CMSParRemarkTask tsk(this, | |
| 5386 cms_space, perm_space, | |
| 5387 n_workers, workers, task_queues()); | |
| 5388 | |
| 5389 // Set up for parallel process_strong_roots work. | |
| 5390 gch->set_par_threads(n_workers); | |
| 5391 gch->change_strong_roots_parity(); | |
| 5392 // We won't be iterating over the cards in the card table updating | |
| 5393 // the younger_gen cards, so we shouldn't call the following else | |
| 5394 // the verification code as well as subsequent younger_refs_iterate | |
| 5395 // code would get confused. XXX | |
| 5396 // gch->rem_set()->prepare_for_younger_refs_iterate(true); // parallel | |
| 5397 | |
| 5398 // The young gen rescan work will not be done as part of | |
| 5399 // process_strong_roots (which currently doesn't knw how to | |
| 5400 // parallelize such a scan), but rather will be broken up into | |
| 5401 // a set of parallel tasks (via the sampling that the [abortable] | |
| 5402 // preclean phase did of EdenSpace, plus the [two] tasks of | |
| 5403 // scanning the [two] survivor spaces. Further fine-grain | |
| 5404 // parallelization of the scanning of the survivor spaces | |
| 5405 // themselves, and of precleaning of the younger gen itself | |
| 5406 // is deferred to the future. | |
| 5407 initialize_sequential_subtasks_for_young_gen_rescan(n_workers); | |
| 5408 | |
| 5409 // The dirty card rescan work is broken up into a "sequence" | |
| 5410 // of parallel tasks (per constituent space) that are dynamically | |
| 5411 // claimed by the parallel threads. | |
| 5412 cms_space->initialize_sequential_subtasks_for_rescan(n_workers); | |
| 5413 perm_space->initialize_sequential_subtasks_for_rescan(n_workers); | |
| 5414 | |
| 5415 // It turns out that even when we're using 1 thread, doing the work in a | |
| 5416 // separate thread causes wide variance in run times. We can't help this | |
| 5417 // in the multi-threaded case, but we special-case n=1 here to get | |
| 5418 // repeatable measurements of the 1-thread overhead of the parallel code. | |
| 5419 if (n_workers > 1) { | |
| 5420 // Make refs discovery MT-safe | |
| 5421 ReferenceProcessorMTMutator mt(ref_processor(), true); | |
| 5422 workers->run_task(&tsk); | |
| 5423 } else { | |
| 5424 tsk.work(0); | |
| 5425 } | |
| 5426 gch->set_par_threads(0); // 0 ==> non-parallel. | |
| 5427 // restore, single-threaded for now, any preserved marks | |
| 5428 // as a result of work_q overflow | |
| 5429 restore_preserved_marks_if_any(); | |
| 5430 } | |
| 5431 | |
| 5432 // Non-parallel version of remark | |
| 5433 void CMSCollector::do_remark_non_parallel() { | |
| 5434 ResourceMark rm; | |
| 5435 HandleMark hm; | |
| 5436 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 5437 MarkRefsIntoAndScanClosure | |
| 5438 mrias_cl(_span, ref_processor(), &_markBitMap, &_modUnionTable, | |
| 5439 &_markStack, &_revisitStack, this, | |
| 5440 false /* should_yield */, false /* not precleaning */); | |
| 5441 MarkFromDirtyCardsClosure | |
| 5442 markFromDirtyCardsClosure(this, _span, | |
| 5443 NULL, // space is set further below | |
| 5444 &_markBitMap, &_markStack, &_revisitStack, | |
| 5445 &mrias_cl); | |
| 5446 { | |
| 5447 TraceTime t("grey object rescan", PrintGCDetails, false, gclog_or_tty); | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
5448 // Iterate over the dirty cards, setting the corresponding bits in the |
|
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
5449 // mod union table. |
| 0 | 5450 { |
| 5451 ModUnionClosure modUnionClosure(&_modUnionTable); | |
| 5452 _ct->ct_bs()->dirty_card_iterate( | |
| 5453 _cmsGen->used_region(), | |
| 5454 &modUnionClosure); | |
| 5455 _ct->ct_bs()->dirty_card_iterate( | |
| 5456 _permGen->used_region(), | |
| 5457 &modUnionClosure); | |
| 5458 } | |
| 5459 // Having transferred these marks into the modUnionTable, we just need | |
| 5460 // to rescan the marked objects on the dirty cards in the modUnionTable. | |
| 5461 // The initial marking may have been done during an asynchronous | |
| 5462 // collection so there may be dirty bits in the mod-union table. | |
| 5463 const int alignment = | |
| 5464 CardTableModRefBS::card_size * BitsPerWord; | |
| 5465 { | |
| 5466 // ... First handle dirty cards in CMS gen | |
| 5467 markFromDirtyCardsClosure.set_space(_cmsGen->cmsSpace()); | |
| 5468 MemRegion ur = _cmsGen->used_region(); | |
| 5469 HeapWord* lb = ur.start(); | |
| 5470 HeapWord* ub = (HeapWord*)round_to((intptr_t)ur.end(), alignment); | |
| 5471 MemRegion cms_span(lb, ub); | |
| 5472 _modUnionTable.dirty_range_iterate_clear(cms_span, | |
| 5473 &markFromDirtyCardsClosure); | |
| 5474 verify_work_stacks_empty(); | |
| 5475 if (PrintCMSStatistics != 0) { | |
| 5476 gclog_or_tty->print(" (re-scanned "SIZE_FORMAT" dirty cards in cms gen) ", | |
| 5477 markFromDirtyCardsClosure.num_dirty_cards()); | |
| 5478 } | |
| 5479 } | |
| 5480 { | |
| 5481 // .. and then repeat for dirty cards in perm gen | |
| 5482 markFromDirtyCardsClosure.set_space(_permGen->cmsSpace()); | |
| 5483 MemRegion ur = _permGen->used_region(); | |
| 5484 HeapWord* lb = ur.start(); | |
| 5485 HeapWord* ub = (HeapWord*)round_to((intptr_t)ur.end(), alignment); | |
| 5486 MemRegion perm_span(lb, ub); | |
| 5487 _modUnionTable.dirty_range_iterate_clear(perm_span, | |
| 5488 &markFromDirtyCardsClosure); | |
| 5489 verify_work_stacks_empty(); | |
| 5490 if (PrintCMSStatistics != 0) { | |
| 5491 gclog_or_tty->print(" (re-scanned "SIZE_FORMAT" dirty cards in perm gen) ", | |
| 5492 markFromDirtyCardsClosure.num_dirty_cards()); | |
| 5493 } | |
| 5494 } | |
| 5495 } | |
| 5496 if (VerifyDuringGC && | |
| 5497 GenCollectedHeap::heap()->total_collections() >= VerifyGCStartAt) { | |
| 5498 HandleMark hm; // Discard invalid handles created during verification | |
| 5499 Universe::verify(true); | |
| 5500 } | |
| 5501 { | |
| 5502 TraceTime t("root rescan", PrintGCDetails, false, gclog_or_tty); | |
| 5503 | |
| 5504 verify_work_stacks_empty(); | |
| 5505 | |
| 5506 gch->rem_set()->prepare_for_younger_refs_iterate(false); // Not parallel. | |
| 5507 gch->gen_process_strong_roots(_cmsGen->level(), | |
| 5508 true, // younger gens as roots | |
| 5509 true, // collecting perm gen | |
| 5510 SharedHeap::ScanningOption(roots_scanning_options()), | |
| 5511 NULL, &mrias_cl); | |
| 5512 } | |
| 5513 verify_work_stacks_empty(); | |
| 5514 // Restore evacuated mark words, if any, used for overflow list links | |
| 5515 if (!CMSOverflowEarlyRestoration) { | |
| 5516 restore_preserved_marks_if_any(); | |
| 5517 } | |
| 5518 verify_overflow_empty(); | |
| 5519 } | |
| 5520 | |
| 5521 //////////////////////////////////////////////////////// | |
| 5522 // Parallel Reference Processing Task Proxy Class | |
| 5523 //////////////////////////////////////////////////////// | |
| 5524 class CMSRefProcTaskProxy: public AbstractGangTask { | |
| 5525 typedef AbstractRefProcTaskExecutor::ProcessTask ProcessTask; | |
| 5526 CMSCollector* _collector; | |
| 5527 CMSBitMap* _mark_bit_map; | |
|
143
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
5528 const MemRegion _span; |
| 0 | 5529 OopTaskQueueSet* _task_queues; |
| 5530 ParallelTaskTerminator _term; | |
| 5531 ProcessTask& _task; | |
| 5532 | |
| 5533 public: | |
| 5534 CMSRefProcTaskProxy(ProcessTask& task, | |
| 5535 CMSCollector* collector, | |
| 5536 const MemRegion& span, | |
| 5537 CMSBitMap* mark_bit_map, | |
| 5538 int total_workers, | |
| 5539 OopTaskQueueSet* task_queues): | |
| 5540 AbstractGangTask("Process referents by policy in parallel"), | |
| 5541 _task(task), | |
| 5542 _collector(collector), _span(span), _mark_bit_map(mark_bit_map), | |
| 5543 _task_queues(task_queues), | |
| 5544 _term(total_workers, task_queues) | |
|
143
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
5545 { |
|
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
5546 assert(_collector->_span.equals(_span) && !_span.is_empty(), |
|
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
5547 "Inconsistency in _span"); |
|
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
5548 } |
| 0 | 5549 |
| 5550 OopTaskQueueSet* task_queues() { return _task_queues; } | |
| 5551 | |
| 5552 OopTaskQueue* work_queue(int i) { return task_queues()->queue(i); } | |
| 5553 | |
| 5554 ParallelTaskTerminator* terminator() { return &_term; } | |
| 5555 | |
| 5556 void do_work_steal(int i, | |
| 5557 CMSParDrainMarkingStackClosure* drain, | |
| 5558 CMSParKeepAliveClosure* keep_alive, | |
| 5559 int* seed); | |
| 5560 | |
| 5561 virtual void work(int i); | |
| 5562 }; | |
| 5563 | |
| 5564 void CMSRefProcTaskProxy::work(int i) { | |
|
143
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
5565 assert(_collector->_span.equals(_span), "Inconsistency in _span"); |
| 0 | 5566 CMSParKeepAliveClosure par_keep_alive(_collector, _span, |
| 5567 _mark_bit_map, work_queue(i)); | |
| 5568 CMSParDrainMarkingStackClosure par_drain_stack(_collector, _span, | |
| 5569 _mark_bit_map, work_queue(i)); | |
|
143
b5489bb705c9
6662086: 6u4+, 7b11+: CMS never clears referents when -XX:+ParallelRefProcEnabled
ysr
parents:
113
diff
changeset
|
5570 CMSIsAliveClosure is_alive_closure(_span, _mark_bit_map); |
| 0 | 5571 _task.work(i, is_alive_closure, par_keep_alive, par_drain_stack); |
| 5572 if (_task.marks_oops_alive()) { | |
| 5573 do_work_steal(i, &par_drain_stack, &par_keep_alive, | |
| 5574 _collector->hash_seed(i)); | |
| 5575 } | |
| 5576 assert(work_queue(i)->size() == 0, "work_queue should be empty"); | |
| 5577 assert(_collector->_overflow_list == NULL, "non-empty _overflow_list"); | |
| 5578 } | |
| 5579 | |
| 5580 class CMSRefEnqueueTaskProxy: public AbstractGangTask { | |
| 5581 typedef AbstractRefProcTaskExecutor::EnqueueTask EnqueueTask; | |
| 5582 EnqueueTask& _task; | |
| 5583 | |
| 5584 public: | |
| 5585 CMSRefEnqueueTaskProxy(EnqueueTask& task) | |
| 5586 : AbstractGangTask("Enqueue reference objects in parallel"), | |
| 5587 _task(task) | |
| 5588 { } | |
| 5589 | |
| 5590 virtual void work(int i) | |
| 5591 { | |
| 5592 _task.work(i); | |
| 5593 } | |
| 5594 }; | |
| 5595 | |
| 5596 CMSParKeepAliveClosure::CMSParKeepAliveClosure(CMSCollector* collector, | |
| 5597 MemRegion span, CMSBitMap* bit_map, OopTaskQueue* work_queue): | |
| 5598 _collector(collector), | |
| 5599 _span(span), | |
| 5600 _bit_map(bit_map), | |
| 5601 _work_queue(work_queue), | |
| 5602 _mark_and_push(collector, span, bit_map, work_queue), | |
| 5603 _low_water_mark(MIN2((uint)(work_queue->max_elems()/4), | |
| 5604 (uint)(CMSWorkQueueDrainThreshold * ParallelGCThreads))) | |
| 5605 { } | |
| 5606 | |
| 5607 // . see if we can share work_queues with ParNew? XXX | |
| 5608 void CMSRefProcTaskProxy::do_work_steal(int i, | |
| 5609 CMSParDrainMarkingStackClosure* drain, | |
| 5610 CMSParKeepAliveClosure* keep_alive, | |
| 5611 int* seed) { | |
| 5612 OopTaskQueue* work_q = work_queue(i); | |
| 5613 NOT_PRODUCT(int num_steals = 0;) | |
| 5614 oop obj_to_scan; | |
| 5615 size_t num_from_overflow_list = | |
| 5616 MIN2((size_t)work_q->max_elems()/4, | |
| 5617 (size_t)ParGCDesiredObjsFromOverflowList); | |
| 5618 | |
| 5619 while (true) { | |
| 5620 // Completely finish any left over work from (an) earlier round(s) | |
| 5621 drain->trim_queue(0); | |
| 5622 // Now check if there's any work in the overflow list | |
| 5623 if (_collector->par_take_from_overflow_list(num_from_overflow_list, | |
| 5624 work_q)) { | |
| 5625 // Found something in global overflow list; | |
| 5626 // not yet ready to go stealing work from others. | |
| 5627 // We'd like to assert(work_q->size() != 0, ...) | |
| 5628 // because we just took work from the overflow list, | |
| 5629 // but of course we can't, since all of that might have | |
| 5630 // been already stolen from us. | |
| 5631 continue; | |
| 5632 } | |
| 5633 // Verify that we have no work before we resort to stealing | |
| 5634 assert(work_q->size() == 0, "Have work, shouldn't steal"); | |
| 5635 // Try to steal from other queues that have work | |
| 5636 if (task_queues()->steal(i, seed, /* reference */ obj_to_scan)) { | |
| 5637 NOT_PRODUCT(num_steals++;) | |
| 5638 assert(obj_to_scan->is_oop(), "Oops, not an oop!"); | |
| 5639 assert(_mark_bit_map->isMarked((HeapWord*)obj_to_scan), "Stole an unmarked oop?"); | |
| 5640 // Do scanning work | |
| 5641 obj_to_scan->oop_iterate(keep_alive); | |
| 5642 // Loop around, finish this work, and try to steal some more | |
| 5643 } else if (terminator()->offer_termination()) { | |
| 5644 break; // nirvana from the infinite cycle | |
| 5645 } | |
| 5646 } | |
| 5647 NOT_PRODUCT( | |
| 5648 if (PrintCMSStatistics != 0) { | |
| 5649 gclog_or_tty->print("\n\t(%d: stole %d oops)", i, num_steals); | |
| 5650 } | |
| 5651 ) | |
| 5652 } | |
| 5653 | |
| 5654 void CMSRefProcTaskExecutor::execute(ProcessTask& task) | |
| 5655 { | |
| 5656 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 5657 WorkGang* workers = gch->workers(); | |
| 5658 assert(workers != NULL, "Need parallel worker threads."); | |
| 5659 int n_workers = workers->total_workers(); | |
| 5660 CMSRefProcTaskProxy rp_task(task, &_collector, | |
| 5661 _collector.ref_processor()->span(), | |
| 5662 _collector.markBitMap(), | |
| 5663 n_workers, _collector.task_queues()); | |
| 5664 workers->run_task(&rp_task); | |
| 5665 } | |
| 5666 | |
| 5667 void CMSRefProcTaskExecutor::execute(EnqueueTask& task) | |
| 5668 { | |
| 5669 | |
| 5670 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 5671 WorkGang* workers = gch->workers(); | |
| 5672 assert(workers != NULL, "Need parallel worker threads."); | |
| 5673 CMSRefEnqueueTaskProxy enq_task(task); | |
| 5674 workers->run_task(&enq_task); | |
| 5675 } | |
| 5676 | |
| 5677 void CMSCollector::refProcessingWork(bool asynch, bool clear_all_soft_refs) { | |
| 5678 | |
| 5679 ResourceMark rm; | |
| 5680 HandleMark hm; | |
| 5681 ReferencePolicy* soft_ref_policy; | |
| 5682 | |
| 5683 assert(!ref_processor()->enqueuing_is_done(), "Enqueuing should not be complete"); | |
| 5684 // Process weak references. | |
| 5685 if (clear_all_soft_refs) { | |
| 5686 soft_ref_policy = new AlwaysClearPolicy(); | |
| 5687 } else { | |
| 5688 #ifdef COMPILER2 | |
| 5689 soft_ref_policy = new LRUMaxHeapPolicy(); | |
| 5690 #else | |
| 5691 soft_ref_policy = new LRUCurrentHeapPolicy(); | |
| 5692 #endif // COMPILER2 | |
| 5693 } | |
| 5694 verify_work_stacks_empty(); | |
| 5695 | |
| 5696 ReferenceProcessor* rp = ref_processor(); | |
| 5697 assert(rp->span().equals(_span), "Spans should be equal"); | |
| 5698 CMSKeepAliveClosure cmsKeepAliveClosure(this, _span, &_markBitMap, | |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
5699 &_markStack, false /* !preclean */); |
| 0 | 5700 CMSDrainMarkingStackClosure cmsDrainMarkingStackClosure(this, |
| 5701 _span, &_markBitMap, &_markStack, | |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
5702 &cmsKeepAliveClosure, false /* !preclean */); |
| 0 | 5703 { |
| 5704 TraceTime t("weak refs processing", PrintGCDetails, false, gclog_or_tty); | |
| 5705 if (rp->processing_is_mt()) { | |
| 5706 CMSRefProcTaskExecutor task_executor(*this); | |
| 5707 rp->process_discovered_references(soft_ref_policy, | |
| 5708 &_is_alive_closure, | |
| 5709 &cmsKeepAliveClosure, | |
| 5710 &cmsDrainMarkingStackClosure, | |
| 5711 &task_executor); | |
| 5712 } else { | |
| 5713 rp->process_discovered_references(soft_ref_policy, | |
| 5714 &_is_alive_closure, | |
| 5715 &cmsKeepAliveClosure, | |
| 5716 &cmsDrainMarkingStackClosure, | |
| 5717 NULL); | |
| 5718 } | |
| 5719 verify_work_stacks_empty(); | |
| 5720 } | |
| 5721 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
5722 if (should_unload_classes()) { |
| 0 | 5723 { |
| 5724 TraceTime t("class unloading", PrintGCDetails, false, gclog_or_tty); | |
| 5725 | |
| 5726 // Follow SystemDictionary roots and unload classes | |
| 5727 bool purged_class = SystemDictionary::do_unloading(&_is_alive_closure); | |
| 5728 | |
| 5729 // Follow CodeCache roots and unload any methods marked for unloading | |
| 5730 CodeCache::do_unloading(&_is_alive_closure, | |
| 5731 &cmsKeepAliveClosure, | |
| 5732 purged_class); | |
| 5733 | |
| 5734 cmsDrainMarkingStackClosure.do_void(); | |
| 5735 verify_work_stacks_empty(); | |
| 5736 | |
| 5737 // Update subklass/sibling/implementor links in KlassKlass descendants | |
| 5738 assert(!_revisitStack.isEmpty(), "revisit stack should not be empty"); | |
| 5739 oop k; | |
| 5740 while ((k = _revisitStack.pop()) != NULL) { | |
| 5741 ((Klass*)(oopDesc*)k)->follow_weak_klass_links( | |
| 5742 &_is_alive_closure, | |
| 5743 &cmsKeepAliveClosure); | |
| 5744 } | |
| 5745 assert(!ClassUnloading || | |
| 5746 (_markStack.isEmpty() && overflow_list_is_empty()), | |
| 5747 "Should not have found new reachable objects"); | |
| 5748 assert(_revisitStack.isEmpty(), "revisit stack should have been drained"); | |
| 5749 cmsDrainMarkingStackClosure.do_void(); | |
| 5750 verify_work_stacks_empty(); | |
| 5751 } | |
| 5752 | |
| 5753 { | |
| 5754 TraceTime t("scrub symbol & string tables", PrintGCDetails, false, gclog_or_tty); | |
| 5755 // Now clean up stale oops in SymbolTable and StringTable | |
| 5756 SymbolTable::unlink(&_is_alive_closure); | |
| 5757 StringTable::unlink(&_is_alive_closure); | |
| 5758 } | |
| 5759 } | |
| 5760 | |
| 5761 verify_work_stacks_empty(); | |
| 5762 // Restore any preserved marks as a result of mark stack or | |
| 5763 // work queue overflow | |
| 5764 restore_preserved_marks_if_any(); // done single-threaded for now | |
| 5765 | |
| 5766 rp->set_enqueuing_is_done(true); | |
| 5767 if (rp->processing_is_mt()) { | |
| 5768 CMSRefProcTaskExecutor task_executor(*this); | |
| 5769 rp->enqueue_discovered_references(&task_executor); | |
| 5770 } else { | |
| 5771 rp->enqueue_discovered_references(NULL); | |
| 5772 } | |
| 5773 rp->verify_no_references_recorded(); | |
| 5774 assert(!rp->discovery_enabled(), "should have been disabled"); | |
| 5775 | |
| 5776 // JVMTI object tagging is based on JNI weak refs. If any of these | |
| 5777 // refs were cleared then JVMTI needs to update its maps and | |
| 5778 // maybe post ObjectFrees to agents. | |
| 5779 JvmtiExport::cms_ref_processing_epilogue(); | |
| 5780 } | |
| 5781 | |
| 5782 #ifndef PRODUCT | |
| 5783 void CMSCollector::check_correct_thread_executing() { | |
| 5784 Thread* t = Thread::current(); | |
| 5785 // Only the VM thread or the CMS thread should be here. | |
| 5786 assert(t->is_ConcurrentGC_thread() || t->is_VM_thread(), | |
| 5787 "Unexpected thread type"); | |
| 5788 // If this is the vm thread, the foreground process | |
| 5789 // should not be waiting. Note that _foregroundGCIsActive is | |
| 5790 // true while the foreground collector is waiting. | |
| 5791 if (_foregroundGCShouldWait) { | |
| 5792 // We cannot be the VM thread | |
| 5793 assert(t->is_ConcurrentGC_thread(), | |
| 5794 "Should be CMS thread"); | |
| 5795 } else { | |
| 5796 // We can be the CMS thread only if we are in a stop-world | |
| 5797 // phase of CMS collection. | |
| 5798 if (t->is_ConcurrentGC_thread()) { | |
| 5799 assert(_collectorState == InitialMarking || | |
| 5800 _collectorState == FinalMarking, | |
| 5801 "Should be a stop-world phase"); | |
| 5802 // The CMS thread should be holding the CMS_token. | |
| 5803 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 5804 "Potential interference with concurrently " | |
| 5805 "executing VM thread"); | |
| 5806 } | |
| 5807 } | |
| 5808 } | |
| 5809 #endif | |
| 5810 | |
| 5811 void CMSCollector::sweep(bool asynch) { | |
| 5812 assert(_collectorState == Sweeping, "just checking"); | |
| 5813 check_correct_thread_executing(); | |
| 5814 verify_work_stacks_empty(); | |
| 5815 verify_overflow_empty(); | |
| 5816 incrementSweepCount(); | |
| 5817 _sweep_timer.stop(); | |
| 5818 _sweep_estimate.sample(_sweep_timer.seconds()); | |
| 5819 size_policy()->avg_cms_free_at_sweep()->sample(_cmsGen->free()); | |
| 5820 | |
| 5821 // PermGen verification support: If perm gen sweeping is disabled in | |
| 5822 // this cycle, we preserve the perm gen object "deadness" information | |
| 5823 // in the perm_gen_verify_bit_map. In order to do that we traverse | |
| 5824 // all blocks in perm gen and mark all dead objects. | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
5825 if (verifying() && !should_unload_classes()) { |
| 0 | 5826 assert(perm_gen_verify_bit_map()->sizeInBits() != 0, |
| 5827 "Should have already been allocated"); | |
| 5828 MarkDeadObjectsClosure mdo(this, _permGen->cmsSpace(), | |
| 5829 markBitMap(), perm_gen_verify_bit_map()); | |
|
7
2faf283ce688
6621144: CMS: assertion failure "is_cms_thread == Thread::current()->is_ConcurrentGC_thread()"
ysr
parents:
0
diff
changeset
|
5830 if (asynch) { |
|
2faf283ce688
6621144: CMS: assertion failure "is_cms_thread == Thread::current()->is_ConcurrentGC_thread()"
ysr
parents:
0
diff
changeset
|
5831 CMSTokenSyncWithLocks ts(true, _permGen->freelistLock(), |
|
2faf283ce688
6621144: CMS: assertion failure "is_cms_thread == Thread::current()->is_ConcurrentGC_thread()"
ysr
parents:
0
diff
changeset
|
5832 bitMapLock()); |
|
2faf283ce688
6621144: CMS: assertion failure "is_cms_thread == Thread::current()->is_ConcurrentGC_thread()"
ysr
parents:
0
diff
changeset
|
5833 _permGen->cmsSpace()->blk_iterate(&mdo); |
|
2faf283ce688
6621144: CMS: assertion failure "is_cms_thread == Thread::current()->is_ConcurrentGC_thread()"
ysr
parents:
0
diff
changeset
|
5834 } else { |
|
2faf283ce688
6621144: CMS: assertion failure "is_cms_thread == Thread::current()->is_ConcurrentGC_thread()"
ysr
parents:
0
diff
changeset
|
5835 // In the case of synchronous sweep, we already have |
|
2faf283ce688
6621144: CMS: assertion failure "is_cms_thread == Thread::current()->is_ConcurrentGC_thread()"
ysr
parents:
0
diff
changeset
|
5836 // the requisite locks/tokens. |
|
2faf283ce688
6621144: CMS: assertion failure "is_cms_thread == Thread::current()->is_ConcurrentGC_thread()"
ysr
parents:
0
diff
changeset
|
5837 _permGen->cmsSpace()->blk_iterate(&mdo); |
|
2faf283ce688
6621144: CMS: assertion failure "is_cms_thread == Thread::current()->is_ConcurrentGC_thread()"
ysr
parents:
0
diff
changeset
|
5838 } |
| 0 | 5839 } |
| 5840 | |
| 5841 if (asynch) { | |
| 5842 TraceCPUTime tcpu(PrintGCDetails, true, gclog_or_tty); | |
| 5843 CMSPhaseAccounting pa(this, "sweep", !PrintGCDetails); | |
| 5844 // First sweep the old gen then the perm gen | |
| 5845 { | |
| 5846 CMSTokenSyncWithLocks ts(true, _cmsGen->freelistLock(), | |
| 5847 bitMapLock()); | |
| 5848 sweepWork(_cmsGen, asynch); | |
| 5849 } | |
| 5850 | |
| 5851 // Now repeat for perm gen | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
5852 if (should_unload_classes()) { |
| 0 | 5853 CMSTokenSyncWithLocks ts(true, _permGen->freelistLock(), |
| 5854 bitMapLock()); | |
| 5855 sweepWork(_permGen, asynch); | |
| 5856 } | |
| 5857 | |
| 5858 // Update Universe::_heap_*_at_gc figures. | |
| 5859 // We need all the free list locks to make the abstract state | |
| 5860 // transition from Sweeping to Resetting. See detailed note | |
| 5861 // further below. | |
| 5862 { | |
| 5863 CMSTokenSyncWithLocks ts(true, _cmsGen->freelistLock(), | |
| 5864 _permGen->freelistLock()); | |
| 5865 // Update heap occupancy information which is used as | |
| 5866 // input to soft ref clearing policy at the next gc. | |
| 5867 Universe::update_heap_info_at_gc(); | |
| 5868 _collectorState = Resizing; | |
| 5869 } | |
| 5870 } else { | |
| 5871 // already have needed locks | |
| 5872 sweepWork(_cmsGen, asynch); | |
| 5873 | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
5874 if (should_unload_classes()) { |
| 0 | 5875 sweepWork(_permGen, asynch); |
| 5876 } | |
| 5877 // Update heap occupancy information which is used as | |
| 5878 // input to soft ref clearing policy at the next gc. | |
| 5879 Universe::update_heap_info_at_gc(); | |
| 5880 _collectorState = Resizing; | |
| 5881 } | |
| 5882 verify_work_stacks_empty(); | |
| 5883 verify_overflow_empty(); | |
| 5884 | |
| 5885 _sweep_timer.reset(); | |
| 5886 _sweep_timer.start(); | |
| 5887 | |
| 5888 update_time_of_last_gc(os::javaTimeMillis()); | |
| 5889 | |
| 5890 // NOTE on abstract state transitions: | |
| 5891 // Mutators allocate-live and/or mark the mod-union table dirty | |
| 5892 // based on the state of the collection. The former is done in | |
| 5893 // the interval [Marking, Sweeping] and the latter in the interval | |
| 5894 // [Marking, Sweeping). Thus the transitions into the Marking state | |
| 5895 // and out of the Sweeping state must be synchronously visible | |
| 5896 // globally to the mutators. | |
| 5897 // The transition into the Marking state happens with the world | |
| 5898 // stopped so the mutators will globally see it. Sweeping is | |
| 5899 // done asynchronously by the background collector so the transition | |
| 5900 // from the Sweeping state to the Resizing state must be done | |
| 5901 // under the freelistLock (as is the check for whether to | |
| 5902 // allocate-live and whether to dirty the mod-union table). | |
| 5903 assert(_collectorState == Resizing, "Change of collector state to" | |
| 5904 " Resizing must be done under the freelistLocks (plural)"); | |
| 5905 | |
| 5906 // Now that sweeping has been completed, if the GCH's | |
| 5907 // incremental_collection_will_fail flag is set, clear it, | |
| 5908 // thus inviting a younger gen collection to promote into | |
| 5909 // this generation. If such a promotion may still fail, | |
| 5910 // the flag will be set again when a young collection is | |
| 5911 // attempted. | |
| 5912 // I think the incremental_collection_will_fail flag's use | |
| 5913 // is specific to a 2 generation collection policy, so i'll | |
| 5914 // assert that that's the configuration we are operating within. | |
| 5915 // The use of the flag can and should be generalized appropriately | |
| 5916 // in the future to deal with a general n-generation system. | |
| 5917 | |
| 5918 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 5919 assert(gch->collector_policy()->is_two_generation_policy(), | |
| 5920 "Resetting of incremental_collection_will_fail flag" | |
| 5921 " may be incorrect otherwise"); | |
| 5922 gch->clear_incremental_collection_will_fail(); | |
| 5923 gch->update_full_collections_completed(_collection_count_start); | |
| 5924 } | |
| 5925 | |
| 5926 // FIX ME!!! Looks like this belongs in CFLSpace, with | |
| 5927 // CMSGen merely delegating to it. | |
| 5928 void ConcurrentMarkSweepGeneration::setNearLargestChunk() { | |
| 5929 double nearLargestPercent = 0.999; | |
| 5930 HeapWord* minAddr = _cmsSpace->bottom(); | |
| 5931 HeapWord* largestAddr = | |
| 5932 (HeapWord*) _cmsSpace->dictionary()->findLargestDict(); | |
| 5933 if (largestAddr == 0) { | |
| 5934 // The dictionary appears to be empty. In this case | |
| 5935 // try to coalesce at the end of the heap. | |
| 5936 largestAddr = _cmsSpace->end(); | |
| 5937 } | |
| 5938 size_t largestOffset = pointer_delta(largestAddr, minAddr); | |
| 5939 size_t nearLargestOffset = | |
| 5940 (size_t)((double)largestOffset * nearLargestPercent) - MinChunkSize; | |
| 5941 _cmsSpace->set_nearLargestChunk(minAddr + nearLargestOffset); | |
| 5942 } | |
| 5943 | |
| 5944 bool ConcurrentMarkSweepGeneration::isNearLargestChunk(HeapWord* addr) { | |
| 5945 return addr >= _cmsSpace->nearLargestChunk(); | |
| 5946 } | |
| 5947 | |
| 5948 FreeChunk* ConcurrentMarkSweepGeneration::find_chunk_at_end() { | |
| 5949 return _cmsSpace->find_chunk_at_end(); | |
| 5950 } | |
| 5951 | |
| 5952 void ConcurrentMarkSweepGeneration::update_gc_stats(int current_level, | |
| 5953 bool full) { | |
| 5954 // The next lower level has been collected. Gather any statistics | |
| 5955 // that are of interest at this point. | |
| 5956 if (!full && (current_level + 1) == level()) { | |
| 5957 // Gather statistics on the young generation collection. | |
| 5958 collector()->stats().record_gc0_end(used()); | |
| 5959 } | |
| 5960 } | |
| 5961 | |
| 5962 CMSAdaptiveSizePolicy* ConcurrentMarkSweepGeneration::size_policy() { | |
| 5963 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 5964 assert(gch->kind() == CollectedHeap::GenCollectedHeap, | |
| 5965 "Wrong type of heap"); | |
| 5966 CMSAdaptiveSizePolicy* sp = (CMSAdaptiveSizePolicy*) | |
| 5967 gch->gen_policy()->size_policy(); | |
| 5968 assert(sp->is_gc_cms_adaptive_size_policy(), | |
| 5969 "Wrong type of size policy"); | |
| 5970 return sp; | |
| 5971 } | |
| 5972 | |
| 5973 void ConcurrentMarkSweepGeneration::rotate_debug_collection_type() { | |
| 5974 if (PrintGCDetails && Verbose) { | |
| 5975 gclog_or_tty->print("Rotate from %d ", _debug_collection_type); | |
| 5976 } | |
| 5977 _debug_collection_type = (CollectionTypes) (_debug_collection_type + 1); | |
| 5978 _debug_collection_type = | |
| 5979 (CollectionTypes) (_debug_collection_type % Unknown_collection_type); | |
| 5980 if (PrintGCDetails && Verbose) { | |
| 5981 gclog_or_tty->print_cr("to %d ", _debug_collection_type); | |
| 5982 } | |
| 5983 } | |
| 5984 | |
| 5985 void CMSCollector::sweepWork(ConcurrentMarkSweepGeneration* gen, | |
| 5986 bool asynch) { | |
| 5987 // We iterate over the space(s) underlying this generation, | |
| 5988 // checking the mark bit map to see if the bits corresponding | |
| 5989 // to specific blocks are marked or not. Blocks that are | |
| 5990 // marked are live and are not swept up. All remaining blocks | |
| 5991 // are swept up, with coalescing on-the-fly as we sweep up | |
| 5992 // contiguous free and/or garbage blocks: | |
| 5993 // We need to ensure that the sweeper synchronizes with allocators | |
| 5994 // and stop-the-world collectors. In particular, the following | |
| 5995 // locks are used: | |
| 5996 // . CMS token: if this is held, a stop the world collection cannot occur | |
| 5997 // . freelistLock: if this is held no allocation can occur from this | |
| 5998 // generation by another thread | |
| 5999 // . bitMapLock: if this is held, no other thread can access or update | |
| 6000 // | |
| 6001 | |
| 6002 // Note that we need to hold the freelistLock if we use | |
| 6003 // block iterate below; else the iterator might go awry if | |
| 6004 // a mutator (or promotion) causes block contents to change | |
| 6005 // (for instance if the allocator divvies up a block). | |
| 6006 // If we hold the free list lock, for all practical purposes | |
| 6007 // young generation GC's can't occur (they'll usually need to | |
| 6008 // promote), so we might as well prevent all young generation | |
| 6009 // GC's while we do a sweeping step. For the same reason, we might | |
| 6010 // as well take the bit map lock for the entire duration | |
| 6011 | |
| 6012 // check that we hold the requisite locks | |
| 6013 assert(have_cms_token(), "Should hold cms token"); | |
| 6014 assert( (asynch && ConcurrentMarkSweepThread::cms_thread_has_cms_token()) | |
| 6015 || (!asynch && ConcurrentMarkSweepThread::vm_thread_has_cms_token()), | |
| 6016 "Should possess CMS token to sweep"); | |
| 6017 assert_lock_strong(gen->freelistLock()); | |
| 6018 assert_lock_strong(bitMapLock()); | |
| 6019 | |
| 6020 assert(!_sweep_timer.is_active(), "Was switched off in an outer context"); | |
| 6021 gen->cmsSpace()->beginSweepFLCensus((float)(_sweep_timer.seconds()), | |
| 6022 _sweep_estimate.padded_average()); | |
| 6023 gen->setNearLargestChunk(); | |
| 6024 | |
| 6025 { | |
| 6026 SweepClosure sweepClosure(this, gen, &_markBitMap, | |
| 6027 CMSYield && asynch); | |
| 6028 gen->cmsSpace()->blk_iterate_careful(&sweepClosure); | |
| 6029 // We need to free-up/coalesce garbage/blocks from a | |
| 6030 // co-terminal free run. This is done in the SweepClosure | |
| 6031 // destructor; so, do not remove this scope, else the | |
| 6032 // end-of-sweep-census below will be off by a little bit. | |
| 6033 } | |
| 6034 gen->cmsSpace()->sweep_completed(); | |
| 6035 gen->cmsSpace()->endSweepFLCensus(sweepCount()); | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
6036 if (should_unload_classes()) { // unloaded classes this cycle, |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
6037 _concurrent_cycles_since_last_unload = 0; // ... reset count |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
6038 } else { // did not unload classes, |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
6039 _concurrent_cycles_since_last_unload++; // ... increment count |
|
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
6040 } |
| 0 | 6041 } |
| 6042 | |
| 6043 // Reset CMS data structures (for now just the marking bit map) | |
| 6044 // preparatory for the next cycle. | |
| 6045 void CMSCollector::reset(bool asynch) { | |
| 6046 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 6047 CMSAdaptiveSizePolicy* sp = size_policy(); | |
| 6048 AdaptiveSizePolicyOutput(sp, gch->total_collections()); | |
| 6049 if (asynch) { | |
| 6050 CMSTokenSyncWithLocks ts(true, bitMapLock()); | |
| 6051 | |
| 6052 // If the state is not "Resetting", the foreground thread | |
| 6053 // has done a collection and the resetting. | |
| 6054 if (_collectorState != Resetting) { | |
| 6055 assert(_collectorState == Idling, "The state should only change" | |
| 6056 " because the foreground collector has finished the collection"); | |
| 6057 return; | |
| 6058 } | |
| 6059 | |
| 6060 // Clear the mark bitmap (no grey objects to start with) | |
| 6061 // for the next cycle. | |
| 6062 TraceCPUTime tcpu(PrintGCDetails, true, gclog_or_tty); | |
| 6063 CMSPhaseAccounting cmspa(this, "reset", !PrintGCDetails); | |
| 6064 | |
| 6065 HeapWord* curAddr = _markBitMap.startWord(); | |
| 6066 while (curAddr < _markBitMap.endWord()) { | |
| 6067 size_t remaining = pointer_delta(_markBitMap.endWord(), curAddr); | |
| 6068 MemRegion chunk(curAddr, MIN2(CMSBitMapYieldQuantum, remaining)); | |
| 6069 _markBitMap.clear_large_range(chunk); | |
| 6070 if (ConcurrentMarkSweepThread::should_yield() && | |
| 6071 !foregroundGCIsActive() && | |
| 6072 CMSYield) { | |
| 6073 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 6074 "CMS thread should hold CMS token"); | |
| 6075 assert_lock_strong(bitMapLock()); | |
| 6076 bitMapLock()->unlock(); | |
| 6077 ConcurrentMarkSweepThread::desynchronize(true); | |
| 6078 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 6079 stopTimer(); | |
| 6080 if (PrintCMSStatistics != 0) { | |
| 6081 incrementYields(); | |
| 6082 } | |
| 6083 icms_wait(); | |
| 6084 | |
| 6085 // See the comment in coordinator_yield() | |
| 6086 for (unsigned i = 0; i < CMSYieldSleepCount && | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6087 ConcurrentMarkSweepThread::should_yield() && |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6088 !CMSCollector::foregroundGCIsActive(); ++i) { |
| 0 | 6089 os::sleep(Thread::current(), 1, false); |
| 6090 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 6091 } | |
| 6092 | |
| 6093 ConcurrentMarkSweepThread::synchronize(true); | |
| 6094 bitMapLock()->lock_without_safepoint_check(); | |
| 6095 startTimer(); | |
| 6096 } | |
| 6097 curAddr = chunk.end(); | |
| 6098 } | |
| 6099 _collectorState = Idling; | |
| 6100 } else { | |
| 6101 // already have the lock | |
| 6102 assert(_collectorState == Resetting, "just checking"); | |
| 6103 assert_lock_strong(bitMapLock()); | |
| 6104 _markBitMap.clear_all(); | |
| 6105 _collectorState = Idling; | |
| 6106 } | |
| 6107 | |
| 6108 // Stop incremental mode after a cycle completes, so that any future cycles | |
| 6109 // are triggered by allocation. | |
| 6110 stop_icms(); | |
| 6111 | |
| 6112 NOT_PRODUCT( | |
| 6113 if (RotateCMSCollectionTypes) { | |
| 6114 _cmsGen->rotate_debug_collection_type(); | |
| 6115 } | |
| 6116 ) | |
| 6117 } | |
| 6118 | |
| 6119 void CMSCollector::do_CMS_operation(CMS_op_type op) { | |
| 6120 gclog_or_tty->date_stamp(PrintGC && PrintGCDateStamps); | |
| 6121 TraceCPUTime tcpu(PrintGCDetails, true, gclog_or_tty); | |
| 6122 TraceTime t("GC", PrintGC, !PrintGCDetails, gclog_or_tty); | |
| 6123 TraceCollectorStats tcs(counters()); | |
| 6124 | |
| 6125 switch (op) { | |
| 6126 case CMS_op_checkpointRootsInitial: { | |
| 6127 checkpointRootsInitial(true); // asynch | |
| 6128 if (PrintGC) { | |
| 6129 _cmsGen->printOccupancy("initial-mark"); | |
| 6130 } | |
| 6131 break; | |
| 6132 } | |
| 6133 case CMS_op_checkpointRootsFinal: { | |
| 6134 checkpointRootsFinal(true, // asynch | |
| 6135 false, // !clear_all_soft_refs | |
| 6136 false); // !init_mark_was_synchronous | |
| 6137 if (PrintGC) { | |
| 6138 _cmsGen->printOccupancy("remark"); | |
| 6139 } | |
| 6140 break; | |
| 6141 } | |
| 6142 default: | |
| 6143 fatal("No such CMS_op"); | |
| 6144 } | |
| 6145 } | |
| 6146 | |
| 6147 #ifndef PRODUCT | |
| 6148 size_t const CMSCollector::skip_header_HeapWords() { | |
| 6149 return FreeChunk::header_size(); | |
| 6150 } | |
| 6151 | |
| 6152 // Try and collect here conditions that should hold when | |
| 6153 // CMS thread is exiting. The idea is that the foreground GC | |
| 6154 // thread should not be blocked if it wants to terminate | |
| 6155 // the CMS thread and yet continue to run the VM for a while | |
| 6156 // after that. | |
| 6157 void CMSCollector::verify_ok_to_terminate() const { | |
| 6158 assert(Thread::current()->is_ConcurrentGC_thread(), | |
| 6159 "should be called by CMS thread"); | |
| 6160 assert(!_foregroundGCShouldWait, "should be false"); | |
| 6161 // We could check here that all the various low-level locks | |
| 6162 // are not held by the CMS thread, but that is overkill; see | |
| 6163 // also CMSThread::verify_ok_to_terminate() where the CGC_lock | |
| 6164 // is checked. | |
| 6165 } | |
| 6166 #endif | |
| 6167 | |
| 6168 size_t CMSCollector::block_size_using_printezis_bits(HeapWord* addr) const { | |
| 6169 assert(_markBitMap.isMarked(addr) && _markBitMap.isMarked(addr + 1), | |
| 6170 "missing Printezis mark?"); | |
| 6171 HeapWord* nextOneAddr = _markBitMap.getNextMarkedWordAddress(addr + 2); | |
| 6172 size_t size = pointer_delta(nextOneAddr + 1, addr); | |
| 6173 assert(size == CompactibleFreeListSpace::adjustObjectSize(size), | |
| 6174 "alignment problem"); | |
| 6175 assert(size >= 3, "Necessary for Printezis marks to work"); | |
| 6176 return size; | |
| 6177 } | |
| 6178 | |
| 6179 // A variant of the above (block_size_using_printezis_bits()) except | |
| 6180 // that we return 0 if the P-bits are not yet set. | |
| 6181 size_t CMSCollector::block_size_if_printezis_bits(HeapWord* addr) const { | |
| 6182 if (_markBitMap.isMarked(addr)) { | |
| 6183 assert(_markBitMap.isMarked(addr + 1), "Missing Printezis bit?"); | |
| 6184 HeapWord* nextOneAddr = _markBitMap.getNextMarkedWordAddress(addr + 2); | |
| 6185 size_t size = pointer_delta(nextOneAddr + 1, addr); | |
| 6186 assert(size == CompactibleFreeListSpace::adjustObjectSize(size), | |
| 6187 "alignment problem"); | |
| 6188 assert(size >= 3, "Necessary for Printezis marks to work"); | |
| 6189 return size; | |
| 6190 } else { | |
| 6191 assert(!_markBitMap.isMarked(addr + 1), "Bit map inconsistency?"); | |
| 6192 return 0; | |
| 6193 } | |
| 6194 } | |
| 6195 | |
| 6196 HeapWord* CMSCollector::next_card_start_after_block(HeapWord* addr) const { | |
| 6197 size_t sz = 0; | |
| 6198 oop p = (oop)addr; | |
| 187 | 6199 if (p->klass_or_null() != NULL && p->is_parsable()) { |
| 0 | 6200 sz = CompactibleFreeListSpace::adjustObjectSize(p->size()); |
| 6201 } else { | |
| 6202 sz = block_size_using_printezis_bits(addr); | |
| 6203 } | |
| 6204 assert(sz > 0, "size must be nonzero"); | |
| 6205 HeapWord* next_block = addr + sz; | |
| 6206 HeapWord* next_card = (HeapWord*)round_to((uintptr_t)next_block, | |
| 6207 CardTableModRefBS::card_size); | |
| 6208 assert(round_down((uintptr_t)addr, CardTableModRefBS::card_size) < | |
| 6209 round_down((uintptr_t)next_card, CardTableModRefBS::card_size), | |
| 6210 "must be different cards"); | |
| 6211 return next_card; | |
| 6212 } | |
| 6213 | |
| 6214 | |
| 6215 // CMS Bit Map Wrapper ///////////////////////////////////////// | |
| 6216 | |
| 6217 // Construct a CMS bit map infrastructure, but don't create the | |
| 6218 // bit vector itself. That is done by a separate call CMSBitMap::allocate() | |
| 6219 // further below. | |
| 6220 CMSBitMap::CMSBitMap(int shifter, int mutex_rank, const char* mutex_name): | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
6221 _bm(), |
| 0 | 6222 _shifter(shifter), |
| 6223 _lock(mutex_rank >= 0 ? new Mutex(mutex_rank, mutex_name, true) : NULL) | |
| 6224 { | |
| 6225 _bmStartWord = 0; | |
| 6226 _bmWordSize = 0; | |
| 6227 } | |
| 6228 | |
| 6229 bool CMSBitMap::allocate(MemRegion mr) { | |
| 6230 _bmStartWord = mr.start(); | |
| 6231 _bmWordSize = mr.word_size(); | |
| 6232 ReservedSpace brs(ReservedSpace::allocation_align_size_up( | |
| 6233 (_bmWordSize >> (_shifter + LogBitsPerByte)) + 1)); | |
| 6234 if (!brs.is_reserved()) { | |
| 6235 warning("CMS bit map allocation failure"); | |
| 6236 return false; | |
| 6237 } | |
| 6238 // For now we'll just commit all of the bit map up fromt. | |
| 6239 // Later on we'll try to be more parsimonious with swap. | |
| 6240 if (!_virtual_space.initialize(brs, brs.size())) { | |
| 6241 warning("CMS bit map backing store failure"); | |
| 6242 return false; | |
| 6243 } | |
| 6244 assert(_virtual_space.committed_size() == brs.size(), | |
| 6245 "didn't reserve backing store for all of CMS bit map?"); | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
6246 _bm.set_map((BitMap::bm_word_t*)_virtual_space.low()); |
| 0 | 6247 assert(_virtual_space.committed_size() << (_shifter + LogBitsPerByte) >= |
| 6248 _bmWordSize, "inconsistency in bit map sizing"); | |
| 6249 _bm.set_size(_bmWordSize >> _shifter); | |
| 6250 | |
| 6251 // bm.clear(); // can we rely on getting zero'd memory? verify below | |
| 6252 assert(isAllClear(), | |
| 6253 "Expected zero'd memory from ReservedSpace constructor"); | |
| 6254 assert(_bm.size() == heapWordDiffToOffsetDiff(sizeInWords()), | |
| 6255 "consistency check"); | |
| 6256 return true; | |
| 6257 } | |
| 6258 | |
| 6259 void CMSBitMap::dirty_range_iterate_clear(MemRegion mr, MemRegionClosure* cl) { | |
| 6260 HeapWord *next_addr, *end_addr, *last_addr; | |
| 6261 assert_locked(); | |
| 6262 assert(covers(mr), "out-of-range error"); | |
| 6263 // XXX assert that start and end are appropriately aligned | |
| 6264 for (next_addr = mr.start(), end_addr = mr.end(); | |
| 6265 next_addr < end_addr; next_addr = last_addr) { | |
| 6266 MemRegion dirty_region = getAndClearMarkedRegion(next_addr, end_addr); | |
| 6267 last_addr = dirty_region.end(); | |
| 6268 if (!dirty_region.is_empty()) { | |
| 6269 cl->do_MemRegion(dirty_region); | |
| 6270 } else { | |
| 6271 assert(last_addr == end_addr, "program logic"); | |
| 6272 return; | |
| 6273 } | |
| 6274 } | |
| 6275 } | |
| 6276 | |
| 6277 #ifndef PRODUCT | |
| 6278 void CMSBitMap::assert_locked() const { | |
| 6279 CMSLockVerifier::assert_locked(lock()); | |
| 6280 } | |
| 6281 | |
| 6282 bool CMSBitMap::covers(MemRegion mr) const { | |
| 6283 // assert(_bm.map() == _virtual_space.low(), "map inconsistency"); | |
| 6284 assert((size_t)_bm.size() == (_bmWordSize >> _shifter), | |
| 6285 "size inconsistency"); | |
| 6286 return (mr.start() >= _bmStartWord) && | |
| 6287 (mr.end() <= endWord()); | |
| 6288 } | |
| 6289 | |
| 6290 bool CMSBitMap::covers(HeapWord* start, size_t size) const { | |
| 6291 return (start >= _bmStartWord && (start + size) <= endWord()); | |
| 6292 } | |
| 6293 | |
| 6294 void CMSBitMap::verifyNoOneBitsInRange(HeapWord* left, HeapWord* right) { | |
| 6295 // verify that there are no 1 bits in the interval [left, right) | |
| 6296 FalseBitMapClosure falseBitMapClosure; | |
| 6297 iterate(&falseBitMapClosure, left, right); | |
| 6298 } | |
| 6299 | |
| 6300 void CMSBitMap::region_invariant(MemRegion mr) | |
| 6301 { | |
| 6302 assert_locked(); | |
| 6303 // mr = mr.intersection(MemRegion(_bmStartWord, _bmWordSize)); | |
| 6304 assert(!mr.is_empty(), "unexpected empty region"); | |
| 6305 assert(covers(mr), "mr should be covered by bit map"); | |
| 6306 // convert address range into offset range | |
| 6307 size_t start_ofs = heapWordToOffset(mr.start()); | |
| 6308 // Make sure that end() is appropriately aligned | |
| 6309 assert(mr.end() == (HeapWord*)round_to((intptr_t)mr.end(), | |
| 6310 (1 << (_shifter+LogHeapWordSize))), | |
| 6311 "Misaligned mr.end()"); | |
| 6312 size_t end_ofs = heapWordToOffset(mr.end()); | |
| 6313 assert(end_ofs > start_ofs, "Should mark at least one bit"); | |
| 6314 } | |
| 6315 | |
| 6316 #endif | |
| 6317 | |
| 6318 bool CMSMarkStack::allocate(size_t size) { | |
| 6319 // allocate a stack of the requisite depth | |
| 6320 ReservedSpace rs(ReservedSpace::allocation_align_size_up( | |
| 6321 size * sizeof(oop))); | |
| 6322 if (!rs.is_reserved()) { | |
| 6323 warning("CMSMarkStack allocation failure"); | |
| 6324 return false; | |
| 6325 } | |
| 6326 if (!_virtual_space.initialize(rs, rs.size())) { | |
| 6327 warning("CMSMarkStack backing store failure"); | |
| 6328 return false; | |
| 6329 } | |
| 6330 assert(_virtual_space.committed_size() == rs.size(), | |
| 6331 "didn't reserve backing store for all of CMS stack?"); | |
| 6332 _base = (oop*)(_virtual_space.low()); | |
| 6333 _index = 0; | |
| 6334 _capacity = size; | |
| 6335 NOT_PRODUCT(_max_depth = 0); | |
| 6336 return true; | |
| 6337 } | |
| 6338 | |
| 6339 // XXX FIX ME !!! In the MT case we come in here holding a | |
| 6340 // leaf lock. For printing we need to take a further lock | |
| 6341 // which has lower rank. We need to recallibrate the two | |
| 6342 // lock-ranks involved in order to be able to rpint the | |
| 6343 // messages below. (Or defer the printing to the caller. | |
| 6344 // For now we take the expedient path of just disabling the | |
| 6345 // messages for the problematic case.) | |
| 6346 void CMSMarkStack::expand() { | |
| 6347 assert(_capacity <= CMSMarkStackSizeMax, "stack bigger than permitted"); | |
| 6348 if (_capacity == CMSMarkStackSizeMax) { | |
| 6349 if (_hit_limit++ == 0 && !CMSConcurrentMTEnabled && PrintGCDetails) { | |
| 6350 // We print a warning message only once per CMS cycle. | |
| 6351 gclog_or_tty->print_cr(" (benign) Hit CMSMarkStack max size limit"); | |
| 6352 } | |
| 6353 return; | |
| 6354 } | |
| 6355 // Double capacity if possible | |
| 6356 size_t new_capacity = MIN2(_capacity*2, CMSMarkStackSizeMax); | |
| 6357 // Do not give up existing stack until we have managed to | |
| 6358 // get the double capacity that we desired. | |
| 6359 ReservedSpace rs(ReservedSpace::allocation_align_size_up( | |
| 6360 new_capacity * sizeof(oop))); | |
| 6361 if (rs.is_reserved()) { | |
| 6362 // Release the backing store associated with old stack | |
| 6363 _virtual_space.release(); | |
| 6364 // Reinitialize virtual space for new stack | |
| 6365 if (!_virtual_space.initialize(rs, rs.size())) { | |
| 6366 fatal("Not enough swap for expanded marking stack"); | |
| 6367 } | |
| 6368 _base = (oop*)(_virtual_space.low()); | |
| 6369 _index = 0; | |
| 6370 _capacity = new_capacity; | |
| 6371 } else if (_failed_double++ == 0 && !CMSConcurrentMTEnabled && PrintGCDetails) { | |
| 6372 // Failed to double capacity, continue; | |
| 6373 // we print a detail message only once per CMS cycle. | |
| 6374 gclog_or_tty->print(" (benign) Failed to expand marking stack from "SIZE_FORMAT"K to " | |
| 6375 SIZE_FORMAT"K", | |
| 6376 _capacity / K, new_capacity / K); | |
| 6377 } | |
| 6378 } | |
| 6379 | |
| 6380 | |
| 6381 // Closures | |
| 6382 // XXX: there seems to be a lot of code duplication here; | |
| 6383 // should refactor and consolidate common code. | |
| 6384 | |
| 6385 // This closure is used to mark refs into the CMS generation in | |
| 6386 // the CMS bit map. Called at the first checkpoint. This closure | |
| 6387 // assumes that we do not need to re-mark dirty cards; if the CMS | |
| 6388 // generation on which this is used is not an oldest (modulo perm gen) | |
| 6389 // generation then this will lose younger_gen cards! | |
| 6390 | |
| 6391 MarkRefsIntoClosure::MarkRefsIntoClosure( | |
| 6392 MemRegion span, CMSBitMap* bitMap, bool should_do_nmethods): | |
| 6393 _span(span), | |
| 6394 _bitMap(bitMap), | |
| 6395 _should_do_nmethods(should_do_nmethods) | |
| 6396 { | |
| 6397 assert(_ref_processor == NULL, "deliberately left NULL"); | |
| 6398 assert(_bitMap->covers(_span), "_bitMap/_span mismatch"); | |
| 6399 } | |
| 6400 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6401 void MarkRefsIntoClosure::do_oop(oop obj) { |
| 0 | 6402 // if p points into _span, then mark corresponding bit in _markBitMap |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6403 assert(obj->is_oop(), "expected an oop"); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6404 HeapWord* addr = (HeapWord*)obj; |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6405 if (_span.contains(addr)) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6406 // this should be made more efficient |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6407 _bitMap->mark(addr); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6408 } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6409 } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6410 |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6411 void MarkRefsIntoClosure::do_oop(oop* p) { MarkRefsIntoClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6412 void MarkRefsIntoClosure::do_oop(narrowOop* p) { MarkRefsIntoClosure::do_oop_work(p); } |
| 0 | 6413 |
| 6414 // A variant of the above, used for CMS marking verification. | |
| 6415 MarkRefsIntoVerifyClosure::MarkRefsIntoVerifyClosure( | |
| 6416 MemRegion span, CMSBitMap* verification_bm, CMSBitMap* cms_bm, | |
| 6417 bool should_do_nmethods): | |
| 6418 _span(span), | |
| 6419 _verification_bm(verification_bm), | |
| 6420 _cms_bm(cms_bm), | |
| 6421 _should_do_nmethods(should_do_nmethods) { | |
| 6422 assert(_ref_processor == NULL, "deliberately left NULL"); | |
| 6423 assert(_verification_bm->covers(_span), "_verification_bm/_span mismatch"); | |
| 6424 } | |
| 6425 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6426 void MarkRefsIntoVerifyClosure::do_oop(oop obj) { |
| 0 | 6427 // if p points into _span, then mark corresponding bit in _markBitMap |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6428 assert(obj->is_oop(), "expected an oop"); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6429 HeapWord* addr = (HeapWord*)obj; |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6430 if (_span.contains(addr)) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6431 _verification_bm->mark(addr); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6432 if (!_cms_bm->isMarked(addr)) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6433 oop(addr)->print(); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6434 gclog_or_tty->print_cr(" (" INTPTR_FORMAT " should have been marked)", addr); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6435 fatal("... aborting"); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6436 } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6437 } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6438 } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6439 |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6440 void MarkRefsIntoVerifyClosure::do_oop(oop* p) { MarkRefsIntoVerifyClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6441 void MarkRefsIntoVerifyClosure::do_oop(narrowOop* p) { MarkRefsIntoVerifyClosure::do_oop_work(p); } |
| 0 | 6442 |
| 6443 ////////////////////////////////////////////////// | |
| 6444 // MarkRefsIntoAndScanClosure | |
| 6445 ////////////////////////////////////////////////// | |
| 6446 | |
| 6447 MarkRefsIntoAndScanClosure::MarkRefsIntoAndScanClosure(MemRegion span, | |
| 6448 ReferenceProcessor* rp, | |
| 6449 CMSBitMap* bit_map, | |
| 6450 CMSBitMap* mod_union_table, | |
| 6451 CMSMarkStack* mark_stack, | |
| 6452 CMSMarkStack* revisit_stack, | |
| 6453 CMSCollector* collector, | |
| 6454 bool should_yield, | |
| 6455 bool concurrent_precleaning): | |
| 6456 _collector(collector), | |
| 6457 _span(span), | |
| 6458 _bit_map(bit_map), | |
| 6459 _mark_stack(mark_stack), | |
| 6460 _pushAndMarkClosure(collector, span, rp, bit_map, mod_union_table, | |
| 6461 mark_stack, revisit_stack, concurrent_precleaning), | |
| 6462 _yield(should_yield), | |
| 6463 _concurrent_precleaning(concurrent_precleaning), | |
| 6464 _freelistLock(NULL) | |
| 6465 { | |
| 6466 _ref_processor = rp; | |
| 6467 assert(_ref_processor != NULL, "_ref_processor shouldn't be NULL"); | |
| 6468 } | |
| 6469 | |
| 6470 // This closure is used to mark refs into the CMS generation at the | |
| 6471 // second (final) checkpoint, and to scan and transitively follow | |
| 6472 // the unmarked oops. It is also used during the concurrent precleaning | |
| 6473 // phase while scanning objects on dirty cards in the CMS generation. | |
| 6474 // The marks are made in the marking bit map and the marking stack is | |
| 6475 // used for keeping the (newly) grey objects during the scan. | |
| 6476 // The parallel version (Par_...) appears further below. | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6477 void MarkRefsIntoAndScanClosure::do_oop(oop obj) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6478 if (obj != NULL) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6479 assert(obj->is_oop(), "expected an oop"); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6480 HeapWord* addr = (HeapWord*)obj; |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6481 assert(_mark_stack->isEmpty(), "pre-condition (eager drainage)"); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6482 assert(_collector->overflow_list_is_empty(), |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6483 "overflow list should be empty"); |
| 0 | 6484 if (_span.contains(addr) && |
| 6485 !_bit_map->isMarked(addr)) { | |
| 6486 // mark bit map (object is now grey) | |
| 6487 _bit_map->mark(addr); | |
| 6488 // push on marking stack (stack should be empty), and drain the | |
| 6489 // stack by applying this closure to the oops in the oops popped | |
| 6490 // from the stack (i.e. blacken the grey objects) | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6491 bool res = _mark_stack->push(obj); |
| 0 | 6492 assert(res, "Should have space to push on empty stack"); |
| 6493 do { | |
| 6494 oop new_oop = _mark_stack->pop(); | |
| 6495 assert(new_oop != NULL && new_oop->is_oop(), "Expected an oop"); | |
| 6496 assert(new_oop->is_parsable(), "Found unparsable oop"); | |
| 6497 assert(_bit_map->isMarked((HeapWord*)new_oop), | |
| 6498 "only grey objects on this stack"); | |
| 6499 // iterate over the oops in this oop, marking and pushing | |
| 6500 // the ones in CMS heap (i.e. in _span). | |
| 6501 new_oop->oop_iterate(&_pushAndMarkClosure); | |
| 6502 // check if it's time to yield | |
| 6503 do_yield_check(); | |
| 6504 } while (!_mark_stack->isEmpty() || | |
| 6505 (!_concurrent_precleaning && take_from_overflow_list())); | |
| 6506 // if marking stack is empty, and we are not doing this | |
| 6507 // during precleaning, then check the overflow list | |
| 6508 } | |
| 6509 assert(_mark_stack->isEmpty(), "post-condition (eager drainage)"); | |
| 6510 assert(_collector->overflow_list_is_empty(), | |
| 6511 "overflow list was drained above"); | |
| 6512 // We could restore evacuated mark words, if any, used for | |
| 6513 // overflow list links here because the overflow list is | |
| 6514 // provably empty here. That would reduce the maximum | |
| 6515 // size requirements for preserved_{oop,mark}_stack. | |
| 6516 // But we'll just postpone it until we are all done | |
| 6517 // so we can just stream through. | |
| 6518 if (!_concurrent_precleaning && CMSOverflowEarlyRestoration) { | |
| 6519 _collector->restore_preserved_marks_if_any(); | |
| 6520 assert(_collector->no_preserved_marks(), "No preserved marks"); | |
| 6521 } | |
| 6522 assert(!CMSOverflowEarlyRestoration || _collector->no_preserved_marks(), | |
| 6523 "All preserved marks should have been restored above"); | |
| 6524 } | |
| 6525 } | |
| 6526 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6527 void MarkRefsIntoAndScanClosure::do_oop(oop* p) { MarkRefsIntoAndScanClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6528 void MarkRefsIntoAndScanClosure::do_oop(narrowOop* p) { MarkRefsIntoAndScanClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6529 |
| 0 | 6530 void MarkRefsIntoAndScanClosure::do_yield_work() { |
| 6531 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 6532 "CMS thread should hold CMS token"); | |
| 6533 assert_lock_strong(_freelistLock); | |
| 6534 assert_lock_strong(_bit_map->lock()); | |
| 6535 // relinquish the free_list_lock and bitMaplock() | |
| 6536 _bit_map->lock()->unlock(); | |
| 6537 _freelistLock->unlock(); | |
| 6538 ConcurrentMarkSweepThread::desynchronize(true); | |
| 6539 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 6540 _collector->stopTimer(); | |
| 6541 GCPauseTimer p(_collector->size_policy()->concurrent_timer_ptr()); | |
| 6542 if (PrintCMSStatistics != 0) { | |
| 6543 _collector->incrementYields(); | |
| 6544 } | |
| 6545 _collector->icms_wait(); | |
| 6546 | |
| 6547 // See the comment in coordinator_yield() | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6548 for (unsigned i = 0; |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6549 i < CMSYieldSleepCount && |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6550 ConcurrentMarkSweepThread::should_yield() && |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6551 !CMSCollector::foregroundGCIsActive(); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6552 ++i) { |
| 0 | 6553 os::sleep(Thread::current(), 1, false); |
| 6554 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 6555 } | |
| 6556 | |
| 6557 ConcurrentMarkSweepThread::synchronize(true); | |
| 6558 _freelistLock->lock_without_safepoint_check(); | |
| 6559 _bit_map->lock()->lock_without_safepoint_check(); | |
| 6560 _collector->startTimer(); | |
| 6561 } | |
| 6562 | |
| 6563 /////////////////////////////////////////////////////////// | |
| 6564 // Par_MarkRefsIntoAndScanClosure: a parallel version of | |
| 6565 // MarkRefsIntoAndScanClosure | |
| 6566 /////////////////////////////////////////////////////////// | |
| 6567 Par_MarkRefsIntoAndScanClosure::Par_MarkRefsIntoAndScanClosure( | |
| 6568 CMSCollector* collector, MemRegion span, ReferenceProcessor* rp, | |
| 6569 CMSBitMap* bit_map, OopTaskQueue* work_queue, CMSMarkStack* revisit_stack): | |
| 6570 _span(span), | |
| 6571 _bit_map(bit_map), | |
| 6572 _work_queue(work_queue), | |
| 6573 _low_water_mark(MIN2((uint)(work_queue->max_elems()/4), | |
| 6574 (uint)(CMSWorkQueueDrainThreshold * ParallelGCThreads))), | |
| 6575 _par_pushAndMarkClosure(collector, span, rp, bit_map, work_queue, | |
| 6576 revisit_stack) | |
| 6577 { | |
| 6578 _ref_processor = rp; | |
| 6579 assert(_ref_processor != NULL, "_ref_processor shouldn't be NULL"); | |
| 6580 } | |
| 6581 | |
| 6582 // This closure is used to mark refs into the CMS generation at the | |
| 6583 // second (final) checkpoint, and to scan and transitively follow | |
| 6584 // the unmarked oops. The marks are made in the marking bit map and | |
| 6585 // the work_queue is used for keeping the (newly) grey objects during | |
| 6586 // the scan phase whence they are also available for stealing by parallel | |
| 6587 // threads. Since the marking bit map is shared, updates are | |
| 6588 // synchronized (via CAS). | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6589 void Par_MarkRefsIntoAndScanClosure::do_oop(oop obj) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6590 if (obj != NULL) { |
| 0 | 6591 // Ignore mark word because this could be an already marked oop |
| 6592 // that may be chained at the end of the overflow list. | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
6593 assert(obj->is_oop(true), "expected an oop"); |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6594 HeapWord* addr = (HeapWord*)obj; |
| 0 | 6595 if (_span.contains(addr) && |
| 6596 !_bit_map->isMarked(addr)) { | |
| 6597 // mark bit map (object will become grey): | |
| 6598 // It is possible for several threads to be | |
| 6599 // trying to "claim" this object concurrently; | |
| 6600 // the unique thread that succeeds in marking the | |
| 6601 // object first will do the subsequent push on | |
| 6602 // to the work queue (or overflow list). | |
| 6603 if (_bit_map->par_mark(addr)) { | |
| 6604 // push on work_queue (which may not be empty), and trim the | |
| 6605 // queue to an appropriate length by applying this closure to | |
| 6606 // the oops in the oops popped from the stack (i.e. blacken the | |
| 6607 // grey objects) | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6608 bool res = _work_queue->push(obj); |
| 0 | 6609 assert(res, "Low water mark should be less than capacity?"); |
| 6610 trim_queue(_low_water_mark); | |
| 6611 } // Else, another thread claimed the object | |
| 6612 } | |
| 6613 } | |
| 6614 } | |
| 6615 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6616 void Par_MarkRefsIntoAndScanClosure::do_oop(oop* p) { Par_MarkRefsIntoAndScanClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6617 void Par_MarkRefsIntoAndScanClosure::do_oop(narrowOop* p) { Par_MarkRefsIntoAndScanClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6618 |
| 0 | 6619 // This closure is used to rescan the marked objects on the dirty cards |
| 6620 // in the mod union table and the card table proper. | |
| 6621 size_t ScanMarkedObjectsAgainCarefullyClosure::do_object_careful_m( | |
| 6622 oop p, MemRegion mr) { | |
| 6623 | |
| 6624 size_t size = 0; | |
| 6625 HeapWord* addr = (HeapWord*)p; | |
| 6626 DEBUG_ONLY(_collector->verify_work_stacks_empty();) | |
| 6627 assert(_span.contains(addr), "we are scanning the CMS generation"); | |
| 6628 // check if it's time to yield | |
| 6629 if (do_yield_check()) { | |
| 6630 // We yielded for some foreground stop-world work, | |
| 6631 // and we have been asked to abort this ongoing preclean cycle. | |
| 6632 return 0; | |
| 6633 } | |
| 6634 if (_bitMap->isMarked(addr)) { | |
| 6635 // it's marked; is it potentially uninitialized? | |
| 187 | 6636 if (p->klass_or_null() != NULL) { |
| 0 | 6637 if (CMSPermGenPrecleaningEnabled && !p->is_parsable()) { |
| 6638 // Signal precleaning to redirty the card since | |
| 6639 // the klass pointer is already installed. | |
| 6640 assert(size == 0, "Initial value"); | |
| 6641 } else { | |
| 6642 assert(p->is_parsable(), "must be parsable."); | |
| 6643 // an initialized object; ignore mark word in verification below | |
| 6644 // since we are running concurrent with mutators | |
| 6645 assert(p->is_oop(true), "should be an oop"); | |
| 6646 if (p->is_objArray()) { | |
| 6647 // objArrays are precisely marked; restrict scanning | |
| 6648 // to dirty cards only. | |
| 187 | 6649 size = CompactibleFreeListSpace::adjustObjectSize( |
| 6650 p->oop_iterate(_scanningClosure, mr)); | |
| 0 | 6651 } else { |
| 6652 // A non-array may have been imprecisely marked; we need | |
| 6653 // to scan object in its entirety. | |
| 6654 size = CompactibleFreeListSpace::adjustObjectSize( | |
| 6655 p->oop_iterate(_scanningClosure)); | |
| 6656 } | |
| 6657 #ifdef DEBUG | |
| 6658 size_t direct_size = | |
| 6659 CompactibleFreeListSpace::adjustObjectSize(p->size()); | |
| 6660 assert(size == direct_size, "Inconsistency in size"); | |
| 6661 assert(size >= 3, "Necessary for Printezis marks to work"); | |
| 6662 if (!_bitMap->isMarked(addr+1)) { | |
| 6663 _bitMap->verifyNoOneBitsInRange(addr+2, addr+size); | |
| 6664 } else { | |
| 6665 _bitMap->verifyNoOneBitsInRange(addr+2, addr+size-1); | |
| 6666 assert(_bitMap->isMarked(addr+size-1), | |
| 6667 "inconsistent Printezis mark"); | |
| 6668 } | |
| 6669 #endif // DEBUG | |
| 6670 } | |
| 6671 } else { | |
| 6672 // an unitialized object | |
| 6673 assert(_bitMap->isMarked(addr+1), "missing Printezis mark?"); | |
| 6674 HeapWord* nextOneAddr = _bitMap->getNextMarkedWordAddress(addr + 2); | |
| 6675 size = pointer_delta(nextOneAddr + 1, addr); | |
| 6676 assert(size == CompactibleFreeListSpace::adjustObjectSize(size), | |
| 6677 "alignment problem"); | |
| 6678 // Note that pre-cleaning needn't redirty the card. OopDesc::set_klass() | |
| 6679 // will dirty the card when the klass pointer is installed in the | |
| 6680 // object (signalling the completion of initialization). | |
| 6681 } | |
| 6682 } else { | |
| 6683 // Either a not yet marked object or an uninitialized object | |
| 187 | 6684 if (p->klass_or_null() == NULL || !p->is_parsable()) { |
| 0 | 6685 // An uninitialized object, skip to the next card, since |
| 6686 // we may not be able to read its P-bits yet. | |
| 6687 assert(size == 0, "Initial value"); | |
| 6688 } else { | |
| 6689 // An object not (yet) reached by marking: we merely need to | |
| 6690 // compute its size so as to go look at the next block. | |
| 6691 assert(p->is_oop(true), "should be an oop"); | |
| 6692 size = CompactibleFreeListSpace::adjustObjectSize(p->size()); | |
| 6693 } | |
| 6694 } | |
| 6695 DEBUG_ONLY(_collector->verify_work_stacks_empty();) | |
| 6696 return size; | |
| 6697 } | |
| 6698 | |
| 6699 void ScanMarkedObjectsAgainCarefullyClosure::do_yield_work() { | |
| 6700 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 6701 "CMS thread should hold CMS token"); | |
| 6702 assert_lock_strong(_freelistLock); | |
| 6703 assert_lock_strong(_bitMap->lock()); | |
| 6704 // relinquish the free_list_lock and bitMaplock() | |
| 6705 _bitMap->lock()->unlock(); | |
| 6706 _freelistLock->unlock(); | |
| 6707 ConcurrentMarkSweepThread::desynchronize(true); | |
| 6708 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 6709 _collector->stopTimer(); | |
| 6710 GCPauseTimer p(_collector->size_policy()->concurrent_timer_ptr()); | |
| 6711 if (PrintCMSStatistics != 0) { | |
| 6712 _collector->incrementYields(); | |
| 6713 } | |
| 6714 _collector->icms_wait(); | |
| 6715 | |
| 6716 // See the comment in coordinator_yield() | |
| 6717 for (unsigned i = 0; i < CMSYieldSleepCount && | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6718 ConcurrentMarkSweepThread::should_yield() && |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6719 !CMSCollector::foregroundGCIsActive(); ++i) { |
| 0 | 6720 os::sleep(Thread::current(), 1, false); |
| 6721 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 6722 } | |
| 6723 | |
| 6724 ConcurrentMarkSweepThread::synchronize(true); | |
| 6725 _freelistLock->lock_without_safepoint_check(); | |
| 6726 _bitMap->lock()->lock_without_safepoint_check(); | |
| 6727 _collector->startTimer(); | |
| 6728 } | |
| 6729 | |
| 6730 | |
| 6731 ////////////////////////////////////////////////////////////////// | |
| 6732 // SurvivorSpacePrecleanClosure | |
| 6733 ////////////////////////////////////////////////////////////////// | |
| 6734 // This (single-threaded) closure is used to preclean the oops in | |
| 6735 // the survivor spaces. | |
| 6736 size_t SurvivorSpacePrecleanClosure::do_object_careful(oop p) { | |
| 6737 | |
| 6738 HeapWord* addr = (HeapWord*)p; | |
| 6739 DEBUG_ONLY(_collector->verify_work_stacks_empty();) | |
| 6740 assert(!_span.contains(addr), "we are scanning the survivor spaces"); | |
| 187 | 6741 assert(p->klass_or_null() != NULL, "object should be initializd"); |
| 0 | 6742 assert(p->is_parsable(), "must be parsable."); |
| 6743 // an initialized object; ignore mark word in verification below | |
| 6744 // since we are running concurrent with mutators | |
| 6745 assert(p->is_oop(true), "should be an oop"); | |
| 6746 // Note that we do not yield while we iterate over | |
| 6747 // the interior oops of p, pushing the relevant ones | |
| 6748 // on our marking stack. | |
| 6749 size_t size = p->oop_iterate(_scanning_closure); | |
| 6750 do_yield_check(); | |
| 6751 // Observe that below, we do not abandon the preclean | |
| 6752 // phase as soon as we should; rather we empty the | |
| 6753 // marking stack before returning. This is to satisfy | |
| 6754 // some existing assertions. In general, it may be a | |
| 6755 // good idea to abort immediately and complete the marking | |
| 6756 // from the grey objects at a later time. | |
| 6757 while (!_mark_stack->isEmpty()) { | |
| 6758 oop new_oop = _mark_stack->pop(); | |
| 6759 assert(new_oop != NULL && new_oop->is_oop(), "Expected an oop"); | |
| 6760 assert(new_oop->is_parsable(), "Found unparsable oop"); | |
| 6761 assert(_bit_map->isMarked((HeapWord*)new_oop), | |
| 6762 "only grey objects on this stack"); | |
| 6763 // iterate over the oops in this oop, marking and pushing | |
| 6764 // the ones in CMS heap (i.e. in _span). | |
| 6765 new_oop->oop_iterate(_scanning_closure); | |
| 6766 // check if it's time to yield | |
| 6767 do_yield_check(); | |
| 6768 } | |
| 6769 unsigned int after_count = | |
| 6770 GenCollectedHeap::heap()->total_collections(); | |
| 6771 bool abort = (_before_count != after_count) || | |
| 6772 _collector->should_abort_preclean(); | |
| 6773 return abort ? 0 : size; | |
| 6774 } | |
| 6775 | |
| 6776 void SurvivorSpacePrecleanClosure::do_yield_work() { | |
| 6777 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 6778 "CMS thread should hold CMS token"); | |
| 6779 assert_lock_strong(_bit_map->lock()); | |
| 6780 // Relinquish the bit map lock | |
| 6781 _bit_map->lock()->unlock(); | |
| 6782 ConcurrentMarkSweepThread::desynchronize(true); | |
| 6783 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 6784 _collector->stopTimer(); | |
| 6785 GCPauseTimer p(_collector->size_policy()->concurrent_timer_ptr()); | |
| 6786 if (PrintCMSStatistics != 0) { | |
| 6787 _collector->incrementYields(); | |
| 6788 } | |
| 6789 _collector->icms_wait(); | |
| 6790 | |
| 6791 // See the comment in coordinator_yield() | |
| 6792 for (unsigned i = 0; i < CMSYieldSleepCount && | |
| 6793 ConcurrentMarkSweepThread::should_yield() && | |
| 6794 !CMSCollector::foregroundGCIsActive(); ++i) { | |
| 6795 os::sleep(Thread::current(), 1, false); | |
| 6796 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 6797 } | |
| 6798 | |
| 6799 ConcurrentMarkSweepThread::synchronize(true); | |
| 6800 _bit_map->lock()->lock_without_safepoint_check(); | |
| 6801 _collector->startTimer(); | |
| 6802 } | |
| 6803 | |
| 6804 // This closure is used to rescan the marked objects on the dirty cards | |
| 6805 // in the mod union table and the card table proper. In the parallel | |
| 6806 // case, although the bitMap is shared, we do a single read so the | |
| 6807 // isMarked() query is "safe". | |
| 6808 bool ScanMarkedObjectsAgainClosure::do_object_bm(oop p, MemRegion mr) { | |
| 6809 // Ignore mark word because we are running concurrent with mutators | |
| 6810 assert(p->is_oop_or_null(true), "expected an oop or null"); | |
| 6811 HeapWord* addr = (HeapWord*)p; | |
| 6812 assert(_span.contains(addr), "we are scanning the CMS generation"); | |
| 6813 bool is_obj_array = false; | |
| 6814 #ifdef DEBUG | |
| 6815 if (!_parallel) { | |
| 6816 assert(_mark_stack->isEmpty(), "pre-condition (eager drainage)"); | |
| 6817 assert(_collector->overflow_list_is_empty(), | |
| 6818 "overflow list should be empty"); | |
| 6819 | |
| 6820 } | |
| 6821 #endif // DEBUG | |
| 6822 if (_bit_map->isMarked(addr)) { | |
| 6823 // Obj arrays are precisely marked, non-arrays are not; | |
| 6824 // so we scan objArrays precisely and non-arrays in their | |
| 6825 // entirety. | |
| 6826 if (p->is_objArray()) { | |
| 6827 is_obj_array = true; | |
| 6828 if (_parallel) { | |
| 6829 p->oop_iterate(_par_scan_closure, mr); | |
| 6830 } else { | |
| 6831 p->oop_iterate(_scan_closure, mr); | |
| 6832 } | |
| 6833 } else { | |
| 6834 if (_parallel) { | |
| 6835 p->oop_iterate(_par_scan_closure); | |
| 6836 } else { | |
| 6837 p->oop_iterate(_scan_closure); | |
| 6838 } | |
| 6839 } | |
| 6840 } | |
| 6841 #ifdef DEBUG | |
| 6842 if (!_parallel) { | |
| 6843 assert(_mark_stack->isEmpty(), "post-condition (eager drainage)"); | |
| 6844 assert(_collector->overflow_list_is_empty(), | |
| 6845 "overflow list should be empty"); | |
| 6846 | |
| 6847 } | |
| 6848 #endif // DEBUG | |
| 6849 return is_obj_array; | |
| 6850 } | |
| 6851 | |
| 6852 MarkFromRootsClosure::MarkFromRootsClosure(CMSCollector* collector, | |
| 6853 MemRegion span, | |
| 6854 CMSBitMap* bitMap, CMSMarkStack* markStack, | |
| 6855 CMSMarkStack* revisitStack, | |
| 6856 bool should_yield, bool verifying): | |
| 6857 _collector(collector), | |
| 6858 _span(span), | |
| 6859 _bitMap(bitMap), | |
| 6860 _mut(&collector->_modUnionTable), | |
| 6861 _markStack(markStack), | |
| 6862 _revisitStack(revisitStack), | |
| 6863 _yield(should_yield), | |
| 6864 _skipBits(0) | |
| 6865 { | |
| 6866 assert(_markStack->isEmpty(), "stack should be empty"); | |
| 6867 _finger = _bitMap->startWord(); | |
| 6868 _threshold = _finger; | |
| 6869 assert(_collector->_restart_addr == NULL, "Sanity check"); | |
| 6870 assert(_span.contains(_finger), "Out of bounds _finger?"); | |
| 6871 DEBUG_ONLY(_verifying = verifying;) | |
| 6872 } | |
| 6873 | |
| 6874 void MarkFromRootsClosure::reset(HeapWord* addr) { | |
| 6875 assert(_markStack->isEmpty(), "would cause duplicates on stack"); | |
| 6876 assert(_span.contains(addr), "Out of bounds _finger?"); | |
| 6877 _finger = addr; | |
| 6878 _threshold = (HeapWord*)round_to( | |
| 6879 (intptr_t)_finger, CardTableModRefBS::card_size); | |
| 6880 } | |
| 6881 | |
| 6882 // Should revisit to see if this should be restructured for | |
| 6883 // greater efficiency. | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
6884 bool MarkFromRootsClosure::do_bit(size_t offset) { |
| 0 | 6885 if (_skipBits > 0) { |
| 6886 _skipBits--; | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
6887 return true; |
| 0 | 6888 } |
| 6889 // convert offset into a HeapWord* | |
| 6890 HeapWord* addr = _bitMap->startWord() + offset; | |
| 6891 assert(_bitMap->endWord() && addr < _bitMap->endWord(), | |
| 6892 "address out of range"); | |
| 6893 assert(_bitMap->isMarked(addr), "tautology"); | |
| 6894 if (_bitMap->isMarked(addr+1)) { | |
| 6895 // this is an allocated but not yet initialized object | |
| 6896 assert(_skipBits == 0, "tautology"); | |
| 6897 _skipBits = 2; // skip next two marked bits ("Printezis-marks") | |
| 6898 oop p = oop(addr); | |
| 187 | 6899 if (p->klass_or_null() == NULL || !p->is_parsable()) { |
| 0 | 6900 DEBUG_ONLY(if (!_verifying) {) |
| 6901 // We re-dirty the cards on which this object lies and increase | |
| 6902 // the _threshold so that we'll come back to scan this object | |
| 6903 // during the preclean or remark phase. (CMSCleanOnEnter) | |
| 6904 if (CMSCleanOnEnter) { | |
| 6905 size_t sz = _collector->block_size_using_printezis_bits(addr); | |
| 6906 HeapWord* end_card_addr = (HeapWord*)round_to( | |
| 6907 (intptr_t)(addr+sz), CardTableModRefBS::card_size); | |
|
283
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
6908 MemRegion redirty_range = MemRegion(addr, end_card_addr); |
| 0 | 6909 assert(!redirty_range.is_empty(), "Arithmetical tautology"); |
| 6910 // Bump _threshold to end_card_addr; note that | |
| 6911 // _threshold cannot possibly exceed end_card_addr, anyhow. | |
| 6912 // This prevents future clearing of the card as the scan proceeds | |
| 6913 // to the right. | |
| 6914 assert(_threshold <= end_card_addr, | |
| 6915 "Because we are just scanning into this object"); | |
| 6916 if (_threshold < end_card_addr) { | |
| 6917 _threshold = end_card_addr; | |
| 6918 } | |
| 187 | 6919 if (p->klass_or_null() != NULL) { |
| 0 | 6920 // Redirty the range of cards... |
| 6921 _mut->mark_range(redirty_range); | |
| 6922 } // ...else the setting of klass will dirty the card anyway. | |
| 6923 } | |
| 6924 DEBUG_ONLY(}) | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
6925 return true; |
| 0 | 6926 } |
| 6927 } | |
| 6928 scanOopsInOop(addr); | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
6929 return true; |
| 0 | 6930 } |
| 6931 | |
| 6932 // We take a break if we've been at this for a while, | |
| 6933 // so as to avoid monopolizing the locks involved. | |
| 6934 void MarkFromRootsClosure::do_yield_work() { | |
| 6935 // First give up the locks, then yield, then re-lock | |
| 6936 // We should probably use a constructor/destructor idiom to | |
| 6937 // do this unlock/lock or modify the MutexUnlocker class to | |
| 6938 // serve our purpose. XXX | |
| 6939 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 6940 "CMS thread should hold CMS token"); | |
| 6941 assert_lock_strong(_bitMap->lock()); | |
| 6942 _bitMap->lock()->unlock(); | |
| 6943 ConcurrentMarkSweepThread::desynchronize(true); | |
| 6944 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 6945 _collector->stopTimer(); | |
| 6946 GCPauseTimer p(_collector->size_policy()->concurrent_timer_ptr()); | |
| 6947 if (PrintCMSStatistics != 0) { | |
| 6948 _collector->incrementYields(); | |
| 6949 } | |
| 6950 _collector->icms_wait(); | |
| 6951 | |
| 6952 // See the comment in coordinator_yield() | |
| 6953 for (unsigned i = 0; i < CMSYieldSleepCount && | |
| 6954 ConcurrentMarkSweepThread::should_yield() && | |
| 6955 !CMSCollector::foregroundGCIsActive(); ++i) { | |
| 6956 os::sleep(Thread::current(), 1, false); | |
| 6957 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 6958 } | |
| 6959 | |
| 6960 ConcurrentMarkSweepThread::synchronize(true); | |
| 6961 _bitMap->lock()->lock_without_safepoint_check(); | |
| 6962 _collector->startTimer(); | |
| 6963 } | |
| 6964 | |
| 6965 void MarkFromRootsClosure::scanOopsInOop(HeapWord* ptr) { | |
| 6966 assert(_bitMap->isMarked(ptr), "expected bit to be set"); | |
| 6967 assert(_markStack->isEmpty(), | |
| 6968 "should drain stack to limit stack usage"); | |
| 6969 // convert ptr to an oop preparatory to scanning | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6970 oop obj = oop(ptr); |
| 0 | 6971 // Ignore mark word in verification below, since we |
| 6972 // may be running concurrent with mutators. | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6973 assert(obj->is_oop(true), "should be an oop"); |
| 0 | 6974 assert(_finger <= ptr, "_finger runneth ahead"); |
| 6975 // advance the finger to right end of this object | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
6976 _finger = ptr + obj->size(); |
| 0 | 6977 assert(_finger > ptr, "we just incremented it above"); |
| 6978 // On large heaps, it may take us some time to get through | |
| 6979 // the marking phase (especially if running iCMS). During | |
| 6980 // this time it's possible that a lot of mutations have | |
| 6981 // accumulated in the card table and the mod union table -- | |
| 6982 // these mutation records are redundant until we have | |
| 6983 // actually traced into the corresponding card. | |
| 6984 // Here, we check whether advancing the finger would make | |
| 6985 // us cross into a new card, and if so clear corresponding | |
| 6986 // cards in the MUT (preclean them in the card-table in the | |
| 6987 // future). | |
| 6988 | |
| 6989 DEBUG_ONLY(if (!_verifying) {) | |
| 6990 // The clean-on-enter optimization is disabled by default, | |
| 6991 // until we fix 6178663. | |
| 6992 if (CMSCleanOnEnter && (_finger > _threshold)) { | |
| 6993 // [_threshold, _finger) represents the interval | |
| 6994 // of cards to be cleared in MUT (or precleaned in card table). | |
| 6995 // The set of cards to be cleared is all those that overlap | |
| 6996 // with the interval [_threshold, _finger); note that | |
| 6997 // _threshold is always kept card-aligned but _finger isn't | |
| 6998 // always card-aligned. | |
| 6999 HeapWord* old_threshold = _threshold; | |
| 7000 assert(old_threshold == (HeapWord*)round_to( | |
| 7001 (intptr_t)old_threshold, CardTableModRefBS::card_size), | |
| 7002 "_threshold should always be card-aligned"); | |
| 7003 _threshold = (HeapWord*)round_to( | |
| 7004 (intptr_t)_finger, CardTableModRefBS::card_size); | |
| 7005 MemRegion mr(old_threshold, _threshold); | |
| 7006 assert(!mr.is_empty(), "Control point invariant"); | |
| 7007 assert(_span.contains(mr), "Should clear within span"); | |
| 7008 // XXX When _finger crosses from old gen into perm gen | |
| 7009 // we may be doing unnecessary cleaning; do better in the | |
| 7010 // future by detecting that condition and clearing fewer | |
| 7011 // MUT/CT entries. | |
| 7012 _mut->clear_range(mr); | |
| 7013 } | |
| 7014 DEBUG_ONLY(}) | |
| 7015 | |
| 7016 // Note: the finger doesn't advance while we drain | |
| 7017 // the stack below. | |
| 7018 PushOrMarkClosure pushOrMarkClosure(_collector, | |
| 7019 _span, _bitMap, _markStack, | |
| 7020 _revisitStack, | |
| 7021 _finger, this); | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7022 bool res = _markStack->push(obj); |
| 0 | 7023 assert(res, "Empty non-zero size stack should have space for single push"); |
| 7024 while (!_markStack->isEmpty()) { | |
| 7025 oop new_oop = _markStack->pop(); | |
| 7026 // Skip verifying header mark word below because we are | |
| 7027 // running concurrent with mutators. | |
| 7028 assert(new_oop->is_oop(true), "Oops! expected to pop an oop"); | |
| 7029 // now scan this oop's oops | |
| 7030 new_oop->oop_iterate(&pushOrMarkClosure); | |
| 7031 do_yield_check(); | |
| 7032 } | |
| 7033 assert(_markStack->isEmpty(), "tautology, emphasizing post-condition"); | |
| 7034 } | |
| 7035 | |
| 7036 Par_MarkFromRootsClosure::Par_MarkFromRootsClosure(CMSConcMarkingTask* task, | |
| 7037 CMSCollector* collector, MemRegion span, | |
| 7038 CMSBitMap* bit_map, | |
| 7039 OopTaskQueue* work_queue, | |
| 7040 CMSMarkStack* overflow_stack, | |
| 7041 CMSMarkStack* revisit_stack, | |
| 7042 bool should_yield): | |
| 7043 _collector(collector), | |
| 7044 _whole_span(collector->_span), | |
| 7045 _span(span), | |
| 7046 _bit_map(bit_map), | |
| 7047 _mut(&collector->_modUnionTable), | |
| 7048 _work_queue(work_queue), | |
| 7049 _overflow_stack(overflow_stack), | |
| 7050 _revisit_stack(revisit_stack), | |
| 7051 _yield(should_yield), | |
| 7052 _skip_bits(0), | |
| 7053 _task(task) | |
| 7054 { | |
| 7055 assert(_work_queue->size() == 0, "work_queue should be empty"); | |
| 7056 _finger = span.start(); | |
| 7057 _threshold = _finger; // XXX Defer clear-on-enter optimization for now | |
| 7058 assert(_span.contains(_finger), "Out of bounds _finger?"); | |
| 7059 } | |
| 7060 | |
| 7061 // Should revisit to see if this should be restructured for | |
| 7062 // greater efficiency. | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7063 bool Par_MarkFromRootsClosure::do_bit(size_t offset) { |
| 0 | 7064 if (_skip_bits > 0) { |
| 7065 _skip_bits--; | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7066 return true; |
| 0 | 7067 } |
| 7068 // convert offset into a HeapWord* | |
| 7069 HeapWord* addr = _bit_map->startWord() + offset; | |
| 7070 assert(_bit_map->endWord() && addr < _bit_map->endWord(), | |
| 7071 "address out of range"); | |
| 7072 assert(_bit_map->isMarked(addr), "tautology"); | |
| 7073 if (_bit_map->isMarked(addr+1)) { | |
| 7074 // this is an allocated object that might not yet be initialized | |
| 7075 assert(_skip_bits == 0, "tautology"); | |
| 7076 _skip_bits = 2; // skip next two marked bits ("Printezis-marks") | |
| 7077 oop p = oop(addr); | |
| 187 | 7078 if (p->klass_or_null() == NULL || !p->is_parsable()) { |
| 0 | 7079 // in the case of Clean-on-Enter optimization, redirty card |
| 7080 // and avoid clearing card by increasing the threshold. | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7081 return true; |
| 0 | 7082 } |
| 7083 } | |
| 7084 scan_oops_in_oop(addr); | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7085 return true; |
| 0 | 7086 } |
| 7087 | |
| 7088 void Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord* ptr) { | |
| 7089 assert(_bit_map->isMarked(ptr), "expected bit to be set"); | |
| 7090 // Should we assert that our work queue is empty or | |
| 7091 // below some drain limit? | |
| 7092 assert(_work_queue->size() == 0, | |
| 7093 "should drain stack to limit stack usage"); | |
| 7094 // convert ptr to an oop preparatory to scanning | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7095 oop obj = oop(ptr); |
| 0 | 7096 // Ignore mark word in verification below, since we |
| 7097 // may be running concurrent with mutators. | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7098 assert(obj->is_oop(true), "should be an oop"); |
| 0 | 7099 assert(_finger <= ptr, "_finger runneth ahead"); |
| 7100 // advance the finger to right end of this object | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7101 _finger = ptr + obj->size(); |
| 0 | 7102 assert(_finger > ptr, "we just incremented it above"); |
| 7103 // On large heaps, it may take us some time to get through | |
| 7104 // the marking phase (especially if running iCMS). During | |
| 7105 // this time it's possible that a lot of mutations have | |
| 7106 // accumulated in the card table and the mod union table -- | |
| 7107 // these mutation records are redundant until we have | |
| 7108 // actually traced into the corresponding card. | |
| 7109 // Here, we check whether advancing the finger would make | |
| 7110 // us cross into a new card, and if so clear corresponding | |
| 7111 // cards in the MUT (preclean them in the card-table in the | |
| 7112 // future). | |
| 7113 | |
| 7114 // The clean-on-enter optimization is disabled by default, | |
| 7115 // until we fix 6178663. | |
| 7116 if (CMSCleanOnEnter && (_finger > _threshold)) { | |
| 7117 // [_threshold, _finger) represents the interval | |
| 7118 // of cards to be cleared in MUT (or precleaned in card table). | |
| 7119 // The set of cards to be cleared is all those that overlap | |
| 7120 // with the interval [_threshold, _finger); note that | |
| 7121 // _threshold is always kept card-aligned but _finger isn't | |
| 7122 // always card-aligned. | |
| 7123 HeapWord* old_threshold = _threshold; | |
| 7124 assert(old_threshold == (HeapWord*)round_to( | |
| 7125 (intptr_t)old_threshold, CardTableModRefBS::card_size), | |
| 7126 "_threshold should always be card-aligned"); | |
| 7127 _threshold = (HeapWord*)round_to( | |
| 7128 (intptr_t)_finger, CardTableModRefBS::card_size); | |
| 7129 MemRegion mr(old_threshold, _threshold); | |
| 7130 assert(!mr.is_empty(), "Control point invariant"); | |
| 7131 assert(_span.contains(mr), "Should clear within span"); // _whole_span ?? | |
| 7132 // XXX When _finger crosses from old gen into perm gen | |
| 7133 // we may be doing unnecessary cleaning; do better in the | |
| 7134 // future by detecting that condition and clearing fewer | |
| 7135 // MUT/CT entries. | |
| 7136 _mut->clear_range(mr); | |
| 7137 } | |
| 7138 | |
| 7139 // Note: the local finger doesn't advance while we drain | |
| 7140 // the stack below, but the global finger sure can and will. | |
| 7141 HeapWord** gfa = _task->global_finger_addr(); | |
| 7142 Par_PushOrMarkClosure pushOrMarkClosure(_collector, | |
| 7143 _span, _bit_map, | |
| 7144 _work_queue, | |
| 7145 _overflow_stack, | |
| 7146 _revisit_stack, | |
| 7147 _finger, | |
| 7148 gfa, this); | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7149 bool res = _work_queue->push(obj); // overflow could occur here |
| 0 | 7150 assert(res, "Will hold once we use workqueues"); |
| 7151 while (true) { | |
| 7152 oop new_oop; | |
| 7153 if (!_work_queue->pop_local(new_oop)) { | |
| 7154 // We emptied our work_queue; check if there's stuff that can | |
| 7155 // be gotten from the overflow stack. | |
| 7156 if (CMSConcMarkingTask::get_work_from_overflow_stack( | |
| 7157 _overflow_stack, _work_queue)) { | |
| 7158 do_yield_check(); | |
| 7159 continue; | |
| 7160 } else { // done | |
| 7161 break; | |
| 7162 } | |
| 7163 } | |
| 7164 // Skip verifying header mark word below because we are | |
| 7165 // running concurrent with mutators. | |
| 7166 assert(new_oop->is_oop(true), "Oops! expected to pop an oop"); | |
| 7167 // now scan this oop's oops | |
| 7168 new_oop->oop_iterate(&pushOrMarkClosure); | |
| 7169 do_yield_check(); | |
| 7170 } | |
| 7171 assert(_work_queue->size() == 0, "tautology, emphasizing post-condition"); | |
| 7172 } | |
| 7173 | |
| 7174 // Yield in response to a request from VM Thread or | |
| 7175 // from mutators. | |
| 7176 void Par_MarkFromRootsClosure::do_yield_work() { | |
| 7177 assert(_task != NULL, "sanity"); | |
| 7178 _task->yield(); | |
| 7179 } | |
| 7180 | |
| 7181 // A variant of the above used for verifying CMS marking work. | |
| 7182 MarkFromRootsVerifyClosure::MarkFromRootsVerifyClosure(CMSCollector* collector, | |
| 7183 MemRegion span, | |
| 7184 CMSBitMap* verification_bm, CMSBitMap* cms_bm, | |
| 7185 CMSMarkStack* mark_stack): | |
| 7186 _collector(collector), | |
| 7187 _span(span), | |
| 7188 _verification_bm(verification_bm), | |
| 7189 _cms_bm(cms_bm), | |
| 7190 _mark_stack(mark_stack), | |
| 7191 _pam_verify_closure(collector, span, verification_bm, cms_bm, | |
| 7192 mark_stack) | |
| 7193 { | |
| 7194 assert(_mark_stack->isEmpty(), "stack should be empty"); | |
| 7195 _finger = _verification_bm->startWord(); | |
| 7196 assert(_collector->_restart_addr == NULL, "Sanity check"); | |
| 7197 assert(_span.contains(_finger), "Out of bounds _finger?"); | |
| 7198 } | |
| 7199 | |
| 7200 void MarkFromRootsVerifyClosure::reset(HeapWord* addr) { | |
| 7201 assert(_mark_stack->isEmpty(), "would cause duplicates on stack"); | |
| 7202 assert(_span.contains(addr), "Out of bounds _finger?"); | |
| 7203 _finger = addr; | |
| 7204 } | |
| 7205 | |
| 7206 // Should revisit to see if this should be restructured for | |
| 7207 // greater efficiency. | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7208 bool MarkFromRootsVerifyClosure::do_bit(size_t offset) { |
| 0 | 7209 // convert offset into a HeapWord* |
| 7210 HeapWord* addr = _verification_bm->startWord() + offset; | |
| 7211 assert(_verification_bm->endWord() && addr < _verification_bm->endWord(), | |
| 7212 "address out of range"); | |
| 7213 assert(_verification_bm->isMarked(addr), "tautology"); | |
| 7214 assert(_cms_bm->isMarked(addr), "tautology"); | |
| 7215 | |
| 7216 assert(_mark_stack->isEmpty(), | |
| 7217 "should drain stack to limit stack usage"); | |
| 7218 // convert addr to an oop preparatory to scanning | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7219 oop obj = oop(addr); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7220 assert(obj->is_oop(), "should be an oop"); |
| 0 | 7221 assert(_finger <= addr, "_finger runneth ahead"); |
| 7222 // advance the finger to right end of this object | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7223 _finger = addr + obj->size(); |
| 0 | 7224 assert(_finger > addr, "we just incremented it above"); |
| 7225 // Note: the finger doesn't advance while we drain | |
| 7226 // the stack below. | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7227 bool res = _mark_stack->push(obj); |
| 0 | 7228 assert(res, "Empty non-zero size stack should have space for single push"); |
| 7229 while (!_mark_stack->isEmpty()) { | |
| 7230 oop new_oop = _mark_stack->pop(); | |
| 7231 assert(new_oop->is_oop(), "Oops! expected to pop an oop"); | |
| 7232 // now scan this oop's oops | |
| 7233 new_oop->oop_iterate(&_pam_verify_closure); | |
| 7234 } | |
| 7235 assert(_mark_stack->isEmpty(), "tautology, emphasizing post-condition"); | |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7236 return true; |
| 0 | 7237 } |
| 7238 | |
| 7239 PushAndMarkVerifyClosure::PushAndMarkVerifyClosure( | |
| 7240 CMSCollector* collector, MemRegion span, | |
| 7241 CMSBitMap* verification_bm, CMSBitMap* cms_bm, | |
| 7242 CMSMarkStack* mark_stack): | |
| 7243 OopClosure(collector->ref_processor()), | |
| 7244 _collector(collector), | |
| 7245 _span(span), | |
| 7246 _verification_bm(verification_bm), | |
| 7247 _cms_bm(cms_bm), | |
| 7248 _mark_stack(mark_stack) | |
| 7249 { } | |
| 7250 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7251 void PushAndMarkVerifyClosure::do_oop(oop* p) { PushAndMarkVerifyClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7252 void PushAndMarkVerifyClosure::do_oop(narrowOop* p) { PushAndMarkVerifyClosure::do_oop_work(p); } |
| 0 | 7253 |
| 7254 // Upon stack overflow, we discard (part of) the stack, | |
| 7255 // remembering the least address amongst those discarded | |
| 7256 // in CMSCollector's _restart_address. | |
| 7257 void PushAndMarkVerifyClosure::handle_stack_overflow(HeapWord* lost) { | |
| 7258 // Remember the least grey address discarded | |
| 7259 HeapWord* ra = (HeapWord*)_mark_stack->least_value(lost); | |
| 7260 _collector->lower_restart_addr(ra); | |
| 7261 _mark_stack->reset(); // discard stack contents | |
| 7262 _mark_stack->expand(); // expand the stack if possible | |
| 7263 } | |
| 7264 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7265 void PushAndMarkVerifyClosure::do_oop(oop obj) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7266 assert(obj->is_oop_or_null(), "expected an oop or NULL"); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7267 HeapWord* addr = (HeapWord*)obj; |
| 0 | 7268 if (_span.contains(addr) && !_verification_bm->isMarked(addr)) { |
| 7269 // Oop lies in _span and isn't yet grey or black | |
| 7270 _verification_bm->mark(addr); // now grey | |
| 7271 if (!_cms_bm->isMarked(addr)) { | |
| 7272 oop(addr)->print(); | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7273 gclog_or_tty->print_cr(" (" INTPTR_FORMAT " should have been marked)", |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7274 addr); |
| 0 | 7275 fatal("... aborting"); |
| 7276 } | |
| 7277 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7278 if (!_mark_stack->push(obj)) { // stack overflow |
| 0 | 7279 if (PrintCMSStatistics != 0) { |
| 7280 gclog_or_tty->print_cr("CMS marking stack overflow (benign) at " | |
| 7281 SIZE_FORMAT, _mark_stack->capacity()); | |
| 7282 } | |
| 7283 assert(_mark_stack->isFull(), "Else push should have succeeded"); | |
| 7284 handle_stack_overflow(addr); | |
| 7285 } | |
| 7286 // anything including and to the right of _finger | |
| 7287 // will be scanned as we iterate over the remainder of the | |
| 7288 // bit map | |
| 7289 } | |
| 7290 } | |
| 7291 | |
| 7292 PushOrMarkClosure::PushOrMarkClosure(CMSCollector* collector, | |
| 7293 MemRegion span, | |
| 7294 CMSBitMap* bitMap, CMSMarkStack* markStack, | |
| 7295 CMSMarkStack* revisitStack, | |
| 7296 HeapWord* finger, MarkFromRootsClosure* parent) : | |
| 7297 OopClosure(collector->ref_processor()), | |
| 7298 _collector(collector), | |
| 7299 _span(span), | |
| 7300 _bitMap(bitMap), | |
| 7301 _markStack(markStack), | |
| 7302 _revisitStack(revisitStack), | |
| 7303 _finger(finger), | |
| 7304 _parent(parent), | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
7305 _should_remember_klasses(collector->should_unload_classes()) |
| 0 | 7306 { } |
| 7307 | |
| 7308 Par_PushOrMarkClosure::Par_PushOrMarkClosure(CMSCollector* collector, | |
| 7309 MemRegion span, | |
| 7310 CMSBitMap* bit_map, | |
| 7311 OopTaskQueue* work_queue, | |
| 7312 CMSMarkStack* overflow_stack, | |
| 7313 CMSMarkStack* revisit_stack, | |
| 7314 HeapWord* finger, | |
| 7315 HeapWord** global_finger_addr, | |
| 7316 Par_MarkFromRootsClosure* parent) : | |
| 7317 OopClosure(collector->ref_processor()), | |
| 7318 _collector(collector), | |
| 7319 _whole_span(collector->_span), | |
| 7320 _span(span), | |
| 7321 _bit_map(bit_map), | |
| 7322 _work_queue(work_queue), | |
| 7323 _overflow_stack(overflow_stack), | |
| 7324 _revisit_stack(revisit_stack), | |
| 7325 _finger(finger), | |
| 7326 _global_finger_addr(global_finger_addr), | |
| 7327 _parent(parent), | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
7328 _should_remember_klasses(collector->should_unload_classes()) |
| 0 | 7329 { } |
| 7330 | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
7331 // Assumes thread-safe access by callers, who are |
|
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
7332 // responsible for mutual exclusion. |
| 0 | 7333 void CMSCollector::lower_restart_addr(HeapWord* low) { |
| 7334 assert(_span.contains(low), "Out of bounds addr"); | |
| 7335 if (_restart_addr == NULL) { | |
| 7336 _restart_addr = low; | |
| 7337 } else { | |
| 7338 _restart_addr = MIN2(_restart_addr, low); | |
| 7339 } | |
| 7340 } | |
| 7341 | |
| 7342 // Upon stack overflow, we discard (part of) the stack, | |
| 7343 // remembering the least address amongst those discarded | |
| 7344 // in CMSCollector's _restart_address. | |
| 7345 void PushOrMarkClosure::handle_stack_overflow(HeapWord* lost) { | |
| 7346 // Remember the least grey address discarded | |
| 7347 HeapWord* ra = (HeapWord*)_markStack->least_value(lost); | |
| 7348 _collector->lower_restart_addr(ra); | |
| 7349 _markStack->reset(); // discard stack contents | |
| 7350 _markStack->expand(); // expand the stack if possible | |
| 7351 } | |
| 7352 | |
| 7353 // Upon stack overflow, we discard (part of) the stack, | |
| 7354 // remembering the least address amongst those discarded | |
| 7355 // in CMSCollector's _restart_address. | |
| 7356 void Par_PushOrMarkClosure::handle_stack_overflow(HeapWord* lost) { | |
| 7357 // We need to do this under a mutex to prevent other | |
|
340
ebeb6490b814
6722116: CMS: Incorrect overflow handling when using parallel concurrent marking
ysr
parents:
283
diff
changeset
|
7358 // workers from interfering with the work done below. |
| 0 | 7359 MutexLockerEx ml(_overflow_stack->par_lock(), |
| 7360 Mutex::_no_safepoint_check_flag); | |
| 7361 // Remember the least grey address discarded | |
| 7362 HeapWord* ra = (HeapWord*)_overflow_stack->least_value(lost); | |
| 7363 _collector->lower_restart_addr(ra); | |
| 7364 _overflow_stack->reset(); // discard stack contents | |
| 7365 _overflow_stack->expand(); // expand the stack if possible | |
| 7366 } | |
| 7367 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7368 void PushOrMarkClosure::do_oop(oop obj) { |
| 0 | 7369 // Ignore mark word because we are running concurrent with mutators. |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7370 assert(obj->is_oop_or_null(true), "expected an oop or NULL"); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7371 HeapWord* addr = (HeapWord*)obj; |
| 0 | 7372 if (_span.contains(addr) && !_bitMap->isMarked(addr)) { |
| 7373 // Oop lies in _span and isn't yet grey or black | |
| 7374 _bitMap->mark(addr); // now grey | |
| 7375 if (addr < _finger) { | |
| 7376 // the bit map iteration has already either passed, or | |
| 7377 // sampled, this bit in the bit map; we'll need to | |
| 7378 // use the marking stack to scan this oop's oops. | |
| 7379 bool simulate_overflow = false; | |
| 7380 NOT_PRODUCT( | |
| 7381 if (CMSMarkStackOverflowALot && | |
| 7382 _collector->simulate_overflow()) { | |
| 7383 // simulate a stack overflow | |
| 7384 simulate_overflow = true; | |
| 7385 } | |
| 7386 ) | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7387 if (simulate_overflow || !_markStack->push(obj)) { // stack overflow |
| 0 | 7388 if (PrintCMSStatistics != 0) { |
| 7389 gclog_or_tty->print_cr("CMS marking stack overflow (benign) at " | |
| 7390 SIZE_FORMAT, _markStack->capacity()); | |
| 7391 } | |
| 7392 assert(simulate_overflow || _markStack->isFull(), "Else push should have succeeded"); | |
| 7393 handle_stack_overflow(addr); | |
| 7394 } | |
| 7395 } | |
| 7396 // anything including and to the right of _finger | |
| 7397 // will be scanned as we iterate over the remainder of the | |
| 7398 // bit map | |
| 7399 do_yield_check(); | |
| 7400 } | |
| 7401 } | |
| 7402 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7403 void PushOrMarkClosure::do_oop(oop* p) { PushOrMarkClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7404 void PushOrMarkClosure::do_oop(narrowOop* p) { PushOrMarkClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7405 |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7406 void Par_PushOrMarkClosure::do_oop(oop obj) { |
| 0 | 7407 // Ignore mark word because we are running concurrent with mutators. |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7408 assert(obj->is_oop_or_null(true), "expected an oop or NULL"); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7409 HeapWord* addr = (HeapWord*)obj; |
| 0 | 7410 if (_whole_span.contains(addr) && !_bit_map->isMarked(addr)) { |
| 7411 // Oop lies in _span and isn't yet grey or black | |
| 7412 // We read the global_finger (volatile read) strictly after marking oop | |
| 7413 bool res = _bit_map->par_mark(addr); // now grey | |
| 7414 volatile HeapWord** gfa = (volatile HeapWord**)_global_finger_addr; | |
| 7415 // Should we push this marked oop on our stack? | |
| 7416 // -- if someone else marked it, nothing to do | |
| 7417 // -- if target oop is above global finger nothing to do | |
| 7418 // -- if target oop is in chunk and above local finger | |
| 7419 // then nothing to do | |
| 7420 // -- else push on work queue | |
| 7421 if ( !res // someone else marked it, they will deal with it | |
| 7422 || (addr >= *gfa) // will be scanned in a later task | |
| 7423 || (_span.contains(addr) && addr >= _finger)) { // later in this chunk | |
| 7424 return; | |
| 7425 } | |
| 7426 // the bit map iteration has already either passed, or | |
| 7427 // sampled, this bit in the bit map; we'll need to | |
| 7428 // use the marking stack to scan this oop's oops. | |
| 7429 bool simulate_overflow = false; | |
| 7430 NOT_PRODUCT( | |
| 7431 if (CMSMarkStackOverflowALot && | |
| 7432 _collector->simulate_overflow()) { | |
| 7433 // simulate a stack overflow | |
| 7434 simulate_overflow = true; | |
| 7435 } | |
| 7436 ) | |
| 7437 if (simulate_overflow || | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7438 !(_work_queue->push(obj) || _overflow_stack->par_push(obj))) { |
| 0 | 7439 // stack overflow |
| 7440 if (PrintCMSStatistics != 0) { | |
| 7441 gclog_or_tty->print_cr("CMS marking stack overflow (benign) at " | |
| 7442 SIZE_FORMAT, _overflow_stack->capacity()); | |
| 7443 } | |
| 7444 // We cannot assert that the overflow stack is full because | |
| 7445 // it may have been emptied since. | |
| 7446 assert(simulate_overflow || | |
| 7447 _work_queue->size() == _work_queue->max_elems(), | |
| 7448 "Else push should have succeeded"); | |
| 7449 handle_stack_overflow(addr); | |
| 7450 } | |
| 7451 do_yield_check(); | |
| 7452 } | |
| 7453 } | |
| 7454 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7455 void Par_PushOrMarkClosure::do_oop(oop* p) { Par_PushOrMarkClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7456 void Par_PushOrMarkClosure::do_oop(narrowOop* p) { Par_PushOrMarkClosure::do_oop_work(p); } |
| 0 | 7457 |
| 7458 PushAndMarkClosure::PushAndMarkClosure(CMSCollector* collector, | |
| 7459 MemRegion span, | |
| 7460 ReferenceProcessor* rp, | |
| 7461 CMSBitMap* bit_map, | |
| 7462 CMSBitMap* mod_union_table, | |
| 7463 CMSMarkStack* mark_stack, | |
| 7464 CMSMarkStack* revisit_stack, | |
| 7465 bool concurrent_precleaning): | |
| 7466 OopClosure(rp), | |
| 7467 _collector(collector), | |
| 7468 _span(span), | |
| 7469 _bit_map(bit_map), | |
| 7470 _mod_union_table(mod_union_table), | |
| 7471 _mark_stack(mark_stack), | |
| 7472 _revisit_stack(revisit_stack), | |
| 7473 _concurrent_precleaning(concurrent_precleaning), | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
7474 _should_remember_klasses(collector->should_unload_classes()) |
| 0 | 7475 { |
| 7476 assert(_ref_processor != NULL, "_ref_processor shouldn't be NULL"); | |
| 7477 } | |
| 7478 | |
| 7479 // Grey object rescan during pre-cleaning and second checkpoint phases -- | |
| 7480 // the non-parallel version (the parallel version appears further below.) | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7481 void PushAndMarkClosure::do_oop(oop obj) { |
|
342
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7482 // Ignore mark word verification. If during concurrent precleaning, |
|
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7483 // the object monitor may be locked. If during the checkpoint |
|
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7484 // phases, the object may already have been reached by a different |
|
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7485 // path and may be at the end of the global overflow list (so |
|
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7486 // the mark word may be NULL). |
|
37f87013dfd8
6711316: Open source the Garbage-First garbage collector
ysr
parents:
143
diff
changeset
|
7487 assert(obj->is_oop_or_null(true /* ignore mark word */), |
| 0 | 7488 "expected an oop or NULL"); |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7489 HeapWord* addr = (HeapWord*)obj; |
| 0 | 7490 // Check if oop points into the CMS generation |
| 7491 // and is not marked | |
| 7492 if (_span.contains(addr) && !_bit_map->isMarked(addr)) { | |
| 7493 // a white object ... | |
| 7494 _bit_map->mark(addr); // ... now grey | |
| 7495 // push on the marking stack (grey set) | |
| 7496 bool simulate_overflow = false; | |
| 7497 NOT_PRODUCT( | |
| 7498 if (CMSMarkStackOverflowALot && | |
| 7499 _collector->simulate_overflow()) { | |
| 7500 // simulate a stack overflow | |
| 7501 simulate_overflow = true; | |
| 7502 } | |
| 7503 ) | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7504 if (simulate_overflow || !_mark_stack->push(obj)) { |
| 0 | 7505 if (_concurrent_precleaning) { |
|
283
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7506 // During precleaning we can just dirty the appropriate card(s) |
| 0 | 7507 // in the mod union table, thus ensuring that the object remains |
|
283
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7508 // in the grey set and continue. In the case of object arrays |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7509 // we need to dirty all of the cards that the object spans, |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7510 // since the rescan of object arrays will be limited to the |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7511 // dirty cards. |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7512 // Note that no one can be intefering with us in this action |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7513 // of dirtying the mod union table, so no locking or atomics |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7514 // are required. |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7515 if (obj->is_objArray()) { |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7516 size_t sz = obj->size(); |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7517 HeapWord* end_card_addr = (HeapWord*)round_to( |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7518 (intptr_t)(addr+sz), CardTableModRefBS::card_size); |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7519 MemRegion redirty_range = MemRegion(addr, end_card_addr); |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7520 assert(!redirty_range.is_empty(), "Arithmetical tautology"); |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7521 _mod_union_table->mark_range(redirty_range); |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7522 } else { |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7523 _mod_union_table->mark(addr); |
|
9199f248b0ee
6722112: CMS: Incorrect encoding of overflown object arrays during concurrent precleaning
ysr
parents:
271
diff
changeset
|
7524 } |
| 0 | 7525 _collector->_ser_pmc_preclean_ovflw++; |
| 7526 } else { | |
| 7527 // During the remark phase, we need to remember this oop | |
| 7528 // in the overflow list. | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7529 _collector->push_on_overflow_list(obj); |
| 0 | 7530 _collector->_ser_pmc_remark_ovflw++; |
| 7531 } | |
| 7532 } | |
| 7533 } | |
| 7534 } | |
| 7535 | |
| 7536 Par_PushAndMarkClosure::Par_PushAndMarkClosure(CMSCollector* collector, | |
| 7537 MemRegion span, | |
| 7538 ReferenceProcessor* rp, | |
| 7539 CMSBitMap* bit_map, | |
| 7540 OopTaskQueue* work_queue, | |
| 7541 CMSMarkStack* revisit_stack): | |
| 7542 OopClosure(rp), | |
| 7543 _collector(collector), | |
| 7544 _span(span), | |
| 7545 _bit_map(bit_map), | |
| 7546 _work_queue(work_queue), | |
| 7547 _revisit_stack(revisit_stack), | |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
7548 _should_remember_klasses(collector->should_unload_classes()) |
| 0 | 7549 { |
| 7550 assert(_ref_processor != NULL, "_ref_processor shouldn't be NULL"); | |
| 7551 } | |
| 7552 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7553 void PushAndMarkClosure::do_oop(oop* p) { PushAndMarkClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7554 void PushAndMarkClosure::do_oop(narrowOop* p) { PushAndMarkClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7555 |
| 0 | 7556 // Grey object rescan during second checkpoint phase -- |
| 7557 // the parallel version. | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7558 void Par_PushAndMarkClosure::do_oop(oop obj) { |
| 0 | 7559 // In the assert below, we ignore the mark word because |
| 7560 // this oop may point to an already visited object that is | |
| 7561 // on the overflow stack (in which case the mark word has | |
| 7562 // been hijacked for chaining into the overflow stack -- | |
| 7563 // if this is the last object in the overflow stack then | |
| 7564 // its mark word will be NULL). Because this object may | |
| 7565 // have been subsequently popped off the global overflow | |
| 7566 // stack, and the mark word possibly restored to the prototypical | |
| 7567 // value, by the time we get to examined this failing assert in | |
| 7568 // the debugger, is_oop_or_null(false) may subsequently start | |
| 7569 // to hold. | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7570 assert(obj->is_oop_or_null(true), |
| 0 | 7571 "expected an oop or NULL"); |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7572 HeapWord* addr = (HeapWord*)obj; |
| 0 | 7573 // Check if oop points into the CMS generation |
| 7574 // and is not marked | |
| 7575 if (_span.contains(addr) && !_bit_map->isMarked(addr)) { | |
| 7576 // a white object ... | |
| 7577 // If we manage to "claim" the object, by being the | |
| 7578 // first thread to mark it, then we push it on our | |
| 7579 // marking stack | |
| 7580 if (_bit_map->par_mark(addr)) { // ... now grey | |
| 7581 // push on work queue (grey set) | |
| 7582 bool simulate_overflow = false; | |
| 7583 NOT_PRODUCT( | |
| 7584 if (CMSMarkStackOverflowALot && | |
| 7585 _collector->par_simulate_overflow()) { | |
| 7586 // simulate a stack overflow | |
| 7587 simulate_overflow = true; | |
| 7588 } | |
| 7589 ) | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7590 if (simulate_overflow || !_work_queue->push(obj)) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7591 _collector->par_push_on_overflow_list(obj); |
| 0 | 7592 _collector->_par_pmc_remark_ovflw++; // imprecise OK: no need to CAS |
| 7593 } | |
| 7594 } // Else, some other thread got there first | |
| 7595 } | |
| 7596 } | |
| 7597 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7598 void Par_PushAndMarkClosure::do_oop(oop* p) { Par_PushAndMarkClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7599 void Par_PushAndMarkClosure::do_oop(narrowOop* p) { Par_PushAndMarkClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
7600 |
| 0 | 7601 void PushAndMarkClosure::remember_klass(Klass* k) { |
| 7602 if (!_revisit_stack->push(oop(k))) { | |
| 7603 fatal("Revisit stack overflowed in PushAndMarkClosure"); | |
| 7604 } | |
| 7605 } | |
| 7606 | |
| 7607 void Par_PushAndMarkClosure::remember_klass(Klass* k) { | |
| 7608 if (!_revisit_stack->par_push(oop(k))) { | |
| 7609 fatal("Revist stack overflowed in Par_PushAndMarkClosure"); | |
| 7610 } | |
| 7611 } | |
| 7612 | |
| 7613 void CMSPrecleanRefsYieldClosure::do_yield_work() { | |
| 7614 Mutex* bml = _collector->bitMapLock(); | |
| 7615 assert_lock_strong(bml); | |
| 7616 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 7617 "CMS thread should hold CMS token"); | |
| 7618 | |
| 7619 bml->unlock(); | |
| 7620 ConcurrentMarkSweepThread::desynchronize(true); | |
| 7621 | |
| 7622 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 7623 | |
| 7624 _collector->stopTimer(); | |
| 7625 GCPauseTimer p(_collector->size_policy()->concurrent_timer_ptr()); | |
| 7626 if (PrintCMSStatistics != 0) { | |
| 7627 _collector->incrementYields(); | |
| 7628 } | |
| 7629 _collector->icms_wait(); | |
| 7630 | |
| 7631 // See the comment in coordinator_yield() | |
| 7632 for (unsigned i = 0; i < CMSYieldSleepCount && | |
| 7633 ConcurrentMarkSweepThread::should_yield() && | |
| 7634 !CMSCollector::foregroundGCIsActive(); ++i) { | |
| 7635 os::sleep(Thread::current(), 1, false); | |
| 7636 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 7637 } | |
| 7638 | |
| 7639 ConcurrentMarkSweepThread::synchronize(true); | |
| 7640 bml->lock(); | |
| 7641 | |
| 7642 _collector->startTimer(); | |
| 7643 } | |
| 7644 | |
| 7645 bool CMSPrecleanRefsYieldClosure::should_return() { | |
| 7646 if (ConcurrentMarkSweepThread::should_yield()) { | |
| 7647 do_yield_work(); | |
| 7648 } | |
| 7649 return _collector->foregroundGCIsActive(); | |
| 7650 } | |
| 7651 | |
| 7652 void MarkFromDirtyCardsClosure::do_MemRegion(MemRegion mr) { | |
| 7653 assert(((size_t)mr.start())%CardTableModRefBS::card_size_in_words == 0, | |
| 7654 "mr should be aligned to start at a card boundary"); | |
| 7655 // We'd like to assert: | |
| 7656 // assert(mr.word_size()%CardTableModRefBS::card_size_in_words == 0, | |
| 7657 // "mr should be a range of cards"); | |
| 7658 // However, that would be too strong in one case -- the last | |
| 7659 // partition ends at _unallocated_block which, in general, can be | |
| 7660 // an arbitrary boundary, not necessarily card aligned. | |
| 7661 if (PrintCMSStatistics != 0) { | |
| 7662 _num_dirty_cards += | |
| 7663 mr.word_size()/CardTableModRefBS::card_size_in_words; | |
| 7664 } | |
| 7665 _space->object_iterate_mem(mr, &_scan_cl); | |
| 7666 } | |
| 7667 | |
| 7668 SweepClosure::SweepClosure(CMSCollector* collector, | |
| 7669 ConcurrentMarkSweepGeneration* g, | |
| 7670 CMSBitMap* bitMap, bool should_yield) : | |
| 7671 _collector(collector), | |
| 7672 _g(g), | |
| 7673 _sp(g->cmsSpace()), | |
| 7674 _limit(_sp->sweep_limit()), | |
| 7675 _freelistLock(_sp->freelistLock()), | |
| 7676 _bitMap(bitMap), | |
| 7677 _yield(should_yield), | |
| 7678 _inFreeRange(false), // No free range at beginning of sweep | |
| 7679 _freeRangeInFreeLists(false), // No free range at beginning of sweep | |
| 7680 _lastFreeRangeCoalesced(false), | |
| 7681 _freeFinger(g->used_region().start()) | |
| 7682 { | |
| 7683 NOT_PRODUCT( | |
| 7684 _numObjectsFreed = 0; | |
| 7685 _numWordsFreed = 0; | |
| 7686 _numObjectsLive = 0; | |
| 7687 _numWordsLive = 0; | |
| 7688 _numObjectsAlreadyFree = 0; | |
| 7689 _numWordsAlreadyFree = 0; | |
| 7690 _last_fc = NULL; | |
| 7691 | |
| 7692 _sp->initializeIndexedFreeListArrayReturnedBytes(); | |
| 7693 _sp->dictionary()->initializeDictReturnedBytes(); | |
| 7694 ) | |
| 7695 assert(_limit >= _sp->bottom() && _limit <= _sp->end(), | |
| 7696 "sweep _limit out of bounds"); | |
| 7697 if (CMSTraceSweeper) { | |
| 7698 gclog_or_tty->print("\n====================\nStarting new sweep\n"); | |
| 7699 } | |
| 7700 } | |
| 7701 | |
| 7702 // We need this destructor to reclaim any space at the end | |
| 7703 // of the space, which do_blk below may not have added back to | |
| 7704 // the free lists. [basically dealing with the "fringe effect"] | |
| 7705 SweepClosure::~SweepClosure() { | |
| 7706 assert_lock_strong(_freelistLock); | |
| 7707 // this should be treated as the end of a free run if any | |
| 7708 // The current free range should be returned to the free lists | |
| 7709 // as one coalesced chunk. | |
| 7710 if (inFreeRange()) { | |
| 7711 flushCurFreeChunk(freeFinger(), | |
| 7712 pointer_delta(_limit, freeFinger())); | |
| 7713 assert(freeFinger() < _limit, "the finger pointeth off base"); | |
| 7714 if (CMSTraceSweeper) { | |
| 7715 gclog_or_tty->print("destructor:"); | |
| 7716 gclog_or_tty->print("Sweep:put_free_blk 0x%x ("SIZE_FORMAT") " | |
| 7717 "[coalesced:"SIZE_FORMAT"]\n", | |
| 7718 freeFinger(), pointer_delta(_limit, freeFinger()), | |
| 7719 lastFreeRangeCoalesced()); | |
| 7720 } | |
| 7721 } | |
| 7722 NOT_PRODUCT( | |
| 7723 if (Verbose && PrintGC) { | |
| 7724 gclog_or_tty->print("Collected "SIZE_FORMAT" objects, " | |
| 7725 SIZE_FORMAT " bytes", | |
| 7726 _numObjectsFreed, _numWordsFreed*sizeof(HeapWord)); | |
| 7727 gclog_or_tty->print_cr("\nLive "SIZE_FORMAT" objects, " | |
| 7728 SIZE_FORMAT" bytes " | |
| 7729 "Already free "SIZE_FORMAT" objects, "SIZE_FORMAT" bytes", | |
| 7730 _numObjectsLive, _numWordsLive*sizeof(HeapWord), | |
| 7731 _numObjectsAlreadyFree, _numWordsAlreadyFree*sizeof(HeapWord)); | |
| 7732 size_t totalBytes = (_numWordsFreed + _numWordsLive + _numWordsAlreadyFree) * | |
| 7733 sizeof(HeapWord); | |
| 7734 gclog_or_tty->print_cr("Total sweep: "SIZE_FORMAT" bytes", totalBytes); | |
| 7735 | |
| 7736 if (PrintCMSStatistics && CMSVerifyReturnedBytes) { | |
| 7737 size_t indexListReturnedBytes = _sp->sumIndexedFreeListArrayReturnedBytes(); | |
| 7738 size_t dictReturnedBytes = _sp->dictionary()->sumDictReturnedBytes(); | |
| 7739 size_t returnedBytes = indexListReturnedBytes + dictReturnedBytes; | |
| 7740 gclog_or_tty->print("Returned "SIZE_FORMAT" bytes", returnedBytes); | |
| 7741 gclog_or_tty->print(" Indexed List Returned "SIZE_FORMAT" bytes", | |
| 7742 indexListReturnedBytes); | |
| 7743 gclog_or_tty->print_cr(" Dictionary Returned "SIZE_FORMAT" bytes", | |
| 7744 dictReturnedBytes); | |
| 7745 } | |
| 7746 } | |
| 7747 ) | |
| 7748 // Now, in debug mode, just null out the sweep_limit | |
| 7749 NOT_PRODUCT(_sp->clear_sweep_limit();) | |
| 7750 if (CMSTraceSweeper) { | |
| 7751 gclog_or_tty->print("end of sweep\n================\n"); | |
| 7752 } | |
| 7753 } | |
| 7754 | |
| 7755 void SweepClosure::initialize_free_range(HeapWord* freeFinger, | |
| 7756 bool freeRangeInFreeLists) { | |
| 7757 if (CMSTraceSweeper) { | |
| 7758 gclog_or_tty->print("---- Start free range 0x%x with free block [%d] (%d)\n", | |
| 7759 freeFinger, _sp->block_size(freeFinger), | |
| 7760 freeRangeInFreeLists); | |
| 7761 } | |
| 7762 assert(!inFreeRange(), "Trampling existing free range"); | |
| 7763 set_inFreeRange(true); | |
| 7764 set_lastFreeRangeCoalesced(false); | |
| 7765 | |
| 7766 set_freeFinger(freeFinger); | |
| 7767 set_freeRangeInFreeLists(freeRangeInFreeLists); | |
| 7768 if (CMSTestInFreeList) { | |
| 7769 if (freeRangeInFreeLists) { | |
| 7770 FreeChunk* fc = (FreeChunk*) freeFinger; | |
| 7771 assert(fc->isFree(), "A chunk on the free list should be free."); | |
| 7772 assert(fc->size() > 0, "Free range should have a size"); | |
| 7773 assert(_sp->verifyChunkInFreeLists(fc), "Chunk is not in free lists"); | |
| 7774 } | |
| 7775 } | |
| 7776 } | |
| 7777 | |
| 7778 // Note that the sweeper runs concurrently with mutators. Thus, | |
| 7779 // it is possible for direct allocation in this generation to happen | |
| 7780 // in the middle of the sweep. Note that the sweeper also coalesces | |
| 7781 // contiguous free blocks. Thus, unless the sweeper and the allocator | |
| 7782 // synchronize appropriately freshly allocated blocks may get swept up. | |
| 7783 // This is accomplished by the sweeper locking the free lists while | |
| 7784 // it is sweeping. Thus blocks that are determined to be free are | |
| 7785 // indeed free. There is however one additional complication: | |
| 7786 // blocks that have been allocated since the final checkpoint and | |
| 7787 // mark, will not have been marked and so would be treated as | |
| 7788 // unreachable and swept up. To prevent this, the allocator marks | |
| 7789 // the bit map when allocating during the sweep phase. This leads, | |
| 7790 // however, to a further complication -- objects may have been allocated | |
| 7791 // but not yet initialized -- in the sense that the header isn't yet | |
| 7792 // installed. The sweeper can not then determine the size of the block | |
| 7793 // in order to skip over it. To deal with this case, we use a technique | |
| 7794 // (due to Printezis) to encode such uninitialized block sizes in the | |
| 7795 // bit map. Since the bit map uses a bit per every HeapWord, but the | |
| 7796 // CMS generation has a minimum object size of 3 HeapWords, it follows | |
| 7797 // that "normal marks" won't be adjacent in the bit map (there will | |
| 7798 // always be at least two 0 bits between successive 1 bits). We make use | |
| 7799 // of these "unused" bits to represent uninitialized blocks -- the bit | |
| 7800 // corresponding to the start of the uninitialized object and the next | |
| 7801 // bit are both set. Finally, a 1 bit marks the end of the object that | |
| 7802 // started with the two consecutive 1 bits to indicate its potentially | |
| 7803 // uninitialized state. | |
| 7804 | |
| 7805 size_t SweepClosure::do_blk_careful(HeapWord* addr) { | |
| 7806 FreeChunk* fc = (FreeChunk*)addr; | |
| 7807 size_t res; | |
| 7808 | |
| 7809 // check if we are done sweepinrg | |
| 7810 if (addr == _limit) { // we have swept up to the limit, do nothing more | |
| 7811 assert(_limit >= _sp->bottom() && _limit <= _sp->end(), | |
| 7812 "sweep _limit out of bounds"); | |
| 7813 // help the closure application finish | |
| 7814 return pointer_delta(_sp->end(), _limit); | |
| 7815 } | |
| 7816 assert(addr <= _limit, "sweep invariant"); | |
| 7817 | |
| 7818 // check if we should yield | |
| 7819 do_yield_check(addr); | |
| 7820 if (fc->isFree()) { | |
| 7821 // Chunk that is already free | |
| 7822 res = fc->size(); | |
| 7823 doAlreadyFreeChunk(fc); | |
| 7824 debug_only(_sp->verifyFreeLists()); | |
| 7825 assert(res == fc->size(), "Don't expect the size to change"); | |
| 7826 NOT_PRODUCT( | |
| 7827 _numObjectsAlreadyFree++; | |
| 7828 _numWordsAlreadyFree += res; | |
| 7829 ) | |
| 7830 NOT_PRODUCT(_last_fc = fc;) | |
| 7831 } else if (!_bitMap->isMarked(addr)) { | |
| 7832 // Chunk is fresh garbage | |
| 7833 res = doGarbageChunk(fc); | |
| 7834 debug_only(_sp->verifyFreeLists()); | |
| 7835 NOT_PRODUCT( | |
| 7836 _numObjectsFreed++; | |
| 7837 _numWordsFreed += res; | |
| 7838 ) | |
| 7839 } else { | |
| 7840 // Chunk that is alive. | |
| 7841 res = doLiveChunk(fc); | |
| 7842 debug_only(_sp->verifyFreeLists()); | |
| 7843 NOT_PRODUCT( | |
| 7844 _numObjectsLive++; | |
| 7845 _numWordsLive += res; | |
| 7846 ) | |
| 7847 } | |
| 7848 return res; | |
| 7849 } | |
| 7850 | |
| 7851 // For the smart allocation, record following | |
| 7852 // split deaths - a free chunk is removed from its free list because | |
| 7853 // it is being split into two or more chunks. | |
| 7854 // split birth - a free chunk is being added to its free list because | |
| 7855 // a larger free chunk has been split and resulted in this free chunk. | |
| 7856 // coal death - a free chunk is being removed from its free list because | |
| 7857 // it is being coalesced into a large free chunk. | |
| 7858 // coal birth - a free chunk is being added to its free list because | |
| 7859 // it was created when two or more free chunks where coalesced into | |
| 7860 // this free chunk. | |
| 7861 // | |
| 7862 // These statistics are used to determine the desired number of free | |
| 7863 // chunks of a given size. The desired number is chosen to be relative | |
| 7864 // to the end of a CMS sweep. The desired number at the end of a sweep | |
| 7865 // is the | |
| 7866 // count-at-end-of-previous-sweep (an amount that was enough) | |
| 7867 // - count-at-beginning-of-current-sweep (the excess) | |
| 7868 // + split-births (gains in this size during interval) | |
| 7869 // - split-deaths (demands on this size during interval) | |
| 7870 // where the interval is from the end of one sweep to the end of the | |
| 7871 // next. | |
| 7872 // | |
| 7873 // When sweeping the sweeper maintains an accumulated chunk which is | |
| 7874 // the chunk that is made up of chunks that have been coalesced. That | |
| 7875 // will be termed the left-hand chunk. A new chunk of garbage that | |
| 7876 // is being considered for coalescing will be referred to as the | |
| 7877 // right-hand chunk. | |
| 7878 // | |
| 7879 // When making a decision on whether to coalesce a right-hand chunk with | |
| 7880 // the current left-hand chunk, the current count vs. the desired count | |
| 7881 // of the left-hand chunk is considered. Also if the right-hand chunk | |
| 7882 // is near the large chunk at the end of the heap (see | |
| 7883 // ConcurrentMarkSweepGeneration::isNearLargestChunk()), then the | |
| 7884 // left-hand chunk is coalesced. | |
| 7885 // | |
| 7886 // When making a decision about whether to split a chunk, the desired count | |
| 7887 // vs. the current count of the candidate to be split is also considered. | |
| 7888 // If the candidate is underpopulated (currently fewer chunks than desired) | |
| 7889 // a chunk of an overpopulated (currently more chunks than desired) size may | |
| 7890 // be chosen. The "hint" associated with a free list, if non-null, points | |
| 7891 // to a free list which may be overpopulated. | |
| 7892 // | |
| 7893 | |
| 7894 void SweepClosure::doAlreadyFreeChunk(FreeChunk* fc) { | |
| 7895 size_t size = fc->size(); | |
| 7896 // Chunks that cannot be coalesced are not in the | |
| 7897 // free lists. | |
| 7898 if (CMSTestInFreeList && !fc->cantCoalesce()) { | |
| 7899 assert(_sp->verifyChunkInFreeLists(fc), | |
| 7900 "free chunk should be in free lists"); | |
| 7901 } | |
| 7902 // a chunk that is already free, should not have been | |
| 7903 // marked in the bit map | |
| 7904 HeapWord* addr = (HeapWord*) fc; | |
| 7905 assert(!_bitMap->isMarked(addr), "free chunk should be unmarked"); | |
| 7906 // Verify that the bit map has no bits marked between | |
| 7907 // addr and purported end of this block. | |
| 7908 _bitMap->verifyNoOneBitsInRange(addr + 1, addr + size); | |
| 7909 | |
| 7910 // Some chunks cannot be coalesced in under any circumstances. | |
| 7911 // See the definition of cantCoalesce(). | |
| 7912 if (!fc->cantCoalesce()) { | |
| 7913 // This chunk can potentially be coalesced. | |
| 7914 if (_sp->adaptive_freelists()) { | |
| 7915 // All the work is done in | |
| 7916 doPostIsFreeOrGarbageChunk(fc, size); | |
| 7917 } else { // Not adaptive free lists | |
| 7918 // this is a free chunk that can potentially be coalesced by the sweeper; | |
| 7919 if (!inFreeRange()) { | |
| 7920 // if the next chunk is a free block that can't be coalesced | |
| 7921 // it doesn't make sense to remove this chunk from the free lists | |
| 7922 FreeChunk* nextChunk = (FreeChunk*)(addr + size); | |
| 7923 assert((HeapWord*)nextChunk <= _limit, "sweep invariant"); | |
| 7924 if ((HeapWord*)nextChunk < _limit && // there's a next chunk... | |
| 7925 nextChunk->isFree() && // which is free... | |
| 7926 nextChunk->cantCoalesce()) { // ... but cant be coalesced | |
| 7927 // nothing to do | |
| 7928 } else { | |
| 7929 // Potentially the start of a new free range: | |
| 7930 // Don't eagerly remove it from the free lists. | |
| 7931 // No need to remove it if it will just be put | |
| 7932 // back again. (Also from a pragmatic point of view | |
| 7933 // if it is a free block in a region that is beyond | |
| 7934 // any allocated blocks, an assertion will fail) | |
| 7935 // Remember the start of a free run. | |
| 7936 initialize_free_range(addr, true); | |
| 7937 // end - can coalesce with next chunk | |
| 7938 } | |
| 7939 } else { | |
| 7940 // the midst of a free range, we are coalescing | |
| 7941 debug_only(record_free_block_coalesced(fc);) | |
| 7942 if (CMSTraceSweeper) { | |
| 7943 gclog_or_tty->print(" -- pick up free block 0x%x (%d)\n", fc, size); | |
| 7944 } | |
| 7945 // remove it from the free lists | |
| 7946 _sp->removeFreeChunkFromFreeLists(fc); | |
| 7947 set_lastFreeRangeCoalesced(true); | |
| 7948 // If the chunk is being coalesced and the current free range is | |
| 7949 // in the free lists, remove the current free range so that it | |
| 7950 // will be returned to the free lists in its entirety - all | |
| 7951 // the coalesced pieces included. | |
| 7952 if (freeRangeInFreeLists()) { | |
| 7953 FreeChunk* ffc = (FreeChunk*) freeFinger(); | |
| 7954 assert(ffc->size() == pointer_delta(addr, freeFinger()), | |
| 7955 "Size of free range is inconsistent with chunk size."); | |
| 7956 if (CMSTestInFreeList) { | |
| 7957 assert(_sp->verifyChunkInFreeLists(ffc), | |
| 7958 "free range is not in free lists"); | |
| 7959 } | |
| 7960 _sp->removeFreeChunkFromFreeLists(ffc); | |
| 7961 set_freeRangeInFreeLists(false); | |
| 7962 } | |
| 7963 } | |
| 7964 } | |
| 7965 } else { | |
| 7966 // Code path common to both original and adaptive free lists. | |
| 7967 | |
| 7968 // cant coalesce with previous block; this should be treated | |
| 7969 // as the end of a free run if any | |
| 7970 if (inFreeRange()) { | |
| 7971 // we kicked some butt; time to pick up the garbage | |
| 7972 assert(freeFinger() < addr, "the finger pointeth off base"); | |
| 7973 flushCurFreeChunk(freeFinger(), pointer_delta(addr, freeFinger())); | |
| 7974 } | |
| 7975 // else, nothing to do, just continue | |
| 7976 } | |
| 7977 } | |
| 7978 | |
| 7979 size_t SweepClosure::doGarbageChunk(FreeChunk* fc) { | |
| 7980 // This is a chunk of garbage. It is not in any free list. | |
| 7981 // Add it to a free list or let it possibly be coalesced into | |
| 7982 // a larger chunk. | |
| 7983 HeapWord* addr = (HeapWord*) fc; | |
| 7984 size_t size = CompactibleFreeListSpace::adjustObjectSize(oop(addr)->size()); | |
| 7985 | |
| 7986 if (_sp->adaptive_freelists()) { | |
| 7987 // Verify that the bit map has no bits marked between | |
| 7988 // addr and purported end of just dead object. | |
| 7989 _bitMap->verifyNoOneBitsInRange(addr + 1, addr + size); | |
| 7990 | |
| 7991 doPostIsFreeOrGarbageChunk(fc, size); | |
| 7992 } else { | |
| 7993 if (!inFreeRange()) { | |
| 7994 // start of a new free range | |
| 7995 assert(size > 0, "A free range should have a size"); | |
| 7996 initialize_free_range(addr, false); | |
| 7997 | |
| 7998 } else { | |
| 7999 // this will be swept up when we hit the end of the | |
| 8000 // free range | |
| 8001 if (CMSTraceSweeper) { | |
| 8002 gclog_or_tty->print(" -- pick up garbage 0x%x (%d) \n", fc, size); | |
| 8003 } | |
| 8004 // If the chunk is being coalesced and the current free range is | |
| 8005 // in the free lists, remove the current free range so that it | |
| 8006 // will be returned to the free lists in its entirety - all | |
| 8007 // the coalesced pieces included. | |
| 8008 if (freeRangeInFreeLists()) { | |
| 8009 FreeChunk* ffc = (FreeChunk*)freeFinger(); | |
| 8010 assert(ffc->size() == pointer_delta(addr, freeFinger()), | |
| 8011 "Size of free range is inconsistent with chunk size."); | |
| 8012 if (CMSTestInFreeList) { | |
| 8013 assert(_sp->verifyChunkInFreeLists(ffc), | |
| 8014 "free range is not in free lists"); | |
| 8015 } | |
| 8016 _sp->removeFreeChunkFromFreeLists(ffc); | |
| 8017 set_freeRangeInFreeLists(false); | |
| 8018 } | |
| 8019 set_lastFreeRangeCoalesced(true); | |
| 8020 } | |
| 8021 // this will be swept up when we hit the end of the free range | |
| 8022 | |
| 8023 // Verify that the bit map has no bits marked between | |
| 8024 // addr and purported end of just dead object. | |
| 8025 _bitMap->verifyNoOneBitsInRange(addr + 1, addr + size); | |
| 8026 } | |
| 8027 return size; | |
| 8028 } | |
| 8029 | |
| 8030 size_t SweepClosure::doLiveChunk(FreeChunk* fc) { | |
| 8031 HeapWord* addr = (HeapWord*) fc; | |
| 8032 // The sweeper has just found a live object. Return any accumulated | |
| 8033 // left hand chunk to the free lists. | |
| 8034 if (inFreeRange()) { | |
| 8035 if (_sp->adaptive_freelists()) { | |
| 8036 flushCurFreeChunk(freeFinger(), | |
| 8037 pointer_delta(addr, freeFinger())); | |
| 8038 } else { // not adaptive freelists | |
| 8039 set_inFreeRange(false); | |
| 8040 // Add the free range back to the free list if it is not already | |
| 8041 // there. | |
| 8042 if (!freeRangeInFreeLists()) { | |
| 8043 assert(freeFinger() < addr, "the finger pointeth off base"); | |
| 8044 if (CMSTraceSweeper) { | |
| 8045 gclog_or_tty->print("Sweep:put_free_blk 0x%x (%d) " | |
| 8046 "[coalesced:%d]\n", | |
| 8047 freeFinger(), pointer_delta(addr, freeFinger()), | |
| 8048 lastFreeRangeCoalesced()); | |
| 8049 } | |
| 8050 _sp->addChunkAndRepairOffsetTable(freeFinger(), | |
| 8051 pointer_delta(addr, freeFinger()), lastFreeRangeCoalesced()); | |
| 8052 } | |
| 8053 } | |
| 8054 } | |
| 8055 | |
| 8056 // Common code path for original and adaptive free lists. | |
| 8057 | |
| 8058 // this object is live: we'd normally expect this to be | |
| 8059 // an oop, and like to assert the following: | |
| 8060 // assert(oop(addr)->is_oop(), "live block should be an oop"); | |
| 8061 // However, as we commented above, this may be an object whose | |
| 8062 // header hasn't yet been initialized. | |
| 8063 size_t size; | |
| 8064 assert(_bitMap->isMarked(addr), "Tautology for this control point"); | |
| 8065 if (_bitMap->isMarked(addr + 1)) { | |
| 8066 // Determine the size from the bit map, rather than trying to | |
| 8067 // compute it from the object header. | |
| 8068 HeapWord* nextOneAddr = _bitMap->getNextMarkedWordAddress(addr + 2); | |
| 8069 size = pointer_delta(nextOneAddr + 1, addr); | |
| 8070 assert(size == CompactibleFreeListSpace::adjustObjectSize(size), | |
| 8071 "alignment problem"); | |
| 8072 | |
| 8073 #ifdef DEBUG | |
| 187 | 8074 if (oop(addr)->klass_or_null() != NULL && |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
8075 ( !_collector->should_unload_classes() |
| 0 | 8076 || oop(addr)->is_parsable())) { |
| 8077 // Ignore mark word because we are running concurrent with mutators | |
| 8078 assert(oop(addr)->is_oop(true), "live block should be an oop"); | |
| 8079 assert(size == | |
| 8080 CompactibleFreeListSpace::adjustObjectSize(oop(addr)->size()), | |
| 8081 "P-mark and computed size do not agree"); | |
| 8082 } | |
| 8083 #endif | |
| 8084 | |
| 8085 } else { | |
| 8086 // This should be an initialized object that's alive. | |
| 187 | 8087 assert(oop(addr)->klass_or_null() != NULL && |
|
94
0834225a7916
6634032: CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction
ysr
parents:
9
diff
changeset
|
8088 (!_collector->should_unload_classes() |
| 0 | 8089 || oop(addr)->is_parsable()), |
| 8090 "Should be an initialized object"); | |
| 8091 // Ignore mark word because we are running concurrent with mutators | |
| 8092 assert(oop(addr)->is_oop(true), "live block should be an oop"); | |
| 8093 // Verify that the bit map has no bits marked between | |
| 8094 // addr and purported end of this block. | |
| 8095 size = CompactibleFreeListSpace::adjustObjectSize(oop(addr)->size()); | |
| 8096 assert(size >= 3, "Necessary for Printezis marks to work"); | |
| 8097 assert(!_bitMap->isMarked(addr+1), "Tautology for this control point"); | |
| 8098 DEBUG_ONLY(_bitMap->verifyNoOneBitsInRange(addr+2, addr+size);) | |
| 8099 } | |
| 8100 return size; | |
| 8101 } | |
| 8102 | |
| 8103 void SweepClosure::doPostIsFreeOrGarbageChunk(FreeChunk* fc, | |
| 8104 size_t chunkSize) { | |
| 8105 // doPostIsFreeOrGarbageChunk() should only be called in the smart allocation | |
| 8106 // scheme. | |
| 8107 bool fcInFreeLists = fc->isFree(); | |
| 8108 assert(_sp->adaptive_freelists(), "Should only be used in this case."); | |
| 8109 assert((HeapWord*)fc <= _limit, "sweep invariant"); | |
| 8110 if (CMSTestInFreeList && fcInFreeLists) { | |
| 8111 assert(_sp->verifyChunkInFreeLists(fc), | |
| 8112 "free chunk is not in free lists"); | |
| 8113 } | |
| 8114 | |
| 8115 | |
| 8116 if (CMSTraceSweeper) { | |
| 8117 gclog_or_tty->print_cr(" -- pick up another chunk at 0x%x (%d)", fc, chunkSize); | |
| 8118 } | |
| 8119 | |
| 8120 HeapWord* addr = (HeapWord*) fc; | |
| 8121 | |
| 8122 bool coalesce; | |
| 8123 size_t left = pointer_delta(addr, freeFinger()); | |
| 8124 size_t right = chunkSize; | |
| 8125 switch (FLSCoalescePolicy) { | |
| 8126 // numeric value forms a coalition aggressiveness metric | |
| 8127 case 0: { // never coalesce | |
| 8128 coalesce = false; | |
| 8129 break; | |
| 8130 } | |
| 8131 case 1: { // coalesce if left & right chunks on overpopulated lists | |
| 8132 coalesce = _sp->coalOverPopulated(left) && | |
| 8133 _sp->coalOverPopulated(right); | |
| 8134 break; | |
| 8135 } | |
| 8136 case 2: { // coalesce if left chunk on overpopulated list (default) | |
| 8137 coalesce = _sp->coalOverPopulated(left); | |
| 8138 break; | |
| 8139 } | |
| 8140 case 3: { // coalesce if left OR right chunk on overpopulated list | |
| 8141 coalesce = _sp->coalOverPopulated(left) || | |
| 8142 _sp->coalOverPopulated(right); | |
| 8143 break; | |
| 8144 } | |
| 8145 case 4: { // always coalesce | |
| 8146 coalesce = true; | |
| 8147 break; | |
| 8148 } | |
| 8149 default: | |
| 8150 ShouldNotReachHere(); | |
| 8151 } | |
| 8152 | |
| 8153 // Should the current free range be coalesced? | |
| 8154 // If the chunk is in a free range and either we decided to coalesce above | |
| 8155 // or the chunk is near the large block at the end of the heap | |
| 8156 // (isNearLargestChunk() returns true), then coalesce this chunk. | |
| 8157 bool doCoalesce = inFreeRange() && | |
| 8158 (coalesce || _g->isNearLargestChunk((HeapWord*)fc)); | |
| 8159 if (doCoalesce) { | |
| 8160 // Coalesce the current free range on the left with the new | |
| 8161 // chunk on the right. If either is on a free list, | |
| 8162 // it must be removed from the list and stashed in the closure. | |
| 8163 if (freeRangeInFreeLists()) { | |
| 8164 FreeChunk* ffc = (FreeChunk*)freeFinger(); | |
| 8165 assert(ffc->size() == pointer_delta(addr, freeFinger()), | |
| 8166 "Size of free range is inconsistent with chunk size."); | |
| 8167 if (CMSTestInFreeList) { | |
| 8168 assert(_sp->verifyChunkInFreeLists(ffc), | |
| 8169 "Chunk is not in free lists"); | |
| 8170 } | |
| 8171 _sp->coalDeath(ffc->size()); | |
| 8172 _sp->removeFreeChunkFromFreeLists(ffc); | |
| 8173 set_freeRangeInFreeLists(false); | |
| 8174 } | |
| 8175 if (fcInFreeLists) { | |
| 8176 _sp->coalDeath(chunkSize); | |
| 8177 assert(fc->size() == chunkSize, | |
| 8178 "The chunk has the wrong size or is not in the free lists"); | |
| 8179 _sp->removeFreeChunkFromFreeLists(fc); | |
| 8180 } | |
| 8181 set_lastFreeRangeCoalesced(true); | |
| 8182 } else { // not in a free range and/or should not coalesce | |
| 8183 // Return the current free range and start a new one. | |
| 8184 if (inFreeRange()) { | |
| 8185 // In a free range but cannot coalesce with the right hand chunk. | |
| 8186 // Put the current free range into the free lists. | |
| 8187 flushCurFreeChunk(freeFinger(), | |
| 8188 pointer_delta(addr, freeFinger())); | |
| 8189 } | |
| 8190 // Set up for new free range. Pass along whether the right hand | |
| 8191 // chunk is in the free lists. | |
| 8192 initialize_free_range((HeapWord*)fc, fcInFreeLists); | |
| 8193 } | |
| 8194 } | |
| 8195 void SweepClosure::flushCurFreeChunk(HeapWord* chunk, size_t size) { | |
| 8196 assert(inFreeRange(), "Should only be called if currently in a free range."); | |
| 8197 assert(size > 0, | |
| 8198 "A zero sized chunk cannot be added to the free lists."); | |
| 8199 if (!freeRangeInFreeLists()) { | |
| 8200 if(CMSTestInFreeList) { | |
| 8201 FreeChunk* fc = (FreeChunk*) chunk; | |
| 8202 fc->setSize(size); | |
| 8203 assert(!_sp->verifyChunkInFreeLists(fc), | |
| 8204 "chunk should not be in free lists yet"); | |
| 8205 } | |
| 8206 if (CMSTraceSweeper) { | |
| 8207 gclog_or_tty->print_cr(" -- add free block 0x%x (%d) to free lists", | |
| 8208 chunk, size); | |
| 8209 } | |
| 8210 // A new free range is going to be starting. The current | |
| 8211 // free range has not been added to the free lists yet or | |
| 8212 // was removed so add it back. | |
| 8213 // If the current free range was coalesced, then the death | |
| 8214 // of the free range was recorded. Record a birth now. | |
| 8215 if (lastFreeRangeCoalesced()) { | |
| 8216 _sp->coalBirth(size); | |
| 8217 } | |
| 8218 _sp->addChunkAndRepairOffsetTable(chunk, size, | |
| 8219 lastFreeRangeCoalesced()); | |
| 8220 } | |
| 8221 set_inFreeRange(false); | |
| 8222 set_freeRangeInFreeLists(false); | |
| 8223 } | |
| 8224 | |
| 8225 // We take a break if we've been at this for a while, | |
| 8226 // so as to avoid monopolizing the locks involved. | |
| 8227 void SweepClosure::do_yield_work(HeapWord* addr) { | |
| 8228 // Return current free chunk being used for coalescing (if any) | |
| 8229 // to the appropriate freelist. After yielding, the next | |
| 8230 // free block encountered will start a coalescing range of | |
| 8231 // free blocks. If the next free block is adjacent to the | |
| 8232 // chunk just flushed, they will need to wait for the next | |
| 8233 // sweep to be coalesced. | |
| 8234 if (inFreeRange()) { | |
| 8235 flushCurFreeChunk(freeFinger(), pointer_delta(addr, freeFinger())); | |
| 8236 } | |
| 8237 | |
| 8238 // First give up the locks, then yield, then re-lock. | |
| 8239 // We should probably use a constructor/destructor idiom to | |
| 8240 // do this unlock/lock or modify the MutexUnlocker class to | |
| 8241 // serve our purpose. XXX | |
| 8242 assert_lock_strong(_bitMap->lock()); | |
| 8243 assert_lock_strong(_freelistLock); | |
| 8244 assert(ConcurrentMarkSweepThread::cms_thread_has_cms_token(), | |
| 8245 "CMS thread should hold CMS token"); | |
| 8246 _bitMap->lock()->unlock(); | |
| 8247 _freelistLock->unlock(); | |
| 8248 ConcurrentMarkSweepThread::desynchronize(true); | |
| 8249 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 8250 _collector->stopTimer(); | |
| 8251 GCPauseTimer p(_collector->size_policy()->concurrent_timer_ptr()); | |
| 8252 if (PrintCMSStatistics != 0) { | |
| 8253 _collector->incrementYields(); | |
| 8254 } | |
| 8255 _collector->icms_wait(); | |
| 8256 | |
| 8257 // See the comment in coordinator_yield() | |
| 8258 for (unsigned i = 0; i < CMSYieldSleepCount && | |
| 8259 ConcurrentMarkSweepThread::should_yield() && | |
| 8260 !CMSCollector::foregroundGCIsActive(); ++i) { | |
| 8261 os::sleep(Thread::current(), 1, false); | |
| 8262 ConcurrentMarkSweepThread::acknowledge_yield_request(); | |
| 8263 } | |
| 8264 | |
| 8265 ConcurrentMarkSweepThread::synchronize(true); | |
| 8266 _freelistLock->lock(); | |
| 8267 _bitMap->lock()->lock_without_safepoint_check(); | |
| 8268 _collector->startTimer(); | |
| 8269 } | |
| 8270 | |
| 8271 #ifndef PRODUCT | |
| 8272 // This is actually very useful in a product build if it can | |
| 8273 // be called from the debugger. Compile it into the product | |
| 8274 // as needed. | |
| 8275 bool debug_verifyChunkInFreeLists(FreeChunk* fc) { | |
| 8276 return debug_cms_space->verifyChunkInFreeLists(fc); | |
| 8277 } | |
| 8278 | |
| 8279 void SweepClosure::record_free_block_coalesced(FreeChunk* fc) const { | |
| 8280 if (CMSTraceSweeper) { | |
| 8281 gclog_or_tty->print("Sweep:coal_free_blk 0x%x (%d)\n", fc, fc->size()); | |
| 8282 } | |
| 8283 } | |
| 8284 #endif | |
| 8285 | |
| 8286 // CMSIsAliveClosure | |
| 8287 bool CMSIsAliveClosure::do_object_b(oop obj) { | |
| 8288 HeapWord* addr = (HeapWord*)obj; | |
| 8289 return addr != NULL && | |
| 8290 (!_span.contains(addr) || _bit_map->isMarked(addr)); | |
| 8291 } | |
| 8292 | |
| 8293 // CMSKeepAliveClosure: the serial version | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8294 void CMSKeepAliveClosure::do_oop(oop obj) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8295 HeapWord* addr = (HeapWord*)obj; |
| 0 | 8296 if (_span.contains(addr) && |
| 8297 !_bit_map->isMarked(addr)) { | |
| 8298 _bit_map->mark(addr); | |
| 8299 bool simulate_overflow = false; | |
| 8300 NOT_PRODUCT( | |
| 8301 if (CMSMarkStackOverflowALot && | |
| 8302 _collector->simulate_overflow()) { | |
| 8303 // simulate a stack overflow | |
| 8304 simulate_overflow = true; | |
| 8305 } | |
| 8306 ) | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8307 if (simulate_overflow || !_mark_stack->push(obj)) { |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8308 if (_concurrent_precleaning) { |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8309 // We dirty the overflown object and let the remark |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8310 // phase deal with it. |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8311 assert(_collector->overflow_list_is_empty(), "Error"); |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8312 // In the case of object arrays, we need to dirty all of |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8313 // the cards that the object spans. No locking or atomics |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8314 // are needed since no one else can be mutating the mod union |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8315 // table. |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8316 if (obj->is_objArray()) { |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8317 size_t sz = obj->size(); |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8318 HeapWord* end_card_addr = |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8319 (HeapWord*)round_to((intptr_t)(addr+sz), CardTableModRefBS::card_size); |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8320 MemRegion redirty_range = MemRegion(addr, end_card_addr); |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8321 assert(!redirty_range.is_empty(), "Arithmetical tautology"); |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8322 _collector->_modUnionTable.mark_range(redirty_range); |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8323 } else { |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8324 _collector->_modUnionTable.mark(addr); |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8325 } |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8326 _collector->_ser_kac_preclean_ovflw++; |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8327 } else { |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8328 _collector->push_on_overflow_list(obj); |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8329 _collector->_ser_kac_ovflw++; |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8330 } |
| 0 | 8331 } |
| 8332 } | |
| 8333 } | |
| 8334 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8335 void CMSKeepAliveClosure::do_oop(oop* p) { CMSKeepAliveClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8336 void CMSKeepAliveClosure::do_oop(narrowOop* p) { CMSKeepAliveClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8337 |
| 0 | 8338 // CMSParKeepAliveClosure: a parallel version of the above. |
| 8339 // The work queues are private to each closure (thread), | |
| 8340 // but (may be) available for stealing by other threads. | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8341 void CMSParKeepAliveClosure::do_oop(oop obj) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8342 HeapWord* addr = (HeapWord*)obj; |
| 0 | 8343 if (_span.contains(addr) && |
| 8344 !_bit_map->isMarked(addr)) { | |
| 8345 // In general, during recursive tracing, several threads | |
| 8346 // may be concurrently getting here; the first one to | |
| 8347 // "tag" it, claims it. | |
| 8348 if (_bit_map->par_mark(addr)) { | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8349 bool res = _work_queue->push(obj); |
| 0 | 8350 assert(res, "Low water mark should be much less than capacity"); |
| 8351 // Do a recursive trim in the hope that this will keep | |
| 8352 // stack usage lower, but leave some oops for potential stealers | |
| 8353 trim_queue(_low_water_mark); | |
| 8354 } // Else, another thread got there first | |
| 8355 } | |
| 8356 } | |
| 8357 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8358 void CMSParKeepAliveClosure::do_oop(oop* p) { CMSParKeepAliveClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8359 void CMSParKeepAliveClosure::do_oop(narrowOop* p) { CMSParKeepAliveClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8360 |
| 0 | 8361 void CMSParKeepAliveClosure::trim_queue(uint max) { |
| 8362 while (_work_queue->size() > max) { | |
| 8363 oop new_oop; | |
| 8364 if (_work_queue->pop_local(new_oop)) { | |
| 8365 assert(new_oop != NULL && new_oop->is_oop(), "Expected an oop"); | |
| 8366 assert(_bit_map->isMarked((HeapWord*)new_oop), | |
| 8367 "no white objects on this stack!"); | |
| 8368 assert(_span.contains((HeapWord*)new_oop), "Out of bounds oop"); | |
| 8369 // iterate over the oops in this oop, marking and pushing | |
| 8370 // the ones in CMS heap (i.e. in _span). | |
| 8371 new_oop->oop_iterate(&_mark_and_push); | |
| 8372 } | |
| 8373 } | |
| 8374 } | |
| 8375 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8376 void CMSInnerParMarkAndPushClosure::do_oop(oop obj) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8377 HeapWord* addr = (HeapWord*)obj; |
| 0 | 8378 if (_span.contains(addr) && |
| 8379 !_bit_map->isMarked(addr)) { | |
| 8380 if (_bit_map->par_mark(addr)) { | |
| 8381 bool simulate_overflow = false; | |
| 8382 NOT_PRODUCT( | |
| 8383 if (CMSMarkStackOverflowALot && | |
| 8384 _collector->par_simulate_overflow()) { | |
| 8385 // simulate a stack overflow | |
| 8386 simulate_overflow = true; | |
| 8387 } | |
| 8388 ) | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8389 if (simulate_overflow || !_work_queue->push(obj)) { |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8390 _collector->par_push_on_overflow_list(obj); |
| 0 | 8391 _collector->_par_kac_ovflw++; |
| 8392 } | |
| 8393 } // Else another thread got there already | |
| 8394 } | |
| 8395 } | |
| 8396 | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8397 void CMSInnerParMarkAndPushClosure::do_oop(oop* p) { CMSInnerParMarkAndPushClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8398 void CMSInnerParMarkAndPushClosure::do_oop(narrowOop* p) { CMSInnerParMarkAndPushClosure::do_oop_work(p); } |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8399 |
| 0 | 8400 ////////////////////////////////////////////////////////////////// |
| 8401 // CMSExpansionCause ///////////////////////////// | |
| 8402 ////////////////////////////////////////////////////////////////// | |
| 8403 const char* CMSExpansionCause::to_string(CMSExpansionCause::Cause cause) { | |
| 8404 switch (cause) { | |
| 8405 case _no_expansion: | |
| 8406 return "No expansion"; | |
| 8407 case _satisfy_free_ratio: | |
| 8408 return "Free ratio"; | |
| 8409 case _satisfy_promotion: | |
| 8410 return "Satisfy promotion"; | |
| 8411 case _satisfy_allocation: | |
| 8412 return "allocation"; | |
| 8413 case _allocate_par_lab: | |
| 8414 return "Par LAB"; | |
| 8415 case _allocate_par_spooling_space: | |
| 8416 return "Par Spooling Space"; | |
| 8417 case _adaptive_size_policy: | |
| 8418 return "Ergonomics"; | |
| 8419 default: | |
| 8420 return "unknown"; | |
| 8421 } | |
| 8422 } | |
| 8423 | |
| 8424 void CMSDrainMarkingStackClosure::do_void() { | |
| 8425 // the max number to take from overflow list at a time | |
| 8426 const size_t num = _mark_stack->capacity()/4; | |
|
452
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8427 assert(!_concurrent_precleaning || _collector->overflow_list_is_empty(), |
|
00b023ae2d78
6722113: CMS: Incorrect overflow handling during precleaning of Reference lists
ysr
parents:
360
diff
changeset
|
8428 "Overflow list should be NULL during concurrent phases"); |
| 0 | 8429 while (!_mark_stack->isEmpty() || |
| 8430 // if stack is empty, check the overflow list | |
| 8431 _collector->take_from_overflow_list(num, _mark_stack)) { | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8432 oop obj = _mark_stack->pop(); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8433 HeapWord* addr = (HeapWord*)obj; |
| 0 | 8434 assert(_span.contains(addr), "Should be within span"); |
| 8435 assert(_bit_map->isMarked(addr), "Should be marked"); | |
|
113
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8436 assert(obj->is_oop(), "Should be an oop"); |
|
ba764ed4b6f2
6420645: Create a vm that uses compressed oops for up to 32gb heapsizes
coleenp
parents:
94
diff
changeset
|
8437 obj->oop_iterate(_keep_alive); |
| 0 | 8438 } |
| 8439 } | |
| 8440 | |
| 8441 void CMSParDrainMarkingStackClosure::do_void() { | |
| 8442 // drain queue | |
| 8443 trim_queue(0); | |
| 8444 } | |
| 8445 | |
| 8446 // Trim our work_queue so its length is below max at return | |
| 8447 void CMSParDrainMarkingStackClosure::trim_queue(uint max) { | |
| 8448 while (_work_queue->size() > max) { | |
| 8449 oop new_oop; | |
| 8450 if (_work_queue->pop_local(new_oop)) { | |
| 8451 assert(new_oop->is_oop(), "Expected an oop"); | |
| 8452 assert(_bit_map->isMarked((HeapWord*)new_oop), | |
| 8453 "no white objects on this stack!"); | |
| 8454 assert(_span.contains((HeapWord*)new_oop), "Out of bounds oop"); | |
| 8455 // iterate over the oops in this oop, marking and pushing | |
| 8456 // the ones in CMS heap (i.e. in _span). | |
| 8457 new_oop->oop_iterate(&_mark_and_push); | |
| 8458 } | |
| 8459 } | |
| 8460 } | |
| 8461 | |
| 8462 //////////////////////////////////////////////////////////////////// | |
| 8463 // Support for Marking Stack Overflow list handling and related code | |
| 8464 //////////////////////////////////////////////////////////////////// | |
| 8465 // Much of the following code is similar in shape and spirit to the | |
| 8466 // code used in ParNewGC. We should try and share that code | |
| 8467 // as much as possible in the future. | |
| 8468 | |
| 8469 #ifndef PRODUCT | |
| 8470 // Debugging support for CMSStackOverflowALot | |
| 8471 | |
| 8472 // It's OK to call this multi-threaded; the worst thing | |
| 8473 // that can happen is that we'll get a bunch of closely | |
| 8474 // spaced simulated oveflows, but that's OK, in fact | |
| 8475 // probably good as it would exercise the overflow code | |
| 8476 // under contention. | |
| 8477 bool CMSCollector::simulate_overflow() { | |
| 8478 if (_overflow_counter-- <= 0) { // just being defensive | |
| 8479 _overflow_counter = CMSMarkStackOverflowInterval; | |
| 8480 return true; | |
| 8481 } else { | |
| 8482 return false; | |
| 8483 } | |
| 8484 } | |
| 8485 | |
| 8486 bool CMSCollector::par_simulate_overflow() { | |
| 8487 return simulate_overflow(); | |
| 8488 } | |
| 8489 #endif | |
| 8490 | |
| 8491 // Single-threaded | |
| 8492 bool CMSCollector::take_from_overflow_list(size_t num, CMSMarkStack* stack) { | |
| 8493 assert(stack->isEmpty(), "Expected precondition"); | |
| 8494 assert(stack->capacity() > num, "Shouldn't bite more than can chew"); | |
| 8495 size_t i = num; | |
| 8496 oop cur = _overflow_list; | |
| 8497 const markOop proto = markOopDesc::prototype(); | |
| 8498 NOT_PRODUCT(size_t n = 0;) | |
| 8499 for (oop next; i > 0 && cur != NULL; cur = next, i--) { | |
| 8500 next = oop(cur->mark()); | |
| 8501 cur->set_mark(proto); // until proven otherwise | |
| 8502 assert(cur->is_oop(), "Should be an oop"); | |
| 8503 bool res = stack->push(cur); | |
| 8504 assert(res, "Bit off more than can chew?"); | |
| 8505 NOT_PRODUCT(n++;) | |
| 8506 } | |
| 8507 _overflow_list = cur; | |
| 8508 #ifndef PRODUCT | |
| 8509 assert(_num_par_pushes >= n, "Too many pops?"); | |
| 8510 _num_par_pushes -=n; | |
| 8511 #endif | |
| 8512 return !stack->isEmpty(); | |
| 8513 } | |
| 8514 | |
| 8515 // Multi-threaded; use CAS to break off a prefix | |
| 8516 bool CMSCollector::par_take_from_overflow_list(size_t num, | |
| 8517 OopTaskQueue* work_q) { | |
| 8518 assert(work_q->size() == 0, "That's the current policy"); | |
| 8519 assert(num < work_q->max_elems(), "Can't bite more than we can chew"); | |
| 8520 if (_overflow_list == NULL) { | |
| 8521 return false; | |
| 8522 } | |
| 8523 // Grab the entire list; we'll put back a suffix | |
| 8524 oop prefix = (oop)Atomic::xchg_ptr(NULL, &_overflow_list); | |
| 8525 if (prefix == NULL) { // someone grabbed it before we did ... | |
| 8526 // ... we could spin for a short while, but for now we don't | |
| 8527 return false; | |
| 8528 } | |
| 8529 size_t i = num; | |
| 8530 oop cur = prefix; | |
| 8531 for (; i > 1 && cur->mark() != NULL; cur = oop(cur->mark()), i--); | |
| 8532 if (cur->mark() != NULL) { | |
| 8533 oop suffix_head = cur->mark(); // suffix will be put back on global list | |
| 8534 cur->set_mark(NULL); // break off suffix | |
| 8535 // Find tail of suffix so we can prepend suffix to global list | |
| 8536 for (cur = suffix_head; cur->mark() != NULL; cur = (oop)(cur->mark())); | |
| 8537 oop suffix_tail = cur; | |
| 8538 assert(suffix_tail != NULL && suffix_tail->mark() == NULL, | |
| 8539 "Tautology"); | |
| 8540 oop observed_overflow_list = _overflow_list; | |
| 8541 do { | |
| 8542 cur = observed_overflow_list; | |
| 8543 suffix_tail->set_mark(markOop(cur)); | |
| 8544 observed_overflow_list = | |
| 8545 (oop) Atomic::cmpxchg_ptr(suffix_head, &_overflow_list, cur); | |
| 8546 } while (cur != observed_overflow_list); | |
| 8547 } | |
| 8548 | |
| 8549 // Push the prefix elements on work_q | |
| 8550 assert(prefix != NULL, "control point invariant"); | |
| 8551 const markOop proto = markOopDesc::prototype(); | |
| 8552 oop next; | |
| 8553 NOT_PRODUCT(size_t n = 0;) | |
| 8554 for (cur = prefix; cur != NULL; cur = next) { | |
| 8555 next = oop(cur->mark()); | |
| 8556 cur->set_mark(proto); // until proven otherwise | |
| 8557 assert(cur->is_oop(), "Should be an oop"); | |
| 8558 bool res = work_q->push(cur); | |
| 8559 assert(res, "Bit off more than we can chew?"); | |
| 8560 NOT_PRODUCT(n++;) | |
| 8561 } | |
| 8562 #ifndef PRODUCT | |
| 8563 assert(_num_par_pushes >= n, "Too many pops?"); | |
| 8564 Atomic::add_ptr(-(intptr_t)n, &_num_par_pushes); | |
| 8565 #endif | |
| 8566 return true; | |
| 8567 } | |
| 8568 | |
| 8569 // Single-threaded | |
| 8570 void CMSCollector::push_on_overflow_list(oop p) { | |
| 8571 NOT_PRODUCT(_num_par_pushes++;) | |
| 8572 assert(p->is_oop(), "Not an oop"); | |
| 8573 preserve_mark_if_necessary(p); | |
| 8574 p->set_mark((markOop)_overflow_list); | |
| 8575 _overflow_list = p; | |
| 8576 } | |
| 8577 | |
| 8578 // Multi-threaded; use CAS to prepend to overflow list | |
| 8579 void CMSCollector::par_push_on_overflow_list(oop p) { | |
| 8580 NOT_PRODUCT(Atomic::inc_ptr(&_num_par_pushes);) | |
| 8581 assert(p->is_oop(), "Not an oop"); | |
| 8582 par_preserve_mark_if_necessary(p); | |
| 8583 oop observed_overflow_list = _overflow_list; | |
| 8584 oop cur_overflow_list; | |
| 8585 do { | |
| 8586 cur_overflow_list = observed_overflow_list; | |
| 8587 p->set_mark(markOop(cur_overflow_list)); | |
| 8588 observed_overflow_list = | |
| 8589 (oop) Atomic::cmpxchg_ptr(p, &_overflow_list, cur_overflow_list); | |
| 8590 } while (cur_overflow_list != observed_overflow_list); | |
| 8591 } | |
| 8592 | |
| 8593 // Single threaded | |
| 8594 // General Note on GrowableArray: pushes may silently fail | |
| 8595 // because we are (temporarily) out of C-heap for expanding | |
| 8596 // the stack. The problem is quite ubiquitous and affects | |
| 8597 // a lot of code in the JVM. The prudent thing for GrowableArray | |
| 8598 // to do (for now) is to exit with an error. However, that may | |
| 8599 // be too draconian in some cases because the caller may be | |
| 8600 // able to recover without much harm. For suych cases, we | |
| 8601 // should probably introduce a "soft_push" method which returns | |
| 8602 // an indication of success or failure with the assumption that | |
| 8603 // the caller may be able to recover from a failure; code in | |
| 8604 // the VM can then be changed, incrementally, to deal with such | |
| 8605 // failures where possible, thus, incrementally hardening the VM | |
| 8606 // in such low resource situations. | |
| 8607 void CMSCollector::preserve_mark_work(oop p, markOop m) { | |
| 8608 int PreserveMarkStackSize = 128; | |
| 8609 | |
| 8610 if (_preserved_oop_stack == NULL) { | |
| 8611 assert(_preserved_mark_stack == NULL, | |
| 8612 "bijection with preserved_oop_stack"); | |
| 8613 // Allocate the stacks | |
| 8614 _preserved_oop_stack = new (ResourceObj::C_HEAP) | |
| 8615 GrowableArray<oop>(PreserveMarkStackSize, true); | |
| 8616 _preserved_mark_stack = new (ResourceObj::C_HEAP) | |
| 8617 GrowableArray<markOop>(PreserveMarkStackSize, true); | |
| 8618 if (_preserved_oop_stack == NULL || _preserved_mark_stack == NULL) { | |
| 8619 vm_exit_out_of_memory(2* PreserveMarkStackSize * sizeof(oop) /* punt */, | |
| 8620 "Preserved Mark/Oop Stack for CMS (C-heap)"); | |
| 8621 } | |
| 8622 } | |
| 8623 _preserved_oop_stack->push(p); | |
| 8624 _preserved_mark_stack->push(m); | |
| 8625 assert(m == p->mark(), "Mark word changed"); | |
| 8626 assert(_preserved_oop_stack->length() == _preserved_mark_stack->length(), | |
| 8627 "bijection"); | |
| 8628 } | |
| 8629 | |
| 8630 // Single threaded | |
| 8631 void CMSCollector::preserve_mark_if_necessary(oop p) { | |
| 8632 markOop m = p->mark(); | |
| 8633 if (m->must_be_preserved(p)) { | |
| 8634 preserve_mark_work(p, m); | |
| 8635 } | |
| 8636 } | |
| 8637 | |
| 8638 void CMSCollector::par_preserve_mark_if_necessary(oop p) { | |
| 8639 markOop m = p->mark(); | |
| 8640 if (m->must_be_preserved(p)) { | |
| 8641 MutexLockerEx x(ParGCRareEvent_lock, Mutex::_no_safepoint_check_flag); | |
| 8642 // Even though we read the mark word without holding | |
| 8643 // the lock, we are assured that it will not change | |
| 8644 // because we "own" this oop, so no other thread can | |
| 8645 // be trying to push it on the overflow list; see | |
| 8646 // the assertion in preserve_mark_work() that checks | |
| 8647 // that m == p->mark(). | |
| 8648 preserve_mark_work(p, m); | |
| 8649 } | |
| 8650 } | |
| 8651 | |
| 8652 // We should be able to do this multi-threaded, | |
| 8653 // a chunk of stack being a task (this is | |
| 8654 // correct because each oop only ever appears | |
| 8655 // once in the overflow list. However, it's | |
| 8656 // not very easy to completely overlap this with | |
| 8657 // other operations, so will generally not be done | |
| 8658 // until all work's been completed. Because we | |
| 8659 // expect the preserved oop stack (set) to be small, | |
| 8660 // it's probably fine to do this single-threaded. | |
| 8661 // We can explore cleverer concurrent/overlapped/parallel | |
| 8662 // processing of preserved marks if we feel the | |
| 8663 // need for this in the future. Stack overflow should | |
| 8664 // be so rare in practice and, when it happens, its | |
| 8665 // effect on performance so great that this will | |
| 8666 // likely just be in the noise anyway. | |
| 8667 void CMSCollector::restore_preserved_marks_if_any() { | |
| 8668 if (_preserved_oop_stack == NULL) { | |
| 8669 assert(_preserved_mark_stack == NULL, | |
| 8670 "bijection with preserved_oop_stack"); | |
| 8671 return; | |
| 8672 } | |
| 8673 | |
| 8674 assert(SafepointSynchronize::is_at_safepoint(), | |
| 8675 "world should be stopped"); | |
| 8676 assert(Thread::current()->is_ConcurrentGC_thread() || | |
| 8677 Thread::current()->is_VM_thread(), | |
| 8678 "should be single-threaded"); | |
| 8679 | |
| 8680 int length = _preserved_oop_stack->length(); | |
| 8681 assert(_preserved_mark_stack->length() == length, "bijection"); | |
| 8682 for (int i = 0; i < length; i++) { | |
| 8683 oop p = _preserved_oop_stack->at(i); | |
| 8684 assert(p->is_oop(), "Should be an oop"); | |
| 8685 assert(_span.contains(p), "oop should be in _span"); | |
| 8686 assert(p->mark() == markOopDesc::prototype(), | |
| 8687 "Set when taken from overflow list"); | |
| 8688 markOop m = _preserved_mark_stack->at(i); | |
| 8689 p->set_mark(m); | |
| 8690 } | |
| 8691 _preserved_mark_stack->clear(); | |
| 8692 _preserved_oop_stack->clear(); | |
| 8693 assert(_preserved_mark_stack->is_empty() && | |
| 8694 _preserved_oop_stack->is_empty(), | |
| 8695 "stacks were cleared above"); | |
| 8696 } | |
| 8697 | |
| 8698 #ifndef PRODUCT | |
| 8699 bool CMSCollector::no_preserved_marks() const { | |
| 8700 return ( ( _preserved_mark_stack == NULL | |
| 8701 && _preserved_oop_stack == NULL) | |
| 8702 || ( _preserved_mark_stack->is_empty() | |
| 8703 && _preserved_oop_stack->is_empty())); | |
| 8704 } | |
| 8705 #endif | |
| 8706 | |
| 8707 CMSAdaptiveSizePolicy* ASConcurrentMarkSweepGeneration::cms_size_policy() const | |
| 8708 { | |
| 8709 GenCollectedHeap* gch = (GenCollectedHeap*) GenCollectedHeap::heap(); | |
| 8710 CMSAdaptiveSizePolicy* size_policy = | |
| 8711 (CMSAdaptiveSizePolicy*) gch->gen_policy()->size_policy(); | |
| 8712 assert(size_policy->is_gc_cms_adaptive_size_policy(), | |
| 8713 "Wrong type for size policy"); | |
| 8714 return size_policy; | |
| 8715 } | |
| 8716 | |
| 8717 void ASConcurrentMarkSweepGeneration::resize(size_t cur_promo_size, | |
| 8718 size_t desired_promo_size) { | |
| 8719 if (cur_promo_size < desired_promo_size) { | |
| 8720 size_t expand_bytes = desired_promo_size - cur_promo_size; | |
| 8721 if (PrintAdaptiveSizePolicy && Verbose) { | |
| 8722 gclog_or_tty->print_cr(" ASConcurrentMarkSweepGeneration::resize " | |
| 8723 "Expanding tenured generation by " SIZE_FORMAT " (bytes)", | |
| 8724 expand_bytes); | |
| 8725 } | |
| 8726 expand(expand_bytes, | |
| 8727 MinHeapDeltaBytes, | |
| 8728 CMSExpansionCause::_adaptive_size_policy); | |
| 8729 } else if (desired_promo_size < cur_promo_size) { | |
| 8730 size_t shrink_bytes = cur_promo_size - desired_promo_size; | |
| 8731 if (PrintAdaptiveSizePolicy && Verbose) { | |
| 8732 gclog_or_tty->print_cr(" ASConcurrentMarkSweepGeneration::resize " | |
| 8733 "Shrinking tenured generation by " SIZE_FORMAT " (bytes)", | |
| 8734 shrink_bytes); | |
| 8735 } | |
| 8736 shrink(shrink_bytes); | |
| 8737 } | |
| 8738 } | |
| 8739 | |
| 8740 CMSGCAdaptivePolicyCounters* ASConcurrentMarkSweepGeneration::gc_adaptive_policy_counters() { | |
| 8741 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 8742 CMSGCAdaptivePolicyCounters* counters = | |
| 8743 (CMSGCAdaptivePolicyCounters*) gch->collector_policy()->counters(); | |
| 8744 assert(counters->kind() == GCPolicyCounters::CMSGCAdaptivePolicyCountersKind, | |
| 8745 "Wrong kind of counters"); | |
| 8746 return counters; | |
| 8747 } | |
| 8748 | |
| 8749 | |
| 8750 void ASConcurrentMarkSweepGeneration::update_counters() { | |
| 8751 if (UsePerfData) { | |
| 8752 _space_counters->update_all(); | |
| 8753 _gen_counters->update_all(); | |
| 8754 CMSGCAdaptivePolicyCounters* counters = gc_adaptive_policy_counters(); | |
| 8755 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 8756 CMSGCStats* gc_stats_l = (CMSGCStats*) gc_stats(); | |
| 8757 assert(gc_stats_l->kind() == GCStats::CMSGCStatsKind, | |
| 8758 "Wrong gc statistics type"); | |
| 8759 counters->update_counters(gc_stats_l); | |
| 8760 } | |
| 8761 } | |
| 8762 | |
| 8763 void ASConcurrentMarkSweepGeneration::update_counters(size_t used) { | |
| 8764 if (UsePerfData) { | |
| 8765 _space_counters->update_used(used); | |
| 8766 _space_counters->update_capacity(); | |
| 8767 _gen_counters->update_all(); | |
| 8768 | |
| 8769 CMSGCAdaptivePolicyCounters* counters = gc_adaptive_policy_counters(); | |
| 8770 GenCollectedHeap* gch = GenCollectedHeap::heap(); | |
| 8771 CMSGCStats* gc_stats_l = (CMSGCStats*) gc_stats(); | |
| 8772 assert(gc_stats_l->kind() == GCStats::CMSGCStatsKind, | |
| 8773 "Wrong gc statistics type"); | |
| 8774 counters->update_counters(gc_stats_l); | |
| 8775 } | |
| 8776 } | |
| 8777 | |
| 8778 // The desired expansion delta is computed so that: | |
| 8779 // . desired free percentage or greater is used | |
| 8780 void ASConcurrentMarkSweepGeneration::compute_new_size() { | |
| 8781 assert_locked_or_safepoint(Heap_lock); | |
| 8782 | |
| 8783 GenCollectedHeap* gch = (GenCollectedHeap*) GenCollectedHeap::heap(); | |
| 8784 | |
| 8785 // If incremental collection failed, we just want to expand | |
| 8786 // to the limit. | |
| 8787 if (incremental_collection_failed()) { | |
| 8788 clear_incremental_collection_failed(); | |
| 8789 grow_to_reserved(); | |
| 8790 return; | |
| 8791 } | |
| 8792 | |
| 8793 assert(UseAdaptiveSizePolicy, "Should be using adaptive sizing"); | |
| 8794 | |
| 8795 assert(gch->kind() == CollectedHeap::GenCollectedHeap, | |
| 8796 "Wrong type of heap"); | |
| 8797 int prev_level = level() - 1; | |
| 8798 assert(prev_level >= 0, "The cms generation is the lowest generation"); | |
| 8799 Generation* prev_gen = gch->get_gen(prev_level); | |
| 8800 assert(prev_gen->kind() == Generation::ASParNew, | |
| 8801 "Wrong type of young generation"); | |
| 8802 ParNewGeneration* younger_gen = (ParNewGeneration*) prev_gen; | |
| 8803 size_t cur_eden = younger_gen->eden()->capacity(); | |
| 8804 CMSAdaptiveSizePolicy* size_policy = cms_size_policy(); | |
| 8805 size_t cur_promo = free(); | |
| 8806 size_policy->compute_tenured_generation_free_space(cur_promo, | |
| 8807 max_available(), | |
| 8808 cur_eden); | |
| 8809 resize(cur_promo, size_policy->promo_size()); | |
| 8810 | |
| 8811 // Record the new size of the space in the cms generation | |
| 8812 // that is available for promotions. This is temporary. | |
| 8813 // It should be the desired promo size. | |
| 8814 size_policy->avg_cms_promo()->sample(free()); | |
| 8815 size_policy->avg_old_live()->sample(used()); | |
| 8816 | |
| 8817 if (UsePerfData) { | |
| 8818 CMSGCAdaptivePolicyCounters* counters = gc_adaptive_policy_counters(); | |
| 8819 counters->update_cms_capacity_counter(capacity()); | |
| 8820 } | |
| 8821 } | |
| 8822 | |
| 8823 void ASConcurrentMarkSweepGeneration::shrink_by(size_t desired_bytes) { | |
| 8824 assert_locked_or_safepoint(Heap_lock); | |
| 8825 assert_lock_strong(freelistLock()); | |
| 8826 HeapWord* old_end = _cmsSpace->end(); | |
| 8827 HeapWord* unallocated_start = _cmsSpace->unallocated_block(); | |
| 8828 assert(old_end >= unallocated_start, "Miscalculation of unallocated_start"); | |
| 8829 FreeChunk* chunk_at_end = find_chunk_at_end(); | |
| 8830 if (chunk_at_end == NULL) { | |
| 8831 // No room to shrink | |
| 8832 if (PrintGCDetails && Verbose) { | |
| 8833 gclog_or_tty->print_cr("No room to shrink: old_end " | |
| 8834 PTR_FORMAT " unallocated_start " PTR_FORMAT | |
| 8835 " chunk_at_end " PTR_FORMAT, | |
| 8836 old_end, unallocated_start, chunk_at_end); | |
| 8837 } | |
| 8838 return; | |
| 8839 } else { | |
| 8840 | |
| 8841 // Find the chunk at the end of the space and determine | |
| 8842 // how much it can be shrunk. | |
| 8843 size_t shrinkable_size_in_bytes = chunk_at_end->size(); | |
| 8844 size_t aligned_shrinkable_size_in_bytes = | |
| 8845 align_size_down(shrinkable_size_in_bytes, os::vm_page_size()); | |
| 8846 assert(unallocated_start <= chunk_at_end->end(), | |
| 8847 "Inconsistent chunk at end of space"); | |
| 8848 size_t bytes = MIN2(desired_bytes, aligned_shrinkable_size_in_bytes); | |
| 8849 size_t word_size_before = heap_word_size(_virtual_space.committed_size()); | |
| 8850 | |
| 8851 // Shrink the underlying space | |
| 8852 _virtual_space.shrink_by(bytes); | |
| 8853 if (PrintGCDetails && Verbose) { | |
| 8854 gclog_or_tty->print_cr("ConcurrentMarkSweepGeneration::shrink_by:" | |
| 8855 " desired_bytes " SIZE_FORMAT | |
| 8856 " shrinkable_size_in_bytes " SIZE_FORMAT | |
| 8857 " aligned_shrinkable_size_in_bytes " SIZE_FORMAT | |
| 8858 " bytes " SIZE_FORMAT, | |
| 8859 desired_bytes, shrinkable_size_in_bytes, | |
| 8860 aligned_shrinkable_size_in_bytes, bytes); | |
| 8861 gclog_or_tty->print_cr(" old_end " SIZE_FORMAT | |
| 8862 " unallocated_start " SIZE_FORMAT, | |
| 8863 old_end, unallocated_start); | |
| 8864 } | |
| 8865 | |
| 8866 // If the space did shrink (shrinking is not guaranteed), | |
| 8867 // shrink the chunk at the end by the appropriate amount. | |
| 8868 if (((HeapWord*)_virtual_space.high()) < old_end) { | |
| 8869 size_t new_word_size = | |
| 8870 heap_word_size(_virtual_space.committed_size()); | |
| 8871 | |
| 8872 // Have to remove the chunk from the dictionary because it is changing | |
| 8873 // size and might be someplace elsewhere in the dictionary. | |
| 8874 | |
| 8875 // Get the chunk at end, shrink it, and put it | |
| 8876 // back. | |
| 8877 _cmsSpace->removeChunkFromDictionary(chunk_at_end); | |
| 8878 size_t word_size_change = word_size_before - new_word_size; | |
| 8879 size_t chunk_at_end_old_size = chunk_at_end->size(); | |
| 8880 assert(chunk_at_end_old_size >= word_size_change, | |
| 8881 "Shrink is too large"); | |
| 8882 chunk_at_end->setSize(chunk_at_end_old_size - | |
| 8883 word_size_change); | |
| 8884 _cmsSpace->freed((HeapWord*) chunk_at_end->end(), | |
| 8885 word_size_change); | |
| 8886 | |
| 8887 _cmsSpace->returnChunkToDictionary(chunk_at_end); | |
| 8888 | |
| 8889 MemRegion mr(_cmsSpace->bottom(), new_word_size); | |
| 8890 _bts->resize(new_word_size); // resize the block offset shared array | |
| 8891 Universe::heap()->barrier_set()->resize_covered_region(mr); | |
| 8892 _cmsSpace->assert_locked(); | |
| 8893 _cmsSpace->set_end((HeapWord*)_virtual_space.high()); | |
| 8894 | |
| 8895 NOT_PRODUCT(_cmsSpace->dictionary()->verify()); | |
| 8896 | |
| 8897 // update the space and generation capacity counters | |
| 8898 if (UsePerfData) { | |
| 8899 _space_counters->update_capacity(); | |
| 8900 _gen_counters->update_all(); | |
| 8901 } | |
| 8902 | |
| 8903 if (Verbose && PrintGCDetails) { | |
| 8904 size_t new_mem_size = _virtual_space.committed_size(); | |
| 8905 size_t old_mem_size = new_mem_size + bytes; | |
| 8906 gclog_or_tty->print_cr("Shrinking %s from %ldK by %ldK to %ldK", | |
| 8907 name(), old_mem_size/K, bytes/K, new_mem_size/K); | |
| 8908 } | |
| 8909 } | |
| 8910 | |
| 8911 assert(_cmsSpace->unallocated_block() <= _cmsSpace->end(), | |
| 8912 "Inconsistency at end of space"); | |
| 8913 assert(chunk_at_end->end() == _cmsSpace->end(), | |
| 8914 "Shrinking is inconsistent"); | |
| 8915 return; | |
| 8916 } | |
| 8917 } | |
| 8918 | |
| 8919 // Transfer some number of overflown objects to usual marking | |
| 8920 // stack. Return true if some objects were transferred. | |
| 8921 bool MarkRefsIntoAndScanClosure::take_from_overflow_list() { | |
| 8922 size_t num = MIN2((size_t)_mark_stack->capacity()/4, | |
| 8923 (size_t)ParGCDesiredObjsFromOverflowList); | |
| 8924 | |
| 8925 bool res = _collector->take_from_overflow_list(num, _mark_stack); | |
| 8926 assert(_collector->overflow_list_is_empty() || res, | |
| 8927 "If list is not empty, we should have taken something"); | |
| 8928 assert(!res || !_mark_stack->isEmpty(), | |
| 8929 "If we took something, it should now be on our stack"); | |
| 8930 return res; | |
| 8931 } | |
| 8932 | |
| 8933 size_t MarkDeadObjectsClosure::do_blk(HeapWord* addr) { | |
| 8934 size_t res = _sp->block_size_no_stall(addr, _collector); | |
| 8935 assert(res != 0, "Should always be able to compute a size"); | |
| 8936 if (_sp->block_is_obj(addr)) { | |
| 8937 if (_live_bit_map->isMarked(addr)) { | |
| 8938 // It can't have been dead in a previous cycle | |
| 8939 guarantee(!_dead_bit_map->isMarked(addr), "No resurrection!"); | |
| 8940 } else { | |
| 8941 _dead_bit_map->mark(addr); // mark the dead object | |
| 8942 } | |
| 8943 } | |
| 8944 return res; | |
| 8945 } |