HBasecompaction有什么用
这篇文章主要为大家展示了“HBase compaction有什么用”,内容简而易懂,条理清晰,希望能够帮助大家解决疑惑,下面让小编带领大家一起研究并学习一下“HBase compaction有什么用”这篇文章吧。
创新互联公司坚持“要么做到,要么别承诺”的工作理念,服务领域包括:网站建设、成都网站制作、企业官网、英文网站、手机端网站、网站推广等服务,满足客户于互联网时代的长宁网站设计、移动媒体设计的需求,帮助企业找到有效的互联网解决方案。努力成为您成熟可靠的网络建设合作伙伴!
compaction是将多个HFile合并为一个HFile操作,
进行compaction有如下几个作用:
减少HFile文件的个数,
HFile减少可能提高读取性能,
清除过期和删除数据。
compaction有二种类型:minor 和 major
1)Minor操作只用来做部分文件的合并操作以及包括minVersion=0并且设置ttl的过期版本清理,不做任何删除数据、多版本数据的清理工作。
2)Major操作是对Region下的HStore下的所有StoreFile执行合并操作,会做删除数据,多版本数据清理工作,最终的结果是整理合并出一个文件。
HRegionServer在启动的时候,会启动compactionChecker线程,compactionChecker会检测region是否需要compaction。
主要执行的逻辑如下:
protected void chore() { for (HRegion r : this.instance.onlineRegions.values()) { if (r == null) continue; for (Store s : r.getStores().values()) { try { long multiplier = s.getCompactionCheckMultiplier(); assert multiplier > 0; if (iteration % multiplier != 0) continue; if (s.needsCompaction()) { // Queue a compaction. Will recognize if major is needed. this.instance.compactSplitThread.requestSystemCompaction(r, s, getName() + " requests compaction"); } else if (s.isMajorCompaction()) { if (majorCompactPriority == DEFAULT_PRIORITY || majorCompactPriority > r.getCompactPriority()) { this.instance.compactSplitThread.requestCompaction(r, s, getName() + " requests major compaction; use default priority", null); } else { this.instance.compactSplitThread.requestCompaction(r, s, getName() + " requests major compaction; use configured priority", this.majorCompactPriority, null); } } } catch (IOException e) { LOG.warn("Failed major compaction check on " + r, e); } } } iteration = (iteration == Long.MAX_VALUE) ? 0 : (iteration + 1); } }
如代码,遍历onlineRegions,获取每个region的Store进行判断,其中needsCompaction逻辑如下:
public boolean needsCompaction(final CollectionstoreFiles, final List filesCompacting) { int numCandidates = storeFiles.size() - filesCompacting.size(); return numCandidates >= comConf.getMinFilesToCompact(); }
minFilesToCompact由hbase.hstore.compaction.min(老版本是:hbase.hstore.compactionThreshold)控制,默认值为3,即store下面的storeFiles数量 减去 正在compaction的数量 >=3是,需要做compaction。
needsCompaction为true后,会调用compactSplitThread.requestSystemCompaction方法,发送compaction请求,具体内容会在compactSplitThread线程里分析。
needsCompaction为false后,会判断是否为isMajorCompaction,具体逻辑如下:
/* * @param filesToCompact Files to compact. Can be null. * @return True if we should run a major compaction. */ public boolean isMajorCompaction(final CollectionfilesToCompact) throws IOException { boolean result = false; long mcTime = getNextMajorCompactTime(filesToCompact); if (filesToCompact == null || filesToCompact.isEmpty() || mcTime == 0) { return result; } // TODO: Use better method for determining stamp of last major (HBASE-2990) long lowTimestamp = StoreUtils.getLowestTimestamp(filesToCompact); long now = System.currentTimeMillis(); if (lowTimestamp > 0l && lowTimestamp < (now - mcTime)) { // Major compaction time has elapsed. long cfTtl = this.storeConfigInfo.getStoreFileTtl(); if (filesToCompact.size() == 1) { // Single file StoreFile sf = filesToCompact.iterator().next(); Long minTimestamp = sf.getMinimumTimestamp(); long oldest = (minTimestamp == null) ? Long.MIN_VALUE : now - minTimestamp.longValue(); if (sf.isMajorCompaction() && (cfTtl == HConstants.FOREVER || oldest < cfTtl)) { if (LOG.isDebugEnabled()) { LOG.debug("Skipping major compaction of " + this + " because one (major) compacted file only and oldestTime " + oldest + "ms is < ttl=" + cfTtl); } } else if (cfTtl != HConstants.FOREVER && oldest > cfTtl) { LOG.debug("Major compaction triggered on store " + this + ", because keyvalues outdated; time since last major compaction " + (now - lowTimestamp) + "ms"); result = true; } } else { if (LOG.isDebugEnabled()) { LOG.debug("Major compaction triggered on store " + this + "; time since last major compaction " + (now - lowTimestamp) + "ms"); } result = true; } } return result; }
以上是“HBase compaction有什么用”这篇文章的所有内容,感谢各位的阅读!相信大家都有了一定的了解,希望分享的内容对大家有所帮助,如果还想学习更多知识,欢迎关注创新互联行业资讯频道!
文章标题:HBasecompaction有什么用
文章起源:http://myzitong.com/article/pdoipj.html