Apps In The Open

30 May 2017

Merge Empty HBase Regions

Sometimes you see HBase tables with ascending row keys (eg an increasing sequence, or a date stamp) for storing time series data, and the table is setup to have a TTL set on the rows to age out old data. With a setup like this, the new data will always be added to the last region in the table, which will split when it reaches a certain size leading to a new last region.

As the older rows age out, you will end up with regions that are empty, but HBase will not automatically remove them, potentially leaving you with tables with 100's or even 1000's of empty regions.

The only way to remove these regions is to merge adjacent regions over and over until you get the number down to a manageable level, but with many regions, this process can take a long time.

The following script can make this process much easier. What it does is:

  • Get a list of all regions in the table
  • Then get a list of all non-empty regions, ie those with a store files of over 1MB
  • Remove the non empty regions from the first list
  • Make one pass over the empty regions, merging adjacent pairs so that the number of empty regions should be halved on each run

After completing one run of the script, simply run it again to make another pass over the empty regions. It would be possible to make this script loop until all empty regions have been merged, but the following works and suits my needs OK.

To run the script, you should first run it in test mode so it will print out what it plans to do:

hbase org.jruby.Main merge_empty_regions.rb namespace.tablename

When you are happy the output looks OK, add the 'merge' option to actually do the merge:

hbase org.jruby.Main merge_empty_regions.rb namespace.tablename merge

To test the script out, you can create a table with a given number of empty regions as follows:

hbase(main):002:0> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'}
0 row(s) in 2.4890 seconds

hbase(main):004:0> put 't1', 'key1', 'f1:c1', 'val'
0 row(s) in 0.1420 seconds

hbase(main):006:0> flush 't1'
0 row(s) in 0.3560 seconds

Now we have a table with 15 regions, 14 of which are empty, but all are under 1MB and so will so run the merge script in test mode:

hbase org.jruby.Main merge_empty_regions.rb t1
...
Total Table Regions: 15
Total non empty regions: 0
Total regions to consider for Merge: 15
3cc4e2dc6fb5878aaf5eb7588ae367d3 is adjacent to d316a9284372859a94d97c3532c1bd85
3df36d4f3ffcec81e4119fb2cfc23401 is adjacent to 7fdfabff5e7391e51afd91a3a7bd3196
5d625b3d5c9761c8280c6d6a802e443a is adjacent to 21b363273911448aba8df7a1e9b4a13d
f1da23259ced30138aba15cf6fed5406 is adjacent to d9e5c55fe2990679ca2b9f0078af65f1
a42cc7197467665eff4ef4eadc5299ff is adjacent to 325b36c018a34c6ee524f99c0624d1d0
dfc021c2aedc58b3497529bc625ea2b1 is adjacent to 030d525ef0e9ae77180b2b8b9325f36c
b3cf0198943c7562bd05622bbb1227e2 is adjacent to ff556c5c60ff71ceb81c9b68133fa2cc

Finally merge the regions:

hbase org.jruby.Main merge_empty_regions.rb t1 merge
...
Total Table Regions: 15
Total non empty regions: 0
Total regions to consider for Merge: 15
3cc4e2dc6fb5878aaf5eb7588ae367d3 is adjacent to d316a9284372859a94d97c3532c1bd85
Successfully Merged 3cc4e2dc6fb5878aaf5eb7588ae367d3 with d316a9284372859a94d97c3532c1bd85
3df36d4f3ffcec81e4119fb2cfc23401 is adjacent to 7fdfabff5e7391e51afd91a3a7bd3196
Successfully Merged 3df36d4f3ffcec81e4119fb2cfc23401 with 7fdfabff5e7391e51afd91a3a7bd3196
5d625b3d5c9761c8280c6d6a802e443a is adjacent to 21b363273911448aba8df7a1e9b4a13d
Successfully Merged 5d625b3d5c9761c8280c6d6a802e443a with 21b363273911448aba8df7a1e9b4a13d
f1da23259ced30138aba15cf6fed5406 is adjacent to d9e5c55fe2990679ca2b9f0078af65f1
Successfully Merged f1da23259ced30138aba15cf6fed5406 with d9e5c55fe2990679ca2b9f0078af65f1
a42cc7197467665eff4ef4eadc5299ff is adjacent to 325b36c018a34c6ee524f99c0624d1d0
Successfully Merged a42cc7197467665eff4ef4eadc5299ff with 325b36c018a34c6ee524f99c0624d1d0
dfc021c2aedc58b3497529bc625ea2b1 is adjacent to 030d525ef0e9ae77180b2b8b9325f36c
Successfully Merged dfc021c2aedc58b3497529bc625ea2b1 with 030d525ef0e9ae77180b2b8b9325f36c
b3cf0198943c7562bd05622bbb1227e2 is adjacent to ff556c5c60ff71ceb81c9b68133fa2cc
Successfully Merged b3cf0198943c7562bd05622bbb1227e2 with ff556c5c60ff71ceb81c9b68133fa2cc

The script to perform this merge is below:

# Test Mode:
#
# hbase org.jruby.Main merge_empty_regions.rb namespace.tablename
#
# Non Test - ie actually do the merge:
#
# hbase org.jruby.Main merge_empty_regions.rb namespace.tablename merge
#
# Note: Please replace namespace.tablename with your namespace and table, eg NS1.MyTable. This value is case sensitive.

require 'digest'
require 'java'
java_import org.apache.hadoop.hbase.HBaseConfiguration
java_import org.apache.hadoop.hbase.client.HBaseAdmin
java_import org.apache.hadoop.hbase.TableName
java_import org.apache.hadoop.hbase.HRegionInfo;
java_import org.apache.hadoop.hbase.client.Connection
java_import org.apache.hadoop.hbase.client.ConnectionFactory
java_import org.apache.hadoop.hbase.client.Table
java_import org.apache.hadoop.hbase.util.Bytes

def list_non_empty_regions(admin, table)
  cluster_status = admin.getClusterStatus()
  master = cluster_status.getMaster()
  non_empty = []
  cluster_status.getServers.each do |s|
    cluster_status.getLoad(s).getRegionsLoad.each do |r|
      # getRegionsLoad returns an array of arrays, where each array
      # is 2 elements

      # Filter out any regions that don't match the requested
      # tablename
      next unless r[1].get_name_as_string =~ /#{table}\,/
      if r[1].getStorefileSizeMB() > 0
        if r[1].get_name_as_string =~ /\.([^\.]+)\.$/
          non_empty.push $1
        else
          raise "Failed to get the encoded name for #{r[1].get_name_as_string}"
        end
      end
    end
  end
  non_empty
end

# Handle command line parameters
table_name = ARGV[0]
do_merge = false
if ARGV[1] == 'merge'
  do_merge = true
end

config = HBaseConfiguration.create();
connection = ConnectionFactory.createConnection(config);
admin = HBaseAdmin.new(connection);

non_empty_regions = list_non_empty_regions(admin, table_name)
regions = admin.getTableRegions(Bytes.toBytes(table_name));

puts "Total Table Regions: #{regions.length}"
puts "Total non empty regions: #{non_empty_regions.length}"

filtered_regions = regions.reject do |r|
  non_empty_regions.include?(r.get_encoded_name)
end

puts "Total regions to consider for Merge: #{filtered_regions.length}"

if filtered_regions.length < 2
  puts "There are not enough regions to merge"
end

r1, r2 = nil
filtered_regions.each do |r|
  if r1.nil?
    r1 = r
    next
  end
  if r2.nil?
    r2 = r
  end
  # Skip any region that is a split region
  if r1.is_split()
    r1 = r2
    r2 = nil
    next
  end
  if r2.is_split()
    r2 = nil
    next
  end
  if HRegionInfo.are_adjacent(r1, r2)
    # only merge regions that are adjacent
    puts "#{r1.get_encoded_name} is adjacent to #{r2.get_encoded_name}"
    if do_merge
      admin.mergeRegions(r1.getEncodedNameAsBytes, r2.getEncodedNameAsBytes, false)
      puts "Successfully Merged #{r1.get_encoded_name} with #{r2.get_encoded_name}"
      sleep 2
    end
    r1, r2 = nil
  else
    # Regions are not adjacent, so drop the first one and iterate again
    r1 = r2
    r2 = nil
  end
end
admin.close
blog comments powered by Disqus