Apache Big Data EU 2015 - HBase

State
of
the
Database

@HBase

h)p://hbase.apache.org

2015-‐09-‐28

Nick
Dimiduk
(@xefyr)

h)p://n10k.com

#apachebigdata

Agenda

o State
of
the
Project

o State
of
the
SoMware

o State
of
the
Ecosystem

o Latest
Releases

o Bonus
Content!

o Q
&
A

STATE
OF
THE
PROJECT

Who
we
are,
what
we
do,
why
we
do
it

Project:
Vision

Simple,
steady,
and
powerful:
“A
ﬁrst
class
high

performance
horizontally
scalable
data
storage

engine
for
Big
Data,
suitable
as
the
store
of

record
for
mission
cri]cal
data.”

Project:
Usage

o  Data
access
for
medium-‐
and
high-‐scale
services

o Hundreds
of
enterprises
and
startups

o Some
of
the
largest
Internet
companies
in
the
world

o  Running
major
produc]on
workloads
since
2011

o  Use-‐cases

o messaging,
security,
measurement/“IoT”,

collabora]on,
digital
media,
digital
adver]sing,

telecommunica]ons,
computa]onal
biology,
clinical

informa]cs/healthcare,
insurance

Apache Big Data EU 2015 - HBase

Project:
Goals

o Availability:
Always
more,
always
faster

o Stability
and
operability

o Scaling
up,
scaling
down

o Up-‐to-‐date
with
“commodity”
hardware

o Mul]-‐tenancy

o Diversity
of
ecosystem

STATE
OF
THE
SOFTWARE

Regarding
the
codebase

State
of
the
SoMware

o Mature
codebase

o 100+
contributors
(40+
commi)ers)

o 1.1M
lines
of
code
(each
ac]ve
branch)

o est.
1200+
human-‐years’
eﬀort

o  Clusters
sizes
from
10
to
1000+
machines

o  that
we
know
of!

o Runs
on
HDFS,
MapR,
Gluster,
GPFS,
Lustre

o HBase
as
a
Service

o AWS/EMR,
HDInsight,
Qubole,
Google
(sort-‐of)

SoMware:
Seman]c
Versioning

MAJOR-‐MINOR-‐PATCH[-‐iden]ﬁer]

o Client/Server
wire
compa]bility

o Server/Server
feature
compa]bility

o API
compliance
guarantees

o ABI
compliance
guarantees

h)p://hbase.apache.org/book.html#hbase.versioning

SoMware:
Ac]ve
Development

o Smaller
regions,
more
regions

o Less
write
ampliﬁca]on

o 1M+
region
clusters

o Stability

o ProcedureV2

o Assignment
improvements/stability

o Backup,
restore
tools

o Built
on
snapshots,
easier
opera]ons

SoMware:
Ac]ve
Development

o Adap]on:
Workloads

o HBase
as
Medium
Object
Store
(MOB)

o Tunable
Availability

o Region
replicas

o TIMELINE
consistency

o Coprocessor
API
stability

o Less
GC,
more
RAM
(oﬀ-‐heap)

SoMware:
Ac]ve
Development

o Mul]-‐tenancy

o Table
groups

o Quotas

o Priori]es

o Improved
machine
u]liza]on

o More
RAM
(100’s
of
GB)

o IOPS

o Be)er
concurrency

STATE
OF
THE
ECOSYSTEM

The
whole
enchilada

State
of
the
Ecosystem

o OpenTSDB

o Transac]on
Managers

o Themis,
Tephera,
Omid2,
LeanXcale

o Graph
engines

o Titan,
Giraph,
Zen,
S2Graph

o Myriad
SQL’s

o Other
Hadoop
components

o Google
Cloud
Bigtable

Ecosystem:
Hadoop
Components

o YARN-‐2928
Applica]on
Timeline
Service

o HIVE-‐9452
HBase
to
store
Hive
metadata

o AMBARI-‐5707
Ambari
Metrics
System

LATEST
RELEASES

Come
and
get
it!

Release:
0.94

o Last
(ﬁnal?)
release:
0.94.27,
2015-‐03-‐26

o “ancient
history”

o No
new
deployments

o Exis]ng
users
highly
encouraged
to
upgrade

o Requires
down]me
to
upgrade

😫
😡
(╯°□°)╯︵
┻━┻

Release:
0.98

o Last
release:
0.98.14,
2015-‐08-‐31

o “legacy”

o Most
produc]on
deploys
(probably)

o Largest
produc]on
clusters
(probably)

o New
features
back-‐ported
when
possible

Release
1.x

o Last
release:
1.1.2,
2015-‐09-‐01

o “stable”

o Produc]on
deploys
moving
here

o Ac]ve
development

o Rolling
upgrade
from
0.98.x

😄
😍
ヽ(´ー｀)ﾉ

Release
1.0

o Released
1.0.0,
2015-‐02-‐24

o Adop]ng
seman]c
versioning

o Patch
releases
don’t
quite
follow
spec
yet

o Client
/
Server
API
cleanup

o Interfaces,
builder
pa)ern,
@InterfaceAudience

o Region
Replicas

o Trade
Consistency,
resources
for
Availability

github.com/ndimiduk/hbase-‐1.0-‐api-‐examples

Region
Replicas

o Mul]ple
Region
Servers
host
each
region

o Primary
+
N
read
replicas
(usually
N=2)

o Primary
is
authority
on
reads
and
writes

o Replicas
tail
replicate
edits,
oﬀer
TIMELINE
view

o Client’s
choice

o Read
primary
only
for
“classic”
strong
consistency

o Fan-‐out
reads
for
faster,
poten]ally
TIMELINE

results

Release
1.1

o  Release
1.1.0,
2015-‐05-‐15

o  Async
RPC
client

o  Scanner
improvements

o RPC
chunking,
heartbeat
messages,
API

o  RPC
thro)ling

o quotas
for
per
user,
table,
namespace

o  Compac]on
thro)ling,
monitoring

o  ProcedureV2

o Improved
opera]onal
reliability

ProcedureV2

o Distributed,
fault-‐tolerant
opera]ons

o Mul]ple
steps
on
mul]ple
machines

o Roll-‐back
in
case
of
failure

o Coordina]on
of
long-‐running
procedures

o Compac]ons,
splits,
&c.

o Progress
tracking

o No]ﬁca]ons
across
mul]ple
machines

o Current
status
inquiries

Branch-‐1.2

o Next
up
in
1.x
line

o Java
8
support

o Na]ve
checksums

o SyncTable

o Flush-‐per-‐store

o ProcV2
all
the
things!

o (More)
Compac]on

improvements

o Region
normalizer

Region
Normalizer

o  An]-‐entropy
for
region
size

o  Converge
towards
uniform
size

o  Compliments
balancer
working
toward
uniform
distribu]on

o  Managed
by
Master,
runs
in
the
background
(like
balancer)

o  Pluggable
normaliza]on
strategies
(“simple”
default)

o  Use-‐cases

o  Merge
away
regions
from
expired
]meseries
data

o  Smooth
uneven
bulk
loads

o  Correct
operator
ini]al
split
guesses

o  Ease
upgrades
from
ancient
versions
(0.92/1g
vs.
today/20g)

Thanks!

@HBase

h)p://hbase.apache.org

2015-‐09-‐28

Nick
Dimiduk
(@xefyr)

h)p://n10k.com

#apachebigdata

BONUS
CONTENT!

Ask
and
you
shall
receive

Agenda

o Replica]on

o Filters

o Coprocessors

Replica]on

o  Keep
data
synchronized
between
clusters

o  Supports
mulOple
desOnaOons

o  Cyclical
graphs
supported

o  Conﬁgurable
at
Column
Family
granularity

o  Uses
WAL
shipping
to
propagate
data

o  Replica]on
state,
status
stored
in
ZooKeeper

o  General
purpose
interface
for
asynchronously
shipping

edits
from
a
cluster

o  Other
HBase
clusters,
Region
Replicas,
SOLR/Elas]cSearch

hbase.apache.org/book.html#_cluster_replica]on

Filters

o  Addi]onal
applied
to
reads

o Use
in
conjunc]on
with
specifying
start,
end
rows,
&c.

o  Run
on
the
Region
Servers

o Included
in
GET,
SCAN
request

o  Explicitly
exclude
data
based
on
criteria

o I.E.,
value
>=
10

o  Implicitly
exclude
data
by
hin]ng
seeks

o INCLUDE_AND_NEXT_COL,
NEXT_ROW,

SEEK_NEXT_USING_HINT

o  Operate
on
data
read
from
BlockCache

Filters

o 30+
Filters
included
in
distribu]on

o Mini-‐language
for
use
in
ThriM,
REST

o  "(PrefixFilter ('row2') AND (QualifierFilter (>=,
'binary:xyz'))) AND (TimestampsFilter ( 123, 456))"
o  hbase.apache.org/book.html#thriM.ﬁlter_language

o Simple
interface,
Implement
your
own!

public class PageFilter extends FilterBase {
public PageFilter(long pageSize) {…}
public boolean filterRowKey(Cell c) {
return false;
}
public ReturnCode filterKeyValue(Cell c) {
return ReturnCode.INCLUDE;
}
public boolean filterAllRemaining() {
return this.rowsAccepted >= this.pageSize;
}
public filterRow() {
this.rowsAccepted++;
return this.rowsAccepted > this.pageSize;
}
}

Coprocessors

o  Extension
points
for
HBase

o Think
Linux
Kernel
Module,
not
Stored
Procedure

o I.E.,
customize
compac]ons,
Table
constraints

o  Observers

o pre-‐
and
post-‐execu]on
logic

o I.E.,
MasterObserver#preTruncateTable,

RegionObserver#postScannerNext

o  Endpoints

o Cluster
RPC
extensions

o I.E.,
RowCountEndpoint,
BulkDeleteEndpoint

public class RowCountEndpoint implements
ExampleProtos.RowCountService {
public void getRowCount(…) {
Scan = new Scan();
InternalScanner scanner =
env.getRegion().getScanner(scan);
…
do {
count++;
} while (scanner.next());
// return count
}
}

Apache Big Data EU 2015 - HBase

More Related Content

What's hot (20)

Viewers also liked (13)

Similar to Apache Big Data EU 2015 - HBase (20)

More from Nick Dimiduk (10)

Recently uploaded (20)

Apache Big Data EU 2015 - HBase