Apache Calcite: One planner fits all
15 likes•6,894 views
This document discusses how Apache Calcite makes it easier to write database management systems (DBMS) by decomposing them into modular components like a query parser, catalog, algorithms, and storage engines. It presents Calcite as a framework that allows these components to be mixed and matched, with a core relational algebra and rule-based optimization. Calcite powers systems like Apache Hive, Drill, Phoenix, and Kylin by translating SQL and other queries to relational algebra and optimizing queries using over 100 rules before executing them using configurable engines and data sources.
![SELECT products.name, COUNT(*)
FROM sales
JOIN products USING (productId)
WHERE sales.discount IS NOT NULL
GROUP BY products.name
ORDER BY COUNT(*) DESC
scan
[products]
scan
[sales]
join
filter
aggregate
sort
scan
[products]
scan
[sales]
filter’
join’
aggregate
sort
FilterIntoJoinRule
translate SQL
to relational
algebra](https://image.slidesharecdn.com/calcite-xldb-2015-150521211311-lva1-app6891/85/Apache-Calcite-One-planner-fits-all-8-320.jpg)
Apache Calcite: One planner fits all
- 1. One planner fits all (How Apache Calcite makes it easier to write a DBMS) Lightning talk at XLDB 2015 Stanford, California Julian Hyde (Hortonworks)
- 2. –Mike Stonebraker (2005) “One size fits all” is an idea whose time has come and gone
- 3. • Hadoop and other open source technologies have deconstructed the DBMS • Query parser/API + catalog + authorization + algorithms + scheduler + engine + data format + storage
- 7. Apache Calcite DB framework Parser Algebra Engine Data Engine Data Engine Data Schema SPI Operators, Rules, Statistics, Cost model
- 8. SELECT products.name, COUNT(*) FROM sales JOIN products USING (productId) WHERE sales.discount IS NOT NULL GROUP BY products.name ORDER BY COUNT(*) DESC scan [products] scan [sales] join filter aggregate sort scan [products] scan [sales] filter’ join’ aggregate sort FilterIntoJoinRule translate SQL to relational algebra
- 9. Relational algebra • Robust • Allows re-use • Complex cost-based optimization • Multiple front-ends & back- ends • Not just for “flat” relations SQL Other QL API Algebra Rules Engine A Engine B
- 10. Thank you! Download: http://calcite.incubator.apache.org Use Calcite to build your next database! Calcite powers Apache Hive, Drill, Phoenix, Kylin An Apache Incubator project since May 2014 @julianhyde What’s in the box? • SQL parser & AST • JDBC/ODBC framework • Built-in operators (project, filter, …) • In-memory engine • 100+ rules • Planning engines • Adapters (CSV, JDBC, Mongo, …) • Streaming SQL • Materialized views Apache Calcite Apache Calcite