Apache Druid 101
1 like541 views
Apache Druid is a high-performance, real-time analytics database designed for low-latency and high concurrency, suitable for a variety of data applications. It was initially developed for digital advertising by Metamarkets and has since expanded across multiple sectors, allowing for fast ingestion and complex queries on large datasets. Druid's architecture is distinguished from traditional data warehouses by providing real-time data access, flexible schemas, and support for ad-hoc query environments.
[3,4] (00011000)
[5,6,7](0000111)
DATA0
0
0
1
1
2
2
2
DICT
Melbourne = 0
Perth = 1
Sydney = 2](https://image.slidesharecdn.com/dataconlasarreldruid101-201103193754/85/Apache-Druid-101-28-320.jpg)
Apache Druid 101
- 1. Apache Druid 101: 1 Fast, Real-time, Open Source Analytics
- 2. Matt Sarrel Developer Evangelist ● Former IT leadership/CIO roles ● Focus on data management, data analysis, network infrastructure and network security ● 15 years of startup experience (mostly open source infrastructure and datastores) ● Former PCMag Tech Director, GigaOm Pro analyst, eWeek and InfoWorld contributor ● BA (History) and MPH (ID Epi) ● CISSP certified ● Cook Competitive BBQ (KCBS) [email protected] @msarrel on Twitter @matt on ASF #druid Slack
- 3. Agenda 3 What is Druid? Why was Druid created? Best Uses for Druid How Druid Works
- 4. Druid is a high performance real-time analytics database
- 5. Apache Druid Powers Interactive Applications 5
- 6. 1st gen: on-prem data warehouses 6 The 1st gen architecture was unscalable, complex, and expensive. Data Sources Processing Store and Compute BI tools Reporting Analytics Data Data Data ETL Data Warehouse
- 7. 2nd gen: cloud data warehouses 7 The 2nd gen, while cheaper and more flexible, still has many latency restrictions. Data Sources Storage Compute Processing BI tools Reporting Analytics Data Data Data Data Lake (S3, Blob store, etc) Data Warehouse ELT (Spark, Hadoop, etc)
- 8. 3rd gen: Apache Druid/Imply 8 The 3rd gen architecture is designed for an increasingly low latency world. Data Sources Storage Processing Next gen data apps Interactive GUIs Real-time Analytics Data Data Data Message Bus (Kafka, Kinesis, Pub/Sub) ELT (Spark Streaming, Kafka Streams, Apache Flink) Druid Data Warehouse Archiving Reporting
- 9. MetaMarkets: The first use case Druid was created at a startup called Metamarkets (now part of Snapchat) Druid was created to power an interactive app for digital advertisers Advertisers loaded impressions and clicks data Advertisers used the app to optimize user/ad engagement Druid has since expanded to many new verticals and use cases
- 10. Challenges • Scale: millions events/sec (batch and real-time) • Complexity: high dimensionality & high cardinality • Structure: semi-structured (nested, evolving schemas, etc.)Data: • Drill downs: static reports aren’t enough (BI tools not enough) • Multi-tenancy: thousands of concurrent users • Self-service: many users are non-technical App:
- 11. Core Design ● Real-time ingestion ● Flexible schema ● Full text search ● Batch ingestion ● Efficient storage ● Fast analytic queries ● Optimized storage for time-based datasets ● Time-based functions SEARCH PLATFORM TIME SERIES DB OLAP
- 12. Key features Column oriented High concurrency Scalable to 1000s of servers, millions of messages/sec Continuous, real-time ingest Query through SQL via API Target query latency sub-second to a few seconds 12
- 14. User activity Data sets: clickstreams, view streams, activity streams Group users along any attributes, without pre-computation or pre-definition Compare groups of users against each other Define interesting groupings quickly through top lists Count number of users matching any criteria
- 15. Network flows Data set: netflow logs View relationships between source & dest addresses Measure flows based on protocol, interface, IP address, or any other attribute Burstable billing: 95th percentile flow rates in 5 min buckets Troubleshoot bottlenecks
- 16. Digital advertising Data sets: bids, clicks, impressions, etc. Analyze campaign performance on ad-hoc groupings of participants Compute quantiles and histograms for bid prices Calculate conversion rates (impressions → clicks)
- 17. Server metrics Data sets: server logs, application metrics, etc. Track CPU load on servers, numbers of cache requests/hits/misses, data center performance, etc. Aggregate time series on the fly Compute latency %iles over ad hoc groups of events (all ‘foo’ servers; all ‘/v1/bar’ API calls; all servers in rack 10; etc)
- 18. Druid…it’s out there The original Druid cluster: • >500 TB of segments (>50 trillion raw events, >50 PB raw data) • mean 500ms query time • 90%ile < 1s • 95%ile < 5s • 99%ile < 10s Netflix Druid Cluster: • 100 billion+ rows/day • 1+ trillion rows, retained for at least a year • 100s of servers • Sub-second to a few seconds query response • Relies on combination of streaming and batch ingestion
- 19. Druid is designed for performance Data sourced from: Correia, José & Costa, Carlos & Santos, Maribel. (2019). Challenging SQL-on-Hadoop Performance with Apache Druid.
- 20. There is no such thing as too fast
- 21. Is Druid Right For My Project? Data Characteristics Timestamp dimension Streaming Denormalized Many attributes (30+ dimensions) High cardinality Use Case Characteristics Large dataset Fast query response (<1s) Low latency data ingestion Interactive, ad-hoc queries Arbitrary slicing and dicing (OLAP) Query real-time & historical data Infrequent updates
- 22. Druid in Data Pipeline Data lakes Message buses Raw data Staging (and Processing) Analytics Database End User Application clicks, ad impressions network telemetry application events
- 23. Druid and Data Warehouses Druid is not a DW Druid augments DW to provide • consistent, sub-second SLA • pre-aggregation/metrics generation upon ingest • simple schema • high concurrency reads Druid is for hot queries (sub-second queries on fresh data) • Slice and dice OLAP • Dashboards that fire dozens of queries at once DW is for cold queries (second+ queries on historical data)
- 27. Druid segments Enables global index and write once consistency.
- 28. Engine and data format are tightly integrated 28 Secondary indexes Operate on compressed data Late materializationCompression INDEX [0,1,2](11100000) [3,4] (00011000) [5,6,7](0000111) DATA0 0 0 1 1 2 2 2 DICT Melbourne = 0 Perth = 1 Sydney = 2
- 29. Querying Query libraries: • JSON over HTTP • SQL • R • Python • Ruby
- 30. Query Processing in Parallel 30 Historical Indexer Historical Indexer Data server Broker Query server Segments
- 31. Apache Druid Unified Console
- 32. Resources druid.apache.org druid.apache.org/community Imply Meetup Groups https://www.meetup.com/pro/apache-druid ASF #druid Slack channel @druidio on Twitter Apache Distribution https://github.com/apache/druid Imply Distribution https://imply.io/get-started