Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
Store SnapShot method does not buffer read the database which could be an issue if database size is large #273
Comments
Thanks -- it's a weakness, I can look into addressing. |
Hi again, Philip! |
Is it actually causing you an issue?
… On Feb 14, 2017, at 11:46 AM, Ivan ***@***.***> wrote:
Hi again, Philip!
Do you happen to know how to fix this problem? I guess it's a "bug" in hashicorp/raft implementation.
Because as far as I understood, hashicorp/raft doesn't call Snapshot() and Persist() concurrently, so I don't know of a way to communicate chunks of the database between Snapshot() and Persist() - do you have an idea how to do this?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
I mean, are you actually encountering a problem in practice?
… On Feb 14, 2017, at 11:46 AM, Ivan ***@***.***> wrote:
Hi again, Philip!
Do you happen to know how to fix this problem? I guess it's a "bug" in hashicorp/raft implementation.
Because as far as I understood, hashicorp/raft doesn't call Snapshot() and Persist() concurrently, so I don't know of a way to communicate chunks of the database between Snapshot() and Persist() - do you have an idea how to do this?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
otoolep, thanks a lot! I've sorted it out with the raft library authors - here |
Thank you @JeanGolang -- I will look into using the new interface exposed by the Raft package. |
Hello Philip!
I think there is an issue with this part of the code of rqlite (store/store.go).
As far as I understood from your code it's a support function for creating a snapshot of the database. However, this implementation reads the whole database into memory, in your particular case in the variable fsm.database, err = s.Database(false), so if the size of your database is larger than RAM available, I guess the program would crash.
I guess it's better to read the database in buffers or send chunks of the database via a channel when writing it to the so called SnapshotSink.
Thanks!