I followed the README to set up the environment using uv and successfully downloaded the dataset via Hugging Face:
lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v5")
However, when running the evaluation command:
python -m lcb_runner.runner.main --local_model_path {model_name} --scenario codegeneration --evaluate --release_version release_v5
I encountered the following error:
ValueError: Couldn't find cache for livecodebench/code_generation_lite for config 'default-version_tag=release_latest'
Available configs in the cache: ['release_latest-version_tag=release_v5']
I noticed this is caused by the load_dataset call in code_generation.py:
https://github.com/LiveCodeBench/LiveCodeBench/blob/main/lcb_runner/benchmarks/code_generation.py
As suggested in #108, I already removed trust_remote_code=True and updated the call to:
load_dataset("livecodebench/code_generation_lite", release_version, split="test")
or equivalently:
load_dataset("livecodebench/code_generation_lite", split="test", version_tag=release_version)
Despite this change, the ValueError persists. It seems the code is still trying to look for a config named default-version_tag=release_latest, which doesn't match the cached config release_latest-version_tag=release_v5.
Any guidance would be appreciated. Thanks!