Colorectal cancer is a common malignant tumor with the second incidence rate and the third mortality rate worldwide. In this study, we identified and validated an immune-related gene signature, explored the clinical and molecular characteristics of the signature-defined risk groups, and assessed its ability in predicting prognosis, immune cell infiltration and immunotherapy responses. The Cancer Genome Atlas database was used as the training set while GSE39582 database as the validation set. Immune-related hub genes were selected by the Least Absolute Shrinkage and Selection Operator-penalized Cox regression model, and the signature was then constructed by the selected genes and their relevant coefficients. Prognostic performance of the signature and the signature-base nomogram models were assessed by time-dependent receiver operating characteristic curves and calibration plots in both training and validation cohorts. Clinical and mutation-related data were downloaded and analyzed to explore their associations with signature-defined risk groups. Proportions of infiltrated immune cells was estimated via CIBERSORT algorithm and immunotherapy response was evaluated by immunophenoscore and tumor immune dysfunction and exclusion scores. Seven among 790 immune-related differentially-expressed genes were selected and use to construct the signature. The signature and signature-base nomograms showed promising prognostic performance in both training and validation cohorts. Signature-defined high-risk group was associated with advanced disease, poor pathological prognostic factors and less active immune infiltration microenvironment. Besides, the response to immunotherapy of high-risk group was predicted to be poorer by immunophenoscore and tumor immune dysfunction and exclusion scores. Our signature proved its efficacy in predicting prognosis, tumor immune microenvironment and responses to immunotherapy in colorectal cancer.