Inflammatory cells are an essential component of the tumor microenvironment. Neutrophils have emerged as important players in the orchestration and effector phase of innate and adaptive immunity. The significance of tumor‐associated neutrophils (TAN) in colorectal cancer (CRC) has been the subject of conflicting reports and the present study was designed to set up a reliable methodology to assess TAN infiltration in CRC and to evaluate their clinical significance. CD66b and myeloperoxidase (MPO) were assessed as candidate neutrophil markers in CRC using immunohistochemistry. CD66b was found to be a reliable marker to identify TAN in CRC tissues, whereas MPO also identified a subset of CD68⁺ macrophages. CRC patients (n = 271) (Stages I–IV) were investigated retrospectively by computer‐assisted imaging on whole tumor sections. TAN density dramatically decreases in Stage IV patients as compared to Stage I–III. At Cox analysis, higher TAN density was associated with better prognosis. Importantly, multivariate analysis showed that prognostic significance of TAN can be influenced by clinical stage and 5‐fluorouracil(5‐FU)‐based chemotherapy. On separate analysis of Stage III patients (n = 178), TAN density had a dual clinical significance depending on the use of 5‐FU‐based chemotherapy. Unexpectedly, higher TAN density was associated with better response to 5‐FU‐based chemotherapy. Thus, TAN are an important component of the immune cell infiltrate in CRC and assessment of TAN infiltration may help identify patients likely to benefit from 5‐FU‐based chemotherapy. These results call for a reassessment of the role of neutrophils in cancer using rigorous quantitative methodology.