Surface color appearance depends on both local surface chromaticity and global context. How are these inter-dependencies supported by cortical networks? Combining functional imaging and psychophysics, we examined if color from long-range filling-in engages distinct pathways from responses caused by a field of uniform chromaticity. We find that color from filling-in is best classified and best correlated with appearance by two dorsal areas, V3A and V3B/KO. In contrast, a field of uniform chromaticity is best classified by ventral areas hV4 and LO. Dynamic causal modeling revealed feedback modulation from area V3A to areas V1 and LO for filling-in, contrasting with feedback from LO modulating areas V1 and V3A for a matched uniform chromaticity. These results indicate a dorsal stream role in color filling-in via feedback modulation of area V1 coupled with a cross-stream modulation of ventral areas suggesting that local and contextual influences on color appearance engage distinct neural networks.