Human observers generally perceive a stable and coherent visual scene despite the fact that sensory information is ambiguous and often incomplete. Perceptual filling-in provides an interesting example of how the visual system realizes perceptual inferences from incomplete information. Vision scientists have a long history studying filling-in phenomena in the context of surface color filling-in. While significant progress has been achieved with behavioral experiments, little is known about the neural substrate of perceptual filling-in. We explored several hypotheses that have been considered to determine how the neural representation of edge-induced filling-in percepts arises in the brain. We argue that the visual system uses distinct extra-striate pathways for the processing of chromatic surfaces generated by edge-dependent filling-in that indirectly influence color pathways via a contour integration mechanism. Such a mechanism may play an important role in figure/ground segregation. Finally, we highlight the potential role of feedback projections that are ignored in many models despite the fact that they are as numerous as feedforward projections.