Oh boy... so I told him that I am Hispanic and he back pedals away from what he said.
The issue isn't that he's racist, as I suspect most of us are. The issue is that he doesn't know that he's racist. I mean, you probably have some biases too (https://implicit.harvard.edu/implicit/demo/selectatest.html), Lord knows I do. But to not know about them is the worst kind of mistake. Like an engine, you can't fix what you don't know is wrong.
My neighbor was probably just associating visible crimes with people of color, which is probably a fair association. People of color are generally poorer than white counter-parts and poor people generally commit the more visible kinds of crimes (theft, fights, disorderly conduct etc.). White people commit crimes too, but they are of the "download movies illegally", or "abuse my family", or something like that. Those kinds of crimes are not as visible and go unnoticed by society and therefore don't get the stigma that "people of color" do.
Aside: that's really a dumb phrase, "people of color". It just implies that "white" is "normal" and "other" requires some kind of additional nomenclature. Everyone has a color.
In any case, as I told my neighbor: trash comes in all colors.