Well, I used them for a cobblestreet once, yonks ago when I were a wee lad, and threw the base away after oh... only 20 years.
Just don't soak them first, like you would if you were to eat them, and you might want to "seal" them with a layer of FFA or what-not. Though I didn't do that (what did I know) and it still looked good. Btw: the reason I threw it out was because over the years I lost most of the peas (I used plastic cement tube glue back then, again, what did I know, I'd use something else now), not because of any mould.
But I agree with you on the edible/rottable bit. The only other foodstuff I ever used was cacao powder dusted on thinned whiteglue to imitate dried mud. That was before the time when we knew about pigments. I still have that base, I could make a pic if anyone thinks it's worth it.