The comments offered by these referees would not be delete-able except by the one who commented.
Each comment would itself have room for a single two-fold response by each debater: an upvote/downvote, and a short text-response (115 characters). Since there are only ever 2 debaters, the range of possible scores would be -2 to +2. The score would not affect the display of the comment in any way, but would serve as a good marker for comments which both parties agree were exceptionally fair/astute, or exceptionally bad.