Mice will wander randomly most of the time, but will make a special effort to 'eat' the players crops, which involves trampling them. It is unknown if they will continue doing this in 1.1, where crops may no longer be trampled.
If attacked Mice will flee, but they are not targeted by any animal other than Cats, so this rarely happens unless triggered by the player.
Mice will also climb smaller walls and fences to get to your crops.
Mice can be useful to the player if long grass is scarce, for seeds. Birds also serve this purpose.