When the client disconnects we need to remove all events from our loop.
Prior to the introduction of the file monitor we only needed to remove
the client pipes.  Now we need to remove the file monitor events too.

Without this the event loop may continue to block on file monitor
events.  If one does eventually come in then the event handler may try
to report it to the disconnected client and crash because our internal
structures for writing to the client have been freed.

This addresses a failure of the `Server` test on some machines.