You've removed the brake boss along with the brake itself. That Allen key head to the left of the Dia Compe logo is the head of a bolt that threads into the brake boss. Normally the bolt comes out and the brake slides off the boss. If it's all seized together, you might have to put the threaded part of the brake boss in the vice, then loosen the Allen key bolt and work the V-Brake off the post. You would be sacrificing the brake boss though, as the threads would get damaged.