It's very secure with the shim. Klein used to sell a shim for the task, almost identical in thickness to the seatpost shim I used.
I have tried twisting my bars and it is not coming off.
Also, the fork is glued into the headset bearings. So does not come off easily.