Interaction between route-map and access-group on powerconnect vlan

I'm doing PBR on a vlan to route out a secondary ISP as describing in the PBR Guide. This works fine. On this vlan, I also need to implement port restrictions to other vlans. If I add a access-group to the vlan interface, the route-map stops working, even if the access-group only has one entry to permit everything. Is there a trick to get this to work?