Technically, the person named on a medical power of attorney or health care proxy has a duty only to the patient, not to the patient's family. That said, it's certainly better for all concerned if decisions can be made as a team. As you suggest, the question of taking your mother outside is not necessarily a medical issue, so it’s a question as to whether it comes under the scope of the medical power of attorney.
Depending on your state’s laws, this may be a gray area, which is all the more reason for everyone to work together. I’d suggest asking the social worker at your mother’s facility to convene a meeting with you, your brother, and the medical power of attorney to discuss how to best care for your mother.
Get more information about health care proxies in this article.