When acting as agent for an elderly relative, the power granted is supposed to make life simpler for both of you. Power of attorney can be useful, but also quite stressful. It's important to respect the wishes of your relative no matter what their mental state.
Many people grant power of attorney to a relative even if they are of sound mind and body. They do this to make it easier for business or legal proceedings to go forward. Power of attorney is not carte blanche for the agent to do whatever he or she wishes. I remember one example of a woman who decided to make the move and downsize. She had moved some of her possessions into a smaller apartment while leaving others in her house, until she would decide what to do with them at a later date. When she went back to her house to gather the remainder of her belongings, her POA had already cleaned out the house and listed it on the market. Her POA did not do this maliciously however she did overstep her wishes. The actions of the power of attorney are supposed to be consistent with the wishes of the principle.
If you have been given power of attorney and your principle is no longer able to make decisions, due to physical or mental impairment, it is still your job to know and respect their wishes. You should consult with other family members and the principle's lawyer to try and determine what the principle wanted. It is important also to check the will. Consulting the will can help direct your actions if it is clear your principle left specific possessions to family members or organizations.
People with power of attorney often fall into the same trap. They tend to get drawn into doing too much, and end up focusing on practical or legal minutia. The POA wants to be helpful and supportive of their relative, but they can often end up not being emotionally supportive. That's the hard stuff, and it's easy to put off. It's also important to take care of your own mental health. If you are overstressed and emotionally drained, you are not going to be performing your best. Talk to your relative. Understanding their needs, and being able to explain yours, will make the situation much easier to handle if or when the time comes.