As long as you are not in the same field or department, it should be all right. I know people who worked at the phone Bank, got fired and went to work at a branch, just let the temp agency know.