IS DHS can challenge the decision of BIA? or reverse the decision.

Yes they can, is your AOS case based on css/lulac?
did you won the appeal?

BIA decision is considered to be the final administrative decision.
For DHS to reverse it there must be substantial grounds and evidence that didn't exist on record at the time when BIA made it's decision.

You will not get on public forums reliable advise on complicated immigration issues.
Nor would it be possible for anyone to give you proper advise without learning all the facts of your case, including the written BIA opinion.

You need to contact and consult professional immigration attorney for further assistance.