Jesse Williams did not have a very merry Christmas. The actor is accusing his ex-wife of keeping their two children away from him over the holiday season.

According to TMZ.com, Williams filed a petition to have a judge modify the custody agreement he and Aryn Drake-Lee agreed to this past November. Williams, who is seeking custody for two weekdays, accused his ex of violating the overnight agreement their custody schedule initially set in place.

The Grey’s star says Drake-Lee did not allow him to see his 4-year-old daughter Sadie and 2-year-old son Maceo on Christmas Eve, Christmas and New Year’s Eve. Williams also says his ex only “answers one out of about every three FaceTime calls” and only allows him to speak to their children for minutes at a time whenever she does answer.

The former couple filed for divorce in April 2017 after five years of marriage.