The genealogy purists would probably say you would add you father's first wife since your half-siblings are blood / genetic relatives of yours.

Conversally, you wouldn't add your mother's 1st marriage since there is no children for you to have a relationship to.

Family historians prone people would add both 1st marriages in order to build a more complete picture of both your mother and father.

Personally, I'd also add in any low hanging fruit that pops up. I'd probably start with the important fact of your mother's 1st marriage. Maybe look at the marriage certificate for that date, which is moderately important. As low hanging fruit, the marriage cert might have his birth date, and his mother's maiden name along with his father's name.

If your mother and her ex lived in the same town as kids, you might stumble upon him in the census; if so, add that information. I wouldn't bother looking for his family specifically though.