Open mail relays are configured SMTP servers that enable most anyone with an online connection to send email through it (hence it being an "open mail" relay, as its name suggests). Since such relays don't bother in authenticating email senders, it's therefore prone to address spoofing and many other hacking techniques that seek to exploit this flawed nature of the SMTP server. On one hand, it's open to everyone and it's basically the default configuration of many mail servers back in the day since it's easier to facilitate email sending in this free-for-all manner. On the other hand, this practice of allowing anyone, not just mail originating from or destined to known users, eventually ended as the dangers of the Internet became even more well-known.
Open Relays and Blacklists
It's much easier to send unsolicited spam to mail servers that don't bother to authenticate whether a user is genuine or not (thus allowing unknown users to send anything to (and receive anything from) you when compared to today's more sophisticated server configurations. Even the Internet itself was initially set up the same way as open mail relays until privacy and hacking concerns compelled people to find better solutions when it comes to sending and receiving mail. The current unpopularity of open mail relays came about due to spammer abuse as well as the spread of worms and various malicious software. Many servers placed open relays in blacklists (at least those that weren't outright closed down suffered this fate instead).
Upon installation, UNIX systems used open relay as the default configuration until around the Nineties; it was in fact a common mail server configuration, which also led to many spammers and hackers abusing this fact to the point that email had gotten a hard-to-shake reputation as a prime resource for viruses and other types of malware. The formerly ubiquitous nature of open relay was due to the fact that it's the most straightforward method of getting email to where it's supposed to go (i.e. via storage and forwarding). There are certain servers that still make use of open relay configurations, which is why it's imperative that you test your relay with the SecPoint Penetrator or Protector to ensure its safety and integrity.