There are 4 main classes of attackers, grouped by what sorts of security measures are appropriate for them.
These are extremely unmotivated people like asshole flatmates or nosy neighbors (who end up on your WAN). They will rarely even try anything, and if they do, they will make the tiniest of efforts and give up at the first sign of difficulty.
Just put good passwords on your shit and don't tell them to anyone, and it will be enough for this group.
These are attackers who actually have a strong motive to get your data, but aren't competent and don't have the resources to make a serious attack. Say a burglar steals your computer. Of course he will try turning it on and seeing if there's anything interesting inside. He will not hesitate at all, unlike the previous group. Hopefully, he will see a password prompt, and after a few tries, give up on trying to guess it. However, again unlike the previous group, he will not stop there. If at all computer literate, he will try to plug the hard drive into another computer, or boot from a LiveCD. At that point it's possible that you will be the victim of identity theft.
Technically, this group has physical access, so by the common maxim of computer security, you have already lost. But in practice, they could only defeat your security theoretically. In practice, they don't know how to actually leverage that physical access, and probably won't bother trying.
Full disk encryption will deter this group, since they don't have the resources to defeat, and will simply give up and just sell the hardware at that point. Encryption takes a lot of work to defeat, and most of the shit people encrypt honestly isn't valuable or worth bothering with.
Within even the first day of your server's uptime, you will realize that there's tons people constantly trying to hack your machine. There is a sliding scale: There are tons of shitty script kiddies just randomly scanning ports over a range of IPs, and when they see your server responding, they start trying to bruteforce the SSH. This is extremely common and very easy to protect against. On the other hand, there are some extremely determined, very skilled hackers who buy 0days on the black market and use them, but these will probably be rarer, and they will prefer go after juicier targets like banks before coming after your home server full of animu porn.
The way to defend against this group is to configure your server for strong security, always keep your software up to date as new exploits appear, and keep a close eye on signs of intrusion. Defending against this group is probably the bulk of your security related maintenance workload.
At the same time, you have to recognize that you the only way to have perfect security is to turn off your server and disconnect it from the internet. Consider something like the Heartbleed bug: There is nothing you could have done to protect against it, even if you acted the moment the news broke (you could have been attacked before the news were out). That said, if a bad guy got hold of Heartbleed, he probably wouldn't attack you (there are much better targets), and just because you can't have perfect security, doesn't mean you shouldn't have any security. Lockpicks exist, but we still lock our doors.
The last class is attackers with very high competence, extensive resources, who are highly motivated to come after you. This is basically a government agency (CIA/NSA/FBI/Chinese spies) or a serious hacker (eg. hired by a company) targeting specifically you. Note that, as opposed to the 0day hacker from the previous class, these attackers are targeting specifically you. So the argument that they have "bigger fish to fry" does not apply anymore: Either the government is after you because you did something they don't like, or a someone paid a hacker to get you, either way the attacker will not stop until they've tried every trick in the book (and it's a very big book).
There isn't much you can do about this group. You could try to take some measures, but they will be very inconvenient, and some may even be dangerous or borderline illegal. For one, the government could easily gain physical access to your system by producing a warrant, and unlike a burglar, they actually could and would take advantage of that physical access. Given this, your best option is to not attract any attacks in the first place: Don't do anything illegal, don't be an important person targeted by cyber warfare, and don't piss off any hackers, or anyone who would pay a hacker. Anything beyond that is out of this document's scope.