Unique RPG Mechanics?

Traveller (the older the version the better)
WEG Star Wars d6
Genesys (for wacky dice)
Apocaypse World or Monster of the Week (PbtA before they all crawled up their own ass)
Seconding Mythras, but check out Runequest of some sort to see which serial numbers they filed off.

Check out Conan 2d20

There's Nechronica with it's tactical part based combat and battle map.

Just ran Blade Runner RPG recently and it's become one of my favorite quick and easy dice systems with very fast and brutal ranged and melee combat. Four attributes (Strength, Agility, Intelligence, Empathy) with three skills tied to each. Each attribute and skill has a rating from D to A with die sizes d6 (D), d8 (C), d10 (B) and d12 (A). You roll both your attribute and skill die together when making a check. 6+ on a single die is one success, 10+ on a single die is two successes. Advantage on a check adds a die of the lowest size to the roll, disadvantage removes it so you instead only roll one. Typically one success is enough to succeed on a check, but two successes or more often grants you extra benefits. For example, two successes when shooting someone is an automatic roll on the critical hit chart regardless of their current HP, with a fairly high chance of killing or getting killed instantly.

There's some other nuance to the die system like being able to push a roll, which is being able to reroll once if you're Human, or up to twice if you're a Replicant. If you rolled any 1's, you can't push/reroll those die, and if you pushed and have any 1's at your check result, you receive either 1 physical or stress damage depending on the attribute. eg. you're pursuing someone and try to smash through a flimsy wall with Strength (Force) to give yourself a shortcut, so you roll your two dice (rolling a 1 and 5, both fail). You push to reroll the 5, knowing you'll automatically take a damage because you already have a 1, but you really need this check to succeed, because failure means they'll probably get away. The second die gets a 6+ after the reroll which lets you smash through the wall, giving you the shortcut, but you suffer 1 physical damage as you bust through the flimsy wall due to the 1.

Since all you've played are d20 games, the concept of dice pools will probably blow your frickin' mind. Look into Shadowrun or World of Darkness.

There are lots and lots of different games out there, where the mechanics exist to facilitate very particular types of gameplay or emulate certain other things that exist in fiction.

As an earlier Anon noted, Apocalypse World and other games based on its mechanics (Powered by the Apocalypse) are narrativist games. You roleplay the fiction of the game, and when you trigger the conditions for moves, you roll 2d6+bonuses and gauge degrees of success (full hit, partial hit, miss) to see how well it goes and if there are consequences. The GM effectively just lets the players act and the world responds. There are a vast number of PbtA games out there, each one with a different genre or other niche purpose. Some of them are kind of just lazy and bad, but there are some very legitimately good ones.

Another game of note is Dogs in the Vineyard (and its settings-agnostic copy DOGS). Characters have traits, bonds, and equipment with dice values, and when those things come up in a conflict, you roll those dice and add the results to a pool. Then during the conflict, you sort of put up those numbers in a back and forth against what the problem is, in a style that is vaguely reminiscent of playing hands of cards. When you get rekt, those dice become a pool you roll at the end to see what consequences you took, which can also lead to growth. You are incentivized to take risk while trying to manage that risk.

If you've never seen it, you should check out Call of Cthulu or its other game Delta Green. This uses the Basic Roleplaying: Universal Game System. You attempt to roll under a value of your skill, and there's room for gradual growth. Those specific games that I mentioned also have stats for your characters going mad from encounters with things that humans can't handle or comprehend safely, which grates on them gradually until it potentially spells their doom by insanity.

Absolutely fantastic, thank you to everyone who posted here. Making a list, and am going to look through all of them when I have more time. Being able to break away from 5E is a blessing, so I look forward to it, and don't want to frick up the execution.

Wrath & Glory has a "Wrath dice" system. All rolls are done with a number of D6s wqual to the skills being used, but one dice is always marked apart from the other as the Wrath dice. This is what determines is the roll is a critical success (6) or failure (1). This means that a player can roll low enough to fail, but still gain something if the Wrath duce was a 6. Likewise, a roll can succeed, but if thr Wrath dice was a 1 something unfortunate happens as well. Other notable features of the system are how initiative is handled. One of the PCs always go first (barring some narrative reason like the BBEG making his appearance), and it alternates between PC and enemies. Another is Glory tokens, which reset each session. There is a numberbof them equal to the players, and these can be used for rerolls or adding something reasonable to a scene, but anyone can take them any time. Since they reset each session players are encouraged to ise them and be involved in the storytelling beyond combat.