The UI is the bridge between the player and the character they are controlling. I have to say that my favorite UI's are the ones that are so well integrated in the game that they are in game, much like the health meter for Spyro the Dragon. Whenever Spyro took a hit the dragon fly's colors changed from gold, to blue, to green, and ultimately disappearing entirely, leaving Spyro vulnerable to defeat upon the next hit. To prevent that, Spyro attacked small creatures called "fodder" to release a Butterfly, which Sparx devours, to recover one hit point in the process. The health meter was a character, a essential part of the story.
Another great example is Dead Space 2, where the character's health meter is displayed on his back as part of the character's suit. It seems amazing how these people can integrate the HUD inside the game, which can help immerse the players even more into that world. For me, personally, that improves the experience immensely. If I can achieve something that would immerse our players even more into our story and world, then I've done it right. I don't think I'm quite there yet, I still have a lot more exploring to do, but so far I've been having fun.
Here are a few sketches I've come up for the Gremlins and Gears UI. Most of it is rather traditional, but I tried my best to add the GG flare to it.