“The prime motive was the desire of a tale-teller to try his hand at a really long story that would hold the attention of readers, amuse them, delight them, and at times maybe excite them or deeply move them. As a guide I had only my own feelings for what is appealing or moving, and for many, the guide was inevitably often at fault.”

- J.R.R. Tolkien, Foreword to the second edition of The Fellowship of the Ring

Why did J.R.R. Tolkien consider his own universe to be woefully flawed and ultimately incomplete? Why are there so many inconsistencies in world-building between The Hobbit and The Lord of the Rings? Why are there no Hobbits in The Silmarillion?

And who REALLY wrote The Lord of the Rings?

The answers to these questions lie buried beneath a century’s worth of history waiting to be uncovered. And in digging for the truth, we’ll discover the real John Ronald Tolkien: World War I lieutenant, scholar, and writer, haunted all his life by a creative vision that would never come to fruition during his lifetime.

Here’s how one man’s creative adaptability in the face of failure led to the creation of his masterpiece.

Tolkien at War and the Lost Tales

Our story starts with a kid named John who liked fairytales. He was born in South Africa In 1892, but by the time he was 12, both his father and mother had died, leaving him and his younger brother orphans. They were sent to live under the care of a Catholic priest in England. John’s love of fairytales didn’t fade with age, and he became keenly interested in languages, even inventing some of his own.

Eventually, John would undertake a course of study at Oxford, where his interest in language led to studies in English literature. He and several of his undergraduate friends, brought together by their similar interests, formed a sort of literary club. But everything changed when John’s generation was called to serve his nation in The Great War.

A quick side-note here: You’ve got to understand that World War One was war like no one had ever seen it before. This conflict came at a time when the tactics of war had not yet caught up with innovations of war, like the machine gun, the flamethrower, chemical weapons, and the tank. And the innovations didn’t develop evenly; while some countries were sending cavalrymen into battle on horseback, the same battlefields were being strafed by machine gunner fire from planes.

The art of killing had become highly efficient. And in a war where battles were still being fought using traditional tactics, soldiers assembling in orderly lines and marching forward across open battlefields, entire companies could be mowed down in a few quick machine gun blasts. There are accounts of German machine gunners refusing orders to fire at retreating British soldiers out of pity for the victims of the slaughter.

This was the war that John stood up and volunteered to serve in.

John served as a Second Lieutenant. His unit was at the Battle of the Somme, an offensive between the forces of England, France, and Germany. Out of the estimated 3 million men who fought at the Somme, 1 million were either wounded or killed, making it, to this day, one of the deadliest battles in all of human history. John’s unit was almost completely wiped out at the Somme, and he might well have numbered among the casualties had a bad infection of trench fever not caused him to be transferred home just days before the worst of the fighting.

The Quest for Publication

By the time the war was over, nearly every one of his childhood friends — including most of the members of his undergraduate literary club — were dead.

John had kept up writing, even during the war.

He had started working on a collection of stories he called The Book of Lost Tales. In it, his early love of fairytales was combined with a new appreciation for a recently published Finnish book of mythology known as the Kalevala. One of the stories in The Book of Lost Tales, “The Fall of Gondolin,” featured a race of elves making their last stand against an invading force, but ultimately being defeated. As John would later write, “A real taste for fairy stories was wakened by philology on the threshold of manhood and quickened to full life by war.” Overall, the Lost Tales read more like fictional history than novels, and those who read them found them striking if a little odd.

Recurring illness kept John out of most of the rest of the war. Afterward, John began an academic career, eventually taking a position as a Professor of Anglo-Saxon at Oxford. For a time, he worked as one of the writers of the Oxford English Dictionary, and he translated Middle-English works like Sir Gawain and the Green Knight and Beowulf.

But all the while, he was writing his own fiction, too.

Inspired by fairytales, the Kalevala, and Anglo-Saxon literature, John began to write what he conceived of as a fictional mythology of England. The concept was that this story of his was really a translation of recovered texts taken from a time before history, which told of a sunken continent, elves, a creator god with a pantheon of angelic beings, and even his own invented languages.

Now, you might assume that someone with John’s academic qualifications and connections would have no trouble getting his fiction published. Not so. His work was imaginative, expansive, rich in lore — and very dense. John had a few early successes with having his fiction published, but when it came to trying to sell his wider, overarching narrative, the answer was always the same. It was too grandiose. Too dry. There was no market for it.

For decades, John continued to work on his mythology privately. There are boxes upon boxes worth of handwritten papers from this time. Much of these were written during the years of World War II, when there was paper rationing in effect, so some of his work is written on the backs of old scrap paper. Other papers, you can see, he erased entirely, from top to bottom, and rewrote over just to salvage some writing space.

And yet, for decades, he was met with nothing but rejection. Few people read his work except for his wife Edith and a group of close personal friends that met weekly at a local pub in Oxford.

Well, that and his four children.

The Creation of The Hobbit

Aside from his great mythology, John also created bedtime stories for his kids. Stories he would make up for them on the fly. They were a lot like the fairytales and imaginative adventures he had enjoyed as a child.

Then one day, John (now better known by his academic title, Professor Tolkien), was grading exam papers. It was a process he described as “laborious” and “boring.” In the middle of his work, he picked up the next page in the huge stack of essays and found that the page had been left blank. In his own words:

“Glorious! Nothing to read. So I scribbled on it, I can’t think why, ‘In a hole in the ground there lived a Hobbit.’”

Professor Tolkien’s spontaneous idea for a creature called a “hobbit” became the inspiration for a new bedtime story for his children.

Tolkien had made up a story for his children about a funny little fellow called Bilbo Baggins who was recruited by a wizard to steal treasure from a dragon. As the story expanded, he began to have trouble remembering it all. When his children took to correcting him on the details, he realized he would have to start writing it down. Before long, it was a lengthy narrative with a beginning, middle, and end.

Tolkien suddenly had a children’s book on his hands.

As John Garth writes in Tolkien and the Great War: The Threshold of Middle Earth, “Tolkien had started writing stories for their children as early as 1920, when he first sent John a letter purporting to be from Father Christmas. That year, Edith had a second son, Michael, and in 1924, a third Christopher. In 1929, a daughter, Priscilla, was born. It was for their entertainment that he wrote The Hobbit, showing it to an enthusiastic C.S. Lewis in 1933.”

Side note: I’m never surprised when somebody tells me they tried to read The Hobbit but just couldn’t get into it. It’s quite unlike a modern fantasy novel. In fact, especially in the earlier chapters, it’s very episodic in nature, with a series of self-contained misadventures. Almost like, you know, someone was coming up with quick, digestible stories to get their kids to go to sleep.

Tolkien wasn’t a children’s writer. Not in his mind, anyway. The epic, sprawling mythologies he’d created on his own time used esoteric, grandiose language and told tales of valor, romance, and tragedy. LOTS of tragedy. It was all very serious — and frankly, pretty difficult to read. His biggest problem was that his work just wasn’t very accessible to a non-academic audience.

Compare that to the stories he was telling his children. Those, he filled with silliness, comically inept characters, songs, jokes, riddles… And he was getting feedback from his children in real-time, so he knew which parts were keeping the kids engaged with thrills, scares, and laughs. The story was still epic in scale and full of adventure, but it was also lighthearted and, well, entertaining. Frankly, it was just plain fun! Plus, the main character was sort of kid-like. He wasn’t a great warrior. He was totally unprepared for the quest he was on. All he wanted to do was go home and eat. Sounds like some kids I know. Yet he eventually became a greater hero than any of the other characters around him.

This more approachable style is what led to The Hobbit, a new kind of fairytale book for children, being published in 1937, and just like that, J.R.R. Tolkien became a household name.

But here is where the story gets interesting.

All the time that Tolkien was writing The Hobbit and becoming a huge success, he hadn’t given up on his original idea: his grand mythology. By now, it had a name. The Silmarillion.

To Tolkien, children’s books were well and good, but The Silmarillion was his passion. And since The Hobbit was now a best-seller, surely he would be able to share his real masterpiece with the world.

Nope.

Tolkien was met with nothing but rejection. Not even his wild success as a best-selling author was enough to convince publishers to take a chance on his odd book of legends, battles, and… elves. They didn’t want all that. What they wanted was a sequel to The Hobbit.

Even after all of the runaway success of his first book, he couldn’t sell his passion project. At this point, anything with Tolkien’s name on it would have sold! They must really have not been interested.

That’s what you call failure.

"I Don't Much Approve of The Hobbit"

Tolkien was understandably discouraged. In letters to friends and colleagues, it’s clear that while he appreciated The Hobbit for what it was, he considered it altogether separate from his serious work. In one letter in 1937, he even wrote, “I don't much approve of The Hobbit myself, preferring my own mythology.”

Readers (and publishers) wanted hobbits. And The Silmarillion was no place for hobbits.

It seemed like this thing Tolkien had been working on for decades would never see the light of day.

Reluctantly, and halfheartedly, Tolkien took up his pen and began writing a sequel to The Hobbit. There were many false starts, and he seems to have gotten discouraged and abandoned work on it entirely at several points along the way. At one point in 1938, he wrote, “The Hobbit sequel is still where it was, and I have only the vaguest notions of how to proceed. Not ever intending any sequel, I fear I squandered all my favorite motifs and characters on the original Hobbit.”

But then, Tolkien made a choice that would change the entire landscape of literature forever. He decided, they want hobbits? Fine.

I’ll give them hobbits…

Writing the First Lord of the Rings

The “Hobbit sequel,” as it was initially referred to, began to take on a very different form. Soon, it was no longer a simple sequel. It grew ever more expansive as he went. Sometimes, it was like he was discovering the story even as he wrote it. In a letter on June 7, 1955, he wrote, “I met a lot of things on the way that astonished me. Tom Bombadil I knew already; but I had never been to Bree. Strider sitting in the corner at the inn was a shock, and I had no more idea who he was than had Frodo. The Mines of Moria had been a mere name; and of Lothlórien no word had reached my mortal ears till I came there. Far away I knew there were the Horse- lords on the confines of an ancient Kingdom of Men, but Fangorn Forest was an unforeseen adventure. I had never heard of the House of Eorl nor of the Stewards of Gondor. Most disquieting of all, Saruman had never been revealed to me, and I was as mystified as Frodo at Gandalf's failure to appear on September 22.”

Tolkien wrote and wrote, and rewrote and rewrote. Like, I really can’t stress the rewriting part enough. A great deal of his work exists in multiple versions, especially the earliest chapters of his work, all written in careful longhand in some places, and in barely comprehensible scribbles when he got excited. He would write a chapter, set it aside, return to it, add ideas in the margins, decide he wanted to change it, then write the entire thing over again from scratch, beginning to end. Several times. The papers still exist with these many versions of chapters on them. In fact, in February of 1938, he wrote, “They say it is the first step that costs the effort. I do not find it so. I am sure I could write unlimited first chapters. I have indeed written many.”

When all was said and done, it had taken Tolkien twelve years to complete his “Hobbit sequel.” And the final result was not the single-volume children’s book his publishers had been expecting. Instead, Tolkien presented them with a quest that spanned six books. After some pushback from the publishers and a whole lot of editing, these six books were trimmed down to three. But it was still a heck of a long story.

He had given them hobbits, all right. But those hobbits were now wrapped up in a much larger world.

With the world of The Silmarillion as a backdrop, and a seemingly unimportant item from The Hobbit as a plot device (a magic ring Bilbo had accidentally stumbled upon in a cave), he tied the two stories together, creating an epic quest for his protagonists that placed the fate of the entire world in their little hands.

The rest, of course, is history. The three volumes that would be collectively entitled The Lord of the Rings, published between 1954 and 1955, basically invented the modern fantasy genre and would go on to sell over 150 million copies worldwide, making them the best-selling novels of the 20th century. Only one book outsold them. The Bible.

Tolkien’s Two Separate Universes: No Connection with The Hobbit

Can we just pause for a moment to appreciate what we’ve just discovered? Because even after reading about this and studying Tolkien’s work, this fact still blows me away:

The Hobbit was never supposed to be a part of The Lord of the Rings world. These two stories were intended to take place in two separate, unconnected worlds.

Don’t believe me? Here it is in Tolkien’s own words!

The following is from a letter written by J.R.R. Tolkien in July of 1964.

“I returned to Oxford in Jan. 1926, and by the time The Hobbit appeared (1937) this 'matter of the Elder Days' (The Silmarillion) was in coherent form. The Hobbit was not intended to have anything to do with it. I had the habit while my children were still young of inventing and telling orally, sometimes of writing down, 'children's stories' for their private amusement. The Hobbit was intended to be one of them. It had no necessary connexion with the 'mythology.’”

Even though Tolkien created a huge world with its own mythology and history and geography and languages, it was not his intention that The Hobbit would be a part of that world. But no one would give his greater work a chance.

So… he improvised.

Tolkien crammed the world of The Hobbit into Middle Earth like a puzzle piece jammed right smack in the center, and he adjusted his plan to fit everything else around it. If that was what it would take to tell his story, that’s what he was going to do.

Did Tolkien have to do this? No. Could he have kept plugging away at The Silmarillion, refusing to compromise on his artistic vision? Sure.

But if he had, The Lord of the Rings, some of the highest-selling books of all time, the precursor to the entire fantasy genre, never would have happened.

Once you realize that The Hobbit wasn’t originally meant to be connected to Tolkien’s greater work, you can see the evidence for it everywhere.

Gandalf is a wizard in The Hobbit. Well, this presented a problem, because there weren’t meant to be wizards in Tolkien’s mythology. Those were for kids’ stories! So, “wizard” became something that the simpler folk of Middle Earth called the type of being that Gandalf actually was (which you can look up if you want – it’s pretty interesting).

How about the dwarves? For me, the differences between the dwarves in The Hobbit and Gimli in The Lord of the Rings has always been a huge disconnect. The Peter Jackson Hobbit films had to make some changes for sake of consistency by making the dwarves more like Gimli. But the dwarves in the book are comically inept. They have no idea what they are doing. They keep getting into trouble at every turn, they’re almost eaten, they get lost, they get captured a few times — at the start of the book, they don’t even have swords or anything! Compare that to Gimli in The Lord of the Rings, who is a one-dwarf army just mowing down orcs and keeping a tally of his kills. This difference always seemed so odd to me. But now we know why. Originally, these were two separate universes, with two very different ideas of what the essential qualities of dwarves were.

World-Building Inconsistencies in Middle-Earth

As much as it pains me to say it, there are some major world-building inconsistencies in Tolkien’s work. Some of the stuff mentioned in the Shire is several hundred years ahead of what we find in the rest of Middle-Earth. Tolkien even said at one point that the Shire was supposed to be a typical rural English village around the “Queen’s Diamond Jubilee,” which would make it the era of Victorian England. I mean, Hobbits use umbrellas and have indoor plumbing. To top it off, The Hobbit makes a reference to the game of golf being invented. Whereas, in The Lord of the Rings, you get Rohan and Gondor, which feel more like Anglo-Saxon England (pre-1066).

These differences can be jarring at times, and Tolkien knew this. At one point, he had actually planned to go back and revise The Hobbit to give it a tone and style that more closely matched The Lord of the Rings. But he never did. So these disconnects remain as the legacy.

And this is just one of the reasons that Tolkien, all his life, considered his stories to be woefully flawed and incomplete.

“The most critical reader of all, myself, now finds many defects, minor and major,” Tolkien wrote in his Foreword to the second edition of The Fellowship of the Ring, “but being fortunately under no obligation either to review the book or to write it again, he will pass over these in silence, except one that has been noted by others: the book is too short.”

Unfortunately, this is where the story gets a little sad. Because Tolkien’s real work, the thing he had dreamed of sharing with the world… He never managed to complete it. It remained unpublished in his lifetime.

As to the reason, I’ll defer to fantasy literature scholar Dimitra Fimi:

“I'm often asked why Tolkien didn't manage to finish revising The Silmarillion for publication. Well, Tolkien’s turn from writing a mythology to writing fantasy novels from the late 1920s onwards transformed his invented world and created an additional layer of complexity in his already many times revised legendarium. As we saw, he tried, he really did, to revise the earlier mythology so that it could now be consistent and integrated with The Lord of the Rings. But the vision had changed too much. As Christopher Tolkien once noted, it was not that his father could not finish The Silmarillion, what he could not finish was the post-Lord of the Rings Silmarillion. He could not satisfactorily integrate two different visions of his world: the mythological, the bigger picture and vision of a large canvas, and the novelistic, with its minute details and specifics. When Christopher Tolkien eventually edited The Silmarillion, he selected versions of the main stories, some from earlier periods, in which case he had to regularize things, and most from later versions.”

So, what’s at the core of this story? Is Tolkien a genius for imaginatively reframing his work? Or did he compromise on his artistic vision? How do we reconcile these two ideas?

Here’s the thing about The Silmarillion. Tolkien considered it to be his real legacy: his great life’s work. He loved it… But nobody else did.

Meanwhile, The Hobbit unexpectedly became a runaway success. So, over here was the thing he loved that nobody wanted. And over here was the thing that everybody wanted but that he didn’t really love that much.

Sure, Tolkien could have stuck to his guns and refused to compromise on his vision. But instead, his creative adaptability ended up turning his failure into a massive success.

All that said, I’m sorry to say that Tolkien never was able to finish The Silmarillion during his lifetime. There’s a whole side story about why this is, and it has to do with his inability to settle a few key problems between the purely secondary world of Middle Earth and the prehistorical, mythology of England he’d originally envisioned for The Silmarillion. Apparently, in his later years, he kept going back and forth over which version to make official and just couldn’t figure out how to make it work to his satisfaction.

So, in the end, he never did finish it. His son Christopher made it his mission to complete it, which he did, and J.R.R. Tolkien’s true life’s work would be published in 1977, years after his death.

And you want to know the biggest complaints people have about The Silmarillion? It is dry, grandiose, and difficult to read.

Who Really Wrote The Lord of the Rings?

As with everything J.R.R. Tolkien did, there was a story behind the story. His original idea of presenting a mythology as if it had been recovered from lost writings? He ran with it.

According to Dimitra Fimi, “The supposed source for The Hobbit and The Lord of the Rings is the Red Book of Westmarch. Tolkien describes the textual transmission of the Red Book of Westmarch as follows. First, the original manuscript included Bilbo's private diary, so the story of The Hobbit, Frodo's account of the War of the Ring completed by Sam, so the story of The Lord of the Rings, Bilbo’s translations from Elvish sources in Rivendell, and Hobbit lore collected in the Shire. Then we're told the Red Book was copied by scribes in Minas Tirith after the War of the Ring. When they copied it, they also included annotations and corrections and added parts of the tale of Aragon and Arwen, a story that is told in the appendices of The Lord of the Rings. A copy of this particular augmented compilation was then made in Minas Tirith and brought back to the Shire, where the hobbits added marginalia, scribbles on the margins. This, claims Tolkien, is the source he used for The Hobbit, The Lord of the Rings, as well as his later poetry collection, The Adventures of Tom Bombadil, tellingly subtitled, And Other Verses from the Red Book.”

That’s a lot to take in. And it’s why, when people say Tolkien’s work is unlike anything else ever written, it’s not just rhetoric. He thought of everything. If you check the appendices in the Lord of the Rings, you’ll see that every main character has a family tree tracing their lineage back multiple generations. But Tolkien even had a family tree in mind for the imaginary translation he was supposedly using to tell his story! Incredible.

In terms of how this all fits into the wider legendarium, again, I’m going to quote Dimitra Fini: “I think what Tolkien was planning, but he didn't live long enough to do it, was to present The Silmarillion, when he would finish it at last, as the bit from the Red Book that is unaccounted for, Bilbo's translation from elvish material in Rivendell. Christopher Tolkien published The Silmarillion “neat.” When you open The Silmarillion, it begins as if you're reading the Bible of the elves, with the Godhead creating the vision of the world. But towards the end of The Lord of the Rings, Bilbo gives Frodo three books of lore that he had made at various times, written in his spidery hand, and labeled on the red backs, ‘Translations from the Elvish by B.B.’ These three books were surely meant to be the framework by which The Silmarillion could have been presented, and that framework perhaps would have made the book more accessible, less obscure, less difficult. In fact, Christopher Tolkien later wrote that he was almost certain that was the case, but he had been reluctant to add this framework himself when his father had never managed to do so.”

Uninvited Guests at an Unexpected Party

If you ever do manage to read The Silmarillion, you may notice something rather interesting. Despite the fact that The Silmarillion is the equivalent of Tolkien’s Bible for his fantasy world, including the account of the creation of the universe, thousands of years’ worth of history, the origins of every species including humans, elves, dwarves, and others, something’s missing. Can you guess what it is?

Hobbits. There is not a single mention of hobbits.

You will not find a hobbit in the entire text. There’s nothing in there to explain what hobbits are, where they come from, how they are related to the other species of the world, how they fit into the history — not a thing about them.

Now you know why. Hobbits were the uninvited guests at an unexpected party.

Personally, on the question of whether Tolkien was a genius for reframing his work or if he compromised on his vision, I tend to fall into the first category. Partly just because I’m a fan of his work (The Fellowship of the Ring is my favorite book, and I’ve read the dry, grandiose Silmarillion multiple times). But mostly, it’s because, without an audience, art is purely self-indulgent. And that’s fine. Art purely for the sake of the enjoyment of the artist is a great thing.

But if you want an audience, which many creators do, then you might have to be willing to take some feedback and revise. That’s just what Tolkien did. And I’d say it paid off.

Because…

Here’s the thing: Hobbits are what really make The Lord of the Rings work!

Think about it! Frodo, Sam, Merry, and Pippin are like uninvited guests in Tolkien’s wider mythology. But, in a strange way, it is this juxtaposition between a sprawling, epic world full of heroes and history and magic, and a viewpoint of the story seen through the eyes of a few outsiders — literal outsiders, they were never meant to even be in the story to begin with — that gives The Lord of the Rings some of its essential qualities.

Yes, we feel lost in this huge world and totally out of our element. And so do the main characters. All they want to do is go home! Wouldn’t you?

We can relate to the lost, hapless hobbits better than we can relate to any other characters in the story. Tolkien framed The Lord of the Rings as the hobbits’ memoirs about going on an adventure that the folks back home would hardly believe. We feel like we’re discovering much of the story right alongside the main characters as they are swept up in this world. Without this dynamic to sort of anchor the entire story, it wouldn’t feel quite the same.

From Failure to Masterpiece

So, I guess the question is, was Tolkien’s inability to get his great masterpiece published in his lifetime a failure? Well, I think it was a setback, certainly. Much the same way that a case of trench fever was a setback that just happened to pull Tolkien from the front lines only a few days before he might have been killed in one of the deadliest battles ever fought.

How about you?

Will you be defined by your failures? Or will it be your ability to adapt to them, and revise in the face of rejection, that leads to your true masterpiece?

A special thanks to my Patreon patrons, who help make my content (my books, my blog, and my videos) possible.

Also, thank you to the following sources. John Garth’s Tolkien and the Great War, which is so interesting I’ve read it multiple times now. Dimitra Fini’s series of lectures from The Great Courses entitled The World of J.R.R. Tolkien. I also drew on Tolkien’s own words in the form of his essay entitled On Fairy-Stories. Thank you to Corey Olsen, aka The Tolkien Professor, who makes his lectures available as a podcast. And, most of all the unbelievable scholarship of Christopher Tolkien’s 12-volume series, The History of Middle Earth.

The following is a list of the books mentioned in this post, the sources used during my research, and some recommended further reading. Please note that as an Amazon Associate, I earn from qualifying purchases on the following product links.

The Silmarillion: https://amzn.to/3Zo83J0

The Lord of the Rings: https://amzn.to/3xWrEEF

The Hobbit: https://amzn.to/3Y88MwP

Exploring J.R.R. Tolkien’s The Hobbit by Corey Olsen: https://amzn.to/41wRhJJ

The History of Middle-Earth by Christopher Tolkien: https://amzn.to/3xZWQCN

The World of J.R.R. Tolkien by Dimitra Fimi: https://amzn.to/3KLuK5V

Tolkien and the Great War: The Threshold of Middle Earth: https://amzn.to/3EGAERN

Check out my novels here:

The Fallen Odyssey: https://amzn.to/3YdGSj4

The Fallen Aeneid: https://amzn.to/41ybBKE

Shadows of the Fallen: https://amzn.to/3L2Fhdh

A Knife in the Dark: https://amzn.to/41wOMqP