How do they know the title of each book in the citadel library in Game of Thrones if the book's back faces the wall?

Having books with the spine facing out is a relatively recent innovation.

For the record, when you tuck a book away with the title-bearing spine pointing out, you’re carrying on a tradition that began roughly 480 years ago. “The first spine with printing dates from 1535, and it was then that books began to spin into the position we’re familiar with,” says Mari.

But before book, there were scrolls, and that’s where Mari’s story starts.

---

<sub>- Libraries Used to Chain Their Books to Shelves, With the Spines Hidden Away
- Smithsonian</sub>

Since Game of thrones is a fantasy and seems to be (comparatively) set in what would have been something close to Earth's "Middle Ages" with knights and kings all warring constantly this seems to be a pretty accurate depiction of a library.

As scrolls gave way to books, new shelves and a new organizational system were in order.

For the next fourteen hundred or so years, books, as Henry Petroski, a professor of civic engineering and history at Duke, writes in The Book on the Bookshelf, were shelved every which way but straight up, spine out. Engravings of private studies show books piled horizontally, standing on the edge opposite their spine (their fore edge), as well as turned fore edge out.

Before the printing press books were ornate constructions, and in comparison to what came after they were both highly valuable and in short supply.

In the Middle Ages, when monasteries were the closest equivalent to a public library, monks kept works in their carrels. To increase circulation, these works were eventually chained to inclined desks, or lecterns, thus giving ownership of a work to a particular lectern rather than a particular monk.

When space got tight the monks moved their books to shelves, but they stacked them with the spines hidden. Which, as you can imagine, would have been quite confusing. The solution, Mari says: “Sometimes an identifying design was drawn across the thick of the pages.”

---

<sub>- Libraries Used to Chain Their Books to Shelves, With the Spines Hidden Away
- Smithsonian</sub>

So, to answer your question about how they were identified: there were sometimes identifying markings on "The thick of the pages".

Originally the shelf marks and titling of a book was written on its fore-edge, the paper part of the book, not on the back (the spine) the way they are today.

The GOT library is modelled after the chained library of Hereford Cathedral. In that library the books have only shelf marks and they are written on the fore-edge, so the fore-edge is the part of the book facing outwards.

_{Image shared by Hereford Cathedral's official Twitter account to mark
Book Week 2018}

The chaining of books was the most widespread and effective security system in European libraries from the Middle Ages to the 18th century, and Hereford Cathedral’s 17th-century Chained Library is the largest to survive with all its chains, rods and locks intact.

A chain is attached at one end to the front cover of each book; the other end is slotted on to a rod running along the bottom of each shelf. The system allows a book to be taken from the shelf and read at the desk, but not to be removed from the bookcase.

The books are shelved with their foredges, rather than their spines, facing the reader (the wrong way round to us); this allows the book to be lifted down and opened without needing to be turned around – thus avoiding tangling the chain.

_{Quote from the official Hereford Cathedral's page on the Chained Library}

Our eyes are playing tricks on us. We look at these shelves with all the books and our mind is telling us that the books are turned around, when in fact they are representing books the way they were bound in that era.

Book binding has evolved over the years. Modern books have covers that wrap from the front and around one side to the back with pages sewn or glued to the side spine. But years ago, there was no spine. Pages were often tied together with heavy wood or (other material) for front and back covers. (see image below)

When we see the books on the shelves of the Citadel, we "see" them as turned around because that is how we are accustomed to seeing our modern day books (i.e.; no spine = other side of the book). What we are really seeing is a more accurate depiction of a library in that age and time, with no spines at all. Yes, Sam is holding books with a spine. Some books in the library will be older than others. Hence, different binding processes.

Wikipedia Bookbinding