bindery/README.md

Bindery

Bindery downloads web fiction into a Calibre-Web-Automated (CWA) library and keeps the books up to date. It uses FanFicFare, so it supports the sites that FanFicFare supports. The tested site is RoyalRoad.

How it differs from FanFicFare

FanFicFare builds a book from the chapter list on the site, so a chapter that the author removes also goes from your book. Bindery keeps every chapter that it downloads in its own database, and builds each ePub from that database.

• Chapters that the site removes stay in the book. A new chapter goes after the chapter that comes before it on the site. For example, you have chapters 1 to 60, and the site now lists only 1 to 4 and 61. The book then has chapters 1 to 61.
• A changed chapter keeps its old text as a revision. You can compare the revisions and choose the one that goes in the book.
• Replacement rules run when Bindery builds the book. You can change a rule and rebuild the book without a new download.

Quick start

You need CWA v4.0.1 or later. Older versions can import an update as a new book. See "Limitations".

1. Get the code, and make the folder for Bindery's data.

   git clone <this repository> bindery
   cd bindery
   mkdir config
   

2. Edit docker-compose.yml or environment variables. Set the CWA ingest folder, the CWA address and account, and the user and group of the container.

3. Build and start the container.

   docker compose up -d --build
   

4. Open http://localhost:8085, and log in with a CWA user.

The container runs as the user in docker-compose.yml. That user must be able to write to ./config and to the CWA ingest folder. Usually this is the same user and group as CWA.

Configuration

Environment variables set the configuration. Restart the container after a change.

Variable	Description	Default value
CWA_URL	The address of CWA, for example http://calibre-web-automated:8083	None
CWA_USERNAME, CWA_PASSWORD	The CWA account of Bindery	None
CWA_DB_PATH	Calibre-Web's app.db, for logins. See "CWA logins".	None
CALIBRE_DB_PATH	The calibre metadata.db, for book ids. See "How books reach the library".	None
UPDATE_INTERVAL_HOURS	Hours between checks for new chapters. 0 turns them off.	24
DEFAULT_LOCKED_FIELDS	Fields that new stories lock: title, authors, comments, tags, series, pubdate, languages	title
AUTH_METHOD	none, cwa or proxy. See "Logins".	none
CWA_ALLOWED_USERS	The CWA users that can log in. Empty allows every CWA user.	Empty
PROXY_AUTH_USER_HEADER	The header in which a reverse proxy sends the user name	X-Remote-User
PROXY_AUTH_LOGOUT_URL	Where the Log out button goes in proxy mode	None
BINDERY_API_KEY	A key for scripts. See "Scripts".	None
SESSION_COOKIE_SECURE	true if users reach Bindery only over HTTPS	false
TRUST_PROXY	true if a reverse proxy sends the client address. See "Lockouts and sessions".	false
URL_BASE	A path prefix, such as /bindery. See "Reverse proxy".	None
INGEST_DIR	The CWA ingest folder	/cwa-book-ingest
CONFIG_DIR	The story database and personal.ini	/config
TMP_DIR	The folder for ePubs while Bindery builds them	/tmp/bindery
LOG_LEVEL	The log level	INFO

Separate the values in a list with commas.

Logins

AUTH_METHOD sets how users log in:

• none: Bindery asks for no login. Use this only on a private network, or behind a proxy that does the login.
• cwa: Users log in with their CWA user name and password. See "CWA logins".
• proxy: A reverse proxy, such as Rauthy, Authelia or Authentik, logs users in and sends the user name in PROXY_AUTH_USER_HEADER. Make sure that users can reach Bindery only through the proxy, because Bindery trusts this header. If a request gets a 401 because the proxy session ended, Bindery reloads the page, so the proxy can step in again.

If the login settings are incomplete or wrong, Bindery refuses every request, also with the API key, and shows the reason.

CWA logins

AUTH_METHOD=cwa needs CWA_URL, CWA_DB_PATH or both. Without CWA_DB_PATH, Bindery sends the user name and password to CWA_URL/opds as an HTTP Basic login. This is the OPDS login.

With CWA_DB_PATH, Bindery checks the password against the account in Calibre-Web's app.db. A wrong password or an unknown account fails at once. If Bindery cannot read the file or its password hash, it uses the OPDS login.

CWA_DB_PATH is the path of app.db inside the container. To use it, mount the CWA config folder read-only into the container. Mount the whole folder, so that SQLite can read the journal files beside the database. Bindery does not write this database, and access should ideally be read-only.

Database logins check only the password. CWA's external login providers and multi-factor authentication do not apply. If they must apply, use the proxy mode.

• The CWA guest user cannot log in.
• If CWA_ALLOWED_USERS is set, only those users can log in. Bindery checks this list on each request.
• CWA_USERNAME and CWA_PASSWORD are only for the library lookups of Bindery.

Lockouts and sessions

The cwa mode locks out password guesses:

• After 5 wrong passwords for one user name from one address in 15 minutes, Bindery locks that user name for 15 minutes, for that address only. The real user can still log in from another address.
• After 20 wrong passwords from one address in 15 minutes, Bindery locks that address for 15 minutes.

Behind a reverse proxy, every request comes from the address of the proxy, so all users share one lock. To give each client its own lock, set TRUST_PROXY=true. Bindery then takes the client address from the last entry of X-Forwarded-For. Set it only if clients can reach Bindery through the proxy alone. Otherwise a client can send a false address with each guess.

A session ends when the user logs out, and always after one day. Without "Keep me logged in", it also ends when the user closes the browser.

Bindery checks a password only at login. A deleted CWA user or a changed CWA password keeps its open session until that session ends. To stop a CWA user at once, remove the user from CWA_ALLOWED_USERS and restart Bindery. To end all sessions, delete /config/.flask_secret and restart Bindery.

Scripts

Scripts can use the JSON API of the web interface.

• In the cwa and proxy login modes, a script needs the API key. Make a long random key, for example with openssl rand -hex 32, and set it in BINDERY_API_KEY. Send it with each request as Authorization: Bearer <key> or X-Api-Key: <key>.
• In the none login mode, a script needs no key, but it must send X-Requested-With: bindery with each change.

This example adds a story. If Bindery tracks the story already, it downloads the new chapters.

curl -X POST http://localhost:8085/api/stories \
  -H "Authorization: Bearer $BINDERY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://www.royalroad.com/fiction/12345"]}'

The answer lists the queued jobs and the unsupported URLs: {"jobs": [...], "errors": [...]}.

Request	What it does
GET /api/stories	Lists the tracked stories, with their ids
POST /api/stories	Adds stories. The body is {"urls": [...]}.
POST /api/stories/<id>/download	Queues a job. The body is {"mode": "new"}. The modes are new, recheck (with "chapter_ids": [...]), recheck_all, full, metadata and rebuild.
POST /api/stories/import-ePub	Imports an ePub file, for example with curl -F file=@story.epub. Add the form field check=false to skip the check of the site. For one story, use /api/stories/<id>/import-ePub.
PUT /api/stories/<id>/metadata	Edits metadata. The body is {"values": {"title": "New title", "tags": ["a", "b"]}}. null removes an edit. The description, comments, is HTML.
POST /api/stories/<id>/approve	Approves the library link of a held story. The body is {"uuid": "<uuid of the library book>"}, or {"uuid": null} for a new book.
PUT /api/settings	Changes settings. The body {"pause_uploads": true} pauses all uploads to CWA.
GET /api/jobs	Lists the recent jobs. Add ?story_id=<id> for one story.

If Bindery runs under URL_BASE, put the prefix before /api.

FanFicFare settings

On first start, Bindery copies data/personal.ini to /config/personal.ini. It uses the format of personal.ini in the FanFicFare calibre plugin, so you can copy your plugin settings into it. The INI File page of the FanFicFare wiki explains the format, and defaults.ini lists every setting. Bindery reads the file for each job, so a change applies to the next job.

You can edit the file on the Settings page. Bindery refuses a file that FanFicFare cannot read, and shows FanFicFare's warnings for other problems. The file can hold passwords for story sites, and every user who can log in to Bindery can read them.

Bindery always sets always_reload_first_chapter, mark_new_chapters, continue_on_chapter_error, make_directories, zip_output and calibre_series_meta, because it needs fixed values for them.

Each book gets the cover from the site. FanFicFare downloads the cover only if include_images is on. If it is off, Bindery sets it to coveronly, which keeps the cover but leaves out the images in the story text.

Reverse proxy

To serve Bindery under a path, such as /bindery/, do one of these:

• Make the proxy remove the path before it forwards the request.
• Make the proxy forward the full path, and set URL_BASE=/bindery.

The health check at /api/health works with and without the prefix.

How books reach the library

Bindery gives every book to CWA through the ingest folder. CWA makes all changes to the library.

1. Bindery puts the ePub of a new story into the ingest folder. CWA imports it as a new book.
2. At the next update, Bindery finds the CWA book of the story. If you set CALIBRE_DB_PATH, it first looks for the story URL in the library database. Without a single match there, it searches the CWA OPDS feed (using CWA_URL, CWA_USERNAME and CWA_PASSWORD) for the story title. The search ignores the parts of a title in parentheses or brackets, such as tags or "(Book 1 Complete)". Bindery downloads up to 5 ePubs with a matching title, closest title first. The book whose ePub holds the story URL is the match. Bindery saves the id and uuid of the book with the story.
3. For the update, Bindery puts two files into the ingest folder: the new ePub, and a sidecar file <name>.epub.cwa.json that holds {"action": "add_format", "book_id": N}. CWA then replaces the ePub of book N.

Before each later update, Bindery finds the book by its uuid, first in the library database if you set CALIBRE_DB_PATH, then through CWA. If you deleted the book in CWA, Bindery unlinks the story.

CALIBRE_DB_PATH is optional. It is the path of the calibre metadata.db inside the container, and it lets Bindery find a book by its story URL instead of its title. To use it, mount the library folder read-only into the container. Mount the whole folder, so that SQLite can read the journal files beside the database. The Compose file has an example. Bindery does not write to this database, and access should ideally be read-only.

Be sure to use the library that gets the ingest files. A different library can make Bindery update the wrong book. Database lookups also find books that the CWA account of Bindery cannot see. If Bindery cannot read the database, it uses CWA for every lookup. Keep CWA_URL, CWA_USERNAME and CWA_PASSWORD set, because library imports and CWA lookups use them.

A book that the FanFicFare calibre plugin made holds its story URL. If you add its URL, or import it from the library, Bindery links that book and imports its chapters from the ePub.

If Bindery cannot find the book

The update fails, and the story page shows the error "Bindery cannot find this story in CWA". The chapters stay stored, and the next successful update includes them. This happens if:

• CWA did not import the ePub.
• The title of the book changed in CWA before Bindery linked it.
• The CWA account of Bindery cannot see the book.
• You deleted the book in CWA.

To fix it, go to "Library link" on the story page, and do one of these:

• Search for the book, and click "Link". Then click "Rebuild book" to send the update.
• Click "Send to CWA as a new book". Use this if CWA did not import the first version, or if you deleted the book.

Hold uploads

Two settings on the Settings page stop books from going to CWA. Bindery still downloads the chapters and builds each book. The story list marks a book that waits with "held".

• Hold new links for approval. Use this to catch a story that is linked to the wrong CWA book. The first upload of a new story waits for your approval. The first upload after a change of the library link also waits. This includes a link that you make, and a book that Bindery finds after the linked book is gone. The story page names the book that gets the update, or says that the story goes to CWA as a new book.
• Pause all uploads. No book goes to CWA. When you turn the pause off, Bindery rebuilds each book that waits and sends it.

To approve a link, open the story page and do one of these:

• If the book is right, click "Approve and send".
• If the book is wrong, search for the right book under "Library link", and click "Link". Then approve the new link.
• If CWA does not have the story, click "Send as a new book instead".

The book that CWA imports from the first upload of a story needs no approval. After an approval, later updates go to CWA as usual.

The CWA account

Make a CWA user for Bindery with these settings:

• The download permission. Bindery reads story URLs and chapters from the ePubs.
• "Show Publisher Section", in the sidebar settings of the user. CWA shows its list of publishers in OPDS only to users with this setting. "Import from the library" uses the list to find the books of story sites, because FanFicFare sets the publisher of a book to the site name.
• Access to all books. Bindery finds only the books that this account can see.

Bindery sends the password of this account with each OPDS request. Use a password that no other service uses.

Import an ePub file

If the site removed chapters before Bindery tracked the story, import an ePub that you made earlier with FanFicFare. Bindery stores its chapters as if it downloaded them. Chapters that the site publishes later go after them.

• To start to track a story from its ePub, use "Import an ePub file" on the stories page. For a story that Bindery tracks already, it adds the chapters that it does not have.
• To add chapters to one story, use "Import chapters from an ePub" on the story page. The ePub must be of that story.

Bindery accepts only ePubs that FanFicFare made, because it reads the story URL and chapter data that FanFicFare writes into each ePub. A chapter that Bindery stored already keeps its stored text. The other chapters go in the order of the ePub.

After the import, Bindery checks the site for new chapters and sends the book to CWA. If you clear "Check the site for new chapters after import", Bindery only rebuilds the book of a story that it downloaded before.

Download modes

Mode	What it does
Download new chapters	Downloads the chapters that are not stored yet. The default, and what scheduled updates use.
Recheck selected chapters	Downloads the selected chapters again. Changed text becomes a new revision.
Recheck all chapters	Downloads every chapter on the site again, and new chapters too.
Download and replace entire book	Downloads every chapter again. Chapters that the site no longer lists leave the book. They stay stored, and you can put them back.
Update metadata from site	Updates only the metadata in the ePub.
Rebuild book	Builds the book again from the stored chapters, for example after you change a rule.

Find edited chapters

Scheduled updates download only new chapters, so they miss most edits. To find edits, run a recheck, or use one of these settings:

• On the Settings page, "also recheck the last N chapters" rechecks the last N chapters with each download of new chapters. Authors most often fix their latest chapters. The default is 0, which turns it off.
• In the Updates section of a story page, "Recheck all chapters every N days" rechecks the whole story. This recheck takes the place of the next scheduled update. The story then ignores the "last N chapters" setting.

Each rechecked chapter is one request to the site, so keep the numbers small for long stories.

If the site changes the title of a chapter, "Download new chapters" and "Recheck selected chapters" download that chapter again. The chapter gets the new title only if the download works. On sites that number chapter URLs, such as FanFiction.net, this also finds chapters that moved when the author inserted a chapter.

Revisions

A recheck compares the text of a chapter with its newest revision, which is the text that the site sent last. Bindery stores the HTML as the site sends it, before the rules, so a rule never makes a chapter look changed.

• If the text changed, the new revision goes in the book. If you chose an older revision, it stays in the book until the site changes the chapter again.
• On the page of a chapter, you can compare revisions, choose the one that goes in the book, and delete the others. The revision in the book and the newest revision cannot be deleted.
• A chapter imported from an ePub holds its text after the FanFicFare settings and rules of that build. If the current rules turn the site text into the imported text, Bindery counts the chapter as unchanged, and stores the site text in its place. Other differences, for example from settings in personal.ini, show as a change.

Restarts

Bindery stores each chapter as soon as it downloads it, and keeps the job queue in its database. After a restart, the waiting jobs run. A job that was running starts again at the front of the queue, and skips the chapters that it stored. If Bindery stops during the same job three times, the job fails.

Backups

Once a day, Bindery copies its database and personal.ini to /config/backups, and keeps the copies of the last 7 days. The database is the only copy of the chapters that sites removed. The backups are on the same disk, so also copy /config to another disk or machine.

To restore a backup, do these steps:

1. Stop Bindery.
2. Copy the backup, for example /config/backups/bindery-2026-09-26.db, to /config/bindery.db.
3. If /config/bindery.db-wal or /config/bindery.db-shm exist, delete them. They belong to the old database.
4. Start Bindery.

Metadata

Bindery keeps the metadata of each story in its database, and writes it into each ePub. The fields are title, authors, description, tags, series, published date and language. Each field is locked or unlocked for each story:

• An unlocked field takes the site value on every update.
• A locked field keeps its stored value. The first download sets that value. For a story that you import from the library or from an ePub file, the value comes from that ePub.
• To update a locked field once, select it under "Also take these locked or edited fields from the site" when you start a download. The field stays locked.

You can edit each field under "Metadata" on the story page. An edit of a locked field replaces its stored value. An edit of an unlocked field stays until the site changes that field. To go back to the site value at once, click "Use the site value" or unlock the field. If Bindery sent the book to CWA before, it rebuilds the book after an edit.

• Separate authors with &, and tags with commas.
• Put the number of a series in brackets, for example Saga [2].
• The description is plain text, with a blank line between paragraphs.

DEFAULT_LOCKED_FIELDS sets the locks for new stories. The default is title. The publisher is always the site name.

CWA reads the metadata and the cover of an ePub only when it imports a new book. An update replaces only the ePub file. So edits that you make in CWA stay, and later changes on the site show in the ePub but not in the CWA catalog.

Replacement rules

A rule is a regular expression, a replacement and optional flags (i, s, m, x). Bindery uses the Python regex module, which accepts the syntax of Python's re module. For an introduction, read the Regular Expression HOWTO. The replacement can use groups, such as \1 or \g<name>. When Bindery builds a book, it runs the global rules from the Settings page, then the rules of the story from its page. The rules run on the HTML of each chapter. Each rule editor has a preview that shows one chapter before and after the rules.

If a rule takes longer than 10 seconds on one chapter, the build stops with an error that names the rule.

Some sites separate paragraphs with <br> tags. For those, turn on "Turn line breaks into paragraphs before the rules run" in a rule editor. Bindery then puts each piece of text between <br> tags in its own <p>, and drops empty pieces. A <br> inside other markup, such as italics, stays. This step runs before the global rules. The global setting is off by default, and applies to each story that has no setting of its own. To see the new paragraphs in the preview, select "Compare HTML source".

Limitations

• Bindery identifies a chapter by its URL. Most sites, such as RoyalRoad and AO3, keep the URL of a chapter. FanFiction.net and similar sites number chapter URLs by position. When an author inserts a chapter there, Bindery finds the moved chapters by their new titles. If the titles stay the same, run "Recheck all chapters". When an author removes a chapter there, the book loses that chapter, and the last chapter appears twice. The removed chapter stays as an older revision of the chapter that took its URL.
• "Stop tracking" deletes only the data of Bindery. The book stays in CWA.
• Updates need CWA v4.0.1 or later. CWA v4.0.0 added the .cwa.json sidecar files. But v4.0.0 can delete a sidecar file before it imports the ePub, and then imports the update as a new book. Versions before v4.0.0 cannot read sidecar files, so each update becomes a new book.

Development

You need uv. The frontend tests also need Node 22 or later.

Tests

Run these commands in the project folder:

uv sync                               # install the Python dependencies
uv run pytest                         # Python tests
uv run ruff check .                   # lint
node --test "tests/js/*.test.mjs"     # frontend tests

The Python tests use a fake CWA server and temporary folders, so they need no CWA and no story sites. The frontend tests use the built-in test runner of Node, and need no npm packages. Also check the pages in a browser.

Run locally without CWA

Without CWA, Bindery puts the first download of each story into its ingest folder as an ePub. Later downloads fail, because Bindery needs CWA to find the book, but their chapters stay stored. This is enough to work on downloads, chapters, rules and the web interface.

CONFIG_DIR=dev/config INGEST_DIR=dev/ingest TMP_DIR=dev/tmp uv run python -m bindery

Open http://127.0.0.1:8085. The folders go in dev/, which git ignores. To look at an ePub, use "Download ePub" on the story page.

Run locally with a test CWA

To test library lookups and updates, run a separate CWA container with its own empty folders. Do not use your real library.

1. Start a test CWA on port 8083. It shares dev/ingest with Bindery, and makes an empty library in dev/cwa-library on first start.

   mkdir -p dev/cwa-config dev/cwa-library dev/ingest
   docker run -d --name cwa-dev -p 8083:8083 \
     -e PUID=$(id -u) -e PGID=$(id -g) \
     -v "$PWD/dev/cwa-config:/config" \
     -v "$PWD/dev/ingest:/cwa-book-ingest" \
     -v "$PWD/dev/cwa-library:/calibre-library" \
     crocodilestick/calibre-web-automated:latest
   

2. Open http://localhost:8083, and log in as admin with the password admin123. These are the default credentials of CWA.

3. Start Bindery with the test CWA. AUTH_METHOD=cwa makes Bindery ask for a CWA login, so you can test logins too.

   CONFIG_DIR=dev/config INGEST_DIR=dev/ingest TMP_DIR=dev/tmp \
   CWA_URL=http://localhost:8083 CWA_USERNAME=admin CWA_PASSWORD=admin123 AUTH_METHOD=cwa \
   uv run python -m bindery
   

To remove the test CWA, run docker rm -f cwa-dev, then delete dev/.

The web interface is in bindery/static, as plain JavaScript modules, so a browser reload shows each change. After a change to the Python code, restart python -m bindery.

Keep gunicorn at one worker. The job threads run inside that worker.

License

Apache License 2.0. See LICENSE.