bindery/README.md

# Bindery

Bindery downloads web fiction into a
[Calibre-Web-Automated](https://github.com/crocodilestick/Calibre-Web-Automated) (CWA) library and
keeps the books up to date. It uses [FanFicFare](https://github.com/JimmXinu/FanFicFare), so it
supports the sites that FanFicFare supports. The tested site is RoyalRoad.
CWA is optional. You can download ePubs from Bindery without it.

## How it differs from FanFicFare

FanFicFare builds a book from the chapter list on the site, so a chapter that the author removes
also goes from your book. **Bindery keeps every chapter that it downloads** in its own database, and
builds each ePub from that database.

- **Chapters that the site removes stay in the book.** A new chapter goes after the chapter that
  comes before it on the site. For example, you have chapters 1 to 60, and the site now lists only 1
  to 4 and 61. The book then has chapters 1 to 61.
- **A changed chapter keeps its old text as a revision.** You can compare the revisions and choose
  the one that goes in the book.
- **Replacement rules run when Bindery builds the book.** You can change a rule and rebuild the book
  without a new download.

## Quick start

For uploads to CWA, use CWA v4.0.1 or later. Older versions can import an update as a new book. See
"Limitations".

1. Get the code, and make the folder for Bindery's data.

   ```bash
   git clone <this repository> bindery
   cd bindery
   mkdir config
   ```

2. Edit `docker-compose.yml` or environment variables. Set the CWA ingest folder, the CWA address and account, and the user
   and group of the container.

3. Build and start the container.

   ```bash
   docker compose up -d --build
   ```

4. Open `http://localhost:8085`, and log in with a CWA user.

The container runs as the `user` in `docker-compose.yml`. **That user must be able to write to
`./config` and to the CWA ingest folder.** Usually this is the same user and group as CWA.

## Configuration

Environment variables set the configuration. **Restart the container after a change.**

| Variable | Description | Default value |
| --- | --- | --- |
| `CWA_ENABLED` | `false` if you have no CWA. See "Without CWA". | `true` |
| `CWA_URL` | The address of CWA, for example `http://calibre-web-automated:8083` | None |
| `CWA_USERNAME`, `CWA_PASSWORD` | The CWA account of Bindery. Leave them unset if CWA asks for no login. | None |
| `CWA_DB_PATH` | Calibre-Web's `app.db`, for logins. See "CWA logins". | None |
| `CALIBRE_DB_PATH` | The calibre `metadata.db`, for book ids. See "How books reach the library". | None |
| `UPDATE_INTERVAL_HOURS` | Hours between checks for new chapters. `0` turns them off. | `24` |
| `DEFAULT_LOCKED_FIELDS` | Fields that new stories lock: `title`, `authors`, `comments`, `tags`, `series`, `pubdate`, `languages` | `title` |
| `AUTH_METHOD` | `none`, `local`, `cwa` or `proxy`. See "Logins". | `none` |
| `BINDERY_USERNAME` | The user name for `AUTH_METHOD=local`. Case-sensitive. | None |
| `BINDERY_PASSWORD_HASH` | The password hash for `AUTH_METHOD=local`. See "Local login". | None |
| `CWA_ALLOWED_USERS` | The CWA users that can log in. Empty allows every CWA user. | Empty |
| `BINDERY_ADMIN_USERS` | The users that can change the global settings. See "Admins". | Empty |
| `PROXY_AUTH_USER_HEADER` | The header in which a reverse proxy sends the user name | `X-Remote-User` |
| `PROXY_AUTH_TRUSTED_IPS` | The addresses and networks that can send the user header. See "Logins". | Loopback and private addresses |
| `PROXY_AUTH_LOGOUT_URL` | Where the Log out button goes in proxy mode | None |
| `BINDERY_API_KEY` | A key for scripts. See "Scripts". | None |
| `ALLOWED_HOSTS` | The host names that Bindery answers to. See "Host names". | None |
| `SESSION_COOKIE_SECURE` | `true` if users reach Bindery only over HTTPS | `false` |
| `TRUST_PROXY` | `true` if a reverse proxy sends the client address. See "Lockouts and sessions". | `false` |
| `URL_BASE` | A path prefix, such as `/bindery`. See "Reverse proxy". | None |
| `INGEST_DIR` | The CWA ingest folder. If unset, all uploads stay paused. Bindery does not make this folder. Without CWA, a folder for the built ePubs. | None |
| `CONFIG_DIR` | The story database and `personal.ini` | `/config` |
| `TMP_DIR` | The folder for the built ePubs. Bindery keeps the last build of each story. | `/tmp/bindery` |
| `LOG_LEVEL` | The log level | `INFO` |

Separate the values in a list with commas.

### Without CWA

Set `CWA_ENABLED=false` to run Bindery without CWA. Bindery still downloads chapters and builds
ePubs. Download the books from each story page.

With `CWA_ENABLED=false`:

- Bindery makes no request to CWA. It ignores `CWA_URL`, `CWA_USERNAME`, `CWA_PASSWORD`,
  `CWA_DB_PATH` and `CALIBRE_DB_PATH`.
- The web interface and the log show nothing about CWA. The pause, the library links and the
  approvals do not apply.
- If `INGEST_DIR` is set, Bindery puts each changed book in that folder as a plain ePub. Use this to
  import the books into other software. Each file gets a new name, so Bindery never replaces a file
  in the folder.
- `AUTH_METHOD=cwa` does not work. Use `local` or `proxy`. On a private network, you can use `none`.

If you use the example Compose file, remove its CWA connection values. Keep `INGEST_DIR` and its
volume mount only if you want the ePubs in a folder.

Without `CWA_ENABLED=false`, Bindery expects CWA. At each start, the log then has a warning for each
function that a missing `INGEST_DIR`, `CWA_URL` or `CALIBRE_DB_PATH` turns off. Bindery still
downloads chapters and builds ePubs.

Without `INGEST_DIR`, Settings shows "Pause all uploads" checked and a highlighted explanation.
You cannot clear that checkbox until you set `INGEST_DIR` and restart Bindery.
If neither `CWA_URL` nor `CALIBRE_DB_PATH` is set, library linking is off and its checkbox is disabled.
Set either value and restart Bindery to enable the checkbox. Bindery keeps your saved linking preference.
Saved CWA book IDs still work for uploads if you configure an ingest folder.

To export ePubs for other software, set `INGEST_DIR` and turn off "Link books to CWA library" in Settings.
Each build without a saved CWA book ID then puts a plain ePub in that folder, including updates.
"Pause all uploads" still stops these exports. If CWA imports the files, updates will create duplicate books.

### Logins

`AUTH_METHOD` sets how users log in:

- `none`: Bindery asks for no login. Use this only on a private network, or behind a proxy that does
  the login.
- `local`: One user logs in with a user name and password that you set. See "Local login".
- `cwa`: Users log in with their CWA user name and password. See "CWA logins".
- `proxy`: A reverse proxy, such as [Rauthy](https://github.com/sebadob/rauthy),
  [Authelia](https://github.com/authelia/authelia) or
  [Authentik](https://github.com/goauthentik/authentik), logs users in and sends the user name in
  `PROXY_AUTH_USER_HEADER`. **Make sure that users can reach Bindery only through the proxy**,
  because Bindery trusts this header. If a request gets a 401 because the proxy session ended,
  Bindery reloads the page, so the proxy can step in again.

  Bindery accepts the header only from the addresses in `PROXY_AUTH_TRUSTED_IPS`. The default is the
  loopback and private addresses, such as `127.0.0.1`, `10.x`, `172.16.x` to `172.31.x` and
  `192.168.x`. This stops a request that comes straight from the internet, but not one from your own
  network. **Set it to the address of the proxy**, for example `PROXY_AUTH_TRUSTED_IPS=172.18.0.3`,
  or publish the port of Bindery only on the host, as `127.0.0.1:8085:8085` in the Compose file.
  `TRUST_PROXY` does not change this check. It is only for the lockouts.

**If the login settings are incomplete or wrong, Bindery refuses every request**, also with the API
key, and shows the reason.

#### Local login

`AUTH_METHOD=local` gives one Bindery account without CWA. You must set `BINDERY_USERNAME` and
`BINDERY_PASSWORD_HASH`. The hash is a one-way form of the password.

1. In the folder with your `docker-compose.yaml` file, run this command to make the hash. The command hides the
   password as you type it. Use a password that is not empty.

   ```bash
   docker compose run --rm --no-deps bindery python -c 'from getpass import getpass; from werkzeug.security import generate_password_hash; print(generate_password_hash(getpass("Password: ")))'
   ```

   In a source checkout, you can use:

    ```bash
    uv run python -c 'from getpass import getpass; from werkzeug.security import generate_password_hash; print(generate_password_hash(getpass("Password: ")))'
    ```

2. Make a `.env` file beside the `docker-compose.yaml` file, with your user name and the hash:

   ```dotenv
   BINDERY_USERNAME=reader
   BINDERY_PASSWORD_HASH='paste-the-generated-hash-here'
   ```

   **Keep the single quotes around the hash.** They stop Docker from reading its `$` characters as
   variables. Keep `.env` private, and do not commit it.

   If you put the hash in `docker-compose.yml` instead, **write each `$` as `$$`**, for example
   `'scrypt:32768:8:1$$salt$$digest'`. In the Compose file, quotes do not stop Compose from reading
   `$` as the start of a variable. Bindery then gets a shortened hash, and it reports that the hash
   is not valid.

3. In the Compose file, set `AUTH_METHOD: local`, and uncomment `BINDERY_USERNAME` and
   `BINDERY_PASSWORD_HASH`. Restart the container.

4. Log in with the user name and the password, not the hash.

To change the account, change the user name or make a new hash, and restart Bindery. This ends the
open sessions.

#### CWA logins

`AUTH_METHOD=cwa` needs `CWA_URL`, `CWA_DB_PATH` or both. Without `CWA_DB_PATH`, Bindery sends the
user name and password to `CWA_URL/opds` as an HTTP Basic login. This is the OPDS login.

**The OPDS login works only if CWA refuses a wrong password.** A CWA with anonymous browsing can
accept every OPDS login. Bindery tests this with a login that cannot exist. If CWA accepts it,
Bindery refuses each OPDS login, and the log names the reason. Set `CWA_DB_PATH` then, or use
another `AUTH_METHOD`.

With `CWA_DB_PATH`, Bindery checks the password against the account in Calibre-Web's `app.db`. A
wrong password or an unknown account fails at once. If Bindery cannot read the file or its password
hash, it uses the OPDS login.

`CWA_DB_PATH` is the path of `app.db` inside the container. To use it, mount the CWA config folder
read-only into the container. Mount the whole folder, so that SQLite can read the journal files
beside the database. **Bindery does not write this database.**

**Database logins check only the password.** CWA's external login providers and multi-factor
authentication do not apply. If they must apply, use the `proxy` mode.

- The CWA guest user cannot log in.
- If `CWA_ALLOWED_USERS` is set, only those users can log in. Bindery checks this list on each
  request.
- `CWA_USERNAME` and `CWA_PASSWORD` are only for the library lookups of Bindery.

#### Lockouts and sessions

The `local` and `cwa` modes lock out password guesses:

- After 5 wrong passwords for one user name from one address in 15 minutes, Bindery locks that user
  name for 15 minutes, for that address only. The real user can still log in from another address.
- After 20 wrong passwords from one address in 15 minutes, Bindery locks that address for 15
  minutes.

Behind a reverse proxy, every request comes from the address of the proxy, so all users share one
lock. To give each client its own lock, set `TRUST_PROXY=true`. Bindery then takes the client
address from the last entry of `X-Forwarded-For`. **Set it only if clients can reach Bindery through
the proxy alone.** Otherwise a client can send a false address with each guess.

A session ends when the user logs out, and **always after one day**. Without "Keep me logged in", it
also ends when the user closes the browser.

**Bindery checks a password only at login.** A deleted CWA user or a changed CWA password keeps its
open session until that session ends. To stop a CWA user at once, remove the user from
`CWA_ALLOWED_USERS` and restart Bindery. To end all sessions, delete `/config/.flask_secret` and
restart Bindery.

#### Admins

In the `cwa` and `proxy` modes, `BINDERY_ADMIN_USERS` lists the users that are admins, for example
`BINDERY_ADMIN_USERS=alice, bob`. Only an admin can do these things:

- Open the Settings page, and change the settings on it.
- Change the global replacement rules.
- Read and change `personal.ini`, which can hold passwords for story sites.

Each other user can still add, update and delete stories, and change the settings and the rules of a
story. Bindery ignores the case of the names.

If the list is empty, each user is an admin. The `none` and `local` modes have only one user, who is
always an admin. The API key is always an admin too.

### Host names

Without a login, a web page that you visit can try to reach Bindery from your browser, under a host
name of its own that points at the address of Bindery. To stop this, the `none` login mode answers
only to these host names:

- An address, such as `192.168.1.5`.
- `localhost`, and a name without a dot, such as `nas` or the name of a container.
- A name of a private network. These end in `.local`, `.lan`, `.home`, `.home.arpa`, `.internal`,
  `.localhost` or `.localdomain`.

For each other name, Bindery answers with an error that shows the name. If you use such a name, for
example behind a reverse proxy, list it in `ALLOWED_HOSTS`:

- `ALLOWED_HOSTS=bindery.example.com` allows that name. Bindery then answers only to the names in
  the list, in each login mode. Write the names without a port.
- A leading dot, as in `.example.com`, also allows the subdomains.
- `ALLOWED_HOSTS=*` allows each name.

The other login modes answer to each name, unless you set a list. The health check at `/api/health`
works under each name.

### Scripts

Scripts can use the JSON API of the web interface.

- In the `local`, `cwa` and `proxy` login modes, a script needs the API key. Make a long random key,
  for example with `openssl rand -hex 32`, and set it in `BINDERY_API_KEY`. Send it with each
  request as `Authorization: Bearer <key>` or `X-Api-Key: <key>`.
- In the `none` login mode, a script needs no key, but it must send `X-Requested-With: bindery` with
  each change.

This example adds a story. If Bindery tracks the story already, it downloads the new chapters.

```bash
curl -X POST http://localhost:8085/api/stories \
  -H "Authorization: Bearer $BINDERY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://www.royalroad.com/fiction/12345"]}'
```

The answer lists the queued jobs and the unsupported URLs: `{"jobs": [...], "errors": [...]}`.

| Request | What it does |
| --- | --- |
| `GET /api/stories` | Lists the tracked stories, with their ids |
| `POST /api/stories` | Adds stories. The body is `{"urls": [...]}`. |
| `POST /api/stories/<id>/download` | Queues a job. The body is `{"mode": "new"}`. The modes are `new`, `recheck` (with `"chapter_ids": [...]`), `recheck_all`, `full`, `metadata` and `rebuild`. |
| `POST /api/stories/import-ePub` | Imports an ePub file, for example with `curl -F file=@story.epub`. Add the form field `check=false` to skip the check of the site. For one story, use `/api/stories/<id>/import-ePub`. |
| `PUT /api/stories/<id>/metadata` | Edits metadata. The body is `{"values": {"title": "New title", "tags": ["a", "b"]}}`. `null` removes an edit. The description, `comments`, is HTML. |
| `POST /api/stories/<id>/approve` | Approves the library link of a held story. The body is `{"uuid": "<uuid of the library book>"}`, or `{"uuid": null}` for a new book. |
| `PUT /api/settings` | Changes settings. The body `{"pause_uploads": true}` pauses all uploads to CWA. |
| `GET /api/jobs` | Lists the recent jobs. Add `?story_id=<id>` for one story. |

If Bindery runs under `URL_BASE`, put the prefix before `/api`.

### FanFicFare settings

On first start, Bindery copies `data/personal.ini` to `/config/personal.ini`. It uses the format of
`personal.ini` in the FanFicFare calibre plugin, so you can copy your plugin settings into it. The
[INI File](https://github.com/JimmXinu/FanFicFare/wiki/INI-File) page of the FanFicFare wiki
explains the format, and
[defaults.ini](https://github.com/JimmXinu/FanFicFare/blob/main/fanficfare/defaults.ini) lists every
setting. Bindery reads the file for each job, so a change applies to the next job.

You can edit the file on the Settings page. Bindery refuses a file that FanFicFare cannot read, and
shows FanFicFare's warnings for other problems. **The file can hold passwords for story sites, and
each admin can read them.** If `BINDERY_ADMIN_USERS` is not set, each user is an admin. See "Admins".

Bindery always sets `always_reload_first_chapter`, `mark_new_chapters`, `continue_on_chapter_error`,
`make_directories`, `zip_output` and `calibre_series_meta`, because it needs fixed values for them.

**Each book gets the cover from the site.** FanFicFare downloads the cover only if `include_images`
is on. If it is off, Bindery sets it to `coveronly`, which keeps the cover but leaves out the images
in the story text.

### Reverse proxy

To serve Bindery under a path, such as `/bindery/`, do one of these:

- Make the proxy remove the path before it forwards the request.
- Make the proxy forward the full path, and set `URL_BASE=/bindery`.

The health check at `/api/health` works with and without the prefix.

## How books reach the library

Bindery gives every book to CWA through the ingest folder. CWA makes all changes to the library.

**Bindery does not make the ingest folder.** If the folder in `INGEST_DIR` does not exist, most often
because its volume mount is missing, the log shows an error at the start and each upload fails. The
books go to CWA with the next update after you fix the mount.

1. Bindery puts the ePub of a new story into the ingest folder. CWA imports it as a new book.
2. At the next update, Bindery finds the CWA book of the story. If you set `CALIBRE_DB_PATH`, it
   first looks for the story URL in the library database. Without a single match there, it searches
   the CWA OPDS feed (using `CWA_URL`, `CWA_USERNAME` and `CWA_PASSWORD`) for the story title. The
   search ignores the parts of a title in parentheses or brackets, such as tags or "(Book 1 Complete)".
   Bindery downloads up to 5 ePubs with a matching title, closest title first. The book whose ePub
   holds the story URL is the match. Bindery saves the id and uuid of the book with the story.
3. For the update, Bindery puts two files into the ingest folder: the new ePub, and a sidecar file
   `<name>.epub.cwa.json` that holds `{"action": "add_format", "book_id": N}`. CWA then replaces the
   ePub of book N.

Before each later update, Bindery finds the book by its uuid, first in the library database if you
set `CALIBRE_DB_PATH`, then through CWA. If you deleted the book in CWA, Bindery unlinks the story.

`CALIBRE_DB_PATH` is optional. It is the path of the calibre `metadata.db` inside the container, and
it lets Bindery find a book by its story URL instead of its title. To use it, mount the library
folder read-only into the container. Mount the whole folder, so that SQLite can read the journal
files beside the database. The Compose file has an example. **Bindery does not write to this database.**

Be sure to use the library that gets the ingest files. A different library can make Bindery update the
wrong book. Database lookups also find books that the CWA account of Bindery cannot see. If Bindery
cannot read the database, it uses CWA for every lookup. Keep `CWA_URL` and the CWA account set,
because library imports and CWA lookups use them.

A book that the FanFicFare calibre plugin made holds its story URL. If you add its URL, or import it
from the library, Bindery links that book and imports its chapters from the ePub.

### If Bindery cannot find the book

If library matching is on and configured, an update fails when Bindery cannot find the book.
The story page shows the error "Cannot find this story in CWA. Link to library book, or send as new book." The
chapters stay stored, and the next successful update includes them. This happens if:

- CWA did not import the ePub.
- The title of the book changed in CWA before Bindery linked it.
- The CWA account of Bindery cannot see the book.
- You deleted the book in CWA.

To fix it, go to "Library link" on the story page, and do one of these:

- Search for the book, and click "Link". Then click "Rebuild book" to send the update.
- Click "Send to CWA as a new book". Use this if CWA did not import the first version, or if you
  deleted the book.

### Upload settings

The "Uploads to CWA" section controls library matching and uploads. Bindery still downloads chapters
and builds books when uploads stop. The story list marks a built book with "held" only when it waits
for approval, library linking is on, and uploads are not paused. Bindery keeps track of all unsent books.

- Link books to CWA library. This defaults to on when `CWA_URL` or `CALIBRE_DB_PATH` is set.
  Without either value, it stays off and disabled. Turn it off to skip matching with a configured library connection.
  Uploads use saved CWA book IDs when available. Without an ID, every build goes to `INGEST_DIR` as a plain ePub.
  If CWA imports these files, updates will create duplicate books. Settings and the story page show a highlighted warning.
  A change of this setting sends the unsent books, unless uploads are paused.
- **Hold new links for approval.** This setting applies only when library linking is on.
  Use this to catch a story that is linked to the wrong CWA book.
  The first upload of a new story waits for your approval. The first upload after a change of the
  library link also waits. This includes a link that you make, and a book that Bindery finds after
  the linked book is gone. The story page names the book that gets the update, or says that the
  story goes to CWA as a new book.
- **Pause all uploads.** No book goes to CWA. This is always checked when `INGEST_DIR` is unset.
  When you turn the pause off, Bindery sends the unsent books. The other upload settings still apply.
  While paused, use "Approve" or "Approve as a new book instead" to save a decision without sending the book.

To approve a link, open the story page and do one of these:

- If the book is right, click "Approve and send".
- If the book is wrong, search for the right book under "Library link", and click "Link". Then
  approve the new link.
- If CWA does not have the story, click "Send as a new book instead".

The book that CWA imports from the first upload of a story needs no approval. After an approval,
later updates go to CWA as usual.

### The CWA account

Make a CWA user for Bindery with these settings:

- The download permission. Bindery reads story URLs and chapters from the ePubs.
- "Show Publisher Section", in the sidebar settings of the user. CWA shows its list of publishers in
  OPDS only to users with this setting. "Import from the library" uses the list to find the books of
  story sites, because FanFicFare sets the publisher of a book to the site name.
- Access to all books. Bindery finds only the books that this account can see.

Bindery sends the password of this account with each OPDS request. **Use a password that no other
service uses.**

If CWA allows anonymous browsing, Bindery needs no account. Leave `CWA_USERNAME` and `CWA_PASSWORD`
unset, and give the guest user of CWA the settings above.

## Import an ePub file

If the site removed chapters before Bindery tracked the story, import an ePub that you made earlier
with FanFicFare. Bindery stores its chapters as if it downloaded them. Chapters that the site
publishes later go after them.

- To start to track a story from its ePub, use "Import an ePub file" on the stories page. For a
  story that Bindery tracks already, it adds the chapters that it does not have.
- To add chapters to one story, use "Import chapters from an ePub" on the story page. The ePub must
  be of that story.

**Bindery accepts only ePubs that FanFicFare made**, because it reads the story URL and chapter data
that FanFicFare writes into each ePub. A chapter that Bindery stored already keeps its stored text.
The other chapters go in the order of the ePub.

After the import, Bindery checks the site for new chapters and sends the book to CWA. If you clear
"Check the site for new chapters after import", Bindery only rebuilds the book of a story that it
downloaded before.

## File names of downloaded books

"Download ePub" on a story page sends the book to your browser with a file name. A template gives
the name. The default template is `{title} - {authors}`, as in Calibre. For "The Book" by Ann
Writer, it gives `Book, The - Ann Writer.epub`.

An admin can change the template under "Book Downloads" in Settings. To go back to the default,
clear the field and save. The template follows
the rules of [Calibre's templates](https://manual.calibre-ebook.com/template_lang.html) for file
names, so a template from Calibre gives the same name in Bindery. Bindery adds `.epub`. For example,
`{series:|| - }{series_index:0>2s|| - }{title} - {authors}` gives
`Saga, The - 02 - Book, The - Ann Writer.epub`.

| Field | Value |
| --- | --- |
| `{title}` | The title. A leading "The", "A" or "An" moves to the end, as in Calibre: `Book, The`. |
| `{title_sort}` | The same value as `{title}`. |
| `{authors}`, `{author}` | The authors, with ` & ` between them. |
| `{author_sort}` | The authors with the last name first: `Writer, Ann`. |
| `{series}` | The series. A leading article moves to the end. |
| `{series_index}` | The series number: `2`, or `2.5`. |
| `{tags}` | The tags in sorted order, with `, ` between them. |
| `{languages}` | The language, as Bindery stores it. |
| `{publisher}` | The story site. |
| `{pubdate}` | The published date. |
| `{timestamp}` | The date when Bindery added the story. |
| `{last_modified}` | The date when the chapters last changed. |
| `{id}` | The CWA book ID. For a story with no library book, and without CWA, the Bindery story ID. |

- A field with no value becomes empty text.
- `{field:format|text before|text after}` adds the two texts only when the field has a value. Each
  of the three parts can be empty. `{series:|| - }` adds ` - ` after the series.
- The format is a Python format. `{series_index:0>3s}` gives `002`, and `{title:.20s}` gives the
  first 20 characters. `{author_sort[0]}` gives the first character.
- `{{` and `}}` give braces. Bindery makes each run of spaces one space.
- A download has one file name. If the template has folders, Bindery uses the part after the last
  `/`. That is the name that Calibre gives the file.
- Bindery does not support template functions, such as `{title:uppercase()}`, or the program modes
  of Calibre. It refuses such a template when you save it.
- If the template fails for a book, or gives an empty name, Bindery names the file after the title.
  The log has a warning.
- Bindery replaces each character that a file name cannot have with `_`. A name has 100 characters
  and 200 bytes at most.

"Date format" sets the form of `{pubdate}`, `{timestamp}` and `{last_modified}`. It takes the
[codes of Python's strftime](https://docs.python.org/3/library/datetime.html#format-codes). `%d`
gives the day, `%b` the month name, `%m` the month number and `%Y` the year. The default is
`%b, %Y`, as in Calibre. It gives `May, 2020`. To go back to the default, clear the field and save.
A `/` in a date becomes `_`, as in Calibre.

Bindery follows Calibre 9.2.1 with its default tweaks, for English. The order of tags with letters
outside ASCII can differ from Calibre. The template applies only to downloads in the browser. It
does not change the names of the files in `INGEST_DIR`. The browser can ignore the name.

## Download modes

| Mode | What it does |
| --- | --- |
| Download new chapters | Downloads the chapters that are not stored yet. The default, and what scheduled updates use. |
| Recheck selected chapters | Downloads the selected chapters again. Changed text becomes a new revision. |
| Recheck all chapters | Downloads every chapter on the site again, and new chapters too. |
| Download and replace entire book | Downloads every chapter again. Chapters that the site no longer lists leave the book. They stay stored, and you can put them back. |
| Update metadata from site | Updates only the metadata in the ePub. |
| Rebuild book | Builds the book again from the stored chapters, and sends it. |

**Bindery builds and sends a book only if the book changed.** A change is a new, changed, moved or
renamed chapter, a changed rule, or changed metadata. A job that finds no change answers "No changes"
and sends nothing to CWA. Only "Rebuild book" builds and sends a book that did not change.

### Find edited chapters

**Scheduled updates download only new chapters, so they miss most edits.** To find edits, run a
recheck, or use one of these settings:

- On the Settings page, "also recheck the last N chapters" rechecks the last N chapters with each
  download of new chapters. Authors most often fix their latest chapters. The default is 0, which
  turns it off.
- In the "Checking for new chapters" section of a story page, "Recheck all chapters every N days" rechecks the whole
  story. This recheck takes the place of the next scheduled update. The story then ignores the "last
  N chapters" setting.

Each rechecked chapter is one request to the site, so keep the numbers small for long stories.

If the site changes the title of a chapter, "Download new chapters" and "Recheck selected chapters"
download that chapter again. The chapter gets the new title only if the download works. On sites
that number chapter URLs, such as FanFiction.net, this also finds chapters that moved when the
author inserted a chapter.

### Revisions

A recheck compares the text of a chapter with its newest revision, which is the text that the site
sent last. Bindery stores the HTML as the site sends it, before the rules, so a rule never makes a
chapter look changed.

- If the text changed, the new revision goes in the book. If you chose an older revision, it
  stays in the book until the site changes the chapter again.
- On the page of a chapter, you can compare revisions, choose the one that goes in the book, and
  delete the others. The revision in the book and the newest revision cannot be deleted.
- A chapter imported from an ePub holds its text after the FanFicFare settings and rules of that
  build. If the current rules turn the site text into the imported text, Bindery counts the chapter
  as unchanged, and stores the site text in its place. Other differences, for example from settings
  in `personal.ini`, show as a change.

### Restarts

Bindery stores each chapter as soon as it downloads it, and keeps the job queue in its database.
After a restart, the waiting jobs run. A job that was running starts again at the front of the
queue, and skips the chapters that it stored. If Bindery stops during the same job three times, the
job fails.

If a job thread stops, `/api/health` answers 503, so the container shows as unhealthy. Restart
Bindery to start the thread again.

### Backups

Once a day, Bindery copies its database and `personal.ini` to `/config/backups`, and keeps the
copies of the last 7 days. **The database is the only copy of the chapters that sites removed.** The
backups are on the same disk, so also copy `/config` to another disk or machine.

To restore a backup, do these steps:

1. Stop Bindery.
2. Copy the backup, for example `/config/backups/bindery-2026-09-26.db`, to `/config/bindery.db`.
3. If `/config/bindery.db-wal` or `/config/bindery.db-shm` exist, delete them. They belong to the
   old database.
4. Start Bindery.

## Metadata

Bindery keeps the metadata of each story in its database, and writes it into each ePub. The fields
are title, authors, description, tags, series, published date and language. Each field is locked or
unlocked for each story:

- An unlocked field takes the site value on every update.
- A locked field keeps its stored value. The first download sets that value. For a story that you
  import from the library or from an ePub file, the value comes from that ePub.
- To update a locked field once, select it under "Also take these locked or edited fields from the
  site" when you start a download. The field stays locked.

You can edit each field under "Metadata" on the story page. An edit of a locked field replaces its
stored value. An edit of an unlocked field stays until the site changes that field. To go back to
the site value at once, click "Use the site value" or unlock the field. If Bindery sent the book to
CWA before, it rebuilds the book after an edit.

- Separate authors with `&`, and tags with commas.
- Put the number of a series in brackets, for example `Saga [2]`.
- The description is plain text, with a blank line between paragraphs.

`DEFAULT_LOCKED_FIELDS` sets the locks for new stories. The default is `title`. The publisher is
always the site name.

**CWA reads the metadata and the cover of an ePub only when it imports a new book.** An update
replaces only the ePub file. So edits that you make in CWA stay, and later changes on the site show
in the ePub but not in the CWA catalog.

## Replacement rules

A rule is a regular expression, a replacement and optional flags (`i`, `s`, `m`, `x`). Bindery uses
the Python [`regex`](https://pypi.org/project/regex/) module, which accepts the [syntax of Python's
`re` module](https://docs.python.org/3/library/re.html#regular-expression-syntax). For an
introduction, read the [Regular Expression HOWTO](https://docs.python.org/3/howto/regex.html). The
replacement can use groups, such as `\1` or `\g<name>`. When Bindery builds a book, it runs the
global rules from the Settings page, then the rules of the story from its page. The rules run on the
HTML of each chapter. Each rule editor has a preview that shows one chapter before and after the
rules.

**If a rule takes longer than 10 seconds on one chapter, the build stops** with an error that names
the rule.

Some sites separate paragraphs with `<br>` tags. For those, turn on "Turn line breaks into
paragraphs before the rules run" in a rule editor. Bindery then puts each piece of text between
`<br>` tags in its own `<p>`, and drops empty pieces. A `<br>` inside other markup, such as italics,
stays. This step runs before the global rules. The global setting is off by default, and applies to
each story that has no setting of its own. To see the new paragraphs in the preview, select "Compare
HTML source".

## Limitations

- **Bindery identifies a chapter by its URL.** Most sites, such as RoyalRoad and AO3, keep the URL
  of a chapter. FanFiction.net and similar sites number chapter URLs by position. When an author
  inserts a chapter there, Bindery finds the moved chapters by their new titles. If the titles stay
  the same, run "Recheck all chapters". When an author removes a chapter there, the book loses that
  chapter, and the last chapter appears twice. The removed chapter stays as an older revision of the
  chapter that took its URL.
- "Stop tracking" deletes only the data of Bindery. The book stays in CWA.
- **Updates need CWA v4.0.1 or later.** CWA v4.0.0 added the `.cwa.json` sidecar files. But v4.0.0
  can delete a sidecar file before it imports the ePub, and then imports the update as a new book.
  Versions before v4.0.0 cannot read sidecar files, so each update becomes a new book.

## Development

You need [uv](https://docs.astral.sh/uv/). The frontend tests also need Node 22 or later.

### Tests

Run these commands in the project folder:

```bash
uv sync                               # install the Python dependencies
uv run pytest                         # Python tests
uv run ruff check .                   # lint
node --test "tests/js/*.test.mjs"     # frontend tests
```

The Python tests use a fake CWA server and temporary folders, so they need no CWA and no story
sites. The frontend tests use the built-in test runner of Node, and need no npm packages. Also check
the pages in a browser.

### Run locally without CWA

Without CWA, Bindery puts each new build of a story into its ingest folder as an ePub. This is
enough to work on downloads, chapters, rules and the web interface.

```bash
mkdir -p dev/ingest
CWA_ENABLED=false CONFIG_DIR=dev/config INGEST_DIR=dev/ingest TMP_DIR=dev/tmp uv run python -m bindery
```

Open `http://127.0.0.1:8085`. The folders go in `dev/`, which git ignores. To look at an ePub, use
"Download ePub" on the story page.

### Run locally with a test CWA

To test library lookups and updates, run a separate CWA container with its own empty folders. **Do
not use your real library.**

1. Start a test CWA on port 8083. It shares `dev/ingest` with Bindery, and makes an empty library in
   `dev/cwa-library` on first start.

   ```bash
   mkdir -p dev/cwa-config dev/cwa-library dev/ingest
   docker run -d --name cwa-dev -p 8083:8083 \
     -e PUID=$(id -u) -e PGID=$(id -g) \
     -v "$PWD/dev/cwa-config:/config" \
     -v "$PWD/dev/ingest:/cwa-book-ingest" \
     -v "$PWD/dev/cwa-library:/calibre-library" \
     crocodilestick/calibre-web-automated:latest
   ```

2. Open `http://localhost:8083`, and log in as `admin` with the password `admin123`. These are the
   default credentials of CWA.

3. Start Bindery with the test CWA. `AUTH_METHOD=cwa` makes Bindery ask for a CWA login, so you can
   test logins too.

   ```bash
   CONFIG_DIR=dev/config INGEST_DIR=dev/ingest TMP_DIR=dev/tmp \
   CWA_URL=http://localhost:8083 CWA_USERNAME=admin CWA_PASSWORD=admin123 AUTH_METHOD=cwa \
   uv run python -m bindery
   ```

To remove the test CWA, run `docker rm -f cwa-dev`, then delete `dev/`.

The web interface is in `bindery/static`, as plain JavaScript modules, so a browser reload shows
each change. After a change to the Python code, restart `python -m bindery`.

**Keep gunicorn at one worker.** The job threads run inside that worker.

## License

Apache License 2.0. See `LICENSE`.