tim/bindery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Compare commits

base: tim/bindery:v0.1
Branches Tags
tim/bindery:main
tim/bindery:v0.6 tim/bindery:v0.5 tim/bindery:v0.4 tim/bindery:v0.3 tim/bindery:v0.2 tim/bindery:v0.1
..
compare: tim/bindery:main
Branches Tags
tim/bindery:main
tim/bindery:v0.6 tim/bindery:v0.5 tim/bindery:v0.4 tim/bindery:v0.3 tim/bindery:v0.2 tim/bindery:v0.1
55 Commits
v0.1 .. main
Author SHA1 Message Date
tim 2f59b0e125 Put the core flow first in the README 
The README starts with the flow from a story URL to the OPDS catalog. One section near the end holds everything about CWA.

The example Compose file asks for no login and has the CWA values as comments.
 2026-10-06 16:59:15 +02:00
tim ca3d730129 Turn CWA off unless CWA_ENABLED is true 
CWA_ENABLED now defaults to false. If a CWA variable is set while CWA is off, the start log has a warning that names it.

On the Settings page, the alert about INGEST_DIR moves to Book Downloads and shows with CWA on or off. The Uploads to CWA card moves below Book Downloads.
 2026-10-06 16:59:15 +02:00
tim 9a48356155 Filter the import search from the list of library books 
The search through CWA gave no results for books that the full list shows. The search now filters the list that "Show all stories in the library" loads, by title, author and site.

The buttons stay disabled until the answer comes. A book that the user imports leaves the list.
 2026-10-06 16:59:09 +02:00
tim eea6adcc91 Fix Last Changed updates during ePub rebuilds 
Update Last Changed only when Bindery stores a new chapter or adds a new
chapter revision. Manual ePub generation, rebuilds, metadata edits and
rule changes leave it unchanged.

Save the timestamp in the same transaction as the chapter or revision.
This preserves the date if a later build fails and prevents retries from
moving it forward without another chapter change.

Add regression tests for new chapters, revisions, forced rebuilds, failed
builds, retries and transaction rollback. Adjust OPDS test data setup and
document what Last Changed records.
 2026-10-06 16:54:33 +02:00
tim 46de77c228 Show an estimated page count for each built book 
Bindery counts the pages as the last step of a build, from the text of the ePub. The story list and the OPDS catalog show the count. A story has no count until its book is built.

The method follows "Paragraphs (APNX Accurate)" of the Count Pages plugin for calibre. Bindery counts each paragraph that a reader sees, so the same text gives the same count as <p> elements, as <div> elements and as lines with <br> between them. An empty paragraph is not counted.
 2026-10-04 02:05:47 +02:00
tim b73b2808ca Search the library in the import section 
The search shows the matches from a story site that Bindery does not track yet. The button that lists every such book stays.
 2026-10-04 02:04:16 +02:00
tim d28f69d93a Show the story sort headings as chips on narrow screens 2026-10-03 11:53:47 +02:00
tim 60424e8e74 Name the publishing site in the README 
"The site" could mean the publishing site of a story or the Bindery
site. Name the publishing site, its publisher and its published text.
 2026-10-03 11:23:28 +02:00
tim 5ac56de09d Document the OPDS catalog 
Cover the catalog, its logins in each mode, and the paths that the
proxy must let through.
 2026-10-03 11:23:28 +02:00
tim 3662372c38 Add the OPDS settings to the settings page 
The Book Downloads card has an "Enable OPDS" box. In the proxy mode, it
also shows the user name for the reader and a field to set the OPDS
password. The field is grey until the box is checked. Disabled text
fields now look grey.
 2026-10-03 11:23:28 +02:00
tim 28cd0dcde9 Add an OPDS catalog 
A reader app can browse the stored books by title, author and last
change, search them, and download them. The download and cover
routes share their code with the story page.

An admin turns the catalog on in the settings. The login follows
AUTH_METHOD: none needs no login, local and cwa take the account
login, and proxy takes the user name from the proxy with an OPDS
password that each user sets. The API key does not open the catalog.
 2026-10-03 11:23:28 +02:00
tim ece59c2558 Add {series_sort} to the download template 
It gives the series with a leading article at the end, as Calibre's default does for {series}.
 2026-10-03 08:09:41 +02:00
tim dcdd8e4ac7 Keep the article in {title} and {series} 
The default template is now {title_sort} - {authors}. It gives the same name as Calibre's default.
 2026-10-03 08:09:41 +02:00
tim 42e7287389 Fold each group of settings into an open card 
The long help for the download template and the date format now folds shut.
 2026-10-03 08:09:41 +02:00
tim e38c8d655a Add a date format for download names 
The new setting gives the strftime format of {pubdate}, {timestamp} and {last_modified}.
The default is Calibre's "%b, %Y".
 2026-10-03 08:09:41 +02:00
tim 2c8a1bb187 Use {title} - {authors} as the default download template 
This is Calibre's default. An empty template now restores the default.
 2026-10-03 08:09:41 +02:00
tim 323bec3c04 Name no template fields that Bindery does not have 2026-10-03 08:09:41 +02:00
tim 3e29ab6bf0 Name downloaded books with a template 
The new "Book Downloads" group in Settings holds a Calibre template for the file name of a
direct download. Without a template, the name stays the title. The name is also the title if
the template fails or gives no name for a book.
 2026-10-03 08:09:41 +02:00
tim 9e7a488b50 Add Calibre's file name templates 
The module prepares the metadata of a book and expands a template as Calibre 9.2.1 does with
its default tweaks. It takes fields, formats, indexes, and text before and after a field. It
refuses template functions and the program modes.
 2026-10-03 08:09:41 +02:00
tim ccd3b18791 Share the folder setup of the configuration tests 2026-10-03 08:09:41 +02:00
tim 6a1c92bc62 Remove duplicates in the web interface 
The settings page has one Save button helper. The story list sorts by
the sort key that the server sends, so the rule for a leading article
exists once. The story page takes its list of download modes from the
mode hints.
 2026-10-03 08:09:41 +02:00
tim 2ce3de6b58 Remove unused store and CWA client code 
The two functions that add chapters share one insert. Nothing used
Store.current_revisions and CwaClient.check.
 2026-10-03 08:09:41 +02:00
tim 5f682d209b Remove small duplicates in the settings, the logins and the names 
One function reads a true or false value. The download name uses the
file name cleaning of the ingest folder. The login code reads the login
mode from the configuration, and has one lockout message. The title
helper of FanFicFare is now unescape_title, because cwa.py has another
plain_title.
 2026-10-03 08:09:41 +02:00
tim 96cf1e4f03 Split the site job into smaller functions 
A new story, the chapter list of the site and the choice of chapters to
download each have a function. One helper downloads a chapter for both
loops.
 2026-10-03 08:09:41 +02:00
tim 2ada3b3d7e Share the checks and the job builders of the orchestrator 
One function checks a whole number of a request, and one checks the
metadata field names. save_settings takes the settings by name. A rebuild
job and a story from an epub each have one builder.
 2026-10-03 08:09:41 +02:00
tim 89709f875a Give a library book one form in the API 
A book from the search, from the list of library candidates and from a
lookup now has the same fields, with the authors as a list.
 2026-10-03 08:09:41 +02:00
tim 22478c3a56 Simplify the upload state and the story page data 
The pause of the user and a missing ingest folder are separate checks.
One function gives the upload state to the settings and the story page.
A route that returns only a story no longer loads its chapters or asks
CWA for its book. The story page data loses the values that no page reads.
 2026-10-03 08:09:41 +02:00
tim 2b1a26d1da Name Bindery alone in the default personal.ini 2026-10-03 08:09:41 +02:00
tim 05ef1dbfad Add CWA_ENABLED to run Bindery without CWA 
With CWA_ENABLED=false, Bindery ignores the CWA settings, makes no request
to CWA, and shows nothing about CWA in the log and the web interface. A
set INGEST_DIR still gets each changed book as a plain epub.

Without it, the log names each CWA function that a missing setting turns
off.
 2026-10-02 23:39:58 +02:00
tim bb6c87cdd5 Let only admins change the global settings 
BINDERY_ADMIN_USERS lists the admins of the cwa and proxy modes. Only an
admin can change the global settings and the global rules, and read or
change personal.ini. Without a list, each user is an admin.
 2026-10-02 23:37:33 +02:00
tim 751b52da77 Answer only to the host names of Bindery 
Without a login, a web page that the user visits could reach Bindery
under a name of its own that points at the address of Bindery, and read
and change everything. The "none" mode now answers only to addresses
and to names of a private network. ALLOWED_HOSTS sets the names for
each mode.
 2026-10-02 23:37:33 +02:00
tim 2d9796d77c Accept the proxy user header only from trusted addresses 
In proxy mode, each client that reached the port could send the user
header. PROXY_AUTH_TRUSTED_IPS now lists the addresses and networks that
can send it. The default is the loopback and private addresses, so a
request that comes straight from the internet cannot log in.
 2026-10-02 23:37:33 +02:00
tim 9196dd75a8 Refuse OPDS logins when CWA accepts every password 
A CWA with anonymous browsing can answer an OPDS request whatever the
password, so each password logged a user in to Bindery. Bindery now
tests CWA with a login that cannot exist, and refuses OPDS logins if
CWA accepts it. A login check no longer shares the session of the
account of Bindery. The CWA account stays optional for a CWA that asks
for no login, and a missing account now gets its own message.
 2026-10-02 23:37:33 +02:00
tim 869e56eaf0 Do not make the ingest folder 
A folder that Bindery made hid a missing volume mount. The books went
to a folder inside the container, and CWA never saw them. A missing
folder is now an error in the log at the start, and each upload fails
with a message that names the mount.
 2026-10-02 23:37:33 +02:00
tim 679c1ede9f Serve a download from the built book, and delete stale uploads 
A download built the book again each time. It now sends the kept build,
and builds only if the book changed after it. Stop tracking deletes the
built book. A cancelled import deletes its uploaded file, and the start
deletes the uploads that no job needs.
 2026-10-02 23:37:33 +02:00
tim ff2fa0bf90 Keep a move or a delete of chapters that the user makes during a job 
A job placed each new chapter with the chapter order from its start, so
a move by hand during the job went back. The store now places a new
chapter against the order that the story has at that moment, in one
transaction. A recheck of a chapter that the user deleted no longer
stops the job.
 2026-10-02 23:37:33 +02:00
tim 81efc03b75 Keep the job thread alive after an error of the queue 
An error outside a job, such as a full disk, stopped the thread, and no
job ran until a restart. The health check now answers 503 when a job
thread stopped.
 2026-10-02 23:37:33 +02:00
tim c0504135b2 Follow the writing rules in the README and two messages 
Remove "should" from the README and the semicolons from two messages.
Correct the note about a local run without CWA.
 2026-10-02 23:37:33 +02:00
tim 9c9c5b7dac Load a job record that holds unknown fields 
A downgrade of Bindery no longer stops at the start.
 2026-10-02 23:37:33 +02:00
tim 98c4a66da1 Answer 400 for a request value of the wrong type 
A value such as chapter_ids that is not a list of numbers gave a 500.
 2026-10-02 23:37:33 +02:00
tim 90457610f6 Wait only 5 seconds for CWA on the story page 
A job still waits for the full timeout.
 2026-10-02 23:37:33 +02:00
tim e196d1688e Keep server details out of API answers 
An unexpected error now sends a general message, and the log keeps the
text. A job no longer shows the path of its uploaded file. A cover is
cached only by the browser of the user.
 2026-10-02 23:37:33 +02:00
tim 0edfa1308c Stop the start with a clear message for a bad setting 
A bad UPDATE_INTERVAL_HOURS or LOG_LEVEL no longer ends in a traceback.
Warn about a CWA_PASSWORD without a CWA_USERNAME.
 2026-10-02 23:37:33 +02:00
tim 0506086533 Check every setting before the first one is stored 
A bad value in a settings request no longer leaves the other values of
that request saved.
 2026-10-02 23:37:33 +02:00
tim 7a36e26751 Build and send a book only when it changed 
A key of the chapters, rules and metadata of a story tells if its book
changed. A check without a change no longer builds the book again, also
for a book that waits for an upload. A held book goes out without a new
build. The Rebuild button still builds and sends in each case.
 2026-10-02 23:37:33 +02:00
tim f961e4781e Warn when the ingest folder has no library to match 
Without CWA_URL and CALIBRE_DB_PATH, each update goes to CWA as a new book.
 2026-10-02 21:04:49 +02:00
tim c188795852 Make CWA linking and uploads optional 2026-10-02 20:25:45 +02:00
tim 34ff432aac Remember story sorting in local storage 2026-10-01 16:08:46 +02:00
tim e4a17029c1 Sort stories through column headings 2026-10-01 16:08:42 +02:00
tim b536a1f116 Show story change times and compact list details 2026-10-01 15:42:48 +02:00
timandClaude Sonnet 5 13d7360f60 Show last update time on the story page 
The story page already showed last check time. Add last update time
next to it, from the existing last_changed field.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
 2026-09-28 21:28:48 +02:00
tim 8faab2463b Move and delete chapters by hand in the chapter list 
The merge cannot always tell where new chapters go. For example, a site
can list an announcement before the chapters of an old book. You can now
select chapters and move them, in their order, to after another chapter
or to the start. You can also delete them with their revisions. A story
keeps at least one chapter. The book is then rebuilt. A deleted chapter
that the site still lists comes back with the next download.

The Move… and Delete… buttons open a dialog that names the selected
chapters before anything changes. A shift-click selects a range of
chapters.

The Select all checkbox moves into the first heading of the chapter
list. On a phone, that heading shows with the text "Select all".
 2026-09-27 23:38:28 +02:00
tim ee611cd283 Sort the story list without a leading The, A or An 2026-09-27 23:38:28 +02:00
tim df9959a078 Add a script that cleans up smart punctuation in chapter revisions 
Some revisions differ from others only by curly quotes, curly
apostrophes and the ellipsis. The script deletes those revisions and
keeps the one with the least smart punctuation. In revisions with other
text changes, it makes the punctuation straight and stores the new text
hash. It does not change the newest revision if it came from the site.

The script uses only the Python standard library, so you can run it on
the host against bindery.db. It is a dry run unless you give --apply,
and --apply makes a backup first.
 2026-09-27 23:38:28 +02:00
tim 5df2806694 Add a script that fills a gap in a story from an ePub without chapter links 
A FanFicFare ePub records the site address of each chapter, so Bindery
can import it. Other ePubs, such as the author's own ebook of a story,
have no chapter links. The script asks which stored chapters come before
and after the gap, and which ePub chapters match them. It suggests
matches by title and puts the suggestion on the input line. It then adds
the ePub chapters between the matches, with their images, in the order
of the ePub.

Each added chapter gets a stable URL with the #bindery-import- marker
and the revision source "epub-import". A second run with the same ePub
does not add the same chapters again. --remove deletes the chapters that
the script added.
 2026-09-27 23:38:28 +02:00
61 changed files with 8304 additions and 793 deletions
Expand all files Collapse all files
AGENTS.md
+3 -1
View File
@@ -17,7 +17,8 @@ Keep the quotes around the glob in the node command. Without them, node reads `t
Start a development server with no CWA:
```bash
CONFIG_DIR=dev/config INGEST_DIR=dev/ingest TMP_DIR=dev/tmp uv run python -m bindery
mkdir -p dev/ingest
CWA_ENABLED=false CONFIG_DIR=dev/config INGEST_DIR=dev/ingest TMP_DIR=dev/tmp uv run python -m bindery
```
The server has no reloader. If you change Python code, restart it. Flask reads `bindery/static` from disk on each request, so a server that is not restarted sends new JavaScript to old Python routes.
@@ -27,6 +28,7 @@ The server has no reloader. If you change Python code, restart it. Flask reads `
- `bindery/fff.py` is the only module that calls FanFicFare.
- `bindery/cwa.py` talks to CWA through OPDS. Bindery never opens the CWA database.
- `bindery/diffing.py` hashes and diffs text, not HTML.
- `bindery/filename_template.py` expands Calibre's file name templates. It never runs template text as code.
- `bindery/jobs.py` keeps the job queue in SQLite, so jobs survive a restart.
- `tests/fake_cwa.py` is a fake CWA server.
README.md
+420 -154
View File
@@ -1,9 +1,13 @@
# Bindery
Bindery downloads web fiction into a
[Calibre-Web-Automated](https://github.com/crocodilestick/Calibre-Web-Automated) (CWA) library and
keeps the books up to date. It uses [FanFicFare](https://github.com/JimmXinu/FanFicFare), so it
supports the sites that FanFicFare supports. The tested site is RoyalRoad.
Bindery downloads web fiction and keeps each story up to date as an ePub. You give it the URL of a
story. Bindery downloads the chapters, checks for new chapters each day, and builds the book again
after each change. Your reader app gets the books from the OPDS catalog of Bindery.
Bindery uses [FanFicFare](https://github.com/JimmXinu/FanFicFare), so it supports the sites that
FanFicFare supports. The tested site is RoyalRoad.
Bindery can also send each book to a [Calibre-Web-Automated](https://github.com/crocodilestick/Calibre-Web-Automated) (CWA) library. See "Calibre-Web-Automated".
## How it differs from FanFicFare
@@ -11,8 +15,8 @@ FanFicFare builds a book from the chapter list on the site, so a chapter that th
also goes from your book. **Bindery keeps every chapter that it downloads** in its own database, and
builds each ePub from that database.
- **Chapters that the site removes stay in the book.** A new chapter goes after the chapter that
comes before it on the site. For example, you have chapters 1 to 60, and the site now lists only 1
- **Chapters that the publisher removes stay in the book.** A new chapter goes after the chapter that
comes before it on the publishing site. For example, you have chapters 1 to 60, and the publishing site now lists only 1
to 4 and 61. The book then has chapters 1 to 61.
- **A changed chapter keeps its old text as a revision.** You can compare the revisions and choose
the one that goes in the book.
@@ -21,9 +25,6 @@ builds each ePub from that database.
## Quick start
**You need CWA v4.0.1 or later.** Older versions can import an update as a new book. See
"Limitations".
1. Get the code, and make the folder for Bindery's data.
```bash
@@ -32,8 +33,8 @@ builds each ePub from that database.
mkdir config
```
2. Edit `docker-compose.yml` or environment variables. Set the CWA ingest folder, the CWA address and account, and the user
and group of the container.
2. Edit `docker-compose.yml`. Set the user and group of the container. **That user must be able to
write to `./config`.**
3. Build and start the container.
@@ -41,10 +42,54 @@ builds each ePub from that database.
docker compose up -d --build
```
4. Open `http://localhost:8085`, and log in with a CWA user.
4. Open `http://localhost:8085`. Under "Add stories", enter the URL of a story, and click "Add".
Bindery downloads the chapters and builds the book. To get the book at once, click "Download ePub"
on the story page.
The container runs as the `user` in `docker-compose.yml`. **That user must be able to write to
`./config` and to the CWA ingest folder.** Usually this is the same user and group as CWA.
5. Open Settings. Under "Book Downloads", check "Enable OPDS", and save. In your reader app, add the
catalog `http://localhost:8085/opds`, with your server address in place of `localhost`.
Bindery then checks each story for new chapters every 24 hours. When a story changes, Bindery builds
its book again, and the catalog has the new book.
The example Compose file asks for no login. Set a login before other people can reach Bindery. See
"Logins".
## OPDS catalog
OPDS lets a reader app browse and download books from Bindery. An admin turns it on with "Enable OPDS"
under Settings > Book Downloads. The catalog URL is `http://localhost:8085/opds`, with your server
address in place of `localhost`. If Bindery runs under `URL_BASE`, put it before `/opds`.
The catalog has three lists, with 50 books per page:
- By Title, in title order without a leading "The", "A" or "An".
- By Author, in surname order. Each author leads to their books. A book shows under each of its authors.
- By Last Changed, with the newest change first.
Last Changed records when Bindery stores a new chapter or a new revision of an existing chapter.
Rebuilds, metadata edits and rule changes do not update this date.
A book appears when it has a stored, included chapter. It shows its authors, series, description, tags,
language and cover, with your kept values in place of the published values. A reader can search by title,
author, series and tag. A download uses the stored chapters, the current ePub rules and the download
template for the file name.
`AUTH_METHOD` sets how a reader logs in:
- `none`: The reader needs no login.
- `local`: The reader sends your Bindery user name and password with HTTP Basic authentication.
A browser with a Bindery session works too.
- `cwa`: The reader sends your CWA user name and password with HTTP Basic authentication.
A browser with a Bindery session works too.
- `proxy`: Each user sets an OPDS password under Settings > Book Downloads. Settings shows the user
name for the reader, which is the name in `PROXY_AUTH_USER_HEADER`. The reader sends that name and
password with HTTP Basic authentication. The password has 8 to 256 characters, and Bindery keeps a
hash of it. Set the proxy to skip its login for the catalog. See "Paths without the proxy login".
Bindery keeps a good login for five minutes. Wrong passwords count toward the lockouts in
"Lockouts and sessions". With `TRUST_PROXY=true`, each reader has its own lockout.
Use HTTPS outside a trusted local network, because HTTP Basic sends the password with each request.
## Configuration
@@ -52,28 +97,26 @@ Environment variables set the configuration. **Restart the container after a cha
| Variable | Description | Default value |
| --- | --- | --- |
| `CWA_URL` | The address of CWA, for example `http://calibre-web-automated:8083` | None |
| `CWA_USERNAME`, `CWA_PASSWORD` | The CWA account of Bindery | None |
| `CWA_DB_PATH` | Calibre-Web's `app.db`, for logins. See "CWA logins". | None |
| `CALIBRE_DB_PATH` | The calibre `metadata.db`, for book ids. See "How books reach the library". | None |
| `UPDATE_INTERVAL_HOURS` | Hours between checks for new chapters. `0` turns them off. | `24` |
| `DEFAULT_LOCKED_FIELDS` | Fields that new stories lock: `title`, `authors`, `comments`, `tags`, `series`, `pubdate`, `languages` | `title` |
| `AUTH_METHOD` | `none`, `local`, `cwa` or `proxy`. See "Logins". | `none` |
| `BINDERY_USERNAME` | The user name for `AUTH_METHOD=local`. Case-sensitive. | None |
| `BINDERY_PASSWORD_HASH` | The password hash for `AUTH_METHOD=local`. See "Local login". | None |
| `CWA_ALLOWED_USERS` | The CWA users that can log in. Empty allows every CWA user. | Empty |
| `BINDERY_ADMIN_USERS` | The users that can change the global settings. See "Admins". | Empty |
| `PROXY_AUTH_USER_HEADER` | The header in which a reverse proxy sends the user name | `X-Remote-User` |
| `PROXY_AUTH_TRUSTED_IPS` | The addresses and networks that can send the user header. See "Logins". | Loopback and private addresses |
| `PROXY_AUTH_LOGOUT_URL` | Where the Log out button goes in proxy mode | None |
| `BINDERY_API_KEY` | A key for scripts. See "Scripts". | None |
| `ALLOWED_HOSTS` | The host names that Bindery answers to. See "Host names". | None |
| `SESSION_COOKIE_SECURE` | `true` if users reach Bindery only over HTTPS | `false` |
| `TRUST_PROXY` | `true` if a reverse proxy sends the client address. See "Lockouts and sessions". | `false` |
| `URL_BASE` | A path prefix, such as `/bindery`. See "Reverse proxy". | None |
| `INGEST_DIR` | The CWA ingest folder | `/cwa-book-ingest` |
| `INGEST_DIR` | A folder that gets each built ePub. With CWA, the CWA ingest folder. Bindery does not make this folder. | None |
| `CONFIG_DIR` | The story database and `personal.ini` | `/config` |
| `TMP_DIR` | The folder for ePubs while Bindery builds them | `/tmp/bindery` |
| `TMP_DIR` | The folder for the built ePubs. Bindery keeps the last build of each story. | `/tmp/bindery` |
| `LOG_LEVEL` | The log level | `INFO` |
Separate the values in a list with commas.
Separate the values in a list with commas. The variables for CWA are in "Calibre-Web-Automated".
### Logins
@@ -90,6 +133,24 @@ Separate the values in a list with commas.
because Bindery trusts this header. If a request gets a 401 because the proxy session ended,
Bindery reloads the page, so the proxy can step in again.
Bindery accepts the header only from the addresses in `PROXY_AUTH_TRUSTED_IPS`. The default is the
loopback and private addresses, such as `127.0.0.1`, `10.x`, `172.16.x` to `172.31.x` and
`192.168.x`. This stops a request that comes straight from the internet, but not one from your own
network. **Set it to the address of the proxy**, for example `PROXY_AUTH_TRUSTED_IPS=172.18.0.3`,
or publish the port of Bindery only on the host, as `127.0.0.1:8085:8085` in the Compose file.
`TRUST_PROXY` does not change this check. It is only for the lockouts.
#### Paths without the proxy login
An OPDS reader cannot use the login page of the proxy, such as OIDC. Set the proxy to skip its login
for `/opds` and each path below it. If you set `URL_BASE`, put it before the path. As a regular
expression, this is `^/opds([/?].*)?$`. In Authelia, add an `access_control` rule with
`policy: bypass` and this expression in `resources`. In Authentik, add it to "Unauthenticated
Paths" of the proxy provider.
On these paths, Bindery checks the OPDS login itself. See "OPDS catalog". Keep the proxy login on
`/api` and on all other paths. The health check at `/api/health` needs no login.
**If the login settings are incomplete or wrong, Bindery refuses every request**, also with the API
key, and shows the reason.
@@ -134,27 +195,6 @@ key, and shows the reason.
To change the account, change the user name or make a new hash, and restart Bindery. This ends the
open sessions.
#### CWA logins
`AUTH_METHOD=cwa` needs `CWA_URL`, `CWA_DB_PATH` or both. Without `CWA_DB_PATH`, Bindery sends the
user name and password to `CWA_URL/opds` as an HTTP Basic login. This is the OPDS login.
With `CWA_DB_PATH`, Bindery checks the password against the account in Calibre-Web's `app.db`. A
wrong password or an unknown account fails at once. If Bindery cannot read the file or its password
hash, it uses the OPDS login.
`CWA_DB_PATH` is the path of `app.db` inside the container. To use it, mount the CWA config folder
read-only into the container. Mount the whole folder, so that SQLite can read the journal files
beside the database. **Bindery does not write this database**, and access should ideally be read-only.
**Database logins check only the password.** CWA's external login providers and multi-factor
authentication do not apply. If they must apply, use the `proxy` mode.
- The CWA guest user cannot log in.
- If `CWA_ALLOWED_USERS` is set, only those users can log in. Bindery checks this list on each
request.
- `CWA_USERNAME` and `CWA_PASSWORD` are only for the library lookups of Bindery.
#### Lockouts and sessions
The `local` and `cwa` modes lock out password guesses:
@@ -177,6 +217,43 @@ open session until that session ends. To stop a CWA user at once, remove the use
`CWA_ALLOWED_USERS` and restart Bindery. To end all sessions, delete `/config/.flask_secret` and
restart Bindery.
#### Admins
In the `cwa` and `proxy` modes, `BINDERY_ADMIN_USERS` lists the users that are admins, for example
`BINDERY_ADMIN_USERS=alice, bob`. Only an admin can do these things:
- Open the Settings page, and change the settings on it.
- Change the global replacement rules.
- Read and change `personal.ini`, which can hold passwords for publishing sites.
Each other user can still add, update and delete stories, and change the settings and the rules of a
story. Bindery ignores the case of the names.
If the list is empty, each user is an admin. The `none` and `local` modes have only one user, who is
always an admin. The API key is always an admin too.
### Host names
Without a login, a web page that you visit can try to reach Bindery from your browser, under a host
name of its own that points at the address of Bindery. To stop this, the `none` login mode answers
only to these host names:
- An address, such as `192.168.1.5`.
- `localhost`, and a name without a dot, such as `nas` or the name of a container.
- A name of a private network. These end in `.local`, `.lan`, `.home`, `.home.arpa`, `.internal`,
`.localhost` or `.localdomain`.
For each other name, Bindery answers with an error that shows the name. If you use such a name, for
example behind a reverse proxy, list it in `ALLOWED_HOSTS`:
- `ALLOWED_HOSTS=bindery.example.com` allows that name. Bindery then answers only to the names in
the list, in each login mode. Write the names without a port.
- A leading dot, as in `.example.com`, also allows the subdomains.
- `ALLOWED_HOSTS=*` allows each name.
The other login modes answer to each name, unless you set a list. The health check at `/api/health`
works under each name.
### Scripts
Scripts can use the JSON API of the web interface.
@@ -221,13 +298,13 @@ explains the format, and
setting. Bindery reads the file for each job, so a change applies to the next job.
You can edit the file on the Settings page. Bindery refuses a file that FanFicFare cannot read, and
shows FanFicFare's warnings for other problems. **The file can hold passwords for story sites, and
every user who can log in to Bindery can read them.**
shows FanFicFare's warnings for other problems. **The file can hold passwords for publishing sites, and
each admin can read them.** If `BINDERY_ADMIN_USERS` is not set, each user is an admin. See "Admins".
Bindery always sets `always_reload_first_chapter`, `mark_new_chapters`, `continue_on_chapter_error`,
`make_directories`, `zip_output` and `calibre_series_meta`, because it needs fixed values for them.
**Each book gets the cover from the site.** FanFicFare downloads the cover only if `include_images`
**Each book gets the cover from the publisher.** FanFicFare downloads the cover only if `include_images`
is on. If it is off, Bindery sets it to `coveronly`, which keeps the cover but leaves out the images
in the story text.
@@ -240,93 +317,9 @@ To serve Bindery under a path, such as `/bindery/`, do one of these:
The health check at `/api/health` works with and without the prefix.
## How books reach the library
Bindery gives every book to CWA through the ingest folder. CWA makes all changes to the library.
1. Bindery puts the ePub of a new story into the ingest folder. CWA imports it as a new book.
2. At the next update, Bindery finds the CWA book of the story. If you set `CALIBRE_DB_PATH`, it
first looks for the story URL in the library database. Without a single match there, it searches
the CWA OPDS feed (using `CWA_URL`, `CWA_USERNAME` and `CWA_PASSWORD`) for the story title. The
search ignores the parts of a title in parentheses or brackets, such as tags or "(Book 1 Complete)".
Bindery downloads up to 5 ePubs with a matching title, closest title first. The book whose ePub
holds the story URL is the match. Bindery saves the id and uuid of the book with the story.
3. For the update, Bindery puts two files into the ingest folder: the new ePub, and a sidecar file
`<name>.epub.cwa.json` that holds `{"action": "add_format", "book_id": N}`. CWA then replaces the
ePub of book N.
Before each later update, Bindery finds the book by its uuid, first in the library database if you
set `CALIBRE_DB_PATH`, then through CWA. If you deleted the book in CWA, Bindery unlinks the story.
`CALIBRE_DB_PATH` is optional. It is the path of the calibre `metadata.db` inside the container, and
it lets Bindery find a book by its story URL instead of its title. To use it, mount the library
folder read-only into the container. Mount the whole folder, so that SQLite can read the journal
files beside the database. The Compose file has an example. **Bindery does not write to this database**,
and access should ideally be read-only.
Be sure to use the library that gets the ingest files. A different library can make Bindery update the
wrong book. Database lookups also find books that the CWA account of Bindery cannot see. If Bindery
cannot read the database, it uses CWA for every lookup. Keep `CWA_URL`, `CWA_USERNAME` and
`CWA_PASSWORD` set, because library imports and CWA lookups use them.
A book that the FanFicFare calibre plugin made holds its story URL. If you add its URL, or import it
from the library, Bindery links that book and imports its chapters from the ePub.
### If Bindery cannot find the book
The update fails, and the story page shows the error "Bindery cannot find this story in CWA". The
chapters stay stored, and the next successful update includes them. This happens if:
- CWA did not import the ePub.
- The title of the book changed in CWA before Bindery linked it.
- The CWA account of Bindery cannot see the book.
- You deleted the book in CWA.
To fix it, go to "Library link" on the story page, and do one of these:
- Search for the book, and click "Link". Then click "Rebuild book" to send the update.
- Click "Send to CWA as a new book". Use this if CWA did not import the first version, or if you
deleted the book.
### Hold uploads
Two settings on the Settings page stop books from going to CWA. Bindery still downloads the chapters
and builds each book. The story list marks a book that waits with "held".
- **Hold new links for approval.** Use this to catch a story that is linked to the wrong CWA book.
The first upload of a new story waits for your approval. The first upload after a change of the
library link also waits. This includes a link that you make, and a book that Bindery finds after
the linked book is gone. The story page names the book that gets the update, or says that the
story goes to CWA as a new book.
- **Pause all uploads.** No book goes to CWA. When you turn the pause off, Bindery rebuilds each
book that waits and sends it.
To approve a link, open the story page and do one of these:
- If the book is right, click "Approve and send".
- If the book is wrong, search for the right book under "Library link", and click "Link". Then
approve the new link.
- If CWA does not have the story, click "Send as a new book instead".
The book that CWA imports from the first upload of a story needs no approval. After an approval,
later updates go to CWA as usual.
### The CWA account
Make a CWA user for Bindery with these settings:
- The download permission. Bindery reads story URLs and chapters from the ePubs.
- "Show Publisher Section", in the sidebar settings of the user. CWA shows its list of publishers in
OPDS only to users with this setting. "Import from the library" uses the list to find the books of
story sites, because FanFicFare sets the publisher of a book to the site name.
- Access to all books. Bindery finds only the books that this account can see.
Bindery sends the password of this account with each OPDS request. **Use a password that no other
service uses.**
## Import an ePub file
If the site removed chapters before Bindery tracked the story, import an ePub that you made earlier
If the publishing site removed chapters before Bindery tracked the story, import an ePub that you made earlier
with FanFicFare. Bindery stores its chapters as if it downloaded them. Chapters that the site
publishes later go after them.
@@ -339,20 +332,85 @@ publishes later go after them.
that FanFicFare writes into each ePub. A chapter that Bindery stored already keeps its stored text.
The other chapters go in the order of the ePub.
After the import, Bindery checks the site for new chapters and sends the book to CWA. If you clear
After the import, Bindery checks the publishing site for new chapters and builds the book. If you clear
"Check the site for new chapters after import", Bindery only rebuilds the book of a story that it
downloaded before.
## File names of downloaded books
"Download ePub" on a story page and each download from the OPDS catalog send the book with a file
name. A template gives the name. The default template is `{title_sort} - {authors}`. For "The Book"
by Ann Writer, it gives `Book, The - Ann Writer.epub`. That is the name that Calibre's default template gives.
An admin can change the template under "Book Downloads" in Settings. To go back to the default,
clear the field and save. The template follows
the rules of [Calibre's templates](https://manual.calibre-ebook.com/template_lang.html) for file
names. Bindery adds `.epub`. For example, `{series:|| - }{series_index:0>2s|| - }{title} - {authors}`
gives `The Saga - 02 - The Book - Ann Writer.epub`.
In Calibre, `{title}` and `{series}` give the title sort in a file name, for example `Book, The`.
In Bindery, they keep the article. This is the same as Calibre with the tweak
`save_template_title_series_sorting` set to `strictly_alphabetic`. Use `{title_sort}` and
`{series_sort}` for the title and the series with the article at the end. `{series_sort}` is a
Bindery field. Calibre does not have it, and gives empty text for it.
| Field | Value |
| --- | --- |
| `{title}` | The title: `The Book`. |
| `{title_sort}` | The title with a leading "The", "A" or "An" at the end: `Book, The`. |
| `{authors}`, `{author}` | The authors, with ` & ` between them. |
| `{author_sort}` | The authors with the last name first: `Writer, Ann`. |
| `{series}` | The series. |
| `{series_sort}` | The series with a leading article at the end: `Saga, The`. |
| `{series_index}` | The series number: `2`, or `2.5`. |
| `{tags}` | The tags in sorted order, with `, ` between them. |
| `{languages}` | The language, as Bindery stores it. |
| `{publisher}` | The publishing site. |
| `{pubdate}` | The published date. |
| `{timestamp}` | The date when Bindery added the story. |
| `{last_modified}` | The date when the chapters last changed. |
| `{id}` | The Bindery story ID. With CWA on, the CWA book ID of a story that has a library book. |
- A field with no value becomes empty text.
- `{field:format|text before|text after}` adds the two texts only when the field has a value. Each
of the three parts can be empty. `{series:|| - }` adds ` - ` after the series.
- The format is a Python format. `{series_index:0>3s}` gives `002`, and `{title:.20s}` gives the
first 20 characters. `{author_sort[0]}` gives the first character.
- `{{` and `}}` give braces. Bindery makes each run of spaces one space.
- A download has one file name. If the template has folders, Bindery uses the part after the last
`/`. That is the name that Calibre gives the file.
- Bindery does not support template functions, such as `{title:uppercase()}`, or the program modes
of Calibre. It refuses such a template when you save it.
- If the template fails for a book, or gives an empty name, Bindery names the file after the title.
The log has a warning.
- Bindery replaces each character that a file name cannot have with `_`. A name has 100 characters
and 200 bytes at most.
"Date format" sets the form of `{pubdate}`, `{timestamp}` and `{last_modified}`. It takes the
[codes of Python's strftime](https://docs.python.org/3/library/datetime.html#format-codes). `%d`
gives the day, `%b` the month name, `%m` the month number and `%Y` the year. The default is
`%b, %Y`, as in Calibre. It gives `May, 2020`. To go back to the default, clear the field and save.
A `/` in a date becomes `_`, as in Calibre.
Bindery follows Calibre 9.2.1 for English, with the tweak above. The order of tags with letters
outside ASCII can differ from Calibre. The template applies only to downloads in the browser. It
does not change the names of the files in `INGEST_DIR`. The browser can ignore the name.
## Download modes
| Mode | What it does |
| --- | --- |
| Download new chapters | Downloads the chapters that are not stored yet. The default, and what scheduled updates use. |
| Recheck selected chapters | Downloads the selected chapters again. Changed text becomes a new revision. |
| Recheck all chapters | Downloads every chapter on the site again, and new chapters too. |
| Download and replace entire book | Downloads every chapter again. Chapters that the site no longer lists leave the book. They stay stored, and you can put them back. |
| Recheck all chapters | Downloads every chapter on the publishing site again, and new chapters too. |
| Download and replace entire book | Downloads every chapter again. Chapters that the publishing site no longer lists leave the book. They stay stored, and you can put them back. |
| Update metadata from site | Updates only the metadata in the ePub. |
| Rebuild book | Builds the book again from the stored chapters, for example after you change a rule. |
| Rebuild book | Builds the book again from the stored chapters. |
**Bindery builds a book only if the book changed.** A change is a new, changed, moved or renamed
chapter, a changed rule, or changed metadata. A job that finds no change answers "No changes". Only
"Rebuild book" builds a book that did not change. With CWA on, Bindery sends each book that it
builds to CWA.
### Find edited chapters
@@ -366,26 +424,26 @@ recheck, or use one of these settings:
story. This recheck takes the place of the next scheduled update. The story then ignores the "last
N chapters" setting.
Each rechecked chapter is one request to the site, so keep the numbers small for long stories.
Each rechecked chapter is one request to the publishing site, so keep the numbers small for long stories.
If the site changes the title of a chapter, "Download new chapters" and "Recheck selected chapters"
If the publishing site changes the title of a chapter, "Download new chapters" and "Recheck selected chapters"
download that chapter again. The chapter gets the new title only if the download works. On sites
that number chapter URLs, such as FanFiction.net, this also finds chapters that moved when the
author inserted a chapter.
### Revisions
A recheck compares the text of a chapter with its newest revision, which is the text that the site
sent last. Bindery stores the HTML as the site sends it, before the rules, so a rule never makes a
A recheck compares the text of a chapter with its newest revision, which is the text that the publishing site
sent last. Bindery stores the HTML as the publishing site sends it, before the rules, so a rule never makes a
chapter look changed.
- If the text changed, the new revision goes in the book. If you chose an older revision, it
stays in the book until the site changes the chapter again.
stays in the book until the publishing site changes the chapter again.
- On the page of a chapter, you can compare revisions, choose the one that goes in the book, and
delete the others. The revision in the book and the newest revision cannot be deleted.
- A chapter imported from an ePub holds its text after the FanFicFare settings and rules of that
build. If the current rules turn the site text into the imported text, Bindery counts the chapter
as unchanged, and stores the site text in its place. Other differences, for example from settings
build. If the current rules turn the published text into the imported text, Bindery counts the chapter
as unchanged, and stores the published text in its place. Other differences, for example from settings
in `personal.ini`, show as a change.
### Restarts
@@ -395,10 +453,13 @@ After a restart, the waiting jobs run. A job that was running starts again at th
queue, and skips the chapters that it stored. If Bindery stops during the same job three times, the
job fails.
If a job thread stops, `/api/health` answers 503, so the container shows as unhealthy. Restart
Bindery to start the thread again.
### Backups
Once a day, Bindery copies its database and `personal.ini` to `/config/backups`, and keeps the
copies of the last 7 days. **The database is the only copy of the chapters that sites removed.** The
copies of the last 7 days. **The database is the only copy of the chapters that publishing sites removed.** The
backups are on the same disk, so also copy `/config` to another disk or machine.
To restore a backup, do these steps:
@@ -415,7 +476,7 @@ Bindery keeps the metadata of each story in its database, and writes it into eac
are title, authors, description, tags, series, published date and language. Each field is locked or
unlocked for each story:
- An unlocked field takes the site value on every update.
- An unlocked field takes the published value on every update.
- A locked field keeps its stored value. The first download sets that value. For a story that you
import from the library or from an ePub file, the value comes from that ePub.
- To update a locked field once, select it under "Also take these locked or edited fields from the
@@ -433,9 +494,36 @@ CWA before, it rebuilds the book after an edit.
`DEFAULT_LOCKED_FIELDS` sets the locks for new stories. The default is `title`. The publisher is
always the site name.
**CWA reads the metadata and the cover of an ePub only when it imports a new book.** An update
replaces only the ePub file. So edits that you make in CWA stay, and later changes on the site show
in the ePub but not in the CWA catalog.
## Page count
The story list shows an estimated page count beside the chapter count. The OPDS catalog shows it in the
description of a book. Bindery counts the pages as the last step when it builds an ePub, so the count
includes your replacement rules and the line-break setting. A story shows no page count until Bindery
builds its book.
The method comes from "Paragraphs (APNX Accurate)" in the
[Count Pages plugin](https://github.com/kiwidude68/calibre_plugins/wiki/Count-Pages) for calibre:
- A paragraph start is one line. Each 70 characters of text is one more line.
- A page is 31 lines.
- A page is at most 2400 characters. This limit sets the count of a book with almost no paragraphs.
The plugin counts either the `<p>` elements or the `<div>` elements of a book as its paragraphs. Bindery
counts each paragraph that a reader sees:
- Text starts a paragraph after a `<br>`, and after the start or the end of a block element such as `<p>`,
`<div>`, `<li>` or a heading.
- A run of `<br>` elements is one paragraph break.
- An element that only holds other blocks is not a paragraph.
- The cells of a table row share one line.
- A paragraph with no text is not counted. This includes a paragraph that holds only a no-break space,
which some stories put between paragraphs as a blank line.
So the same text gives the same count as `<p>` elements, as `<div>` elements, and as lines with `<br>`
between them. For a book that uses `<p>` elements and has no empty paragraphs, the count is within about 1%
of the count of the plugin. The plugin counts each empty paragraph as a line, so its count is higher for a
book that has many of them.
The count is an estimate and differs from the page count of a printed book.
## Replacement rules
@@ -458,6 +546,184 @@ stays. This step runs before the global rules. The global setting is off by defa
each story that has no setting of its own. To see the new paragraphs in the preview, select "Compare
HTML source".
## Calibre-Web-Automated
Bindery can send each book that it builds to a [Calibre-Web-Automated](https://github.com/crocodilestick/Calibre-Web-Automated) (CWA) library, and update that book when the
story changes. CWA is off by default. To turn it on:
1. Set `CWA_ENABLED=true`.
2. Set `INGEST_DIR` to the CWA ingest folder. Set `CWA_URL`, `CALIBRE_DB_PATH` or both.
3. Restart Bindery.
Use CWA v4.0.1 or later. Older versions can import an update as a new book. See "Limitations".
An earlier Bindery used CWA unless `CWA_ENABLED` was `false`. After an upgrade, set `CWA_ENABLED=true`
to keep the uploads to CWA.
| Variable | Description | Default value |
| --- | --- | --- |
| `CWA_ENABLED` | `true` to use CWA | `false` |
| `CWA_URL` | The address of CWA, for example `http://calibre-web-automated:8083` | None |
| `CWA_USERNAME`, `CWA_PASSWORD` | The CWA account of Bindery. Leave them unset if CWA asks for no login. | None |
| `CALIBRE_DB_PATH` | The calibre `metadata.db`, for book ids. See "How books reach the library". | None |
| `CWA_DB_PATH` | Calibre-Web's `app.db`, for logins. See "CWA logins". | None |
| `CWA_ALLOWED_USERS` | The CWA users that can log in. Empty allows every CWA user. | Empty |
The example Compose file has these values as comments. The user of the container must be able to write
to the CWA ingest folder. Usually this is the same user and group as CWA.
Settings shows an alert for each missing value:
- Neither `CWA_URL` nor `CALIBRE_DB_PATH` is set. The alert is under "Uploads to CWA". Library linking
is then off, and its checkbox is disabled. Set either value and restart Bindery. Bindery keeps your
saved linking preference.
- `INGEST_DIR` is not set. The alert is under "Book Downloads". "Pause all uploads" is then checked
until you set `INGEST_DIR` and restart Bindery.
At each start, the log has a warning for each missing value.
### Without CWA
Without `CWA_ENABLED=true`:
- Bindery makes no request to CWA. It ignores `CWA_URL`, `CWA_USERNAME`, `CWA_PASSWORD`,
`CWA_DB_PATH` and `CALIBRE_DB_PATH`. If one of them is set, the log has a warning at each start.
- The web interface shows nothing about CWA. The pause, the library links and the approvals do not
apply.
- If `INGEST_DIR` is set, Bindery puts each changed book in that folder as a plain ePub. Use this to
import the books into other software. Each file gets a new name, so the earlier files stay in the
folder. If `INGEST_DIR` is not set, Settings shows an alert under "Book Downloads".
- `AUTH_METHOD=cwa` does not work. Use `local` or `proxy`. On a private network, you can use `none`.
### How books reach the library
Bindery gives every book to CWA through the ingest folder. CWA makes all changes to the library.
**Bindery does not make the ingest folder.** If the folder in `INGEST_DIR` does not exist, most often
because its volume mount is missing, the log shows an error at the start and each upload fails. The
books go to CWA with the next update after you fix the mount.
1. Bindery puts the ePub of a new story into the ingest folder. CWA imports it as a new book.
2. At the next update, Bindery finds the CWA book of the story. If you set `CALIBRE_DB_PATH`, it
first looks for the story URL in the library database. Without a single match there, it searches
the CWA OPDS feed (using `CWA_URL`, `CWA_USERNAME` and `CWA_PASSWORD`) for the story title. The
search ignores the parts of a title in parentheses or brackets, such as tags or "(Book 1 Complete)".
Bindery downloads up to 5 ePubs with a matching title, closest title first. The book whose ePub
holds the story URL is the match. Bindery saves the id and uuid of the book with the story.
3. For the update, Bindery puts two files into the ingest folder: the new ePub, and a sidecar file
`<name>.epub.cwa.json` that holds `{"action": "add_format", "book_id": N}`. CWA then replaces the
ePub of book N.
Before each later update, Bindery finds the book by its uuid, first in the library database if you
set `CALIBRE_DB_PATH`, then through CWA. If you deleted the book in CWA, Bindery unlinks the story.
`CALIBRE_DB_PATH` is optional. It is the path of the calibre `metadata.db` inside the container, and
it lets Bindery find a book by its story URL instead of its title. To use it, mount the library
folder read-only into the container. Mount the whole folder, so that SQLite can read the journal
files beside the database. The Compose file has an example. **Bindery does not write to this database.**
Be sure to use the library that gets the ingest files. A different library can make Bindery update the
wrong book. Database lookups also find books that the CWA account of Bindery cannot see. If Bindery
cannot read the database, it uses CWA for every lookup. Keep `CWA_URL` and the CWA account set,
because library imports and CWA lookups use them.
A book that the FanFicFare calibre plugin made holds its story URL. If you add its URL, or import it
from the library, Bindery links that book and imports its chapters from the ePub.
**CWA reads the metadata and the cover of an ePub only when it imports a new book.** An update
replaces only the ePub file. So edits that you make in CWA stay, and later changes on the site show
in the ePub but not in the CWA catalog.
### If Bindery cannot find the book
If library matching is on and configured, an update fails when Bindery cannot find the book.
The story page shows the error "Cannot find this story in CWA. Link to library book, or send as new book." The
chapters stay stored, and the next successful update includes them. This happens if:
- CWA did not import the ePub.
- The title of the book changed in CWA before Bindery linked it.
- The CWA account of Bindery cannot see the book.
- You deleted the book in CWA.
To fix it, go to "Library link" on the story page, and do one of these:
- Search for the book, and click "Link". Then click "Rebuild book" to send the update.
- Click "Send to CWA as a new book". Use this if CWA did not import the first version, or if you
deleted the book.
### Upload settings
The "Uploads to CWA" section of Settings controls library matching and uploads. Bindery still downloads chapters
and builds books when uploads stop. The story list marks a built book with "held" only when it waits
for approval, library linking is on, and uploads are not paused. Bindery keeps track of all unsent books.
- Link books to CWA library. This defaults to on when `CWA_URL` or `CALIBRE_DB_PATH` is set.
Without either value, it stays off and disabled. Turn it off to skip matching with a configured library connection.
Uploads use saved CWA book IDs when available. Without an ID, every build goes to `INGEST_DIR` as a plain ePub.
If CWA imports these files, updates will create duplicate books. Settings and the story page show a highlighted warning.
A change of this setting sends the unsent books, unless uploads are paused.
- **Hold new links for approval.** This setting applies only when library linking is on.
Use this to catch a story that is linked to the wrong CWA book.
The first upload of a new story waits for your approval. The first upload after a change of the
library link also waits. This includes a link that you make, and a book that Bindery finds after
the linked book is gone. The story page names the book that gets the update, or says that the
story goes to CWA as a new book.
- **Pause all uploads.** No book goes to CWA. This is always checked when `INGEST_DIR` is unset.
When you turn the pause off, Bindery sends the unsent books. The other upload settings still apply.
While paused, use "Approve" or "Approve as a new book instead" to save a decision without sending the book.
To approve a link, open the story page and do one of these:
- If the book is right, click "Approve and send".
- If the book is wrong, search for the right book under "Library link", and click "Link". Then
approve the new link.
- If CWA does not have the story, click "Send as a new book instead".
The book that CWA imports from the first upload of a story needs no approval. After an approval,
later updates go to CWA as usual.
### The CWA account
Make a CWA user for Bindery with these settings:
- The download permission. Bindery reads story URLs and chapters from the ePubs.
- "Show Publisher Section", in the sidebar settings of the user. CWA shows its list of publishers in
OPDS only to users with this setting. "Import from the library" uses the list to find the books of
publishing sites, because FanFicFare sets the publisher of a book to the site name.
- Access to all books. Bindery finds only the books that this account can see.
Bindery sends the password of this account with each OPDS request. **Use a password that no other
service uses.**
If CWA allows anonymous browsing, Bindery needs no account. Leave `CWA_USERNAME` and `CWA_PASSWORD`
unset, and give the guest user of CWA the settings above.
### CWA logins
`AUTH_METHOD=cwa` needs `CWA_ENABLED=true`, and `CWA_URL`, `CWA_DB_PATH` or both. Without `CWA_DB_PATH`, Bindery sends the
user name and password to `CWA_URL/opds` as an HTTP Basic login. This is the OPDS login.
**The OPDS login works only if CWA refuses a wrong password.** A CWA with anonymous browsing can
accept every OPDS login. Bindery tests this with a login that cannot exist. If CWA accepts it,
Bindery refuses each OPDS login, and the log names the reason. Set `CWA_DB_PATH` then, or use
another `AUTH_METHOD`.
With `CWA_DB_PATH`, Bindery checks the password against the account in Calibre-Web's `app.db`. A
wrong password or an unknown account fails at once. If Bindery cannot read the file or its password
hash, it uses the OPDS login.
`CWA_DB_PATH` is the path of `app.db` inside the container. To use it, mount the CWA config folder
read-only into the container. Mount the whole folder, so that SQLite can read the journal files
beside the database. **Bindery does not write this database.**
**Database logins check only the password.** CWA's external login providers and multi-factor
authentication do not apply. If they must apply, use the `proxy` mode.
- The CWA guest user cannot log in.
- If `CWA_ALLOWED_USERS` is set, only those users can log in. Bindery checks this list on each
request.
- `CWA_USERNAME` and `CWA_PASSWORD` are only for the library lookups of Bindery.
## Limitations
- **Bindery identifies a chapter by its URL.** Most sites, such as RoyalRoad and AO3, keep the URL
@@ -466,8 +732,8 @@ HTML source".
the same, run "Recheck all chapters". When an author removes a chapter there, the book loses that
chapter, and the last chapter appears twice. The removed chapter stays as an older revision of the
chapter that took its URL.
- "Stop tracking" deletes only the data of Bindery. The book stays in CWA.
- **Updates need CWA v4.0.1 or later.** CWA v4.0.0 added the `.cwa.json` sidecar files. But v4.0.0
- "Stop tracking" deletes only the data of Bindery. A book in CWA stays there.
- **Updates of a CWA library need CWA v4.0.1 or later.** CWA v4.0.0 added the `.cwa.json` sidecar files. But v4.0.0
can delete a sidecar file before it imports the ePub, and then imports the update as a new book.
Versions before v4.0.0 cannot read sidecar files, so each update becomes a new book.
@@ -490,13 +756,13 @@ The Python tests use a fake CWA server and temporary folders, so they need no CW
sites. The frontend tests use the built-in test runner of Node, and need no npm packages. Also check
the pages in a browser.
### Run locally without CWA
### Run locally
Without CWA, Bindery puts the first download of each story into its ingest folder as an ePub. Later
downloads fail, because Bindery needs CWA to find the book, but their chapters stay stored. This is
enough to work on downloads, chapters, rules and the web interface.
CWA is off by default. Bindery puts each new build of a story into its ingest folder as an
ePub. This is enough to work on downloads, chapters, rules and the web interface.
```bash
mkdir -p dev/ingest
CONFIG_DIR=dev/config INGEST_DIR=dev/ingest TMP_DIR=dev/tmp uv run python -m bindery
```
@@ -528,7 +794,7 @@ not use your real library.**
test logins too.
```bash
CONFIG_DIR=dev/config INGEST_DIR=dev/ingest TMP_DIR=dev/tmp \
CWA_ENABLED=true CONFIG_DIR=dev/config INGEST_DIR=dev/ingest TMP_DIR=dev/tmp \
CWA_URL=http://localhost:8083 CWA_USERNAME=admin CWA_PASSWORD=admin123 AUTH_METHOD=cwa \
uv run python -m bindery
```
bindery/auth.py
+211 -27
View File
@@ -1,20 +1,26 @@
"""Logins: none, one local account, CWA accounts, or a proxy. API keys work in every mode.
"""Logins: none, one local account, CWA accounts, or a proxy. API keys work in every mode. The OPDS catalog takes
the account login, and in proxy mode a password of its own.
CWA accounts use an optional read-only app.db, with OPDS as the fallback when
the database cannot answer. A rejected database login never falls back.
the database cannot answer. A rejected database login never falls back. The
OPDS login works only with a CWA that refuses a wrong password.
"""
import base64
import hashlib
import hmac
import ipaddress
import logging
import secrets
import threading
import time
from collections.abc import Mapping
from collections.abc import Callable, Mapping
from dataclasses import dataclass
from functools import lru_cache
from typing import Any
from urllib.parse import urlsplit
from werkzeug.security import check_password_hash
from werkzeug.security import check_password_hash, generate_password_hash
from bindery import calibre_db, config
from bindery.cwa import CwaClient, CwaError
@@ -25,6 +31,9 @@ AUTH_NONE, AUTH_LOCAL, AUTH_CWA, AUTH_PROXY = "none", "local", "cwa", "proxy"
MAX_FAILURES = 5 # for one user name from one address
MAX_ADDRESS_FAILURES = 20 # for all user names from one address
LOCKOUT_SECONDS = 15 * 60
LOCKED = "Too many wrong passwords. Try again in 15 minutes."
# Name endings that only a private network can resolve.
LOCAL_SUFFIXES = (".local", ".lan", ".home", ".home.arpa", ".internal", ".localhost", ".localdomain")
class AuthError(Exception):
@@ -53,6 +62,48 @@ def _valid_local_hash(value: str) -> bool:
return False
@lru_cache(maxsize=8)
def _networks(values: tuple[str, ...]) -> tuple[ipaddress.IPv4Network | ipaddress.IPv6Network, ...]:
"""The networks of a list of addresses and networks. Raises ValueError for a value that is neither."""
return tuple(ipaddress.ip_network(v, strict=False) for v in values)
def _trusted_proxy(address: str | None) -> bool:
"""Whether the request comes from an address that can send the user header."""
try:
ip = ipaddress.ip_address(address or "")
except ValueError:
return False
if ip.version == 6 and ip.ipv4_mapped:
ip = ip.ipv4_mapped
return any(ip.version == n.version and ip in n for n in _networks(tuple(config.PROXY_AUTH_TRUSTED_IPS)))
def host_allowed(host: str) -> bool:
"""Whether Bindery answers a request with this Host header.
Without a login, each web page that the user visits could reach Bindery under
a name of its own that points at the address of Bindery (DNS rebinding), and
read and change everything. Such a name is a public name, so the "none" mode
answers only to addresses and to names of a private network.
"""
try:
name = (urlsplit(f"//{host}").hostname or "").rstrip(".")
except ValueError:
return False
if config.ALLOWED_HOSTS:
return any(
h == "*" or name == h.lstrip(".") or (h.startswith(".") and name.endswith(h)) for h in config.ALLOWED_HOSTS
)
if config.AUTH_METHOD != AUTH_NONE:
return True
try:
ipaddress.ip_address(name)
return True
except ValueError:
return bool(name) and ("." not in name or name.endswith(LOCAL_SUFFIXES))
def local_password_version() -> str:
"""Invalidate local sessions when the configured password hash changes."""
return hashlib.sha256(config.LOCAL_PASSWORD_HASH.encode()).hexdigest()
@@ -71,22 +122,38 @@ def config_error() -> str | None:
return "AUTH_METHOD=local needs BINDERY_USERNAME and BINDERY_PASSWORD_HASH"
if not _valid_local_hash(config.LOCAL_PASSWORD_HASH):
return "BINDERY_PASSWORD_HASH must be a Werkzeug scrypt or PBKDF2 hash of a nonempty password"
if method == AUTH_CWA and not config.CWA_ENABLED:
return "AUTH_METHOD=cwa needs CWA_ENABLED=true"
if method == AUTH_CWA and not (config.CWA_URL or config.CWA_DB_PATH):
return "AUTH_METHOD=cwa needs CWA_URL or CWA_DB_PATH"
if method == AUTH_PROXY and not config.PROXY_AUTH_USER_HEADER:
return "AUTH_METHOD=proxy needs PROXY_AUTH_USER_HEADER"
if method == AUTH_PROXY:
if not config.PROXY_AUTH_USER_HEADER:
return "AUTH_METHOD=proxy needs PROXY_AUTH_USER_HEADER"
if not config.PROXY_AUTH_TRUSTED_IPS:
return "AUTH_METHOD=proxy needs PROXY_AUTH_TRUSTED_IPS, the addresses of the proxy"
try:
_networks(tuple(config.PROXY_AUTH_TRUSTED_IPS))
except ValueError as e:
return f"PROXY_AUTH_TRUSTED_IPS must be a list of addresses and networks: {e}"
return None
def active_mode() -> str:
return config.AUTH_METHOD
def _allowed(name: str) -> bool:
names = {n.casefold() for n in config.CWA_ALLOWED_USERS}
return not names or name.casefold() in names
def is_admin(identity: Identity) -> bool:
"""Whether this user can change the global settings, the global rules and personal.ini.
Only the cwa and proxy modes have more than one user. The API key is always an admin.
"""
names = {n.casefold() for n in config.ADMIN_USERS}
if not names or identity.via in ("api_key", "none") or config.AUTH_METHOD == AUTH_LOCAL:
return True
return identity.name.casefold() in names
def _api_key_identity(headers: Mapping[str, str]) -> Identity | None:
if not config.API_KEY:
return None
@@ -99,19 +166,9 @@ def _api_key_identity(headers: Mapping[str, str]) -> Identity | None:
return None
def current_identity(headers: Mapping[str, str], session: Mapping[str, Any]) -> Identity | None:
"""Who makes this request, or None when a login is needed."""
if config_error():
return None
identity = _api_key_identity(headers)
if identity:
return identity
mode = active_mode()
if mode == AUTH_NONE:
return Identity("", "none")
if mode == AUTH_PROXY:
name = (headers.get(config.PROXY_AUTH_USER_HEADER) or "").strip()
return Identity(name, "proxy") if name else None
def _session_identity(session: Mapping[str, Any]) -> Identity | None:
"""The user of a web session in the local or CWA mode."""
mode = config.AUTH_METHOD
name = session.get("user")
if not name or session.get("mode") != mode:
return None
@@ -123,6 +180,27 @@ def current_identity(headers: Mapping[str, str], session: Mapping[str, Any]) ->
return None
def current_identity(
headers: Mapping[str, str], session: Mapping[str, Any], remote_addr: str | None = None
) -> Identity | None:
"""Who makes this request, or None when a login is needed. `remote_addr` is the address that connects."""
if config_error():
return None
identity = _api_key_identity(headers)
if identity:
return identity
mode = config.AUTH_METHOD
if mode == AUTH_NONE:
return Identity("", "none")
if mode == AUTH_PROXY:
name = (headers.get(config.PROXY_AUTH_USER_HEADER) or "").strip()
if name and not _trusted_proxy(remote_addr):
logger.warning(f"Ignored the user header from {remote_addr}, which is not in PROXY_AUTH_TRUSTED_IPS")
return None
return Identity(name, "proxy") if name else None
return _session_identity(session)
def client_address(headers: Mapping[str, str], remote_addr: str | None) -> str:
"""The address of the client. With TRUST_PROXY, the last X-Forwarded-For entry, which the proxy added."""
if config.TRUST_PROXY:
@@ -164,6 +242,11 @@ class LoginLimiter:
kept = [x for x in self._failures.get(address, []) if x[1] != name.casefold()]
self._failures[address] = kept
def forget(self, name: str) -> None:
"""Clear the wrong passwords of a user name from every address."""
with self._lock:
self._failures = {a: [x for x in f if x[1] != name.casefold()] for a, f in self._failures.items()}
limiter = LoginLimiter()
@@ -174,17 +257,22 @@ def _login_ok(username: str, password: str, client: CwaClient) -> bool:
return calibre_db.login_ok(config.CWA_DB_PATH, username, password)
except calibre_db.DatabaseUnavailable as e:
logger.warning(f"Cannot read the Calibre-Web login database. Trying OPDS: {e}")
if client.accepts_any_login():
raise CwaError(
"CWA accepts every OPDS login, most often because anonymous browsing is on, so Bindery cannot check "
"a password through OPDS. Set CWA_DB_PATH, or use another AUTH_METHOD."
)
return client.login_ok(username, password)
def login_local(username: str, password: str, address: str) -> Identity:
username = username.strip()
if active_mode() != AUTH_LOCAL:
if config.AUTH_METHOD != AUTH_LOCAL:
raise AuthError("Bindery does not use local logins")
if error := config_error():
raise AuthError(error)
if limiter.locked(username, address):
raise AuthError("Too many wrong passwords. Try again in 15 minutes.")
raise AuthError(LOCKED)
# Check the password even for a wrong user name, so both failures take the same work.
password_ok = check_password_hash(config.LOCAL_PASSWORD_HASH, password)
if not (password_ok and password and hmac.compare_digest(username.encode(), config.LOCAL_USERNAME.encode())):
@@ -197,10 +285,10 @@ def login_local(username: str, password: str, address: str) -> Identity:
def login_cwa(username: str, password: str, client: CwaClient, address: str) -> Identity:
"""Check a CWA login. Raises AuthError with a message for the user."""
username = username.strip()
if active_mode() != AUTH_CWA:
if config.AUTH_METHOD != AUTH_CWA:
raise AuthError("Bindery does not use CWA logins")
if limiter.locked(username, address):
raise AuthError("Too many wrong passwords. Try again in 15 minutes.")
raise AuthError(LOCKED)
try:
ok = (
_allowed(username)
@@ -217,3 +305,99 @@ def login_cwa(username: str, password: str, client: CwaClient, address: str) ->
raise AuthError("Wrong user name or password")
limiter.clear(username, address)
return Identity(username, "session")
KEPT_SECONDS = 5 * 60
KEPT_LIMIT = 100
MAX_PASSWORD = 256
_kept_key = secrets.token_bytes(32)
_kept_logins: dict[bytes, float] = {} # keyed hash of a good login: end time
_kept_lock = threading.Lock()
def _basic_login(headers: Mapping[str, str]) -> tuple[str, str] | None:
"""The user name and password of an HTTP Basic login, or None."""
scheme, _, value = headers.get("Authorization", "").partition(" ")
if scheme.lower() != "basic":
return None
try:
username, colon, password = base64.b64decode(value.strip(), validate=True).decode().partition(":")
except ValueError:
return None
return (username, password) if colon and password and len(password) <= MAX_PASSWORD else None
def _kept(parts: tuple[Any, ...], check: Callable[[], bool]) -> bool:
"""Whether `check` accepts the login. Bindery keeps a good login for five minutes, so a reader that loads 50
covers checks a password hash once. `parts` holds everything that the check depends on."""
key = hmac.digest(_kept_key, repr(parts).encode(), "sha256")
with _kept_lock:
if _kept_logins.get(key, 0) > time.monotonic():
return True
if not check():
return False
with _kept_lock:
if len(_kept_logins) >= KEPT_LIMIT:
_kept_logins.clear()
_kept_logins[key] = time.monotonic() + KEPT_SECONDS
return True
def _account_login(username: str, password: str, address: str, client: Callable[[], CwaClient]) -> bool:
"""Whether this is a good login of the local or the CWA account. A wrong password counts toward the lockout."""
try:
if config.AUTH_METHOD == AUTH_LOCAL:
login_local(username, password, address)
else:
login_cwa(username, password, client(), address)
except AuthError:
return False
return True
@lru_cache(maxsize=1)
def _dummy_hash() -> str:
return generate_password_hash("not a password")
def _own_password_login(username: str, password: str, address: str, password_hash: str | None) -> bool:
"""Whether this is the OPDS password of this user. A wrong password counts toward the lockout."""
if limiter.locked(username, address):
return False
# A user without a password meets the same work as one with a password, so the time shows no difference.
if not (check_password_hash(password_hash or _dummy_hash(), password) and password_hash):
limiter.failed(username, address)
return False
limiter.clear(username, address)
return True
def catalog_login(
headers: Mapping[str, str],
session: Mapping[str, Any],
address: str,
password_hash_of: Callable[[str], str | None],
client: Callable[[], CwaClient],
) -> bool:
"""Whether the request may open the OPDS catalog.
The local and CWA modes take the web session or the account login in HTTP Basic. The proxy mode takes the user
name that the proxy gives for the web login, with the OPDS password of that user, because the proxy skips its
login on these paths and a client can send any user header there. The API key opens the catalog in no mode.
"""
mode = config.AUTH_METHOD
if mode == AUTH_NONE:
return True
if mode in (AUTH_LOCAL, AUTH_CWA) and _session_identity(session):
return True
login = _basic_login(headers)
if not login:
return False
username, password = login
if mode == AUTH_PROXY:
username = username.strip()
password_hash = password_hash_of(username)
parts = (mode, username.casefold(), password_hash, password)
return _kept(parts, lambda: _own_password_login(username, password, address, password_hash))
parts = (mode, username, password, local_password_version(), tuple(config.CWA_ALLOWED_USERS))
return _kept(parts, lambda: _account_login(username, password, address, client))
bindery/config.py
+87 -9
View File
@@ -8,8 +8,18 @@ from pathlib import Path
logger = logging.getLogger(__name__)
def _bool(name: str, default: str = "false") -> bool:
return os.getenv(name, default).strip().lower() in ("true", "yes", "1", "y", "on")
def flag(value: str, default: bool = False) -> bool:
"""A text value as true or false. A value that is neither gives the default."""
value = value.strip().lower()
if value in ("true", "yes", "1", "y", "on"):
return True
if value in ("false", "no", "0", "n", "off"):
return False
return default
def _bool(name: str, default: bool = False) -> bool:
return flag(os.getenv(name, ""), default)
def _list(name: str, default: str = "") -> list[str]:
@@ -27,6 +37,22 @@ def _optional_path(name: str) -> Path | None:
return Path(value).absolute() if value else None
# A bad value stops the start with one clear line, not with a traceback.
def _number(name: str, default: str) -> float:
value = os.getenv(name, default).strip()
try:
return float(value)
except ValueError:
raise SystemExit(f"{name} must be a number, not {value!r}") from None
def _choice(name: str, default: str, choices: tuple[str, ...]) -> str:
value = os.getenv(name, default).strip().upper()
if value not in choices:
raise SystemExit(f"{name} must be one of {', '.join(choices)}, not {value!r}")
return value
def _base_path(value: str) -> str:
path = value.strip().strip("/")
return f"/{path}" if path else ""
@@ -35,9 +61,9 @@ def _base_path(value: str) -> str:
PROJECT_ROOT = Path(__file__).resolve().parent.parent
CONFIG_DIR = _path("CONFIG_DIR", "/config")
INGEST_DIR = _path("INGEST_DIR", "/cwa-book-ingest")
INGEST_DIR = _optional_path("INGEST_DIR")
TMP_DIR = _path("TMP_DIR", "/tmp/bindery")
LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO").strip().upper()
LOG_LEVEL = _choice("LOG_LEVEL", "INFO", ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"))
# A path prefix, such as /bindery, for a reverse proxy that forwards the full path.
URL_BASE = _base_path(os.getenv("URL_BASE", ""))
@@ -48,8 +74,16 @@ CWA_PASSWORD = os.getenv("CWA_PASSWORD", "")
# Independent, optional read-only databases. HTTP remains the fallback.
CWA_DB_PATH = _optional_path("CWA_DB_PATH") # Calibre-Web app.db, for logins
CALIBRE_DB_PATH = _optional_path("CALIBRE_DB_PATH") # calibre metadata.db, for book IDs
# true if the user has CWA. Without it, Bindery ignores the CWA settings above, and shows nothing about CWA. INGEST_DIR
# still gets each built book, as a plain epub. Only a clear "true" turns CWA on.
CWA_ENABLED = _bool("CWA_ENABLED")
CWA_SETTINGS = ("CWA_URL", "CWA_USERNAME", "CWA_PASSWORD", "CWA_DB_PATH", "CALIBRE_DB_PATH")
IGNORED_SETTINGS = [] if CWA_ENABLED else [name for name in CWA_SETTINGS if os.getenv(name, "").strip()]
if not CWA_ENABLED:
CWA_URL = CWA_USERNAME = CWA_PASSWORD = ""
CWA_DB_PATH = CALIBRE_DB_PATH = None
UPDATE_INTERVAL_HOURS = float(os.getenv("UPDATE_INTERVAL_HOURS", "24"))
UPDATE_INTERVAL_HOURS = _number("UPDATE_INTERVAL_HOURS", "24")
DEFAULT_LOCKED_FIELDS = _list("DEFAULT_LOCKED_FIELDS", "title")
# Logins: none, local, cwa or proxy. An API key works in every mode.
@@ -57,15 +91,25 @@ AUTH_METHOD = os.getenv("AUTH_METHOD", "none").strip().lower()
LOCAL_USERNAME = os.getenv("BINDERY_USERNAME", "").strip()
LOCAL_PASSWORD_HASH = os.getenv("BINDERY_PASSWORD_HASH", "").strip()
CWA_ALLOWED_USERS = _list("CWA_ALLOWED_USERS")
# The users of the cwa and proxy modes that can change the global settings. Without a list, each user can.
ADMIN_USERS = _list("BINDERY_ADMIN_USERS")
PROXY_AUTH_USER_HEADER = os.getenv("PROXY_AUTH_USER_HEADER", "X-Remote-User").strip()
PROXY_AUTH_LOGOUT_URL = os.getenv("PROXY_AUTH_LOGOUT_URL", "").strip()
# The addresses and networks that can send the user header. The default is each loopback and private address, so
# a request that comes straight from the internet cannot send it.
PROXY_AUTH_TRUSTED_IPS = _list(
"PROXY_AUTH_TRUSTED_IPS", "127.0.0.0/8, ::1, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7, fe80::/10"
)
API_KEY = os.getenv("BINDERY_API_KEY", "").strip()
# The host names that Bindery answers to. A leading dot also allows the subdomains, and * allows every name.
# Without a list, the "none" login mode answers only to addresses and local names. The other modes answer to all.
ALLOWED_HOSTS = [h.lower() for h in _list("ALLOWED_HOSTS")]
SESSION_COOKIE_SECURE = _bool("SESSION_COOKIE_SECURE")
# Take the client address from the last entry of X-Forwarded-For, which a reverse proxy adds.
TRUST_PROXY = _bool("TRUST_PROXY")
# Tests turn off the job threads.
START_THREADS = _bool("BINDERY_START_THREADS", "true")
START_THREADS = _bool("BINDERY_START_THREADS", True)
SITE_DELAY = 5 # seconds between two jobs that go to a story site
JOB_LIST_MAX_HOURS = 24 * 30 # the highest allowed value of a "jobs from the last N hours" setting
@@ -83,10 +127,44 @@ BACKUP_COUNT = 7 # daily backups to keep
def prepare() -> None:
"""Make the folders, and give the user a personal.ini to edit on first start."""
for directory in (CONFIG_DIR, UPLOAD_DIR, TMP_DIR, INGEST_DIR):
for directory in (CONFIG_DIR, UPLOAD_DIR, TMP_DIR):
directory.mkdir(parents=True, exist_ok=True)
# The ingest folder belongs to CWA. A folder that Bindery makes hides a missing volume mount, and CWA
# never sees its files.
if INGEST_DIR and not INGEST_DIR.is_dir():
logger.error(
f"The ingest folder {INGEST_DIR} does not exist. Check INGEST_DIR and the volume mount. "
"Uploads fail until the folder exists."
)
if not PERSONAL_INI.exists():
shutil.copy(DEFAULT_PERSONAL_INI, PERSONAL_INI)
logger.info(f"Created the default FanFicFare configuration at {PERSONAL_INI}")
if not CWA_URL:
logger.warning("CWA_URL is not set. OPDS imports and HTTP fallbacks are unavailable.")
if not CWA_ENABLED:
if IGNORED_SETTINGS:
logger.warning(
f"CWA_ENABLED is not true. Bindery ignores {', '.join(IGNORED_SETTINGS)}. To use CWA, set "
"CWA_ENABLED=true."
)
return
if CWA_PASSWORD and not CWA_USERNAME:
logger.warning("CWA_PASSWORD is set, but CWA_USERNAME is not. Bindery sends no login to CWA.")
missing = _missing_cwa()
for warning in missing:
logger.warning(warning)
if missing:
logger.warning("If you do not use CWA, remove CWA_ENABLED. Bindery then shows no messages about CWA.")
def _missing_cwa() -> list[str]:
"""What Bindery cannot do with CWA, because a setting is not set."""
missing = []
if INGEST_DIR is None:
missing.append("INGEST_DIR is not set. Bindery sends no book to CWA.")
if not (CWA_URL or CALIBRE_DB_PATH):
effect = ", so each update adds a new book to CWA." if INGEST_DIR else "."
missing.append(
"CWA_URL and CALIBRE_DB_PATH are not set. Bindery cannot find the library book of a story" + effect
)
elif not CWA_URL:
missing.append("CWA_URL is not set. Bindery cannot search the CWA library, or import books from it.")
return missing
bindery/cwa.py
+32 -10
View File
@@ -7,7 +7,9 @@ by title, then reads the story URL inside each candidate epub.
import logging
import re
import secrets
import threading
import time
import uuid
from collections.abc import Callable
from dataclasses import dataclass, field
@@ -30,6 +32,8 @@ ACQUISITION = "http://opds-spec.org/acquisition"
# The most epubs that one search downloads to find a story.
MAX_DOWNLOADS = 5
MAX_PAGES = 100
# The seconds that Bindery trusts its last test of the logins of CWA.
LOGIN_TEST_SECONDS = 10 * 60
class CwaError(Exception):
@@ -52,11 +56,12 @@ class Book:
formats: dict[str, str] = field(default_factory=dict) # "epub" -> download href
def to_dict(self) -> dict[str, Any]:
"""The book in the form of get_book, with its publisher."""
return {
"id": self.id,
"uuid": self.uuid,
"title": self.title,
"authors": " & ".join(self.authors),
"authors": self.authors,
"publisher": self.publisher,
"formats": sorted(self.formats),
}
@@ -71,30 +76,35 @@ class CwaClient:
self.auth = (username, password or "") if username else None
self.timeout = timeout
self.session = requests.Session()
self._login_test: tuple[float, bool] | None = None # (time, CWA accepted a login that cannot exist)
def available(self) -> bool:
return bool(self.base_url)
def _get(self, href: str, auth: Any = None, **kwargs: Any) -> requests.Response:
"""Get a page as the account of Bindery, or with `auth` as a user who logs in."""
if not self.available():
raise CwaError("CWA address is not set. Set CWA_URL.")
# hrefs from the feed are absolute paths, and include any path prefix of CWA.
url = urljoin(self.base_url, href)
timeout = kwargs.pop("timeout", self.timeout)
try:
response = self.session.get(url, auth=auth or self.auth, timeout=self.timeout, **kwargs)
if auth:
# Without the session, so a cookie of the account of Bindery cannot stand in for the password.
response = requests.get(url, auth=auth, timeout=timeout, **kwargs)
else:
response = self.session.get(url, auth=self.auth, timeout=timeout, **kwargs)
except requests.RequestException as e:
raise CwaError(f"Cannot reach CWA at {self.base_url}: {e}") from e
if response.status_code == 401:
if not (auth or self.auth):
# A CWA with anonymous browsing needs no account. This one does.
raise CwaLoginError("CWA asks for a login. Set CWA_USERNAME and CWA_PASSWORD.")
raise CwaLoginError("CWA refused the login")
if response.status_code >= 400:
raise CwaError(f"CWA answered {response.status_code} for {url}")
return response
def check(self) -> str:
"""Log in to the OPDS feed. Returns the catalog title."""
root = ElementTree.fromstring(self._get("opds").content)
return (root.findtext(f"{ATOM}title") or "CWA").strip()
def login_ok(self, username: str, password: str) -> bool:
"""True if CWA accepts this user name and password."""
try:
@@ -103,6 +113,18 @@ class CwaClient:
return False
return True
def accepts_any_login(self) -> bool:
"""True if CWA accepts a login that cannot exist.
A CWA with anonymous browsing can answer every OPDS request, whatever the
password. Its OPDS feed then cannot check the password of a user.
"""
test = self._login_test
if test is None or time.monotonic() - test[0] > LOGIN_TEST_SECONDS:
accepted = self.login_ok(f"bindery-test-{secrets.token_hex(8)}", secrets.token_hex(16))
test = self._login_test = (time.monotonic(), accepted)
return test[1]
def _feed(self, href: str, params: dict[str, str] | None = None) -> tuple[list[Book], list[tuple[str, str]]]:
"""Books and navigation entries (title, href) of an OPDS feed, with all its pages."""
books: list[Book] = []
@@ -153,8 +175,8 @@ class CwaClient:
return []
return self._feed("opds/search", params={"query": query.strip()})[0]
def get_book(self, book_uuid: str) -> dict[str, Any] | None:
"""A book by its calibre uuid, or None if CWA has no such book."""
def get_book(self, book_uuid: str, timeout: int | None = None) -> dict[str, Any] | None:
"""A book by its calibre uuid, or None if CWA has no such book. `timeout` is for a page that waits."""
# The uuid can come from a request. Other characters could change the path.
if not re.fullmatch(r"[0-9A-Za-z-]+", book_uuid):
return None
@@ -165,7 +187,7 @@ class CwaClient:
return book
except calibre_db.DatabaseUnavailable as e:
logger.warning(f"Cannot read the calibre library database. Trying CWA: {e}")
response = self._get(f"ajax/book/{book_uuid}")
response = self._get(f"ajax/book/{book_uuid}", timeout=timeout or self.timeout)
if not response.text.strip():
return None
try:
bindery/epub.py
+27 -5
View File
@@ -1,9 +1,11 @@
"""Read the OPF metadata of an epub that came from CWA."""
import posixpath
import re
import zipfile
from pathlib import Path
from typing import Any
from urllib.parse import unquote
from xml.etree.ElementTree import Element
from defusedxml import ElementTree
@@ -17,14 +19,19 @@ class EpubError(Exception):
pass
def _opf(path: Path, z: zipfile.ZipFile) -> tuple[str, Element]:
"""The name of the OPF file in the epub, and its content."""
container = ElementTree.fromstring(z.read("META-INF/container.xml"))
rootfile = container.find(f".//{CONTAINER}rootfile")
if rootfile is None or not rootfile.get("full-path"):
raise EpubError(f"{path.name} has no OPF file")
return rootfile.get("full-path"), ElementTree.fromstring(z.read(rootfile.get("full-path")))
def _metadata(path: Path) -> Element:
try:
with zipfile.ZipFile(path) as z:
container = ElementTree.fromstring(z.read("META-INF/container.xml"))
rootfile = container.find(f".//{CONTAINER}rootfile")
if rootfile is None or not rootfile.get("full-path"):
raise EpubError(f"{path.name} has no OPF file")
opf = ElementTree.fromstring(z.read(rootfile.get("full-path")))
opf = _opf(path, z)[1]
except (zipfile.BadZipFile, KeyError, ElementTree.ParseError) as e:
raise EpubError(f"{path.name} is not a readable epub: {e}") from e
metadata = opf.find(f"{OPF}metadata")
@@ -33,6 +40,21 @@ def _metadata(path: Path) -> Element:
return metadata
def spine_documents(path: Path) -> list[str]:
"""The text of each document in the spine of an epub, in reading order."""
try:
with zipfile.ZipFile(path) as z:
name, opf = _opf(path, z)
hrefs = {item.get("id"): item.get("href") for item in opf.iter(f"{OPF}item")}
files = [
posixpath.join(posixpath.dirname(name), unquote(hrefs[ref.get("idref")]))
for ref in opf.iter(f"{OPF}itemref")
]
return [z.read(file).decode("utf-8", "replace") for file in files]
except (zipfile.BadZipFile, KeyError, ElementTree.ParseError) as e:
raise EpubError(f"{path.name} is not a readable epub: {e}") from e
def epub_urls(path: Path) -> list[str]:
"""Every URL in the identifiers and the source of an epub's OPF.
bindery/fff.py
+11 -6
View File
@@ -90,7 +90,7 @@ def escape_title(text: str) -> str:
return html.escape(text, quote=False)
def plain_title(title: str) -> str:
def unescape_title(title: str) -> str:
"""A stored chapter title as text, for a message."""
return html.unescape(title)
@@ -148,7 +148,7 @@ def check_personal_ini(text: str) -> list[str]:
try:
configuration.read_file(io.StringIO(text))
except configparser.ParsingError as e:
lines = "; ".join(f"line {n}: {line.strip()}" for n, line in e.errors)
lines = ", ".join(f"line {n}: {line.strip()}" for n, line in e.errors)
raise ValueError(f"FanFicFare cannot read this personal.ini. Check {lines}") from e
except configparser.Error as e:
raise ValueError(f"FanFicFare cannot read this personal.ini: {e}") from e
@@ -238,16 +238,21 @@ def word_count(chapters: list[dict[str, Any]]) -> int:
return sum(len(block.split()) for c in chapters for block in html_to_blocks(c["html"]))
def effective_overrides(overrides: dict[str, Any]) -> dict[str, Any]:
"""The kept values that replace a site value. An empty title, author list or language does not."""
return {k: v for k, v in overrides.items() if v or k not in ("title", "authors", "languages")}
def apply_overrides(story, overrides: dict[str, Any]) -> None:
"""Put the kept values of locked fields into FanFicFare's story metadata.
The epub writer takes the OPF metadata from these entries, and CWA takes
the book record from the OPF on the first import.
"""
for key, value in overrides.items():
if key == "title" and value:
for key, value in effective_overrides(overrides).items():
if key == "title":
story.setMetadata("title", value)
elif key == "authors" and value:
elif key == "authors":
for n, author in enumerate(value):
story.addToList("author", author, clear=(n == 0))
elif key == "comments":
@@ -263,7 +268,7 @@ def apply_overrides(story, overrides: dict[str, Any]) -> None:
story.setMetadata("datePublished", datetime.fromisoformat(value) if value else None)
except ValueError:
logger.warning(f"Ignoring a kept publication date that is not a date: {value!r}")
elif key == "languages" and value:
elif key == "languages":
story.setMetadata("langcode", value)
elif key == "tags":
tags = list(value or [])
bindery/filename_template.py
+321
View File
@@ -0,0 +1,321 @@
"""Calibre's templates for file names, for the name of a downloaded book.
Bindery follows Calibre 9.2.1 with its default tweaks and the English articles. prepare() takes the tweak for
the order of titles and series as an argument. Dates take a strftime format, with Calibre's "%b, %Y" as the
default. Bindery takes {field}, {field[0]}, {field:format|prefix|suffix} and {{ }}. It refuses template
functions and the program modes. It never runs template text as code.
{series_sort} is a Bindery field. It is to {series} what {title_sort} is to {title}. Calibre has no such field.
The tags sort close to Calibre's ICU rules, but their order can differ for characters outside ASCII.
"""
import re
import unicodedata
from datetime import datetime
from string import Formatter
from typing import Any
from bindery.ingest import sanitize_filename
# The name that Calibre's default template, {title} - {authors}, gives with its default tweaks.
DEFAULT_TEMPLATE = "{title_sort} - {authors}"
DEFAULT_DATE_FORMAT = "%b, %Y"
MAX_TEMPLATE_LENGTH = 1024
MAX_DATE_FORMAT_LENGTH = 100
# The most characters that a format can ask for, as a width or a precision.
MAX_WIDTH = 255
MAX_NAME_BYTES = 200
# The fields that Bindery has, with the help text for each. Any other field is empty, as it is in Calibre for
# a book without that value.
FIELDS = {
"title": "The title",
"title_sort": 'The title with a leading "The", "A" or "An" at the end, such as "Book, The"',
"authors": "The authors",
"author": "The authors",
"author_sort": "The author sort string. To use only the first letter of the name, use {author_sort[0]}",
"series": "The series",
"series_sort": 'The series with a leading "The", "A" or "An" at the end, such as "Saga, The". '
"Calibre does not have this field",
"series_index": "The series number. To get leading zeros, use {series_index:0>3s}",
"tags": "The tags",
"languages": "The language",
"publisher": "The story site",
"pubdate": "The published date",
"timestamp": "The date when Bindery added the story",
"last_modified": "The date when the chapters last changed",
"id": "The Bindery story ID",
}
class TemplateError(ValueError):
"""A template that Bindery cannot use, or that fails for a book."""
# Metadata, as Calibre prepares it for a file name
_ARTICLE = re.compile(r"^(A\s+|The\s+|An\s+)", re.IGNORECASE)
# Calibre's quote_pairs: each opening quote, with the quotes that close it.
_QUOTE_PAIRS = {
'"': '"',
"'": "'",
"\u201c": "\u201d\u201c",
"\u201d": "\u201d",
"\u201e": "\u201d\u201c",
"\u201a": "\u2019\u2018",
"\u2019": "\u2019\u2018",
"\u2018": "\u2019\u2018",
"\u2039": "\u203a",
"\u203a": "\u2039",
"\u300a": "\u300b",
"\u3008": "\u3009",
"\u00bb": "\u00ab\u00bb",
"\u00ab": "\u00ab\u00bb",
"\u300c": "\u300d",
"\u300e": "\u300f",
}
# The ASCII punctuation in the order of Calibre's sort. Digits come next, then letters.
_PUNCTUATION = " _-,;:!?.'\"()[]{}@*/\\&#%`^+<=>|~$"
# Calibre's default tweaks for author names.
_NAME_PREFIXES = {"mr", "mrs", "ms", "dr", "prof"}
_NAME_SUFFIXES = {"jr", "sr", "inc", "ph.d", "phd", "md", "m.d", "i", "ii", "iii", "iv", "junior", "senior"}
_COPY_WORDS = {
"agency",
"corporation",
"company",
"co.",
"council",
"committee",
"inc.",
"institute",
"national",
"society",
"club",
"team",
"software",
"games",
"entertainment",
"media",
"studios",
}
_BRACKETS = {"(": ")", "[": "]", "{": "}"}
def _unquote(title: str) -> str:
closers = _QUOTE_PAIRS.get(title[:1])
if closers:
title = title[1:]
if title and title[-1] in closers:
title = title[:-1]
return title
def title_sort(title: str) -> str:
"""Calibre's title_sort(): "The Book" gives "Book, The"."""
title = _unquote(title.strip())
match = _ARTICLE.search(title)
if match:
title = _unquote(f"{title[match.end() :]}, {match.group(1)}")
return title.strip()
def _without_brackets(text: str) -> str:
depth = dict.fromkeys(_BRACKETS, 0)
closers = {close: opener for opener, close in _BRACKETS.items()}
kept = []
for char in text:
if char in depth:
depth[char] += 1
elif char in closers:
depth[closers[char]] = max(0, depth[closers[char]] - 1)
elif not any(depth.values()):
kept.append(char)
return "".join(kept)
def author_sort(author: str) -> str:
"""Calibre's author_to_author_sort() with its default tweaks: "Ann Writer" gives "Writer, Ann"."""
plain = _without_brackets(author).strip()
tokens = plain.split()
lower = [t.lower().removesuffix(".") for t in tokens]
if "," in plain or len(tokens) < 2 or _COPY_WORDS.intersection(t.lower() for t in tokens):
return author
first = next((n for n, t in enumerate(lower) if t not in _NAME_PREFIXES), None)
if first is None:
return author
last = next((n for n in range(len(tokens) - 1, first - 1, -1) if lower[n] not in _NAME_SUFFIXES), None)
if last is None:
return author
names = [tokens[last], *tokens[first:last]]
if len(names) > 1:
names[0] += ","
return " ".join(names + tokens[last + 1 :])
def series_index(index: Any) -> str:
"""Calibre's fmt_sidx(): 2.0 gives "2", and 2.345 gives "2.35"."""
try:
index = float(index)
except (TypeError, ValueError):
index = 1.0
if index.is_integer():
return str(int(index))
return f"{index:.2f}".rstrip("0")
def check_date_format(date_format: str) -> None:
"""Raise TemplateError if Bindery cannot use the date format."""
if len(date_format) > MAX_DATE_FORMAT_LENGTH:
raise TemplateError(f"A date format can have {MAX_DATE_FORMAT_LENGTH} characters at most")
try:
datetime(2020, 5, 6).strftime(date_format)
except ValueError as e:
raise TemplateError(f"The date format is not valid: {e}") from e
def _date(value: Any, date_format: str) -> str:
try:
return datetime.fromisoformat(value).strftime(date_format)
except (TypeError, ValueError):
return ""
def _tag_key(tag: str) -> tuple[list[tuple[int, str]], str]:
"""Close to the key of Calibre's sort: no case, punctuation before digits before letters, accents last."""
text = unicodedata.normalize("NFKD", tag.casefold())
ranks = len(_PUNCTUATION)
letters = [
(_PUNCTUATION.index(c), "") if c in _PUNCTUATION else (ranks + c.isalpha(), c)
for c in text
if not unicodedata.combining(c)
]
return letters, text
def prepare(
meta: dict[str, Any], book_id: int, order: str = "library_order", date_format: str = DEFAULT_DATE_FORMAT
) -> dict[str, Any]:
"""Calibre's get_component_metadata(): the values of a book in the form that a file name takes.
`order` is Calibre's tweak save_template_title_series_sorting. With "strictly_alphabetic", the title and the
series keep a leading article. `date_format` is Calibre's timefmt.
"""
library = order == "library_order"
values: dict[str, Any] = {"id": str(book_id), "publisher": meta.get("publisher") or ""}
title, authors, series = meta.get("title"), [a for a in meta.get("authors") or [] if a], meta.get("series")
if title:
values["title_sort"] = title_sort(title)
values["title"] = values["title_sort"] if library else title
if authors:
values["authors"] = values["author"] = " & ".join(a.replace("&", "&&") for a in authors)
values["author_sort"] = " & ".join(author_sort(a) for a in authors)
if series:
values["series_sort"] = title_sort(series)
values["series"] = values["series_sort"] if library else series.strip()
if meta.get("series_index") is not None:
values["series_index"] = series_index(meta["series_index"])
values["tags"] = ", ".join(sorted(meta.get("tags") or [], key=_tag_key)).removeprefix("/")
values["languages"] = meta.get("languages") or ""
for name in ("pubdate", "timestamp", "last_modified"):
values[name] = _date(meta.get(name), date_format)
return values
# Templates
_FIELD = re.compile(r"([^.\[\]]*)((?:\[\d+\])*)")
_AFFIXES = re.compile(r"^(.*)\|([^|]*)\|(.*)$", re.DOTALL)
_INT_CODES, _FLOAT_CODES = "bcdoxXn", "eEfFgG%"
def _format(value: str, fmt: str) -> str:
"""Calibre's _do_format(): a number code takes the text as a number."""
if not fmt or not value:
return value
code = fmt[-1]
try:
number = int(value) if code in _INT_CODES else float(value) if code in _FLOAT_CODES else value
return format(number, fmt)
except (ValueError, OverflowError) as e:
raise TemplateError(f"The format {fmt!r} cannot take the value {value!r}: {e}") from e
def _parse(template: str) -> list[tuple[str, str | None, list[int], str, str, str]]:
"""The parts of a template. Each part is the text before a field, then the name of the field, its indexes,
its format, its prefix and its suffix. The name is None for text with no field after it."""
if len(template) > MAX_TEMPLATE_LENGTH:
raise TemplateError(f"A template can have {MAX_TEMPLATE_LENGTH} characters at most")
if template.startswith(("program:", "python:")):
raise TemplateError("Bindery does not support the program modes of Calibre")
try:
fields = list(Formatter().parse(template))
except ValueError as e:
raise TemplateError(f"The braces of the template do not match: {e}") from e
parts = []
for text, name, spec, conversion in fields:
if name is None:
parts.append((text, None, [], "", "", ""))
continue
field = _FIELD.fullmatch(name)
if not field or conversion or "{" in spec:
raise TemplateError(f"Bindery does not support the field {{{name}}} in this form")
if not field.group(1) or field.group(1).isdigit():
raise TemplateError("A field needs a name, such as {title}")
affixes = _AFFIXES.match(spec)
fmt, prefix, suffix = affixes.groups() if affixes else (spec, "", "")
if fmt.startswith("'") or (":'" in fmt and fmt.endswith("'")) or ("(" in fmt and fmt.endswith(")")):
raise TemplateError("Bindery does not support template functions, such as {title:uppercase()}")
if any(int(n) > MAX_WIDTH for n in re.findall(r"\d+", fmt)):
raise TemplateError(f"A format can ask for {MAX_WIDTH} characters at most")
_format("1", fmt)
indexes = [int(n) for n in re.findall(r"\d+", field.group(2))]
parts.append((text, field.group(1).lower(), indexes, fmt, prefix, suffix))
return parts
def check(template: str) -> None:
"""Raise TemplateError if Bindery cannot use the template."""
_parse(template)
def expand(template: str, values: dict[str, Any]) -> str:
"""The text of a template for the prepared values of a book, as Calibre gives it before it makes a path."""
out = []
for text, name, indexes, fmt, prefix, suffix in _parse(template):
out.append(text)
if name is None:
continue
value = values.get(name, "")
if isinstance(value, list):
value = ",".join(value)
value = value.replace("/", "_").replace("\\", "_")
try:
for index in indexes:
value = value[index]
except IndexError as e:
raise TemplateError(f"The value of {{{name}}} is too short for [{index}]") from e
if value:
value = _format(value, fmt)
if value:
out += prefix, value, suffix
return re.sub(r"\s+", " ", "".join(out)).strip(" ")
_RESERVED = re.compile(r"(?:CON|PRN|AUX|NUL|COM\d|LPT\d)(?:\.|$)", re.IGNORECASE)
def file_name(template: str, values: dict[str, Any]) -> str:
"""The name of one file from a template, without the extension. Empty if the template gives no name.
A download has one name, so Bindery uses the last part of a template with folders. That is the name that
Calibre gives the file. Windows cannot use some names, such as CON. Those get a leading underscore.
"""
parts = [part.strip() for part in expand(template, values).split("/")]
name = next((part for part in reversed(parts) if part), "")
if not name:
return ""
name = sanitize_filename(name)
if _RESERVED.match(name):
name = f"_{name}"
return name.encode()[:MAX_NAME_BYTES].decode(errors="ignore").rstrip(" .")
bindery/ingest.py
+11 -2
View File
@@ -26,10 +26,19 @@ def sanitize_filename(name: str) -> str:
return name[:MAX_NAME_LENGTH].strip(" .") or "story"
class IngestError(Exception):
"""The ingest folder cannot take a book."""
class IngestFolder:
def __init__(self, path: Path) -> None:
self.path = Path(path)
def _check(self) -> None:
# Bindery does not make the folder. A missing folder is most often a missing volume mount.
if not self.path.is_dir():
raise IngestError(f"The ingest folder {self.path} does not exist. Check INGEST_DIR and the volume mount.")
def _place(self, epub: Path, final_path: Path) -> Path:
"""Copy an epub in under a temporary .part name, then rename it.
@@ -43,7 +52,7 @@ class IngestFolder:
def ingest(self, epub: Path, name: str) -> Path:
"""Send a new book to CWA."""
self.path.mkdir(parents=True, exist_ok=True)
self._check()
# A short unique suffix, so a file that CWA has not imported yet is never replaced.
stem = sanitize_filename(f"{name[:80]} {uuid.uuid4().hex[:6]}")
final_path = self._place(epub, self.path / f"{stem}.epub")
@@ -52,7 +61,7 @@ class IngestFolder:
def send_update(self, epub: Path, book_id: int, name: str) -> Path:
"""Send a new epub for an existing book. CWA replaces the book's epub."""
self.path.mkdir(parents=True, exist_ok=True)
self._check()
# A unique name, so two updates of one book never share a sidecar file.
stem = sanitize_filename(f"{name[:60]} {book_id}-{uuid.uuid4().hex[:8]}")
final_path = self.path / f"{stem}.epub"
bindery/jobs.py
+15 -2
View File
@@ -4,7 +4,7 @@ import json
import sqlite3
import uuid
from collections import deque
from dataclasses import asdict, dataclass, field
from dataclasses import asdict, dataclass, field, fields
from datetime import datetime, timedelta
from enum import StrEnum
from pathlib import Path
@@ -87,6 +87,8 @@ class Job:
def to_dict(self) -> dict[str, Any]:
d = asdict(self)
# The path of an uploaded file is for the job only.
d["options"] = {k: v for k, v in self.options.items() if k != "path"}
d["mode"] = self.mode.value
d["mode_label"] = MODE_LABELS[self.mode.value]
d["status"] = self.status.value
@@ -102,7 +104,9 @@ class Job:
@classmethod
def from_record(cls, record: str) -> "Job":
d = json.loads(record)
# A record of a newer Bindery can hold fields that this one does not know.
known = {f.name for f in fields(cls)}
d = {k: v for k, v in json.loads(record).items() if k in known}
d["mode"] = DownloadMode(d["mode"])
d["status"] = QueueStatus(d["status"])
d["created"] = datetime.fromisoformat(d["created"])
@@ -221,6 +225,15 @@ class JobQueue:
self._save(job)
return True
def get(self, job_id: str) -> Job | None:
with self._lock:
return self._jobs.get(job_id)
def active(self) -> list[Job]:
"""The jobs that wait or run."""
with self._lock:
return [j for j in self._jobs.values() if j.status in (QueueStatus.QUEUED, QueueStatus.RUNNING)]
def has_active(self, story_id: int) -> bool:
with self._lock:
return any(
bindery/opds.py
+301
View File
@@ -0,0 +1,301 @@
"""An OPDS 1.2 catalog of the books stored in Bindery."""
import hashlib
import io
import os
import re
from collections.abc import Callable
from datetime import UTC, datetime
from typing import Any
from urllib.parse import quote
from xml.etree import ElementTree as ET
from flask import Blueprint, Response, abort, request, send_file, url_for
from bindery import diffing, filename_template, orchestrator
from bindery.store import sort_title
bp = Blueprint("opds", __name__)
ATOM = "http://www.w3.org/2005/Atom"
DC = "http://purl.org/dc/terms/"
OPENSEARCH = "http://a9.com/-/spec/opensearch/1.1/"
NAVIGATION = "application/atom+xml;profile=opds-catalog;kind=navigation"
ACQUISITION = "application/atom+xml;profile=opds-catalog;kind=acquisition"
EPUB = "application/epub+zip"
PAGE_SIZE = 50
UNKNOWN_AUTHOR = "Unknown" # as in Calibre
# Each list: its title, its kind, and the text of its entry in the start feed.
LISTS = {
"title": ("By Title", ACQUISITION, "All books in title order"),
"author": ("By Author", NAVIGATION, "The authors, with the books of each"),
"changed": ("By Last Changed", ACQUISITION, "All books, with the most recently changed books first"),
}
# XML 1.0 excludes control characters, even in escaped text.
INVALID_XML = re.compile("[^\x09\x0a\x0d\x20-퟿-�\U00010000-\U0010ffff]")
def _element(parent: ET.Element, tag: str, text: str = "", **attributes: str) -> ET.Element:
element = ET.SubElement(parent, tag, attributes)
element.text = INVALID_XML.sub("", text)
return element
def _updated(book: dict[str, Any]) -> str:
# Store timestamps without an offset use the server's local time.
date = datetime.fromisoformat(book["last_changed"] or book["created_at"])
return date.astimezone(UTC).isoformat(timespec="seconds").replace("+00:00", "Z")
def _link(parent: ET.Element, rel: str, href: str, media_type: str) -> None:
_element(parent, "link", rel=rel, href=href, type=media_type)
def _feed(title: str, identifier: str, updated: str, href: str, media_type: str) -> ET.Element:
feed = ET.Element("feed", {"xmlns": ATOM, "xmlns:dc": DC})
_element(feed, "id", identifier)
_element(feed, "title", title)
_element(feed, "updated", updated)
_element(_element(feed, "author"), "name", "Bindery")
_link(feed, "self", href, media_type)
_link(feed, "start", url_for("opds.catalog"), NAVIGATION)
_link(feed, "search", url_for("opds.search_description"), "application/opensearchdescription+xml")
# Many readers take the search template from the feed and never read the description.
_link(feed, "search", f"{url_for('opds.search')}?q={{searchTerms}}", ACQUISITION)
return feed
def _response(root: ET.Element, media_type: str) -> Response:
response = Response(ET.tostring(root, encoding="utf-8", xml_declaration=True), content_type=media_type)
response.headers["Cache-Control"] = "private, no-store"
return response
def _languages(value: Any) -> list[str]:
return [value] if isinstance(value, str) else list(value or [])
def _books() -> list[dict[str, Any]]:
"""The books of the catalog, in title order."""
books = []
page_counts = orchestrator.store.story_settings("page_count")
for story in orchestrator.store.catalog():
meta = orchestrator.book_metadata(story)
title = (meta["title"] or "").strip() or story["url"]
authors = [a.strip() for a in meta["authors"] if a and a.strip()] or [UNKNOWN_AUTHOR]
cover = story["cover_key"] if story["cover_mime"] in orchestrator.COVER_TYPES else None
books.append(
{
**story,
"meta": meta,
"title": title,
"authors": authors,
"updated": _updated(story),
"page_count": page_counts.get(story["id"]),
"cover": hashlib.sha256(cover.encode()).hexdigest()[:16] if cover else None,
}
)
# The ID breaks ties, so a book keeps its page.
return sorted(books, key=lambda book: (sort_title(book["title"]), book["id"]))
def _latest(books: list[dict[str, Any]]) -> str:
return max((book["updated"] for book in books), default="1970-01-01T00:00:00Z")
def _content(book: dict[str, Any]) -> str:
meta = book["meta"]
lines = []
if meta.get("series"):
index = filename_template.series_index(meta.get("series_index"))
lines.append(f"Series: {meta['series']}" + (f", book {index}" if index else ""))
lines.append(f"Chapters: {book['chapter_count']}")
if book["page_count"] is not None:
lines.append(f"Pages: {book['page_count']}")
description = "\n\n".join(diffing.html_to_blocks(meta.get("comments") or ""))
return "\n".join(lines) + (f"\n\n{description}" if description else "")
def _book_entry(feed: ET.Element, book: dict[str, Any]) -> None:
meta = book["meta"]
entry = _element(feed, "entry")
_element(entry, "id", book["url"])
_element(entry, "title", book["title"])
_element(entry, "updated", book["updated"])
for name in book["authors"]:
author = _element(entry, "author")
_element(author, "name", name)
_element(author, "uri", url_for("opds.books", order="author", name=name))
_element(entry, "dc:publisher", book["site"])
if meta.get("pubdate"):
_element(entry, "dc:issued", meta["pubdate"][:10])
for language in _languages(meta.get("languages")):
_element(entry, "dc:language", language)
for tag in meta.get("tags") or []:
_element(entry, "category", term=tag, label=tag)
_element(entry, "content", _content(book), type="text")
_link(entry, "http://opds-spec.org/acquisition", url_for("opds.download", story_id=book["id"]), EPUB)
if book["cover"]:
href = url_for("opds.cover", story_id=book["id"], v=book["cover"])
for rel in ("http://opds-spec.org/image", "http://opds-spec.org/image/thumbnail"):
_link(entry, rel, href, book["cover_mime"])
def _author_entry(feed: ET.Element, author: dict[str, Any]) -> None:
entry = _element(feed, "entry")
_element(entry, "id", f"urn:bindery:author:{quote(author['name'])}")
_element(entry, "title", author["name"])
_element(entry, "updated", _latest(author["books"]))
count = len(author["books"])
_element(entry, "content", f"{count} book" + ("" if count == 1 else "s"), type="text")
_link(entry, "subsection", url_for("opds.books", order="author", name=author["name"]), ACQUISITION)
def _authors(books: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""The authors of the books, in Calibre's author sort order. A name in other capitals is the same author."""
authors: dict[str, dict[str, Any]] = {}
for book in books:
for name in book["authors"]:
authors.setdefault(name.casefold(), {"name": name, "books": []})["books"].append(book)
return sorted(authors.values(), key=lambda a: (filename_template.author_sort(a["name"]).casefold(), a["name"]))
def _page() -> int:
text = request.args.get("page", "1")
if not re.fullmatch(r"[0-9]{1,9}", text) or int(text) < 1:
raise ValueError("The page must be a positive whole number")
return int(text)
def _paged(
title: str,
identifier: str,
items: list[dict[str, Any]],
kind: str,
entry: Callable[[ET.Element, dict[str, Any]], None],
up: str,
endpoint: str,
**args: str,
) -> Response:
"""A feed that shows one page of the items. `entry` adds the entry of one item."""
page = _page()
pages = max(1, (len(items) + PAGE_SIZE - 1) // PAGE_SIZE)
if page > pages:
abort(404)
books = [book for item in items for book in item.get("books", [item])]
feed = _feed(title, identifier, _latest(books), url_for(endpoint, **args, page=page), kind)
_link(feed, "up", up, NAVIGATION)
if page > 1:
_link(feed, "previous", url_for(endpoint, **args, page=page - 1), kind)
if page < pages:
_link(feed, "next", url_for(endpoint, **args, page=page + 1), kind)
for item in items[(page - 1) * PAGE_SIZE : page * PAGE_SIZE]:
entry(feed, item)
return _response(feed, kind)
@bp.get("/opds/")
@bp.get("/opds")
def catalog() -> Response:
updated = _latest(_books())
feed = _feed("Bindery", "urn:bindery:catalog", updated, url_for("opds.catalog"), NAVIGATION)
for order, (title, kind, text) in LISTS.items():
entry = _element(feed, "entry")
_element(entry, "id", f"urn:bindery:catalog:{order}")
_element(entry, "title", title)
_element(entry, "updated", updated)
_element(entry, "content", text, type="text")
_link(entry, "subsection", url_for("opds.books", order=order), kind)
return _response(feed, NAVIGATION)
@bp.get("/opds/<order>")
def books(order: str) -> Response:
if order not in LISTS:
abort(404)
books = _books()
up = url_for("opds.catalog")
if order == "author":
authors = _authors(books)
name = request.args.get("name")
if name is None:
title, kind, _ = LISTS[order]
return _paged(
title, "urn:bindery:catalog:author", authors, kind, _author_entry, up, "opds.books", order=order
)
author = next((a for a in authors if a["name"].casefold() == name.casefold()), None)
if author is None:
abort(404)
identifier = f"urn:bindery:author:{quote(author['name'])}"
up = url_for("opds.books", order="author")
return _paged(
author["name"],
identifier,
author["books"],
ACQUISITION,
_book_entry,
up,
"opds.books",
order=order,
name=author["name"],
)
if order == "changed":
books.sort(key=lambda book: book["updated"], reverse=True)
identifier = f"urn:bindery:catalog:{order}"
return _paged(LISTS[order][0], identifier, books, ACQUISITION, _book_entry, up, "opds.books", order=order)
@bp.get("/opds/search")
def search() -> Response:
"""The books whose title, authors, series or tags hold each word of the search, in title order."""
query = request.args.get("q", "")
words = query.casefold().split()
found = []
for book in _books() if words else []:
meta = book["meta"]
text = " ".join([book["title"], *book["authors"], meta.get("series") or "", *(meta.get("tags") or [])])
if all(word in text.casefold() for word in words):
found.append(book)
identifier = f"urn:bindery:search:{quote(query)}"
up = url_for("opds.catalog")
return _paged(f"Search: {query}", identifier, found, ACQUISITION, _book_entry, up, "opds.search", q=query)
@bp.get("/opds/search.xml")
def search_description() -> Response:
root = ET.Element("OpenSearchDescription", xmlns=OPENSEARCH)
_element(root, "ShortName", "Bindery")
_element(root, "Description", "Search the books in Bindery by title, author, series or tag")
_element(root, "InputEncoding", "UTF-8")
_element(root, "OutputEncoding", "UTF-8")
_element(root, "Url", type=ACQUISITION, template=f"{url_for('opds.search')}?q={{searchTerms}}")
return _response(root, "application/opensearchdescription+xml")
def send_epub(story_id: int) -> Response:
"""The ePub of a story as a download, built from the stored chapters."""
file, name = orchestrator.build_for_download(story_id)
response = send_file(file, download_name=name, as_attachment=True, mimetype=EPUB)
response.content_length = os.fstat(file.fileno()).st_size
return response
def send_cover(story_id: int) -> Response:
cover = orchestrator.story_cover(story_id)
if not cover:
raise KeyError("This story has no cover")
# The URL holds the cover version, so a client can keep the image for a year.
response = send_file(io.BytesIO(cover["data"]), mimetype=cover["mime"], max_age=365 * 24 * 60 * 60)
# A cover needs a login, so a shared cache must not keep it.
response.cache_control.public = False
response.cache_control.private = True
return response
@bp.get("/opds/books/<int:story_id>.epub")
def download(story_id: int) -> Response:
return send_epub(story_id)
@bp.get("/opds/books/<int:story_id>/cover")
def cover(story_id: int) -> Response:
return send_cover(story_id)
bindery/orchestrator.py
+636 -281
View File
File diff suppressed because it is too large Load Diff
bindery/pages.py
+78
View File
@@ -0,0 +1,78 @@
"""An estimate of the pages of a built epub, after the "Paragraphs (APNX Accurate)" method of the Count Pages
plugin for calibre.
The method counts lines. A paragraph start is one line, and so is each 70 characters of text. A page is 31
lines. A page is also at most 2400 characters, which sets the count of a book with almost no paragraphs.
The plugin counts either the <p> or the <div> elements of a book as its paragraphs. Bindery counts what a
reader sees as a paragraph: text that follows a <br> or the start or end of a block element. So the same text
gives the same count as <p> elements, as <div> elements, and as lines with <br> between them. A paragraph with
no text, or with only a no-break space, is not counted.
"""
import html as htmllib
import re
from pathlib import Path
from typing import NamedTuple
from bindery.epub import spine_documents
LINE_CHARS = 70
LINES_PER_PAGE = 31
CHARS_PER_PAGE = 2400
# Elements that stay in the line of the text around them. Each other element, and <br>, ends a paragraph.
# The cells of a table row share a line.
INLINE = frozenset(
"a abbr acronym b bdi bdo big cite code data del dfn em font i img ins kbd mark q rp rt ruby s samp small "
"span strike strong sub sup td th time tt u var wbr".split()
)
# Elements whose text is not text of the book.
HIDDEN = frozenset(("head", "script", "style"))
_TOKEN = re.compile(r"<(/?)([a-zA-Z][a-zA-Z0-9]*)[^>]*>|<[^>]*>|([^<]+)")
class Measure(NamedTuple):
paragraphs: int = 0
chars: int = 0
def measure(html: str) -> Measure:
"""The paragraphs and the characters of text of one document."""
paragraphs = chars = 0
hidden = False
new_paragraph = True
for match in _TOKEN.finditer(html):
closing, name, text = match.groups()
if name:
name = name.lower()
if name in HIDDEN:
hidden = not closing and not match.group().endswith("/>")
elif name not in INLINE:
new_paragraph = True
elif text and not hidden:
if "&" in text:
text = htmllib.unescape(text)
# A no-break space is white space here, so a paragraph that only spaces the text is no paragraph.
if text.strip():
if new_paragraph:
paragraphs += 1
new_paragraph = False
text = text.lstrip()
chars += len(text)
elif not new_paragraph:
chars += 1 # the space between two inline elements
return Measure(paragraphs, chars)
def page_count(total: Measure) -> int:
"""The pages of a book, from the sums of its documents. A book with no text has no pages."""
if not total.chars:
return 0
lines = total.paragraphs + total.chars // LINE_CHARS
return max(lines // LINES_PER_PAGE, total.chars // CHARS_PER_PAGE + 1)
def epub_pages(path: Path) -> int:
"""The estimated pages of an epub, from each document in its spine."""
measures = [measure(text) for text in spine_documents(path)]
return page_count(Measure(sum(m.paragraphs for m in measures), sum(m.chars for m in measures)))
bindery/static/js/main.js
+16 -4
View File
@@ -11,7 +11,7 @@ const ROUTES = [
[/^\/$/, storiesPage, []],
[/^\/story\/(\d+)$/, storyPage, ['storyId']],
[/^\/story\/(\d+)\/chapter\/(\d+)$/, chapterPage, ['storyId', 'chapterId']],
[/^\/settings$/, settingsPage, []],
[/^\/settings$/, settingsPage, [], true],
];
const app = document.getElementById('bindery-app');
@@ -23,6 +23,9 @@ let nav = null;
let storyTab = null;
let storyTabShown = false;
// The server refuses the changes of a user who is not an admin. The interface only hides what that user cannot use.
const isAdmin = () => auth.admin !== false;
const currentPath = () => window.location.hash.replace(/^#/, '') || '/';
// The story of #/story/3 and of its chapter pages, such as #/story/3/chapter/5.
const currentStoryId = () => currentPath().match(/^\/story\/(\d+)(?:\/|$)/)?.[1];
@@ -44,6 +47,14 @@ function notFound(main) {
fill(main, h('section', { class: 'card' }, h('h1', {}, 'Page not found'), h('p', {}, h('a', { href: '#/' }, 'Go to the story list'))));
}
function adminOnly(main) {
document.title = 'Settings · Bindery';
fill(
main,
h('section', { class: 'card' }, h('h1', {}, 'Settings'), h('p', {}, 'Only an admin can open the settings. BINDERY_ADMIN_USERS lists the admins.')),
);
}
function route() {
const main = document.getElementById('main');
if (!main) return;
@@ -54,9 +65,10 @@ function route() {
if (storyTab?.getAttribute('href') !== `#/story/${currentStoryId()}`) hideStoryTab();
const found = ROUTES.find(([pattern]) => pattern.test(path));
if (found) {
const [pattern, page, names] = found;
const [pattern, page, names, admin] = found;
const match = path.match(pattern);
cleanup = page(main, Object.fromEntries(names.map((name, i) => [name, match[i + 1]])), config) || null;
if (admin && !isAdmin()) adminOnly(main);
else cleanup = page(main, Object.fromEntries(names.map((name, i) => [name, match[i + 1]])), config) || null;
} else {
notFound(main);
}
@@ -81,7 +93,7 @@ function shell() {
(auth.mode === 'cwa' || auth.mode === 'local' || auth.logout_url) &&
h('button', { type: 'button', onclick: () => logout().catch(showError) }, 'Log out'),
];
nav = h('nav', { 'aria-label': 'Main navigation' }, h('a', { href: '#/' }, 'Stories'), h('a', { href: '#/settings' }, 'Settings'));
nav = h('nav', { 'aria-label': 'Main navigation' }, h('a', { href: '#/' }, 'Stories'), isAdmin() && h('a', { href: '#/settings' }, 'Settings'));
storyTab = h('a', { class: 'story-tab' });
storyTabShown = false;
fill(
bindery/static/js/rules.js
+3 -3
View File
@@ -227,9 +227,9 @@ export function globalRulesCard() {
.catch(showError);
return h(
'section',
{ class: 'card' },
h('h2', {}, 'Global replacement rules'),
'details',
{ class: 'card', open: true },
h('summary', {}, h('h2', {}, 'Global replacement rules')),
h(
'p',
{ class: 'hint' },
bindery/static/js/settings.js
+278 -51
View File
@@ -1,16 +1,113 @@
// Settings: the recheck of recent chapters, the uploads to CWA, the global rules, and FanFicFare's personal.ini.
// Settings: the recheck of recent chapters, the names of downloaded books, the uploads to CWA, the global rules,
// and FanFicFare's personal.ini.
import { api, button, checkbox, externalLink, fill, h, showError, toast } from './lib.js';
import { globalRulesCard } from './rules.js';
const INI_DOCS = 'https://github.com/JimmXinu/FanFicFare/wiki/INI-File';
const DEFAULTS_INI = 'https://github.com/JimmXinu/FanFicFare/blob/main/fanficfare/defaults.ini';
const TEMPLATE_DOCS = 'https://manual.calibre-ebook.com/template_lang.html';
const DATE_CODES = 'https://docs.python.org/3/library/datetime.html#format-codes';
const code = (text) => h('code', {}, text);
// How a reader logs in to the catalog, for each login mode.
const OPDS_LOGIN = {
none: 'Bindery asks for no login, so the catalog needs none.',
local: 'The reader sends your Bindery user name and password with HTTP Basic authentication.',
cwa: 'The reader sends your CWA user name and password with HTTP Basic authentication.',
proxy:
'The proxy skips its login for the catalog. Each user sets an OPDS password below. The reader sends that ' +
'password and the user name from the proxy with HTTP Basic authentication.',
};
// A card of settings that folds shut. It starts open.
const card = (title, ...children) =>
h('details', { class: 'card', open: true }, h('summary', {}, h('h2', {}, title)), ...children);
export function settingsPage(main, _params, config) {
document.title = 'Settings · Bindery';
const recent = h('input', { id: 'recent-chapters', type: 'number', min: 0, step: 1 });
const holdNewLinks = checkbox(false, () => {});
const libraryNote = h('p', { class: 'notice', id: 'library-note', hidden: true },
'CWA_URL and CALIBRE_DB_PATH are not set. Set one or both and restart Bindery to enable library linking.');
const linkNote = h('p', { class: 'notice', id: 'link-note', hidden: true },
'Updates without a saved CWA book ID create duplicate books if CWA imports them.');
const showLinkState = (enabled) => {
holdNewLinks.disabled = !enabled;
linkNote.hidden = enabled;
linkBooks.setAttribute('aria-describedby',
[linkBooks.disabled && 'library-note', !enabled && 'link-note'].filter(Boolean).join(' '));
};
const linkBooks = checkbox(true, showLinkState);
const pauseUploads = checkbox(false, () => {});
const ingestNote = h('p', { class: 'notice', id: 'ingest-note', hidden: true },
'INGEST_DIR is not set. Bindery sends no book to the ingest folder until you set it and restart Bindery.');
const showUploadSettings = (settings) => {
linkBooks.disabled = settings.linking_available === false;
linkBooks.checked = !linkBooks.disabled && (settings.link_books ?? true);
libraryNote.hidden = !linkBooks.disabled;
holdNewLinks.checked = settings.hold_new_links;
showLinkState(linkBooks.checked);
pauseUploads.disabled = settings.ingest_available === false;
pauseUploads.checked = pauseUploads.disabled || settings.pause_uploads;
pauseUploads.setAttribute('aria-describedby', pauseUploads.disabled ? 'ingest-note' : '');
ingestNote.hidden = !pauseUploads.disabled;
};
const opdsEnabled = checkbox(false, (checked) => showPassword(passwordSet, checked), {
id: 'opds-enabled',
'aria-describedby': 'opds-note',
});
let passwordSet = false;
const opdsUser = h('input', {
id: 'opds-user',
type: 'text',
readonly: true,
spellcheck: 'false',
autocomplete: 'off',
'aria-describedby': 'opds-password-status',
});
const opdsPassword = h('input', {
id: 'opds-password',
type: 'password',
minlength: 8,
maxlength: 256,
autocomplete: 'new-password',
});
const opdsPasswordStatus = h('p', { class: 'hint', id: 'opds-password-status' });
const setPassword = button('Set password', async () => {
await api('settings/opds-password', 'PUT', { password: opdsPassword.value });
opdsPassword.value = '';
showPassword(true, opdsEnabled.checked);
toast('Saved');
});
// The password applies only when the catalog is on, so the field is grey while the box is not checked.
const showPassword = (isSet, enabled) => {
passwordSet = isSet;
opdsPassword.disabled = !enabled;
setPassword.disabled = !enabled;
setPassword.textContent = isSet ? 'Reset password' : 'Set password';
fill(
opdsPasswordStatus,
isSet
? 'You have an OPDS password. Bindery keeps only a hash of it. ' +
'Enter the user name and the password in your reader.'
: 'You have no OPDS password yet. Set one, then enter the user name and the password in your reader.',
);
};
const template = h('input', {
id: 'download-template',
class: 'template',
type: 'text',
maxlength: 1024,
spellcheck: 'false',
autocomplete: 'off',
});
const dateFormat = h('input', {
id: 'download-date-format',
class: 'template',
type: 'text',
maxlength: 100,
spellcheck: 'false',
autocomplete: 'off',
});
const hoursField = (id) => h('input', { id, type: 'number', min: 0, max: config.job_list_max_hours, step: 1 });
const countField = (id) => h('input', { id, type: 'number', min: 0, max: config.job_list_max_count, step: 1 });
const storyJobsHours = hoursField('story-jobs-hours');
@@ -24,6 +121,13 @@ export function settingsPage(main, _params, config) {
'aria-label': 'personal.ini',
});
const warnings = h('div');
// A Save button for some of the settings. `values` gives the settings to send, and `saved` gets the answer.
const saveButton = (values, saved = () => toast('Saved')) =>
h(
'div',
{ class: 'controls' },
button('Save', async () => saved(await api('settings', 'PUT', values())), { class: 'primary' }),
);
const showWarnings = (list) =>
fill(
@@ -40,9 +144,13 @@ export function settingsPage(main, _params, config) {
api('settings')
.then((settings) => {
recent.value = settings.recent_chapters;
holdNewLinks.checked = settings.hold_new_links;
pauseUploads.checked = settings.pause_uploads;
showUploadSettings(settings);
ini.value = settings.personal_ini;
opdsEnabled.checked = settings.opds_enabled ?? false;
opdsUser.value = settings.opds_user ?? '';
showPassword(settings.opds_password_set ?? false, opdsEnabled.checked);
template.value = settings.download_template;
dateFormat.value = settings.download_date_format;
storyJobsHours.value = settings.story_jobs_hours;
storyJobsCount.value = settings.story_jobs_count;
mainJobsHours.value = settings.main_jobs_hours;
@@ -52,11 +160,9 @@ export function settingsPage(main, _params, config) {
fill(
main,
h(
'section',
{ class: 'card' },
h('h1', {}, 'Settings'),
h('h2', {}, 'Recheck recent chapters'),
h('h1', {}, 'Settings'),
card(
'Recheck recent chapters',
h(
'div',
{ class: 'controls' },
@@ -70,19 +176,153 @@ export function settingsPage(main, _params, config) {
'Authors most often fix their latest chapters. Each rechecked chapter is one more request to the site. ' +
'Enter 0 to turn this off. A story with its own interval for full rechecks does not use this setting.',
),
saveButton(() => ({ recent_chapters: recent.value === '' ? 0 : Number(recent.value) })),
),
card(
'Book Downloads',
h('label', {}, opdsEnabled, ' Enable OPDS'),
h(
'div',
{ class: 'controls' },
button('Save', async () => {
await api('settings', 'PUT', { recent_chapters: recent.value === '' ? 0 : Number(recent.value) });
'p',
{ class: 'hint', id: 'opds-note' },
'OPDS lets a reader browse and download books. It is off by default. After you enable it and save, use ',
h('a', { href: 'opds' }, 'the OPDS catalog'),
'. ',
OPDS_LOGIN[config.auth_mode] ?? OPDS_LOGIN.local,
' Use HTTPS outside a trusted local network.',
),
config.auth_mode === 'proxy' && [
h(
'div',
{ class: 'credentials' },
h('label', { for: 'opds-user' }, 'OPDS user name'),
opdsUser,
h('label', { for: 'opds-password' }, 'OPDS password'),
h('div', { class: 'controls' }, opdsPassword, setPassword),
),
opdsPasswordStatus,
],
h('div', { class: 'controls' }, h('label', { for: 'download-template' }, 'File name template'), template),
h(
'p',
{ class: 'hint' },
'This template gives the file name of a book that you download from a story page or OPDS. ' +
'To go back to the default, ',
code('{title_sort} - {authors}'),
', clear the field and save.',
),
h(
'details',
{ class: 'help' },
h('summary', {}, 'File name template help'),
h(
'p',
{ class: 'hint' },
'The template follows the ',
externalLink(TEMPLATE_DOCS, 'template rules of Calibre'),
' for file names. Bindery replaces each field in braces with the value of the book, and adds .epub. ' +
'A field with no value becomes empty text. For example, ',
code('{series:|| - }{series_index:0>2s|| - }{title} - {authors}'),
' gives "The Saga - 02 - The Book - Ann Writer.epub".',
),
h('p', { class: 'hint' }, 'You can use these fields:'),
h(
'dl',
{ class: 'fields' },
Object.entries(config.download_fields).map(([name, text]) => [h('dt', {}, code(`{${name}}`)), h('dd', {}, text)]),
),
h(
'ul',
{ class: 'hint' },
h('li', {}, code('{title:.20s}'), ' gives the first 20 characters of the title.'),
h('li', {}, code('{author_sort[0]}'), ' gives the first character of the author sort string.'),
h(
'li',
{},
code('{series:|| - }'),
' adds " - " after the series, but only for a book with a series. The form is ',
code('{field:format|text before|text after}'),
'.',
),
h('li', {}, code('{{'), ' and ', code('}}'), ' give braces.'),
h(
'li',
{},
'{title} and {series} keep a leading "The", "A" or "An". {title_sort} and {series_sort} move it to the ' +
'end, so ' +
'"The Book" becomes "Book, The". Calibre puts the title sort in {title} as well, unless you set its ' +
'tweak save_template_title_series_sorting to strictly_alphabetic.',
),
h(
'li',
{},
'A download has one file name. If the template has folders, Bindery uses the part after the last ',
code('/'),
'.',
),
h(
'li',
{},
'Bindery does not support template functions, such as ',
code('{title:uppercase()}'),
', or the program modes of Calibre.',
),
h('li', {}, 'If the template fails or gives no name for a book, Bindery names the file after the title.'),
),
),
h('div', { class: 'controls' }, h('label', { for: 'download-date-format' }, 'Date format'), dateFormat),
h(
'p',
{ class: 'hint' },
'The form of template dates in {pubdate}, {timestamp} and {last_modified}. To go back to the default, ',
code('%b, %Y'),
', clear the field and save.',
),
h(
'details',
{ class: 'help' },
h('summary', {}, 'Date format help'),
h(
'p',
{ class: 'hint' },
code('%d'),
' gives the day, ',
code('%b'),
' the month name, ',
code('%m'),
' the month number and ',
code('%Y'),
' the year. See the ',
externalLink(DATE_CODES, 'list of all codes'),
'. The default gives "May, 2020". A / in a date becomes _, as in Calibre.',
),
),
ingestNote,
saveButton(
() => ({
opds_enabled: opdsEnabled.checked,
download_template: template.value,
download_date_format: dateFormat.value,
}),
(settings) => {
opdsEnabled.checked = settings.opds_enabled ?? false;
showPassword(passwordSet, opdsEnabled.checked);
template.value = settings.download_template;
dateFormat.value = settings.download_date_format;
toast('Saved');
}, { class: 'primary' }),
},
),
),
h(
'section',
{ class: 'card' },
h('h2', {}, 'Uploads to CWA'),
config.cwa_enabled && card(
'Uploads to CWA',
h('label', {}, linkBooks, ' Link books to CWA library'),
h(
'p',
{ class: 'hint' },
'Find library books through CWA_URL or CALIBRE_DB_PATH. When this is off, uploads use saved CWA book IDs ' +
'or send a plain ePub to INGEST_DIR.',
),
libraryNote,
linkNote,
h('label', {}, holdNewLinks, ' Hold new links for approval'),
h(
'p',
@@ -98,19 +338,16 @@ export function settingsPage(main, _params, config) {
'Bindery downloads chapters and builds books, but sends no book to CWA. ' +
'When you turn the pause off, Bindery sends the books that wait.',
),
h(
'div',
{ class: 'controls' },
button('Save', async () => {
await api('settings', 'PUT', { hold_new_links: holdNewLinks.checked, pause_uploads: pauseUploads.checked });
saveButton(
() => ({ link_books: linkBooks.checked, hold_new_links: holdNewLinks.checked, pause_uploads: pauseUploads.checked }),
(settings) => {
showUploadSettings(settings);
toast('Saved');
}, { class: 'primary' }),
},
),
),
h(
'section',
{ class: 'card' },
h('h2', {}, 'Jobs list'),
card(
'Jobs list',
h(
'div',
{ class: 'controls' },
@@ -135,25 +372,16 @@ export function settingsPage(main, _params, config) {
`A queued or running job always shows. The most either field allows is ${config.job_list_max_hours} hours ` +
`and ${config.job_list_max_count} jobs.`,
),
h(
'div',
{ class: 'controls' },
button('Save', async () => {
await api('settings', 'PUT', {
story_jobs_hours: storyJobsHours.value,
story_jobs_count: storyJobsCount.value,
main_jobs_hours: mainJobsHours.value,
main_jobs_count: mainJobsCount.value,
});
toast('Saved');
}, { class: 'primary' }),
),
saveButton(() => ({
story_jobs_hours: storyJobsHours.value,
story_jobs_count: storyJobsCount.value,
main_jobs_hours: mainJobsHours.value,
main_jobs_count: mainJobsCount.value,
})),
),
globalRulesCard(),
h(
'section',
{ class: 'card' },
h('h2', {}, 'FanFicFare settings (personal.ini)'),
card(
'FanFicFare settings (personal.ini)',
h(
'p',
{ class: 'hint' },
@@ -161,18 +389,17 @@ export function settingsPage(main, _params, config) {
externalLink(INI_DOCS, 'personal.ini documentation'),
' and the ',
externalLink(DEFAULTS_INI, 'list of all settings with their defaults'),
'. The file can hold passwords for story sites. Each user who can log in to Bindery can read them here.',
'. The file can hold passwords for story sites. Each admin can read them here. If BINDERY_ADMIN_USERS is not ' +
'set, each user is an admin.',
),
ini,
warnings,
h(
'div',
{ class: 'controls' },
button('Save', async () => {
const saved = await api('settings', 'PUT', { personal_ini: ini.value });
saveButton(
() => ({ personal_ini: ini.value }),
(saved) => {
showWarnings(saved.warnings);
toast(saved.warnings.length ? 'Saved, with warnings' : 'Saved');
}, { class: 'primary' }),
},
),
),
);
bindery/static/js/stories.js
+159 -24
View File
@@ -3,6 +3,18 @@
import { jobPanel, refreshJobs, watchJobs } from './jobs.js';
import { api, badge, button, checkbox, fill, fmtDate, h, link, showError, toast } from './lib.js';
const STORY_SORT_KEY = 'bindery.story-sort';
const STORY_COLUMNS = [
{ label: 'Title', key: 'title', type: 'text' },
{ label: 'Author', key: 'author', type: 'text' },
{ label: 'Site', key: 'site', type: 'text' },
{ label: 'Chapters', key: 'chapter_count', type: 'number' },
{ label: 'Pages', key: 'page_count', type: 'number' },
{ label: 'Status', key: 'status', type: 'text' },
{ label: 'Last change', key: 'last_changed', type: 'date' },
{ label: 'Last check', key: 'last_checked', type: 'date' },
];
function libraryImport(config) {
if (!config.library_available) {
return h('p', { class: 'hint' }, 'Set CWA_URL to import books from the library.');
@@ -10,11 +22,14 @@ function libraryImport(config) {
let candidates = [];
let selected = new Set();
let check = true;
let empty = '';
// The load of the list, kept for each search, because CWA takes a while to send the list.
let list;
const results = h('div');
const render = () => {
if (!candidates.length) {
fill(results, h('p', { class: 'empty' }, 'Every library book from a story site is tracked already.'));
fill(results, h('p', { class: 'empty' }, empty));
return;
}
const toggle = (id, on) => {
@@ -39,7 +54,7 @@ function libraryImport(config) {
'th',
{},
checkbox(selected.size === candidates.length, (on) => {
selected = new Set(on ? candidates.map((c) => c.calibre_id) : []);
selected = new Set(on ? candidates.map((c) => c.id) : []);
render();
}, { 'aria-label': 'Select all' }),
),
@@ -55,10 +70,10 @@ function libraryImport(config) {
h(
'tr',
{},
h('td', {}, checkbox(selected.has(c.calibre_id), (on) => toggle(c.calibre_id, on), { 'aria-label': c.title })),
h('td', {}, checkbox(selected.has(c.id), (on) => toggle(c.id, on), { 'aria-label': c.title })),
h('td', {}, c.title),
h('td', {}, c.authors),
h('td', {}, c.site),
h('td', {}, c.authors.join(' & ')),
h('td', {}, c.publisher),
),
),
),
@@ -68,11 +83,16 @@ function libraryImport(config) {
'Import selected',
async () => {
const books = candidates
.filter((c) => selected.has(c.calibre_id))
.map((c) => ({ id: c.calibre_id, uuid: c.uuid, title: c.title }));
if (!books.length) return;
.filter((c) => selected.has(c.id))
.map(({ id, uuid, title }) => ({ id, uuid, title }));
if (!books.length) {
toast('Select books first', true);
return;
}
const jobs = await api('library/import', 'POST', { books, check });
toast(`Queued ${jobs.length} imports`);
const queued = new Set(books.map((b) => b.id));
list = list?.then((all) => all.filter((c) => !queued.has(c.id)));
fill(results);
refreshJobs();
},
@@ -81,22 +101,68 @@ function libraryImport(config) {
);
};
// The full list starts with each book selected. A search starts with no book selected.
const show = (books, all) => {
candidates = books;
selected = new Set(all ? books.map((c) => c.id) : []);
empty = all
? 'Every library book from a story site is tracked already.'
: 'No library books from a story site match. A book that Bindery tracks already does not show.';
render();
};
const query = h('input', {
type: 'search',
placeholder: 'Search the library by title, author or site',
'aria-label': 'Search the library',
});
const load = () => {
list ??= api('library/candidates').catch((e) => {
list = undefined;
throw e;
});
return list;
};
// A book matches if its title, authors and site hold each word of the search.
const search = async () => {
const words = query.value.toLowerCase().split(/\s+/).filter(Boolean);
if (!words.length) return;
const books = await load();
show(books.filter((c) => words.every((w) => [c.title, ...c.authors, c.publisher].join(' ').toLowerCase().includes(w))), false);
};
const submit = h('button', { type: 'submit' }, 'Search');
return h(
'div',
{},
h(
'p',
{ class: 'hint' },
'Finds library books whose publisher is a story site, for example books from the FanFicFare calibre plugin. ' +
'Search the library, or show all the stories in it. Both find only books whose publisher is a story site, ' +
'for example books from the FanFicFare calibre plugin, and that Bindery does not track yet. ' +
'Importing downloads each ePub from CWA and keeps its chapters, including chapters the site no longer has.',
),
h(
'form',
{
class: 'library-search',
onsubmit: (e) => {
e.preventDefault();
if (submit.disabled) return;
submit.disabled = true;
search()
.catch(showError)
.finally(() => (submit.disabled = false));
},
},
query,
submit,
),
h(
'div',
{ class: 'controls' },
button('Find stories in the library', async () => {
candidates = await api('library/candidates');
selected = new Set(candidates.map((c) => c.calibre_id));
render();
button('Show all stories in the library', async () => {
list = undefined;
show(await load(), true);
}),
h('label', {}, checkbox(true, (on) => (check = on)), ' Check the site for new chapters after import'),
),
@@ -160,7 +226,21 @@ export function epubImport(storyId, title = 'Import an ePub file') {
return card;
}
function storyDate(label, value) {
const date = new Date(value);
let content = fmtDate(value);
if (value && !Number.isNaN(date.getTime())) {
content = h('time', { datetime: value, title: content }, content);
}
return h('td', { class: 'date', 'data-label': label }, content);
}
function storyRow(s) {
const status = s.status && badge(s.status, s.status, s.message);
if (status && s.message) {
status.setAttribute('tabindex', '0');
status.setAttribute('aria-description', s.message);
}
return h(
'tr',
{},
@@ -171,18 +251,19 @@ function storyRow(s) {
'td',
{ 'data-label': 'Chapters' },
String(s.chapter_count),
s.offsite_count > 0 && [' ', badge('warn', `${s.offsite_count} removed`, 'Chapters the site no longer lists')],
s.revised_count > 0 && [' ', badge('info', `${s.revised_count} revised`, 'Chapters the site changed since Bindery added them')],
s.offsite_count > 0 && h('small', { title: 'Chapters the site no longer lists' }, `${s.offsite_count} removed`),
s.revised_count > 0 && h('small', { title: 'Chapters the site changed since Bindery added them' }, `${s.revised_count} revised`),
),
// A story has no page count until Bindery builds its book.
h('td', { 'data-label': 'Pages', title: 'Estimated from the built book' }, s.page_count != null && String(s.page_count)),
h(
'td',
{ 'data-label': 'Status' },
s.status && badge(s.status),
s.upload_held && [' ', badge('warn', 'held', 'The book did not go to CWA. See the story page.')],
' ',
h('small', {}, s.message),
status,
s.approval_held && [' ', badge('warn', 'held', 'The book waits for your approval. See the story page.')],
),
h('td', { class: 'date', 'data-label': 'Last check' }, fmtDate(s.last_checked)),
storyDate('Last change', s.last_changed),
storyDate('Last check', s.last_checked),
);
}
@@ -190,20 +271,72 @@ export function storiesPage(main, _params, config) {
document.title = 'Stories · Bindery';
let stories = null;
let filter = '';
let sort = 'title';
let direction = 1;
try {
const saved = JSON.parse(window.localStorage.getItem(STORY_SORT_KEY));
if (saved && STORY_COLUMNS.some(({ key }) => key === saved.column) && [1, -1].includes(saved.direction)) {
sort = saved.column;
direction = saved.direction;
}
} catch {
// Keep the default if storage is blocked or its value is invalid.
}
const sortButtons = new Map();
const table = h('div', {}, h('p', { class: 'empty' }, 'Loading…'));
const jobs = jobPanel({ hours: config.main_jobs_hours, count: config.main_jobs_count });
const render = () => {
const focusedSort = [...sortButtons].find(([, control]) => control === document.activeElement)?.[0];
const q = filter.trim().toLowerCase();
const shown = q
? stories.filter((s) => [s.title, s.author, s.site, s.url].some((v) => v?.toLowerCase().includes(q)))
: stories;
: [...stories];
const sortType = STORY_COLUMNS.find(({ key }) => key === sort).type;
shown.sort((a, b) => {
if (sortType === 'date') {
const left = Date.parse(a[sort]);
const right = Date.parse(b[sort]);
if (Number.isNaN(left) !== Number.isNaN(right)) return Number.isNaN(left) ? 1 : -1;
const order = (left - right) * direction;
if (order) return order;
} else if (sort !== 'title') {
// A story with no number goes last in each direction.
if (sortType === 'number' && (a[sort] == null) !== (b[sort] == null)) return a[sort] == null ? 1 : -1;
const order = sortType === 'number'
? a[sort] - b[sort]
: (a[sort] || '').localeCompare(b[sort] || '', undefined, { sensitivity: 'base' });
if (order) return order * direction;
}
const order = a.sort_title.localeCompare(b.sort_title, undefined, { sensitivity: 'base' }) || a.id - b.id;
return sort === 'title' ? order * direction : order;
});
if (!shown.length) {
const text = stories.length ? 'No stories match.' : 'No stories yet. Add a URL above, or import an ePub.';
fill(table, h('p', { class: 'empty' }, text));
return;
}
const head = ['Title', 'Author', 'Site', 'Chapters', 'Status', 'Last check'].map((t) => h('th', {}, t));
const heading = ({ label, key, type }) => {
const active = sort === key;
const nextDirection = active ? -direction : type === 'text' ? 1 : -1;
const orders = type === 'text' ? ['A to Z', 'Z to A']
: type === 'date' ? ['oldest first', 'newest first'] : ['fewest first', 'most first'];
const action = `Sort by ${label.toLowerCase()}, ${orders[nextDirection === 1 ? 0 : 1]}`;
const control = button(label, () => {
sort = key;
direction = nextDirection;
try {
window.localStorage.setItem(STORY_SORT_KEY, JSON.stringify({ column: sort, direction }));
} catch {
// Sorting still works when the browser blocks storage.
}
render();
sortButtons.get(key).focus();
}, { title: action, 'aria-label': action });
sortButtons.set(key, control);
return h('th', { scope: 'col', 'aria-sort': active ? direction === 1 ? 'ascending' : 'descending' : 'none' }, control);
};
const head = STORY_COLUMNS.map(heading);
fill(
table,
h(
@@ -212,6 +345,7 @@ export function storiesPage(main, _params, config) {
h('table', {}, h('thead', {}, h('tr', {}, head)), h('tbody', {}, shown.map(storyRow))),
),
);
if (focusedSort) sortButtons.get(focusedSort).focus();
};
const load = () =>
@@ -262,7 +396,7 @@ export function storiesPage(main, _params, config) {
'div',
{ class: 'controls' },
add,
h(
config.cwa_enabled && h(
'span',
{ class: 'hint' },
'If CWA already has this story, Bindery links that book and keeps its chapters, then downloads only the chapters that are missing.',
@@ -292,7 +426,8 @@ export function storiesPage(main, _params, config) {
),
h('details', { class: 'card', open: true }, h('summary', {}, h('h2', {}, 'Jobs')), jobs.el),
epubImport(),
h('details', { class: 'card' }, h('summary', {}, h('h2', {}, 'Import from the library')), libraryImport(config)),
config.cwa_enabled &&
h('details', { class: 'card' }, h('summary', {}, h('h2', {}, 'Import from the library')), libraryImport(config)),
);
load();
bindery/static/js/story.js
+130 -34
View File
@@ -24,7 +24,6 @@ import {
import { regexDocs, rulesEditor, rulesPreview } from './rules.js';
import { epubImport } from './stories.js';
const SITE_MODES = ['new', 'recheck', 'recheck_all', 'full', 'metadata'];
const MODE_HINTS = {
new:
@@ -55,7 +54,7 @@ const editText = (key, value) => {
return formatField(key, value);
};
const authorsText = (authors) => (Array.isArray(authors) ? authors.join(' & ') : authors);
const authorsText = (authors) => authors.join(' & ');
// On a narrow screen, the rows stack instead of scrolling sideways. `extra` names the wrapper class
// that gives a table's stacked rows their own layout, such as picking out the row's first cell.
@@ -69,6 +68,9 @@ const table = (headings, rows, extra = '') =>
export function storyPage(main, { storyId }, config) {
const id = Number(storyId);
const fields = config.metadata_fields;
// Without CWA_ENABLED=true, the page shows nothing about CWA.
const cwa = config.cwa_enabled;
const libraryStays = cwa ? ' The library book stays.' : '';
let details = null;
let selected = new Set();
let mode = 'new';
@@ -87,7 +89,12 @@ export function storyPage(main, { storyId }, config) {
const metadata = h('div', { class: 'metadata' });
const updates = h('div');
const chapters = h('div');
const selectAll = h('span', { class: 'chapter-select-all' });
// One dialog for Move… and Delete…. It names the selected chapters before anything changes.
const dialog = h('dialog', { class: 'chapter-dialog', 'aria-labelledby': 'chapter-dialog-title' });
let moveAfterId = null;
// A shift-click on a chapter checkbox sets every chapter from the last one clicked.
let anchor = null;
let shift = false;
const jobs = jobPanel({ hours: config.story_jobs_hours, count: config.story_jobs_count });
const linked = h('div');
const searchResults = h('div');
@@ -128,7 +135,7 @@ export function storyPage(main, { storyId }, config) {
let library = 'Not in the library yet';
if (book) library = `Library book ${book.id}`;
else if (story.calibre_id) library = `Library book ${story.calibre_id}`;
else if (story.ingested_at) library = 'Sent to CWA, unlinked';
else if (story.ingested_at) library = details.link_books === false ? 'Sent to ingest folder' : 'Sent to CWA, unlinked';
fill(
summary,
// The version in the URL changes with the image, so the browser keeps the cover until the site changes it.
@@ -142,7 +149,7 @@ export function storyPage(main, { storyId }, config) {
{ class: 'byline' },
`${story.author} · `,
externalLink(story.url, story.site),
` · ${library}`,
cwa && ` · ${library}`,
),
h(
'p',
@@ -150,6 +157,7 @@ export function storyPage(main, { storyId }, config) {
story.status && badge(story.status),
` ${story.message || ''}`,
story.last_checked && h('span', { class: 'last-check' }, ` Last check ${fmtDate(story.last_checked)}.`),
story.last_changed && h('span', { class: 'last-update' }, ` Last update ${fmtDate(story.last_changed)}.`),
),
details.library_error && h('p', { class: 'error' }, `CWA: ${details.library_error}`),
holdNotice(),
@@ -159,15 +167,18 @@ export function storyPage(main, { storyId }, config) {
const approve = async (uuid) => {
const result = await api(`stories/${id}/approve`, 'POST', { uuid });
toast(result.job ? 'Approved. Bindery rebuilds the book and sends it.' : 'Approved');
toast(result.job ? 'Approved. Rebuild queued.' : 'Approved');
if (result.job) refreshJobs();
await load();
};
const approveAsNew = async () => {
const message =
'Unlink the library book, and send this story to CWA as a new book? If CWA already has the book, you get a ' +
'duplicate. In that case, search for the book under "Library link" and link it instead.';
const action = details.pause_uploads
? 'Unlink the library book, and approve this story as a new book? Uploads remain paused. '
: 'Unlink the library book, and send this story to CWA as a new book? ';
const message = action +
'If CWA already has the book, you get a duplicate. ' +
'In that case, search for the book under "Library link" and link it instead.';
if (!window.confirm(message)) return;
await api(`stories/${id}/link`, 'POST', { uuid: null });
await approve(null);
@@ -175,16 +186,19 @@ export function storyPage(main, { storyId }, config) {
// Why the book did not go to CWA, and the approval of a new library link.
function holdNotice() {
if (!cwa) return null;
const { story, book } = details;
const paused = story.upload_held && details.uploads_paused;
// A story that went to CWA without a link gets its book at the next update, and the approval comes then.
const approval = story.needs_approval && (story.calibre_uuid || !story.ingested_at);
if (!paused && !approval) return null;
let target = 'The next upload goes to CWA as a new book.';
const linking = details.link_books !== false;
const unlinked = !linking && story.calibre_id == null;
// A previously uploaded book needs a known target before approval.
const approval = linking && story.needs_approval && (story.calibre_id != null || !story.ingested_at);
const paused = details.pause_uploads && (story.upload_held || approval);
if (!paused && !approval && !unlinked) return null;
let target = 'Approve this story as a new CWA book.';
if (book) {
target = `The next upload replaces the ePub of library book ${book.id}: ${book.title} · ${authorsText(book.authors)}.`;
} else if (story.calibre_id) {
target = `The next upload replaces the ePub of library book ${story.calibre_id}.`;
target = `Approve replacing the ePub of library book ${book.id}: ${book.title} · ${authorsText(book.authors)}.`;
} else if (story.calibre_id != null) {
target = `Approve replacing the ePub of library book ${story.calibre_id}.`;
}
return h(
'div',
@@ -193,19 +207,27 @@ export function storyPage(main, { storyId }, config) {
h(
'p',
{},
'Uploads to CWA are paused. Bindery sends this book when you turn the pause off in ',
link('/settings', 'Settings'),
'.',
details.ingest_available === false
? 'Set INGEST_DIR and restart Bindery to enable uploads.'
: ['Uploads to CWA are paused. Resume them in ', link('/settings', 'Settings'), '.'],
),
unlinked &&
h(
'p',
{},
'Uploads of this book have no CWA book ID. If CWA imports them, updates will create duplicate books.',
),
approval && [
h('p', {}, `${story.upload_held ? 'The book waits for your approval. ' : ''}${target} Make sure that this is right.`),
h('p', {}, target),
h(
'div',
{ class: 'controls' },
button(story.upload_held ? 'Approve and send' : 'Approve', () => approve(story.calibre_uuid), {
button(story.upload_held && !details.pause_uploads ? 'Approve and send' : 'Approve', () => approve(story.calibre_uuid), {
class: 'primary',
}),
story.calibre_uuid && button('Send as a new book instead', approveAsNew),
story.calibre_uuid && button(
details.pause_uploads ? 'Approve as a new book instead' : 'Send as a new book instead', approveAsNew,
),
),
],
);
@@ -381,8 +403,17 @@ export function storyPage(main, { storyId }, config) {
function renderChapters() {
const list = details.chapters;
const ids = list.map((c) => c.id);
const pick = (i, on) => {
const from = shift ? ids.indexOf(anchor) : -1;
const range = from < 0 ? [ids[i]] : ids.slice(Math.min(from, i), Math.max(from, i) + 1);
for (const cid of range) on ? selected.add(cid) : selected.delete(cid);
anchor = ids[i];
shift = false;
if (range.length > 1) renderChapters();
};
let number = 0;
const rows = list.map((c) => {
const rows = list.map((c, i) => {
if (c.included) number += 1;
const route = `/story/${id}/chapter/${c.id}`;
return h(
@@ -391,8 +422,10 @@ export function storyPage(main, { storyId }, config) {
h(
'td',
{},
checkbox(selected.has(c.id), (on) => (on ? selected.add(c.id) : selected.delete(c.id)), {
checkbox(selected.has(c.id), (on) => pick(i, on), {
'aria-label': `Select ${chapterTitle(c.title)}`,
title: 'Shift-click to select all chapters from the last one you clicked',
onclick: (e) => (shift = e.shiftKey),
}),
),
h('td', { 'data-label': '#' }, c.included ? String(number) : ''),
@@ -412,11 +445,12 @@ export function storyPage(main, { storyId }, config) {
selected = new Set(on ? list.map((c) => c.id) : []);
renderChapters();
}, { 'aria-label': 'Select all chapters' });
fill(selectAll, h('label', {}, all, ' Select all'));
// A phone hides the other headings, so the text shows only there.
const selectAll = h('label', {}, all, h('span', { class: 'select-all-text' }, ' Select all'));
fill(
chapters,
table(
['', '#', 'Title', 'State', 'Revisions', 'Added'],
[selectAll, '#', 'Title', 'State', 'Revisions', 'Added'],
rows.length ? rows : h('tr', {}, h('td', { colspan: 6, class: 'empty' }, 'No chapters stored yet.')),
'chapters-table',
),
@@ -462,8 +496,66 @@ export function storyPage(main, { storyId }, config) {
await load();
};
const finish = async (message) => {
dialog.close();
toast(message);
refreshJobs();
await load();
};
const moveForm = (picked, chapterIds) => {
const others = details.chapters.filter((c) => !selected.has(c.id));
const after = h(
'select',
{ id: 'move-after' },
h('option', { value: '' }, 'The start'),
others.map((c) => h('option', { value: String(c.id) }, chapterTitle(c.title))),
);
after.value = others.some((c) => c.id === moveAfterId) ? String(moveAfterId) : '';
return [
h('div', { class: 'controls' }, h('label', { for: 'move-after' }, picked.length === 1 ? 'Put it after' : 'Put them after'), after),
button('Move', async () => {
moveAfterId = after.value === '' ? null : Number(after.value);
await api(`stories/${id}/chapters/move`, 'POST', { chapter_ids: chapterIds, after: moveAfterId });
await finish('Chapters moved. The book is being rebuilt.');
}, { class: 'primary' }),
];
};
const deleteForm = (picked, chapterIds) => [
h(
'p',
{},
`Bindery deletes ${picked.length === 1 ? 'this chapter and its' : 'these chapters and their'} revisions. `,
'You cannot undo this.',
picked.some((c) => c.on_site) ? ' A chapter that the site still lists comes back with the next download of new chapters.' : '',
),
button('Delete', async () => {
await api(`stories/${id}/chapters/delete`, 'POST', { chapter_ids: chapterIds });
await finish('Chapters deleted. The book is being rebuilt.');
}, { class: 'danger' }),
];
const openDialog = (verb) => {
if (!selected.size) {
toast('Select chapters first', true);
return;
}
const picked = details.chapters.filter((c) => selected.has(c.id));
const chapterIds = picked.map((c) => c.id);
const [body, action] = (verb === 'Move' ? moveForm : deleteForm)(picked, chapterIds);
fill(
dialog,
h('h3', { id: 'chapter-dialog-title' }, `${verb} ${plural(picked.length, 'chapter')}`),
h('ol', { class: 'dialog-chapters' }, picked.map((c) => h('li', {}, chapterTitle(c.title)))),
body,
h('div', { class: 'dialog-actions' }, h('button', { type: 'button', onclick: () => dialog.close() }, 'Cancel'), action),
);
dialog.showModal();
};
const modeHint = h('p', { class: 'hint' }, MODE_HINTS[mode]);
const modes = SITE_MODES.map((m) =>
const modes = Object.keys(MODE_HINTS).map((m) =>
h(
'label',
{},
@@ -574,8 +666,10 @@ export function storyPage(main, { storyId }, config) {
{ class: 'hint' },
'Bindery writes this metadata into the ePub. You can edit each field. A locked field keeps its value on ' +
'update, unless you take it from the site above. An unlocked field keeps an edited value until the site ' +
"changes that field. CWA reads the ePub's metadata only when it first imports a book, so later changes " +
'show in the ePub but not in the CWA catalog.',
'changes that field.',
cwa &&
" CWA reads the ePub's metadata only when it first imports a book, so later changes show in the ePub " +
'but not in the CWA catalog.',
),
metadata,
),
@@ -601,7 +695,6 @@ export function storyPage(main, { storyId }, config) {
'details',
{ class: 'card', open: true },
h('summary', {}, h('h2', {}, 'Chapters')),
selectAll,
h(
'div',
{ class: 'controls chapter-actions' },
@@ -610,12 +703,15 @@ export function storyPage(main, { storyId }, config) {
),
button('Take out of book', () => setIncluded(false)),
button('Put back in book', () => setIncluded(true)),
button('Move…', () => openDialog('Move')),
button('Delete…', () => openDialog('Delete'), { class: 'danger' }),
),
chapters,
dialog,
),
epubImport(id, 'Import chapters from an ePub'),
h('details', { class: 'card', open: true }, h('summary', {}, h('h2', {}, 'Recent jobs')), jobs.el),
h(
cwa && h(
'details',
{ class: 'card' },
h('summary', {}, h('h2', {}, 'Library link')),
@@ -647,11 +743,11 @@ export function storyPage(main, { storyId }, config) {
'details',
{ class: 'card card-danger' },
h('summary', {}, h('h2', {}, 'Stop tracking')),
h('p', { class: 'hint' }, 'Deletes the stored chapters, revisions and rules for this story. The library book stays.'),
h('p', { class: 'hint' }, `Deletes the stored chapters, revisions and rules for this story.${libraryStays}`),
button(
'Stop tracking this story',
async () => {
const message = 'Stop tracking this story? Stored chapters, revisions and rules are deleted. The library book stays.';
const message = `Stop tracking this story? Stored chapters, revisions and rules are deleted.${libraryStays}`;
if (!window.confirm(message)) return;
await api(`stories/${id}`, 'DELETE');
window.location.hash = '#/';
bindery/static/style.css
+213 -11
View File
@@ -390,9 +390,8 @@ details.card[open] > summary .summary-action {
justify-content: space-between;
}
.chapter-select-all {
display: block;
margin-top: 8px;
.select-all-text {
display: none;
}
/* Stays visible while the chapter list scrolls, so the buttons need no scrolling back up to
@@ -428,6 +427,10 @@ label:has(> input[type='radio']) {
display: block;
}
label:has(> input[type='checkbox']:disabled) {
color: var(--muted);
}
.library-results {
list-style: none;
margin: 8px 0 0;
@@ -448,6 +451,25 @@ label:has(> input[type='radio']) {
margin: 0;
}
ul.hint {
padding-left: 20px;
}
/* The fields of the download template: each name with its meaning beside it. */
.fields {
display: grid;
grid-template-columns: max-content 1fr;
gap: 2px 12px;
max-width: 80ch;
margin: 4px 0 8px;
color: var(--muted);
font-size: var(--text-sm);
dd {
margin: 0;
}
}
.notice {
margin: 12px 0;
padding: 10px 14px;
@@ -467,6 +489,7 @@ label:has(> input[type='radio']) {
.byline,
.last-check,
.last-update,
.story-site,
.field-default,
small,
@@ -552,6 +575,11 @@ button.danger:hover:not(:disabled) {
background: var(--err-bg);
}
input:not([type='checkbox'], [type='radio']):disabled {
opacity: 0.55;
cursor: not-allowed;
}
button:disabled {
opacity: 0.55;
cursor: default;
@@ -695,6 +723,35 @@ textarea {
line-height: 1.45;
}
/* A user name over a password, with the labels in one column. */
.credentials {
display: grid;
grid-template-columns: max-content minmax(0, 1fr);
align-items: center;
gap: 8px 12px;
max-width: 48em;
margin-top: 8px;
input {
width: 18em;
}
input[readonly] {
background: var(--card-bg);
font-family: var(--mono);
cursor: default;
}
.controls {
margin-top: 0;
}
}
.template {
flex: 1 1 24em;
max-width: 48em;
}
.metadata input:not([type='checkbox']) {
width: 100%;
max-width: 32em;
@@ -745,6 +802,56 @@ textarea {
overflow-x: auto;
}
.stories-table th {
padding: 0;
}
.stories-table th button {
inline-size: 100%;
justify-content: flex-start;
padding: 6px 8px;
border: 0;
border-radius: 4px;
background: none;
color: inherit;
font: inherit;
}
.stories-table th button:is(:hover, :focus-visible) {
background: var(--info-bg);
color: var(--info);
}
.stories-table th[aria-sort] button::after {
content: '↕';
display: inline-block;
inline-size: 1em;
margin-inline-start: 0.3em;
}
.stories-table th[aria-sort='ascending'] button::after {
content: '↑';
}
.stories-table th[aria-sort='descending'] button::after {
content: '↓';
}
.stories-table th[aria-sort]:not([aria-sort='none']) {
color: var(--text);
}
.stories-table td[data-label='Chapters'] small {
display: block;
white-space: nowrap;
}
@media (min-width: 601px) {
.stories-table .date {
inline-size: 1%;
}
}
/* The story list and the chapter list stack into cards below 600px, instead of scrolling
sideways. data-label repeats each column heading in front of its value. */
@media (max-width: 600px) {
@@ -768,14 +875,43 @@ textarea {
display: block;
}
.stories-table thead,
.chapters-table thead {
position: absolute;
width: 1px;
height: 1px;
overflow: hidden;
white-space: nowrap;
clip-path: inset(50%);
/* The column headings become a row of sort chips. The chip of the sorted column is tinted and
shows the direction. The other chips show no arrow. */
.stories-table thead tr {
display: flex;
flex-wrap: wrap;
gap: 6px;
margin-bottom: 10px;
}
.stories-table thead tr::before {
content: 'Sort by';
flex-basis: 100%;
color: var(--muted);
font-size: var(--text-sm);
}
.stories-table th {
padding: 0;
border: 0;
}
.stories-table th button {
inline-size: auto;
padding: 3px 10px;
border: 1px solid var(--border);
border-radius: 999px;
background: var(--surface);
font-size: var(--text-sm);
}
.stories-table th[aria-sort='none'] button::after {
display: none;
}
.stories-table th:not([aria-sort='none']) button {
background: var(--info-bg);
color: var(--info);
}
.stories-table tbody tr,
@@ -804,6 +940,11 @@ textarea {
color: var(--muted);
}
/* A story with no page count yet shows no line for it. */
.stories-table td:empty {
display: none;
}
/* The chapter list has far more rows than the story list, so each row packs its cells onto
three lines instead of one line per cell: checkbox and number, title, then the rest. */
.chapters-table tbody tr {
@@ -823,6 +964,57 @@ textarea {
flex-basis: 100%;
font-weight: 600;
}
/* Of the chapter list headings, only the Select all checkbox shows. */
/* The left padding is the 1px border and 12px padding of a row card, so the checkboxes line up. */
.chapters-table th {
border-bottom: 0;
padding: 0 0 8px 13px;
}
.chapters-table th:not(:first-child) {
display: none;
}
.select-all-text {
display: inline;
}
}
/* The Move… and Delete… dialog of the chapter list. */
.chapter-dialog {
width: min(32rem, calc(100vw - 32px));
max-height: calc(100dvh - 32px);
padding: 16px 20px;
border: 1px solid var(--border);
border-radius: 8px;
background: var(--surface);
color: var(--text);
}
.chapter-dialog::backdrop {
background: rgb(0 0 0 / 0.4);
}
.chapter-dialog h3 {
margin: 0 0 8px;
}
.dialog-chapters {
max-height: 40dvh;
overflow-y: auto;
margin: 0 0 12px;
padding: 8px 8px 8px 32px;
border: 1px solid var(--border);
border-radius: 6px;
background: var(--card-bg);
}
.dialog-actions {
display: flex;
justify-content: flex-end;
gap: 8px;
margin-top: 16px;
}
/* Jobs */
@@ -932,6 +1124,16 @@ textarea {
margin-top: 12px;
}
/* Help that folds shut under a setting, such as the help for the download template. */
.help {
margin: 0 0 12px;
> summary {
color: var(--muted);
font-size: var(--text-sm);
}
}
/* Toasts */
#toasts {
bindery/store.py
+97 -42
View File
@@ -6,8 +6,9 @@ site without losing text.
"""
import json
import re
import sqlite3
from collections.abc import Iterator
from collections.abc import Callable, Iterator
from contextlib import contextmanager
from datetime import datetime
from pathlib import Path
@@ -120,6 +121,14 @@ def _story(row: sqlite3.Row | None) -> dict[str, Any] | None:
return d
_ARTICLE = re.compile(r"^(?:the|an?)\s+", re.I)
def sort_title(title: str | None) -> str:
"""The key that sorts a story list by title, without a leading "The", "A" or "An"."""
return _ARTICLE.sub("", (title or "").strip()).casefold()
class Store:
def __init__(self, path: Path) -> None:
self.path = Path(path)
@@ -190,7 +199,21 @@ class Store:
AND c.on_site = 0) AS offsite_count,
(SELECT COUNT(*) FROM chapters c WHERE c.story_id = s.id AND c.included = 1
AND (SELECT COUNT(*) FROM revisions r WHERE r.chapter_id = c.id) > 1) AS revised_count
FROM stories s ORDER BY s.title COLLATE NOCASE""").fetchall()
FROM stories s""").fetchall()
# The story list of the web interface sorts by the same key.
stories = [dict(_story(r), sort_title=sort_title(r["title"] or r["url"])) for r in rows]
return sorted(stories, key=lambda s: s["sort_title"])
def catalog(self) -> list[dict[str, Any]]:
"""The stories with included chapters, with the values of the OPDS catalog. A story has one cover at most,
and `cover_key` changes with it."""
with self._connect() as conn:
rows = conn.execute("""
SELECT s.id, s.url, s.site, s.title, s.author, s.pinned, s.site_values, s.last_changed, s.created_at,
(SELECT COUNT(*) FROM chapters c WHERE c.story_id = s.id AND c.included = 1) AS chapter_count,
i.mime AS cover_mime, i.url || ' ' || length(i.data) AS cover_key
FROM stories s LEFT JOIN images i ON i.story_id = s.id AND i.is_cover = 1
WHERE EXISTS (SELECT 1 FROM chapters c WHERE c.story_id = s.id AND c.included = 1)""").fetchall()
return [_story(r) for r in rows]
def update_story(self, story_id: int, **fields: Any) -> None:
@@ -233,18 +256,6 @@ class Store:
).fetchall()
return [dict(r) for r in rows]
def current_revisions(self, story_id: int) -> dict[int, dict[str, Any]]:
"""The rev, text_hash and source of the current revision of each chapter, by chapter id."""
with self._connect() as conn:
rows = conn.execute(
"""
SELECT c.id, r.rev, r.text_hash, r.source FROM chapters c
JOIN revisions r ON r.chapter_id = c.id AND r.rev = c.current_rev
WHERE c.story_id = ?""",
(story_id,),
).fetchall()
return {r["id"]: {"rev": r["rev"], "text_hash": r["text_hash"], "source": r["source"]} for r in rows}
def replace_revision(self, chapter_id: int, rev: int, html: str, text_hash: str) -> None:
with self._connect() as conn:
conn.execute(
@@ -257,41 +268,54 @@ class Store:
row = conn.execute("SELECT * FROM chapters WHERE id = ?", (chapter_id,)).fetchone()
return dict(row) if row else None
@staticmethod
def _insert_chapter(
conn: sqlite3.Connection, story_id: int, chapter: dict[str, str], source: str, position: int
) -> int:
"""Store a chapter with its first revision. The chapter needs url, title, html and text_hash."""
changed_at = now()
cur = conn.execute(
"INSERT INTO chapters (story_id, position, url, title, added_at) VALUES (?, ?, ?, ?, ?)",
(story_id, position, chapter["url"], chapter["title"], changed_at),
)
chapter_id = int(cur.lastrowid)
conn.execute(
"INSERT INTO revisions (chapter_id, rev, html, text_hash, source, fetched_at) VALUES (?, 1, ?, ?, ?, ?)",
(chapter_id, chapter["html"], chapter["text_hash"], source, changed_at),
)
conn.execute("UPDATE stories SET last_changed = ? WHERE id = ?", (changed_at, story_id))
return chapter_id
def add_chapter(
self, story_id: int, url: str, title: str, html: str, text_hash: str, source: str, position: int
) -> int:
"""Add one chapter at a position. The jobs use add_chapters, which also places the chapters."""
chapter = {"url": url, "title": title, "html": html, "text_hash": text_hash}
with self._connect() as conn:
cur = conn.execute(
"INSERT INTO chapters (story_id, position, url, title, added_at) VALUES (?, ?, ?, ?, ?)",
(story_id, position, url, title, now()),
)
chapter_id = int(cur.lastrowid)
conn.execute(
"INSERT INTO revisions (chapter_id, rev, html, text_hash, source, fetched_at) "
"VALUES (?, 1, ?, ?, ?, ?)",
(chapter_id, html, text_hash, source, now()),
)
return chapter_id
return self._insert_chapter(conn, story_id, chapter, source, position)
def add_chapters(self, story_id: int, chapters: list[dict[str, str]], source: str, order: list[str]) -> list[int]:
def add_chapters(
self,
story_id: int,
chapters: list[dict[str, str]],
source: str,
order: list[str] | Callable[[list[str]], list[str]],
) -> list[int]:
"""Add chapters and put the story in the order of `order`, a list of chapter URLs, in one transaction.
Each chapter needs url, title, html and text_hash. Chapters that `order`
leaves out go after the others and keep their order.
`order` can be a function. It gets the URLs of the stored chapters in book
order, and returns the list. It runs in the transaction, so it sees a move or
a delete that the user made a moment before.
"""
with self._connect() as conn:
ids = []
for c in chapters:
cur = conn.execute(
"INSERT INTO chapters (story_id, position, url, title, added_at) VALUES (?, 0, ?, ?, ?)",
(story_id, c["url"], c["title"], now()),
)
ids.append(int(cur.lastrowid))
conn.execute(
"INSERT INTO revisions (chapter_id, rev, html, text_hash, source, fetched_at) "
"VALUES (?, 1, ?, ?, ?, ?)",
(ids[-1], c["html"], c["text_hash"], source, now()),
)
conn.execute("BEGIN IMMEDIATE")
if callable(order):
stored = conn.execute("SELECT url FROM chapters WHERE story_id = ? ORDER BY position, id", (story_id,))
order = order([r["url"] for r in stored])
ids = [self._insert_chapter(conn, story_id, c, source, 0) for c in chapters]
rows = conn.execute(
"SELECT id, url FROM chapters WHERE story_id = ? ORDER BY position, id", (story_id,)
).fetchall()
@@ -304,6 +328,7 @@ class Store:
def add_revision(self, chapter_id: int, html: str, text_hash: str, source: str) -> int:
"""Store a new revision and make it the current one."""
changed_at = now()
with self._connect() as conn:
rev = conn.execute(
"SELECT COALESCE(MAX(rev), 0) + 1 FROM revisions WHERE chapter_id = ?", (chapter_id,)
@@ -311,9 +336,15 @@ class Store:
conn.execute(
"INSERT INTO revisions (chapter_id, rev, html, text_hash, source, fetched_at) "
"VALUES (?, ?, ?, ?, ?, ?)",
(chapter_id, rev, html, text_hash, source, now()),
(chapter_id, rev, html, text_hash, source, changed_at),
)
conn.execute(
"UPDATE chapters SET current_rev = ?, changed_at = ? WHERE id = ?", (rev, changed_at, chapter_id)
)
conn.execute(
"UPDATE stories SET last_changed = ? WHERE id = (SELECT story_id FROM chapters WHERE id = ?)",
(changed_at, chapter_id),
)
conn.execute("UPDATE chapters SET current_rev = ?, changed_at = ? WHERE id = ?", (rev, now(), chapter_id))
return int(rev)
def update_chapter(self, chapter_id: int, **fields: Any) -> None:
@@ -329,11 +360,27 @@ class Store:
(*fields.values(), chapter_id),
)
def set_positions(self, story_id: int, chapter_ids: list[int]) -> None:
def reorder_chapters(self, story_id: int, arrange: Callable[[list[int]], list[int]]) -> None:
"""Give the chapters of a story a new order.
`arrange` gets the chapter ids in book order, and returns them in the new
order. The read and the write are one transaction, so a job that adds a
chapter at the same time cannot come between them.
"""
with self._connect() as conn:
conn.execute("BEGIN IMMEDIATE")
rows = conn.execute("SELECT id FROM chapters WHERE story_id = ? ORDER BY position, id", (story_id,))
order = arrange([r["id"] for r in rows])
conn.executemany(
"UPDATE chapters SET position = ? WHERE id = ? AND story_id = ?",
[(i, cid, story_id) for i, cid in enumerate(chapter_ids)],
[(i, cid, story_id) for i, cid in enumerate(order)],
)
def delete_chapters(self, story_id: int, chapter_ids: list[int]) -> None:
"""Delete chapters of the story and their revisions."""
with self._connect() as conn:
conn.executemany(
"DELETE FROM chapters WHERE id = ? AND story_id = ?", [(cid, story_id) for cid in chapter_ids]
)
def list_revisions(self, chapter_id: int) -> list[dict[str, Any]]:
@@ -372,6 +419,14 @@ class Store:
rows = conn.execute("SELECT url FROM images WHERE story_id = ? AND is_cover = 0", (story_id,))
return {r["url"] for r in rows}
def image_sizes(self, story_id: int) -> list[list[Any]]:
"""The URL, the cover mark and the size of each image, without its data."""
with self._connect() as conn:
rows = conn.execute(
"SELECT url, is_cover, LENGTH(data) FROM images WHERE story_id = ? ORDER BY url, is_cover", (story_id,)
)
return [list(r) for r in rows]
def images(self, story_id: int) -> list[dict[str, Any]]:
with self._connect() as conn:
rows = conn.execute(
bindery/web.py
+143 -49
View File
@@ -2,7 +2,6 @@
import base64
import hashlib
import io
import logging
import os
import re
@@ -13,10 +12,10 @@ from datetime import timedelta
from pathlib import Path
from typing import Any
from flask import Flask, Response, g, jsonify, request, send_file, session
from werkzeug.exceptions import HTTPException
from flask import Flask, Response, g, jsonify, request, session
from werkzeug.exceptions import Forbidden, HTTPException
from bindery import auth, config, orchestrator
from bindery import auth, config, opds, orchestrator
from bindery.cwa import CwaError
from bindery.epub import EpubError
from bindery.fff import StoryError
@@ -26,8 +25,8 @@ from bindery.rules import RuleError
logging.basicConfig(level=config.LOG_LEVEL, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
logger = logging.getLogger(__name__)
# API paths that need no login.
PUBLIC_PATHS = {"/api/health", "/api/auth/check", "/api/auth/login", "/api/auth/logout"}
# API paths that need no login. The login gate also lets /api/health through, before it reads a setting.
PUBLIC_PATHS = {"/api/auth/check", "/api/auth/login", "/api/auth/logout"}
# The browser sends this header with every change. Another site cannot send it
# without a CORS preflight, which Bindery never allows, so it stops cross-site requests.
CSRF_HEADER = "X-Requested-With"
@@ -69,6 +68,7 @@ def _secret_key() -> bytes:
config.prepare()
app = Flask(__name__)
app.register_blueprint(opds.bp)
app.config.update(
SECRET_KEY=_secret_key(),
SEND_FILE_MAX_AGE_DEFAULT=0,
@@ -92,20 +92,83 @@ def _json() -> dict[str, Any]:
return data if isinstance(data, dict) else {}
# Values of a request body. A value of the wrong type is an error of the request, so it gets a 400.
def _list(data: dict[str, Any], name: str) -> list[Any]:
value = data.get(name)
if value is None:
return []
if not isinstance(value, list):
raise ValueError(f"{name} must be a list")
return value
def _int(value: Any, name: str) -> int:
return orchestrator.whole_number(value, name, negative=True)
def _ids(data: dict[str, Any], name: str = "chapter_ids") -> list[int]:
return [_int(v, name) for v in _list(data, name)]
def _rules(data: dict[str, Any]) -> list[dict[str, Any]]:
rules = _list(data, "rules")
if not all(isinstance(r, dict) for r in rules):
raise ValueError("rules must be a list of rules")
return rules
def _names(data: dict[str, Any], name: str) -> list[str] | None:
"""A list of text values, or None if the request leaves it out."""
if data.get(name) is None:
return None
values = _list(data, name)
if not all(isinstance(v, str) for v in values):
raise ValueError(f"{name} must be a list of names")
return values
# Login gate and errors
def _identity() -> auth.Identity | None:
return auth.current_identity(request.headers, session)
return auth.current_identity(request.headers, session, request.remote_addr)
def _admin_only() -> None:
if not auth.is_admin(g.identity):
raise Forbidden("Only an admin can do this. BINDERY_ADMIN_USERS lists the admins.")
@app.before_request
def check_host() -> Any:
if request.path == "/api/health" or auth.host_allowed(request.host):
return None
return jsonify({"error": f"Bindery does not answer to the host name {request.host}. Add it to ALLOWED_HOSTS."}), 400
def _catalog_login() -> Any:
"""The login of the OPDS catalog. It takes no API key. See `auth.catalog_login`."""
if not orchestrator.store.get_setting(None, "opds_enabled"):
return jsonify({"error": "Not found"}), 404
address = auth.client_address(request.headers, request.remote_addr)
if auth.catalog_login(request.headers, session, address, orchestrator.opds_password_hash, orchestrator.cwa):
return None
# The header makes a reader ask for the user name and password.
return jsonify({"error": "Unauthorized"}), 401, {"WWW-Authenticate": 'Basic realm="Bindery", charset="UTF-8"'}
@app.before_request
def check_login() -> Any:
if not request.path.startswith("/api/") or request.path == "/api/health":
catalog = request.path.startswith("/opds/") or request.path == "/opds"
if not (catalog or request.path.startswith("/api/")) or request.path == "/api/health":
return None
error = auth.config_error()
if error:
return jsonify({"error": f"{error}. Fix the setting and restart Bindery."}), 503
if catalog:
return _catalog_login()
identity = _identity()
if (
request.method in MUTATING
@@ -151,8 +214,9 @@ def bad_request(e: Exception) -> Any:
def server_error(e: Exception) -> Any:
if isinstance(e, HTTPException):
return {"error": e.description}, e.code
# The text of an unexpected error can hold paths and addresses, so it goes only to the log.
logger.exception(f"Error in {request.path}: {e}")
return {"error": f"{type(e).__name__}: {e}"}, 500
return {"error": "Bindery hit an unexpected error. The log has the details."}, 500
# Login
@@ -160,17 +224,21 @@ def server_error(e: Exception) -> Any:
@app.get("/api/health")
def health() -> Any:
stopped = orchestrator.stopped_threads()
if stopped:
return {"status": "error", "error": f"Stopped: {', '.join(stopped)}. Restart Bindery."}, 503
return {"status": "ok"}
@app.get("/api/auth/check")
def auth_check() -> Any:
identity = _identity()
mode = auth.active_mode()
mode = config.AUTH_METHOD
return {
"authenticated": identity is not None,
"mode": mode,
"user": identity.name if identity else None,
"admin": identity is not None and auth.is_admin(identity),
"logout_url": config.PROXY_AUTH_LOGOUT_URL or None if mode == auth.AUTH_PROXY else None,
}
@@ -182,7 +250,7 @@ def auth_login() -> Any:
username = str(data.get("username", ""))
password = str(data.get("password", ""))
address = auth.client_address(request.headers, request.remote_addr)
if auth.active_mode() == auth.AUTH_LOCAL:
if config.AUTH_METHOD == auth.AUTH_LOCAL:
identity = auth.login_local(username, password, address)
else:
identity = auth.login_cwa(username, password, orchestrator.cwa(), address)
@@ -190,8 +258,8 @@ def auth_login() -> Any:
return {"error": str(e)}, 401
session.clear()
session["user"] = identity.name
session["mode"] = auth.active_mode()
if auth.active_mode() == auth.AUTH_LOCAL:
session["mode"] = config.AUTH_METHOD
if config.AUTH_METHOD == auth.AUTH_LOCAL:
session["local_password_version"] = auth.local_password_version()
session.permanent = bool(data.get("remember"))
logger.info(f"{identity.name} logged in")
@@ -210,9 +278,12 @@ def app_config() -> Any:
"metadata_fields": METADATA_FIELDS,
"mode_labels": MODE_LABELS,
"library_available": orchestrator.library_available(),
"cwa_enabled": config.CWA_ENABLED,
"update_interval": config.UPDATE_INTERVAL_HOURS,
"job_list_max_hours": config.JOB_LIST_MAX_HOURS,
"job_list_max_count": config.JOB_LIST_MAX_COUNT,
"download_fields": orchestrator.download_fields(),
"auth_mode": config.AUTH_METHOD,
**orchestrator.job_list_settings(),
}
@@ -220,18 +291,31 @@ def app_config() -> Any:
@app.route("/api/settings", methods=["GET", "PUT"])
def settings() -> Any:
if request.method == "PUT":
_admin_only()
data = _json()
return orchestrator.save_settings(
data.get("recent_chapters"),
data.get("personal_ini"),
data.get("hold_new_links"),
data.get("pause_uploads"),
data.get("story_jobs_hours"),
data.get("story_jobs_count"),
data.get("main_jobs_hours"),
data.get("main_jobs_count"),
)
return orchestrator.get_settings()
return _with_opds_user(orchestrator.save_settings(**{name: data.get(name) for name in orchestrator.SETTINGS}))
settings = _with_opds_user(orchestrator.get_settings())
if not auth.is_admin(g.identity):
del settings["personal_ini"] # it can hold passwords for story sites
return settings
def _with_opds_user(settings: dict[str, Any]) -> dict[str, Any]:
"""Adds the user name that the catalog expects from this user, and whether this user has an OPDS password."""
user = g.identity.name if config.AUTH_METHOD == auth.AUTH_PROXY and g.identity.via == "proxy" else None
return {**settings, "opds_user": user, "opds_password_set": bool(user and orchestrator.opds_password_hash(user))}
@app.put("/api/settings/opds-password")
def opds_password() -> Any:
"""Each user in the proxy mode sets their own OPDS password. The user name is the one from the proxy."""
identity = g.identity
if config.AUTH_METHOD != auth.AUTH_PROXY or identity.via != "proxy" or not identity.name:
raise ValueError("The OPDS password is for users who log in through the proxy")
orchestrator.set_opds_password(identity.name, _json().get("password"))
# Tries before the password existed must not lock the user out of the new password.
auth.limiter.forget(identity.name)
return {"opds_user": identity.name, "opds_password_set": True}
# Jobs
@@ -257,9 +341,8 @@ def stories() -> Any:
@app.post("/api/stories")
def add_stories() -> Any:
urls = _json().get("urls", [])
if isinstance(urls, str):
urls = re.split(r"\s+", urls)
data = _json()
urls = re.split(r"\s+", data["urls"]) if isinstance(data.get("urls"), str) else _list(data, "urls")
queued, errors = orchestrator.queue_urls([str(u) for u in urls])
return {"jobs": queued, "errors": errors}
@@ -272,7 +355,7 @@ def import_epub(story_id: int | None = None) -> Any:
raise ValueError("Choose an epub file")
path = config.UPLOAD_DIR / f"{uuid.uuid4().hex}.epub"
upload.save(path)
check = request.form.get("check", "true").lower() not in ("0", "false", "no", "off")
check = config.flag(request.form.get("check", ""), True)
return orchestrator.queue_epub_file(path, os.path.basename(upload.filename), story_id, check)
@@ -290,18 +373,23 @@ def delete_story(story_id: int) -> Any:
@app.post("/api/stories/<int:story_id>/download")
def download(story_id: int) -> Any:
data = _json()
# A rebuild that the user asks for also builds and sends a book that did not change.
return orchestrator.queue_story(
story_id, data.get("mode", DownloadMode.NEW.value), data.get("chapter_ids"), data.get("force_fields")
story_id, str(data.get("mode", DownloadMode.NEW.value)), _ids(data), _names(data, "force_fields"), True
)
@app.put("/api/stories/<int:story_id>/settings")
def story_settings(story_id: int) -> Any:
data = _json()
days = data.get("recheck_days")
orchestrator.update_story_settings(
story_id, data.get("locked_fields"), data.get("auto_update"), data.get("recheck_days")
story_id,
_names(data, "locked_fields"),
data.get("auto_update"),
None if days is None else _int(days, "recheck_days"),
)
return orchestrator.story_details(story_id)["story"]
return orchestrator.story_view(story_id)
@app.put("/api/stories/<int:story_id>/metadata")
@@ -315,37 +403,41 @@ def edit_metadata(story_id: int) -> Any:
@app.post("/api/stories/<int:story_id>/link")
def link(story_id: int) -> Any:
orchestrator.link_story(story_id, _json().get("uuid") or None)
return orchestrator.story_details(story_id)["story"]
return orchestrator.story_view(story_id)
@app.post("/api/stories/<int:story_id>/approve")
def approve_upload(story_id: int) -> Any:
job = orchestrator.approve_upload(story_id, _json().get("uuid") or None)
return {"story": orchestrator.story_details(story_id)["story"], "job": job}
return {"story": orchestrator.story_view(story_id), "job": job}
@app.post("/api/stories/<int:story_id>/chapters")
def chapters_included(story_id: int) -> Any:
data = _json()
return orchestrator.set_chapters_included(
story_id, [int(c) for c in data.get("chapter_ids", [])], bool(data.get("included", True))
)
return orchestrator.set_chapters_included(story_id, _ids(data), bool(data.get("included", True)))
@app.post("/api/stories/<int:story_id>/chapters/move")
def move_chapters(story_id: int) -> Any:
data = _json()
after = data.get("after")
return orchestrator.move_chapters(story_id, _ids(data), None if after is None else _int(after, "after"))
@app.post("/api/stories/<int:story_id>/chapters/delete")
def delete_chapters(story_id: int) -> Any:
return orchestrator.delete_chapters(story_id, _ids(_json()))
@app.get("/api/stories/<int:story_id>/epub")
def story_epub(story_id: int) -> Any:
path, title = orchestrator.build_for_download(story_id)
name = re.sub(r'[\\/:*?"<>|]', "_", title.strip())[:200] or "story"
return send_file(path, download_name=f"{name}.epub", as_attachment=True)
return opds.send_epub(story_id)
@app.get("/api/stories/<int:story_id>/cover")
def story_cover(story_id: int) -> Any:
cover = orchestrator.story_cover(story_id)
if not cover:
raise KeyError("This story has no cover")
# The page puts the cover version in the URL, so the browser can keep the image for a year.
return send_file(io.BytesIO(cover["data"]), mimetype=cover["mime"], max_age=365 * 24 * 60 * 60)
return opds.send_cover(story_id)
# Replacement rules. Story rules run after the global rules.
@@ -355,8 +447,10 @@ def story_cover(story_id: int) -> Any:
@app.route("/api/rules", methods=["GET", "PUT"])
def rules(story_id: int | None = None) -> Any:
if request.method == "PUT":
if story_id is None:
_admin_only()
data = _json()
return orchestrator.save_rules(story_id, data.get("rules", []), data.get("paragraphs", ...))
return orchestrator.save_rules(story_id, _rules(data), data.get("paragraphs", ...))
return orchestrator.get_rules(story_id)
@@ -366,8 +460,8 @@ def rules_preview(story_id: int | None = None) -> Any:
data = _json()
return orchestrator.preview_rules(
story_id,
data.get("rules", []),
int(data.get("chapter_id", 0)),
_rules(data),
_int(data.get("chapter_id", 0), "chapter_id"),
bool(data.get("source")),
data.get("paragraphs"),
)
@@ -385,7 +479,7 @@ def chapter_diff(chapter_id: int) -> Any:
@app.post("/api/chapters/<int:chapter_id>/revision")
def chapter_revision(chapter_id: int) -> Any:
return orchestrator.set_chapter_revision(chapter_id, int(_json().get("rev", 0)))
return orchestrator.set_chapter_revision(chapter_id, _int(_json().get("rev", 0), "rev"))
@app.delete("/api/chapters/<int:chapter_id>/revisions/<int:rev>")
@@ -409,7 +503,7 @@ def library_search() -> Any:
@app.post("/api/library/import")
def library_import() -> Any:
data = _json()
books = [b for b in data.get("books", []) if isinstance(b, dict) and "id" in b]
books = [{**b, "id": _int(b["id"], "id")} for b in _list(data, "books") if isinstance(b, dict) and "id" in b]
return orchestrator.queue_import(books, bool(data.get("check", True)))
data/personal.ini
+1 -1
View File
@@ -1,4 +1,4 @@
## FanFicFare configuration for CWA Bindery.
## FanFicFare configuration for Bindery.
##
## This file has the same format as personal.ini in the FanFicFare calibre
## plugin and CLI. You can copy settings from your plugin configuration.
docker-compose.yml
+26 -13
View File
@@ -2,26 +2,39 @@ services:
bindery:
build: .
image: bindery:latest
# The user and group that own ./config and can write CWA's ingest folder. Usually the same as CWA's.
# The user and group that own ./config.
user: "1000:1000"
environment:
# The address of CWA, as the Bindery container reaches it, and the CWA account that Bindery uses.
CWA_URL: http://calibre-web-automated:8083
CWA_USERNAME: bindery
CWA_PASSWORD: change-me
# none, local (one Bindery account), cwa (CWA accounts), or proxy (a trusted user header)
AUTH_METHOD: cwa
AUTH_METHOD: none
# For local logins, put these values in .env. See "Local login" in the README.
# BINDERY_USERNAME: ${BINDERY_USERNAME:-}
# BINDERY_PASSWORD_HASH: ${BINDERY_PASSWORD_HASH:-}
# URL_BASE: /bindery
# The host names that Bindery answers to. AUTH_METHOD=none needs this for a public name such as this one.
# ALLOWED_HOSTS: bindery.example.com
# true only if clients reach Bindery through a reverse proxy alone. The login lockout then uses X-Forwarded-For.
# TRUST_PROXY: "true"
# For AUTH_METHOD=proxy: the address of the proxy. The default trusts each private address.
# PROXY_AUTH_TRUSTED_IPS: 172.18.0.3
# For the cwa and proxy modes: the users that can change the global settings. Without a list, each user can.
# BINDERY_ADMIN_USERS: alice, bob
# BINDERY_API_KEY: a-long-random-string
# Optional: a folder that gets each built ePub. With CWA, this is CWA's ingest folder.
# INGEST_DIR: /cwa-book-ingest
# Optional: Calibre-Web-Automated. The user above must then be able to write CWA's ingest folder.
# Usually it is the same as CWA's.
# CWA_ENABLED: "true"
# The address of CWA, as the Bindery container reaches it, and the CWA account that Bindery uses.
# Leave the account out if CWA asks for no login.
# CWA_URL: http://calibre-web-automated:8083
# CWA_USERNAME: bindery
# CWA_PASSWORD: change-me
# Optional and independent. app.db is for logins, metadata.db is for book IDs.
# CWA_DB_PATH: /cwa-config/app.db
# CALIBRE_DB_PATH: /calibre-library/metadata.db
# URL_BASE: /bindery
# true only if clients reach Bindery through a reverse proxy alone. The login lockout then uses X-Forwarded-For.
# TRUST_PROXY: "true"
# BINDERY_API_KEY: a-long-random-string
ports:
# If a reverse proxy on this host is the only way in, publish the port only on the host: 127.0.0.1:8085:8085
- 8085:8085
restart: unless-stopped
# /api/health needs no login, and works with and without URL_BASE.
@@ -32,11 +45,11 @@ services:
start_period: 30s
retries: 3
volumes:
# CWA's ingest folder. New stories and updates go here.
- /path/to/cwa/ingest:/cwa-book-ingest
# The story database and personal.ini. Create the folder before the first start.
- ./config:/config
# Optional read-only folders. Include SQLite journal files beside each database.
# Optional: the folder of INGEST_DIR. With CWA, CWA's ingest folder. Bindery does not make the folder.
# - /path/to/cwa/ingest:/cwa-book-ingest
# Optional read-only folders of CWA. Include SQLite journal files beside each database.
# Mount the library that receives the ingest files, not a copy or another library.
# - /path/to/cwa/config:/cwa-config:ro
# - /path/to/calibre/library:/calibre-library:ro
scripts/import_unlinked_epub_chapters.py
+458
View File
@@ -0,0 +1,458 @@
#!/usr/bin/env python3
"""Fill a gap in a tracked story with chapters from an ePub that has no chapter links.
Bindery can import chapters from an ePub that FanFicFare made, because that
ePub records the site address (the link) of each chapter. Other ePubs, such as
the author's own ebook of the story, have no chapter links, so Bindery cannot
tell where their chapters go. Use this script when the site no longer has some
chapters, and you have an ePub without chapter links that has them.
Usage:
CONFIG_DIR=./config uv run python scripts/import_unlinked_epub_chapters.py <story-url> <path-to-epub>
CONFIG_DIR=./config uv run python scripts/import_unlinked_epub_chapters.py <story-url> --remove
Run the script on the host, in the project folder, with CONFIG_DIR set to the
folder that holds bindery.db. The container image does not include the
scripts folder. Stop Bindery before you run the script, so that nothing else
writes to the database at the same time.
The story must be tracked in Bindery, with at least one stored chapter. The
script asks four questions:
1. Which stored chapter comes just before the gap. Type "start" if the gap
is at the start of the story.
2. Which ePub chapter is the same as that stored chapter. After "start", the
question is which ePub chapter to add first, so that you can leave out the
cover and the title pages.
3. Which stored chapter comes just after the gap. Type "end" if the gap is at
the end of the story.
4. Which ePub chapter is the same as that stored chapter. After "end", the
question is which ePub chapter to add last.
The script suggests an answer where the chapter titles match, and puts it on
the input line. Press Enter to accept it, or change it. Type "list" to see all
the chapters again. The script then shows the ePub chapters that go into the
gap, in the order of the ePub, and adds them after you say yes. Their images
come with them. A second run with the same ePub does not add the same chapters
again.
The added chapters have no address on the site, so they show as "removed from
site" on the story page. A "full" download takes them out of the book, like
other chapters that the site does not list. Select them and use "Put back in
book" to put them back. To delete them, use "Delete…" on the story page, or run
the script with --remove. Rebuild the book after each change.
"""
import argparse
import hashlib
import posixpath
import re
import sqlite3
import sys
import warnings
import zipfile
from difflib import SequenceMatcher
from pathlib import Path
from bs4 import BeautifulSoup, XMLParsedAsHTMLWarning
from defusedxml import ElementTree
try:
import readline
except ImportError: # Windows
readline = None
# Run as a file, Python puts scripts/ on the path, not the project folder that holds bindery/.
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from fanficfare.story import imagetypes # noqa: E402
from bindery import config, fff # noqa: E402
from bindery.diffing import text_hash # noqa: E402
from bindery.store import Store # noqa: E402
OPF = "{http://www.idpf.org/2007/opf}"
NCX = "{http://www.daisy.org/z3986/2005/ncx/}"
DC = "{http://purl.org/dc/elements/1.1/}"
CONTAINER = "{urn:oasis:names:tc:opendocument:xmlns:container}"
URL_MARK = "#bindery-import-"
SOURCE = "epub-import"
# Images get a file: URL, so a build never asks the story site for them.
IMAGE_URL = "file:///bindery-epub-import/"
# A long list shows only this many chapters around the one that the question is about.
WINDOW = 12
warnings.filterwarnings("ignore", category=XMLParsedAsHTMLWarning)
class ScriptError(Exception):
"""A problem the user can fix."""
def _resolve(folder: str, href: str) -> str:
"""The path in the zip file of a link from a file in `folder`, without its #fragment."""
return posixpath.normpath(posixpath.join(folder, href.split("#", 1)[0]))
def _toc_titles(z: zipfile.ZipFile, opf, folder: str) -> dict[str, str]:
"""The title of each file in the ePub's table of contents (EPUB 2 NCX or EPUB 3 nav), by path.
The first entry of a file wins. Entries that are only a number, such as page numbers, are left out.
"""
items = list(opf.find(f"{OPF}manifest"))
titles: dict[str, str] = {}
def add(path: str, label: str) -> None:
label = " ".join(label.split())
if label and not label.isdigit():
titles.setdefault(path, label)
ncx = next((i for i in items if i.get("media-type") == "application/x-dtbncx+xml"), None)
if ncx is not None:
ncx_path = _resolve(folder, ncx.get("href"))
nav_map = ElementTree.fromstring(z.read(ncx_path)).find(f"{NCX}navMap")
for point in nav_map.iter(f"{NCX}navPoint") if nav_map is not None else []:
text, content = point.find(f"{NCX}navLabel/{NCX}text"), point.find(f"{NCX}content")
if text is not None and content is not None and content.get("src"):
add(_resolve(posixpath.dirname(ncx_path), content.get("src")), text.text or "")
nav = next((i for i in items if "nav" in (i.get("properties") or "").split()), None)
if nav is not None:
nav_path = _resolve(folder, nav.get("href"))
soup = BeautifulSoup(z.read(nav_path), "html5lib")
toc = soup.find("nav", attrs={"epub:type": "toc"}) or soup.find("nav")
for a in toc.find_all("a", href=True) if toc else []:
add(_resolve(posixpath.dirname(nav_path), a["href"]), a.get_text(" "))
return titles
def _only_links(soup: BeautifulSoup) -> bool:
"""Whether all the text of a file is in links, as in a table of contents."""
texts = [t for t in soup.body.find_all(string=True) if t.strip()] if soup.body else []
return bool(texts) and all(t.find_parent("a") for t in texts)
def _book_id(opf, path: Path) -> str:
"""A short ID of the ePub, from its identifier, or else its file name."""
identifier = opf.find(f"{OPF}metadata/{DC}identifier")
key = identifier.text.strip() if identifier is not None and identifier.text else path.name
return hashlib.sha256(key.encode("utf-8")).hexdigest()[:8]
def _take_images(z: zipfile.ZipFile, soup: BeautifulSoup, folder: str, book: str) -> list[dict]:
"""Point each <img> of a chapter at a URL of its own, and return the images with those URLs.
An image that the ePub does not have, or that FanFicFare cannot use, is left out.
"""
images = []
for img in soup.body.find_all("img", src=True):
if img["src"].startswith(("data:", "http:", "https:")):
continue
path = _resolve(folder, img["src"])
ext = path.rsplit(".", 1)[-1].lower()
if ext in imagetypes and path in z.namelist():
img["src"] = f"{IMAGE_URL}{book}/{path}"
images.append({"url": img["src"], "ext": ext, "mime": imagetypes[ext], "data": z.read(path)})
else:
img.decompose()
return images
def read_epub(path: Path) -> list[dict]:
"""Read the chapters of an ePub, in reading order, with the images that each one uses.
The script reads the OPF manifest and spine directly, so it can read any
ePub. A file that has only links, such as a table of contents, is not a
chapter. Titles come from the table of contents of the ePub, or else from
the first heading of the chapter. Each chapter has a stable `key`: the ID
of the ePub and the path of the chapter in it.
"""
try:
with zipfile.ZipFile(path) as z:
container = ElementTree.fromstring(z.read("META-INF/container.xml"))
rootfile = container.find(f".//{CONTAINER}rootfile")
if rootfile is None or not rootfile.get("full-path"):
raise ScriptError(f"{path.name} has no OPF file")
opf_path = rootfile.get("full-path")
opf = ElementTree.fromstring(z.read(opf_path))
folder = posixpath.dirname(opf_path)
manifest = {item.get("id"): item for item in opf.find(f"{OPF}manifest")}
toc = _toc_titles(z, opf, folder)
book = _book_id(opf, path)
chapters = []
for itemref in opf.find(f"{OPF}spine").findall(f"{OPF}itemref"):
item = manifest.get(itemref.get("idref"))
if itemref.get("linear") == "no" or item is None:
continue
if "nav" in (item.get("properties") or "").split():
continue
href = item.get("href") or ""
if not href.lower().endswith((".xhtml", ".html", ".htm")):
continue
full = _resolve(folder, href)
soup = BeautifulSoup(z.read(full), "html5lib")
if not soup.body or _only_links(soup):
continue
heading = soup.find(["h1", "h2", "h3"])
if full in toc:
title = toc[full]
elif heading and heading.get_text(strip=True):
title = heading.get_text(" ", strip=True)
elif soup.title and soup.title.get_text(strip=True):
title = soup.title.get_text(strip=True)
else:
title = href
images = _take_images(z, soup, posixpath.dirname(full), book)
html = "".join(str(c) for c in soup.body.contents).strip()
if html:
chapters.append({"title": title, "html": html, "key": f"{book}-{full}", "images": images})
except (zipfile.BadZipFile, KeyError, ElementTree.ParseError) as e:
raise ScriptError(f"{path.name} is not a readable ePub: {e}") from e
if not chapters:
raise ScriptError(f"{path.name} has no chapters that the script can read")
return chapters
def suggest_matches(title: str, candidates: list[str], limit: int = 3) -> list[int]:
"""Indexes of the candidates most like `title`, best first."""
ratio = {i: SequenceMatcher(None, title.lower(), c.lower()).ratio() for i, c in enumerate(candidates)}
return sorted((i for i in ratio if ratio[i] > 0.4), key=lambda i: ratio[i], reverse=True)[:limit]
def chapter_url(story_url: str, chapter: dict) -> str:
"""A made-up, stable URL for a chapter that has no link. The same ePub chapter always gets the same URL."""
slug = re.sub(r"[^a-z0-9]+", "-", chapter["key"].lower()).strip("-")
return f"{story_url.rstrip('/')}{URL_MARK}{slug}"
def splice_order(existing_urls: list[str], before_idx: int, new_urls: list[str]) -> list[str]:
"""`existing_urls` with `new_urls` put directly after position `before_idx` (-1 for the start)."""
return existing_urls[: before_idx + 1] + new_urls + existing_urls[before_idx + 1 :]
def _show(items: list[str], marks: set[int] = frozenset(), around: int | None = None) -> None:
"""Print a numbered list. With `around`, a long list shows only the part near that index."""
first, last = 0, len(items)
if around is not None and len(items) > WINDOW * 2:
first = max(0, min(around - WINDOW // 2, len(items) - WINDOW))
last = first + WINDOW
# The numbers go in brackets, padded so that the titles line up. A suggestion has a * in front.
width = len(str(len(items))) + 2
if first:
print(" …")
for i in range(first, last):
print(f"{'*' if i in marks else ' '} {f'[{i + 1}]':<{width}} {items[i]}")
if last < len(items):
print(f" … (type 'list' to see all {len(items)})")
def _ask(prompt: str, default: str) -> str:
"""Read a line. The default is already on the input line, where the terminal allows it."""
prefill = bool(default) and readline is not None and sys.stdin.isatty()
if prefill:
readline.set_startup_hook(lambda: readline.insert_text(default))
elif default:
prompt = f"{prompt[:-2]} [{default}]: "
try:
answer = input(prompt).strip()
finally:
if prefill:
readline.set_startup_hook()
return answer or default
def choose(
prompt: str,
items: list[str],
*,
word: str | None = None,
default: int | None = None,
marks: set[int] = frozenset(),
around: int | None = None,
minimum: int = 0,
) -> int | str:
"""Show `items` and ask for one of them. Returns its index, or `word` if the user typed it.
The prompt says what the question is, so the user does not have to look above a long list for it.
"""
_show(items, marks, around)
words = ["'list'"] + ([f"'{word}'"] if word else [])
while True:
answer = _ask(f"{prompt}: ", "" if default is None else str(default + 1)).lower()
if answer == "list":
_show(items, marks)
elif word and answer == word:
return word
elif answer.isdigit() and minimum < int(answer) <= len(items):
return int(answer) - 1
else:
print(f"Type a number from {minimum + 1} to {len(items)}, or {' or '.join(words)}.")
def _confirm(prompt: str) -> bool:
return input(f"{prompt} [y/N] ").strip().lower() == "y"
def _added_by_script(url: str) -> bool:
return URL_MARK in url
def remove_added_chapters(db_store: Store, story: dict, existing: list[dict]) -> None:
added = [c for c in existing if _added_by_script(c["url"])]
if not added:
print(f"{story['title']} has no chapters that this script added.")
return
print(f"{story['title']} has {_count(len(added), 'chapter')} that this script added:")
for c in added:
print(f" - {fff.unescape_title(c['title'])}")
if not _confirm("Delete them and all their revisions?"):
return
conn = sqlite3.connect(config.DB_PATH)
try:
# Without this, SQLite keeps the revisions of a deleted chapter.
conn.execute("PRAGMA foreign_keys=ON")
with conn:
conn.executemany("DELETE FROM chapters WHERE id = ?", [(c["id"],) for c in added])
finally:
conn.close()
print(f"Deleted {_count(len(added), 'chapter')}. Rebuild {story['title']} in Bindery.")
def _count(n: int, word: str) -> str:
return f"{n} {word}{'' if n == 1 else 's'}"
def ask_for_gap(
story_title: str, titles: list[str], labels: list[str], epub_name: str, epub_titles: list[str]
) -> tuple[int, int, int]:
"""Ask where the gap is. Returns the index of the stored chapter before the gap (-1 for the start),
and the indexes of the first and last ePub chapters to add.
`titles` are the stored chapter titles, and `labels` are the same titles as the lists show them.
"""
print(f"\n{story_title} has {_count(len(titles), 'stored chapter')}:")
before = choose("Stored chapter just BEFORE the gap (or 'start')", labels, word="start")
print(f"\n{epub_name} has {_count(len(epub_titles), 'chapter')}:")
if before == "start":
before = -1
first = choose("FIRST ePub chapter to add", epub_titles)
else:
guess = suggest_matches(titles[before], epub_titles)
match = choose(
f'ePub chapter that matches "{titles[before]}"',
epub_titles,
default=guess[0] if guess else None,
marks=set(guess),
)
first = match + 1
print(f"\n{story_title}, near the gap:")
if before + 1 < len(titles):
after = choose(
"Stored chapter just AFTER the gap (or 'end')",
labels,
word="end",
default=before + 1,
around=before + 1,
minimum=before + 1,
)
else:
print(f' None. The gap is after the last stored chapter, "{titles[before]}".')
after = "end"
print(f"\n{epub_name}, near the gap:")
if after == "end":
last = choose(
"LAST ePub chapter to add", epub_titles, default=len(epub_titles) - 1, around=first, minimum=first
)
else:
if after != before + 1:
skipped = _count(after - before - 1, "stored chapter")
print(f"Note: the {skipped} between those two stay where they are, after the added chapters.")
later = epub_titles[first:]
guess = [first + i for i in suggest_matches(titles[after], later)]
match = choose(
f'ePub chapter that matches "{titles[after]}"',
epub_titles,
default=guess[0] if guess else None,
marks=set(guess),
around=guess[0] if guess else first,
minimum=first,
)
last = match - 1
return before, first, last
def main(argv: list[str] | None = None) -> None:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("story_url", help="The URL of the tracked story")
parser.add_argument("epub", type=Path, nargs="?", help="The ePub that has the missing chapters")
parser.add_argument("--remove", action="store_true", help="Delete the chapters that this script added")
args = parser.parse_args(argv)
normal = fff.normalize_url(args.story_url)
if not normal:
raise ScriptError(f"Not a supported story URL: {args.story_url}")
if not args.remove and (args.epub is None or not args.epub.is_file()):
raise ScriptError(f"No such file: {args.epub}" if args.epub else "Give the path to the ePub.")
db_store = Store(config.DB_PATH)
story = db_store.get_story_by_url(normal[0])
if not story:
raise ScriptError(f"No tracked story at {normal[0]}. Add it in Bindery first.")
existing = db_store.list_chapters(story["id"])
if args.remove:
remove_added_chapters(db_store, story, existing)
return
if not existing:
raise ScriptError(f"{story['title']} has no stored chapters yet. Download it in Bindery first.")
epub_chapters = read_epub(args.epub)
titles = [fff.unescape_title(c["title"]) for c in existing]
labels = [t + ("" if c["on_site"] else " (not on site)") for t, c in zip(titles, existing, strict=True)]
epub_titles = [c["title"] for c in epub_chapters]
before, first, last = ask_for_gap(story["title"], titles, labels, args.epub.name, epub_titles)
stored_urls = {c["url"] for c in existing}
to_add = [c for c in epub_chapters[first : last + 1] if chapter_url(story["url"], c) not in stored_urls]
already = last + 1 - first - len(to_add)
if already:
print(f"\n{_count(already, 'ePub chapter')} of the gap are in the story already, from an earlier run.")
if not to_add:
print("There are no chapters to add.")
return
where = "at the start" if before == -1 else f'after "{titles[before]}"'
print(f"\nThe script will add {'this chapter' if len(to_add) == 1 else f'these {len(to_add)} chapters'} {where}:")
for c in to_add:
print(f" - {c['title']}")
if not _confirm("Add it?" if len(to_add) == 1 else "Add them?"):
return
new_chapters = [
{
"url": chapter_url(story["url"], c),
"title": fff.escape_title(c["title"]),
"html": c["html"],
"text_hash": text_hash(c["html"]),
}
for c in to_add
]
for img in {img["url"]: img for c in to_add for img in c["images"]}.values():
db_store.put_image(story["id"], img["url"], img["ext"], img["mime"], img["data"])
order = splice_order([c["url"] for c in existing], before, [c["url"] for c in new_chapters])
for chapter_id in db_store.add_chapters(story["id"], new_chapters, SOURCE, order):
# The site does not list these chapters. The next download would set this too.
db_store.update_chapter(chapter_id, on_site=0)
print(f"\nAdded {_count(len(new_chapters), 'chapter')} to {story['title']}. Rebuild the book in Bindery.")
if __name__ == "__main__":
try:
main()
except ScriptError as e:
print(f"Error: {e}", file=sys.stderr)
sys.exit(1)
except (KeyboardInterrupt, EOFError):
print("\nStopped. Nothing changed.")
sys.exit(1)
scripts/remove_smart_quote_revisions.py
+271
View File
@@ -0,0 +1,271 @@
#!/usr/bin/env python3
"""Clean up smart punctuation in chapter revisions: curly quotes, curly apostrophes and the ellipsis.
Usage:
python3 scripts/remove_smart_quote_revisions.py /path/to/config/bindery.db
python3 scripts/remove_smart_quote_revisions.py /path/to/config/bindery.db --apply
The first command only lists what it would do. With --apply, the script copies
the database to a backup file next to it, and then changes the revisions. Stop
Bindery before you use --apply.
The script compares the text of the revisions of each chapter, as Bindery does,
so differences in the markup do not count. Smart punctuation is ‘ ’ “ ” and …,
and its straight form is ' " and ...
- If two or more revisions of a chapter differ only by smart punctuation, the
revision with the least smart punctuation stays, and the others go. If a
deleted revision was in the book, the revision that stays goes in the book.
- If a revision has text that no other revision of its chapter has, the script
keeps it, and changes its smart punctuation to straight. Only the text
changes, not the tags or comments. The script stores the new text hash, so
Bindery compares the next site text with the new text.
The script does not delete or change the newest revision of a chapter if it came
from the site. Each recheck compares the site text with the newest revision, so
the next recheck would add the site text again.
Rebuild the stories that the script names, because the text in their books
changed. The script uses only the Python standard library.
"""
import argparse
import hashlib
import re
import sqlite3
import sys
from collections import defaultdict
from datetime import datetime
from html import unescape
from html.parser import HTMLParser
from pathlib import Path
SMART = {"‘": "'", "’": "'", "“": '"', "”": '"', "…": "..."}
STRAIGHT = str.maketrans(SMART)
ENTITY = re.compile(r"&(?:#\d+|#[xX][0-9a-fA-F]+|[a-zA-Z]+);")
MARKUP = re.compile(r"(<!--.*?-->|<[^>]*>)", re.S)
IMPORT_SOURCES = {"import", "epub-import"}
# The same tags as BLOCK_TAGS in bindery/diffing.py.
BLOCK_TAGS = {
"p", "div", "h1", "h2", "h3", "h4", "h5", "h6", "li", "blockquote", "pre",
"tr", "table", "hr", "section", "article", "ul", "ol", "center",
} # fmt: skip
class _TextParser(HTMLParser):
def __init__(self) -> None:
super().__init__(convert_charrefs=True)
self.parts: list[str] = []
self.hidden = 0
def handle_starttag(self, tag: str, attrs: list) -> None:
if tag in ("script", "style"):
self.hidden += 1
elif tag == "br" or tag in BLOCK_TAGS:
self.parts.append("\n")
def handle_endtag(self, tag: str) -> None:
if tag in ("script", "style"):
self.hidden = max(0, self.hidden - 1)
elif tag in BLOCK_TAGS:
self.parts.append("\n")
def handle_data(self, data: str) -> None:
if not self.hidden:
self.parts.append(data)
def text_blocks(markup: str) -> list[str]:
"""The text of a chapter, one string per paragraph, as html_to_blocks in bindery/diffing.py makes it."""
parser = _TextParser()
parser.feed(markup or "")
parser.close()
lines = (re.sub(r"\s+", " ", line).strip() for line in "".join(parser.parts).split("\n"))
return [line for line in lines if line]
def text_hash(markup: str) -> str:
"""The hash of the text of a chapter, as text_hash in bindery/diffing.py makes it."""
return hashlib.sha256("\n".join(text_blocks(markup)).encode("utf-8")).hexdigest()
def smart_count(text: str) -> int:
return sum(text.count(c) for c in SMART)
def straighten_html(markup: str) -> str:
"""Change the smart punctuation in the text of a chapter to straight, also where the HTML has it as an entity.
Tags and comments do not change, so a straight quote cannot end an attribute value.
"""
def entity(match: re.Match) -> str:
return SMART.get(unescape(match.group(0)), match.group(0))
# With a group in the pattern, split() puts the tags and comments at the odd places.
parts = MARKUP.split(markup)
for i in range(0, len(parts), 2):
parts[i] = ENTITY.sub(entity, parts[i]).translate(STRAIGHT)
return "".join(parts)
def plan(conn: sqlite3.Connection) -> list[dict]:
"""For each chapter with revisions to change: what goes, what stays, what gets straight, and what is in the book."""
chapters = conn.execute(
"""
SELECT c.id, c.title, c.current_rev, s.id AS story_id, s.title AS story_title
FROM chapters c JOIN stories s ON s.id = c.story_id
WHERE (SELECT COUNT(*) FROM revisions r WHERE r.chapter_id = c.id) > 1
ORDER BY s.title COLLATE NOCASE, c.position"""
).fetchall()
actions = []
for chapter in chapters:
revisions = conn.execute(
"SELECT rev, source, html FROM revisions WHERE chapter_id = ? ORDER BY rev", (chapter["id"],)
).fetchall()
newest = revisions[-1]["rev"]
sources = {r["rev"]: r["source"] for r in revisions}
markup = {r["rev"]: r["html"] for r in revisions}
def from_site(rev: int, newest: int = newest, sources: dict = sources) -> bool:
return rev == newest and sources[rev] not in IMPORT_SOURCES
groups = defaultdict(list)
smart = {}
for r in revisions:
text = "\n".join(text_blocks(r["html"]))
smart[r["rev"]] = smart_count(text)
groups[text.translate(STRAIGHT)].append(r["rev"])
delete, keep, skipped = [], [], []
for group in groups.values():
least = min(smart[rev] for rev in group)
if all(smart[rev] == least for rev in group):
continue
keep += [rev for rev in group if smart[rev] == least]
for rev in group:
if smart[rev] > least:
(skipped if from_site(rev) else delete).append(rev)
straighten = []
if len(groups) > 1:
for group in groups.values():
left = [rev for rev in group if rev not in delete]
if len(left) == 1 and smart[left[0]] and left[0] not in skipped:
(skipped if from_site(left[0]) else straighten).append(left[0])
if not delete and not straighten and not skipped:
continue
current = chapter["current_rev"]
if current in delete:
# The revision that stays in the group of the deleted one.
same = next(g for g in groups.values() if current in g)
current = max(rev for rev in same if smart[rev] == min(smart[x] for x in same))
straight = {rev: straighten_html(markup[rev]) for rev in straighten}
actions.append(
{
"chapter_id": chapter["id"],
"chapter": unescape(chapter["title"]),
"story": chapter["story_title"],
"delete": sorted(delete),
"keep": sorted(keep),
"straighten": sorted(straighten),
"straight": {rev: (html, text_hash(html)) for rev, html in straight.items()},
"skipped": sorted(skipped),
"sources": sources,
"old_current": chapter["current_rev"],
"new_current": current,
}
)
return actions
def book_changed(action: dict) -> bool:
return action["new_current"] != action["old_current"] or action["new_current"] in action["straighten"]
def apply(conn: sqlite3.Connection, actions: list[dict]) -> None:
now = datetime.now().isoformat(timespec="seconds")
with conn:
for a in actions:
if a["new_current"] != a["old_current"]:
conn.execute(
"UPDATE chapters SET current_rev = ?, changed_at = ? WHERE id = ?",
(a["new_current"], now, a["chapter_id"]),
)
conn.executemany(
"UPDATE revisions SET html = ?, text_hash = ? WHERE chapter_id = ? AND rev = ?",
[(html, digest, a["chapter_id"], rev) for rev, (html, digest) in a["straight"].items()],
)
conn.executemany(
"DELETE FROM revisions WHERE chapter_id = ? AND rev = ?",
[(a["chapter_id"], rev) for rev in a["delete"]],
)
def backup(conn: sqlite3.Connection, db: Path) -> Path:
path = db.with_name(f"{db.stem}.before-quote-cleanup-{datetime.now():%Y%m%d-%H%M%S}{db.suffix}")
target = sqlite3.connect(path)
try:
conn.backup(target)
finally:
target.close()
return path
def _revs(revs: list[int], sources: dict[int, str]) -> str:
return ", ".join(f"{rev} ({sources[rev]})" for rev in revs)
def report(actions: list[dict]) -> None:
for a in actions:
print(f"{a['story']} / {a['chapter']}")
if a["delete"]:
print(f" delete revision {_revs(a['delete'], a['sources'])}, keep {_revs(a['keep'], a['sources'])}")
if a["new_current"] != a["old_current"]:
print(f" revision {a['new_current']} goes in the book instead of revision {a['old_current']}")
if a["straighten"]:
print(f" make the punctuation straight in revision {_revs(a['straighten'], a['sources'])}")
if a["skipped"]:
print(
f" not changed: revision {_revs(a['skipped'], a['sources'])} is the newest text from the site, "
"so the next recheck would add it again"
)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("db", type=Path, help="Path to bindery.db")
parser.add_argument("--apply", action="store_true", help="Back up the database, then change the revisions")
args = parser.parse_args(argv)
if not args.db.is_file():
print(f"Error: no such file: {args.db}", file=sys.stderr)
return 1
conn = sqlite3.connect(args.db, timeout=30)
conn.row_factory = sqlite3.Row
try:
actions = plan(conn)
report(actions)
deletions = sum(len(a["delete"]) for a in actions)
straightened = sum(len(a["straighten"]) for a in actions)
if not deletions and not straightened:
print("No revisions to change.")
return 0
print(f"\n{deletions} revisions to delete, and {straightened} revisions to make straight.")
if not args.apply:
print("Nothing changed. Run again with --apply to change them.")
return 0
print(f"Backup: {backup(conn, args.db)}")
apply(conn, actions)
print(f"Deleted {deletions} revisions, and made {straightened} revisions straight.")
rebuild = sorted({a["story"] for a in actions if book_changed(a)})
if rebuild:
print("The text in the book changed for these stories. Rebuild them:")
for title in rebuild:
print(f" - {title}")
finally:
conn.close()
return 0
if __name__ == "__main__":
sys.exit(main())
tests/conftest.py
+2
View File
@@ -7,8 +7,10 @@ from pathlib import Path
_tmp = Path(tempfile.mkdtemp(prefix="bindery-tests-"))
for name, sub in [("CONFIG_DIR", "config"), ("INGEST_DIR", "ingest"), ("TMP_DIR", "tmp")]:
os.environ[name] = str(_tmp / sub)
(_tmp / "ingest").mkdir() # Bindery does not make the ingest folder
os.environ["BINDERY_START_THREADS"] = "false"
os.environ["UPDATE_INTERVAL_HOURS"] = "0"
os.environ["CWA_ENABLED"] = "true" # most tests are about a Bindery with CWA
for name in (
"CWA_URL",
"CWA_USERNAME",
tests/fake_cwa.py
+3 -1
View File
@@ -26,6 +26,8 @@ class FakeCwa:
self.requests: list[str] = []
# CWA's "Show Publisher Section" setting. Without it, the publisher feed is 404.
self.publisher_view = True
# CWA's anonymous browsing. With it, the fake answers every request, with or without a login.
self.anonymous = False
fake = self
class Handler(BaseHTTPRequestHandler):
@@ -87,7 +89,7 @@ class FakeCwa:
self._send(h, 200, self._feed([self._book_entry(b) for b in page], next_href), "application/atom+xml")
def handle(self, h) -> None:
if not self._authorized(h):
if not self.anonymous and not self._authorized(h):
self._send(h, 401, b"Unauthorized", "text/plain")
return
url = urlparse(h.path)
tests/js/admin.test.mjs
+26
View File
@@ -0,0 +1,26 @@
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { find, install, server, settle } from './flow-helpers.mjs';
const ui = install();
const { stopPolling } = await import('../../bindery/static/js/jobs.js');
test('a user who is not an admin gets no settings link and no settings page', async (t) => {
t.after(stopPolling);
const backend = server(t);
backend.on('GET', 'auth/check', { authenticated: true, mode: 'proxy', user: 'bob', admin: false });
backend.on('GET', 'config', { metadata_fields: {}, mode_labels: {}, library_available: false, download_fields: {} });
backend.on('GET', 'jobs', []);
backend.on('GET', 'stories', []);
await import('../../bindery/static/js/main.js');
await settle();
assert.ok(find(ui.app, 'a', 'Stories'));
assert.equal(find(ui.app, 'a', 'Settings'), undefined);
window.location.hash = '#/settings';
ui.windowEvents.hashchange();
await settle();
assert.match(ui.app.textContent, /Only an admin can open the settings/);
// The page asks the server for no settings and no rules.
assert.deepEqual(backend.sent('GET', 'settings'), []);
assert.deepEqual(backend.sent('GET', 'rules'), []);
});
tests/js/auth-flows.test.mjs
+1 -1
View File
@@ -11,7 +11,7 @@ test('login retry, navigation, session expiry and logout preserve the authentica
const backend = server(t);
let auth = { authenticated: false, mode: 'cwa', user: null };
backend.on('GET', 'auth/check', () => auth);
backend.on('GET', 'config', { metadata_fields: {}, mode_labels: {}, library_available: false });
backend.on('GET', 'config', { metadata_fields: {}, mode_labels: {}, library_available: false, download_fields: {} });
backend.on('GET', 'jobs', []);
backend.on('GET', 'stories', []);
backend.on('GET', 'rules', { rules: [], paragraphs: false });
tests/js/flow-helpers.mjs
+2
View File
@@ -30,6 +30,8 @@ export class Element {
removeAttribute(name) { delete this.attributes[name]; }
addEventListener(name, fn) { this.listeners[name] = fn; }
focus() { document.activeElement = this; }
showModal() { this.open = true; }
close() { this.open = false; }
remove() { if (this.parent) this.parent.children = this.parent.children.filter((child) => child !== this); }
get firstElementChild() { return this.children.find((child) => child instanceof Element); }
get textContent() { return this.children.map((child) => child instanceof Element ? child.textContent : String(child)).join(''); }
tests/js/flows.test.mjs
+238 -11
View File
@@ -14,8 +14,9 @@ const { setUnauthorizedHandler } = await import('../../bindery/static/js/lib.js'
let api, main, details, cleanups;
const config = {
metadata_fields: { title: 'Title', authors: 'Authors', comments: 'Description', series: 'Series', pubdate: 'Published', tags: 'Tags', languages: 'Language' },
mode_labels: { new: 'New chapters' }, library_available: true, update_interval: 2,
mode_labels: { new: 'New chapters' }, library_available: true, cwa_enabled: true, update_interval: 2,
job_list_max_hours: 720, job_list_max_count: 500,
download_fields: { title: 'The title', id: 'The story ID' },
};
const rule = { pattern: 'before', replacement: 'after', flags: 'i', note: '', enabled: true };
const emptyRules = { rules: [], paragraphs: null, global_paragraphs: true };
@@ -54,7 +55,7 @@ for (const value of ['7', '']) {
await settle();
api.on('PUT', 'settings', {});
field(main, 'recent-chapters').value = value;
await click(section(main, 'Settings'), 'Save');
await click(section(main, 'Recheck recent chapters'), 'Save');
assert.deepEqual(api.sent('PUT', 'settings')[0].body, { recent_chapters: value === '' ? 0 : 7 });
assert.match(ui.toasts.textContent, /Saved/);
});
@@ -108,6 +109,52 @@ test('the jobs list settings load and save all four fields', async () => {
assert.match(ui.toasts.textContent, /Saved/);
});
test('each group of settings is an open card, and the download help starts closed', async () => {
settingsPage(main, {}, config);
await settle();
const cards = main.children.filter((el) => el.tagName !== 'h1');
assert.deepEqual(cards.map((el) => [el.tagName, el.className, el.open, el.children[0].textContent]), [
['details', 'card', true, 'Recheck recent chapters'],
['details', 'card', true, 'Book Downloads'],
['details', 'card', true, 'Uploads to CWA'],
['details', 'card', true, 'Jobs list'],
['details', 'card', true, 'Global replacement rules'],
['details', 'card', true, 'FanFicFare settings (personal.ini)'],
]);
const help = nodes(section(main, 'Book Downloads')).filter((el) => el.className === 'help');
assert.deepEqual(help.map((el) => [el.tagName, el.open, el.children[0].textContent]), [
['details', undefined, 'File name template help'],
['details', undefined, 'Date format help'],
]);
});
test('the download template loads, lists its fields, and saves the stored form', async () => {
api.on('GET', 'settings', {
recent_chapters: 0, personal_ini: '', download_template: '{title}', download_date_format: '%b, %Y',
});
settingsPage(main, {}, config);
await settle();
const card = section(main, 'Book Downloads');
assert.equal(field(main, 'download-template').value, '{title}');
assert.equal(field(main, 'download-date-format').value, '%b, %Y');
assert.ok(find(card, 'code', '{id}'));
assert.match(card.textContent, /\{title\}The title\{id\}The story ID/);
field(main, 'download-template').value = ' {title} - {authors} ';
field(main, 'download-date-format').value = '';
api.on('PUT', 'settings', { download_template: '{title} - {authors}', download_date_format: '%b, %Y' });
await click(card, 'Save');
assert.deepEqual(api.sent('PUT', 'settings')[0].body, {
opds_enabled: false, download_template: ' {title} - {authors} ', download_date_format: '',
});
assert.equal(field(main, 'download-template').value, '{title} - {authors}');
assert.equal(field(main, 'download-date-format').value, '%b, %Y');
api.on('PUT', 'settings', { error: 'Bindery does not support template functions' }, 400);
field(main, 'download-template').value = '{title:uppercase()}';
await click(card, 'Save');
assert.equal(field(main, 'download-template').value, '{title:uppercase()}');
assert.match(ui.toasts.textContent, /does not support template functions/);
});
test('adding stories keeps input on rejection and clears it after a mixed result', async () => {
await mountStories({ ...config, library_available: false });
const urls = nodes(main).find((el) => el.tagName === 'textarea');
@@ -186,11 +233,11 @@ test('dropping a file on a closed epub import card opens it and fills the file f
test('library import sends only selected books and handles an empty candidate list', async () => {
await mountStories();
api.on('GET', 'library/candidates', []);
await click(main, 'Find stories in the library');
await click(main, 'Show all stories in the library');
assert.match(main.textContent, /tracked already/);
const books = [{ calibre_id: 8, uuid: 'a', title: 'A' }, { calibre_id: 9, uuid: 'b', title: 'B' }];
const books = [{ id: 8, uuid: 'a', title: 'A', authors: ['W'] }, { id: 9, uuid: 'b', title: 'B', authors: [] }];
api.on('GET', 'library/candidates', books);
await click(main, 'Find stories in the library');
await click(main, 'Show all stories in the library');
change(field(main, 'Select all'), false);
await click(main, 'Import selected');
assert.equal(writes().length, 0);
@@ -205,6 +252,91 @@ test('library import sends only selected books and handles an empty candidate li
assert.deepEqual(api.sent('POST', 'library/import')[0].body, { books: [{ id: 8, uuid: 'a', title: 'A' }], check: false });
});
test('library import search filters the loaded list, selects no book at first and imports the ticked books', async () => {
await mountStories();
const card = section(main, 'Import from the library');
const form = nodes(card).find((el) => el.tagName === 'form');
const submit = find(form, 'button', 'Search');
assert.equal(form.className, 'library-search');
assert.equal(submit.attributes.type, 'submit');
const search = async (text) => {
field(card, 'Search the library').value = text;
emit(form, 'submit');
await settle();
};
const shown = () => ['Ash & Bone', 'Bright Sky', 'Ash Road'].filter((title) => field(card, title));
await search(' ');
assert.equal(api.requests.some((r) => r.path.startsWith('library/candidates')), false);
const books = [
{ id: 8, uuid: 'a', title: 'Ash & Bone', authors: ['Wren'], publisher: 'www.royalroad.com' },
{ id: 9, uuid: 'b', title: 'Bright Sky', authors: ['Ashley Poe', 'Wren'], publisher: 'archiveofourown.org' },
{ id: 10, uuid: 'c', title: 'Ash Road', authors: [], publisher: 'www.royalroad.com' },
];
api.on('GET', 'library/candidates', books);
await search('nothing');
assert.match(card.textContent, /No library books from a story site match/);
assert.equal(find(card, 'button', 'Import selected'), undefined);
// Each word must be in the title, the authors or the site. Capitals do not matter.
await search(' ASH ');
assert.deepEqual(shown(), ['Ash & Bone', 'Bright Sky', 'Ash Road']);
await search('ash royalroad');
assert.deepEqual(shown(), ['Ash & Bone', 'Ash Road']);
await search('wren sky');
assert.deepEqual(shown(), ['Bright Sky']);
assert.equal(api.sent('GET', 'library/candidates').length, 1, 'One load serves each search');
await search('ash');
assert.deepEqual(['Select all', ...shown()].map((label) => field(card, label).checked), [false, false, false, false]);
await click(card, 'Import selected');
assert.equal(writes().length, 0);
change(field(card, 'Bright Sky'), true);
api.on('POST', 'library/import', [{ id: 'job' }]);
await click(card, 'Import selected');
assert.deepEqual(api.sent('POST', 'library/import')[0].body, { books: [{ id: 9, uuid: 'b', title: 'Bright Sky' }], check: true });
// A queued book leaves the list, without another load.
await search('ash');
assert.deepEqual(shown(), ['Ash & Bone', 'Ash Road']);
assert.equal(api.sent('GET', 'library/candidates').length, 1);
// The full list loads again, and selects each book.
await click(card, 'Show all stories in the library');
assert.equal(api.sent('GET', 'library/candidates').length, 2);
assert.deepEqual(['Select all', ...shown()].map((label) => field(card, label).checked), [true, true, true, true]);
});
test('the library import buttons cannot be used again until the answer comes', async () => {
await mountStories();
const card = section(main, 'Import from the library');
const form = nodes(card).find((el) => el.tagName === 'form');
const submit = find(form, 'button', 'Search');
const all = find(card, 'button', 'Show all stories in the library');
let answer;
api.on('GET', 'library/candidates', () => new Promise((resolve) => { answer = resolve; }));
field(card, 'Search the library').value = 'a';
emit(form, 'submit');
await settle();
assert.equal(submit.disabled, true);
emit(form, 'submit');
await settle();
assert.equal(api.sent('GET', 'library/candidates').length, 1);
answer([{ id: 8, uuid: 'a', title: 'A', authors: [] }]);
await settle();
assert.equal(submit.disabled, false);
assert.ok(field(card, 'A'));
emit(all, 'click');
await settle();
assert.equal(all.disabled, true);
answer([]);
await settle();
assert.equal(all.disabled, false);
// A failed load frees the button, and the next search loads again.
api.on('GET', 'library/candidates', { error: 'CWA is offline' }, 502);
await click(card, 'Show all stories in the library');
emit(form, 'submit');
await settle();
assert.equal(submit.disabled, false);
assert.equal(all.disabled, false);
assert.equal(api.sent('GET', 'library/candidates').length, 4);
});
test('rules edits, moves, removal and paragraph choices are saved in order', async () => {
api.on('GET', 'stories/1/rules', { ...emptyRules, rules: [rule, { ...rule, pattern: 'second' }] });
const editor = rulesEditor('stories/1/rules');
@@ -275,13 +407,13 @@ test('the chapter table stacks on a narrow screen instead of scrolling sideways'
await mountStory();
const wrap = nodes(main).find((el) => el.className === 'table-wrap chapters-table');
assert.ok(wrap);
const headings = nodes(wrap).filter((el) => el.tagName === 'th').map((el) => el.textContent);
assert.deepEqual(headings, ['', '#', 'Title', 'State', 'Revisions', 'Added']);
const headings = nodes(wrap).filter((el) => el.tagName === 'th');
assert.deepEqual(headings.map((el) => el.textContent.trim()), ['Select all', '#', 'Title', 'State', 'Revisions', 'Added']);
const dataLabels = nodes(wrap).filter((el) => el.tagName === 'td').map((el) => el.attributes['data-label']);
const perRow = [undefined, '#', undefined, 'State', 'Revisions', 'Added'];
assert.deepEqual(dataLabels, [...perRow, ...perRow]);
// "Select all" moved above the table, so a hidden table header on mobile does not hide it.
assert.ok(nodes(main).find((el) => el.tagName === 'label' && el.textContent.trim() === 'Select all'));
// The Select all checkbox is in the first heading. On a phone, it is the only heading that shows.
assert.ok(nodes(headings[0]).includes(field(main, 'Select all chapters')));
});
test('story list stacks on a narrow screen instead of scrolling sideways', async () => {
@@ -289,7 +421,7 @@ test('story list stacks on a narrow screen instead of scrolling sideways', async
const wrap = nodes(main).find((el) => el.className === 'table-wrap stories-table');
assert.ok(wrap);
const dataLabels = nodes(wrap).filter((el) => el.tagName === 'td').map((el) => el.attributes['data-label']);
assert.deepEqual(dataLabels, [undefined, 'Author', 'Site', 'Chapters', 'Status', 'Last check']);
assert.deepEqual(dataLabels, [undefined, 'Author', 'Site', 'Chapters', 'Pages', 'Status', 'Last change', 'Last check']);
});
test('the story list shows removed and revised chapter counts, only when there are any', async () => {
@@ -348,6 +480,47 @@ test('chapter inclusion changes reload the story after queuing the rebuild', asy
assert.equal(api.sent('GET', 'stories/1').length, 3);
});
test('shift-click selects a range of chapters, and Move sends them with the chapter to go after', async () => {
details.chapters.push({ id: 4, title: 'Three', url: 'chapter-3', included: true, on_site: true, current_rev: 1, revisions: 1 });
await mountStory();
api.on('POST', 'stories/1/chapters/move', {});
const dialog = nodes(main).find((el) => el.tagName === 'dialog');
const texts = (tag) => nodes(dialog).filter((el) => el.tagName === tag).map((el) => el.textContent);
await click(main, 'Move…');
assert.ok(!dialog.open);
const box = (title) => field(main, `Select ${title}`);
const shiftClick = (el, on) => {
el.listeners.click({ shiftKey: true, target: el });
return change(el, on);
};
change(box('One'), true);
shiftClick(box('Three'), true);
assert.equal(box('Two').checked, true);
// The dialog names the chapters, and they are not in the list of chapters to go after.
await click(main, 'Move…');
assert.ok(dialog.open);
assert.deepEqual(texts('h3'), ['Move 3 chapters']);
assert.deepEqual(texts('li'), ['One', 'Two', 'Three']);
assert.deepEqual(texts('option'), ['The start']);
await click(dialog, 'Cancel');
assert.ok(!dialog.open);
assert.equal(writes().length, 0);
await click(main, 'Move…');
await click(dialog, 'Move');
assert.deepEqual(api.sent('POST', 'stories/1/chapters/move').at(-1).body, { chapter_ids: [2, 3, 4], after: null });
assert.ok(!dialog.open);
shiftClick(box('One'), false);
change(box('Two'), true);
await click(main, 'Move…');
assert.deepEqual(texts('option'), ['The start', 'One', 'Three']);
field(dialog, 'move-after').value = '4';
await click(dialog, 'Move');
assert.deepEqual(api.sent('POST', 'stories/1/chapters/move').at(-1).body, { chapter_ids: [3], after: 4 });
// The next dialog starts with the last choice.
await click(main, 'Move…');
assert.equal(field(dialog, 'move-after').value, '4');
});
test('unsaved metadata and update settings survive job completion, then save typed values', async () => {
await mountStory();
startPolling();
@@ -438,7 +611,7 @@ test('library search trims the query and links by UUID', async () => {
await settle();
assert.match(main.textContent, /No library books match/);
api.on('GET', 'library/search?q=A%20%26%20B', [{ id: 7, uuid: 'book-7', title: 'A & B', authors: ['A', 'B'] },
{ id: 8, uuid: 'book-8', title: 'Tracked', authors: 'Writer', tracked_as: 'Other' }]);
{ id: 8, uuid: 'book-8', title: 'Tracked', authors: ['Writer'], tracked_as: 'Other' }]);
emit(form, 'submit');
await settle();
const links = nodes(main).filter((el) => el.tagName === 'button' && el.textContent === 'Link');
@@ -449,6 +622,30 @@ test('library search trims the query and links by UUID', async () => {
assert.equal(api.sent('GET', 'stories/1').length, 2);
});
test('deleting chapters asks in a dialog and warns about chapters that are on the site', async () => {
await mountStory();
api.on('POST', 'stories/1/chapters/delete', {});
const dialog = nodes(main).find((el) => el.tagName === 'dialog');
const text = (tag) => nodes(dialog).find((el) => el.tagName === tag).textContent;
await click(main, 'Delete…');
assert.ok(!dialog.open);
change(field(main, 'Select Two'), true);
await click(main, 'Delete…');
assert.ok(dialog.open);
assert.equal(text('h3'), 'Delete 1 chapter');
assert.equal(text('p'), 'Bindery deletes this chapter and its revisions. You cannot undo this.');
await click(dialog, 'Cancel');
change(field(main, 'Select One'), true);
await click(main, 'Delete…');
assert.equal(text('h3'), 'Delete 2 chapters');
assert.match(text('p'), /^Bindery deletes these chapters and their revisions\..*comes back with the next download/);
assert.equal(writes().length, 0);
await click(dialog, 'Delete');
assert.deepEqual(api.sent('POST', 'stories/1/chapters/delete').at(-1).body, { chapter_ids: [2, 3] });
assert.ok(!dialog.open);
assert.equal(api.sent('GET', 'stories/1').length, 2);
});
test('sending as a new book and stopping tracking require confirmation', async () => {
details.story.ingested_at = '2026-01-01';
await mountStory();
@@ -576,3 +773,33 @@ test('chapter titles show the entities that FanFicFare keeps as text', async ()
render([{ id: 'j', status: 'done', story_id: 1, result: { changed: [{ chapter_id: 2, title: 'A &amp; B' }] } }]);
assert.ok(find(el, 'a', 'A & B'));
});
test('with CWA off, the pages show nothing about CWA', async () => {
await mountStory();
assert.ok(section(main, 'Library link'));
const off = { ...config, cwa_enabled: false, library_available: false };
// Values that show a notice with CWA on.
Object.assign(details, { pause_uploads: true, ingest_available: false, link_books: false });
details.story.upload_held = true;
await mountStory(off);
assert.ok(section(main, 'Metadata'));
assert.equal(section(main, 'Library link'), undefined);
assert.doesNotMatch(main.textContent, /CWA|INGEST_DIR|library/i);
await mountStories(off);
assert.ok(section(main, 'Add stories'));
assert.equal(section(main, 'Import from the library'), undefined);
assert.doesNotMatch(main.textContent, /CWA|INGEST_DIR|library/i);
for (const ingest of [true, false]) {
api.on('GET', 'settings', { recent_chapters: 3, personal_ini: '[defaults]', ingest_available: ingest });
settingsPage(main, {}, off);
await settle();
assert.ok(section(main, 'Jobs list'));
assert.equal(section(main, 'Uploads to CWA'), undefined);
assert.doesNotMatch(main.textContent, /CWA|library/i);
// The alert about INGEST_DIR is under Book Downloads, also with CWA off.
const note = nodes(section(main, 'Book Downloads')).find((el) => el.attributes.id === 'ingest-note');
assert.equal(note.hidden, ingest);
assert.equal(note.className, 'notice');
assert.match(note.textContent, /INGEST_DIR is not set\. Bindery sends no book to the ingest folder/);
}
});
tests/js/frontend.test.mjs
+2 -1
View File
@@ -46,8 +46,9 @@ const text = (node) => node instanceof Element ? node.children.map(text).join(''
const find = (root, tag, label) => elements(root).find((el) => el.tagName === tag && text(el) === label);
const settle = () => new Promise((resolve) => setImmediate(resolve));
const config = {
metadata_fields: { title: 'Title' }, mode_labels: {}, library_available: true, update_interval: 0,
metadata_fields: { title: 'Title' }, mode_labels: {}, library_available: true, cwa_enabled: true, update_interval: 0,
job_list_max_hours: 720, job_list_max_count: 500,
download_fields: { title: 'The title', id: 'The story ID' },
};
const story = {
id: 1, title: 'Story', author: 'Author', url: 'https://example.com/story', site: 'example.com',
tests/js/layout.test.mjs
+77
View File
@@ -87,6 +87,83 @@ test('the Add stories textarea uses the smaller text size without shrinking othe
assert.equal(declarations('textarea')['font-size'], undefined);
});
test('story timestamps fit their text and chapter details use plain small text', () => {
const desktop = css.slice(css.indexOf('@media (min-width: 601px)'));
const date = declarations('.date');
assert.equal(date['white-space'], 'nowrap');
assert.equal(date['min-inline-size'], undefined);
const storyDate = declarations('.stories-table .date', desktop);
assert.equal(storyDate['inline-size'], '1%');
assert.equal(storyDate['min-inline-size'], undefined);
const details = declarations(".stories-table td[data-label='Chapters'] small");
assert.equal(details.display, 'block');
assert.equal(details['white-space'], 'nowrap');
assert.equal(details.background, undefined);
assert.equal(details.border, undefined);
assert.equal(details.padding, undefined);
});
test('sortable headings show direction and remain usable above story cards on narrow screens', () => {
assert.equal(declarations('.stories-table th button').background, 'none');
assert.equal(declarations('.stories-table th[aria-sort] button::after').content, "'↕'");
assert.equal(declarations(".stories-table th[aria-sort='ascending'] button::after").content, "'↑'");
assert.equal(declarations(".stories-table th[aria-sort='descending'] button::after").content, "'↓'");
const mobile = css.slice(css.indexOf('@media (max-width: 600px)'));
assert.equal(declarations('.stories-table thead tr', mobile).display, 'flex');
assert.notEqual(declarations('.stories-table th', mobile).display, 'none');
assert.doesNotMatch(css, /\.stories-table thead\s*\{[^}]*clip-path/);
});
test('on narrow screens a story cell with no value shows no line', () => {
const start = css.indexOf('@media (max-width: 600px)', css.indexOf('.stories-table th button'));
assert.equal(declarations('.stories-table td:empty', css.slice(start)).display, 'none');
assert.doesNotMatch(css.slice(0, start), /\.stories-table td:empty/);
});
test('on narrow screens the headings are sort chips, and only the sorted chip is tinted with an arrow', () => {
const start = css.indexOf('@media (max-width: 600px)', css.indexOf('.stories-table th button'));
const mobile = css.slice(start);
assert.equal(declarations('.stories-table thead tr', mobile)['flex-wrap'], 'wrap');
const caption = declarations('.stories-table thead tr::before', mobile);
assert.equal(caption.content, "'Sort by'");
assert.equal(caption['flex-basis'], '100%');
const chip = declarations('.stories-table th button', mobile);
assert.equal(chip['border-radius'], '999px');
assert.equal(chip['inline-size'], 'auto');
assert.equal(chip.padding, '3px 10px');
assert.equal(chip['font-size'], 'var(--text-sm)');
assert.equal(chip.background, 'var(--surface)');
assert.equal(declarations(".stories-table th[aria-sort='none'] button::after", mobile).display, 'none');
const sorted = declarations(".stories-table th:not([aria-sort='none']) button", mobile);
assert.equal(sorted.background, 'var(--info-bg)');
assert.equal(sorted.color, 'var(--info)');
assert.equal(sorted['border-color'], undefined, 'The sorted chip keeps the soft border');
// The wide layout keeps plain headings.
const wide = css.slice(0, start);
assert.doesNotMatch(wide, /\.stories-table thead tr::before/);
assert.equal(declarations('.stories-table th button', wide).border, '0');
});
test('sortable headings fill the column with one clickable highlight without shifting layout', () => {
assert.equal(declarations('.stories-table th').padding, '0');
const button = declarations('.stories-table th button');
assert.equal(button['inline-size'], '100%');
assert.equal(button['justify-content'], 'flex-start');
assert.equal(button.padding, '6px 8px');
assert.equal(button.margin, undefined);
const highlight = declarations('.stories-table th button:is(:hover, :focus-visible)');
assert.equal(highlight.background, 'var(--info-bg)');
assert.equal(highlight.color, 'var(--info)');
assert.equal(highlight['text-decoration'], undefined);
assert.equal(highlight.padding, undefined);
assert.equal(highlight.border, undefined);
assert.doesNotMatch(css, /\.stories-table th button:hover\s*\{[^}]*underline/);
});
test('disabled checkbox labels use muted text', () => {
assert.equal(declarations("label:has(> input[type='checkbox']:disabled)").color, 'var(--muted)');
});
function luminance(hex) {
const channels = hex.slice(1).match(/../g).map((channel) => {
const value = parseInt(channel, 16) / 255;
tests/js/local-auth.test.mjs
+1 -1
View File
@@ -11,7 +11,7 @@ test('local login, retry, session expiry and logout use the existing form', asyn
const backend = server(t);
let auth = { authenticated: false, mode: 'local', user: null };
backend.on('GET', 'auth/check', () => auth);
backend.on('GET', 'config', { metadata_fields: {}, mode_labels: {}, library_available: false });
backend.on('GET', 'config', { metadata_fields: {}, mode_labels: {}, library_available: false, download_fields: {} });
backend.on('GET', 'stories', []);
backend.on('GET', 'jobs', []);
backend.on('GET', 'expired', { error: 'Session ended' }, 401);
tests/js/opds-settings.test.mjs
+127
View File
@@ -0,0 +1,127 @@
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { Element, change, click, field, find, install, nodes, section, server, settle } from './flow-helpers.mjs';
install();
const { settingsPage } = await import('../../bindery/static/js/settings.js');
async function open(t, settings, authMode = 'proxy') {
install();
t.mock.timers.enable({ apis: ['setTimeout'] });
const backend = server(t);
const stored = { download_template: '{title}', download_date_format: '%Y', personal_ini: '', ...settings };
backend.on('GET', 'settings', stored);
backend.on('GET', 'rules', { rules: [] });
backend.on('GET', 'stories', []);
backend.on('PUT', 'settings', ({ body }) => ({ ...stored, ...body }));
backend.on('PUT', 'settings/opds-password', { opds_user: 'alice', opds_password_set: true });
const main = new Element('main');
settingsPage(main, {}, { cwa_enabled: false, download_fields: {}, auth_mode: authMode });
await settle();
return { backend, main, card: section(main, 'Book Downloads') };
}
for (const enabled of [false, true]) {
test(`OPDS loads ${enabled}, appears first in Book Downloads, and saves both states`, async (t) => {
const { backend, main, card } = await open(t, { opds_enabled: enabled });
const toggle = field(card, 'opds-enabled');
assert.equal(toggle.checked, enabled);
assert.equal(card.children[1].tagName, 'label');
assert.equal(card.children[1].children[0], toggle);
assert.equal(card.children[1].textContent, ' Enable OPDS');
assert.equal(toggle.getAttribute('aria-describedby'), 'opds-note');
assert.match(field(card, 'opds-note').textContent, /off by default/);
assert.equal(find(field(card, 'opds-note'), 'a', 'the OPDS catalog').getAttribute('href'), 'opds');
assert.equal(nodes(main).filter((node) => node.getAttribute('id') === 'opds-enabled').length, 1);
for (const value of [!enabled, enabled]) {
change(toggle, value);
await click(card, 'Save');
assert.deepEqual(backend.sent('PUT', 'settings').at(-1).body, {
opds_enabled: value, download_template: '{title}', download_date_format: '%Y',
});
assert.equal(toggle.checked, value);
}
backend.on('PUT', 'settings', { opds_enabled: false, download_template: '{title}', download_date_format: '%Y' });
change(toggle, true);
await click(card, 'Save');
assert.equal(toggle.checked, false, 'The saved response sets the checkbox state');
});
}
const LOGIN = {
none: /Bindery asks for no login, so the catalog needs none/,
local: /sends your Bindery user name and password/,
cwa: /sends your CWA user name and password/,
proxy: /The proxy skips its login for the catalog.*Each user sets an OPDS password.*user name from the proxy/,
};
for (const [mode, text] of Object.entries(LOGIN)) {
test(`the note names the login of the ${mode} mode, and only the proxy mode shows the password`, async (t) => {
const { card } = await open(t, { opds_enabled: true, opds_password_set: false, opds_user: 'alice' }, mode);
assert.match(field(card, 'opds-note').textContent, text);
assert.doesNotMatch(field(card, 'opds-note').textContent, /BINDERY_API_KEY|Bearer|X-Api-Key/);
for (const id of ['opds-user', 'opds-password', 'opds-password-status']) {
assert.equal(field(card, id) !== undefined, mode === 'proxy', id);
}
assert.equal(find(card, 'button', 'Set password') !== undefined, mode === 'proxy');
});
}
test('the password field is grey until the box is checked, and follows the box at once', async (t) => {
const { card } = await open(t, { opds_enabled: false, opds_password_set: true, opds_user: 'alice' });
const password = field(card, 'opds-password');
const button = find(card, 'button', 'Reset password');
assert.equal(password.disabled, true);
assert.equal(button.disabled, true);
change(field(card, 'opds-enabled'), true);
assert.equal(password.disabled, false);
assert.equal(button.disabled, false);
change(field(card, 'opds-enabled'), false);
assert.equal(password.disabled, true);
assert.equal(button.disabled, true);
});
for (const isSet of [false, true]) {
test(`the OPDS password ${isSet ? 'is set' : 'is not set'}, and the button sets or resets it`, async (t) => {
const { backend, card } = await open(t, { opds_enabled: true, opds_password_set: isSet, opds_user: 'alice' });
const password = field(card, 'opds-password');
const status = field(card, 'opds-password-status');
const user = field(card, 'opds-user');
assert.equal(user.value, 'alice');
assert.equal(user.getAttribute('readonly'), '');
assert.equal(user.disabled, false, 'The user name stays selectable');
assert.equal(user.getAttribute('aria-describedby'), 'opds-password-status');
assert.equal(find(card, 'label', 'OPDS user name').getAttribute('for'), 'opds-user');
assert.equal(find(card, 'label', 'OPDS password').getAttribute('for'), 'opds-password');
assert.equal(password.disabled, false);
assert.equal(password.getAttribute('type'), 'password');
assert.equal(password.getAttribute('autocomplete'), 'new-password');
assert.equal(password.getAttribute('minlength'), '8');
assert.equal(password.value, '');
assert.match(status.textContent, /enter the user name and the password in your reader/i);
assert.match(status.textContent, isSet ? /You have an OPDS password/ : /You have no OPDS password yet/);
assert.ok(find(card, 'button', isSet ? 'Reset password' : 'Set password'));
password.value = 'long enough';
await click(card, isSet ? 'Reset password' : 'Set password');
assert.deepEqual(backend.sent('PUT', 'settings/opds-password')[0].body, { password: 'long enough' });
assert.equal(password.value, '', 'The field is empty after the save');
assert.match(status.textContent, /You have an OPDS password/);
assert.equal(password.disabled, false);
assert.ok(find(card, 'button', 'Reset password'));
// The Save button of the card never sends the password.
password.value = 'not sent';
await click(card, 'Save');
assert.equal(backend.sent('PUT', 'settings/opds-password').length, 1);
assert.equal('password' in backend.sent('PUT', 'settings')[0].body, false);
assert.equal(password.disabled, false, 'A save keeps the field in step with the box');
});
}
test('a rejected OPDS password keeps the field and the status', async (t) => {
const { backend, card } = await open(t, { opds_enabled: true, opds_password_set: false });
backend.on('PUT', 'settings/opds-password', { error: 'The OPDS password must be text of 8 to 256 characters' }, 400);
field(card, 'opds-password').value = 'short';
await click(card, 'Set password');
assert.equal(field(card, 'opds-password').value, 'short');
assert.match(field(card, 'opds-password-status').textContent, /You have no OPDS password yet/);
assert.ok(find(card, 'button', 'Set password'));
});
tests/js/proxy-reload.test.mjs
+1 -1
View File
@@ -15,7 +15,7 @@ test('a 401 while logged in through the proxy reloads the page', async (t) => {
});
const backend = server(t);
backend.on('GET', 'auth/check', { authenticated: true, mode: 'proxy', user: 'alice' });
backend.on('GET', 'config', { metadata_fields: {}, mode_labels: {}, library_available: false });
backend.on('GET', 'config', { metadata_fields: {}, mode_labels: {}, library_available: false, download_fields: {} });
backend.on('GET', 'jobs', []);
backend.on('GET', 'stories', []);
backend.on('GET', 'expired', { error: 'Session ended' }, 401);
tests/js/story-list.test.mjs
+329
View File
@@ -0,0 +1,329 @@
import assert from 'node:assert/strict';
import { beforeEach, test } from 'node:test';
import { click, field, find, input, install, nodes, server, settle } from './flow-helpers.mjs';
const { app, toasts } = install();
const SORT_KEY = 'bindery.story-sort';
beforeEach(() => {
const values = new Map();
window.localStorage = {
getItem: (key) => values.get(key) ?? null,
setItem: (key, value) => values.set(key, value),
};
toasts.replaceChildren();
});
const { storiesPage } = await import('../../bindery/static/js/stories.js');
const { refreshJobs, startPolling, stopPolling } = await import('../../bindery/static/js/jobs.js');
const { fmtDate } = await import('../../bindery/static/js/lib.js');
// The server gives each story its sort key: the title in lower case, without a leading article.
const story = (id, title, changes = {}) => ({
id, title, url: `https://example.com/${id}`,
sort_title: (title || `https://example.com/${id}`).trim().replace(/^(?:the|an?)\s+/i, '').toLowerCase(),
author: 'Reader', site: 'example.com',
chapter_count: 42, page_count: 120, offsite_count: 0, revised_count: 0, status: 'ok', message: 'No changes',
last_changed: null, last_checked: null, ...changes,
});
const rows = () => {
const table = nodes(app).find((el) => el.className === 'table-wrap stories-table');
return nodes(table).find((el) => el.tagName === 'tbody').children;
};
const titles = () => rows().map((row) => row.children[0].textContent);
const cell = (row, label) => row.children.find((el) => el.attributes['data-label'] === label);
const heading = (label) => nodes(app).find((el) => el.tagName === 'th' && el.textContent === label);
async function mount(t, list) {
const backend = server(t);
backend.on('GET', 'stories', list);
backend.on('GET', 'jobs', []);
const cleanup = storiesPage(app, {}, { library_available: false, main_jobs_hours: 24, main_jobs_count: 10 });
t.after(() => { cleanup(); stopPolling(); });
await settle();
return backend;
}
test('column headings toggle sorting, with newest dates first and missing dates always last', async (t) => {
const backend = await mount(t, [
story(3, 'Gamma', { last_changed: '2026-01-01T12:00:00Z', last_checked: '2026-01-03T12:00:00Z' }),
story(2, 'Beta', { last_changed: '2026-01-02T12:00:00Z', last_checked: '2026-01-02T12:00:00Z' }),
story(4, 'No date'),
story(1, 'Alpha', { last_changed: '2026-01-03T12:00:00Z', last_checked: '2026-01-01T12:00:00Z' }),
]);
assert.equal(field(app, 'Sort stories by'), undefined);
assert.equal(heading('Title').attributes['aria-sort'], 'ascending');
assert.equal(heading('Last change').attributes['aria-sort'], 'none');
assert.deepEqual(titles(), ['Alpha', 'Beta', 'Gamma', 'No date']);
assert.equal(find(app, 'button', 'Last change').attributes['aria-label'], 'Sort by last change, newest first');
await click(app, 'Last change');
assert.deepEqual(titles(), ['Alpha', 'Beta', 'Gamma', 'No date']);
assert.equal(heading('Title').attributes['aria-sort'], 'none');
assert.equal(heading('Last change').attributes['aria-sort'], 'descending');
assert.equal(document.activeElement, find(app, 'button', 'Last change'));
assert.equal(find(app, 'button', 'Last change').attributes['aria-label'], 'Sort by last change, oldest first');
await click(app, 'Last change');
assert.deepEqual(titles(), ['Gamma', 'Beta', 'Alpha', 'No date']);
assert.equal(heading('Last change').attributes['aria-sort'], 'ascending');
await click(app, 'Last check');
assert.deepEqual(titles(), ['Gamma', 'Beta', 'Alpha', 'No date']);
assert.equal(heading('Last check').attributes['aria-sort'], 'descending');
await click(app, 'Last check');
assert.deepEqual(titles(), ['Alpha', 'Beta', 'Gamma', 'No date']);
assert.equal(heading('Last check').attributes['aria-sort'], 'ascending');
await click(app, 'Title');
assert.deepEqual(titles(), ['Alpha', 'Beta', 'Gamma', 'No date']);
await click(app, 'Title');
assert.deepEqual(titles(), ['No date', 'Gamma', 'Beta', 'Alpha']);
assert.equal(heading('Title').attributes['aria-sort'], 'descending');
await click(app, 'Last change');
assert.deepEqual(titles(), ['Alpha', 'Beta', 'Gamma', 'No date']);
assert.equal(backend.sent('GET', 'stories').length, 1);
});
test('title sorting and tie breaks keep ignoring leading articles', async (t) => {
await mount(t, ['The Zebra', 'an Owl', 'Anthem', 'A Bee', 'Theory', 'Cat'].map((title, i) => story(i + 1, title)));
const order = ['Anthem', 'A Bee', 'Cat', 'an Owl', 'Theory', 'The Zebra'];
assert.deepEqual(titles(), order);
await click(app, 'Title');
assert.deepEqual(titles(), [...order].reverse());
await click(app, 'Chapters');
assert.deepEqual(titles(), order);
});
test('chapter sorting compares counts numerically and keeps equal counts in title order', async (t) => {
await mount(t, [
story(1, 'Zeta', { chapter_count: 2, offsite_count: 2, revised_count: 1 }),
story(2, 'Alpha', { chapter_count: 10 }),
story(3, 'Beta', { chapter_count: 10 }),
story(4, 'Gamma', { chapter_count: 0 }),
story(5, 'Omega', { chapter_count: 100 }),
]);
assert.equal(find(app, 'button', 'Chapters').attributes['aria-label'], 'Sort by chapters, most first');
await click(app, 'Chapters');
assert.deepEqual(titles(), ['Omega', 'Alpha', 'Beta', 'Zeta', 'Gamma']);
assert.equal(heading('Chapters').attributes['aria-sort'], 'descending');
await click(app, 'Chapters');
assert.deepEqual(titles(), ['Gamma', 'Zeta', 'Alpha', 'Beta', 'Omega']);
assert.equal(heading('Chapters').attributes['aria-sort'], 'ascending');
});
test('pages show beside the chapters and sort numerically', async (t) => {
await mount(t, [
story(1, 'Zeta', { page_count: 9 }),
story(2, 'Alpha', { page_count: 1200 }),
story(3, 'Beta', { page_count: 0 }),
story(4, 'Gamma', { page_count: 9 }),
]);
const cells = nodes(rows()[0]).filter((el) => el.tagName === 'td').map((el) => el.attributes['data-label']);
assert.deepEqual(cells.slice(3, 5), ['Chapters', 'Pages']);
assert.equal(find(app, 'button', 'Pages').attributes['aria-label'], 'Sort by pages, most first');
await click(app, 'Pages');
assert.deepEqual(titles(), ['Alpha', 'Gamma', 'Zeta', 'Beta']);
await click(app, 'Pages');
assert.deepEqual(titles(), ['Beta', 'Gamma', 'Zeta', 'Alpha']);
assert.deepEqual(JSON.parse(window.localStorage.getItem(SORT_KEY)), { column: 'page_count', direction: 1 });
});
test('a story with no built book shows no page count and sorts last in each direction', async (t) => {
await mount(t, [
story(1, 'Alpha', { page_count: null }),
story(2, 'Beta', { page_count: 300 }),
story(3, 'Gamma', { page_count: 0 }),
story(4, 'Delta', { page_count: null }),
]);
const pages = () => rows().map((row) => cell(row, 'Pages'));
assert.deepEqual(pages().map((el) => el.textContent), ['', '300', '', '0']);
assert.deepEqual(pages().map((el) => el.children.length), [0, 1, 0, 1], 'An empty cell hides on a narrow screen');
await click(app, 'Pages');
assert.deepEqual(titles(), ['Beta', 'Gamma', 'Alpha', 'Delta']);
await click(app, 'Pages');
assert.deepEqual(titles(), ['Gamma', 'Beta', 'Alpha', 'Delta']);
});
for (const [label, key] of [['Author', 'author'], ['Site', 'site'], ['Status', 'status']]) {
test(`${label} sorts alphabetically in either direction and uses titles to break ties`, async (t) => {
await mount(t, [
story(1, 'Zeta', { [key]: 'ok' }),
story(3, 'Beta', { [key]: 'error' }),
story(2, 'Alpha', { [key]: 'ERROR' }),
story(4, 'Gamma', { [key]: null }),
story(5, 'Omega', { [key]: 'queued' }),
]);
await click(app, label);
assert.deepEqual(titles(), ['Gamma', 'Alpha', 'Beta', 'Zeta', 'Omega']);
assert.equal(heading(label).attributes['aria-sort'], 'ascending');
await click(app, label);
assert.deepEqual(titles(), ['Omega', 'Zeta', 'Alpha', 'Beta', 'Gamma']);
assert.equal(heading(label).attributes['aria-sort'], 'descending');
});
}
test('sorting saves the column and direction and restores both when the list reopens', async (t) => {
const list = [
story(1, 'Alpha', { author: 'Zed', site: 'z.test', status: 'ok', chapter_count: 2,
last_changed: '2026-01-01T00:00:00Z', last_checked: '2026-01-03T00:00:00Z' }),
story(2, 'Beta', { author: 'Anne', site: 'a.test', status: 'error', chapter_count: 10,
last_changed: '2026-01-02T00:00:00Z', last_checked: '2026-01-02T00:00:00Z' }),
];
await mount(t, list);
for (const [label, column, direction, order] of [
['Last change', 'last_changed', -1, ['Beta', 'Alpha']],
['Last change', 'last_changed', 1, ['Alpha', 'Beta']],
['Last check', 'last_checked', -1, ['Alpha', 'Beta']],
['Last check', 'last_checked', 1, ['Beta', 'Alpha']],
['Title', 'title', 1, ['Alpha', 'Beta']],
['Title', 'title', -1, ['Beta', 'Alpha']],
['Author', 'author', 1, ['Beta', 'Alpha']],
['Author', 'author', -1, ['Alpha', 'Beta']],
['Site', 'site', 1, ['Beta', 'Alpha']],
['Site', 'site', -1, ['Alpha', 'Beta']],
['Status', 'status', 1, ['Beta', 'Alpha']],
['Status', 'status', -1, ['Alpha', 'Beta']],
['Chapters', 'chapter_count', -1, ['Beta', 'Alpha']],
['Chapters', 'chapter_count', 1, ['Alpha', 'Beta']],
]) {
await click(app, label);
assert.deepEqual(JSON.parse(window.localStorage.getItem(SORT_KEY)), { column, direction });
await mount(t, list);
assert.deepEqual(titles(), order);
assert.equal(heading(label).attributes['aria-sort'], direction === 1 ? 'ascending' : 'descending');
}
});
for (const saved of ['not JSON', 'null', '{}', '{"column":"message","direction":-1}',
'{"column":"last_changed","direction":0}', '{"column":"last_checked","direction":"-1"}']) {
test(`invalid saved sorting falls back to title order: ${saved}`, async (t) => {
window.localStorage.setItem(SORT_KEY, saved);
await mount(t, [story(2, 'Beta'), story(1, 'Alpha')]);
assert.deepEqual(titles(), ['Alpha', 'Beta']);
assert.equal(heading('Title').attributes['aria-sort'], 'ascending');
await click(app, 'Title');
assert.deepEqual(JSON.parse(window.localStorage.getItem(SORT_KEY)), { column: 'title', direction: -1 });
});
}
test('sorting works when local storage cannot be read or written', async (t) => {
window.localStorage.getItem = () => { throw new Error('Storage blocked'); };
window.localStorage.setItem = () => { throw new Error('Storage full'); };
await mount(t, [story(2, 'Beta'), story(1, 'Alpha')]);
assert.deepEqual(titles(), ['Alpha', 'Beta']);
await click(app, 'Title');
assert.deepEqual(titles(), ['Beta', 'Alpha']);
assert.equal(heading('Title').attributes['aria-sort'], 'descending');
assert.equal(toasts.textContent, '');
});
test('date sorting compares instants, handles invalid dates and breaks ties by title and id', async (t) => {
await mount(t, [
story(7, 'Missing', { last_changed: 'not a date' }),
story(6, 'Unknown'),
story(5, 'Epoch', { last_changed: '1970-01-01T00:00:00Z' }),
story(4, 'Before epoch', { last_changed: '1969-01-01T00:00:00Z' }),
story(3, 'Zulu', { last_changed: '2026-01-02T00:00:00+02:00' }),
story(2, 'alpha', { last_changed: '2026-01-01T23:00:00Z' }),
story(1, 'Alpha', { last_changed: '2026-01-01T23:00:00Z' }),
story(8, '', { last_changed: '2026-01-01T23:00:00Z' }),
]);
await click(app, 'Last change');
assert.deepEqual(titles(), ['Alpha', 'alpha', 'https://example.com/8', 'Zulu', 'Epoch', 'Before epoch', 'Missing', 'Unknown']);
assert.equal(cell(rows()[6], 'Last change').textContent, 'not a date');
assert.equal(cell(rows()[7], 'Last change').textContent, '');
await click(app, 'Last change');
assert.deepEqual(titles(), ['Before epoch', 'Epoch', 'Zulu', 'Alpha', 'alpha', 'https://example.com/8', 'Missing', 'Unknown']);
});
test('headings sort the filtered list after the initial story request finishes', async (t) => {
let resolve;
await mount(t, () => new Promise((done) => { resolve = done; }));
assert.equal(find(app, 'button', 'Last check'), undefined);
input(field(app, 'Filter stories'), 'Reader');
resolve([
story(1, 'Alpha', { last_checked: '2026-01-01T00:00:00Z' }),
story(2, 'Beta', { last_checked: '2026-01-02T00:00:00Z' }),
]);
await settle();
await click(app, 'Last check');
assert.deepEqual(titles(), ['Beta', 'Alpha']);
});
test('filtering and job refreshes keep the selected sort without dropping hidden stories', async (t) => {
let list = [
story(1, 'Alpha', { last_changed: '2026-01-01T00:00:00Z' }),
story(2, 'Beta', { last_changed: '2026-01-02T00:00:00Z' }),
story(3, 'Gamma', { last_changed: '2026-01-03T00:00:00Z', author: 'Someone else' }),
];
const backend = await mount(t, () => list);
await click(app, 'Last change');
input(field(app, 'Filter stories'), 'Reader');
assert.deepEqual(titles(), ['Beta', 'Alpha']);
backend.on('GET', 'jobs', [{ id: 'job', story_id: 1, status: 'running' }]);
startPolling();
await settle();
list = [{ ...list[0], last_changed: '2026-01-04T00:00:00Z' }, ...list.slice(1)];
backend.on('GET', 'jobs', [{ id: 'job', story_id: 1, status: 'done' }]);
await refreshJobs();
await settle();
assert.equal(backend.sent('GET', 'stories').length, 2);
assert.equal(heading('Last change').attributes['aria-sort'], 'descending');
assert.equal(document.activeElement, find(app, 'button', 'Last change'));
assert.deepEqual(titles(), ['Alpha', 'Beta']);
input(field(app, 'Filter stories'), 'missing');
assert.match(app.textContent, /No stories match/);
input(field(app, 'Filter stories'), '');
assert.deepEqual(titles(), ['Alpha', 'Gamma', 'Beta']);
await click(app, 'Last check');
assert.deepEqual(titles(), ['Alpha', 'Beta', 'Gamma']);
await click(app, 'Last change');
assert.deepEqual(titles(), ['Alpha', 'Gamma', 'Beta']);
});
test('date columns keep each timestamp on one line, with last change before last check', async (t) => {
const changed = '2026-01-02T03:04:00Z';
const checked = '2026-01-03T05:06:00Z';
await mount(t, [story(1, 'Alpha', { last_changed: changed, last_checked: checked })]);
const headings = nodes(app).filter((el) => el.tagName === 'th').map((el) => el.textContent);
assert.deepEqual(headings, ['Title', 'Author', 'Site', 'Chapters', 'Pages', 'Status', 'Last change', 'Last check']);
for (const [label, value] of [['Last change', changed], ['Last check', checked]]) {
const dateCell = cell(rows()[0], label);
const time = nodes(dateCell).find((el) => el.tagName === 'time');
assert.equal(time.attributes.datetime, value);
assert.equal(time.attributes.title, fmtDate(value));
assert.equal(nodes(time).filter((el) => el.tagName === 'br').length, 0);
assert.equal(time.textContent, fmtDate(value));
}
});
test('status messages move into accessible pill descriptions without hiding held or chapter counts', async (t) => {
await mount(t, [
story(1, 'Alpha', { offsite_count: 3, revised_count: 2, message: 'Updated 2 chapters', upload_held: true, approval_held: true }),
story(2, 'Beta', { status: 'error', message: '<script>Connection failed</script>' }),
story(3, 'Gamma'),
story(4, 'New', { status: '', message: '' }),
story(5, 'Queued', { status: 'queued', message: '' }),
]);
for (const [row, label, message] of [
[rows()[0], 'ok', 'Updated 2 chapters'],
[rows()[1], 'error', '<script>Connection failed</script>'],
[rows()[2], 'ok', 'No changes'],
]) {
const status = cell(row, 'Status');
const pill = status.children[0];
assert.equal(pill.textContent, label);
assert.equal(pill.attributes.title, message);
assert.equal(pill.attributes['aria-description'], message);
assert.equal(pill.attributes.tabindex, '0');
assert.ok(!status.textContent.includes(message));
assert.equal(nodes(status).filter((el) => el.tagName === 'small' || el.tagName === 'script').length, 0);
}
assert.match(cell(rows()[0], 'Status').textContent, /held/);
const chapters = cell(rows()[0], 'Chapters');
assert.match(chapters.textContent, /^42/);
assert.equal(nodes(chapters).filter((el) => el.className.startsWith('badge')).length, 0);
const counts = nodes(chapters).filter((el) => el.tagName === 'small');
assert.deepEqual(counts.map((el) => el.textContent), ['3 removed', '2 revised']);
assert.deepEqual(counts.map((el) => el.attributes.title), [
'Chapters the site no longer lists', 'Chapters the site changed since Bindery added them',
]);
assert.equal(cell(rows()[1], 'Chapters').textContent, '42');
assert.equal(cell(rows()[3], 'Status').textContent, '');
assert.equal(cell(rows()[4], 'Status').textContent, 'queued');
});
tests/js/story-tab.test.mjs
+1 -1
View File
@@ -31,7 +31,7 @@ test('an open story has a tab with its title, also on its chapter pages', async
t.after(stopPolling);
const backend = server(t);
backend.on('GET', 'auth/check', { authenticated: true, mode: 'none', user: '' });
backend.on('GET', 'config', { metadata_fields: {}, mode_labels: {}, library_available: false });
backend.on('GET', 'config', { metadata_fields: {}, mode_labels: {}, library_available: false, download_fields: {} });
backend.on('GET', 'jobs', []);
backend.on('GET', 'stories', []);
backend.on('GET', 'settings', { recent_chapters: 0, personal_ini: '' });
tests/js/upload-hold.test.mjs
+245 -18
View File
@@ -7,7 +7,7 @@ const { settingsPage } = await import('../../bindery/static/js/settings.js');
const { storyPage } = await import('../../bindery/static/js/story.js');
const { storiesPage } = await import('../../bindery/static/js/stories.js');
const config = { job_list_max_hours: 720, job_list_max_count: 500 };
const config = { cwa_enabled: true, job_list_max_hours: 720, job_list_max_count: 500, download_fields: {} };
const book = { id: 7, uuid: 'book-7', title: 'Library Title', authors: ['Writer'] };
const details = (story, extra = {}) => ({
story: { id: 1, title: 'Story', author: 'Writer', url: 'https://example.com/story', site: 'example.com',
@@ -15,7 +15,7 @@ const details = (story, extra = {}) => ({
calibre_id: null, calibre_uuid: null, ingested_at: null, upload_held: false, needs_approval: false, ...story },
chapters: [],
recent_chapters: 0,
uploads_paused: false,
pause_uploads: false,
book: null,
...extra,
});
@@ -27,28 +27,135 @@ async function renderStory(t, data) {
backend.on('GET', 'jobs?story_id=1', []);
backend.on('GET', 'jobs', []);
const main = new Element('main');
const cleanup = storyPage(main, { storyId: '1' }, { metadata_fields: {}, mode_labels: {} });
const cleanup = storyPage(main, { storyId: '1' }, { cwa_enabled: true, metadata_fields: {}, mode_labels: {} });
t.after(() => cleanup?.());
await settle();
const summary = nodes(main).find((el) => el.className === 'card story-summary');
return { backend, notice: nodes(summary).find((el) => el.className === 'notice') };
return { backend, summary, notice: nodes(summary).find((el) => el.className === 'notice') };
}
test('the upload switches load and save on the settings page', async (t) => {
const backend = server(t);
backend.on('GET', 'settings', { recent_chapters: 0, hold_new_links: true, pause_uploads: false, personal_ini: '' });
backend.on('PUT', 'settings', {});
backend.on('GET', 'settings', { recent_chapters: 0, link_books: true, hold_new_links: true, pause_uploads: false,
ingest_available: true, personal_ini: '' });
backend.on('PUT', 'settings', { link_books: false, hold_new_links: true, pause_uploads: true, ingest_available: true });
backend.on('GET', 'rules', { rules: [], paragraphs: false });
backend.on('GET', 'stories', []);
const main = new Element('main');
settingsPage(main, {}, config);
await settle();
const card = section(main, 'Uploads to CWA');
const [hold, pause] = nodes(card).filter((el) => el.attributes.type === 'checkbox');
assert.deepEqual([hold.checked, pause.checked], [true, false]);
const [link, hold, pause] = nodes(card).filter((el) => el.attributes.type === 'checkbox');
assert.deepEqual([link.checked, hold.checked, pause.checked], [true, true, false]);
assert.equal(find(card, 'label', ' Link books to CWA library').children[0], link);
assert.match(card.textContent, /send a plain ePub to INGEST_DIR/);
const warning = nodes(card).find((el) => el.attributes.id === 'link-note');
assert.equal(warning.className, 'notice');
assert.equal(warning.hidden, true);
assert.equal(link.attributes['aria-describedby'], '');
assert.match(warning.textContent, /Updates without a saved CWA book ID create duplicate books if CWA imports them/);
assert.equal(nodes(section(main, 'Book Downloads')).find((el) => el.attributes.id === 'ingest-note').hidden, true);
assert.equal(pause.disabled, false);
assert.equal(hold.disabled, false);
assert.equal(link.disabled, false);
assert.equal(nodes(card).find((el) => el.attributes.id === 'library-note').hidden, true);
change(link, false);
assert.equal(warning.hidden, false);
assert.equal(link.attributes['aria-describedby'], warning.attributes.id);
assert.equal(hold.disabled, true);
assert.equal(hold.checked, true);
change(link, true);
assert.equal(warning.hidden, true);
assert.equal(hold.disabled, false);
assert.equal(hold.checked, true);
change(link, false);
change(pause, true);
await click(card, 'Save');
assert.deepEqual(backend.sent('PUT', 'settings')[0].body, { hold_new_links: true, pause_uploads: true });
assert.deepEqual(backend.sent('PUT', 'settings')[0].body, { link_books: false, hold_new_links: true, pause_uploads: true });
assert.deepEqual([link.checked, hold.checked, pause.checked], [false, true, true]);
assert.equal(hold.disabled, true);
assert.equal(warning.hidden, false);
});
for (const ingestAvailable of [true, false]) {
test(`missing library sources disable linking and approval with ingest available ${ingestAvailable}`, async (t) => {
const backend = server(t);
const saved = { link_books: true, linking_available: false, hold_new_links: true,
pause_uploads: false, ingest_available: ingestAvailable };
backend.on('GET', 'settings', saved);
backend.on('PUT', 'settings', saved);
backend.on('GET', 'rules', { rules: [], paragraphs: false });
backend.on('GET', 'stories', []);
const main = new Element('main');
settingsPage(main, {}, config);
await settle();
const card = section(main, 'Uploads to CWA');
const [link, hold, pause] = nodes(card).filter((el) => el.attributes.type === 'checkbox');
const note = nodes(card).find((el) => el.attributes.id === 'library-note');
assert.equal(link.checked, false);
assert.equal(link.disabled, true);
assert.equal(hold.checked, true);
assert.equal(hold.disabled, true);
assert.equal(pause.checked, !ingestAvailable);
assert.equal(pause.disabled, !ingestAvailable);
assert.equal(note.hidden, false);
assert.equal(note.className, 'notice');
assert.match(note.textContent, /CWA_URL and CALIBRE_DB_PATH are not set/);
assert.match(note.textContent, /Set one or both and restart Bindery/);
assert.equal(link.attributes['aria-describedby'], 'library-note link-note');
assert.equal(nodes(card).find((el) => el.attributes.id === 'link-note').hidden, false);
await click(card, 'Save');
assert.equal(backend.sent('PUT', 'settings')[0].body.link_books, false);
assert.equal(link.checked, false);
assert.equal(link.disabled, true);
});
}
for (const linkBooks of [true, false]) {
test(`missing INGEST_DIR forces pause and shows a highlighted note with linking ${linkBooks}`, async (t) => {
const backend = server(t);
const saved = { link_books: linkBooks, hold_new_links: false, pause_uploads: true, ingest_available: false };
backend.on('GET', 'settings', saved);
backend.on('PUT', 'settings', saved);
backend.on('GET', 'rules', { rules: [], paragraphs: false });
backend.on('GET', 'stories', []);
const main = new Element('main');
settingsPage(main, {}, config);
await settle();
const card = section(main, 'Uploads to CWA');
const [link, hold, pause] = nodes(card).filter((el) => el.attributes.type === 'checkbox');
assert.equal(link.checked, linkBooks);
assert.equal(hold.disabled, !linkBooks);
assert.equal(nodes(card).find((el) => el.attributes.id === 'link-note').hidden, linkBooks);
assert.equal(pause.checked, true);
assert.equal(pause.disabled, true);
const note = nodes(section(main, 'Book Downloads')).find((el) => el.attributes.id === 'ingest-note');
assert.equal(note.hidden, false);
assert.equal(note.className, 'notice');
assert.equal(pause.attributes['aria-describedby'], note.attributes.id);
assert.match(note.textContent, /INGEST_DIR is not set/);
await click(card, 'Save');
assert.equal(pause.checked, true);
assert.equal(pause.disabled, true);
assert.equal(backend.sent('PUT', 'settings')[0].body.pause_uploads, true);
});
}
test('a saved approval preference stays checked but disabled while linking is off', async (t) => {
const backend = server(t);
backend.on('GET', 'settings', { link_books: false, hold_new_links: true, pause_uploads: false, ingest_available: true });
backend.on('GET', 'rules', { rules: [], paragraphs: false });
backend.on('GET', 'stories', []);
const main = new Element('main');
settingsPage(main, {}, config);
await settle();
const card = section(main, 'Uploads to CWA');
const [link, hold] = nodes(card).filter((el) => el.attributes.type === 'checkbox');
assert.equal(hold.disabled, true);
assert.equal(hold.checked, true);
change(link, true);
assert.equal(hold.disabled, false);
assert.equal(hold.checked, true);
});
test('a held book with a new link names the book and sends it on approval', async (t) => {
@@ -57,12 +164,12 @@ test('a held book with a new link names the book and sends it on approval', asyn
{ book },
);
const { backend, notice } = await renderStory(t, data);
assert.match(notice.textContent, /waits for your approval\. The next upload replaces the ePub of library book 7: Library Title · Writer\./);
assert.match(notice.textContent, /^Approve replacing the ePub of library book 7: Library Title · Writer\./);
backend.on('POST', 'stories/1/approve', { story: data.story, job: { id: 'j1' } });
await click(notice, 'Approve and send');
await settle();
assert.deepEqual(backend.sent('POST', 'stories/1/approve')[0].body, { uuid: 'book-7' });
assert.match(ui.toasts.textContent, /Approved\. Bindery rebuilds the book and sends it\./);
assert.match(ui.toasts.textContent, /Approved\. Rebuild queued\./);
});
test('a wrong link can go to CWA as a new book instead', async (t) => {
@@ -81,9 +188,50 @@ test('a wrong link can go to CWA as a new book instead', async (t) => {
);
});
for (const uuid of [null, 'book-7']) {
test(`approval while paused saves the decision without promising an upload: ${uuid}`, async (t) => {
const data = details(
{ calibre_id: uuid ? 7 : null, calibre_uuid: uuid, upload_held: true, needs_approval: true },
{ link_books: true, pause_uploads: true },
);
const { backend, notice } = await renderStory(t, data);
assert.ok(find(notice, 'button', 'Approve'));
assert.equal(find(notice, 'button', 'Approve and send'), undefined);
assert.equal(find(notice, 'button', 'Send as a new book instead'), undefined);
backend.on('POST', 'stories/1/approve', { story: data.story, job: null });
await click(notice, 'Approve');
assert.deepEqual(backend.sent('POST', 'stories/1/approve')[0].body, { uuid });
assert.equal(ui.toasts.children.at(-1).textContent, 'Approved');
});
}
test('approval as a new book while paused keeps the pause in the confirmation', async (t) => {
const data = details(
{ calibre_id: 7, calibre_uuid: 'book-7', upload_held: true, needs_approval: true },
{ link_books: true, pause_uploads: true },
);
const { backend, notice } = await renderStory(t, data);
const confirm = t.mock.method(window, 'confirm', () => true);
backend.on('POST', 'stories/1/link', data.story);
backend.on('POST', 'stories/1/approve', { story: data.story, job: null });
await click(notice, 'Approve as a new book instead');
assert.match(confirm.mock.calls[0].arguments[0], /approve this story as a new book\? Uploads remain paused/);
assert.match(confirm.mock.calls[0].arguments[0], /you get a duplicate/);
assert.deepEqual(backend.sent('POST', 'stories/1/approve')[0].body, { uuid: null });
assert.equal(ui.toasts.children.at(-1).textContent, 'Approved');
});
test('a saved ID without a UUID still offers approval', async (t) => {
const { notice } = await renderStory(t, details(
{ calibre_id: 7, ingested_at: '2026-01-01', upload_held: true, needs_approval: true },
));
assert.ok(find(notice, 'button', 'Approve and send'));
assert.match(notice.textContent, /library book 7/);
});
test('a new story asks for approval to go to CWA as a new book', async (t) => {
const { notice } = await renderStory(t, details({ needs_approval: true }));
assert.match(notice.textContent, /The next upload goes to CWA as a new book\./);
assert.match(notice.textContent, /^Approve this story as a new CWA book\./);
assert.ok(find(notice, 'button', 'Approve'));
assert.equal(find(notice, 'button', 'Send as a new book instead'), undefined);
});
@@ -91,25 +239,103 @@ test('a new story asks for approval to go to CWA as a new book', async (t) => {
test('a paused upload points to the settings', async (t) => {
const { notice } = await renderStory(
t,
details({ calibre_id: 7, calibre_uuid: 'book-7', ingested_at: '2026-01-01', upload_held: true }, { book, uploads_paused: true }),
details({ calibre_id: 7, calibre_uuid: 'book-7', ingested_at: '2026-01-01', upload_held: true }, { book, pause_uploads: true }),
);
assert.match(notice.textContent, /Uploads to CWA are paused/);
assert.equal(find(notice, 'a', 'Settings').attributes.href, '#/settings');
assert.equal(nodes(notice).filter((el) => el.tagName === 'button').length, 0);
});
for (const savedId of [null, 7]) {
for (const paused of [false, true]) {
for (const needsApproval of [false, true]) {
test(`linking off hides approval with saved ID ${savedId}, pause ${paused}, and approval ${needsApproval}`, async (t) => {
const { notice } = await renderStory(t, details(
{ calibre_id: savedId, calibre_uuid: savedId ? 'book-7' : null,
upload_held: true, needs_approval: needsApproval },
{ link_books: false, pause_uploads: paused },
));
if (savedId !== null && !paused) {
assert.equal(notice, undefined);
return;
}
assert.equal(nodes(notice).filter((el) => el.tagName === 'button').length, 0);
assert.doesNotMatch(notice.textContent, /waits for your approval|as a new book|Bindery sends this book/);
if (savedId === null) {
assert.match(notice.textContent, /If CWA imports them, updates will create duplicate books/);
assert.equal(notice.className, 'notice');
} else {
assert.doesNotMatch(notice.textContent, /duplicate books/);
}
if (paused) {
assert.match(notice.textContent, /Uploads to CWA are paused/);
assert.equal(find(notice, 'a', 'Settings').attributes.href, '#/settings');
}
});
}
}
}
test('linking off warns that repeated exports without a saved ID create duplicates in CWA', async (t) => {
const { notice, summary } = await renderStory(t, details(
{ ingested_at: '2026-01-01', needs_approval: true, upload_held: true },
{ link_books: false },
));
assert.match(notice.textContent, /If CWA imports them, updates will create duplicate books/);
assert.match(summary.textContent, /Sent to ingest folder/);
assert.doesNotMatch(summary.textContent, /Sent to CWA, unlinked/);
assert.equal(nodes(notice).filter((el) => el.tagName === 'button').length, 0);
});
test('a story that went to CWA without a link shows no approval until Bindery finds its book', async (t) => {
const { notice } = await renderStory(t, details({ ingested_at: '2026-01-01', needs_approval: true }));
assert.equal(notice, undefined);
});
test('the story list marks a held book', async (t) => {
test('missing INGEST_DIR points to the environment rather than the disabled pause switch', async (t) => {
const { notice } = await renderStory(t, details(
{ upload_held: true }, { pause_uploads: true, ingest_available: false },
));
assert.equal(notice.textContent, 'Set INGEST_DIR and restart Bindery to enable uploads.');
assert.equal(nodes(notice).filter((el) => el.tagName === 'button').length, 0);
});
test('missing library sources leave uploads enabled with the duplicate warning', async (t) => {
const { notice } = await renderStory(t, details(
{ ingested_at: '2026-01-01', upload_held: true, needs_approval: true },
{ link_books: false },
));
assert.match(notice.textContent, /If CWA imports them, updates will create duplicate books/);
assert.equal(nodes(notice).filter((el) => el.tagName === 'button').length, 0);
});
test('a new book asks for approval when library linking is enabled', async (t) => {
const { notice } = await renderStory(t, details(
{ upload_held: true, needs_approval: true },
{ link_books: true },
));
assert.match(notice.textContent, /^Approve this story as a new CWA book\./);
assert.ok(find(notice, 'button', 'Approve and send'));
});
test('approval before a build still shows the pause and never offers to send', async (t) => {
const { notice } = await renderStory(t, details(
{ needs_approval: true }, { pause_uploads: true },
));
assert.match(notice.textContent, /Uploads to CWA are paused\. Resume them in Settings\./);
assert.ok(find(notice, 'button', 'Approve'));
assert.equal(find(notice, 'button', 'Approve and send'), undefined);
});
test('the story list marks only an approval hold, not every unsent book', async (t) => {
const backend = server(t);
backend.on('GET', 'stories', [
{ id: 1, url: 'u1', title: 'Held', author: 'A', site: 's', chapter_count: 1, offsite_count: 0, status: 'ok',
message: '', upload_held: true },
{ id: 2, url: 'u2', title: 'Sent', author: 'A', site: 's', chapter_count: 1, offsite_count: 0, status: 'ok',
message: '', upload_held: false },
{ id: 1, url: 'u1', title: 'Held', sort_title: 'held', author: 'A', site: 's', chapter_count: 1, offsite_count: 0, status: 'ok',
message: '', upload_held: true, approval_held: true },
{ id: 2, url: 'u2', title: 'Sent', sort_title: 'sent', author: 'A', site: 's', chapter_count: 1, offsite_count: 0, status: 'ok',
message: '', upload_held: false, approval_held: false },
{ id: 3, url: 'u3', title: 'Paused', sort_title: 'paused', author: 'A', site: 's', chapter_count: 1, offsite_count: 0, status: 'ok',
message: '', upload_held: true, approval_held: false },
]);
backend.on('GET', 'jobs', []);
const main = new Element('main');
@@ -119,4 +345,5 @@ test('the story list marks a held book', async (t) => {
const held = nodes(main).filter((el) => el.tagName === 'tr').filter((row) => find(row, 'span', 'held'));
assert.equal(held.length, 1);
assert.match(held[0].textContent, /^Held/);
assert.match(find(held[0], 'span', 'held').attributes.title, /waits for your approval/);
});
tests/test_api.py
+188
View File
@@ -10,6 +10,7 @@ import pytest
from fake_cwa import FakeCwa
from itsdangerous import TimestampSigner
from test_build import PNG
from werkzeug.security import generate_password_hash
from bindery import auth, config, orchestrator, web
from bindery.store import Store
@@ -70,6 +71,7 @@ def test_cwa_login(client, cwa):
"authenticated": False,
"mode": "cwa",
"user": None,
"admin": False,
"logout_url": None,
}
assert login(client, password="wrong").status_code == 401
@@ -81,6 +83,20 @@ def test_cwa_login(client, cwa):
assert client.get("/api/stories").status_code == 401
def test_cwa_that_accepts_every_login_cannot_check_a_password(client, cwa, caplog):
cwa.anonymous = True
for password in ("pw", "wrong"):
response = login(client, password=password)
assert response.status_code == 401
assert response.json["error"] == "Cannot check the login with CWA. The log has the details."
assert "CWA accepts every OPDS login" in caplog.text
assert client.get("/api/stories").status_code == 401
# These are not wrong passwords, so they lock no one out.
cwa.anonymous = False
orchestrator.cwa()._login_test = None
assert login(client).status_code == 200
def test_a_session_ends_after_one_day(client, cwa, monkeypatch):
response = client.post(
"/api/auth/login", json={"username": "reader", "password": "pw", "remember": True}, headers=H
@@ -155,16 +171,105 @@ def test_proxy_header(client, monkeypatch):
"authenticated": True,
"mode": "proxy",
"user": "alice",
"admin": True,
"logout_url": "https://auth.example.com/logout",
}
@pytest.mark.parametrize(("user", "admin"), [("alice", True), ("Alice", True), ("bob", False)])
def test_only_an_admin_changes_the_global_settings(client, monkeypatch, tmp_path, user, admin):
monkeypatch.setattr(config, "PERSONAL_INI", tmp_path / "personal.ini")
monkeypatch.setattr(config, "AUTH_METHOD", "proxy")
monkeypatch.setattr(config, "ADMIN_USERS", ["alice"])
sid = orchestrator.store.create_story("https://www.royalroad.com/fiction/1", "www.royalroad.com", "S", "W", [])
headers = {**H, "X-Remote-User": user}
status = 200 if admin else 403
assert client.get("/api/auth/check", headers=headers).json["admin"] is admin
assert ("personal_ini" in client.get("/api/settings", headers=headers).json) is admin
for data in ({"recent_chapters": 5}, {"personal_ini": "[defaults]\n"}):
assert client.put("/api/settings", json=data, headers=headers).status_code == status
assert (orchestrator.get_settings()["recent_chapters"] == 5) is admin
assert config.PERSONAL_INI.exists() is admin
rules = {"rules": [{"pattern": "a", "replacement": "b"}]}
response = client.put("/api/rules", json=rules, headers=headers)
assert response.status_code == status
assert bool(orchestrator.store.rules(None)) is admin
if not admin:
assert response.json == {"error": "Only an admin can do this. BINDERY_ADMIN_USERS lists the admins."}
# Each user can read the global rules, and change the rules and the settings of a story.
assert client.get("/api/rules", headers=headers).status_code == 200
assert client.put(f"/api/stories/{sid}/rules", json=rules, headers=headers).status_code == 200
assert client.put(f"/api/stories/{sid}/settings", json={"recheck_days": 3}, headers=headers).status_code == 200
def test_the_api_key_and_single_user_modes_are_admins(client, monkeypatch):
monkeypatch.setattr(config, "ADMIN_USERS", ["alice"])
assert client.get("/api/auth/check").json["admin"] # the none mode has no users
monkeypatch.setattr(config, "AUTH_METHOD", "proxy")
monkeypatch.setattr(config, "API_KEY", "secret")
key = {"X-Api-Key": "secret"}
assert client.get("/api/auth/check", headers=key).json["admin"]
assert client.put("/api/settings", json={"recent_chapters": 2}, headers=key).status_code == 200
assert auth.is_admin(auth.Identity("bob", "proxy")) is False
monkeypatch.setattr(config, "AUTH_METHOD", "local")
assert auth.is_admin(auth.Identity("bob", "session"))
# Without a list, each user is an admin.
monkeypatch.setattr(config, "AUTH_METHOD", "proxy")
monkeypatch.setattr(config, "ADMIN_USERS", [])
assert auth.is_admin(auth.Identity("bob", "proxy"))
@pytest.mark.parametrize(
("trusted", "address", "accepted"),
[
(None, "127.0.0.1", True),
(None, "::1", True),
(None, "172.18.0.3", True),
(None, "192.168.1.20", True),
(None, "fd00::5", True),
(None, "::ffff:10.1.2.3", True),
(None, "203.0.113.9", False),
(None, "2001:db8::1", False),
(None, "", False),
(["172.18.0.3"], "172.18.0.3", True),
(["172.18.0.3"], "172.18.0.4", False),
(["172.18.0.3"], "127.0.0.1", False),
(["10.0.0.0/8", "fd00::/64"], "fd00::7", True),
(["10.0.0.0/8", "fd00::/64"], "192.168.1.20", False),
],
)
def test_only_a_trusted_address_can_send_the_proxy_header(client, monkeypatch, caplog, trusted, address, accepted):
monkeypatch.setattr(config, "AUTH_METHOD", "proxy")
if trusted is not None:
monkeypatch.setattr(config, "PROXY_AUTH_TRUSTED_IPS", trusted)
environ = {"REMOTE_ADDR": address}
response = client.get("/api/stories", headers={"X-Remote-User": "alice"}, environ_base=environ)
assert response.status_code == (200 if accepted else 401)
assert (
client.get("/api/auth/check", headers={"X-Remote-User": "alice"}, environ_base=environ).json["authenticated"]
is accepted
)
assert ("which is not in PROXY_AUTH_TRUSTED_IPS" in caplog.text) is not accepted
# A change needs the header too, from the same address.
change = client.post(
"/api/stories", json={"urls": []}, headers={**H, "X-Remote-User": "alice"}, environ_base=environ
)
assert change.status_code == (200 if accepted else 401)
@pytest.mark.parametrize(
("setting", "value", "error"),
[
("AUTH_METHOD", "cwa", "AUTH_METHOD=cwa needs CWA_URL or CWA_DB_PATH"),
("AUTH_METHOD", "cwa-login", "AUTH_METHOD must be none, local, cwa or proxy, not 'cwa-login'"),
("PROXY_AUTH_USER_HEADER", "", "AUTH_METHOD=proxy needs PROXY_AUTH_USER_HEADER"),
("PROXY_AUTH_TRUSTED_IPS", [], "AUTH_METHOD=proxy needs PROXY_AUTH_TRUSTED_IPS, the addresses of the proxy"),
(
"PROXY_AUTH_TRUSTED_IPS",
["10.0.0.5", "proxy.local"],
"PROXY_AUTH_TRUSTED_IPS must be a list of addresses and networks: "
"'proxy.local' does not appear to be an IPv4 or IPv6 network",
),
],
)
def test_incomplete_login_settings_refuse_every_request(client, monkeypatch, setting, value, error):
@@ -183,6 +288,51 @@ def test_incomplete_login_settings_refuse_every_request(client, monkeypatch, set
assert client.get("/").status_code == 200
@pytest.mark.parametrize(
("mode", "allowed", "host", "accepted"),
[
# Without a login, only an address or a name of a private network.
("none", [], "localhost", True),
("none", [], "127.0.0.1:8085", True),
("none", [], "192.168.1.5:8085", True),
("none", [], "[fd00::5]:8085", True),
("none", [], "bindery:8085", True),
("none", [], "nas.local", True),
("none", [], "NAS.Home.Arpa.", True),
("none", [], "bindery.example.com", False),
("none", [], "rebind.attacker.example:8085", False),
("none", [], "notlocal", True),
("none", [], "evil.com.local.attacker.example", False),
# With a login, each name.
("local", [], "bindery.example.com", True),
("proxy", [], "bindery.example.com", True),
# A list counts in each mode.
("none", ["bindery.example.com"], "bindery.example.com:443", True),
("none", ["bindery.example.com"], "localhost", False),
("none", ["bindery.example.com"], "x.bindery.example.com", False),
("local", [".example.com"], "bindery.example.com", True),
("local", [".example.com"], "example.com", True),
("local", [".example.com"], "badexample.com", False),
("local", ["example.com"], "attacker.example", False),
("none", ["*"], "anything.example", True),
],
)
def test_bindery_answers_only_to_its_host_names(client, monkeypatch, mode, allowed, host, accepted):
monkeypatch.setattr(config, "AUTH_METHOD", mode)
monkeypatch.setattr(config, "LOCAL_USERNAME", "reader")
monkeypatch.setattr(config, "LOCAL_PASSWORD_HASH", generate_password_hash("pw", method="pbkdf2:sha256:1000"))
monkeypatch.setattr(config, "ALLOWED_HOSTS", allowed)
for path in ("/", "/api/auth/check", "/static/style.css"):
response = client.get(path, headers={"Host": host})
assert response.status_code == (200 if accepted else 400), path
if not accepted:
assert response.json == {
"error": f"Bindery does not answer to the host name {host}. Add it to ALLOWED_HOSTS."
}
# The health check of the container works under each name.
assert client.get("/api/health", headers={"Host": host}).status_code == 200
def test_api_key(client, cwa, monkeypatch):
monkeypatch.setattr(config, "API_KEY", "k" * 32)
assert client.get("/api/stories", headers={"X-Api-Key": "wrong"}).status_code == 401
@@ -222,6 +372,7 @@ def test_the_cover_of_a_story(client):
response = client.get(f"/api/stories/{story_id}/cover?v={version}")
assert (response.status_code, response.mimetype, response.data) == (200, "image/png", PNG)
assert response.cache_control.max_age == 365 * 24 * 60 * 60
assert response.cache_control.private and not response.cache_control.public
# A new cover gets a new version, so the browser loads it.
store.put_image(story_id, "https://example.com/new.jpg", "jpg", "image/jpeg", b"new", is_cover=True)
@@ -280,15 +431,52 @@ def test_delete_revisions(client):
assert [r["rev"] for r in client.get(f"/api/chapters/{chapter_id}/diff").json["revisions"]] == [1, 3]
@pytest.mark.parametrize(
"bad",
[
{"recent_chapters": "many"},
{"main_jobs_count": 100000},
{"personal_ini": "[defaults\nfoo"},
{"link_books": "yes"},
],
)
def test_a_bad_setting_changes_no_setting(client, monkeypatch, tmp_path, bad):
ini = tmp_path / "personal.ini"
ini.write_text("[defaults]\n")
monkeypatch.setattr(config, "PERSONAL_INI", ini)
monkeypatch.setattr(config, "CWA_URL", "https://cwa.invalid")
before = client.get("/api/settings").json
good = {
"link_books": False,
"hold_new_links": True,
"pause_uploads": True,
"recent_chapters": 4,
"story_jobs_hours": 1,
"personal_ini": "[defaults]\nis_adult:true\n",
}
assert client.put("/api/settings", json={**good, **bad}, headers=H).status_code == 400
assert client.get("/api/settings").json == before
def test_settings(client, monkeypatch, tmp_path):
ini = tmp_path / "personal.ini"
ini.write_text("[defaults]\nis_adult:true\n")
monkeypatch.setattr(config, "PERSONAL_INI", ini)
monkeypatch.setattr(config, "CWA_URL", "")
monkeypatch.setattr(config, "CALIBRE_DB_PATH", None)
assert client.get("/api/settings").json == {
"recent_chapters": 0,
"link_books": False,
"linking_available": False,
"hold_new_links": False,
"pause_uploads": False,
"ingest_available": True,
"personal_ini": "[defaults]\nis_adult:true\n",
"opds_enabled": False,
"opds_password_set": False,
"opds_user": None,
"download_template": "{title_sort} - {authors}",
"download_date_format": "%b, %Y",
"story_jobs_hours": 36,
"story_jobs_count": 10,
"main_jobs_hours": 6,
tests/test_api_flows.py
+149 -4
View File
@@ -46,6 +46,7 @@ def test_add_list_download_and_cancel_jobs(client, story):
assert result.json["mode"] == "metadata"
app_config = client.get("/api/config").json
assert app_config["metadata_fields"]["title"]
assert app_config["cwa_enabled"] is True
assert (app_config["story_jobs_hours"], app_config["story_jobs_count"]) == (36, 10)
assert (app_config["main_jobs_hours"], app_config["main_jobs_count"]) == (6, 25)
assert (app_config["job_list_max_hours"], app_config["job_list_max_count"]) == (720, 500)
@@ -64,6 +65,14 @@ def test_settings_and_inclusion_persist_and_delete_cascades(client, story):
)
assert response.json["mode"] == "rebuild"
assert not o.store.get_chapter(cid)["included"]
response = client.post(f"/api/stories/{sid}/chapters/move", json={"chapter_ids": [cid], "after": None}, headers=H)
assert response.json["mode"] == "rebuild"
move = client.post(f"/api/stories/{sid}/chapters/move", json={"chapter_ids": [cid], "after": 999}, headers=H)
assert move.status_code == 404
# The story has only this chapter, so it cannot go.
response = client.post(f"/api/stories/{sid}/chapters/delete", json={"chapter_ids": [cid]}, headers=H)
assert response.status_code == 400
assert "at least one chapter" in response.json["error"]
o.store.set_rules(sid, [{"pattern": "a", "replacement": "b", "enabled": True}])
response = client.delete(f"/api/stories/{sid}", headers=H)
assert response.json == {"deleted": sid}
@@ -82,7 +91,7 @@ def test_library_search_link_import_and_unlink(client, story, monkeypatch):
cwa.get_book.return_value = book.to_dict()
monkeypatch.setattr(o, "cwa", lambda: cwa)
monkeypatch.setattr(config, "CWA_URL", "https://cwa.example")
assert client.get("/api/library/candidates").json[0]["calibre_id"] == 7
assert client.get("/api/library/candidates").json[0]["id"] == 7
assert client.get("/api/library/search?q=Title").json[0]["tracked_as"] is None
cwa.search.assert_called_once_with("Title")
linked = client.post(f"/api/stories/{sid}/link", json={"uuid": "book-7"}, headers=H)
@@ -91,6 +100,7 @@ def test_library_search_link_import_and_unlink(client, story, monkeypatch):
assert client.get("/api/library/search?q=Title").json[0]["tracked_as"] == "Story"
cwa.get_book.side_effect = CwaError("offline")
assert client.get(f"/api/stories/{sid}").json["library_error"] == "offline"
cwa.get_book.assert_called_with("book-7", timeout=5) # the page does not wait long for CWA
cwa.get_book.side_effect = None
cwa.get_book.return_value = None
assert client.post(f"/api/stories/{sid}/link", json={"uuid": "missing"}, headers=H).status_code == 404
@@ -132,13 +142,148 @@ def test_download_builds_from_stored_chapters_without_changing_them(client, stor
response = client.get(f"/api/stories/{sid}/epub")
assert response.status_code == 200
assert response.data.startswith(b"PK")
assert "filename=A_B.epub" in response.headers["Content-Disposition"]
assert 'filename="A_B - Writer.epub"' in response.headers["Content-Disposition"]
assert o.store.revision_html(cid, 1) == original
response.close()
def test_unexpected_backend_error_returns_json(client, monkeypatch):
def download_name(client, sid):
response = client.get(f"/api/stories/{sid}/epub")
response.close()
return response.headers["Content-Disposition"]
def test_download_template_names_the_file(client, story):
sid, _ = story
site = {"title": "The Book", "authors": ["Ann Writer"], "series": "The Saga", "series_index": 2.0}
o.store.update_story(sid, metahtml=metahtml(), title="The Book", site_values=site)
assert client.get("/api/config").json["download_fields"]["id"].startswith("The CWA book ID")
saved = client.put("/api/settings", json={"download_template": " {series:|| - }{title} - {id} "}, headers=H)
assert saved.json["download_template"] == "{series:|| - }{title} - {id}"
assert f'filename="The Saga - The Book - {sid}.epub"' in download_name(client, sid)
# The default moves the article to the end, as Calibre's default does.
client.put("/api/settings", json={"download_template": ""}, headers=H)
assert 'filename="Book, The - Ann Writer.epub"' in download_name(client, sid)
client.put("/api/settings", json={"download_template": "{series_sort} - {title} - {id}"}, headers=H)
assert f'filename="Saga, The - The Book - {sid}.epub"' in download_name(client, sid)
# The ID is that of the CWA book, when the story has one.
o.store.update_story(sid, calibre_id=77, pinned={"title": "Caf\u00e9", "series": ["", None]})
client.put("/api/settings", json={"download_template": "{series:|| - }{title} - {id}"}, headers=H)
name = download_name(client, sid)
assert 'filename="Cafe - 77.epub"' in name and "filename*=UTF-8''Caf%C3%A9%20-%2077.epub" in name
# A template with folders gives the name of the file alone.
client.put("/api/settings", json={"download_template": "{author_sort}/{title}"}, headers=H)
assert "filename*=UTF-8''Caf%C3%A9.epub" in download_name(client, sid)
# An empty template restores the default.
saved = client.put("/api/settings", json={"download_template": " "}, headers=H).json
assert saved["download_template"] == "{title_sort} - {authors}"
assert "filename*=UTF-8''Caf%C3%A9%20-%20Ann%20Writer.epub" in download_name(client, sid)
def test_date_format_of_download_names(client, story):
sid, _ = story
o.store.update_story(sid, metahtml=metahtml(), site_values={"pubdate": "2020-05-06T00:00:00"})
client.put("/api/settings", json={"download_template": "{title} {pubdate}"}, headers=H)
assert 'filename="Story May, 2020.epub"' in download_name(client, sid)
saved = client.put("/api/settings", json={"download_date_format": " %Y-%m-%d "}, headers=H).json
assert saved["download_date_format"] == "%Y-%m-%d"
assert "filename=Story 2020-05-06.epub" in download_name(client, sid).replace('"', "")
response = client.put("/api/settings", json={"download_date_format": 5}, headers=H)
assert (response.status_code, response.json) == (400, {"error": "The date format must be text"})
response = client.put("/api/settings", json={"download_date_format": "%Y" * 51}, headers=H)
assert (response.status_code, response.json) == (400, {"error": "A date format can have 100 characters at most"})
# An empty format restores the default.
assert client.put("/api/settings", json={"download_date_format": ""}, headers=H).json["download_date_format"] == (
"%b, %Y"
)
def test_download_template_that_fails_gives_the_title(client, story, caplog):
sid, _ = story
o.store.update_story(sid, metahtml=metahtml())
for template in ("{title:d}", "{series}/{series_index}"):
client.put("/api/settings", json={"download_template": template}, headers=H)
assert "filename=Story.epub" in download_name(client, sid)
assert "The download template fails for https://www.royalroad.com/fiction/1234" in caplog.text
assert caplog.text.count("The download template fails") == 1
def test_bad_download_template_is_not_saved(client):
client.put("/api/settings", json={"download_template": "{title}"}, headers=H)
for template, error in [
("{title:uppercase()}", "Bindery does not support template functions, such as {title:uppercase()}"),
("{title", "The braces of the template do not match: expected '}' before end of string"),
(7, "The download template must be text"),
]:
response = client.put("/api/settings", json={"download_template": template, "recent_chapters": 9}, headers=H)
assert (response.status_code, response.json) == (400, {"error": error})
assert client.get("/api/settings").json["download_template"] == "{title}"
assert client.get("/api/settings").json["recent_chapters"] == 0
def test_unexpected_backend_error_returns_json(client, monkeypatch, caplog):
monkeypatch.setattr(o, "list_stories", Mock(side_effect=RuntimeError("broken")))
response = client.get("/api/stories")
assert response.status_code == 500
assert response.json == {"error": "RuntimeError: broken"}
assert response.json == {"error": "Bindery hit an unexpected error. The log has the details."}
assert "RuntimeError: broken" in caplog.text
@pytest.mark.parametrize(
("method", "path", "body", "error"),
[
("post", "stories", {"urls": 5}, "urls must be a list"),
("post", "stories/{sid}/chapters", {"chapter_ids": 5}, "chapter_ids must be a list"),
("post", "stories/{sid}/chapters/move", {"chapter_ids": [[1]]}, "chapter_ids must be a whole number"),
("post", "stories/{sid}/chapters/move", {"chapter_ids": [1], "after": [2]}, "after must be a whole number"),
("post", "stories/{sid}/chapters/delete", {"chapter_ids": "all"}, "chapter_ids must be a list"),
("post", "stories/{sid}/download", {"mode": "recheck", "chapter_ids": ["x"]}, "chapter_ids must be a whole"),
("post", "stories/{sid}/download", {"force_fields": "title"}, "force_fields must be a list"),
("post", "stories/{sid}/download", {"force_fields": [["title"]]}, "force_fields must be a list of names"),
("put", "stories/{sid}/settings", {"locked_fields": [{}]}, "locked_fields must be a list of names"),
("put", "stories/{sid}/settings", {"recheck_days": [7]}, "recheck_days must be a whole number"),
("put", "rules", {"rules": ["a"]}, "rules must be a list of rules"),
("post", "rules/preview", {"rules": [], "chapter_id": None}, "chapter_id must be a whole number"),
("post", "chapters/{cid}/revision", {"rev": {}}, "rev must be a whole number"),
("post", "library/import", {"books": 7}, "books must be a list"),
("post", "library/import", {"books": [{"id": "x"}]}, "id must be a whole number"),
],
)
def test_a_value_of_the_wrong_type_is_a_bad_request(client, story, method, path, body, error):
sid, cid = story
response = getattr(client, method)("/api/" + path.format(sid=sid, cid=cid), json=body, headers=H)
assert response.status_code == 400
assert error in response.json["error"]
def test_chapter_requests_without_ids_name_what_to_select(client, story):
sid, _ = story
for body in ({}, {"chapter_ids": None}):
response = client.post(f"/api/stories/{sid}/chapters/delete", json=body, headers=H)
assert (response.status_code, response.json["error"]) == (400, "Select the chapters to delete")
def test_a_story_change_does_not_ask_cwa_for_the_book(client, story, monkeypatch):
sid, _ = story
o.store.update_story(sid, calibre_id=7, calibre_uuid="book-7")
monkeypatch.setattr(config, "CWA_URL", "https://cwa.example")
cwa = Mock()
monkeypatch.setattr(o, "cwa", lambda: cwa)
response = client.put(f"/api/stories/{sid}/settings", json={"recheck_days": 4}, headers=H)
assert response.json["recheck_days"] == 4
assert "chapters" not in response.json
cwa.get_book.assert_not_called()
def test_settings_are_saved_by_name(client):
with pytest.raises(ValueError, match="Unknown settings: \\['recent'\\]"):
o.save_settings(recent=3)
# The route passes only the known settings on.
response = client.put("/api/settings", json={"recent": 3, "recent_chapters": 4, "main_jobs_count": 9}, headers=H)
assert (response.status_code, response.json["recent_chapters"], response.json["main_jobs_count"]) == (200, 4, 9)
for data, error in [
({"recent_chapters": -1}, "The number of recent chapters cannot be negative"),
({"main_jobs_count": 501}, "main_jobs_count cannot be more than 500"),
({"story_jobs_hours": "x"}, "story_jobs_hours must be a whole number"),
]:
assert client.put("/api/settings", json=data, headers=H).json == {"error": error}
tests/test_build.py
+1 -1
View File
@@ -76,7 +76,7 @@ def test_a_title_with_entities_builds_and_imports_in_fanficfare_form(tmp_path):
ElementTree.fromstring(z.read(name))
# An import stores the same title as a download, so the next update finds no rename.
assert fff.read_epub(out).chapters[0]["title"] == title
assert fff.plain_title(title) == "Folly & Magic <1>"
assert fff.unescape_title(title) == "Folly & Magic <1>"
assert fff.escape_title("Folly & Magic <1>") == title
tests/test_build_keys.py
+389
View File
@@ -0,0 +1,389 @@
"""A book is built and sent only when its chapters, rules or metadata changed."""
import dataclasses
from unittest.mock import Mock
import pytest
from test_api import H
from test_api import client as client
from test_deliver import STORY_URL, add_story, ingest_files, linked_story, site, update
from test_deliver import fake as fake
from test_deliver import jobs as jobs
from test_deliver import story_epub as story_epub
from bindery import config, orchestrator, pages
from bindery import store as storage
from bindery.jobs import DownloadMode, Job
ONE = [(1, "One", "a")]
@pytest.fixture(autouse=True)
def builds(fake, jobs, monkeypatch, tmp_path):
"""Count the builds, in a folder of built books that starts empty."""
monkeypatch.setattr(config, "TMP_DIR", tmp_path / "tmp")
config.TMP_DIR.mkdir()
spy = Mock(side_effect=orchestrator._build)
monkeypatch.setattr(orchestrator, "_build", spy)
return spy
def story(story_id):
return orchestrator.store.get_story(story_id)
def test_an_unchanged_story_is_not_built_or_sent_again(builds, monkeypatch):
story_id = linked_story(ONE)
site(monkeypatch, ONE)
assert "library book 1" in update(story_id)
changed = story(story_id)["last_changed"]
orchestrator.store.update_story(story_id, last_changed="2020-01-01T00:00:00")
for mode in (DownloadMode.NEW, DownloadMode.RECHECK_ALL, DownloadMode.METADATA, DownloadMode.FULL):
assert update(story_id, mode).startswith("No changes")
assert changed and story(story_id)["last_changed"] == "2020-01-01T00:00:00"
assert builds.call_count == 1
assert len(ingest_files()) == 2 # one epub and its sidecar
@pytest.mark.parametrize(
"change", ["chapter", "revision", "renamed", "rule", "global rule", "paragraphs", "metadata", "included"]
)
def test_each_change_builds_and_sends_the_book(builds, monkeypatch, change):
story_id = linked_story([(1, "One", "a"), (2, "Two", "b")])
site(monkeypatch, [(1, "One", "a"), (2, "Two", "b")])
update(story_id)
previous = "2020-01-01T00:00:00"
orchestrator.store.update_story(story_id, last_changed=previous)
if change == "chapter":
site(monkeypatch, [(1, "One", "a"), (2, "Two", "b"), (3, "Three", "c")])
elif change == "revision":
site(monkeypatch, [(1, "One", "a"), (2, "Two", "new text")])
elif change == "renamed":
site(monkeypatch, [(1, "One", "a"), (2, "Renamed", "b")])
elif change == "rule":
orchestrator.save_rules(story_id, [{"pattern": "a", "replacement": "b"}])
elif change == "global rule":
orchestrator.save_rules(None, [{"pattern": "a", "replacement": "b"}])
elif change == "paragraphs":
orchestrator.save_rules(story_id, [], paragraphs=True)
elif change == "metadata":
orchestrator.store.update_story(story_id, pinned={"title": "Edited"})
else:
chapter = orchestrator.store.list_chapters(story_id)[0]
orchestrator.store.update_chapter(chapter["id"], included=0)
mode = DownloadMode.RECHECK_ALL if change == "revision" else DownloadMode.NEW
assert "library book 1" in update(story_id, mode)
assert builds.call_count == 2
assert len(ingest_files()) == 4
changed_at = story(story_id)["last_changed"]
assert (changed_at != previous) == (change in ("chapter", "revision"))
assert update(story_id).startswith("No changes")
assert story(story_id)["last_changed"] == changed_at
def test_a_held_book_that_did_not_change_is_not_built_again(builds, monkeypatch):
monkeypatch.setattr(config, "INGEST_DIR", None)
site(monkeypatch, ONE)
job = Job(mode=DownloadMode.NEW, url=STORY_URL)
assert "Not sent. INGEST_DIR is not set." in orchestrator._run_site_job(job)
changed = story(job.story_id)["last_changed"]
orchestrator.store.update_story(job.story_id, last_changed="2020-01-01T00:00:00")
assert "Not sent. INGEST_DIR is not set." in update(job.story_id)
assert changed and story(job.story_id)["last_changed"] == "2020-01-01T00:00:00"
assert builds.call_count == 1
# The book changes. Bindery builds it, also while it cannot send it.
site(monkeypatch, [*ONE, (2, "Two", "b")])
update(job.story_id)
assert builds.call_count == 2
assert story(job.story_id)["last_changed"] != "2020-01-01T00:00:00"
def test_a_released_book_goes_out_without_another_build(builds, jobs):
story_id = add_story(ingested_at="2026-01-01T00:00:00", calibre_id=1, calibre_uuid="uuid-1")
orchestrator.save_settings(pause_uploads=True)
orchestrator._run_rebuild(Job(mode=DownloadMode.REBUILD, story_id=story_id))
assert builds.call_count == 1 and ingest_files() == []
orchestrator.save_settings(pause_uploads=False)
assert "library book 1" in orchestrator._run_rebuild(jobs.get_next())
assert builds.call_count == 1
assert len(ingest_files()) == 2
def test_a_lost_file_is_built_again_only_to_send_it(builds):
story_id = add_story(ingested_at="2026-01-01T00:00:00", calibre_id=1, calibre_uuid="uuid-1")
orchestrator.save_settings(pause_uploads=True)
job = Job(mode=DownloadMode.REBUILD, story_id=story_id)
orchestrator._run_rebuild(job)
for path in config.TMP_DIR.iterdir():
path.unlink()
assert "pause" in orchestrator._run_rebuild(job)
assert builds.call_count == 1
orchestrator.store.set_setting(None, "pause_uploads", None)
assert "library book 1" in orchestrator._run_rebuild(job)
assert builds.call_count == 2
@pytest.mark.parametrize("last_changed", [None, "2020-01-01T00:00:00"])
def test_a_rebuild_without_a_change_does_nothing_unless_the_user_asks_for_it(builds, client, jobs, last_changed):
story_id = add_story(ingested_at="2026-01-01T00:00:00", calibre_id=1, calibre_uuid="uuid-1")
job = Job(mode=DownloadMode.REBUILD, story_id=story_id)
assert "library book 1" in orchestrator._run_rebuild(job)
orchestrator.store.update_story(story_id, last_changed=last_changed, message="before")
assert orchestrator._run_rebuild(job) == "No changes"
assert (story(story_id)["last_changed"], story(story_id)["message"]) == (last_changed, "before")
assert builds.call_count == 1
# The Rebuild button of the story page.
queued = client.post(f"/api/stories/{story_id}/download", json={"mode": "rebuild"}, headers=H).json
assert queued["options"] == {"force": True}
assert "library book 1" in orchestrator._run_rebuild(jobs.get_next())
assert builds.call_count == 2
assert len(ingest_files()) == 4
assert story(story_id)["last_changed"] == last_changed
# Other modes take no such option.
queued = client.post(f"/api/stories/{story_id}/download", json={"mode": "new"}, headers=H).json
assert queued["options"] == {}
def test_a_changed_link_gets_the_book_without_a_build(builds, fake):
story_id = add_story(ingested_at="2026-01-01T00:00:00", calibre_id=1, calibre_uuid="uuid-1")
job = Job(mode=DownloadMode.REBUILD, story_id=story_id)
orchestrator._run_rebuild(job)
fake.books.append(dataclasses.replace(fake.books[0], id=2, uuid="uuid-2", title="Other"))
orchestrator.link_story(story_id, "uuid-2")
assert "library book 2" in orchestrator._run_rebuild(job)
assert builds.call_count == 1
def test_a_failed_delivery_is_tried_again_without_a_build(builds, fake, monkeypatch):
story_id = linked_story(ONE)
site(monkeypatch, ONE)
send = orchestrator.IngestFolder.send_update
monkeypatch.setattr(orchestrator.IngestFolder, "send_update", Mock(side_effect=OSError("disk full")))
with pytest.raises(OSError, match="disk full"):
update(story_id)
monkeypatch.setattr(orchestrator.IngestFolder, "send_update", send)
assert "library book 1" in update(story_id)
assert builds.call_count == 1
@pytest.mark.parametrize("change", ["chapter", "revision"])
def test_chapter_timestamp_survives_a_failed_build_and_retry(builds, monkeypatch, change):
story_id = linked_story(ONE)
site(monkeypatch, ONE)
update(story_id)
previous = "2020-01-01T00:00:00"
orchestrator.store.update_story(story_id, last_changed=previous)
clock = ["2024-01-01T00:00:00"]
monkeypatch.setattr(storage, "now", lambda: clock[0])
site(monkeypatch, [*ONE, (2, "Two", "b")] if change == "chapter" else [(1, "One", "revised")])
job = Job(mode=DownloadMode.RECHECK_ALL, story_id=story_id)
build = orchestrator._build
monkeypatch.setattr(orchestrator, "_build", Mock(side_effect=OSError("disk full")))
with pytest.raises(OSError, match="disk full"):
orchestrator._run_site_job(job)
assert story(story_id)["last_changed"] == clock[0]
assert job.result["new"] or job.result["changed"]
clock[0] = "2024-02-01T00:00:00"
monkeypatch.setattr(orchestrator, "_build", build)
assert "library book 1" in orchestrator._run_site_job(job)
assert story(story_id)["last_changed"] == "2024-01-01T00:00:00"
assert update(story_id).startswith("No changes")
assert story(story_id)["last_changed"] == "2024-01-01T00:00:00"
def test_a_new_build_removes_the_older_file(builds, monkeypatch):
story_id = linked_story(ONE)
site(monkeypatch, ONE)
update(story_id)
(first,) = config.TMP_DIR.iterdir()
site(monkeypatch, [*ONE, (2, "Two", "b")])
update(story_id)
(second,) = config.TMP_DIR.iterdir()
assert first != second and second.name.startswith(f"story-{story_id}-")
@pytest.mark.parametrize(
("changes", "held", "built", "sent"),
[
({"calibre_id": 1, "calibre_uuid": "uuid-1"}, False, True, True),
({"ingested_at": "2026-01-01"}, False, True, True),
({"ingested_at": "2026-01-01"}, True, True, False),
({}, False, True, False),
({"ingested_at": "2026-01-01", "status": "error"}, False, False, False),
({"ingested_at": "2026-01-01", "no_metadata": True}, False, False, False),
],
)
def test_stories_of_an_older_bindery_get_their_keys(builds, changes, held, built, sent):
no_metadata = changes.pop("no_metadata", False)
story_id = add_story(**changes)
if no_metadata:
orchestrator.store.update_story(story_id, metahtml=None)
orchestrator.store.set_setting(story_id, "upload_held", held or None)
orchestrator.adopt_build_keys()
settings = orchestrator.store
assert bool(settings.get_setting(story_id, "built_key")) is built
assert bool(settings.get_setting(story_id, "sent_key")) is sent
if sent:
assert orchestrator._run_rebuild(Job(mode=DownloadMode.REBUILD, story_id=story_id)) == "No changes"
assert builds.call_count == 0
def test_a_story_with_a_waiting_job_gets_no_key(builds, jobs):
story_id = add_story(ingested_at="2026-01-01")
jobs.add(Job(mode=DownloadMode.NEW, story_id=story_id))
orchestrator.adopt_build_keys()
assert orchestrator.store.get_setting(story_id, "built_key") is None
def test_the_first_check_after_an_upgrade_sends_nothing(builds, monkeypatch):
story_id = linked_story(ONE)
orchestrator.store.update_story(story_id, pinned={"title": "My Title"}, site_values={})
orchestrator.adopt_build_keys()
site(monkeypatch, ONE)
assert update(story_id).startswith("No changes")
assert builds.call_count == 0 and ingest_files() == []
# Downloads in the browser
def download(client, story_id):
response = client.get(f"/api/stories/{story_id}/epub")
assert response.status_code == 200 and response.data.startswith(b"PK")
assert response.content_length == len(response.data)
response.close()
return response.data
def test_a_download_uses_the_built_book(builds, client, monkeypatch):
story_id = linked_story(ONE)
site(monkeypatch, ONE)
update(story_id)
(built,) = config.TMP_DIR.iterdir()
assert download(client, story_id) == download(client, story_id) == built.read_bytes()
assert builds.call_count == 1
def test_a_download_builds_a_book_that_has_no_file_or_that_changed(builds, client, monkeypatch):
story_id = add_story()
changed_at = story(story_id)["last_changed"]
first = download(client, story_id)
download(client, story_id)
assert builds.call_count == 1
orchestrator.save_rules(story_id, [{"pattern": "Zorian", "replacement": "Zach"}])
assert download(client, story_id) != first
assert builds.call_count == 2
assert len(list(config.TMP_DIR.iterdir())) == 1
# The build of a download is not the build of a job. The job sends the book.
assert orchestrator.store.get_setting(story_id, "built_key") is None
job = Job(mode=DownloadMode.REBUILD, story_id=story_id)
assert "library book 1" in orchestrator._run_rebuild(job)
assert builds.call_count == 2
assert story(story_id)["last_changed"] == changed_at
# Page counts
LONG = "A line of the story. " * 20
def listed_pages(story_id):
return {s["id"]: s["page_count"] for s in orchestrator.list_stories()}[story_id]
def test_a_story_has_no_page_count_until_its_book_is_built(builds, monkeypatch):
story_id = linked_story(ONE)
assert listed_pages(story_id) is None
site(monkeypatch, ONE)
update(story_id)
(built,) = config.TMP_DIR.iterdir()
assert listed_pages(story_id) == pages.epub_pages(built) == 1
def test_each_build_counts_the_pages_of_the_new_book(builds, client, monkeypatch):
story_id = linked_story(ONE)
site(monkeypatch, [*ONE, *[(n, f"Chapter {n}", "<br/>".join([LONG] * 31)) for n in range(2, 6)]])
update(story_id)
long_book = listed_pages(story_id)
# Each of the 124 lines is a paragraph and 6 lines of text.
assert long_book >= 4 * 31 * 7 // 31
# The count is of the built book, so a rule that removes text changes it without a download.
orchestrator.save_rules(story_id, [{"pattern": "(A line of the story. ){19}", "replacement": ""}])
download(client, story_id)
assert builds.call_count == 2
assert 0 < listed_pages(story_id) < long_book / 3
def test_a_book_from_before_the_page_count_is_counted_without_a_build(builds, client, monkeypatch):
story_id = linked_story(ONE)
site(monkeypatch, ONE)
update(story_id)
orchestrator.store.set_setting(story_id, "page_count", None)
download(client, story_id)
assert builds.call_count == 1
assert listed_pages(story_id) == 1
def test_a_book_that_cannot_be_counted_is_still_built(builds, monkeypatch):
def fail(path):
raise orchestrator.EpubError("unreadable")
monkeypatch.setattr(pages, "epub_pages", fail)
story_id = linked_story(ONE)
site(monkeypatch, ONE)
assert "library book 1" in update(story_id)
assert listed_pages(story_id) is None
def test_stop_tracking_deletes_the_built_book(builds, client):
story_id, other = add_story(), orchestrator.store.create_story("https://other", "s", "Other", "W", [])
download(client, story_id)
kept = config.TMP_DIR / f"story-{other}-0123456789abcdef.epub"
kept.write_bytes(b"other")
orchestrator.delete_story(story_id)
assert list(config.TMP_DIR.iterdir()) == [kept]
# Uploaded files
@pytest.fixture
def uploads(monkeypatch, tmp_path):
monkeypatch.setattr(config, "UPLOAD_DIR", tmp_path / "uploads")
config.UPLOAD_DIR.mkdir()
return config.UPLOAD_DIR
def test_a_cancelled_import_deletes_its_uploaded_file(builds, jobs, uploads, story_epub):
path = uploads / "one.epub"
path.write_bytes(story_epub)
queued = orchestrator.queue_epub_file(path, "story.epub")
assert "path" not in queued["options"]
assert orchestrator.cancel_job(queued["id"]) is True
assert not path.exists()
assert orchestrator.cancel_job(queued["id"]) is False
assert orchestrator.cancel_job("no-such-job") is False
def test_the_start_deletes_uploads_that_no_job_needs(builds, jobs, uploads, story_epub):
waiting, stale = uploads / "waiting.epub", uploads / "stale.epub"
for path in (waiting, stale):
path.write_bytes(story_epub)
orchestrator.queue_epub_file(waiting, "story.epub")
orchestrator.remove_stale_uploads()
assert list(uploads.iterdir()) == [waiting]
def test_a_missing_ingest_folder_fails_the_job_until_the_folder_exists(builds, jobs, monkeypatch, tmp_path):
monkeypatch.setattr(config, "INGEST_DIR", tmp_path / "not-mounted")
story_id = linked_story(ONE)
site(monkeypatch, ONE)
monkeypatch.setattr(orchestrator.time, "sleep", Mock())
jobs.add(Job(mode=DownloadMode.NEW, story_id=story_id))
job = orchestrator._run_next_job()
assert job.status == "error" and "does not exist. Check INGEST_DIR and the volume mount." in job.message
assert story(story_id)["status"] == "error"
assert not config.INGEST_DIR.exists()
config.INGEST_DIR.mkdir()
assert "library book 1" in update(story_id)
assert builds.call_count == 1
tests/test_calibre_db.py
+3 -2
View File
@@ -14,7 +14,6 @@ from werkzeug.security import generate_password_hash
from bindery import auth, calibre_db, config, orchestrator
from bindery.cwa import CwaClient
from bindery.ingest import IngestFolder
@pytest.fixture
@@ -54,6 +53,7 @@ def db_login(client, user_db, monkeypatch):
monkeypatch.setattr(config, "CWA_DB_PATH", user_db)
remote = Mock(spec=CwaClient)
remote.login_ok.return_value = True
remote.accepts_any_login.return_value = False
monkeypatch.setattr(orchestrator, "cwa", lambda: remote)
return remote
@@ -250,7 +250,8 @@ def test_library_database_failure_keeps_the_http_lookup(library_db, monkeypatch,
@pytest.mark.parametrize("saved_id,saved_uuid", [(None, None), (1, "uuid-42"), (42, None)])
def test_database_book_id_reaches_the_epub_sidecar(client, library_db, monkeypatch, tmp_path, saved_id, saved_uuid):
monkeypatch.setattr(config, "CALIBRE_DB_PATH", library_db)
monkeypatch.setattr(orchestrator, "ingest_folder", IngestFolder(tmp_path / "ingest"))
monkeypatch.setattr(config, "INGEST_DIR", tmp_path / "ingest")
config.INGEST_DIR.mkdir(exist_ok=True)
remote = CwaClient("")
remote.session.get = Mock(side_effect=AssertionError("No HTTP lookup needed"))
monkeypatch.setattr(orchestrator, "cwa", lambda: remote)
tests/test_config.py
+178 -1
View File
@@ -16,6 +16,14 @@ def reload_config(monkeypatch):
importlib.reload(config)
@pytest.fixture
def folders(monkeypatch, tmp_path):
"""Put the folders of Bindery in a temporary folder. The ingest folder does not exist yet."""
for name, folder in [("CONFIG_DIR", "config"), ("UPLOAD_DIR", "uploads"), ("TMP_DIR", "tmp"), ("INGEST_DIR", "in")]:
monkeypatch.setattr(config, name, tmp_path / folder)
monkeypatch.setattr(config, "PERSONAL_INI", tmp_path / "config" / "personal.ini")
def test_bool(monkeypatch):
for value in ("true", "Yes", " 1 ", "on", "y"):
monkeypatch.setenv("X_TEST", value)
@@ -24,7 +32,10 @@ def test_bool(monkeypatch):
monkeypatch.setenv("X_TEST", value)
assert not config._bool("X_TEST")
monkeypatch.delenv("X_TEST")
assert config._bool("X_TEST", "true")
assert config._bool("X_TEST", True)
# A value that is not clear gives the default.
monkeypatch.setenv("X_TEST", "maybe")
assert config._bool("X_TEST", True) and not config._bool("X_TEST")
def test_list(monkeypatch):
@@ -69,18 +80,22 @@ def test_values_from_the_environment(monkeypatch, reload_config):
monkeypatch.setenv("UPDATE_INTERVAL_HOURS", "1.5")
monkeypatch.setenv("DEFAULT_LOCKED_FIELDS", "title, authors")
monkeypatch.setenv("CWA_ALLOWED_USERS", "alice,bob")
monkeypatch.setenv("BINDERY_ADMIN_USERS", "alice")
monkeypatch.setenv("CONFIG_DIR", "/data/bindery")
monkeypatch.setenv("TRUST_PROXY", "yes")
monkeypatch.setenv("CWA_DB_PATH", " /cwa/app.db ")
monkeypatch.setenv("CALIBRE_DB_PATH", "/library/metadata.db")
monkeypatch.setenv("BINDERY_USERNAME", " reader ")
monkeypatch.setenv("BINDERY_PASSWORD_HASH", " hash ")
monkeypatch.setenv("ALLOWED_HOSTS", "Bindery.Example.com, .home.example")
reload_config()
assert config.ALLOWED_HOSTS == ["bindery.example.com", ".home.example"]
assert config.URL_BASE == "/bindery"
assert config.AUTH_METHOD == "cwa"
assert config.UPDATE_INTERVAL_HOURS == 1.5
assert config.DEFAULT_LOCKED_FIELDS == ["title", "authors"]
assert config.CWA_ALLOWED_USERS == ["alice", "bob"]
assert config.ADMIN_USERS == ["alice"]
assert str(config.DB_PATH) == "/data/bindery/bindery.db"
assert config.TRUST_PROXY is True
assert str(config.CWA_DB_PATH) == "/cwa/app.db"
@@ -97,10 +112,15 @@ def test_defaults(monkeypatch, reload_config):
"DEFAULT_LOCKED_FIELDS",
"LOG_LEVEL",
"PROXY_AUTH_USER_HEADER",
"PROXY_AUTH_TRUSTED_IPS",
"ALLOWED_HOSTS",
"BINDERY_ADMIN_USERS",
"CWA_ENABLED",
"SESSION_COOKIE_SECURE",
"TRUST_PROXY",
"CWA_DB_PATH",
"CALIBRE_DB_PATH",
"INGEST_DIR",
"BINDERY_USERNAME",
"BINDERY_PASSWORD_HASH",
):
@@ -112,9 +132,166 @@ def test_defaults(monkeypatch, reload_config):
assert config.DEFAULT_LOCKED_FIELDS == ["title"]
assert config.LOG_LEVEL == "INFO"
assert config.PROXY_AUTH_USER_HEADER == "X-Remote-User"
assert config.PROXY_AUTH_TRUSTED_IPS == [
"127.0.0.0/8",
"::1",
"10.0.0.0/8",
"172.16.0.0/12",
"192.168.0.0/16",
"fc00::/7",
"fe80::/10",
]
assert config.ALLOWED_HOSTS == []
assert config.ADMIN_USERS == []
assert config.CWA_ENABLED is False
assert config.IGNORED_SETTINGS == []
assert config.SESSION_COOKIE_SECURE is False
assert config.TRUST_PROXY is False
assert config.CWA_DB_PATH is None
assert config.CALIBRE_DB_PATH is None
assert config.INGEST_DIR is None
assert config.LOCAL_USERNAME == ""
assert config.LOCAL_PASSWORD_HASH == ""
@pytest.mark.parametrize("value", ["", " "])
def test_empty_ingest_directory_disables_uploads(monkeypatch, reload_config, value):
monkeypatch.setenv("INGEST_DIR", value)
reload_config()
assert config.INGEST_DIR is None
@pytest.mark.parametrize("value", [None, "", "false", " No ", "0", "maybe"])
def test_cwa_off_ignores_the_cwa_settings_but_not_the_ingest_folder(monkeypatch, reload_config, value):
if value is None:
monkeypatch.delenv("CWA_ENABLED")
else:
monkeypatch.setenv("CWA_ENABLED", value)
monkeypatch.setenv("CWA_URL", "https://cwa.invalid")
monkeypatch.setenv("CWA_USERNAME", "bindery")
monkeypatch.setenv("CWA_PASSWORD", "secret")
monkeypatch.setenv("CWA_DB_PATH", "/cwa/app.db")
monkeypatch.setenv("CALIBRE_DB_PATH", "/library/metadata.db")
monkeypatch.setenv("INGEST_DIR", "/books")
reload_config()
assert config.CWA_ENABLED is False
assert (config.CWA_URL, config.CWA_USERNAME, config.CWA_PASSWORD) == ("", "", "")
assert (config.CWA_DB_PATH, config.CALIBRE_DB_PATH) == (None, None)
assert config.IGNORED_SETTINGS == ["CWA_URL", "CWA_USERNAME", "CWA_PASSWORD", "CWA_DB_PATH", "CALIBRE_DB_PATH"]
assert str(config.INGEST_DIR) == "/books"
@pytest.mark.parametrize("value", ["true", " Yes ", "1", "on"])
def test_only_a_clear_true_turns_cwa_on(monkeypatch, reload_config, value):
monkeypatch.setenv("CWA_ENABLED", value)
monkeypatch.setenv("CWA_URL", "https://cwa.invalid")
reload_config()
assert config.CWA_ENABLED is True
assert config.CWA_URL == "https://cwa.invalid"
@pytest.mark.parametrize("ignored", [[], ["CWA_URL", "CALIBRE_DB_PATH"]])
def test_prepare_with_cwa_off_logs_nothing_about_missing_cwa(folders, monkeypatch, tmp_path, caplog, ignored):
monkeypatch.setattr(config, "CWA_ENABLED", False)
monkeypatch.setattr(config, "IGNORED_SETTINGS", ignored)
monkeypatch.setattr(config, "INGEST_DIR", None)
monkeypatch.setattr(config, "CWA_URL", "")
monkeypatch.setattr(config, "CALIBRE_DB_PATH", None)
with caplog.at_level("INFO"):
config.prepare()
lines = [r.getMessage() for r in caplog.records]
ignores = "CWA_ENABLED is not true. Bindery ignores CWA_URL, CALIBRE_DB_PATH. To use CWA, set CWA_ENABLED=true."
expected = [ignores] if ignored else []
assert [r.levelname for r in caplog.records if "CWA" in r.getMessage()] == ["WARNING"] * len(expected)
assert [line for line in lines if "CWA" in line or "INGEST" in line] == expected
@pytest.mark.parametrize(
("ingest", "url", "database", "expected"),
[
(False, "", None, ["INGEST_DIR is not set", "cannot find the library book of a story.", "remove CWA_ENABLED"]),
(True, "", None, ["so each update adds a new book to CWA.", "remove CWA_ENABLED"]),
(True, "", Path("/library/metadata.db"), ["CWA_URL is not set. Bindery cannot search", "remove CWA_ENABLED"]),
(False, "https://cwa.invalid", None, ["INGEST_DIR is not set", "remove CWA_ENABLED"]),
(True, "https://cwa.invalid", None, []),
],
)
def test_prepare_warns_about_each_missing_cwa_function(
folders, monkeypatch, tmp_path, caplog, ingest, url, database, expected
):
monkeypatch.setattr(config, "CWA_URL", url)
monkeypatch.setattr(config, "CALIBRE_DB_PATH", database)
monkeypatch.setattr(config, "INGEST_DIR", tmp_path if ingest else None)
with caplog.at_level("WARNING"):
config.prepare()
warnings = [r.getMessage() for r in caplog.records]
assert len(warnings) == len(expected)
for warning, text in zip(warnings, expected, strict=True):
assert text in warning
def test_prepare_without_cwa_creates_only_bindery_folders(folders, monkeypatch, tmp_path, caplog):
monkeypatch.setattr(config, "CWA_ENABLED", False)
monkeypatch.setattr(config, "INGEST_DIR", None)
monkeypatch.setattr(config, "CWA_URL", "")
monkeypatch.setattr(config, "CALIBRE_DB_PATH", None)
config.prepare()
assert config.PERSONAL_INI.exists()
assert {p.name for p in tmp_path.iterdir()} == {"config", "uploads", "tmp"}
assert caplog.records == []
@pytest.mark.parametrize(
("url", "database", "warns"),
[("", None, True), ("https://cwa.invalid", None, False), ("", Path("/library/metadata.db"), False)],
)
def test_prepare_warns_about_an_ingest_folder_without_a_library(
folders, monkeypatch, tmp_path, caplog, url, database, warns
):
monkeypatch.setattr(config, "CWA_URL", url)
monkeypatch.setattr(config, "CALIBRE_DB_PATH", database)
with caplog.at_level("WARNING"):
config.prepare()
assert ("each update adds a new book" in caplog.text) is warns
@pytest.mark.parametrize(
("name", "value", "error"),
[
("UPDATE_INTERVAL_HOURS", "daily", "UPDATE_INTERVAL_HOURS must be a number, not 'daily'"),
("LOG_LEVEL", "loud", "LOG_LEVEL must be one of DEBUG, INFO, WARNING, ERROR, CRITICAL, not 'LOUD'"),
],
)
def test_a_bad_value_stops_the_start_with_a_clear_message(monkeypatch, reload_config, name, value, error):
monkeypatch.setenv(name, value)
with pytest.raises(SystemExit) as stop:
reload_config()
assert str(stop.value) == error
def test_log_level_in_lower_case(monkeypatch, reload_config):
monkeypatch.setenv("LOG_LEVEL", " debug ")
reload_config()
assert config.LOG_LEVEL == "DEBUG"
@pytest.mark.parametrize(("username", "warns"), [("", True), ("bindery", False)])
def test_prepare_warns_about_a_cwa_password_without_a_user_name(
folders, monkeypatch, tmp_path, caplog, username, warns
):
monkeypatch.setattr(config, "CWA_URL", "https://cwa.invalid")
monkeypatch.setattr(config, "CWA_USERNAME", username)
monkeypatch.setattr(config, "CWA_PASSWORD", "secret")
with caplog.at_level("WARNING"):
config.prepare()
assert ("CWA_PASSWORD is set" in caplog.text) is warns
@pytest.mark.parametrize("exists", [False, True])
def test_prepare_does_not_make_the_ingest_folder(folders, monkeypatch, tmp_path, caplog, exists):
monkeypatch.setattr(config, "CWA_URL", "https://cwa.invalid")
if exists:
config.INGEST_DIR.mkdir()
config.prepare()
assert config.INGEST_DIR.exists() is exists
assert ("does not exist. Check INGEST_DIR and the volume mount" in caplog.text) is not exists
tests/test_cwa.py
+56 -2
View File
@@ -1,15 +1,17 @@
"""The OPDS client, the book finder, the login check, the ingest folder and the epub reader."""
import json
import time
from unittest.mock import Mock
import pytest
from fake_cwa import FakeBook, FakeCwa
from test_build import URL, chapters, metahtml
from bindery import fff
from bindery.cwa import BookFinder, CwaClient, CwaError, plain_title, search_text, title_rank
from bindery.cwa import LOGIN_TEST_SECONDS, BookFinder, CwaClient, CwaError, plain_title, search_text, title_rank
from bindery.epub import epub_metadata, epub_urls
from bindery.ingest import MAX_NAME_LENGTH, IngestFolder
from bindery.ingest import MAX_NAME_LENGTH, IngestError, IngestFolder
STORY_URL = "https://www.royalroad.com/fiction/1234"
@@ -63,6 +65,35 @@ def test_wrong_password_and_missing_address(fake):
CwaClient("").search("x")
def test_cwa_that_needs_no_login_works_without_an_account(fake):
fake.anonymous = True
open_client = CwaClient(fake.url)
assert [b.id for b in open_client.search("Another")] == [3]
assert open_client.get_book("uuid-2")["id"] == 2
fake.anonymous = False
with pytest.raises(CwaError, match="CWA asks for a login. Set CWA_USERNAME and CWA_PASSWORD."):
open_client.search("Test Story")
def test_a_login_check_does_not_use_the_session_of_the_account(fake, monkeypatch):
c = client(fake)
monkeypatch.setattr(c.session, "get", Mock(side_effect=AssertionError("The login check used the session")))
assert c.login_ok("bindery", "secret")
assert not c.login_ok("bindery", "wrong")
def test_the_test_of_the_logins_of_cwa_is_kept_for_a_time(fake, monkeypatch):
c = client(fake)
assert not c.accepts_any_login()
fake.anonymous = True
assert not c.accepts_any_login() # the last test still counts
assert len([r for r in fake.requests if r == "/opds"]) == 1
clock = time.monotonic()
monkeypatch.setattr(time, "monotonic", lambda: clock + LOGIN_TEST_SECONDS + 1)
assert c.accepts_any_login()
assert CwaClient(fake.url).accepts_any_login()
def test_get_book(fake):
c = client(fake)
assert c.get_book("uuid-2") == {
@@ -75,6 +106,17 @@ def test_get_book(fake):
assert c.get_book("nope") is None
def test_get_book_can_wait_for_a_shorter_time(fake, monkeypatch):
c = client(fake)
get = c.session.get
timeouts = []
monkeypatch.setattr(c.session, "get", lambda url, **kw: timeouts.append(kw["timeout"]) or get(url, **kw))
c.get_book("uuid-2", timeout=5)
c.get_book("uuid-2")
c.search("Story")
assert timeouts == [5, 60, 60]
@pytest.mark.parametrize("book_uuid", ["../opds", "uuid-2?x=1", "uuid-2#x", "uuid%2F2", ""])
def test_get_book_refuses_a_uuid_that_changes_the_path(book_uuid):
# Nothing listens on this port, so a request would raise CwaError.
@@ -176,6 +218,7 @@ def test_epub_reader(story_epub, tmp_path):
def test_ingest_uses_part_names_and_unique_names(tmp_path):
folder = IngestFolder(tmp_path / "ingest")
folder.path.mkdir()
epub = tmp_path / "x.epub"
epub.write_bytes(b"PK")
first = folder.ingest(epub, 'A: "Story" / Name')
@@ -187,6 +230,7 @@ def test_ingest_uses_part_names_and_unique_names(tmp_path):
def test_send_update_writes_sidecar(tmp_path):
folder = IngestFolder(tmp_path / "ingest")
folder.path.mkdir()
epub = tmp_path / "x.epub"
epub.write_bytes(b"PK")
path = folder.send_update(epub, 7, "T" * 300)
@@ -195,3 +239,13 @@ def test_send_update_writes_sidecar(tmp_path):
manifest = json.loads((folder.path / (path.name + ".cwa.json")).read_text())
assert manifest == {"action": "add_format", "book_id": 7}
assert path.read_bytes() == b"PK"
def test_a_missing_ingest_folder_is_an_error_and_is_not_made(tmp_path, story_epub):
source = tmp_path / "book.epub"
source.write_bytes(story_epub)
folder = IngestFolder(tmp_path / "missing" / "ingest")
for send in (lambda: folder.ingest(source, "Title"), lambda: folder.send_update(source, 5, "Title")):
with pytest.raises(IngestError, match="does not exist. Check INGEST_DIR and the volume mount."):
send()
assert not (tmp_path / "missing").exists()
tests/test_deliver.py
+23 -14
View File
@@ -13,7 +13,6 @@ from bindery import config, fff, orchestrator
from bindery.diffing import text_hash
from bindery.epub import EpubError
from bindery.fff import SiteSnapshot, StoryError
from bindery.ingest import IngestFolder
from bindery.jobs import DownloadMode, Job, JobQueue, QueueStatus
from bindery.store import Store
@@ -35,7 +34,8 @@ def fake(story_epub, monkeypatch, tmp_path):
monkeypatch.setattr(config, "CWA_USERNAME", "bindery")
monkeypatch.setattr(config, "CWA_PASSWORD", "secret")
monkeypatch.setattr(orchestrator, "store", Store(tmp_path / "bindery.db"))
monkeypatch.setattr(orchestrator, "ingest_folder", IngestFolder(tmp_path / "ingest"))
monkeypatch.setattr(config, "INGEST_DIR", tmp_path / "ingest")
config.INGEST_DIR.mkdir()
monkeypatch.setattr(orchestrator, "finder", orchestrator.BookFinder())
yield server
server.close()
@@ -51,17 +51,18 @@ def add_story(**changes):
def ingest_files():
folder = orchestrator.ingest_folder.path
folder = config.INGEST_DIR
return sorted(p.name for p in folder.iterdir()) if folder.exists() else []
def opf(name):
with zipfile.ZipFile(orchestrator.ingest_folder.path / name) as z:
with zipfile.ZipFile(config.INGEST_DIR / name) as z:
return z.read("content.opf").decode()
def rebuild(story_id):
return orchestrator._run_rebuild(Job(mode=DownloadMode.REBUILD, story_id=story_id))
"""A rebuild that the user asks for. It builds and sends the book also when the book did not change."""
return orchestrator._run_rebuild(Job(mode=DownloadMode.REBUILD, story_id=story_id, options={"force": True}))
def test_first_version_goes_in_as_a_new_book(fake):
@@ -80,7 +81,7 @@ def test_ingested_story_is_found_and_updated_with_a_sidecar(fake):
assert (story["calibre_id"], story["calibre_uuid"]) == (1, "uuid-1")
epub, manifest = ingest_files()
assert manifest == epub + ".cwa.json"
assert json.loads((orchestrator.ingest_folder.path / manifest).read_text()) == {
assert json.loads((config.INGEST_DIR / manifest).read_text()) == {
"action": "add_format",
"book_id": 1,
}
@@ -94,12 +95,13 @@ def test_update_fails_while_cwa_has_no_such_book(fake):
assert ingest_files() == [] # a second plain epub would make a duplicate book
def test_update_fails_without_cwa_url(fake, monkeypatch):
def test_update_exports_without_a_library_connection(fake, monkeypatch):
monkeypatch.setattr(config, "CWA_URL", "")
story_id = add_story(ingested_at="2026-01-01T00:00:00")
with pytest.raises(orchestrator.StoryError, match="CWA_URL is not set"):
rebuild(story_id)
assert ingest_files() == []
assert rebuild(story_id) == "Sent to ingest folder"
assert orchestrator.store.get_story(story_id)["status"] == "ok"
assert orchestrator.store.get_setting(story_id, "upload_held") is None
assert len(ingest_files()) == 1
def test_linked_story_still_updates_when_cwa_is_down(fake):
@@ -140,7 +142,14 @@ def test_import_from_the_library(fake):
def test_library_candidates_and_search(fake):
candidates = orchestrator.library_candidates()
assert candidates == [
{"calibre_id": 1, "uuid": "uuid-1", "title": "Test Story", "authors": "Writer", "site": "www.royalroad.com"}
{
"id": 1,
"uuid": "uuid-1",
"title": "Test Story",
"authors": ["Writer"],
"publisher": "www.royalroad.com",
"formats": ["epub"],
}
]
story_id = add_story()
orchestrator.link_story(story_id, "uuid-1")
@@ -306,7 +315,7 @@ def test_the_site_cover_goes_in_the_book(fake, jobs, monkeypatch, tmp_path):
assert update(story_id).startswith("1 new chapter")
assert orchestrator.store.cover(story_id)["data"] == PNG
(epub,) = [n for n in ingest_files() if n.endswith(".epub")]
with zipfile.ZipFile(orchestrator.ingest_folder.path / epub) as z:
with zipfile.ZipFile(config.INGEST_DIR / epub) as z:
assert "OEBPS/cover.xhtml" in z.namelist()
@@ -598,8 +607,8 @@ def test_paused_uploads_go_to_cwa_when_the_pause_ends(fake, jobs):
orchestrator.save_settings(pause_uploads=True)
assert "pause" in rebuild(story_id)
assert ingest_files() == []
assert orchestrator.approve_upload(story_id, "uuid-1")["mode"] == DownloadMode.REBUILD.value
assert "pause" in orchestrator._run_rebuild(jobs.get_next())
assert orchestrator.approve_upload(story_id, "uuid-1") is None
assert jobs.get_next() is None
orchestrator.save_settings(pause_uploads=True)
assert jobs.get_next() is None
orchestrator.save_settings(pause_uploads=False)
tests/test_filename_template.py
+228
View File
@@ -0,0 +1,228 @@
"""The file name templates, against the results of Calibre 9.2.1.
Each expected text is what Calibre gives for the same book and template.
"""
import pytest
from bindery import filename_template as ft
BOOK = {
"title": "The Book",
"authors": ["Ann Writer", "Ben Reader"],
"series": "The Saga",
"series_index": 2.345,
"languages": ["eng", "fra"],
"tags": ["Space", "Fiction"],
"pubdate": "2020-05-06T12:00:00+00:00",
"publisher": "site.example",
}
NO_SERIES = {**BOOK, "series": "", "series_index": None}
def expand(template, order="library_order", **changes):
return ft.expand(template, ft.prepare({**BOOK, **changes}, 42, order))
# name, template, text from Calibre, series index
REFERENCE = [
("title", "{title}", "Book, The", 2.345),
("series", "{series}", "Saga, The", 2.345),
("fraction", "{series_index}", "2.35", 2.345),
("languages", "{languages}", "eng,fra", 2.345),
("tags", "{tags}", "Fiction, Space", 2.345),
("send_date", "{pubdate}", "May, 2020", 2.345),
("space_padding", "x{series_index:>5s}x", "x 2x", 2),
("zero_padding", "{series_index:0>3s}", "002", 2),
("numeric_format", "{series_index:03d}", "002", 2),
("first_character", "{author_sort[0]}", "W", 2),
("conditional", "{series:|| - }{title}", "Saga, The - Book, The", 2),
("case_insensitive", "{TITLE}", "Book, The", 2),
("unknown", "x{not_a_field}x", "xx", 2),
("literal_braces", "{{title}}", "{title}", 2),
]
@pytest.mark.parametrize("name,template,expected,index", REFERENCE, ids=[case[0] for case in REFERENCE])
def test_reference_cases(name, template, expected, index):
assert expand(template, series_index=index) == expected
def test_reference_cases_for_a_missing_series_and_for_alphabetic_order():
values = ft.prepare(NO_SERIES, 42)
assert ft.expand("{series}{series_index:| - | - }{title}", values) == "Book, The"
assert expand("{title}", "strictly_alphabetic") == "The Book"
assert expand("{series}", "strictly_alphabetic") == "The Saga"
def test_series_sort_is_a_bindery_field():
assert expand("{series_sort}") == expand("{series_sort}", "strictly_alphabetic") == "Saga, The"
assert ft.expand("x{series_sort}x", ft.prepare(NO_SERIES, 42)) == "xx"
@pytest.mark.parametrize(
"template,expected",
[
("{authors}", "Ann Writer & Ben Reader"),
("{author}", "Ann Writer & Ben Reader"),
("{author_sort}", "Writer, Ann & Reader, Ben"),
("{title_sort}", "Book, The"),
("{publisher} {id}", "site.example 42"),
("{title:.3s|<|>}", "<Boo>"),
("{title:*^15}", "***Book, The***"),
("{id:05d} {id:x} {id:.2f}", "00042 2a 42.00"),
("{title[0]}{title[1]}", "Bo"),
("{title} {{{id}}}", "Book, The {42}"),
(" {title} -\t{authors} ", "Book, The - Ann Writer & Ben Reader"),
("{ title }{other}", ""),
("a|b|c{title:|-|}", "a|b|c-Book, The"),
("{series}/{series_index:0>2s| | - }{title}", "Saga, The/ 2.35 - Book, The"),
],
)
def test_more_templates_give_the_text_of_calibre(template, expected):
assert expand(template) == expected
def test_values_follow_the_rules_of_calibre():
assert expand("{title} - {authors}", title="A/B: C\\D", authors=["X/Y"]) == "A_B: C_D - X_Y"
assert expand("{authors}", authors=["Rock & Roll", "", "Solo"]) == "Rock && Roll & Solo"
assert expand("{tags}", tags=["/root", "alpha", "Beta", "1st", "_x"]) == "_x, _root, 1st, alpha, Beta"
assert expand("{tags}", tags=["F/M", "Hurt-Comfort", "hurt comfort", "Hurt", "x2", "x10", "é", "e", "f"]) == (
"e, é, f, F_M, Hurt, hurt comfort, Hurt-Comfort, x10, x2"
)
assert expand("{tags}|{languages}|{pubdate}", tags=[], languages="", pubdate="") == "||"
assert expand("{languages}", languages="en") == "en"
assert expand("{pubdate} {timestamp}", pubdate="1999-12-31T23:59:59", timestamp="2026-10-03") == (
"Dec, 1999 Oct, 2026"
)
assert expand("{pubdate}", pubdate="not a date") == ""
def test_date_format():
# Calibre's helper_date reference case.
assert ft.expand("{pubdate}", ft.prepare(BOOK, 42, date_format="%b %Y")) == "May 2020"
assert ft.expand("{pubdate}", ft.prepare(BOOK, 42, date_format="%d/%m/%Y")) == "06_05_2020"
assert ft.expand("x{pubdate}x", ft.prepare(BOOK, 42, date_format="")) == "xx"
ft.check_date_format("%Y-%m-%d " * 10)
with pytest.raises(ft.TemplateError, match="A date format can have 100 characters at most"):
ft.check_date_format("x" * 101)
# A series without a number has no series_index, and a book without a series has neither.
assert expand("{series}{series_index:| [|]}", series_index=None) == "Saga, The"
assert ft.expand("{series_index:03d}|{series}", ft.prepare(NO_SERIES, 42)) == "|"
@pytest.mark.parametrize(
"title,expected",
[
("A Tale", "Tale, A"),
("An Apple", "Apple, An"),
("the lower", "lower, the"),
("Theory", "Theory"),
("The", "The"),
(" The Spaced ", "Spaced, The"),
("'The Quoted'", "Quoted, The"),
("“The Curly”", "Curly, The"),
("„The Low“", "Low, The"),
("The 'Inner", "Inner, The"),
('""', ""),
("Le Monde", "Le Monde"),
],
)
def test_title_sort(title, expected):
assert ft.title_sort(title) == expected
@pytest.mark.parametrize(
"author,expected",
[
("Ann Writer", "Writer, Ann"),
("Dr. John Smith Jr.", "Smith, John Jr."),
("John Smith MD PhD", "Smith, John MD PhD"),
("Mr Smith", "Smith"),
("Dr Mr", "Dr Mr"),
("Jr Sr", "Jr Sr"),
("xX_Dark_Xx", "xX_Dark_Xx"),
("Smith, John", "Smith, John"),
("John (Jack) Doe", "Doe, John"),
("A (b c", "A (b c"),
("Acme Media", "Acme Media"),
("Ludwig van Beethoven", "Beethoven, Ludwig van"),
("J. R. R. Tolkien", "Tolkien, J. R. R."),
],
)
def test_author_sort(author, expected):
assert ft.author_sort(author) == expected
@pytest.mark.parametrize(
"index,expected",
[(1, "1"), (2.0, "2"), (0, "0"), (2.5, "2.5"), (1.10, "1.1"), (3.14159, "3.14"), (2.001, "2."), (10.999, "11.")],
)
def test_series_index(index, expected):
assert ft.series_index(index) == expected
@pytest.mark.parametrize(
"template,error",
[
("{title:uppercase()}", "does not support template functions"),
("{series:ifempty(none)}", "does not support template functions"),
("{title:'uppercase($)'}", "does not support template functions"),
("{title:.3s:'uppercase($)'}", "does not support template functions"),
("program: field('title')", "does not support the program modes"),
("python: def evaluate(book, context): return ''", "does not support the program modes"),
("{title!r}", "does not support the field"),
("{title.upper}", "does not support the field"),
("{title[x]}", "does not support the field"),
("{title:{id}}", "does not support the field"),
("{}", "A field needs a name"),
("{0}", "A field needs a name"),
("{title", "The braces of the template do not match"),
("title}", "The braces of the template do not match"),
("{title:5q}", "The format '5q' cannot take"),
("{series:|a|b|c}", "The format '|a' cannot take"),
("{id:.3d}", "The format '.3d' cannot take"),
("{title:>256s}", "A format can ask for 255 characters at most"),
("{id:.999f}", "A format can ask for 255 characters at most"),
("x" * 1025, "A template can have 1024 characters at most"),
],
)
def test_check_refuses_what_bindery_does_not_support(template, error):
with pytest.raises(ft.TemplateError, match=error.replace("|", "\\|")):
ft.check(template)
def test_check_accepts_the_templates_of_calibre():
for template in ("", "{author_sort}/{title}/{title} - {authors}", "{title:>255s}", "x" * 1024):
ft.check(template)
@pytest.mark.parametrize("template", ["{series_index:03d}", "{title:d}", "{title[50]}", "{id:c}{title:c}"])
def test_a_template_can_fail_for_one_book(template):
ft.check(template)
with pytest.raises(ft.TemplateError):
expand(template)
def file_name(template, **changes):
return ft.file_name(template, ft.prepare({**BOOK, **changes}, 42))
def test_file_name_is_the_last_part_of_the_path():
assert file_name("{author_sort}/{title}/{title} - {authors}") == "Book, The - Ann Writer & Ben Reader"
assert file_name("{series}//{title} /") == "Book, The"
assert file_name("{author_sort[0]}/{series}", series="") == "W"
assert file_name("{series}/{other}", series="") == ""
assert file_name(" / ") == ""
def test_file_name_is_safe():
assert file_name("{title}?", title="A/B: C\\D") == "A_B_ C_D_"
assert file_name("<{id:c}>") == "___"
assert file_name("{title}", title="con") == "_con"
assert file_name("{title}.{id}", title="NUL") == "_NUL.42"
assert file_name("{title}", title="Console") == "Console"
assert file_name("{title}...") == "Book, The"
long = file_name("{title}", title="é" * 150)
assert long == "é" * 100
assert len(file_name("{title}", title="\U0001f600" * 150).encode()) == 200
tests/test_import_unlinked_epub_chapters.py
+274
View File
@@ -0,0 +1,274 @@
"""Tests for scripts/import_unlinked_epub_chapters.py, a manual tool."""
import zipfile
import pytest
from bindery import config
from bindery.store import Store
from scripts import import_unlinked_epub_chapters as script
from scripts.import_unlinked_epub_chapters import (
ScriptError,
chapter_url,
main,
read_epub,
splice_order,
suggest_matches,
)
STORY_URL = "https://www.royalroad.com/fiction/1234"
CONTAINER_XML = """<?xml version="1.0"?>
<container xmlns="urn:oasis:names:tc:opendocument:xmlns:container" version="1.0">
<rootfiles><rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/></rootfiles>
</container>"""
def chapter_html(title: str, text: str) -> str:
return f"<html><head><title>{title}</title></head><body><h1>{title}</h1><p>{text}</p></body></html>"
def make_opf(count: int, identifier: str) -> str:
items = "\n".join(
f' <item id="c{i}" href="chap{i}.xhtml" media-type="application/xhtml+xml"/>' for i in range(1, count + 1)
)
refs = "\n".join(f' <itemref idref="c{i}"/>' for i in range(1, count + 1))
return f"""<?xml version="1.0"?>
<package xmlns="http://www.idpf.org/2007/opf" version="3.0">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:identifier>{identifier}</dc:identifier></metadata>
<manifest>
<item id="nav" href="nav.xhtml" media-type="application/xhtml+xml" properties="nav"/>
{items}
</manifest>
<spine>
<itemref idref="nav" linear="no"/>
{refs}
</spine>
</package>"""
def make_epub(path, chapters, identifier="book-1", files=None):
"""An ePub with a chapter file for each (title, text). `files` adds more files, such as images."""
with zipfile.ZipFile(path, "w") as z:
z.writestr("META-INF/container.xml", CONTAINER_XML)
z.writestr("OEBPS/content.opf", make_opf(len(chapters), identifier))
z.writestr("OEBPS/nav.xhtml", "<html><body><nav></nav></body></html>")
for i, (title, text) in enumerate(chapters, start=1):
z.writestr(f"OEBPS/chap{i}.xhtml", chapter_html(title, text))
for name, data in (files or {}).items():
z.writestr(name, data)
return path
def test_read_epub_reads_spine_order_and_skips_nonlinear_items(tmp_path):
path = make_epub(tmp_path / "book.epub", [("Chapter One", "First text"), ("Chapter Two", "Second text")])
chapters = read_epub(path)
assert [c["title"] for c in chapters] == ["Chapter One", "Chapter Two"]
assert "First text" in chapters[0]["html"]
assert "<body" not in chapters[0]["html"]
assert chapters[0]["key"] != chapters[1]["key"]
NCX_OPF = """<?xml version="1.0"?>
<package xmlns="http://www.idpf.org/2007/opf" version="2.0">
<metadata/>
<manifest>
<item id="ncx" href="toc.ncx" media-type="application/x-dtbncx+xml"/>
<item id="contents" href="text/contents.xhtml" media-type="application/xhtml+xml"/>
<item id="c1" href="text/one.xhtml" media-type="application/xhtml+xml"/>
<item id="c2" href="text/two.xhtml" media-type="application/xhtml+xml"/>
</manifest>
<spine toc="ncx"><itemref idref="contents"/><itemref idref="c1"/><itemref idref="c2"/></spine>
</package>"""
NCX_TOC = """<?xml version="1.0"?>
<ncx xmlns="http://www.daisy.org/z3986/2005/ncx/" version="2005-1"><navMap>
<navPoint id="p1"><navLabel><text>CHAPTER ONE: Arrival</text></navLabel><content src="text/one.xhtml"/></navPoint>
<navPoint id="p2"><navLabel><text>12</text></navLabel><content src="text/two.xhtml#page_12"/></navPoint>
</navMap></ncx>"""
def test_read_epub_skips_a_contents_page_and_takes_titles_from_the_ncx(tmp_path):
# As in a calibre conversion: a contents page in the reading order, a title in two headings,
# and a page number in the NCX. Curly quotes stay.
path = tmp_path / "book.epub"
with zipfile.ZipFile(path, "w") as z:
z.writestr("META-INF/container.xml", CONTAINER_XML)
z.writestr("OEBPS/content.opf", NCX_OPF)
z.writestr("OEBPS/toc.ncx", NCX_TOC)
z.writestr(
"OEBPS/text/contents.xhtml",
'<html><body><h1><a href="one.xhtml">CHAPTER ONE: Arrival</a></h1>'
'<p><a href="two.xhtml">CHAPTER TWO: Decision</a></p></body></html>',
)
z.writestr(
"OEBPS/text/one.xhtml", "<html><body><h1>CHAPTER ONE</h1><h1>Arrival</h1><p>“I’m here.”</p></body></html>"
)
z.writestr("OEBPS/text/two.xhtml", "<html><body><h1>CHAPTER TWO</h1><h1>Decision</h1><p>Text</p></body></html>")
chapters = read_epub(path)
assert [c["title"] for c in chapters] == ["CHAPTER ONE: Arrival", "CHAPTER TWO"]
assert "<p>“I’m here.”</p>" in chapters[0]["html"]
def test_read_epub_points_images_at_their_own_urls_and_drops_missing_ones(tmp_path):
path = make_epub(
tmp_path / "book.epub",
[("Map", '<img src="images/map.png"/><img src="images/gone.png"/><img src="images/notes.txt"/>')],
files={"OEBPS/images/map.png": b"png data", "OEBPS/images/notes.txt": b"text"},
)
(chapter,) = read_epub(path)
(image,) = chapter["images"]
assert image["url"].startswith("file:///bindery-epub-import/")
assert image["url"].endswith("/OEBPS/images/map.png")
assert (image["ext"], image["mime"], image["data"]) == ("png", "image/png", b"png data")
assert f'src="{image["url"]}"' in chapter["html"]
assert "gone.png" not in chapter["html"]
assert "notes.txt" not in chapter["html"]
def test_read_epub_rejects_a_bad_file(tmp_path):
path = tmp_path / "not-an-epub.epub"
path.write_text("nope")
with pytest.raises(ScriptError):
read_epub(path)
def test_suggest_matches_ranks_the_closest_title_first_and_drops_weak_ones():
candidates = ["Chapter One: A New Start", "Interlude", "Chapter Two: Trouble Brews"]
assert suggest_matches("Chapter Two", candidates)[0] == 2
assert suggest_matches("Chapter One", ["Completely unrelated title"]) == []
def test_chapter_url_is_stable_and_differs_between_epubs(tmp_path):
one = read_epub(make_epub(tmp_path / "one.epub", [("Prologue", "a")], identifier="book-1"))[0]
again = read_epub(make_epub(tmp_path / "again.epub", [("Prologue", "a")], identifier="book-1"))[0]
other = read_epub(make_epub(tmp_path / "other.epub", [("Prologue", "a")], identifier="book-3"))[0]
url = chapter_url(STORY_URL, one)
assert url.startswith(f"{STORY_URL}#bindery-import-")
assert chapter_url(STORY_URL, again) == url
assert chapter_url(STORY_URL, other) != url
def test_splice_order_puts_the_new_urls_after_the_given_index():
assert splice_order(["a", "b", "c"], 0, ["x", "y"]) == ["a", "x", "y", "b", "c"]
assert splice_order(["a", "b"], -1, ["x"]) == ["x", "a", "b"]
assert splice_order(["a", "b"], 1, ["x"]) == ["a", "b", "x"]
@pytest.fixture
def story(tmp_path, monkeypatch):
monkeypatch.setattr(config, "DB_PATH", tmp_path / "bindery.db")
db_store = Store(config.DB_PATH)
story_id = db_store.create_story(STORY_URL, "www.royalroad.com", "Story", "Writer", [])
db_store.add_chapter(story_id, f"{STORY_URL}/1", "Chapter 1", "<p>one</p>", "h1", "new", 0)
db_store.add_chapter(story_id, f"{STORY_URL}/4", "Chapter 4", "<p>four</p>", "h4", "new", 1)
epub = make_epub(
tmp_path / "book.epub",
[
("Cover", "cover"),
("Chapter 1", "one text"),
("Chapter 2", '<img src="map.png"/>two text'),
("Chapter 3", "three text"),
("Chapter 4", "four text"),
("About the Author", "author"),
],
files={"OEBPS/map.png": b"png"},
)
return db_store, story_id, epub
def answer(monkeypatch, *answers):
"""Answer the prompts in order. A prompt that shows a default in [brackets] gets it for an empty answer."""
replies = iter(answers)
prompts = []
def fake_input(prompt=""):
prompts.append(prompt)
return next(replies)
monkeypatch.setattr("builtins.input", fake_input)
return prompts
def titles(db_store, story_id):
return [c["title"] for c in db_store.list_chapters(story_id)]
def test_main_fills_a_gap_with_the_suggested_matches(story, monkeypatch, capsys):
db_store, story_id, epub = story
# Before the gap: stored chapter 1. Then Enter takes each suggestion.
prompts = answer(monkeypatch, "1", "", "", "", "y")
main([STORY_URL, str(epub)])
# Each prompt says what it asks for, and shows the suggestion. Numbers start at 1.
assert prompts[:4] == [
"Stored chapter just BEFORE the gap (or 'start'): ",
'ePub chapter that matches "Chapter 1" [2]: ',
"Stored chapter just AFTER the gap (or 'end') [2]: ",
'ePub chapter that matches "Chapter 4" [5]: ',
]
assert titles(db_store, story_id) == ["Chapter 1", "Chapter 2", "Chapter 3", "Chapter 4"]
added = db_store.list_chapters(story_id)[1]
assert added["url"].startswith(f"{STORY_URL}#bindery-import-")
assert added["on_site"] == 0
assert db_store.list_revisions(added["id"])[0]["source"] == "epub-import"
html = db_store.revision_html(added["id"], 1)
(image,) = db_store.images(story_id)
assert f'src="{image["url"]}"' in html
assert 'The script will add these 2 chapters after "Chapter 1"' in capsys.readouterr().out
# A second run over the same gap finds the same chapters in the story, and adds nothing.
answer(monkeypatch, "1", "", "4", "")
main([STORY_URL, str(epub)])
assert len(db_store.list_chapters(story_id)) == 4
assert "2 ePub chapters of the gap are in the story already" in capsys.readouterr().out
def test_main_asks_again_for_a_bad_answer_and_can_show_the_whole_list(story, monkeypatch, capsys):
db_store, story_id, epub = story
monkeypatch.setattr(script, "WINDOW", 2)
answer(monkeypatch, "9", "list", "1", "", "1", "", "", "n")
main([STORY_URL, str(epub)])
out = capsys.readouterr().out
assert "Type a number from 1 to 2, or 'list' or 'start'." in out
# The AFTER question cannot go before the BEFORE chapter.
assert "Type a number from 2 to 2, or 'list' or 'end'." in out
assert titles(db_store, story_id) == ["Chapter 1", "Chapter 4"]
def test_main_asks_for_the_first_and_last_epub_chapters_at_the_start_and_end(story, monkeypatch):
db_store, story_id, epub = story
# The gap is at the start: add from ePub chapter 1 up to the match of stored Chapter 1.
prompts = answer(monkeypatch, "start", "1", "", "", "y")
main([STORY_URL, str(epub)])
assert prompts[1] == "FIRST ePub chapter to add: "
assert titles(db_store, story_id) == ["Cover", "Chapter 1", "Chapter 4"]
# The gap is after the last stored chapter, so the script does not ask for the chapter after it.
prompts = answer(monkeypatch, "3", "", "", "y")
main([STORY_URL, str(epub)])
assert prompts[2] == "LAST ePub chapter to add [6]: "
assert titles(db_store, story_id) == ["Cover", "Chapter 1", "Chapter 4", "About the Author"]
def test_main_changes_nothing_when_the_user_says_no(story, monkeypatch):
db_store, story_id, epub = story
answer(monkeypatch, "1", "", "", "", "n")
main([STORY_URL, str(epub)])
assert titles(db_store, story_id) == ["Chapter 1", "Chapter 4"]
assert db_store.images(story_id) == []
def test_remove_deletes_only_the_added_chapters_and_their_revisions(story, monkeypatch):
db_store, story_id, _ = story
added = db_store.add_chapter(story_id, f"{STORY_URL}#bindery-import-x", "Two", "<p>2</p>", "h2", "epub-import", 1)
db_store.add_revision(added, "<p>2b</p>", "h2b", "recheck")
answer(monkeypatch, "n")
main([STORY_URL, "--remove"])
assert len(db_store.list_chapters(story_id)) == 3
answer(monkeypatch, "y")
main([STORY_URL, "--remove"])
assert titles(db_store, story_id) == ["Chapter 1", "Chapter 4"]
assert db_store.list_revisions(added) == []
tests/test_job_races.py
+92
View File
@@ -0,0 +1,92 @@
"""Changes that the user makes to the chapters of a story while a job runs for it."""
import pytest
from test_deliver import CHAPTER, linked_story, site, titles, update
from test_deliver import fake as fake
from test_deliver import jobs as jobs
from test_deliver import story_epub as story_epub
from bindery import fff, orchestrator
from bindery.jobs import DownloadMode
from bindery.store import Store
THREE = [(1, "One", "a"), (2, "Two", "b"), (3, "Three", "c")]
def during_download(monkeypatch, number, action):
"""Run `action` when the job downloads the chapter with this number."""
fetch = fff.fetch_chapter
def fetch_and_act(adapter, url, index):
if url == CHAPTER.format(number):
action()
return fetch(adapter, url, index)
monkeypatch.setattr(fff, "fetch_chapter", fetch_and_act)
def chapter_id(story_id, title):
return next(c["id"] for c in orchestrator.store.list_chapters(story_id) if c["title"] == title)
def test_a_move_during_a_download_stays(fake, jobs, monkeypatch):
story_id = linked_story(THREE)
site(monkeypatch, [*THREE, (4, "Four", "d"), (5, "Five", "e")])
during_download(monkeypatch, 4, lambda: orchestrator.move_chapters(story_id, [chapter_id(story_id, "Three")], None))
assert "2 new chapters" in update(story_id)
assert titles(story_id) == ["Three", "One", "Two", "Four", "Five"]
def test_a_new_chapter_in_the_middle_goes_after_its_moved_predecessor(fake, jobs, monkeypatch):
story_id = linked_story(THREE)
site(monkeypatch, [(1, "One", "a"), (9, "Nine", "x"), (2, "Two", "b"), (3, "Three", "c")])
during_download(
monkeypatch,
9,
lambda: orchestrator.move_chapters(story_id, [chapter_id(story_id, "One")], chapter_id(story_id, "Three")),
)
update(story_id)
assert titles(story_id) == ["Two", "Three", "One", "Nine"]
def test_a_delete_during_a_download_stays(fake, jobs, monkeypatch):
story_id = linked_story(THREE)
site(monkeypatch, [*THREE, (4, "Four", "d")])
during_download(monkeypatch, 4, lambda: orchestrator.delete_chapters(story_id, [chapter_id(story_id, "Two")]))
assert "1 new chapter" in update(story_id)
assert titles(story_id) == ["One", "Three", "Four"]
def test_a_delete_during_a_recheck_does_not_stop_the_job(fake, jobs, monkeypatch):
story_id = linked_story(THREE)
site(monkeypatch, [(1, "One", "a2"), (2, "Two", "b2"), (3, "Three", "c2")])
during_download(monkeypatch, 2, lambda: orchestrator.delete_chapters(story_id, [chapter_id(story_id, "Two")]))
assert "2 of 3 rechecked chapters changed" in update(story_id, DownloadMode.RECHECK_ALL)
assert titles(story_id) == ["One", "Three"]
def test_the_order_function_of_add_chapters_sees_the_stored_order(tmp_path):
store = Store(tmp_path / "db.sqlite")
sid = store.create_story("u", "s", "T", "A", [])
for n, url in enumerate(["u2", "u1"]):
store.add_chapter(sid, url, url, "<p/>", "h", "new", n)
new = [{"url": "u3", "title": "u3", "html": "<p/>", "text_hash": "h"}]
seen = []
store.add_chapters(sid, new, "new", lambda stored: seen.append(stored) or [*stored, "u3"])
assert seen == [["u2", "u1"]]
assert [c["url"] for c in store.list_chapters(sid)] == ["u2", "u1", "u3"]
def test_a_refused_reorder_changes_nothing(tmp_path):
store = Store(tmp_path / "db.sqlite")
sid = store.create_story("u", "s", "T", "A", [])
ids = [store.add_chapter(sid, f"u{n}", str(n), "<p/>", "h", "new", n) for n in range(3)]
def refuse(order):
raise ValueError("no")
with pytest.raises(ValueError, match="no"):
store.reorder_chapters(sid, refuse)
assert [c["id"] for c in store.list_chapters(sid)] == ids
store.reorder_chapters(sid, lambda order: order[::-1])
assert [c["id"] for c in store.list_chapters(sid)] == ids[::-1]
tests/test_jobs.py
+14
View File
@@ -1,5 +1,6 @@
"""The job queue: shared jobs, order, cancelling and the job history."""
import json
from datetime import datetime, timedelta
from bindery import config
@@ -161,3 +162,16 @@ def test_finished_jobs_leave_the_database(tmp_path, monkeypatch):
old.updated = datetime.now() - timedelta(seconds=config.JOB_HISTORY + 1)
queue.get_status()
assert JobQueue(tmp_path / "db.sqlite").get_status() == []
def test_the_api_form_of_a_job_hides_the_path_of_an_upload():
job = Job(mode=DownloadMode.IMPORT_FILE, options={"path": "/config/uploads/x.epub", "name": "x.epub"})
assert job.to_dict()["options"] == {"name": "x.epub"}
assert Job.from_record(job.to_record()).options == job.options
def test_a_record_of_a_newer_bindery_loads_without_its_unknown_fields():
job = Job(mode=DownloadMode.NEW, story_id=3)
record = json.dumps({**json.loads(job.to_record()), "priority": 5})
loaded = Job.from_record(record)
assert (loaded.id, loaded.story_id, loaded.mode) == (job.id, 3, DownloadMode.NEW)
tests/test_last_changed.py
+66
View File
@@ -0,0 +1,66 @@
"""Last Changed records stored chapters and revisions, not changes to the built book."""
import sqlite3
import pytest
from bindery import store as storage
@pytest.fixture
def state(tmp_path, monkeypatch):
clock = ["2024-01-01T00:00:00"]
monkeypatch.setattr(storage, "now", lambda: clock[0])
store = storage.Store(tmp_path / "stories.db")
sid = store.create_story("story", "site", "Title", "Author", [])
return store, sid, clock
@pytest.mark.parametrize("source", ["new", "import"])
def test_new_chapters_and_revisions_set_the_timestamp_in_the_store(state, source):
store, sid, clock = state
other = store.create_story("other", "site", "Other", "Author", [])
assert store.get_story(sid)["last_changed"] is None
first = store.add_chapter(sid, "one", "One", "first", "h1", source, 0)
assert store.get_story(sid)["last_changed"] == clock[0]
assert store.get_chapter(first)["added_at"] == clock[0]
clock[0] = "2024-02-01T00:00:00"
store.add_chapters(sid, [{"url": "two", "title": "Two", "html": "second", "text_hash": "h2"}], source, [])
assert store.get_story(sid)["last_changed"] == clock[0]
clock[0] = "2024-03-01T00:00:00"
store.add_revision(first, "revised", "h3", "recheck")
assert store.get_story(sid)["last_changed"] == clock[0]
assert store.get_chapter(first)["changed_at"] == clock[0]
assert store.list_revisions(first)[-1]["fetched_at"] == clock[0]
assert storage.Store(store.path).get_story(sid)["last_changed"] == clock[0]
assert store.get_story(other)["last_changed"] is None
def test_changes_without_new_chapters_or_revisions_keep_the_timestamp(state):
store, sid, clock = state
first = store.add_chapter(sid, "one", "One", "first", "h1", "new", 0)
second = store.add_chapter(sid, "two", "Two", "second", "h2", "new", 1)
store.add_revision(first, "revised", "h3", "recheck")
previous = clock[0]
clock[0] = "2024-02-01T00:00:00"
store.update_story(sid, title="Edited", pinned={"title": "Edited"})
store.set_rules(sid, [{"pattern": "first", "replacement": "1st"}])
store.update_chapter(first, title="Renamed", included=False, on_site=False, current_rev=1)
store.replace_revision(first, 1, "original site text", "h4")
store.reorder_chapters(sid, lambda order: list(reversed(order)))
store.add_chapters(sid, [], "import", ["one", "two"])
store.delete_revision(first, 2)
store.delete_chapters(sid, [second])
assert store.get_story(sid)["last_changed"] == previous
def test_a_failed_chapter_transaction_does_not_change_the_timestamp(state):
store, sid, clock = state
store.add_chapter(sid, "one", "One", "first", "h1", "new", 0)
previous = clock[0]
clock[0] = "2024-02-01T00:00:00"
chapters = [{"url": url, "title": url, "html": "text", "text_hash": "h"} for url in ("two", "one")]
with pytest.raises(sqlite3.IntegrityError):
store.add_chapters(sid, chapters, "import", [])
assert store.get_story(sid)["last_changed"] == previous
assert [chapter["url"] for chapter in store.list_chapters(sid)] == ["one"]
tests/test_local_auth.py
+1
View File
@@ -37,6 +37,7 @@ def test_local_login_logout_and_password_hashes(client, local, monkeypatch, meth
"authenticated": False,
"mode": "local",
"user": None,
"admin": False,
"logout_url": None,
}
assert client.get("/api/stories").status_code == 401
tests/test_opds.py
+278
View File
@@ -0,0 +1,278 @@
"""OPDS navigation, book lists, downloads and the shared login gate."""
from datetime import UTC, datetime
from xml.etree import ElementTree as ET
import pytest
from test_api import client as client
from test_api import cwa as cwa
from test_build import PNG
from bindery import diffing, fff, opds, orchestrator
NS = {"a": opds.ATOM}
ACQUIRE = "http://opds-spec.org/acquisition"
PATHS = ["/opds", "/opds/", "/opds/title", "/opds/author", "/opds/changed"]
@pytest.fixture(autouse=True)
def enabled_catalog(client):
orchestrator.store.set_setting(None, "opds_enabled", True)
def feed(response):
assert response.status_code == 200
assert response.mimetype == "application/atom+xml"
assert response.headers["Cache-Control"] == "private, no-store"
root = ET.fromstring(response.data)
assert root.tag == f"{{{opds.ATOM}}}feed"
assert root.findtext("a:id", namespaces=NS)
assert root.findtext("a:title", namespaces=NS)
assert datetime.fromisoformat(root.findtext("a:updated", namespaces=NS)).tzinfo is not None
return root
def links(element):
return {link.attrib["rel"]: link.attrib["href"] for link in element.findall("a:link", NS)}
def entries(root):
return root.findall("a:entry", NS)
def titles(root):
return [entry.findtext("a:title", namespaces=NS) for entry in entries(root)]
def add_book(title="A book", author="Writer", changed=None, url=None, included=True, meta=None):
store = orchestrator.store
url = url or f"https://example.com/books/{len(store.list_stories()) + 1}"
sid = store.create_story(url, "example.com", title, author, [])
if included:
html = "<p>Stored text.</p>"
store.add_chapter(sid, url + "/1", "One", html, diffing.text_hash(html), "site", 0)
store.update_story(sid, last_changed=changed, site_values=meta or {})
return sid
@pytest.fixture(autouse=True)
def no_site_requests(monkeypatch):
def fail(*args, **kwargs):
pytest.fail("OPDS must not contact a story site")
for name in ("fetch_story", "fetch_chapter", "collected_images"):
monkeypatch.setattr(fff, name, fail)
def test_navigation_and_empty_lists(client):
root = feed(client.get("/opds"))
assert titles(root) == [title for title, _, _ in opds.LISTS.values()]
assert len({entry.findtext("a:id", namespaces=NS) for entry in entries(root)}) == 3
assert client.get("/opds/").data == client.get("/opds").data
for entry, (_, kind, text) in zip(entries(root), opds.LISTS.values(), strict=True):
assert entry.findtext("a:content", namespaces=NS) == text
link = entry.find("a:link", NS)
assert link.attrib["type"] == kind
books = feed(client.get(link.attrib["href"]))
assert entries(books) == []
assert links(books)["start"] == links(books)["up"] == "/opds"
assert "next" not in links(books)
def test_sort_orders_dates_and_stable_book_ids(client):
add_book("The Zebra", "alice", "2024-01-01T12:00:00+00:00")
add_book("An Apple", "Bob", "2024-03-01T12:00:00+00:00")
add_book("Banana", "ALICE", "2024-02-01T12:00:00+00:00")
expected = {
"title": ["An Apple", "Banana", "The Zebra"],
"author?name=Alice": ["Banana", "The Zebra"],
"changed": ["An Apple", "Banana", "The Zebra"],
}
ids = []
for order, names in expected.items():
root = feed(client.get(f"/opds/{order}"))
assert titles(root) == names
if order != "author?name=Alice":
assert root.findtext("a:updated", namespaces=NS) == "2024-03-01T12:00:00Z"
ids.append({e.findtext("a:id", namespaces=NS) for e in entries(root)})
for entry in entries(root):
assert entry.findtext("a:author/a:name", namespaces=NS)
assert entry.findtext("a:content", namespaces=NS) == "Chapters: 1"
assert entry.find("a:link", NS).attrib["type"] == "application/epub+zip"
assert ids[0] == ids[2] > ids[1]
def test_dates_use_creation_fallback_and_compare_offsets(client):
sid = add_book("Created")
created = orchestrator.store.get_story(sid)["created_at"]
expected = datetime.fromisoformat(created).astimezone(UTC).isoformat().replace("+00:00", "Z")
root = feed(client.get("/opds/changed"))
assert entries(root)[0].findtext("a:updated", namespaces=NS) == expected
orchestrator.store.update_story(sid, last_changed="2024-01-01T12:00:00+02:00")
add_book("Later", changed="2024-01-01T11:00:00+00:00")
assert titles(feed(client.get("/opds/changed"))) == ["Later", "Created"]
def test_xml_escaping_unicode_and_missing_metadata(client):
add_book('Books <&> "日本語"\x00', "A & B <writers>\x0b")
empty = add_book("", "")
root = feed(client.get("/opds/title"))
assert titles(root)[0] == 'Books <&> "日本語"'
assert entries(root)[0].findtext("a:author/a:name", namespaces=NS) == "A & B <writers>"
assert titles(root)[1] == orchestrator.store.get_story(empty)["url"]
assert entries(root)[1].findtext("a:author/a:name", namespaces=NS) == "Unknown"
def test_only_books_with_included_chapters_appear(client):
add_book("Pending", included=False)
sid = add_book("Excluded")
chapter = orchestrator.store.list_chapters(sid)[0]
orchestrator.store.update_chapter(chapter["id"], included=False)
add_book("Ready")
assert titles(feed(client.get("/opds/title"))) == ["Ready"]
@pytest.mark.parametrize("order", ["title", "changed", "author?name=Same", "search?q=same"])
def test_pagination_with_ties(client, monkeypatch, order):
monkeypatch.setattr(opds, "PAGE_SIZE", 2)
for _ in range(5):
add_book("Same", "Same", "2024-01-01T00:00:00+00:00")
first = feed(client.get(f"/opds/{order}"))
second = feed(client.get(links(first)["next"]))
last = feed(client.get(links(second)["next"]))
assert [len(entries(root)) for root in (first, second, last)] == [2, 2, 1]
assert "previous" not in links(first)
assert "next" not in links(last)
assert links(second)["previous"] == links(first)["self"]
assert links(last)["previous"] == links(second)["self"]
ids = [e.findtext("a:id", namespaces=NS) for root in (first, second, last) for e in entries(root)]
assert ids == [f"https://example.com/books/{i}" for i in range(1, 6)]
assert client.get(links(first)["self"].replace("page=1", "page=4")).status_code == 404
@pytest.mark.parametrize("page", ["0", "-1", "1.5", "abc", "", "999999999999999999999"])
def test_bad_page(client, page):
assert client.get("/opds/title", query_string={"page": page}).status_code == 400
def test_unknown_list(client):
assert client.get("/opds/missing").status_code == 404
DC = {"a": opds.ATOM, "dc": opds.DC}
def test_entry_shows_the_book_metadata_and_cover(client):
sid = add_book(
"Stored title",
meta={
"title": "The Book",
"authors": ["Ann Writer", "Ben Reader"],
"comments": "<p>First <b>part</b>.</p><p>Second part.</p>",
"tags": ["Space", "Fiction"],
"series": "The Saga",
"series_index": 2.0,
"pubdate": "2020-05-06T12:00:00+00:00",
"languages": "en",
},
)
orchestrator.store.update_story(sid, pinned={"title": "The Kept Title"})
entry = entries(feed(client.get("/opds/title")))[0]
assert entry.findtext("a:title", namespaces=NS) == "The Kept Title"
assert [a.findtext("a:name", namespaces=NS) for a in entry.findall("a:author", NS)] == ["Ann Writer", "Ben Reader"]
assert entry.find("a:author/a:uri", NS).text == "/opds/author?name=Ann+Writer"
assert entry.find("a:content", NS).attrib["type"] == "text"
assert entry.findtext("a:content", namespaces=NS) == (
"Series: The Saga, book 2\nChapters: 1\n\nFirst part.\n\nSecond part."
)
assert [c.attrib["term"] for c in entry.findall("a:category", NS)] == ["Space", "Fiction"]
assert entry.findtext("dc:language", namespaces=DC) == "en"
assert entry.findtext("dc:issued", namespaces=DC) == "2020-05-06"
assert entry.findtext("dc:publisher", namespaces=DC) == "example.com"
assert "http://opds-spec.org/image" not in links(entry)
orchestrator.store.put_image(sid, "https://example.com/cover.png", "png", "image/png", PNG, is_cover=True)
entry = entries(feed(client.get("/opds/title")))[0]
image = links(entry)["http://opds-spec.org/image"]
assert image == links(entry)["http://opds-spec.org/image/thumbnail"]
assert image.startswith(f"/opds/books/{sid}/cover?v=")
response = client.get(image)
assert response.data == PNG
assert response.mimetype == "image/png"
assert "private" in response.headers["Cache-Control"]
orchestrator.store.put_image(sid, "https://example.com/new.png", "png", "image/png", PNG, is_cover=True)
assert links(entries(feed(client.get("/opds/title")))[0])["http://opds-spec.org/image"] != image
assert client.get(f"/opds/books/{add_book()}/cover").status_code == 404
def test_entry_shows_the_page_count_of_a_built_book(client):
sid = add_book()
orchestrator.store.set_setting(sid, "page_count", 321)
add_book(title="Not built")
contents = {
e.findtext("a:title", namespaces=NS): e.findtext("a:content", namespaces=NS)
for e in entries(feed(client.get("/opds/title")))
}
assert contents == {"A book": "Chapters: 1\nPages: 321", "Not built": "Chapters: 1"}
def test_cover_of_an_unsafe_type_is_left_out(client):
sid = add_book()
orchestrator.store.put_image(sid, "https://example.com/c.svg", "svg", "image/svg+xml", b"<svg/>", is_cover=True)
assert "http://opds-spec.org/image" not in links(entries(feed(client.get("/opds/title")))[0])
assert client.get(f"/opds/books/{sid}/cover").status_code == 404
def test_authors_in_surname_order_with_their_books(client):
add_book("One", "Zed Adams")
add_book("Two", "ann writer")
add_book("Three", "", meta={"authors": ["Ann Writer", "Bob Brown"]})
add_book("Four", "")
root = feed(client.get("/opds/author"))
assert titles(root) == ["Zed Adams", "Bob Brown", "Unknown", "Ann Writer"]
assert [e.findtext("a:content", namespaces=NS) for e in entries(root)] == ["1 book", "1 book", "1 book", "2 books"]
assert links(root)["up"] == "/opds"
writer = feed(client.get(links(entries(root)[3])["subsection"]))
assert titles(writer) == ["Three", "Two"]
assert links(writer)["up"] == "/opds/author"
assert client.get("/opds/author", query_string={"name": "Nobody"}).status_code == 404
def test_author_list_pages(client, monkeypatch):
monkeypatch.setattr(opds, "PAGE_SIZE", 2)
for name in ("A One", "B Two", "C Three"):
add_book(author=name)
first = feed(client.get("/opds/author"))
assert titles(first) == ["A One", "C Three"]
assert titles(feed(client.get(links(first)["next"]))) == ["B Two"]
def test_search(client):
add_book("The Long Road", "Ann Writer", meta={"tags": ["Adventure"]})
add_book("Short Road", "Ben Reader", meta={"series": "Roads"})
add_book("Other", "Cat Smith")
def search(q):
return titles(feed(client.get("/opds/search", query_string={"q": q})))
assert search("road") == ["The Long Road", "Short Road"]
assert search("ROAD writer") == ["The Long Road"]
assert search("adventure") == ["The Long Road"]
assert search("roads") == ["Short Road"]
assert search("missing") == search("") == search(" ") == []
assert feed(client.get("/opds/search")).findtext("a:title", namespaces=NS) == "Search: "
def test_every_feed_links_the_search(client):
add_book()
for path in [*PATHS, "/opds/author?name=Writer", "/opds/search?q=book"]:
searches = [link for link in feed(client.get(path)).findall("a:link", NS) if link.attrib["rel"] == "search"]
found = {link.attrib["type"]: link.attrib["href"] for link in searches}
assert found == {
"application/opensearchdescription+xml": "/opds/search.xml",
opds.ACQUISITION: "/opds/search?q={searchTerms}",
}
response = client.get("/opds/search.xml")
assert response.mimetype == "application/opensearchdescription+xml"
url = ET.fromstring(response.data).find(f"{{{opds.OPENSEARCH}}}Url")
assert url.attrib == {"type": opds.ACQUISITION, "template": "/opds/search?q={searchTerms}"}
tests/test_opds_auth.py
+329
View File
@@ -0,0 +1,329 @@
"""The login of the OPDS catalog: the account login in the local and CWA modes, a password of its own in proxy mode."""
import base64
import io
import zipfile
import pytest
from test_api import H, login
from test_api import client as client
from test_api import cwa as cwa
from test_build import URL, metahtml
from test_opds import PATHS, add_book, enabled_catalog, entries, feed, links, no_site_requests
from werkzeug.security import generate_password_hash
from bindery import auth, config, opds, orchestrator, web
__all__ = ["enabled_catalog", "no_site_requests"] # fixtures that every test here uses
PASSWORD = "reader-password"
CHALLENGE = 'Basic realm="Bindery", charset="UTF-8"'
ACQUIRE = "http://opds-spec.org/acquisition"
GATED = [*PATHS, "/opds/search?q=a", "/opds/search.xml", "/opds/books/1.epub", "/opds/books/1/cover"]
OTHER_LOGINS = [
{"Authorization": "Bearer api-secret"},
{"X-Api-Key": "api-secret"},
{"X-Remote-User": "admin"},
]
def basic(password, user="reader"):
return {"Authorization": "Basic " + base64.b64encode(f"{user}:{password}".encode()).decode()}
def set_password(password=PASSWORD, user="reader"):
hashed = generate_password_hash(password, method="pbkdf2:sha256:1000")
orchestrator.store.set_setting(None, orchestrator._opds_password_name(user), hashed)
def fresh_lockout(monkeypatch):
monkeypatch.setattr(auth, "limiter", auth.LoginLimiter())
@pytest.fixture
def proxy(monkeypatch):
monkeypatch.setattr(config, "AUTH_METHOD", "proxy")
monkeypatch.setattr(config, "PROXY_AUTH_TRUSTED_IPS", ["127.0.0.1"])
monkeypatch.setattr(config, "API_KEY", "api-secret")
set_password()
@pytest.fixture
def local(monkeypatch):
monkeypatch.setattr(config, "AUTH_METHOD", "local")
monkeypatch.setattr(config, "LOCAL_USERNAME", "reader")
monkeypatch.setattr(config, "LOCAL_PASSWORD_HASH", generate_password_hash("web-pw", method="pbkdf2:sha256:1000"))
monkeypatch.setattr(config, "API_KEY", "api-secret")
set_password() # an OPDS password does not apply here
def test_proxy_mode_takes_the_proxy_user_name_with_the_password_of_that_user(client, proxy, monkeypatch):
add_book()
set_password("alice-password", "alice")
for path in GATED:
response = client.get(path)
assert response.status_code == 401
assert response.headers["WWW-Authenticate"] == CHALLENGE
for path in PATHS:
response = client.get(path, headers=basic(PASSWORD, "reader"))
feed(response)
assert "Set-Cookie" not in response.headers
assert client.get("/opds/search.xml", headers=basic("alice-password", "alice")).status_code == 200
# The user name matches without regard to case and spaces. Each user has their own password.
assert client.get("/opds", headers=basic(PASSWORD, " READER ")).status_code == 200
for headers in [
basic(PASSWORD, "alice"),
basic("alice-password", "reader"),
basic(PASSWORD, "nobody"),
basic(PASSWORD, ""),
*OTHER_LOGINS,
basic("api-secret"),
{**OTHER_LOGINS[2], **basic("wrong")},
{"X-Remote-User": "alice", **basic(PASSWORD, "nobody")},
]:
fresh_lockout(monkeypatch)
for path in PATHS[:3]:
assert client.get(path, headers=headers).status_code == 401
def test_the_catalog_password_does_not_open_the_api(client, proxy):
for headers in (basic(PASSWORD), {"Authorization": f"Bearer {PASSWORD}"}, {"X-Api-Key": PASSWORD}):
response = client.get("/api/stories", headers=headers)
assert response.status_code == 401
assert "WWW-Authenticate" not in response.headers
assert client.get("/api/stories", headers={"X-Api-Key": "api-secret"}).status_code == 200
assert client.post("/api/stories", json={"urls": []}, headers={**H, **basic(PASSWORD)}).status_code == 401
def test_a_user_without_a_password_meets_the_same_work_as_one_with_a_password(client, proxy, monkeypatch):
calls = []
real = auth.check_password_hash
monkeypatch.setattr(auth, "check_password_hash", lambda *args: calls.append(args[0]) or real(*args))
for user in ("nobody", "reader"):
calls.clear()
response = client.get("/opds", headers=basic("wrong-password", user))
assert response.status_code == 401
assert response.json == {"error": "Unauthorized"}
assert len(calls) == 1
def test_local_mode_takes_the_account_login(client, local, monkeypatch):
for path in GATED:
response = client.get(path)
assert response.status_code == 401
assert response.headers["WWW-Authenticate"] == CHALLENGE
assert response.json["error"] == "Unauthorized"
for path in PATHS:
response = client.get(path, headers=basic("web-pw"))
feed(response)
assert "Set-Cookie" not in response.headers
for headers in [basic(PASSWORD), basic("web-pw", "other"), basic("wrong"), *OTHER_LOGINS]:
fresh_lockout(monkeypatch)
assert client.get("/opds", headers=headers).status_code == 401
assert client.get("/opds", headers=basic("web-pw", " reader ")).status_code == 200
def test_local_mode_takes_the_web_session_but_not_the_api_key_with_it(client, local):
assert client.get("/opds").status_code == 401
assert login(client, password="web-pw").status_code == 200
for path in PATHS:
feed(client.get(path))
assert client.get("/opds", headers={"X-Api-Key": "api-secret"}).status_code == 200 # the session counts
client.post("/api/auth/logout", headers=H)
assert client.get("/opds").status_code == 401
assert client.get("/opds", headers={"X-Api-Key": "api-secret"}).status_code == 401
def test_cwa_mode_takes_the_cwa_login(client, cwa, monkeypatch):
monkeypatch.setattr(config, "API_KEY", "api-secret")
set_password()
assert client.get("/opds").status_code == 401
assert client.get("/opds", headers=basic("pw")).status_code == 200
for headers in [basic("wrong"), basic("pw", "other"), basic(PASSWORD), *OTHER_LOGINS]:
fresh_lockout(monkeypatch)
assert client.get("/opds", headers=headers).status_code == 401
assert login(client).status_code == 200
feed(client.get("/opds/title"))
client.post("/api/auth/logout", headers=H)
monkeypatch.setattr(config, "CWA_ALLOWED_USERS", ["someone"])
assert client.get("/opds", headers=basic("pw")).status_code == 401
def test_none_mode_needs_no_login(client):
assert orchestrator.store.get_setting(None, "opds_password_hash") is None
for path in PATHS:
feed(client.get(path))
set_password()
feed(client.get("/opds"))
def test_a_good_account_login_is_kept_for_five_minutes(client, local, monkeypatch):
clock = [1000.0]
monkeypatch.setattr(auth.time, "monotonic", lambda: clock[0])
calls = []
real = auth.login_local
monkeypatch.setattr(auth, "login_local", lambda *args: calls.append(args) or real(*args))
for seconds in (0, auth.KEPT_SECONDS - 1, 1):
clock[0] += seconds
assert client.get("/opds", headers=basic("web-pw")).status_code == 200
assert len(calls) == 2
# A wrong password is never kept. A new web password ends a kept login.
assert client.get("/opds", headers=basic("wrong")).status_code == 401
assert client.get("/opds", headers=basic("wrong")).status_code == 401
assert len(calls) == 4
monkeypatch.setattr(config, "LOCAL_PASSWORD_HASH", generate_password_hash("new-pw", method="pbkdf2:sha256:1000"))
assert client.get("/opds", headers=basic("web-pw")).status_code == 401
assert client.get("/opds", headers=basic("new-pw")).status_code == 200
def test_a_good_catalog_password_is_kept_and_a_new_password_ends_it(client, proxy, monkeypatch):
calls = []
real = auth.check_password_hash
monkeypatch.setattr(auth, "check_password_hash", lambda *args: calls.append(args) or real(*args))
for _ in range(3):
assert client.get("/opds", headers=basic(PASSWORD)).status_code == 200
assert len(calls) == 1
set_password("a new password")
assert client.get("/opds", headers=basic(PASSWORD)).status_code == 401
assert client.get("/opds", headers=basic("a new password")).status_code == 200
def test_a_disabled_catalog_answers_404_before_any_login(client, proxy):
orchestrator.store.set_setting(None, "opds_enabled", False)
for headers in ({}, basic(PASSWORD), basic("wrong")):
assert client.get("/opds", headers=headers).status_code == 404
def test_wrong_catalog_passwords_lock_out_a_user_at_an_address(client, proxy):
set_password("alice-password", "alice")
for _ in range(auth.MAX_FAILURES):
assert client.get("/opds", headers=basic("wrong")).status_code == 401
assert client.get("/opds", headers=basic(PASSWORD)).status_code == 401
# Another user at that address, and the same user at another address, can log in.
assert client.get("/opds", headers=basic("alice-password", "alice")).status_code == 200
other = {"environ_base": {"REMOTE_ADDR": "198.51.100.7"}}
assert client.get("/opds", headers=basic(PASSWORD), **other).status_code == 200
def test_wrong_account_passwords_lock_out_an_address(client, local):
for _ in range(auth.MAX_FAILURES):
assert client.get("/opds", headers=basic("wrong")).status_code == 401
assert client.get("/opds", headers=basic("web-pw")).status_code == 401
assert client.get("/opds", headers=basic("web-pw"), environ_base={"REMOTE_ADDR": "198.51.100.7"}).status_code == 200
@pytest.mark.parametrize(
"value",
["Basic", "Basic !!!", "Digest cmVhZGVyOnB3", basic("")["Authorization"], basic("x" * 257)["Authorization"]]
+ ["Basic " + base64.b64encode(text).decode() for text in (b"reader", b"\xff:pw")],
)
def test_bad_basic_headers(client, proxy, monkeypatch, value):
def fail(*args):
pytest.fail("A bad header must not reach the hash")
monkeypatch.setattr(auth, "check_password_hash", fail)
assert client.get("/opds", headers={"Authorization": value}).status_code == 401
def test_bad_config_fails_closed(client, monkeypatch):
set_password()
monkeypatch.setattr(config, "AUTH_METHOD", "invalid")
for path in PATHS:
assert client.get(path, headers=basic(PASSWORD)).status_code == 503
def test_host_protection(client):
assert client.get("/opds", base_url="http://attacker.example").status_code == 400
def test_url_base_and_download_auth(client, proxy, monkeypatch, tmp_path):
monkeypatch.setattr(web.app, "wsgi_app", web.PrefixMiddleware(web.app.wsgi_app, "/bindery"))
monkeypatch.setattr(opds, "PAGE_SIZE", 1)
sid = add_book()
add_book("Second")
headers = basic(PASSWORD)
root = feed(client.get("/bindery/opds", headers=headers))
for entry in entries(root):
books = feed(client.get(links(entry)["subsection"], headers=headers))
assert all(href.startswith("/bindery/") for href in links(books).values())
if entry.find("a:link", {"a": opds.ATOM}).attrib["type"] == opds.ACQUISITION:
assert client.get(links(books)["next"], headers=headers).status_code == 200
books = feed(client.get("/bindery/opds/title", headers=headers))
download = links(entries(books)[0])[ACQUIRE]
assert download == f"/bindery/opds/books/{sid}.epub"
assert client.get(download).status_code == 401
assert client.get(download, headers=basic("wrong")).status_code == 401
file = tmp_path / "book.epub"
file.write_bytes(b"epub bytes")
monkeypatch.setattr(orchestrator, "build_for_download", lambda story_id: (file.open("rb"), "book.epub"))
with client.get(download, headers=headers) as response:
assert response.data == b"epub bytes"
assert response.mimetype == "application/epub+zip"
assert client.get("/opds", headers=headers).status_code == 404
def test_download_builds_from_store_without_changing_chapters(client, local, monkeypatch, tmp_path):
monkeypatch.setattr(config, "TMP_DIR", tmp_path)
sid = add_book(url=URL)
orchestrator.store.update_story(sid, metahtml=metahtml())
before = orchestrator.store.chapters_for_build(sid)
root = feed(client.get("/opds/title", headers=basic("web-pw")))
with client.get(links(entries(root)[0])[ACQUIRE], headers=basic("web-pw")) as response:
assert response.status_code == 200
assert response.mimetype == "application/epub+zip"
with zipfile.ZipFile(io.BytesIO(response.data)) as epub:
assert epub.read("mimetype") == b"application/epub+zip"
assert orchestrator.store.chapters_for_build(sid) == before
def test_the_catalog_takes_no_changes(client, proxy):
assert client.post("/opds", headers=basic(PASSWORD)).status_code == 405
def test_an_opds_download_is_named_by_the_download_template(client, local, monkeypatch, tmp_path):
monkeypatch.setattr(config, "TMP_DIR", tmp_path)
sid = add_book(url=URL, meta={"title": "The Book", "authors": ["Ann Writer"]})
orchestrator.store.update_story(sid, metahtml=metahtml())
root = feed(client.get("/opds/title", headers=basic("web-pw")))
href = links(entries(root)[0])[ACQUIRE]
def disposition():
with client.get(href, headers=basic("web-pw")) as response:
assert response.status_code == 200
return response.headers["Content-Disposition"]
assert 'filename="Book, The - Ann Writer.epub"' in disposition()
orchestrator.store.set_setting(None, "download_template", "{title} - {authors}")
assert 'filename="The Book - Ann Writer.epub"' in disposition()
def test_forget_clears_the_wrong_passwords_of_one_user_from_every_address():
limiter = auth.LoginLimiter()
for address in ("10.0.0.1", "10.0.0.2"):
for _ in range(auth.MAX_FAILURES):
limiter.failed("Alice", address)
limiter.failed("bob", address)
limiter.forget("ALICE")
assert not limiter.locked("alice", "10.0.0.1")
assert not limiter.locked("alice", "10.0.0.2")
assert limiter.locked("bob", "10.0.0.1")
def test_setting_a_password_ends_the_lockout_of_that_user(client, proxy):
set_password("bob-password", "bob")
far = {"environ_base": {"REMOTE_ADDR": "198.51.100.7"}}
# A user without a password tries from two addresses, and meets the lockout.
for kwargs in ({}, far):
for _ in range(auth.MAX_FAILURES):
assert client.get("/opds", headers=basic("guess", "alice"), **kwargs).status_code == 401
for _ in range(auth.MAX_FAILURES):
assert client.get("/opds", headers=basic("guess", "bob")).status_code == 401
assert client.get("/opds", headers=basic("alice-password", "alice")).status_code == 401
saved = client.put(
"/api/settings/opds-password", json={"password": "alice-password"}, headers={**H, "X-Remote-User": "Alice"}
)
assert saved.status_code == 200
for kwargs in ({}, far):
assert client.get("/opds", headers=basic("alice-password", "alice"), **kwargs).status_code == 200
# The lockout of another user stays.
assert client.get("/opds", headers=basic("bob-password", "bob")).status_code == 401
tests/test_opds_settings.py
+186
View File
@@ -0,0 +1,186 @@
"""The OPDS catalog stays off until an admin enables it in Settings. An admin also sets its password there."""
import base64
import pytest
from test_api import H
from test_api import client as client
from werkzeug.security import check_password_hash, generate_password_hash
from bindery import config, orchestrator, web
from bindery.store import Store
PATHS = ["/opds", "/opds/", "/opds/title", "/opds/author", "/opds/changed"]
def test_catalog_defaults_to_off_and_can_be_toggled_without_restart(client, monkeypatch):
assert client.get("/api/settings").json["opds_enabled"] is False
assert orchestrator.store.get_setting(None, "opds_enabled") is None
for enabled in (False, True, False):
if enabled or orchestrator.store.get_setting(None, "opds_enabled") is not None:
saved = client.put("/api/settings", json={"opds_enabled": enabled}, headers=H)
assert saved.status_code == 200
assert saved.json["opds_enabled"] is enabled
for path in PATHS:
for method in (client.get, client.head):
assert method(path).status_code == (200 if enabled else 404)
if enabled:
# A fresh store instance reads the saved setting, as after a restart.
monkeypatch.setattr(orchestrator, "store", Store(orchestrator.store.path))
assert client.get("/api/settings").json["opds_enabled"] is True
unchanged = client.put("/api/settings", json={"recent_chapters": 2}, headers=H)
assert unchanged.json["opds_enabled"] is True
unchanged = client.put("/api/settings", json={"opds_enabled": None}, headers=H)
assert unchanged.json["opds_enabled"] is True
@pytest.mark.parametrize("value", ["false", "true", 0, 1, [], {}])
def test_opds_setting_requires_a_boolean_and_rejects_all_changes(client, value):
before = client.get("/api/settings").json
response = client.put("/api/settings", json={"opds_enabled": value, "recent_chapters": 9}, headers=H)
assert response.status_code == 400
assert "true or false" in response.json["error"]
assert client.get("/api/settings").json == before
def test_invalid_download_settings_do_not_enable_opds(client):
response = client.put(
"/api/settings", json={"opds_enabled": True, "download_template": "{title:uppercase()}"}, headers=H
)
assert response.status_code == 400
assert client.get("/api/settings").json["opds_enabled"] is False
assert client.get("/opds").status_code == 404
def test_only_an_admin_can_enable_opds_and_csrf_header_is_required(client, monkeypatch):
monkeypatch.setattr(config, "AUTH_METHOD", "proxy")
monkeypatch.setattr(config, "ADMIN_USERS", ["admin"])
monkeypatch.setattr(config, "PROXY_AUTH_TRUSTED_IPS", ["127.0.0.1"])
assert client.put("/api/settings", json={"opds_enabled": True}, headers=H).status_code == 401
assert (
client.put("/api/settings", json={"opds_enabled": True}, headers={**H, "X-Remote-User": "reader"}).status_code
== 403
)
assert (
client.put("/api/settings", json={"opds_enabled": True}, headers={"X-Remote-User": "admin"}).status_code == 403
)
assert orchestrator.store.get_setting(None, "opds_enabled") is None
response = client.put("/api/settings", json={"opds_enabled": True}, headers={**H, "X-Remote-User": "admin"})
assert response.status_code == 200
assert response.json["opds_enabled"] is True
assert client.get("/opds").status_code == 401
# The proxy user header does not open the catalog. Only the catalog password does.
assert client.get("/opds", headers={"X-Remote-User": "reader"}).status_code == 401
PASSWORD_URL = "/api/settings/opds-password"
def basic(password, user="reader"):
return {"Authorization": "Basic " + base64.b64encode(f"{user}:{password}".encode()).decode()}
def as_user(name):
return {**H, "X-Remote-User": name}
@pytest.fixture
def proxy(monkeypatch):
monkeypatch.setattr(config, "AUTH_METHOD", "proxy")
monkeypatch.setattr(config, "PROXY_AUTH_TRUSTED_IPS", ["127.0.0.1"])
monkeypatch.setattr(config, "ADMIN_USERS", ["admin"])
monkeypatch.setattr(config, "API_KEY", "key")
def test_disabled_catalog_answers_404_under_url_base_with_any_credentials(client, proxy, monkeypatch):
monkeypatch.setattr(web.app, "wsgi_app", web.PrefixMiddleware(web.app.wsgi_app, "/bindery"))
keyed = {"X-Api-Key": "key"}
for path in PATHS:
for headers in (keyed, basic("reader-password")):
assert client.get("/bindery" + path, headers=headers).status_code == 404
assert client.put("/bindery/api/settings", json={"opds_enabled": True}, headers=keyed).status_code == 200
saved = client.put("/bindery" + PASSWORD_URL, json={"password": "reader-password"}, headers=as_user("reader"))
assert saved.status_code == 200
for path in PATHS:
assert client.get("/bindery" + path, headers=keyed).status_code == 401
assert client.get("/bindery" + path, headers=basic("reader-password")).status_code == 200
def test_each_user_sets_their_own_password_and_it_is_stored_as_a_hash(client, proxy):
client.put("/api/settings", json={"opds_enabled": True}, headers=as_user("admin"))
for user in ("reader", "admin"):
info = client.get("/api/settings", headers=as_user(user)).json
assert (info["opds_user"], info["opds_password_set"]) == (user, False)
saved = client.put(PASSWORD_URL, json={"password": "first-password"}, headers=as_user("Reader"))
assert saved.json == {"opds_user": "Reader", "opds_password_set": True}
stored = orchestrator.store.get_setting(None, "opds_password_hash:reader")
assert "first-password" not in stored
assert check_password_hash(stored, "first-password")
# The answers show the user name and the state, never the password or its hash.
for response in (saved, client.get("/api/settings", headers=as_user("reader"))):
assert "first-password" not in response.text
assert stored not in response.text
assert client.get("/api/settings", headers=as_user("reader")).json["opds_password_set"] is True
assert client.get("/api/settings", headers=as_user("admin")).json["opds_password_set"] is False
assert client.get("/opds", headers=basic("first-password", "reader")).status_code == 200
assert client.get("/opds", headers=basic("first-password", "admin")).status_code == 401
# A new password replaces the old one. Other changes leave it as it is.
client.put(PASSWORD_URL, json={"password": "second-password"}, headers=as_user("reader"))
client.put("/api/settings", json={"recent_chapters": 2}, headers=as_user("admin"))
assert client.get("/opds", headers=basic("first-password")).status_code == 401
assert client.get("/opds", headers=basic("second-password")).status_code == 200
client.put(PASSWORD_URL, json={"password": "admin-password"}, headers=as_user("admin"))
assert client.get("/opds", headers=basic("admin-password", "admin")).status_code == 200
assert client.get("/opds", headers=basic("second-password")).status_code == 200
@pytest.mark.parametrize("value", [None, "", "short", "x" * 7, "x" * 257, 12345678, ["password"], True])
def test_bad_password_changes_nothing(client, proxy, value):
response = client.put(PASSWORD_URL, json={"password": value}, headers=as_user("reader"))
assert response.status_code == 400
assert "OPDS password" in response.json["error"]
assert orchestrator.store.get_setting(None, "opds_password_hash:reader") is None
def test_a_password_of_the_limits_is_accepted(client, proxy):
for password in ("x" * 8, "é" * 256):
assert client.put(PASSWORD_URL, json={"password": password}, headers=as_user("reader")).status_code == 200
def test_the_setting_route_ignores_a_password(client, proxy):
client.put("/api/settings", json={"opds_password": "reader-password"}, headers=as_user("admin"))
assert orchestrator.store.get_setting(None, "opds_password_hash:admin") is None
@pytest.mark.parametrize("mode", ["none", "local", "cwa"])
def test_only_the_proxy_mode_has_an_opds_password(client, monkeypatch, mode):
monkeypatch.setattr(config, "AUTH_METHOD", mode)
monkeypatch.setattr(config, "LOCAL_USERNAME", "reader")
monkeypatch.setattr(config, "LOCAL_PASSWORD_HASH", generate_password_hash("pw", method="pbkdf2:sha256:1000"))
monkeypatch.setattr(config, "CWA_URL", "http://localhost:9999")
monkeypatch.setattr(config, "API_KEY", "key")
keyed = {"X-Api-Key": "key"}
response = client.put(PASSWORD_URL, json={"password": "reader-password"}, headers=keyed)
assert response.status_code == 400
assert "through the proxy" in response.json["error"]
info = client.get("/api/settings", headers=keyed).json
assert (info["opds_user"], info["opds_password_set"]) == (None, False)
def test_a_user_needs_the_login_of_the_proxy_to_set_a_password(client, proxy):
body = {"password": "reader-password"}
assert client.put(PASSWORD_URL, json=body, headers=H).status_code == 401
assert client.put(PASSWORD_URL, json=body, headers={"X-Api-Key": "key"}).status_code == 400
assert client.put(PASSWORD_URL, json=body, headers={"X-Remote-User": "reader"}).status_code == 403 # no CSRF header
assert client.put(PASSWORD_URL, json=body, headers=as_user("reader")).status_code == 200
untrusted = client.put(
PASSWORD_URL, json=body, headers=as_user("reader"), environ_base={"REMOTE_ADDR": "198.51.100.7"}
)
assert untrusted.status_code == 401
def test_config_tells_the_page_the_login_mode(client, monkeypatch):
assert client.get("/api/config").json["auth_mode"] == "none"
monkeypatch.setattr(config, "AUTH_METHOD", "proxy")
monkeypatch.setattr(config, "PROXY_AUTH_TRUSTED_IPS", ["127.0.0.1"])
assert client.get("/api/config", headers={"X-Remote-User": "reader"}).json["auth_mode"] == "proxy"
tests/test_optional_cwa.py
+490
View File
@@ -0,0 +1,490 @@
"""Downloads and builds work with uploads or library matching disabled."""
import json
import os
import subprocess
import sys
from unittest.mock import Mock
import pytest
from test_api import H
from test_api import client as client
from test_deliver import STORY_URL, add_story, rebuild, site, update
from test_deliver import jobs as jobs
from bindery import config, orchestrator
from bindery.cwa import Book
from bindery.ingest import IngestError
from bindery.jobs import DownloadMode, Job
from bindery.store import Store
@pytest.fixture(autouse=True)
def isolated(client, jobs, monkeypatch, tmp_path):
monkeypatch.setattr(config, "INGEST_DIR", tmp_path / "ingest")
config.INGEST_DIR.mkdir()
monkeypatch.setattr(config, "TMP_DIR", tmp_path / "tmp")
config.TMP_DIR.mkdir()
monkeypatch.setattr(config, "CWA_URL", "")
monkeypatch.setattr(config, "CALIBRE_DB_PATH", None)
monkeypatch.setattr(orchestrator, "cwa", Mock(side_effect=AssertionError("Unexpected CWA request")))
monkeypatch.setattr(orchestrator, "_find_book", Mock(side_effect=AssertionError("Unexpected book search")))
@pytest.fixture
def library(isolated, monkeypatch):
monkeypatch.setattr(config, "CWA_URL", "https://cwa.invalid")
client = Mock()
client.get_book.return_value = Book(7, "uuid-7", "Test Story").to_dict()
monkeypatch.setattr(orchestrator, "cwa", Mock(return_value=client))
monkeypatch.setattr(orchestrator, "_find_book", Mock(return_value=None))
@pytest.mark.parametrize("enabled", [None, False, True])
def test_app_starts_without_any_cwa_configuration(tmp_path, enabled):
env = {
**os.environ,
"CONFIG_DIR": str(tmp_path / "standalone"),
"TMP_DIR": str(tmp_path / "builds"),
"AUTH_METHOD": "none",
"BINDERY_START_THREADS": "false",
"URL_BASE": "",
"CWA_ENABLED": str(enabled),
}
# CWA is off unless CWA_ENABLED turns it on.
for name in (
"INGEST_DIR",
"CWA_URL",
"CWA_DB_PATH",
"CALIBRE_DB_PATH",
*(["CWA_ENABLED"] if enabled is None else []),
):
env.pop(name, None)
result = subprocess.run(
[
sys.executable,
"-c",
"from bindery.web import app; "
"client = app.test_client(); "
"assert client.get('/api/health').status_code == 200; "
f"assert client.get('/api/config').json['cwa_enabled'] is {bool(enabled)}; "
"settings = client.get('/api/settings').json; "
"assert settings['pause_uploads'] is True; "
"assert settings['ingest_available'] is False; "
"assert settings['link_books'] is False; "
"assert settings['linking_available'] is False",
],
env=env,
capture_output=True,
text=True,
timeout=30,
)
assert result.returncode == 0, result.stderr
assert "ERROR" not in result.stderr
# Bindery names what is missing only for a user who turned CWA on.
for text in ("WARNING", "INGEST_DIR is not set", "CWA_URL and CALIBRE_DB_PATH are not set", "remove CWA_ENABLED"):
assert (text in result.stderr) is bool(enabled)
if not enabled:
assert "CWA" not in result.stderr and "INGEST" not in result.stderr
@pytest.fixture
def cwa_off(monkeypatch):
monkeypatch.setattr(config, "CWA_ENABLED", False)
def test_without_cwa_and_an_ingest_folder_a_job_only_builds_a_changed_book(monkeypatch, cwa_off):
monkeypatch.setattr(config, "INGEST_DIR", None)
site(monkeypatch, [(1, "One", "First chapter")])
job = Job(mode=DownloadMode.NEW, url=STORY_URL)
assert orchestrator._run_site_job(job).endswith("Built the book")
story_id = job.story_id
story = orchestrator.store.get_story(story_id)
assert (story["status"], story["message"], story["ingested_at"]) == ("ok", "Built the book", None)
assert orchestrator.store.get_setting(story_id, "upload_held") is None
assert len(list(config.TMP_DIR.glob(f"story-{story_id}-*.epub"))) == 1
assert update(story_id).startswith("No changes")
assert orchestrator._run_rebuild(Job(mode=DownloadMode.REBUILD, story_id=story_id)) == "No changes"
assert rebuild(story_id) == "Built the book"
site(monkeypatch, [(1, "One", "First chapter"), (2, "Two", "Second chapter")])
assert update(story_id).endswith("Built the book")
orchestrator.cwa.assert_not_called()
@pytest.mark.parametrize("saved_id", [None, 7])
def test_without_cwa_the_ingest_folder_gets_each_changed_book_as_a_plain_epub(cwa_off, saved_id):
# A pause, a hold and a saved book ID are from a time with CWA. They do not apply.
orchestrator.save_settings(pause_uploads=True, hold_new_links=True)
story_id = add_story(calibre_id=saved_id, calibre_uuid="saved-uuid" if saved_id else None)
orchestrator._link_changed(story_id)
orchestrator.store.set_setting(story_id, "upload_held", True)
assert rebuild(story_id) == "Sent to ingest folder"
assert [p.suffix for p in config.INGEST_DIR.iterdir()] == [".epub"]
assert orchestrator.store.get_story(story_id)["ingested_at"] is None
assert orchestrator.store.get_setting(story_id, "upload_held") is None
assert orchestrator._run_rebuild(Job(mode=DownloadMode.REBUILD, story_id=story_id)) == "No changes"
assert len(list(config.INGEST_DIR.iterdir())) == 1
orchestrator.cwa.assert_not_called()
def test_without_cwa_a_missing_ingest_folder_is_still_an_error(monkeypatch, tmp_path, cwa_off):
monkeypatch.setattr(config, "INGEST_DIR", tmp_path / "missing")
story_id = add_story()
with pytest.raises(IngestError, match="does not exist"):
rebuild(story_id)
# The book counts as sent only when it is in the folder.
assert orchestrator.store.get_setting(story_id, "sent_key") is None
def test_cwa_off_shows_in_the_config_and_refuses_cwa_logins(client, monkeypatch, cwa_off):
assert client.get("/api/config").json["cwa_enabled"] is False
monkeypatch.setattr(config, "AUTH_METHOD", "cwa")
monkeypatch.setattr(config, "CWA_DB_PATH", "/cwa/app.db")
response = client.get("/api/stories")
assert response.status_code == 503
assert response.json["error"].startswith("AUTH_METHOD=cwa needs CWA_ENABLED=true.")
def test_without_cwa_the_download_template_uses_the_story_id(client, cwa_off):
assert client.get("/api/config").json["download_fields"]["id"] == "The Bindery story ID"
story_id = add_story(calibre_id=77)
client.put("/api/settings", json={"download_template": "{title} {id}"}, headers=H)
response = client.get(f"/api/stories/{story_id}/epub")
response.close()
assert f'filename="My Title {story_id}.epub"' in response.headers["Content-Disposition"]
def test_link_setting_defaults_on_and_persists(client, monkeypatch, tmp_path, library):
assert client.get("/api/settings").json["link_books"] is True
response = client.put("/api/settings", json={"link_books": False}, headers=H)
assert response.status_code == 200
assert response.json["link_books"] is False
monkeypatch.setattr(orchestrator, "store", Store(tmp_path / "bindery.db"))
assert client.get("/api/settings").json["link_books"] is False
response = client.put("/api/settings", json={"hold_new_links": True}, headers=H)
assert response.json["link_books"] is False
assert client.put("/api/settings", json={"link_books": True}, headers=H).json["link_books"] is True
@pytest.mark.parametrize("url", ["", "https://cwa.invalid"])
@pytest.mark.parametrize("database", [False, True])
def test_linking_requires_either_library_source(client, monkeypatch, tmp_path, url, database):
monkeypatch.setattr(config, "CWA_URL", url)
monkeypatch.setattr(config, "CALIBRE_DB_PATH", tmp_path / "metadata.db" if database else None)
available = bool(url or database)
settings = client.get("/api/settings").json
assert settings["linking_available"] is available
assert settings["link_books"] is available
settings = client.put("/api/settings", json={"link_books": True}, headers=H).json
assert settings["linking_available"] is available
assert settings["link_books"] is available
@pytest.mark.parametrize("saved", [None, False, True])
def test_missing_sources_force_linking_off_without_changing_preferences(client, monkeypatch, tmp_path, jobs, saved):
monkeypatch.setattr(config, "CWA_DB_PATH", tmp_path / "app.db")
orchestrator.store.set_setting(None, "link_books", saved)
orchestrator.store.set_setting(None, "hold_new_links", True)
story_id = add_story()
orchestrator.store.set_setting(story_id, "upload_held", True)
for requested in (True, False):
response = client.put("/api/settings", json={"link_books": requested}, headers=H)
assert response.status_code == 200
assert response.json["link_books"] is False
assert response.json["linking_available"] is False
assert response.json["hold_new_links"] is True
assert orchestrator.store.get_setting(None, "link_books") is saved
assert jobs.get_next() is None
monkeypatch.setattr(config, "CALIBRE_DB_PATH", tmp_path / "metadata.db")
assert client.get("/api/settings").json["linking_available"] is True
assert client.get("/api/settings").json["link_books"] is (saved is not False)
@pytest.mark.parametrize("value", ["false", 0, [], {}])
def test_link_setting_requires_a_boolean(client, value):
response = client.put("/api/settings", json={"link_books": value}, headers=H)
assert response.status_code == 400
assert client.get("/api/settings").json["link_books"] is False
@pytest.mark.parametrize("paused", [False, True])
def test_missing_ingest_forces_pause_without_changing_saved_preference(client, monkeypatch, tmp_path, paused):
orchestrator.save_settings(pause_uploads=paused)
monkeypatch.setattr(config, "INGEST_DIR", None)
settings = client.get("/api/settings").json
assert settings["pause_uploads"] is True
assert settings["ingest_available"] is False
for requested in (False, True):
response = client.put("/api/settings", json={"pause_uploads": requested}, headers=H)
assert response.status_code == 200
assert response.json["pause_uploads"] is True
monkeypatch.setattr(config, "INGEST_DIR", tmp_path / "enabled")
assert client.get("/api/settings").json["pause_uploads"] is paused
@pytest.mark.parametrize("existing", [False, True])
@pytest.mark.parametrize("link_books", [False, True])
def test_build_without_ingest_succeeds_and_keeps_the_book_local(monkeypatch, tmp_path, existing, link_books, library):
orchestrator.save_settings(link_books=link_books)
monkeypatch.setattr(config, "INGEST_DIR", None)
story_id = add_story(ingested_at="2026-01-01" if existing else None)
assert rebuild(story_id) == "Not sent. INGEST_DIR is not set."
story = orchestrator.store.get_story(story_id)
assert story["status"] == "ok"
assert story["ingested_at"] == ("2026-01-01" if existing else None)
assert list(config.TMP_DIR.glob(f"story-{story_id}-*.epub"))
assert not list((tmp_path / "ingest").iterdir())
details = orchestrator.story_details(story_id)
assert details["pause_uploads"] is True
assert details["story"]["upload_held"] is True
assert details["library_error"] is None
@pytest.mark.parametrize("configured", [False, True])
@pytest.mark.parametrize("saved_id", [None, 7])
@pytest.mark.parametrize("paused", [False, True])
def test_linking_off_exports_books_and_respects_pause(monkeypatch, tmp_path, configured, saved_id, paused):
if configured:
monkeypatch.setattr(config, "CWA_URL", "https://cwa.invalid")
monkeypatch.setattr(config, "CALIBRE_DB_PATH", tmp_path / "unreadable.db")
orchestrator.save_settings(link_books=False, pause_uploads=paused)
story_id = add_story(calibre_id=saved_id, calibre_uuid="saved-uuid", ingested_at="2026-01-01")
message = rebuild(story_id)
assert orchestrator.store.get_story(story_id)["status"] == "ok"
assert orchestrator.story_details(story_id)["library_error"] is None
orchestrator.cwa.assert_not_called()
orchestrator._find_book.assert_not_called()
if not paused:
if saved_id is not None:
assert message == "Updated library book 7"
manifest = next(config.INGEST_DIR.glob("*.cwa.json"))
assert json.loads(manifest.read_text()) == {"action": "add_format", "book_id": 7}
else:
assert message == "Sent to ingest folder"
assert not list(config.INGEST_DIR.glob("*.cwa.json"))
assert len(list(config.INGEST_DIR.glob("*.epub"))) == 1
assert orchestrator.store.get_setting(story_id, "upload_held") is None
else:
assert message.startswith("Not sent.")
assert not list(config.INGEST_DIR.iterdir())
assert orchestrator.store.get_setting(story_id, "upload_held") is True
def test_disabling_linking_releases_approval_holds_for_saved_ids(jobs, library):
orchestrator.save_settings(hold_new_links=True)
story_id = add_story(calibre_id=7, calibre_uuid="uuid-7", ingested_at="2026-01-01")
orchestrator._link_changed(story_id)
assert "Approve link" in rebuild(story_id)
orchestrator.save_settings(link_books=False)
assert orchestrator.get_settings()["hold_new_links"] is True
assert orchestrator.story_details(story_id)["story"]["needs_approval"] is False
job = jobs.get_next()
assert job.story_id == story_id
assert orchestrator._run_rebuild(job) == "Updated library book 7"
@pytest.mark.parametrize("link_books", [True, False])
def test_story_response_reports_linking_and_approval_settings(client, link_books, library):
orchestrator.save_settings(link_books=link_books, hold_new_links=True)
story_id = add_story()
orchestrator._link_changed(story_id)
response = client.get(f"/api/stories/{story_id}")
assert response.status_code == 200
assert response.json["link_books"] is link_books
assert response.json["story"]["needs_approval"] is link_books
def test_enabling_linking_restores_the_saved_approval_preference(library):
orchestrator.save_settings(link_books=False, hold_new_links=True)
story_id = add_story(calibre_id=7, calibre_uuid="uuid-7", ingested_at="2026-01-01")
orchestrator._link_changed(story_id)
assert orchestrator.story_details(story_id)["story"]["needs_approval"] is False
orchestrator.save_settings(link_books=True)
assert orchestrator.get_settings()["hold_new_links"] is True
assert orchestrator.story_details(story_id)["story"]["needs_approval"] is True
assert "Approve link" in rebuild(story_id)
assert not list(config.INGEST_DIR.iterdir())
def test_linking_off_exports_every_rebuild_without_an_id_or_approval():
orchestrator.save_settings(link_books=False, hold_new_links=True)
story_id = add_story()
orchestrator._link_changed(story_id)
for _ in range(2):
assert rebuild(story_id) == "Sent to ingest folder"
assert orchestrator.store.get_story(story_id)["ingested_at"] is not None
assert len(list(config.INGEST_DIR.glob("*.epub"))) == 2
assert not list(config.INGEST_DIR.glob("*.cwa.json"))
assert orchestrator.store.get_setting(story_id, "upload_held") is None
def test_disabling_linking_releases_an_unlinked_book_without_approval(jobs, library):
orchestrator.save_settings(hold_new_links=True)
story_id = add_story()
orchestrator._link_changed(story_id)
assert "Approve as new book" in rebuild(story_id)
orchestrator.save_settings(link_books=False)
assert orchestrator.list_stories()[0]["approval_held"] is False
assert orchestrator._run_rebuild(jobs.get_next()) == "Sent to ingest folder"
assert orchestrator.store.get_setting(story_id, "upload_held") is None
@pytest.mark.parametrize("standalone", [False, True])
def test_new_story_download_skips_library_matching(monkeypatch, standalone):
if standalone:
monkeypatch.setattr(config, "INGEST_DIR", None)
else:
monkeypatch.setattr(config, "CWA_URL", "https://cwa.invalid")
orchestrator.save_settings(link_books=False)
site(monkeypatch, [(1, "One", "First chapter")])
job = Job(mode=DownloadMode.NEW, url=STORY_URL)
message = orchestrator._run_site_job(job)
assert ("Not sent." if standalone else "Sent to ingest folder") in message
assert len(orchestrator.store.list_chapters(job.story_id)) == 1
assert orchestrator.store.get_story(job.story_id)["status"] == "ok"
assert list(config.TMP_DIR.glob(f"story-{job.story_id}-*.epub"))
orchestrator.cwa.assert_not_called()
orchestrator._find_book.assert_not_called()
def test_site_updates_export_without_a_library_or_saved_id(monkeypatch):
assert orchestrator.get_settings()["link_books"] is False
site(monkeypatch, [(1, "One", "First chapter")])
job = Job(mode=DownloadMode.NEW, url=STORY_URL)
assert "Sent to ingest folder" in orchestrator._run_site_job(job)
site(monkeypatch, [(1, "One", "First chapter"), (2, "Two", "Second chapter")])
assert "Sent to ingest folder" in orchestrator._run_site_job(Job(mode=DownloadMode.NEW, story_id=job.story_id))
assert len(orchestrator.store.list_chapters(job.story_id)) == 2
assert len(list(config.INGEST_DIR.glob("*.epub"))) == 2
assert not list(config.INGEST_DIR.glob("*.cwa.json"))
orchestrator.cwa.assert_not_called()
orchestrator._find_book.assert_not_called()
def test_saved_id_without_uuid_works_without_library_configuration():
story_id = add_story(calibre_id=7, ingested_at="2026-01-01")
assert rebuild(story_id) == "Updated library book 7"
orchestrator.cwa.assert_not_called()
orchestrator._find_book.assert_not_called()
def test_paused_unlinked_update_does_not_raise_not_found(monkeypatch):
monkeypatch.setattr(config, "CWA_URL", "https://cwa.invalid")
orchestrator._find_book.return_value = None
orchestrator._find_book.side_effect = None
orchestrator.save_settings(pause_uploads=True)
story_id = add_story(ingested_at="2026-01-01")
assert rebuild(story_id) == "Not sent. Turn off pause to send."
assert orchestrator.store.get_story(story_id)["status"] == "ok"
assert not list(config.INGEST_DIR.iterdir())
def test_enabling_linking_rebuilds_held_books_and_matches_again(monkeypatch, jobs, tmp_path):
monkeypatch.setattr(config, "CWA_URL", "https://cwa.invalid")
orchestrator.save_settings(link_books=False, pause_uploads=True)
story_id = add_story()
assert "Turn off pause" in rebuild(story_id)
downloaded = tmp_path / "library.epub"
downloaded.touch()
orchestrator._find_book.side_effect = None
orchestrator._find_book.return_value = (Book(7, "uuid-7", "Test Story"), downloaded)
orchestrator.save_settings(link_books=True, pause_uploads=False)
job = jobs.get_next()
assert job.mode == DownloadMode.REBUILD
assert job.story_id == story_id
assert orchestrator._run_rebuild(job) == "Updated library book 7"
assert orchestrator.store.get_story(story_id)["calibre_id"] == 7
assert orchestrator.store.get_setting(story_id, "upload_held") is None
@pytest.mark.parametrize("link_books", [False, True])
@pytest.mark.parametrize("hold_new_links", [False, True])
@pytest.mark.parametrize("paused", [False, True])
@pytest.mark.parametrize("saved_id", [None, 7])
def test_upload_settings_matrix(client, link_books, hold_new_links, paused, saved_id, library):
orchestrator.save_settings(link_books=link_books, hold_new_links=hold_new_links, pause_uploads=paused)
story_id = add_story(
calibre_id=saved_id,
calibre_uuid="uuid-7" if saved_id else None,
ingested_at="2026-01-01" if saved_id else None,
)
orchestrator._link_changed(story_id)
assert client.get("/api/stories").json[0]["approval_held"] is False
message = rebuild(story_id)
pending = paused or (link_books and hold_new_links)
assert message.startswith("Not sent.") is pending
listed = client.get("/api/stories").json[0]
assert listed["upload_held"] is pending
assert listed["approval_held"] is (link_books and hold_new_links and not paused)
assert bool(orchestrator.store.get_setting(story_id, "upload_held")) is pending
assert bool(list(config.INGEST_DIR.iterdir())) is (not pending)
@pytest.mark.parametrize("missing_ingest", [False, True])
def test_approval_while_paused_preserves_pending_upload_until_resume(
client, monkeypatch, jobs, missing_ingest, library
):
ingest = config.INGEST_DIR
orchestrator.save_settings(hold_new_links=True, pause_uploads=True)
if missing_ingest:
monkeypatch.setattr(config, "INGEST_DIR", None)
story_id = add_story()
orchestrator._link_changed(story_id)
rebuild(story_id)
response = client.post(f"/api/stories/{story_id}/approve", json={"uuid": None}, headers=H)
assert response.status_code == 200
assert response.json["job"] is None
assert response.json["story"]["needs_approval"] is False
assert response.json["story"]["upload_held"] is True
assert jobs.get_next() is None
assert client.get("/api/stories").json[0]["approval_held"] is False
assert orchestrator.store.get_setting(story_id, "upload_held") is True
monkeypatch.setattr(config, "INGEST_DIR", ingest)
orchestrator.save_settings(pause_uploads=False)
job = jobs.get_next()
assert job.story_id == story_id
assert orchestrator._run_rebuild(job) == "Sent to CWA"
assert orchestrator.store.get_setting(story_id, "upload_held") is None
assert client.get("/api/stories").json[0]["approval_held"] is False
def test_resuming_uploads_does_not_bypass_approval(client, jobs, library):
orchestrator.save_settings(hold_new_links=True, pause_uploads=True)
story_id = add_story(calibre_id=7, calibre_uuid="uuid-7", ingested_at="2026-01-01")
orchestrator._link_changed(story_id)
rebuild(story_id)
assert client.get("/api/stories").json[0]["approval_held"] is False
orchestrator.save_settings(pause_uploads=False)
assert client.get("/api/stories").json[0]["approval_held"] is True
assert "Approve link" in orchestrator._run_rebuild(jobs.get_next())
assert orchestrator.store.get_setting(story_id, "upload_held") is True
assert not list(config.INGEST_DIR.iterdir())
def test_missing_library_connection_is_not_an_approval_hold(client, monkeypatch):
orchestrator.save_settings(hold_new_links=True)
story_id = add_story(ingested_at="2026-01-01")
orchestrator._link_changed(story_id)
assert rebuild(story_id) == "Sent to ingest folder"
assert client.get("/api/stories").json[0]["approval_held"] is False
details = client.get(f"/api/stories/{story_id}").json
assert details["story"]["upload_held"] is False
assert details["story"]["needs_approval"] is False
assert details["link_books"] is False
assert details["ingest_available"] is True
monkeypatch.setattr(config, "INGEST_DIR", None)
assert client.get(f"/api/stories/{story_id}").json["ingest_available"] is False
monkeypatch.setattr(config, "CWA_URL", "https://cwa.invalid")
assert client.get(f"/api/stories/{story_id}").json["link_books"] is True
@pytest.mark.parametrize("missing_ingest", [False, True])
def test_releasing_other_holds_does_not_queue_uploads_while_paused(monkeypatch, jobs, missing_ingest, library):
orchestrator.save_settings(link_books=False, hold_new_links=True, pause_uploads=True)
story_id = add_story()
rebuild(story_id)
if missing_ingest:
monkeypatch.setattr(config, "INGEST_DIR", None)
orchestrator.save_settings(link_books=True, hold_new_links=False)
assert jobs.get_next() is None
assert orchestrator.get_settings()["pause_uploads"] is True
tests/test_pages.py
+122
View File
@@ -0,0 +1,122 @@
"""The page estimate of a document and of a built epub."""
import zipfile
import pytest
from bindery import pages
from bindery.epub import EpubError, spine_documents
from bindery.pages import Measure
TEXT = ["One line of the story.", "Another, longer line of the story that goes on for a while.", "The end."]
CHARS = sum(map(len, TEXT))
def test_a_paragraph_is_text_after_a_block_or_a_line_break():
assert pages.measure("<p>One</p><p>Two</p>") == Measure(2, 6)
assert pages.measure("<div>One<br/><br/>Two<br>Three</div>") == Measure(3, 11)
assert pages.measure("<div><div>One</div><div>Two</div></div>") == Measure(2, 6)
# Text before, between and after the blocks of an element counts too.
assert pages.measure("<div>Intro<p>One</p>Middle<p>Two</p>End</div>") == Measure(5, 20)
assert pages.measure("<h3>Title</h3><ul><li>One</li><li>Two</li></ul><blockquote>Three</blockquote>") == Measure(
4, 16
)
@pytest.mark.parametrize(
"book",
[
"".join(f"<p>{line}</p>" for line in TEXT),
"".join(f'<p class="a">{line}</p>\n' for line in TEXT),
"<div>" + "".join(f"<div>{line}</div>" for line in TEXT) + "</div>",
"<div>" + "<br/><br/>".join(TEXT) + "</div>",
"<div>" + "<br>\n".join(TEXT) + "</div>",
f"<div>{TEXT[0]}<p>{TEXT[1]}</p>{TEXT[2]}</div>",
],
)
def test_the_same_text_gives_the_same_measure_in_each_markup(book):
assert pages.measure(book) == Measure(3, CHARS)
def test_inline_elements_and_table_cells_stay_in_the_line():
assert pages.measure('<p>ab<b>cd</b><a href="x">ef</a> <span><i>gh</i></span></p>') == Measure(1, 9)
assert pages.measure("<table><tr><td>Name</td><td>Value</td></tr><tr><th>a</th><td>b</td></tr></table>") == Measure(
2, 11
)
assert pages.measure('<p>Before<img src="a.png"/>after</p>') == Measure(1, 11)
def test_only_the_text_of_the_book_counts():
document = (
'<?xml version="1.0"?><html><head><title>Hidden title</title><style>p { color: red }</style>'
'<link href="a.css"/></head><body><script>var hidden;</script><script src="x.js"/>'
"<!-- a note --><p>Shown &amp; counted</p>\n\n<p> </p><p></p></body></html>"
)
assert pages.measure(document) == Measure(1, len("Shown & counted"))
assert pages.measure("") == Measure(0, 0)
assert pages.measure("<p>\n</p>") == Measure(0, 0)
def test_a_paragraph_that_only_spaces_the_text_is_not_counted():
spaced = "<p>One</p><p>&nbsp;</p><p>\xa0</p><p> &#160; </p><p><br/></p><p><span>&nbsp;</span></p><p>Two</p>"
assert pages.measure(spaced) == pages.measure("<p>One</p><p>Two</p>") == Measure(2, 6)
assert pages.measure("<div>One<br/>&nbsp;<br/>Two</div>") == Measure(2, 6)
# A no-break space inside a paragraph is a character of its text.
assert pages.measure("<p>One&nbsp;<b>two</b>\xa0three</p>") == Measure(1, 13)
def test_a_page_is_31_lines_and_a_line_is_a_paragraph_or_70_characters():
assert pages.page_count(Measure(62, 620)) == (62 + 620 // 70) // 31 == 2
assert pages.page_count(Measure(31 * 9, 31 * 9)) == 9
assert pages.page_count(Measure(1000, 177_000)) == (1000 + 177_000 // 70) // 31
def test_a_short_book_has_one_page_and_a_book_with_no_text_has_none():
assert pages.page_count(Measure(1, 10)) == 1
assert pages.page_count(Measure(30, 30)) == 1
assert pages.page_count(Measure()) == 0
# One block of text: 2400 characters start the second page before 31 lines do.
assert pages.page_count(Measure(1, 2399)) == 1
assert pages.page_count(Measure(1, 2400)) == 2
def epub(path, documents, spine=None):
names = [f"file {n}.xhtml" for n in range(len(documents))]
items = "".join(f'<item id="f{n}" href="{name.replace(" ", "%20")}"/>' for n, name in enumerate(names))
items += '<item id="css" href="stylesheet.css"/>'
refs = "".join(f'<itemref idref="f{n}"/>' for n in (spine or range(len(documents))))
with zipfile.ZipFile(path, "w") as z:
z.writestr(
"META-INF/container.xml",
'<container xmlns="urn:oasis:names:tc:opendocument:xmlns:container"><rootfiles>'
'<rootfile full-path="OEBPS/content.opf"/></rootfiles></container>',
)
z.writestr(
"OEBPS/content.opf",
f'<package xmlns="http://www.idpf.org/2007/opf"><metadata/><manifest>{items}</manifest>'
f"<spine>{refs}</spine></package>",
)
z.writestr("OEBPS/stylesheet.css", "p { margin: 0 }")
for name, document in zip(names, documents, strict=True):
z.writestr(f"OEBPS/{name}", document)
return path
def test_an_epub_is_counted_from_the_documents_of_its_spine(tmp_path):
chapter = "<html><head><title>T</title></head><body>" + "<p>x</p>" * 31 + "</body></html>"
book = epub(tmp_path / "book.epub", [chapter, "<html><body><p>Not in the spine</p></body></html>", chapter], [2, 0])
assert spine_documents(book) == [chapter, chapter]
# 62 paragraphs of one character.
assert pages.epub_pages(book) == 2
assert pages.epub_pages(epub(tmp_path / "empty.epub", ["<html><body><img src='a.png'/></body></html>"])) == 0
def test_a_file_that_is_not_an_epub_cannot_be_counted(tmp_path):
path = tmp_path / "bad.epub"
path.write_bytes(b"not a zip")
with pytest.raises(EpubError):
pages.epub_pages(path)
with zipfile.ZipFile(path, "w") as z:
z.writestr("META-INF/container.xml", "<container/>")
with pytest.raises(EpubError):
pages.epub_pages(path)
tests/test_remove_smart_quote_revisions.py
+137
View File
@@ -0,0 +1,137 @@
"""Tests for scripts/remove_smart_quote_revisions.py, a manual cleanup tool."""
import pytest
from bindery import diffing
from bindery.store import Store
from scripts.remove_smart_quote_revisions import main, straighten_html, text_blocks, text_hash
CURLY = "<div><p class='a1'>“Hi,” he said.</p><p>It’s ‘fine’…</p></div>"
STRAIGHT = "<div><p class='b2'>\"Hi,\" he said.</p><p>It's 'fine'...</p></div>"
# The same text as CURLY, with the smart punctuation as entities.
ENTITIES = "<div><p>&ldquo;Hi,&#8221; he said.</p><p>It&rsquo;s &#x2018;fine&#x2019;&hellip;</p></div>"
@pytest.mark.parametrize(
"html",
[
CURLY,
ENTITIES,
"<p>One<br/>Two</p><script>x()</script><style>p{}</style>",
"<div>Loose <em>text</em><ul><li>a</li><li>b &amp; c</li></ul> tail </div><hr/>",
"",
],
)
def test_text_and_hash_match_bindery(html):
assert text_blocks(html) == diffing.html_to_blocks(html)
assert text_hash(html) == diffing.text_hash(html)
def test_straighten_html_changes_only_the_text():
html = '<p title="“x”">“Hi,” it’s ‘fine’… &amp; &rsquo;<!-- “note” --></p>'
assert straighten_html(html) == "<p title=\"“x”\">\"Hi,\" it's 'fine'... &amp; '<!-- “note” --></p>"
assert diffing.html_to_blocks(straighten_html(ENTITIES)) == diffing.html_to_blocks(STRAIGHT)
@pytest.fixture
def db(tmp_path):
path = tmp_path / "bindery.db"
store = Store(path)
story_id = store.create_story("https://example.com/s/1", "example.com", "Story", "Writer", [])
return path, store, story_id
def chapter(store, story_id, n, revisions, current=None):
"""A chapter with revisions given as (html, source). The last one is in the book, unless `current` says."""
html, source = revisions[0]
cid = store.add_chapter(story_id, f"u{n}", f"Chapter {n}", html, diffing.text_hash(html), source, n)
for html, source in revisions[1:]:
store.add_revision(cid, html, diffing.text_hash(html), source)
if current:
store.update_chapter(cid, current_rev=current)
return cid
def revs(store, cid):
return [r["rev"] for r in store.list_revisions(cid)]
def test_dry_run_changes_nothing(db, capsys):
path, store, story_id = db
cid = chapter(store, story_id, 1, [(CURLY, "import"), (STRAIGHT, "recheck")])
assert main([str(path)]) == 0
assert revs(store, cid) == [1, 2]
out = capsys.readouterr().out
assert "delete revision 1 (import), keep 2 (recheck)" in out
assert "Run again with --apply" in out
assert not list(path.parent.glob("*before-quote-cleanup*"))
def test_apply_deletes_revisions_that_differ_only_by_smart_punctuation(db, capsys):
path, store, story_id = db
# The smart import, then the straight site text.
older = chapter(store, story_id, 1, [(CURLY, "import"), (STRAIGHT, "recheck")])
# The same, but the user put the smart revision in the book.
chosen = chapter(store, story_id, 2, [(CURLY, "import"), (STRAIGHT, "recheck")], current=1)
# A smart import after the site text. It is the newest, but not from the site.
imported = chapter(store, story_id, 3, [(STRAIGHT, "new"), (ENTITIES, "import")])
# The site sends smart punctuation now. The next recheck would add it again.
site = chapter(store, story_id, 4, [(STRAIGHT, "new"), (CURLY, "recheck")])
assert main([str(path), "--apply"]) == 0
assert revs(store, older) == [2]
assert revs(store, chosen) == [2]
assert store.get_chapter(chosen)["current_rev"] == 2
assert revs(store, imported) == [1]
assert store.get_chapter(imported)["current_rev"] == 1
assert revs(store, site) == [1, 2]
out = capsys.readouterr().out
assert "3 revisions to delete, and 0 revisions to make straight" in out
assert "not changed: revision 2 (recheck) is the newest text from the site" in out
assert "Rebuild them:\n - Story" in out
(backup,) = path.parent.glob("bindery.before-quote-cleanup-*.db")
assert revs(Store(backup), older) == [1, 2]
def test_apply_makes_revisions_with_other_changes_straight(db, capsys):
path, store, story_id = db
changed = CURLY.replace("fine", "good")
# Three different texts. The newest is not from the site, so all of them get straight.
both = chapter(
store, story_id, 1, [(CURLY, "import"), (changed, "recheck"), (ENTITIES.replace("Hi", "Yo"), "import")]
)
# The newest is from the site, so only the older one gets straight.
older = chapter(store, story_id, 2, [(CURLY, "import"), (changed, "recheck")])
# A smart revision that also matches a straight one goes, and the other text gets straight.
mixed = chapter(store, story_id, 3, [(CURLY, "import"), (STRAIGHT, "new"), (changed, "import")], current=1)
assert main([str(path), "--apply"]) == 0
def smart(cid, rev):
text = "\n".join(diffing.html_to_blocks(store.revision_html(cid, rev)))
return any(c in text for c in "‘’“”…")
assert [smart(both, rev) for rev in (1, 2, 3)] == [False, False, False]
assert [smart(older, rev) for rev in (1, 2)] == [False, True]
assert revs(store, mixed) == [2, 3]
assert [smart(mixed, rev) for rev in (2, 3)] == [False, False]
assert store.get_chapter(mixed)["current_rev"] == 2
# Bindery compares the next site text with the stored hash, so it must be the hash of the new text.
for cid in (both, older, mixed):
for r in store.list_revisions(cid):
assert r["text_hash"] == diffing.text_hash(store.revision_html(cid, r["rev"]))
assert "class='a1'" in store.revision_html(both, 1)
out = capsys.readouterr().out
assert "make the punctuation straight in revision 1 (import), 2 (recheck), 3 (import)" in out
assert "1 revisions to delete, and 5 revisions to make straight" in out
# A second run finds nothing to do.
assert main([str(path)]) == 0
assert "not changed: revision 2 (recheck)" in capsys.readouterr().out
def test_no_work_and_a_missing_file(db, tmp_path, capsys):
path, store, story_id = db
chapter(store, story_id, 1, [(STRAIGHT, "new")])
assert main([str(path), "--apply"]) == 0
assert "No revisions to change." in capsys.readouterr().out
assert main([str(tmp_path / "missing.db")]) == 1
tests/test_store.py
+15 -3
View File
@@ -6,13 +6,15 @@ def test_chapters_revisions_and_rules(tmp_path):
sid = store.create_story("https://example/s/1", "example", "T", "A", ["title"])
c1 = store.add_chapter(sid, "u1", "One", "<p>1</p>", "h1", "new", 0)
c2 = store.add_chapter(sid, "u2", "Two", "<p>2</p>", "h2", "new", 1)
store.set_positions(sid, [c2, c1])
store.reorder_chapters(sid, lambda order: [c2, c1])
assert [c["url"] for c in store.chapters_for_build(sid)] == ["u2", "u1"]
assert store.add_revision(c1, "<p>1b</p>", "h1b", "recheck") == 2
assert store.current_revisions(sid)[c1] == {"rev": 2, "text_hash": "h1b", "source": "recheck"}
newest = store.list_revisions(c1)[-1]
assert (newest["rev"], newest["text_hash"], newest["source"]) == (2, "h1b", "recheck")
assert store.get_chapter(c1)["current_rev"] == 2
store.replace_revision(c1, 2, "<p>1c</p>", "h1c")
assert store.current_revisions(sid)[c1]["text_hash"] == "h1c"
assert store.list_revisions(c1)[-1]["text_hash"] == "h1c"
assert store.revision_html(c1, 1) == "<p>1</p>"
store.update_chapter(c1, current_rev=1, included=0)
assert [c["url"] for c in store.chapters_for_build(sid)] == ["u2"]
@@ -47,6 +49,16 @@ def test_list_stories_counts_offsite_and_revised_chapters(tmp_path):
assert story["revised_count"] == 1
def test_list_stories_sorts_titles_without_leading_articles(tmp_path):
store = Store(tmp_path / "db.sqlite")
for n, title in enumerate(["The Zebra", "an Owl", "Anthem", "A Bee", "Theory", "Cat"]):
store.create_story(f"https://example/s/{n}", "example", title, "A", [])
titles = [s["title"] for s in store.list_stories()]
assert titles == ["Anthem", "A Bee", "Cat", "an Owl", "Theory", "The Zebra"]
# The web interface sorts by the same key.
assert [s["sort_title"] for s in store.list_stories()] == ["anthem", "bee", "cat", "owl", "theory", "zebra"]
def test_cover_replaces_previous_cover(tmp_path):
store = Store(tmp_path / "db.sqlite")
sid = store.create_story("u", "s", "T", "A", [])
tests/test_worker_flows.py
+82 -2
View File
@@ -6,7 +6,7 @@ from unittest.mock import Mock
import pytest
from bindery import config, fff
from bindery import config, fff, web
from bindery import orchestrator as o
from bindery.diffing import text_hash
from bindery.jobs import DownloadMode, Job, JobQueue, QueueStatus
@@ -25,6 +25,8 @@ def state(monkeypatch, tmp_path):
monkeypatch.setattr(o, "job_queue", queue)
monkeypatch.setattr(config, "CWA_URL", "")
monkeypatch.setattr(config, "SITE_DELAY", 7)
monkeypatch.setattr(config, "TMP_DIR", tmp_path / "tmp")
config.TMP_DIR.mkdir()
return store, queue
@@ -85,6 +87,42 @@ def test_worker_persists_outcomes_and_continues_after_failure(state, monkeypatch
assert [c.args[0] for c in sleeps.call_args_list] == [2, 7]
def test_worker_goes_on_after_an_error_of_the_queue(state, monkeypatch, caplog):
store, queue = state
job = queue.add(Job(mode=DownloadMode.REBUILD))
monkeypatch.setattr(queue, "get_next", Mock(side_effect=[OSError("disk full"), job, StopLoop()]))
monkeypatch.setattr(queue, "update", Mock(side_effect=[OSError("disk full"), None, None]))
sleeps = Mock()
monkeypatch.setattr(o.time, "sleep", sleeps)
with pytest.raises(StopLoop):
o.download_loop()
assert caplog.text.count("Job loop error: disk full") == 2
assert [c.args[0] for c in sleeps.call_args_list] == [2, 2]
def test_health_reports_a_stopped_job_thread(state, monkeypatch):
client = web.app.test_client()
assert o.stopped_threads() == []
alive, dead = Mock(), Mock()
alive.is_alive.return_value, dead.is_alive.return_value, dead.name = True, False, "download_loop"
monkeypatch.setattr(o, "_threads", [alive, dead])
response = client.get("/api/health")
assert response.status_code == 503
assert response.json == {"status": "error", "error": "Stopped: download_loop. Restart Bindery."}
monkeypatch.setattr(o, "_threads", [alive])
assert client.get("/api/health").json == {"status": "ok"}
def test_start_threads_keeps_the_threads_for_the_health_check(state, monkeypatch):
monkeypatch.setattr(o, "_threads", [])
monkeypatch.setattr(o, "download_loop", lambda: None)
monkeypatch.setattr(o, "scheduler_loop", lambda: None)
o.start_threads()
for thread in o._threads:
thread.join()
assert o.stopped_threads() == ["<lambda>", "<lambda>"]
def test_scheduler_skips_disabled_busy_and_recent_stories(state, monkeypatch):
store, queue = state
now = datetime(2026, 6, 1, 12)
@@ -151,7 +189,8 @@ def site(state, monkeypatch):
calibre={"title": "Title"},
)
fetch = Mock(side_effect=lambda a, url, i: f"<p>fresh {url}</p>")
build, deliver = Mock(), Mock(return_value="Sent")
build = Mock(side_effect=lambda story, overrides, path: path.write_bytes(b"epub"))
deliver = Mock(return_value="Sent")
monkeypatch.setattr(fff, "get_adapter", lambda url: adapter)
monkeypatch.setattr(fff, "fetch_story", lambda a: snap)
monkeypatch.setattr(fff, "fetch_chapter", fetch)
@@ -250,6 +289,47 @@ def test_inclusion_changes_only_the_requested_story_and_queues_a_rebuild(state):
assert len(queue.get_status()) == 1
def test_move_chapters_keeps_their_order_and_queues_a_rebuild(state):
store, queue = state
sid, other = add_story(store), add_story(store, "other")
ids = [store.add_chapter(sid, f"c{n}", str(n), "<p>x</p>", "h", "new", n) for n in range(5)]
foreign = store.add_chapter(other, "foreign", "Other", "<p>x</p>", "h", "new", 0)
def titles():
return [c["title"] for c in store.list_chapters(sid)]
# The selection order does not count, and a chapter of another story is ignored.
assert o.move_chapters(sid, [ids[3], ids[1], foreign], ids[4])["mode"] == "rebuild"
assert titles() == ["0", "2", "4", "1", "3"]
o.move_chapters(sid, [ids[4], ids[3]], None)
assert titles() == ["4", "3", "0", "2", "1"]
assert len(queue.get_status()) == 1
with pytest.raises(ValueError, match="Select"):
o.move_chapters(sid, [foreign], None)
with pytest.raises(ValueError, match="moves with them"):
o.move_chapters(sid, [ids[0], ids[1]], ids[1])
with pytest.raises(KeyError):
o.move_chapters(sid, [ids[0]], foreign)
def test_delete_chapters_removes_their_revisions_and_keeps_one_chapter(state):
store, queue = state
sid, other = add_story(store), add_story(store, "other")
ids = [store.add_chapter(sid, f"c{n}", str(n), "<p>x</p>", "h", "new", n) for n in range(3)]
store.add_revision(ids[1], "<p>y</p>", "h2", "recheck")
foreign = store.add_chapter(other, "foreign", "Other", "<p>x</p>", "h", "new", 0)
assert o.delete_chapters(sid, [ids[1], foreign])["mode"] == "rebuild"
assert [c["id"] for c in store.list_chapters(sid)] == [ids[0], ids[2]]
assert store.list_revisions(ids[1]) == []
assert store.get_chapter(foreign)
with pytest.raises(ValueError, match="Select"):
o.delete_chapters(sid, [foreign])
with pytest.raises(ValueError, match="at least one chapter"):
o.delete_chapters(sid, [ids[0], ids[2]])
assert len(store.list_chapters(sid)) == 2
@pytest.mark.parametrize(
"mode,chapters,fields,message",
[