Build and send a book only when it changed
A key of the chapters, rules and metadata of a story tells if its book changed. A check without a change no longer builds the book again, also for a book that waits for an upload. A held book goes out without a new build. The Rebuild button still builds and sends in each case.
This commit is contained in:
@@ -71,7 +71,7 @@ Environment variables set the configuration. **Restart the container after a cha
|
|||||||
| `URL_BASE` | A path prefix, such as `/bindery`. See "Reverse proxy". | None |
|
| `URL_BASE` | A path prefix, such as `/bindery`. See "Reverse proxy". | None |
|
||||||
| `INGEST_DIR` | The CWA ingest folder. If unset, all uploads stay paused. | None |
|
| `INGEST_DIR` | The CWA ingest folder. If unset, all uploads stay paused. | None |
|
||||||
| `CONFIG_DIR` | The story database and `personal.ini` | `/config` |
|
| `CONFIG_DIR` | The story database and `personal.ini` | `/config` |
|
||||||
| `TMP_DIR` | The folder for ePubs while Bindery builds them | `/tmp/bindery` |
|
| `TMP_DIR` | The folder for the built ePubs. Bindery keeps the last build of each story. | `/tmp/bindery` |
|
||||||
| `LOG_LEVEL` | The log level | `INFO` |
|
| `LOG_LEVEL` | The log level | `INFO` |
|
||||||
|
|
||||||
Separate the values in a list with commas.
|
Separate the values in a list with commas.
|
||||||
@@ -318,7 +318,7 @@ for approval, library linking is on, and uploads are not paused. Bindery keeps t
|
|||||||
Without either value, it stays off and disabled. Turn it off to skip matching with a configured library connection.
|
Without either value, it stays off and disabled. Turn it off to skip matching with a configured library connection.
|
||||||
Uploads use saved CWA book IDs when available. Without an ID, every build goes to `INGEST_DIR` as a plain ePub.
|
Uploads use saved CWA book IDs when available. Without an ID, every build goes to `INGEST_DIR` as a plain ePub.
|
||||||
If CWA imports these files, updates will create duplicate books. Settings and the story page show a highlighted warning.
|
If CWA imports these files, updates will create duplicate books. Settings and the story page show a highlighted warning.
|
||||||
Changing this setting rebuilds unsent books unless uploads are paused.
|
A change of this setting sends the unsent books, unless uploads are paused.
|
||||||
- **Hold new links for approval.** This setting applies only when library linking is on.
|
- **Hold new links for approval.** This setting applies only when library linking is on.
|
||||||
Use this to catch a story that is linked to the wrong CWA book.
|
Use this to catch a story that is linked to the wrong CWA book.
|
||||||
The first upload of a new story waits for your approval. The first upload after a change of the
|
The first upload of a new story waits for your approval. The first upload after a change of the
|
||||||
@@ -326,7 +326,7 @@ for approval, library linking is on, and uploads are not paused. Bindery keeps t
|
|||||||
the linked book is gone. The story page names the book that gets the update, or says that the
|
the linked book is gone. The story page names the book that gets the update, or says that the
|
||||||
story goes to CWA as a new book.
|
story goes to CWA as a new book.
|
||||||
- **Pause all uploads.** No book goes to CWA. This is always checked when `INGEST_DIR` is unset.
|
- **Pause all uploads.** No book goes to CWA. This is always checked when `INGEST_DIR` is unset.
|
||||||
When you turn the pause off, Bindery rebuilds unsent books. The other upload settings still apply.
|
When you turn the pause off, Bindery sends the unsent books. The other upload settings still apply.
|
||||||
While paused, use "Approve" or "Approve as a new book instead" to save a decision without sending the book.
|
While paused, use "Approve" or "Approve as a new book instead" to save a decision without sending the book.
|
||||||
|
|
||||||
To approve a link, open the story page and do one of these:
|
To approve a link, open the story page and do one of these:
|
||||||
@@ -380,7 +380,11 @@ downloaded before.
|
|||||||
| Recheck all chapters | Downloads every chapter on the site again, and new chapters too. |
|
| Recheck all chapters | Downloads every chapter on the site again, and new chapters too. |
|
||||||
| Download and replace entire book | Downloads every chapter again. Chapters that the site no longer lists leave the book. They stay stored, and you can put them back. |
|
| Download and replace entire book | Downloads every chapter again. Chapters that the site no longer lists leave the book. They stay stored, and you can put them back. |
|
||||||
| Update metadata from site | Updates only the metadata in the ePub. |
|
| Update metadata from site | Updates only the metadata in the ePub. |
|
||||||
| Rebuild book | Builds the book again from the stored chapters, for example after you change a rule. |
|
| Rebuild book | Builds the book again from the stored chapters, and sends it. |
|
||||||
|
|
||||||
|
**Bindery builds and sends a book only if the book changed.** A change is a new, changed, moved or
|
||||||
|
renamed chapter, a changed rule, or changed metadata. A job that finds no change answers "No changes"
|
||||||
|
and sends nothing to CWA. Only "Rebuild book" builds and sends a book that did not change.
|
||||||
|
|
||||||
### Find edited chapters
|
### Find edited chapters
|
||||||
|
|
||||||
|
|||||||
+9
-4
@@ -238,16 +238,21 @@ def word_count(chapters: list[dict[str, Any]]) -> int:
|
|||||||
return sum(len(block.split()) for c in chapters for block in html_to_blocks(c["html"]))
|
return sum(len(block.split()) for c in chapters for block in html_to_blocks(c["html"]))
|
||||||
|
|
||||||
|
|
||||||
|
def effective_overrides(overrides: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""The kept values that replace a site value. An empty title, author list or language does not."""
|
||||||
|
return {k: v for k, v in overrides.items() if v or k not in ("title", "authors", "languages")}
|
||||||
|
|
||||||
|
|
||||||
def apply_overrides(story, overrides: dict[str, Any]) -> None:
|
def apply_overrides(story, overrides: dict[str, Any]) -> None:
|
||||||
"""Put the kept values of locked fields into FanFicFare's story metadata.
|
"""Put the kept values of locked fields into FanFicFare's story metadata.
|
||||||
|
|
||||||
The epub writer takes the OPF metadata from these entries, and CWA takes
|
The epub writer takes the OPF metadata from these entries, and CWA takes
|
||||||
the book record from the OPF on the first import.
|
the book record from the OPF on the first import.
|
||||||
"""
|
"""
|
||||||
for key, value in overrides.items():
|
for key, value in effective_overrides(overrides).items():
|
||||||
if key == "title" and value:
|
if key == "title":
|
||||||
story.setMetadata("title", value)
|
story.setMetadata("title", value)
|
||||||
elif key == "authors" and value:
|
elif key == "authors":
|
||||||
for n, author in enumerate(value):
|
for n, author in enumerate(value):
|
||||||
story.addToList("author", author, clear=(n == 0))
|
story.addToList("author", author, clear=(n == 0))
|
||||||
elif key == "comments":
|
elif key == "comments":
|
||||||
@@ -263,7 +268,7 @@ def apply_overrides(story, overrides: dict[str, Any]) -> None:
|
|||||||
story.setMetadata("datePublished", datetime.fromisoformat(value) if value else None)
|
story.setMetadata("datePublished", datetime.fromisoformat(value) if value else None)
|
||||||
except ValueError:
|
except ValueError:
|
||||||
logger.warning(f"Ignoring a kept publication date that is not a date: {value!r}")
|
logger.warning(f"Ignoring a kept publication date that is not a date: {value!r}")
|
||||||
elif key == "languages" and value:
|
elif key == "languages":
|
||||||
story.setMetadata("langcode", value)
|
story.setMetadata("langcode", value)
|
||||||
elif key == "tags":
|
elif key == "tags":
|
||||||
tags = list(value or [])
|
tags = list(value or [])
|
||||||
|
|||||||
+127
-42
@@ -1,6 +1,7 @@
|
|||||||
"""Download modes, metadata locks, delivery to CWA, and the job threads."""
|
"""Download modes, metadata locks, delivery to CWA, and the job threads."""
|
||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
|
import json
|
||||||
import logging
|
import logging
|
||||||
import re
|
import re
|
||||||
import threading
|
import threading
|
||||||
@@ -94,8 +95,13 @@ def queue_urls(urls: list[str]) -> tuple[list[dict[str, Any]], list[str]]:
|
|||||||
|
|
||||||
|
|
||||||
def queue_story(
|
def queue_story(
|
||||||
story_id: int, mode: str, chapter_ids: list[int] | None = None, force_fields: list[str] | None = None
|
story_id: int,
|
||||||
|
mode: str,
|
||||||
|
chapter_ids: list[int] | None = None,
|
||||||
|
force_fields: list[str] | None = None,
|
||||||
|
force: bool = False,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
|
"""Queue a job for a story. With `force`, a rebuild also builds and sends a book that did not change."""
|
||||||
story = _require_story(story_id)
|
story = _require_story(story_id)
|
||||||
mode_enum = DownloadMode(mode)
|
mode_enum = DownloadMode(mode)
|
||||||
if mode_enum == DownloadMode.IMPORT:
|
if mode_enum == DownloadMode.IMPORT:
|
||||||
@@ -111,6 +117,7 @@ def queue_story(
|
|||||||
title=story["title"],
|
title=story["title"],
|
||||||
chapter_ids=sorted(int(c) for c in chapter_ids or []),
|
chapter_ids=sorted(int(c) for c in chapter_ids or []),
|
||||||
force_fields=sorted(force_fields or []),
|
force_fields=sorted(force_fields or []),
|
||||||
|
options={"force": True} if force and mode_enum == DownloadMode.REBUILD else {},
|
||||||
)
|
)
|
||||||
return job_queue.add(job).to_dict()
|
return job_queue.add(job).to_dict()
|
||||||
|
|
||||||
@@ -674,6 +681,60 @@ def _build(story: dict[str, Any], overrides: dict[str, Any], out_path: Path) ->
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Built books. The key of a story is a hash of all that goes into its epub, so the book changed when the key
|
||||||
|
# changed. The name of the built file holds the key. The story settings "built_key" and "sent_key" are the keys of
|
||||||
|
# the last build and of the book that went to CWA.
|
||||||
|
|
||||||
|
_build_lock = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
|
def _book_metadata(site: dict[str, Any], overrides: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""The metadata of the book: the site values, with the kept values in their place."""
|
||||||
|
meta = dict(site)
|
||||||
|
for field, value in fff.effective_overrides(overrides).items():
|
||||||
|
if field == "series":
|
||||||
|
meta["series"], meta["series_index"] = value or ("", None)
|
||||||
|
else:
|
||||||
|
meta[field] = value
|
||||||
|
return meta
|
||||||
|
|
||||||
|
|
||||||
|
def _build_key(story: dict[str, Any], overrides: dict[str, Any], site: dict[str, Any]) -> str:
|
||||||
|
story_id = story["id"]
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
state = [
|
||||||
|
story["url"],
|
||||||
|
_book_metadata(site, overrides),
|
||||||
|
store.rules(None),
|
||||||
|
store.rules(story_id),
|
||||||
|
_paragraphs(story_id),
|
||||||
|
store.image_sizes(story_id),
|
||||||
|
]
|
||||||
|
digest.update(json.dumps(state, sort_keys=True, default=str).encode())
|
||||||
|
for c in store.chapters_for_build(story_id):
|
||||||
|
digest.update(json.dumps([c["url"], c["title"], c["html"]]).encode())
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _built_epub(story: dict[str, Any], overrides: dict[str, Any], key: str, force: bool = False) -> Path:
|
||||||
|
"""The epub of the story for this key. Bindery builds it only if it has no file for the key."""
|
||||||
|
path = config.TMP_DIR / f"story-{story['id']}-{key[:16]}.epub"
|
||||||
|
with _build_lock:
|
||||||
|
if force or not path.exists():
|
||||||
|
# A build under another name first, so a download never reads half a book.
|
||||||
|
partial = config.TMP_DIR / f"building-{uuid.uuid4().hex}.epub"
|
||||||
|
try:
|
||||||
|
_build(story, overrides, partial)
|
||||||
|
partial.replace(path)
|
||||||
|
finally:
|
||||||
|
partial.unlink(missing_ok=True)
|
||||||
|
for old in config.TMP_DIR.glob(f"story-{story['id']}-*.epub"):
|
||||||
|
if old != path:
|
||||||
|
old.unlink(missing_ok=True)
|
||||||
|
store.set_setting(story["id"], "built_key", key)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
def _find_book(story_url: str, titles: list[str | None]) -> tuple[Book, Path] | None:
|
def _find_book(story_url: str, titles: list[str | None]) -> tuple[Book, Path] | None:
|
||||||
"""Search CWA for the book of a story. Returns the book and its downloaded epub."""
|
"""Search CWA for the book of a story. Returns the book and its downloaded epub."""
|
||||||
linked = {s["calibre_id"] for s in store.list_stories() if s["calibre_id"]}
|
linked = {s["calibre_id"] for s in store.list_stories() if s["calibre_id"]}
|
||||||
@@ -744,6 +805,8 @@ def _uploads_paused() -> bool:
|
|||||||
|
|
||||||
def _link_changed(story_id: int) -> None:
|
def _link_changed(story_id: int) -> None:
|
||||||
store.set_setting(story_id, "unapproved_link", True)
|
store.set_setting(story_id, "unapproved_link", True)
|
||||||
|
# Another book, or no book, does not have the epub that Bindery sent.
|
||||||
|
store.set_setting(story_id, "sent_key", None)
|
||||||
|
|
||||||
|
|
||||||
def _needs_approval(story_id: int) -> bool:
|
def _needs_approval(story_id: int) -> bool:
|
||||||
@@ -773,10 +836,6 @@ def _hold_reason(story: dict[str, Any], book: dict[str, Any] | None) -> str | No
|
|||||||
|
|
||||||
|
|
||||||
def _deliver(story: dict[str, Any], epub: Path, book: dict[str, Any] | None, title: str, author: str) -> str:
|
def _deliver(story: dict[str, Any], epub: Path, book: dict[str, Any] | None, title: str, author: str) -> str:
|
||||||
held = _hold_reason(story, book)
|
|
||||||
if held:
|
|
||||||
store.set_setting(story["id"], "upload_held", True)
|
|
||||||
return held
|
|
||||||
# With linking on, do not create a duplicate when the library book cannot be found.
|
# With linking on, do not create a duplicate when the library book cannot be found.
|
||||||
if _link_books() and not book and story["ingested_at"]:
|
if _link_books() and not book and story["ingested_at"]:
|
||||||
raise StoryError(NOT_FOUND)
|
raise StoryError(NOT_FOUND)
|
||||||
@@ -794,6 +853,44 @@ def _deliver(story: dict[str, Any], epub: Path, book: dict[str, Any] | None, tit
|
|||||||
return message
|
return message
|
||||||
|
|
||||||
|
|
||||||
|
def _send(
|
||||||
|
story: dict[str, Any], overrides: dict[str, Any], key: str, title: str, author: str, force: bool = False
|
||||||
|
) -> tuple[str, bool]:
|
||||||
|
"""Build the book if it changed, and deliver it. Returns the message, and whether the book changed.
|
||||||
|
|
||||||
|
A held book that did not change is not built again. With `force`, Bindery builds the book in each case.
|
||||||
|
"""
|
||||||
|
story_id = story["id"]
|
||||||
|
book = _linked_book(story)
|
||||||
|
held = _hold_reason(story, book)
|
||||||
|
changed = key != store.get_setting(story_id, "built_key")
|
||||||
|
if held:
|
||||||
|
if changed or force:
|
||||||
|
_built_epub(story, overrides, key, force)
|
||||||
|
store.set_setting(story_id, "upload_held", True)
|
||||||
|
return held, changed
|
||||||
|
message = _deliver(story, _built_epub(story, overrides, key, force), book, title, author)
|
||||||
|
store.set_setting(story_id, "sent_key", key)
|
||||||
|
return message, changed
|
||||||
|
|
||||||
|
|
||||||
|
def adopt_build_keys() -> None:
|
||||||
|
"""Give the stories of an older Bindery their keys, so the first check does not send each book again."""
|
||||||
|
built = store.story_settings("built_key")
|
||||||
|
held = store.story_settings("upload_held")
|
||||||
|
for listed in store.list_stories():
|
||||||
|
story_id = listed["id"]
|
||||||
|
if story_id in built or listed["status"] == "error" or job_queue.has_active(story_id):
|
||||||
|
continue
|
||||||
|
story = _require_story(story_id)
|
||||||
|
if not story["metahtml"]:
|
||||||
|
continue
|
||||||
|
key = _build_key(story, _epub_overrides(story, story["pinned"]), story["site_values"])
|
||||||
|
store.set_setting(story_id, "built_key", key)
|
||||||
|
if (story["calibre_id"] or story["ingested_at"]) and not held.get(story_id):
|
||||||
|
store.set_setting(story_id, "sent_key", key)
|
||||||
|
|
||||||
|
|
||||||
# Jobs
|
# Jobs
|
||||||
|
|
||||||
|
|
||||||
@@ -920,12 +1017,15 @@ def _run_import_file(job: Job) -> str:
|
|||||||
def _run_rebuild(job: Job) -> str:
|
def _run_rebuild(job: Job) -> str:
|
||||||
story = _require_story(job.story_id)
|
story = _require_story(job.story_id)
|
||||||
job_queue.update(job, title=story["title"])
|
job_queue.update(job, title=story["title"])
|
||||||
book = _linked_book(story)
|
|
||||||
pinned = _resolve_metadata(story, story["site_values"], [])
|
pinned = _resolve_metadata(story, story["site_values"], [])
|
||||||
epub = config.TMP_DIR / f"story-{story['id']}.epub"
|
overrides = _epub_overrides(story, pinned)
|
||||||
_build(story, _epub_overrides(story, pinned), epub)
|
key = _build_key(story, overrides, story["site_values"])
|
||||||
message = _deliver(story, epub, book, story["title"], story["author"])
|
force = bool(job.options.get("force"))
|
||||||
store.update_story(story["id"], pinned=pinned, status="ok", message=message, last_changed=now())
|
if not force and key == store.get_setting(story["id"], "sent_key"):
|
||||||
|
return "No changes"
|
||||||
|
message, changed = _send(story, overrides, key, story["title"], story["author"], force)
|
||||||
|
changes: dict[str, Any] = {"last_changed": now()} if changed or force else {}
|
||||||
|
store.update_story(story["id"], pinned=pinned, status="ok", message=message, **changes)
|
||||||
return message
|
return message
|
||||||
|
|
||||||
|
|
||||||
@@ -1149,46 +1249,27 @@ def _run_site_job(job: Job) -> str:
|
|||||||
job, result={"new": new_count, "changed": changed, "failed": failed, "skipped": skipped, "removed": removed}
|
job, result={"new": new_count, "changed": changed, "failed": failed, "skipped": skipped, "removed": removed}
|
||||||
)
|
)
|
||||||
|
|
||||||
metadata_changed = snap.calibre != story["site_values"]
|
|
||||||
delivered = bool(story["calibre_id"] or story["ingested_at"])
|
|
||||||
# A resumed job builds the book, because the run before the restart can have stored chapters.
|
|
||||||
needs_build = (
|
|
||||||
new_count
|
|
||||||
or changed
|
|
||||||
or renamed_count
|
|
||||||
or job.force_fields
|
|
||||||
or metadata_changed
|
|
||||||
or not delivered
|
|
||||||
or job.options.get("build")
|
|
||||||
or job.restarts
|
|
||||||
or mode in (DownloadMode.FULL, DownloadMode.METADATA)
|
|
||||||
or (mode == DownloadMode.NEW and story["status"] == "error" and not failed)
|
|
||||||
)
|
|
||||||
if mode in (DownloadMode.RECHECK_ALL, DownloadMode.FULL):
|
if mode in (DownloadMode.RECHECK_ALL, DownloadMode.FULL):
|
||||||
store.set_setting(story_id, "last_full_recheck", now())
|
store.set_setting(story_id, "last_full_recheck", now())
|
||||||
if not store.chapters_for_build(story_id):
|
if not store.chapters_for_build(story_id):
|
||||||
raise StoryError("; ".join(summary) + ". No chapters downloaded.")
|
raise StoryError("; ".join(summary) + ". No chapters downloaded.")
|
||||||
if not needs_build:
|
|
||||||
store.update_story(story_id, last_checked=now(), status="ok", message="No changes")
|
|
||||||
return "No changes. " + ", ".join(summary)
|
|
||||||
|
|
||||||
book = _linked_book(story)
|
# The book is built and sent only when its chapters, rules or metadata changed. A job that resumes, or that
|
||||||
|
# follows a failed job, finds the changes of the earlier run through the key.
|
||||||
pinned = _resolve_metadata(story, snap.calibre, job.force_fields)
|
pinned = _resolve_metadata(story, snap.calibre, job.force_fields)
|
||||||
title = _display_title(story, pinned, snap.title)
|
title = _display_title(story, pinned, snap.title)
|
||||||
|
overrides = _epub_overrides(story, pinned)
|
||||||
|
key = _build_key(story, overrides, snap.calibre)
|
||||||
|
checked: dict[str, Any] = {"pinned": pinned, "site_values": snap.calibre, "title": title, "last_checked": now()}
|
||||||
|
if key == store.get_setting(story_id, "sent_key"):
|
||||||
|
store.update_story(story_id, **checked, status="ok", message="No changes")
|
||||||
|
return "No changes. " + ", ".join(summary)
|
||||||
|
|
||||||
_progress(job, "Building epub", 1.0)
|
_progress(job, "Building epub", 1.0)
|
||||||
epub = config.TMP_DIR / f"story-{story_id}.epub"
|
message, book_changed = _send(story, overrides, key, title, snap.author)
|
||||||
_build(story, _epub_overrides(story, pinned), epub)
|
if book_changed:
|
||||||
message = _deliver(story, epub, book, title, snap.author)
|
checked["last_changed"] = now()
|
||||||
store.update_story(
|
store.update_story(story_id, **checked, message=message, status="error" if failed else "ok")
|
||||||
story_id,
|
|
||||||
pinned=pinned,
|
|
||||||
site_values=snap.calibre,
|
|
||||||
title=title,
|
|
||||||
last_checked=now(),
|
|
||||||
last_changed=now(),
|
|
||||||
message=message,
|
|
||||||
status="error" if failed else "ok",
|
|
||||||
)
|
|
||||||
return ", ".join(summary) + ". " + message
|
return ", ".join(summary) + ". " + message
|
||||||
|
|
||||||
|
|
||||||
@@ -1267,5 +1348,9 @@ def scheduler_loop() -> None:
|
|||||||
|
|
||||||
|
|
||||||
def start_threads() -> None:
|
def start_threads() -> None:
|
||||||
|
try:
|
||||||
|
adopt_build_keys()
|
||||||
|
except Exception as e:
|
||||||
|
logger.exception(f"Cannot check the built books: {e}")
|
||||||
for target in (download_loop, scheduler_loop):
|
for target in (download_loop, scheduler_loop):
|
||||||
threading.Thread(target=target, daemon=True, name=target.__name__).start()
|
threading.Thread(target=target, daemon=True, name=target.__name__).start()
|
||||||
|
|||||||
@@ -388,6 +388,14 @@ class Store:
|
|||||||
rows = conn.execute("SELECT url FROM images WHERE story_id = ? AND is_cover = 0", (story_id,))
|
rows = conn.execute("SELECT url FROM images WHERE story_id = ? AND is_cover = 0", (story_id,))
|
||||||
return {r["url"] for r in rows}
|
return {r["url"] for r in rows}
|
||||||
|
|
||||||
|
def image_sizes(self, story_id: int) -> list[list[Any]]:
|
||||||
|
"""The URL, the cover mark and the size of each image, without its data."""
|
||||||
|
with self._connect() as conn:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT url, is_cover, LENGTH(data) FROM images WHERE story_id = ? ORDER BY url, is_cover", (story_id,)
|
||||||
|
)
|
||||||
|
return [list(r) for r in rows]
|
||||||
|
|
||||||
def images(self, story_id: int) -> list[dict[str, Any]]:
|
def images(self, story_id: int) -> list[dict[str, Any]]:
|
||||||
with self._connect() as conn:
|
with self._connect() as conn:
|
||||||
rows = conn.execute(
|
rows = conn.execute(
|
||||||
|
|||||||
+2
-1
@@ -291,8 +291,9 @@ def delete_story(story_id: int) -> Any:
|
|||||||
@app.post("/api/stories/<int:story_id>/download")
|
@app.post("/api/stories/<int:story_id>/download")
|
||||||
def download(story_id: int) -> Any:
|
def download(story_id: int) -> Any:
|
||||||
data = _json()
|
data = _json()
|
||||||
|
# A rebuild that the user asks for also builds and sends a book that did not change.
|
||||||
return orchestrator.queue_story(
|
return orchestrator.queue_story(
|
||||||
story_id, data.get("mode", DownloadMode.NEW.value), data.get("chapter_ids"), data.get("force_fields")
|
story_id, data.get("mode", DownloadMode.NEW.value), data.get("chapter_ids"), data.get("force_fields"), True
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,205 @@
|
|||||||
|
"""A book is built and sent only when its chapters, rules or metadata changed."""
|
||||||
|
|
||||||
|
import dataclasses
|
||||||
|
from unittest.mock import Mock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from test_api import H
|
||||||
|
from test_api import client as client
|
||||||
|
from test_deliver import STORY_URL, add_story, ingest_files, linked_story, site, update
|
||||||
|
from test_deliver import fake as fake
|
||||||
|
from test_deliver import jobs as jobs
|
||||||
|
from test_deliver import story_epub as story_epub
|
||||||
|
|
||||||
|
from bindery import config, orchestrator
|
||||||
|
from bindery.jobs import DownloadMode, Job
|
||||||
|
|
||||||
|
ONE = [(1, "One", "a")]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def builds(fake, jobs, monkeypatch, tmp_path):
|
||||||
|
"""Count the builds, in a folder of built books that starts empty."""
|
||||||
|
monkeypatch.setattr(config, "TMP_DIR", tmp_path / "tmp")
|
||||||
|
config.TMP_DIR.mkdir()
|
||||||
|
spy = Mock(side_effect=orchestrator._build)
|
||||||
|
monkeypatch.setattr(orchestrator, "_build", spy)
|
||||||
|
return spy
|
||||||
|
|
||||||
|
|
||||||
|
def story(story_id):
|
||||||
|
return orchestrator.store.get_story(story_id)
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unchanged_story_is_not_built_or_sent_again(builds, monkeypatch):
|
||||||
|
story_id = linked_story(ONE)
|
||||||
|
site(monkeypatch, ONE)
|
||||||
|
assert "library book 1" in update(story_id)
|
||||||
|
changed = story(story_id)["last_changed"]
|
||||||
|
orchestrator.store.update_story(story_id, last_changed="2020-01-01T00:00:00")
|
||||||
|
for mode in (DownloadMode.NEW, DownloadMode.RECHECK_ALL, DownloadMode.METADATA, DownloadMode.FULL):
|
||||||
|
assert update(story_id, mode).startswith("No changes")
|
||||||
|
assert changed and story(story_id)["last_changed"] == "2020-01-01T00:00:00"
|
||||||
|
assert builds.call_count == 1
|
||||||
|
assert len(ingest_files()) == 2 # one epub and its sidecar
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("change", ["chapter", "rule", "global rule", "paragraphs", "metadata", "included"])
|
||||||
|
def test_each_change_builds_and_sends_the_book(builds, monkeypatch, change):
|
||||||
|
story_id = linked_story([(1, "One", "a"), (2, "Two", "b")])
|
||||||
|
site(monkeypatch, [(1, "One", "a"), (2, "Two", "b")])
|
||||||
|
update(story_id)
|
||||||
|
if change == "chapter":
|
||||||
|
site(monkeypatch, [(1, "One", "a"), (2, "Two", "b"), (3, "Three", "c")])
|
||||||
|
elif change == "rule":
|
||||||
|
orchestrator.save_rules(story_id, [{"pattern": "a", "replacement": "b"}])
|
||||||
|
elif change == "global rule":
|
||||||
|
orchestrator.save_rules(None, [{"pattern": "a", "replacement": "b"}])
|
||||||
|
elif change == "paragraphs":
|
||||||
|
orchestrator.save_rules(story_id, [], paragraphs=True)
|
||||||
|
elif change == "metadata":
|
||||||
|
orchestrator.store.update_story(story_id, pinned={"title": "Edited"})
|
||||||
|
else:
|
||||||
|
chapter = orchestrator.store.list_chapters(story_id)[0]
|
||||||
|
orchestrator.store.update_chapter(chapter["id"], included=0)
|
||||||
|
assert "library book 1" in update(story_id)
|
||||||
|
assert builds.call_count == 2
|
||||||
|
assert len(ingest_files()) == 4
|
||||||
|
assert update(story_id).startswith("No changes")
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_held_book_that_did_not_change_is_not_built_again(builds, monkeypatch):
|
||||||
|
monkeypatch.setattr(config, "INGEST_DIR", None)
|
||||||
|
site(monkeypatch, ONE)
|
||||||
|
job = Job(mode=DownloadMode.NEW, url=STORY_URL)
|
||||||
|
assert "Not sent. INGEST_DIR is not set." in orchestrator._run_site_job(job)
|
||||||
|
changed = story(job.story_id)["last_changed"]
|
||||||
|
orchestrator.store.update_story(job.story_id, last_changed="2020-01-01T00:00:00")
|
||||||
|
assert "Not sent. INGEST_DIR is not set." in update(job.story_id)
|
||||||
|
assert changed and story(job.story_id)["last_changed"] == "2020-01-01T00:00:00"
|
||||||
|
assert builds.call_count == 1
|
||||||
|
# The book changes. Bindery builds it, also while it cannot send it.
|
||||||
|
site(monkeypatch, [*ONE, (2, "Two", "b")])
|
||||||
|
update(job.story_id)
|
||||||
|
assert builds.call_count == 2
|
||||||
|
assert story(job.story_id)["last_changed"] != "2020-01-01T00:00:00"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_released_book_goes_out_without_another_build(builds, jobs):
|
||||||
|
story_id = add_story(ingested_at="2026-01-01T00:00:00", calibre_id=1, calibre_uuid="uuid-1")
|
||||||
|
orchestrator.save_settings(pause_uploads=True)
|
||||||
|
orchestrator._run_rebuild(Job(mode=DownloadMode.REBUILD, story_id=story_id))
|
||||||
|
assert builds.call_count == 1 and ingest_files() == []
|
||||||
|
orchestrator.save_settings(pause_uploads=False)
|
||||||
|
assert "library book 1" in orchestrator._run_rebuild(jobs.get_next())
|
||||||
|
assert builds.call_count == 1
|
||||||
|
assert len(ingest_files()) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_lost_file_is_built_again_only_to_send_it(builds):
|
||||||
|
story_id = add_story(ingested_at="2026-01-01T00:00:00", calibre_id=1, calibre_uuid="uuid-1")
|
||||||
|
orchestrator.save_settings(pause_uploads=True)
|
||||||
|
job = Job(mode=DownloadMode.REBUILD, story_id=story_id)
|
||||||
|
orchestrator._run_rebuild(job)
|
||||||
|
for path in config.TMP_DIR.iterdir():
|
||||||
|
path.unlink()
|
||||||
|
assert "pause" in orchestrator._run_rebuild(job)
|
||||||
|
assert builds.call_count == 1
|
||||||
|
orchestrator.store.set_setting(None, "pause_uploads", None)
|
||||||
|
assert "library book 1" in orchestrator._run_rebuild(job)
|
||||||
|
assert builds.call_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_rebuild_without_a_change_does_nothing_unless_the_user_asks_for_it(builds, client, jobs):
|
||||||
|
story_id = add_story(ingested_at="2026-01-01T00:00:00", calibre_id=1, calibre_uuid="uuid-1")
|
||||||
|
job = Job(mode=DownloadMode.REBUILD, story_id=story_id)
|
||||||
|
assert "library book 1" in orchestrator._run_rebuild(job)
|
||||||
|
orchestrator.store.update_story(story_id, last_changed=None, message="before")
|
||||||
|
assert orchestrator._run_rebuild(job) == "No changes"
|
||||||
|
assert (story(story_id)["last_changed"], story(story_id)["message"]) == (None, "before")
|
||||||
|
assert builds.call_count == 1
|
||||||
|
# The Rebuild button of the story page.
|
||||||
|
queued = client.post(f"/api/stories/{story_id}/download", json={"mode": "rebuild"}, headers=H).json
|
||||||
|
assert queued["options"] == {"force": True}
|
||||||
|
assert "library book 1" in orchestrator._run_rebuild(jobs.get_next())
|
||||||
|
assert builds.call_count == 2
|
||||||
|
assert len(ingest_files()) == 4
|
||||||
|
assert story(story_id)["last_changed"]
|
||||||
|
# Other modes take no such option.
|
||||||
|
queued = client.post(f"/api/stories/{story_id}/download", json={"mode": "new"}, headers=H).json
|
||||||
|
assert queued["options"] == {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_changed_link_gets_the_book_without_a_build(builds, fake):
|
||||||
|
story_id = add_story(ingested_at="2026-01-01T00:00:00", calibre_id=1, calibre_uuid="uuid-1")
|
||||||
|
job = Job(mode=DownloadMode.REBUILD, story_id=story_id)
|
||||||
|
orchestrator._run_rebuild(job)
|
||||||
|
fake.books.append(dataclasses.replace(fake.books[0], id=2, uuid="uuid-2", title="Other"))
|
||||||
|
orchestrator.link_story(story_id, "uuid-2")
|
||||||
|
assert "library book 2" in orchestrator._run_rebuild(job)
|
||||||
|
assert builds.call_count == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_failed_delivery_is_tried_again_without_a_build(builds, fake, monkeypatch):
|
||||||
|
story_id = linked_story(ONE)
|
||||||
|
site(monkeypatch, ONE)
|
||||||
|
send = orchestrator.IngestFolder.send_update
|
||||||
|
monkeypatch.setattr(orchestrator.IngestFolder, "send_update", Mock(side_effect=OSError("disk full")))
|
||||||
|
with pytest.raises(OSError, match="disk full"):
|
||||||
|
update(story_id)
|
||||||
|
monkeypatch.setattr(orchestrator.IngestFolder, "send_update", send)
|
||||||
|
assert "library book 1" in update(story_id)
|
||||||
|
assert builds.call_count == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_new_build_removes_the_older_file(builds, monkeypatch):
|
||||||
|
story_id = linked_story(ONE)
|
||||||
|
site(monkeypatch, ONE)
|
||||||
|
update(story_id)
|
||||||
|
(first,) = config.TMP_DIR.iterdir()
|
||||||
|
site(monkeypatch, [*ONE, (2, "Two", "b")])
|
||||||
|
update(story_id)
|
||||||
|
(second,) = config.TMP_DIR.iterdir()
|
||||||
|
assert first != second and second.name.startswith(f"story-{story_id}-")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("changes", "held", "built", "sent"),
|
||||||
|
[
|
||||||
|
({"calibre_id": 1, "calibre_uuid": "uuid-1"}, False, True, True),
|
||||||
|
({"ingested_at": "2026-01-01"}, False, True, True),
|
||||||
|
({"ingested_at": "2026-01-01"}, True, True, False),
|
||||||
|
({}, False, True, False),
|
||||||
|
({"ingested_at": "2026-01-01", "status": "error"}, False, False, False),
|
||||||
|
({"ingested_at": "2026-01-01", "no_metadata": True}, False, False, False),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_stories_of_an_older_bindery_get_their_keys(builds, changes, held, built, sent):
|
||||||
|
no_metadata = changes.pop("no_metadata", False)
|
||||||
|
story_id = add_story(**changes)
|
||||||
|
if no_metadata:
|
||||||
|
orchestrator.store.update_story(story_id, metahtml=None)
|
||||||
|
orchestrator.store.set_setting(story_id, "upload_held", held or None)
|
||||||
|
orchestrator.adopt_build_keys()
|
||||||
|
settings = orchestrator.store
|
||||||
|
assert bool(settings.get_setting(story_id, "built_key")) is built
|
||||||
|
assert bool(settings.get_setting(story_id, "sent_key")) is sent
|
||||||
|
if sent:
|
||||||
|
assert orchestrator._run_rebuild(Job(mode=DownloadMode.REBUILD, story_id=story_id)) == "No changes"
|
||||||
|
assert builds.call_count == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_story_with_a_waiting_job_gets_no_key(builds, jobs):
|
||||||
|
story_id = add_story(ingested_at="2026-01-01")
|
||||||
|
jobs.add(Job(mode=DownloadMode.NEW, story_id=story_id))
|
||||||
|
orchestrator.adopt_build_keys()
|
||||||
|
assert orchestrator.store.get_setting(story_id, "built_key") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_first_check_after_an_upgrade_sends_nothing(builds, monkeypatch):
|
||||||
|
story_id = linked_story(ONE)
|
||||||
|
orchestrator.store.update_story(story_id, pinned={"title": "My Title"}, site_values={})
|
||||||
|
orchestrator.adopt_build_keys()
|
||||||
|
site(monkeypatch, ONE)
|
||||||
|
assert update(story_id).startswith("No changes")
|
||||||
|
assert builds.call_count == 0 and ingest_files() == []
|
||||||
@@ -60,7 +60,8 @@ def opf(name):
|
|||||||
|
|
||||||
|
|
||||||
def rebuild(story_id):
|
def rebuild(story_id):
|
||||||
return orchestrator._run_rebuild(Job(mode=DownloadMode.REBUILD, story_id=story_id))
|
"""A rebuild that the user asks for. It builds and sends the book also when the book did not change."""
|
||||||
|
return orchestrator._run_rebuild(Job(mode=DownloadMode.REBUILD, story_id=story_id, options={"force": True}))
|
||||||
|
|
||||||
|
|
||||||
def test_first_version_goes_in_as_a_new_book(fake):
|
def test_first_version_goes_in_as_a_new_book(fake):
|
||||||
|
|||||||
@@ -150,7 +150,7 @@ def test_build_without_ingest_succeeds_and_keeps_the_book_local(monkeypatch, tmp
|
|||||||
story = orchestrator.store.get_story(story_id)
|
story = orchestrator.store.get_story(story_id)
|
||||||
assert story["status"] == "ok"
|
assert story["status"] == "ok"
|
||||||
assert story["ingested_at"] == ("2026-01-01" if existing else None)
|
assert story["ingested_at"] == ("2026-01-01" if existing else None)
|
||||||
assert (config.TMP_DIR / f"story-{story_id}.epub").exists()
|
assert list(config.TMP_DIR.glob(f"story-{story_id}-*.epub"))
|
||||||
assert not (tmp_path / "ingest").exists()
|
assert not (tmp_path / "ingest").exists()
|
||||||
details = orchestrator.story_details(story_id)
|
details = orchestrator.story_details(story_id)
|
||||||
assert details["uploads_paused"] is True
|
assert details["uploads_paused"] is True
|
||||||
@@ -260,7 +260,7 @@ def test_new_story_download_skips_library_matching(monkeypatch, standalone):
|
|||||||
assert ("Not sent." if standalone else "Sent to ingest folder") in message
|
assert ("Not sent." if standalone else "Sent to ingest folder") in message
|
||||||
assert len(orchestrator.store.list_chapters(job.story_id)) == 1
|
assert len(orchestrator.store.list_chapters(job.story_id)) == 1
|
||||||
assert orchestrator.store.get_story(job.story_id)["status"] == "ok"
|
assert orchestrator.store.get_story(job.story_id)["status"] == "ok"
|
||||||
assert (config.TMP_DIR / f"story-{job.story_id}.epub").exists()
|
assert list(config.TMP_DIR.glob(f"story-{job.story_id}-*.epub"))
|
||||||
orchestrator.cwa.assert_not_called()
|
orchestrator.cwa.assert_not_called()
|
||||||
orchestrator._find_book.assert_not_called()
|
orchestrator._find_book.assert_not_called()
|
||||||
|
|
||||||
|
|||||||
@@ -25,6 +25,8 @@ def state(monkeypatch, tmp_path):
|
|||||||
monkeypatch.setattr(o, "job_queue", queue)
|
monkeypatch.setattr(o, "job_queue", queue)
|
||||||
monkeypatch.setattr(config, "CWA_URL", "")
|
monkeypatch.setattr(config, "CWA_URL", "")
|
||||||
monkeypatch.setattr(config, "SITE_DELAY", 7)
|
monkeypatch.setattr(config, "SITE_DELAY", 7)
|
||||||
|
monkeypatch.setattr(config, "TMP_DIR", tmp_path / "tmp")
|
||||||
|
config.TMP_DIR.mkdir()
|
||||||
return store, queue
|
return store, queue
|
||||||
|
|
||||||
|
|
||||||
@@ -151,7 +153,8 @@ def site(state, monkeypatch):
|
|||||||
calibre={"title": "Title"},
|
calibre={"title": "Title"},
|
||||||
)
|
)
|
||||||
fetch = Mock(side_effect=lambda a, url, i: f"<p>fresh {url}</p>")
|
fetch = Mock(side_effect=lambda a, url, i: f"<p>fresh {url}</p>")
|
||||||
build, deliver = Mock(), Mock(return_value="Sent")
|
build = Mock(side_effect=lambda story, overrides, path: path.write_bytes(b"epub"))
|
||||||
|
deliver = Mock(return_value="Sent")
|
||||||
monkeypatch.setattr(fff, "get_adapter", lambda url: adapter)
|
monkeypatch.setattr(fff, "get_adapter", lambda url: adapter)
|
||||||
monkeypatch.setattr(fff, "fetch_story", lambda a: snap)
|
monkeypatch.setattr(fff, "fetch_story", lambda a: snap)
|
||||||
monkeypatch.setattr(fff, "fetch_chapter", fetch)
|
monkeypatch.setattr(fff, "fetch_chapter", fetch)
|
||||||
|
|||||||
Reference in New Issue
Block a user