Podcast voices: natural provider TTS, and system-voice hygiene

Two reports, one root: the platform's speech engine.

"Loud cracking at the end of every sentence" is the browser/OS TTS popping
between its internal sentence chunks (an idle cancel makes it worse). Hygiene
for that path: never cancel when nothing is pending, normalise the spoken text
(control characters, ellipses, dashes), and a 140ms beat between sections so
speech never runs into the engine's tail.

The real fix is a second voice source that does not use the platform engine:
- lib/providers carries TTS profiles (OpenAI gpt-4o-mini-tts, Groq
  playai-tts, any OpenAI-compatible /audio/speech behind the custom provider),
  male voices first.
- /api/tts proxies them with the same guards and the key still in headers.
- the picker gains System / Natural, with the provider's voice list.
- lib/podcast-audio plays one clip per section through a hidden <audio>
  element, measures the clip's real duration and gives each word a slice of it
  proportional to its spoken length - the highlight follows the audio exactly,
  and pause/seek/chapters become real audio operations. Clips cache per
  session, the next section generates while the current plays, and a voice
  change disposes the element and its blobs. Speed applies live (setRate)
  without dropping the cache.

podcast-test stubs /api/tts with a real silent WAV and pins the path: the
voice list, the request body, exactly one playing element, and the highlight
riding currentTime. 43 checks. tests/lib.mjs launches Chromium with the
autoplay policy relaxed (a real device has the listener's gesture).
Version 0.7.6, SW v98.
This commit is contained in:
Martin Dvořák 2026-10-16 16:34:46 +00:00
commit 8e1cd0855d
13 changed files with 997 additions and 63 deletions

View file

@ -1492,6 +1492,26 @@ Things I chose while building, with the reasoning:
into view as you type. No strings, so no i18n; the additive change sits on top of the merged into view as you type. No strings, so no i18n; the additive change sits on top of the merged
scene-heat rework unchanged. scene-heat rework unchanged.
104. **Natural voices — and the crack at the end of every sentence.** Two reports, one root:
the platform's speech engine. "There's a loud cracking sound at the end of every sentence" is
the browser/OS TTS (iOS and several Linux engines pop between their internal sentence chunks,
and an unnecessary `cancel()` makes it worse) — nothing in our code touches audio there. Both
halves are answered: the system path got hygiene (never cancel while idle, strip control
characters and normalise ellipses/dashes before speaking, and a 140 ms beat between sections
so speech never runs into the engine's tail), and the real fix is a second voice source that
does not use the platform engine at all. `lib/providers` now carries TTS profiles (OpenAI
`gpt-4o-mini-tts`, Groq `playai-tts`, and any OpenAI-compatible `/audio/speech` the custom
provider points at; male voices first), `/api/tts` proxies them so the key still never leaves
the browser, and the player's picker gains a **System / Natural** switch with the provider's
voice list. The natural engine (`lib/podcast-audio`) plays one clip per section through an
`<audio>` element: the clip's real duration is measured and each word gets a slice of it
proportional to its spoken length, so the highlight follows the audio exactly instead of
estimating — and pause, seek and chapter jumps become real audio operations. Clips are cached
per session, the next section is generated while the current one plays, and switching voice
disposes the old element and its blobs. The suite stubs the TTS endpoint with a real silent
WAV and pins the whole path (voice list, request body, one playing element, the highlight
riding `currentTime`) — 43 checks.
103. **Podcast mode: your notes, read aloud, highlighted word by word.** The third segment of 103. **Podcast mode: your notes, read aloud, highlighted word by word.** The third segment of
the note-powered wizards (Study deck ⇄ Homework ⇄ Podcast) takes the same intake — photos, the note-powered wizards (Study deck ⇄ Homework ⇄ Podcast) takes the same intake — photos,
documents, pasted text, the share target — and turns it into a spoken episode instead of documents, pasted text, the share target — and turns it into a spoken episode instead of

129
app/api/tts/route.ts Normal file
View file

@ -0,0 +1,129 @@
import { getServerConfig } from "@/lib/config";
import { ttsProfile } from "@/lib/providers";
import { resolveModelTarget } from "@/lib/providers-server";
import { rejectCrossOrigin, rejectOversizedBody, requestLocale } from "@/lib/http-guard";
import { clientKey, rateLimit, rateLimitResponse } from "@/lib/rate-limit";
import { acquireModelGate } from "@/lib/model-guard";
import { translate } from "@/lib/i18n-dict";
import { z } from "zod";
export const runtime = "nodejs";
export const dynamic = "force-dynamic";
export const maxDuration = 120;
const requestSchema = z.object({
/** One section of the script — a couple of hundred words at most. */
text: z.string().min(1).max(4000),
voice: z.string().min(1).max(60).optional(),
model: z.string().min(1).max(120).optional(),
});
/** About 1 MB of text; audio comes back in the response, not the request. */
const MAX_BODY_BYTES = 1_048_576;
/**
* Natural podcast voices: the provider's own text-to-speech, proxied so the
* key stays in the browser (it travels in the header, never lands on disk).
* OpenAI-compatible `/audio/speech` only — OpenAI, Groq, and whatever local
* server the custom provider points at.
*/
export async function POST(request: Request) {
const locale = requestLocale(request);
const crossOrigin = rejectCrossOrigin(request);
if (crossOrigin) return crossOrigin;
const oversized = rejectOversizedBody(request, MAX_BODY_BYTES);
if (oversized) return oversized;
const config = getServerConfig();
const limit = rateLimit(`tts:${clientKey(request)}`, config.rateLimits.generate, config.rateLimits.windowMs);
if (!limit.allowed) return rateLimitResponse(limit, locale);
/* TTS is a provider feature: no provider key, no natural voice. */
const resolved = resolveModelTarget(config, request, locale);
if ("error" in resolved) {
return Response.json({ error: resolved.error.message, kind: resolved.error.kind }, { status: resolved.error.status });
}
if (resolved.target.kind !== "provider") {
return Response.json(
{ error: translate(locale, "server.ttsNeedsProvider"), kind: "needs_key" },
{ status: 400 },
);
}
const provider = resolved.target.target;
const providerId = (request.headers.get("x-freezmo-provider") ?? "").trim().toLowerCase();
const profile = ttsProfile(providerId);
if (!profile) {
return Response.json(
{ error: translate(locale, "server.ttsUnsupported", { provider: provider.label }), kind: "bad_provider" },
{ status: 400 },
);
}
let payload: unknown;
try {
payload = await request.json();
} catch {
return Response.json({ error: translate(locale, "server.bodyNotJson") }, { status: 400 });
}
const parsed = requestSchema.safeParse(payload);
if (!parsed.success) {
return Response.json({ error: translate(locale, "server.invalidRequest", { detail: "text" }) }, { status: 400 });
}
const gate = acquireModelGate(request, config, locale);
if ("response" in gate) return gate.response;
const model = parsed.data.model ?? profile.model;
const voice = parsed.data.voice ?? profile.voices.find((entry) => entry.male)?.id ?? profile.voices[0].id;
const abort = new AbortController();
const timer = setTimeout(() => abort.abort(), config.timeoutMs);
const onRequestAbort = () => abort.abort();
if (request.signal.aborted) abort.abort();
else request.signal.addEventListener("abort", onRequestAbort, { once: true });
try {
const upstream = await fetch(`${provider.baseUrl.replace(/\/$/, "")}/audio/speech`, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${provider.apiKey}`,
},
body: JSON.stringify({
model,
voice,
input: parsed.data.text,
response_format: profile.format,
}),
signal: abort.signal,
});
if (!upstream.ok || !upstream.body) {
const detail = await upstream.text().catch(() => "");
/* Never log the text that was spoken; only the status. */
console.error("[api/tts]", upstream.status, detail.slice(0, 200));
const kind = upstream.status === 401 || upstream.status === 403 ? "auth" : upstream.status === 429 ? "usage_limit" : "unknown";
return Response.json(
{ error: translate(locale, "server.ttsFailed", { provider: provider.label }), kind },
{ status: upstream.status === 429 ? 429 : 502 },
);
}
return new Response(upstream.body, {
headers: {
"Content-Type": profile.format === "wav" ? "audio/wav" : "audio/mpeg",
"Cache-Control": "no-store, no-transform",
},
});
} catch (error) {
const timedOut = error instanceof Error && error.name === "AbortError" && !request.signal.aborted;
console.error("[api/tts]", (error as Error)?.name ?? error);
return Response.json(
{ error: translate(locale, timedOut ? "server.ttsTimeout" : "server.ttsFailed", { provider: provider.label }), kind: "network" },
{ status: 502 },
);
} finally {
clearTimeout(timer);
request.signal.removeEventListener("abort", onRequestAbort);
gate.gate.release();
}
}

View file

@ -4,6 +4,16 @@ import Link from "next/link";
import { useCallback, useEffect, useMemo, useRef, useState } from "react"; import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { useI18n } from "@/lib/i18n"; import { useI18n } from "@/lib/i18n";
import { createPodcastAudio, type PodcastAudio } from "@/lib/podcast-audio";
import {
defaultTtsProvider,
readNaturalPref,
ttsProviderLabel,
voiceForProvider,
writeNaturalPref,
} from "@/lib/podcast-natural";
import { TTS_PROFILES } from "@/lib/providers";
import { useProviders } from "@/lib/providers-client";
import { import {
createPodcastSpeech, createPodcastSpeech,
tokenizeSections, tokenizeSections,
@ -54,6 +64,79 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
const [position, setPosition] = useState<PodcastVoice | null>(null); const [position, setPosition] = useState<PodcastVoice | null>(null);
const [unsupported, setUnsupported] = useState(false); const [unsupported, setUnsupported] = useState(false);
const [voiceOpen, setVoiceOpen] = useState(false); const [voiceOpen, setVoiceOpen] = useState(false);
/** "system" = the platform's voices; "natural" = the provider's TTS. */
const [voiceSource, setVoiceSource] = useState<"system" | "natural">(() => readNaturalPref().source);
const [naturalProvider, setNaturalProvider] = useState<string>(() => readNaturalPref().provider);
const [naturalVoiceId, setNaturalVoiceId] = useState<string | null>(() => readNaturalPref().voice);
const [naturalError, setNaturalError] = useState<string | null>(null);
const [naturalBusy, setNaturalBusy] = useState(false);
const providers = useProviders();
/* Either engine drives the same onWord/onEnd events; the transport below
routes to whichever one the listeners picked. */
const natural = useMemo<PodcastAudio | null>(() => {
if (voiceSource !== "natural" || !naturalProvider) return null;
return createPodcastAudio({
sections: episode.sections,
providerId: naturalProvider,
voice: voiceForProvider(naturalProvider, naturalVoiceId),
model: readNaturalPref().model ?? undefined,
rate,
onWord: (next) => {
latest.current = next;
setPosition(next);
},
onPlay: () => {
setPlaying(true);
setNaturalError(null);
window.dispatchEvent(new CustomEvent("freezmo:foreground-audio", { detail: "podcast" }));
},
onPause: () => setPlaying(false),
onEnd: () => {
setPlaying(false);
setPosition(null);
latest.current = null;
savePodcastProgress(episode.id, { section: 0, word: 0 });
},
onError: (message) => {
setPlaying(false);
setNaturalError(message);
},
onBusy: setNaturalBusy,
});
// eslint-disable-next-line react-hooks/exhaustive-deps -- speed is applied live (setRate below), so it must not rebuild the engine
}, [voiceSource, naturalProvider, naturalVoiceId, episode.id, episode.sections]);
/* Speed changes apply to the element, not to the engine's cache. */
useEffect(() => {
natural?.setRate(rate);
}, [natural, rate]);
/* The engine owns an <audio> element: release it when the engine is
replaced (a different voice) or the player closes. */
useEffect(() => {
if (!natural) return;
return () => {
try {
natural.dispose();
} catch {
/* already gone */
}
};
}, [natural]);
useEffect(() => {
return () => naturalRef.current?.dispose();
}, []);
/* The default pick the first time someone opens the natural panel. */
useEffect(() => {
if (naturalProvider) return;
const keys = providers.keys;
const fallback = defaultTtsProvider(keys, providers.generation?.provider);
if (fallback) setNaturalProvider(fallback);
// eslint-disable-next-line react-hooks/exhaustive-deps -- once per mount
}, [providers.keys]);
const transcript = useRef<HTMLDivElement>(null); const transcript = useRef<HTMLDivElement>(null);
const activeWord = useRef<HTMLElement | null>(null); const activeWord = useRef<HTMLElement | null>(null);
/** Line-following state: the last followed line and a manual-scroll pause. */ /** Line-following state: the last followed line and a manual-scroll pause. */
@ -100,6 +183,10 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
} }
return bestVoice(voiceLocale, voiceList); return bestVoice(voiceLocale, voiceList);
}, [voiceURI, availableVoices, voiceLocale, voiceList]); }, [voiceURI, availableVoices, voiceLocale, voiceList]);
const ttsProviderIds = useMemo(
() => Object.keys(TTS_PROFILES).filter((id) => Boolean(providers.keys[id])),
[providers.keys],
);
const roboticOnly = useMemo( const roboticOnly = useMemo(
() => onlyRoboticVoices(voiceLocale, voiceList), () => onlyRoboticVoices(voiceLocale, voiceList),
[voiceLocale, voiceList], [voiceLocale, voiceList],
@ -172,10 +259,14 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
/* The engine instance is rebuilt on a speed change; keep a ref so the /* The engine instance is rebuilt on a speed change; keep a ref so the
unmount cleanup always stops whichever one is current. */ unmount cleanup always stops whichever one is current. */
const speechRef = useRef(speech); const speechRef = useRef(speech);
const naturalRef = useRef<PodcastAudio | null>(natural);
const playingRef = useRef(false); const playingRef = useRef(false);
useEffect(() => { useEffect(() => {
speechRef.current = speech; speechRef.current = speech;
}, [speech]); }, [speech]);
useEffect(() => {
naturalRef.current = natural;
}, [natural]);
useEffect(() => { useEffect(() => {
playingRef.current = playing; playingRef.current = playing;
}, [playing]); }, [playing]);
@ -289,27 +380,33 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
); );
}, [position, tokens]); }, [position, tokens]);
const naturalActive = voiceSource === "natural" && Boolean(natural);
/** The engines share their shape; the transport routes by source. */
const canPlay = naturalActive ? true : supported;
const harness = () => (naturalActive ? natural : speech);
const toggle = () => { const toggle = () => {
if (!speech) return; if (!canPlay) return;
if (playing) { if (playing) {
speech.pause(); harness()?.pause();
persist(); persist();
return; return;
} }
/* Other readers (a card's read-aloud) yield to the episode. */ /* Other readers (a card's read-aloud) yield to the episode. */
stopSpeaking(); stopSpeaking();
speech.play(position ?? { section: 0, word: 0 }); harness()?.play(position ?? { section: 0, word: 0 });
}; };
const seek = (next: PodcastVoice) => { const seek = (next: PodcastVoice) => {
if (!speech) return; if (!canPlay) return;
latest.current = next; latest.current = next;
setPosition(next); setPosition(next);
/* A seek is an instruction to show that place: follow it at once. */ /* A seek is an instruction to show that place: follow it at once. */
lastLineTop.current = null; lastLineTop.current = null;
followPausedUntil.current = 0; followPausedUntil.current = 0;
stopSpeaking(); stopSpeaking();
speech.play(next); harness()?.play(next);
}; };
const jump = (section: number) => seek({ section, word: 0 }); const jump = (section: number) => seek({ section, word: 0 });
@ -375,7 +472,7 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
{sections[currentSection]?.heading ?? ""} {sections[currentSection]?.heading ?? ""}
</p> </p>
</div> </div>
{availableVoices.length > 0 ? ( {availableVoices.length > 0 || ttsProviderIds.length > 0 ? (
<button <button
type="button" type="button"
onClick={() => setVoiceOpen((open) => !open)} onClick={() => setVoiceOpen((open) => !open)}
@ -398,6 +495,14 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
) : null} ) : null}
</button> </button>
) : null} ) : null}
{naturalBusy ? (
<span
role="status"
className="shrink-0 rounded-full bg-[var(--surface-sunken)] px-2.5 py-2 text-[11px] font-semibold text-[var(--text-muted)]"
>
{t("podcast.voiceGenerating")}
</span>
) : null}
<button <button
type="button" type="button"
onClick={() => { onClick={() => {
@ -426,7 +531,7 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
className="no-scrollbar min-h-0 flex-1 overflow-y-auto overscroll-contain px-4 pb-40 pt-24" className="no-scrollbar min-h-0 flex-1 overflow-y-auto overscroll-contain px-4 pb-40 pt-24"
> >
<div className="mx-auto w-full max-w-3xl space-y-6"> <div className="mx-auto w-full max-w-3xl space-y-6">
{!supported || unsupported ? ( {(!supported || unsupported) && !naturalActive ? (
<p className="rounded-2xl border border-amber-300/50 bg-amber-50/70 px-3.5 py-3 text-[12.5px] leading-relaxed text-amber-800 dark:border-amber-400/25 dark:bg-amber-400/10 dark:text-amber-200"> <p className="rounded-2xl border border-amber-300/50 bg-amber-50/70 px-3.5 py-3 text-[12.5px] leading-relaxed text-amber-800 dark:border-amber-400/25 dark:bg-amber-400/10 dark:text-amber-200">
{t("podcast.noVoice")} {t("podcast.noVoice")}
{isAppleMobile() ? ` ${t("podcast.voiceIos")}` : ""} {isAppleMobile() ? ` ${t("podcast.voiceIos")}` : ""}
@ -497,62 +602,147 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
{t("common.close")} {t("common.close")}
</button> </button>
</div> </div>
{voiceURI ? ( {/* Source: the platform's voices, or the provider's TTS. */}
<button <div className="grid grid-cols-2 gap-1.5 px-1 pb-1">
type="button" {(["system", "natural"] as const).map((source) => (
onClick={() => {
setVoiceURI(null);
window.localStorage.removeItem(
"freezmo.podcast-voice.v1",
);
}}
className="nav-row touch-target w-full text-left text-[13px]"
>
{t("podcast.voiceAuto")}
</button>
) : null}
{availableVoices.map((entry) => {
const active = entry.voiceURI === voice?.voiceURI;
return (
<button <button
key={entry.voiceURI} key={source}
type="button" type="button"
aria-pressed={active} aria-pressed={voiceSource === source}
onClick={() => { onClick={() => {
setVoiceURI(entry.voiceURI); setVoiceSource(source);
setStoredVoice(voiceLocale, entry.voiceURI); writeNaturalPref({ source });
setVoiceOpen(false); setNaturalError(null);
/* Swapping engines mid-sentence: stop the other one. */
if (playing) {
harness()?.pause();
setPlaying(false);
}
}} }}
className={cn( className={cn(
"nav-row touch-target flex w-full items-center gap-2 text-left text-[13px]", "touch-target rounded-xl border px-3 py-2 text-[12.5px] font-semibold transition",
active && voiceSource === source
"border-[var(--nav-active-border)] bg-[var(--nav-active-bg)]", ? "border-[var(--nav-active-border)] bg-[var(--nav-active-bg)] text-[var(--text)]"
: "border-[var(--hairline)] bg-[var(--surface-sunken)] text-[var(--text-muted)] hover:bg-[var(--hover)]",
)} )}
> >
<span className="min-w-0 flex-1"> {source === "system" ? t("podcast.voiceSystem") : t("podcast.voiceNatural")}
<span className="block truncate font-medium text-[var(--text)]">
{entry.name}
</span>
<span className="block text-[11.5px] text-[var(--text-subtle)]">
{entry.lang}
{entry.localService
? ` · ${t("podcast.voiceLocal")}`
: ""}
</span>
</span>
{active ? (
<CheckCircleIcon className="size-4 shrink-0 text-accent-600 dark:text-accent-200" />
) : null}
</button> </button>
); ))}
})} </div>
<p className="px-1 pt-1 text-[11.5px] leading-relaxed text-[var(--text-subtle)]">
{iosLimited {voiceSource === "system" ? (
? t("podcast.voiceIos") <>
: roboticOnly {availableVoices.length === 0 ? (
? t("podcast.voiceRobotic") <p className="px-1 pt-1 text-[11.5px] leading-relaxed text-[var(--text-subtle)]">
: t("podcast.voiceNote")} {t("podcast.voiceNoSystem")}
</p> </p>
) : null}
{voiceURI ? (
<button
type="button"
onClick={() => {
setVoiceURI(null);
window.localStorage.removeItem("freezmo.podcast-voice.v1");
}}
className="nav-row touch-target w-full text-left text-[13px]"
>
{t("podcast.voiceAuto")}
</button>
) : null}
{availableVoices.map((entry) => {
const active = entry.voiceURI === voice?.voiceURI;
return (
<button
key={entry.voiceURI}
type="button"
aria-pressed={active}
onClick={() => {
setVoiceURI(entry.voiceURI);
setStoredVoice(voiceLocale, entry.voiceURI);
setVoiceOpen(false);
}}
className={cn(
"nav-row touch-target flex w-full items-center gap-2 text-left text-[13px]",
active && "border-[var(--nav-active-border)] bg-[var(--nav-active-bg)]",
)}
>
<span className="min-w-0 flex-1">
<span className="block truncate font-medium text-[var(--text)]">{entry.name}</span>
<span className="block text-[11.5px] text-[var(--text-subtle)]">
{entry.lang}
{entry.localService ? ` · ${t("podcast.voiceLocal")}` : ""}
</span>
</span>
{active ? <CheckCircleIcon className="size-4 shrink-0 text-accent-600 dark:text-accent-200" /> : null}
</button>
);
})}
<p className="px-1 pt-1 text-[11.5px] leading-relaxed text-[var(--text-subtle)]">
{iosLimited ? t("podcast.voiceIos") : roboticOnly ? t("podcast.voiceRobotic") : t("podcast.voiceNote")}
</p>
</>
) : ttsProviderIds.length === 0 ? (
<p className="px-1 pt-1 text-[11.5px] leading-relaxed text-[var(--text-subtle)]">
{t("podcast.voiceNeedsKey")}
</p>
) : (
<>
{ttsProviderIds.length > 1 ? (
<div className="flex flex-wrap gap-1.5 px-1 pb-1 pt-1">
{ttsProviderIds.map((id) => (
<button
key={id}
type="button"
aria-pressed={naturalProvider === id}
onClick={() => {
setNaturalProvider(id);
setNaturalVoiceId(null);
writeNaturalPref({ provider: id, voice: null });
}}
className={cn(
"rounded-full px-2.5 py-1.5 text-[11.5px] font-semibold transition",
naturalProvider === id
? "bg-[var(--nav-active-bg)] text-[var(--text)] ring-1 ring-[var(--nav-active-border)]"
: "bg-[var(--surface-sunken)] text-[var(--text-muted)] hover:bg-[var(--hover)]",
)}
>
{ttsProviderLabel(id)}
</button>
))}
</div>
) : null}
{(TTS_PROFILES[naturalProvider]?.voices ?? []).map((entry) => {
const active = entry.id === voiceForProvider(naturalProvider, naturalVoiceId);
return (
<button
key={entry.id}
type="button"
aria-pressed={active}
onClick={() => {
setNaturalVoiceId(entry.id);
writeNaturalPref({ voice: entry.id });
}}
className={cn(
"nav-row touch-target flex w-full items-center gap-2 text-left text-[13px]",
active && "border-[var(--nav-active-border)] bg-[var(--nav-active-bg)]",
)}
>
<span className="min-w-0 flex-1">
<span className="block truncate font-medium text-[var(--text)]">{entry.label}</span>
<span className="block text-[11.5px] text-[var(--text-subtle)]">
{entry.male ? t("podcast.voiceMale") : t("podcast.voiceFemale")}
</span>
</span>
{active ? <CheckCircleIcon className="size-4 shrink-0 text-accent-600 dark:text-accent-200" /> : null}
</button>
);
})}
<p className="px-1 pt-1 text-[11.5px] leading-relaxed text-[var(--text-subtle)]">
{naturalError ?? t("podcast.voiceNaturalNote")}
</p>
</>
)}
</div> </div>
</div> </div>
) : null} ) : null}
@ -573,7 +763,7 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
<button <button
type="button" type="button"
onClick={toggle} onClick={toggle}
disabled={!supported} disabled={!canPlay}
aria-label={playing ? t("podcast.pause") : t("podcast.play")} aria-label={playing ? t("podcast.pause") : t("podcast.play")}
title={playing ? t("podcast.pause") : t("podcast.play")} title={playing ? t("podcast.pause") : t("podcast.play")}
className="grid size-12 shrink-0 place-items-center rounded-full grad-ice text-[#06111f] transition disabled:opacity-45" className="grid size-12 shrink-0 place-items-center rounded-full grad-ice text-[#06111f] transition disabled:opacity-45"
@ -605,7 +795,7 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
<button <button
type="button" type="button"
onClick={() => { onClick={() => {
speech?.stop(); harness()?.stop();
setPosition({ section: 0, word: 0 }); setPosition({ section: 0, word: 0 });
latest.current = { section: 0, word: 0 }; latest.current = { section: 0, word: 0 };
savePodcastProgress(episode.id, { section: 0, word: 0 }); savePodcastProgress(episode.id, { section: 0, word: 0 });

View file

@ -291,6 +291,43 @@ export async function generatePodcast(
return { podcast: ready, meta: {} }; return { podcast: ready, meta: {} };
} }
/* ------------------------------------------------------------------ */
/* Natural podcast voices (provider text-to-speech) */
/* ------------------------------------------------------------------ */
/**
* One section of the script, spoken by the picked provider. The key rides in
* the headers exactly like every other model call; the audio comes back as a
* blob URL the player hands to an <audio> element.
*/
export async function synthesizeSpeech(
input: { text: string; voice?: string; model?: string; providerId: string },
options: { signal?: AbortSignal } = {},
): Promise<{ url: string; type: string }> {
const response = await fetch("/api/tts", {
method: "POST",
headers: {
"Content-Type": "application/json",
...authHeaders(),
...providerHeaders(input.providerId),
...localeHeader(),
},
body: JSON.stringify({ text: input.text, voice: input.voice, model: input.model }),
signal: options.signal,
}).catch(() => {
throw new ApiError(t("api.unreachable"), "network");
});
if (!response.ok) {
const body = (await response.json().catch(() => null)) as { error?: string; kind?: string } | null;
throw new ApiError(body?.error ?? t("api.httpError", { status: response.status }), body?.kind);
}
const blob = await response.blob();
if (blob.size === 0) throw new ApiError(t("api.noAudio"), "bad_response");
return { url: URL.createObjectURL(blob), type: blob.type };
}
/* ------------------------------------------------------------------ */ /* ------------------------------------------------------------------ */
/* Make my homework */ /* Make my homework */
/* ------------------------------------------------------------------ */ /* ------------------------------------------------------------------ */

View file

@ -307,6 +307,14 @@ const en: Record<string, string> = {
"podcast.voiceLocal": "installed", "podcast.voiceLocal": "installed",
"podcast.voiceNote": "Voices come from your browser and system — pick the one that sounds best to you.", "podcast.voiceNote": "Voices come from your browser and system — pick the one that sounds best to you.",
"podcast.voiceIos": "On iPhone, Safari offers web pages a single voice per language — the system's preferred one. For more (and better) voices: Settings → Accessibility → Spoken Content → Voices → your language, download a voice in Enhanced or Premium quality and make it the preferred one. Freezmo then reads with it.", "podcast.voiceIos": "On iPhone, Safari offers web pages a single voice per language — the system's preferred one. For more (and better) voices: Settings → Accessibility → Spoken Content → Voices → your language, download a voice in Enhanced or Premium quality and make it the preferred one. Freezmo then reads with it.",
"podcast.voiceSystem": "System voices",
"podcast.voiceNatural": "Natural voice",
"podcast.voiceMale": "male",
"podcast.voiceFemale": "female",
"podcast.voiceNaturalNote": "Generated by your own provider key — the words are timed to the audio, so the highlight still follows the voice.",
"podcast.voiceNeedsKey": "Add an OpenAI or Groq key in Settings → Providers to use natural voices.",
"podcast.voiceNoSystem": "This device reports no system voices. The natural voice works anywhere — or add a voice in your system settings.",
"podcast.voiceGenerating": "Generating the voice…",
"podcast.voiceRobotic": "Only the basic system voices are installed. For a natural host voice, add one in your system settings (Windows: Speech; macOS: Spoken Content; Chrome ships the Google voices).", "podcast.voiceRobotic": "Only the basic system voices are installed. For a natural host voice, add one in your system settings (Windows: Speech; macOS: Spoken Content; Chrome ships the Google voices).",
"podcast.tapToJump": "Tap any word to jump there.", "podcast.tapToJump": "Tap any word to jump there.",
"podcast.noVoice": "This browser has no speech voice installed, so the transcript cannot be read aloud here. You can still read along.", "podcast.noVoice": "This browser has no speech voice installed, so the transcript cannot be read aloud here. You can still read along.",
@ -808,6 +816,8 @@ const en: Record<string, string> = {
"api.streamLost": "The connection dropped mid-run. Check your connection and try again.", "api.streamLost": "The connection dropped mid-run. Check your connection and try again.",
"api.almost": "Almost there…", "api.almost": "Almost there…",
"api.noDeck": "No deck came back from the server. Check your Ollama settings and try again.", "api.noDeck": "No deck came back from the server. Check your Ollama settings and try again.",
"api.noPodcast": "No script came back from the server. Check the model setup and try again.",
"api.noAudio": "The voice came back empty. Try another voice or provider.",
"api.incomplete": "The server finished without sending a deck.", "api.incomplete": "The server finished without sending a deck.",
"api.noActivities": "The generated deck had no usable activities. Try again.", "api.noActivities": "The generated deck had no usable activities. Try again.",
"api.gradingCancelled": "Grading was cancelled.", "api.gradingCancelled": "Grading was cancelled.",
@ -822,6 +832,10 @@ const en: Record<string, string> = {
"server.bodyNotJson": "Request body was not valid JSON.", "server.bodyNotJson": "Request body was not valid JSON.",
"server.invalidRequest": "Invalid request: {detail}", "server.invalidRequest": "Invalid request: {detail}",
"server.providerModelNeeded": "Pick the generation model in Settings → Providers.", "server.providerModelNeeded": "Pick the generation model in Settings → Providers.",
"server.ttsNeedsProvider": "Natural voices come from a provider — add a provider key in Settings → Providers first.",
"server.ttsUnsupported": "{provider} does not offer text-to-speech. OpenAI, Groq or a custom OpenAI-compatible server will.",
"server.ttsFailed": "{provider} could not generate the voice for this part.",
"server.ttsTimeout": "{provider} took too long to generate the voice.",
"server.needContent": "Add at least one photo of your notes or paste some text to build a deck.", "server.needContent": "Add at least one photo of your notes or paste some text to build a deck.",
"server.needAssignment": "server.needAssignment":
"Add the assignment first — paste its text, photograph it, or add a document.", "Add the assignment first — paste its text, photograph it, or add a document.",
@ -1380,6 +1394,14 @@ const cs: Record<string, string> = {
"podcast.voiceLocal": "nainstalovaný", "podcast.voiceLocal": "nainstalovaný",
"podcast.voiceNote": "Hlasy dodává prohlížeč a systém — vyber si ten, co zní nejlíp.", "podcast.voiceNote": "Hlasy dodává prohlížeč a systém — vyber si ten, co zní nejlíp.",
"podcast.voiceIos": "Na iPhonu nabízí Safari webovým stránkám jen jeden hlas na jazyk — ten preferovaný v systému. Pro další (a lepší) hlasy: Nastavení → Zpřístupnění → Mluvený obsah → Hlasy → tvůj jazyk, stáhni si hlas ve verzi Vylepšený nebo Prémiový a nastav ho jako preferovaný. Freezmo pak čte s ním.", "podcast.voiceIos": "Na iPhonu nabízí Safari webovým stránkám jen jeden hlas na jazyk — ten preferovaný v systému. Pro další (a lepší) hlasy: Nastavení → Zpřístupnění → Mluvený obsah → Hlasy → tvůj jazyk, stáhni si hlas ve verzi Vylepšený nebo Prémiový a nastav ho jako preferovaný. Freezmo pak čte s ním.",
"podcast.voiceSystem": "Systémové hlasy",
"podcast.voiceNatural": "Přirozený hlas",
"podcast.voiceMale": "mužský",
"podcast.voiceFemale": "ženský",
"podcast.voiceNaturalNote": "Generuje ho klíč tvého providera — slova jsou časovaná podle audia, takže zvýraznění drží s hlasem.",
"podcast.voiceNeedsKey": "Pro přirozené hlasy si v Nastavení → Provideři přidej klíč OpenAI nebo Groq.",
"podcast.voiceNoSystem": "Tohle zařízení nehlásí žádné systémové hlasy. Přirozený hlas funguje všude — nebo si hlas přidej v nastavení systému.",
"podcast.voiceGenerating": "Generuji hlas…",
"podcast.voiceRobotic": "Jsou nainstalované jen základní systémové hlasy. Pro přirozený hlas moderátora si nějaký přidej v nastavení systému (Windows: Řeč; macOS: Mluvený obsah; Chrome má hlasy Google).", "podcast.voiceRobotic": "Jsou nainstalované jen základní systémové hlasy. Pro přirozený hlas moderátora si nějaký přidej v nastavení systému (Windows: Řeč; macOS: Mluvený obsah; Chrome má hlasy Google).",
"podcast.tapToJump": "Klikni na kterékoli slovo a přeskočíš tam.", "podcast.tapToJump": "Klikni na kterékoli slovo a přeskočíš tam.",
"podcast.noVoice": "Tento prohlížeč nemá nainstalovaný hlas, takže se přepis nedá přečíst nahlas. Číst ho ale můžeš dál.", "podcast.noVoice": "Tento prohlížeč nemá nainstalovaný hlas, takže se přepis nedá přečíst nahlas. Číst ho ale můžeš dál.",
@ -1881,6 +1903,8 @@ const cs: Record<string, string> = {
"api.streamLost": "Připojení se během generování přerušilo. Zkontrolujte připojení a zkuste to znovu.", "api.streamLost": "Připojení se během generování přerušilo. Zkontrolujte připojení a zkuste to znovu.",
"api.almost": "Už to bude…", "api.almost": "Už to bude…",
"api.noDeck": "Ze serveru nepřišla žádná sada. Zkontroluj nastavení Ollamy a zkus to znovu.", "api.noDeck": "Ze serveru nepřišla žádná sada. Zkontroluj nastavení Ollamy a zkus to znovu.",
"api.noPodcast": "Ze serveru nepřišel žádný scénář. Zkontroluj nastavení modelu a zkus to znovu.",
"api.noAudio": "Hlas přišel prázdný. Zkus jiný hlas nebo providera.",
"api.incomplete": "Server skončil, aniž by poslal sadu.", "api.incomplete": "Server skončil, aniž by poslal sadu.",
"api.noActivities": "Vygenerovaná sada neměla žádné použitelné aktivity. Zkus to znovu.", "api.noActivities": "Vygenerovaná sada neměla žádné použitelné aktivity. Zkus to znovu.",
"api.gradingCancelled": "Známkování bylo zrušeno.", "api.gradingCancelled": "Známkování bylo zrušeno.",
@ -1895,6 +1919,10 @@ const cs: Record<string, string> = {
"server.bodyNotJson": "Tělo požadavku nebylo platné JSON.", "server.bodyNotJson": "Tělo požadavku nebylo platné JSON.",
"server.invalidRequest": "Neplatný požadavek: {detail}", "server.invalidRequest": "Neplatný požadavek: {detail}",
"server.providerModelNeeded": "Vyberte model pro generování v Nastavení → Poskytovatelé.", "server.providerModelNeeded": "Vyberte model pro generování v Nastavení → Poskytovatelé.",
"server.ttsNeedsProvider": "Přirozené hlasy dodává provider — nejdřív si v Nastavení → Provideři přidej klíč.",
"server.ttsUnsupported": "{provider} text-to-speech neumí. Umí ho OpenAI, Groq nebo vlastní OpenAI-compatible server.",
"server.ttsFailed": "{provider} nedokázal pro tuhle část vygenerovat hlas.",
"server.ttsTimeout": "{provider} generoval hlas příliš dlouho.",
"server.needContent": "Přidej aspoň jednu fotku poznámek nebo vlož text, ze kterého sada vznikne.", "server.needContent": "Přidej aspoň jednu fotku poznámek nebo vlož text, ze kterého sada vznikne.",
"server.needAssignment": "server.needAssignment":
"Nejdřív přidej zadání — vlož jeho text, vyfoť ho nebo přidej dokument.", "Nejdřív přidej zadání — vlož jeho text, vyfoť ho nebo přidej dokument.",

283
lib/podcast-audio.ts Normal file
View file

@ -0,0 +1,283 @@
"use client";
/**
* The provider-voice playback engine: one section at a time is turned into
* audio by the browser's provider key (/api/tts) and played through an
* <audio> element.
*
* Word highlighting has no boundary events here, but it does not need to
* guess: the clip's real duration is known, so each word gets a slice of it
* proportional to its spoken length. That keeps the highlight honest from the
* first word to the last, and it makes pause/seek real audio operations —
* `currentTime` is the clock.
*
* Sections are cached per session (and the next one is fetched while the
* current plays), so listening to an episode twice costs one generation per
* section, not one per replay.
*/
import { synthesizeSpeech } from "./client-api";
import { tokenizeSections, type PodcastVoice } from "./podcast-speech";
import type { PodcastSection } from "./schemas";
export type PodcastAudioEvents = {
sections: PodcastSection[];
providerId: string;
voice: string;
model?: string;
/** Initial playback speed (0.5–2); `setRate` changes it while playing. */
rate?: number;
onWord: (position: PodcastVoice) => void;
onPlay?: () => void;
onPause?: () => void;
onEnd: () => void;
onError: (message: string) => void;
/** True while a section's voice is being generated. */
onBusy?: (busy: boolean) => void;
};
type Clip = { url: string; duration: number; type: string };
export type PodcastAudio = {
play: (from?: PodcastVoice) => void;
pause: () => void;
stop: () => void;
playing: () => boolean;
position: () => PodcastVoice | null;
/** Warm the cache for a section without playing it. */
prefetch: (index: number) => void;
/** Change the playback speed without losing the generated clips. */
setRate: (rate: number) => void;
/** Drop the element and the cached clips (the player unmounted). */
dispose: () => void;
};
export function createPodcastAudio(events: PodcastAudioEvents): PodcastAudio {
const tokens = tokenizeSections(events.sections);
const clips = new Map<number, Clip>();
const loading = new Map<number, Promise<Clip>>();
const audio = new Audio();
audio.preload = "auto";
/* In the document (hidden): some engines only start audio for attached
elements, and it keeps the element inspectable for debugging. */
audio.setAttribute("data-podcast-audio", "1");
audio.style.display = "none";
if (typeof document !== "undefined") document.body.appendChild(audio);
let playing = false;
let position: PodcastVoice | null = null;
let generation = 0;
let frame = 0;
let currentSection = 0;
let rate = Math.min(2, Math.max(0.5, events.rate ?? 1));
audio.playbackRate = rate;
const wordsOf = (section: number) => tokens.filter((token) => token.section === section);
/** Each word's start time inside the clip, by its share of the text. */
const startsFor = (section: number, duration: number): number[] => {
const words = wordsOf(section);
const weights = words.map((word) => Math.max(1, word.text.replace(/[^\p{L}\p{N}]/gu, "").length));
const total = weights.reduce((sum, weight) => sum + weight, 0) || 1;
const starts: number[] = [];
let cursor = 0;
for (const weight of weights) {
starts.push((cursor / total) * duration);
cursor += weight;
}
return starts;
};
const load = (index: number): Promise<Clip> => {
const cached = clips.get(index);
if (cached) return Promise.resolve(cached);
const pending = loading.get(index);
if (pending) return pending;
const section = events.sections[index];
if (!section) return Promise.reject(new Error("no-section"));
events.onBusy?.(true);
const promise = synthesizeSpeech(
{
text: `${section.heading ? `${section.heading}. ` : ""}${section.text}`,
voice: events.voice,
model: events.model,
providerId: events.providerId,
},
)
.then(async ({ url, type }) => {
const duration = await measure(url);
const clip = { url, duration, type };
clips.set(index, clip);
loading.delete(index);
events.onBusy?.(false);
return clip;
})
.catch((error: unknown) => {
loading.delete(index);
events.onBusy?.(false);
throw error;
});
loading.set(index, promise);
return promise;
};
/** The clip's real length, read before playing so the clock can be built. */
const measure = (url: string): Promise<number> =>
new Promise((resolve) => {
const probe = new Audio();
probe.preload = "metadata";
const done = () => {
const duration = Number.isFinite(probe.duration) && probe.duration > 0 ? probe.duration : 0;
resolve(duration);
};
probe.onloadedmetadata = done;
probe.onerror = () => resolve(0);
probe.src = url;
/* Some engines never fire the event for blob URLs without a nudge. */
window.setTimeout(done, 2500);
});
const stopClock = () => {
if (frame) cancelAnimationFrame(frame);
frame = 0;
};
const startClock = () => {
const mine = generation;
const section = currentSection;
const clip = clips.get(section);
if (!clip) return;
const starts = startsFor(section, clip.duration);
const tick = () => {
if (generation !== mine) return;
const at = audio.currentTime;
/* Last word whose start is behind the cursor. */
let index = 0;
for (let i = 0; i < starts.length; i += 1) {
if (starts[i] <= at) index = i;
else break;
}
const next = { section, word: index };
if (!position || position.section !== section || position.word !== index) {
position = next;
events.onWord(next);
}
if (clip.duration > 0 && at >= clip.duration - 0.05) {
playFrom({ section: section + 1, word: 0 });
return;
}
frame = requestAnimationFrame(tick);
};
frame = requestAnimationFrame(tick);
};
const playFrom = (target: PodcastVoice) => {
const section = events.sections[target.section];
if (!section) {
playing = false;
position = null;
events.onPause?.();
events.onEnd();
return;
}
const mine = ++generation;
stopClock();
playing = false;
void load(target.section)
.then((clip) => {
if (generation !== mine) return;
const words = wordsOf(target.section);
const starts = startsFor(target.section, clip.duration);
const word = Math.min(Math.max(0, target.word), Math.max(0, words.length - 1));
currentSection = target.section;
position = { section: target.section, word };
events.onWord(position);
if (audio.src !== clip.url) audio.src = clip.url;
audio.playbackRate = rate;
try {
audio.currentTime = starts[word] ?? 0;
} catch {
/* not seekable yet: start from the top */
}
audio
.play()
.then(() => {
if (generation !== mine) return;
playing = true;
events.onPlay?.();
startClock();
/* The next part is generated while this one plays. */
if (events.sections[target.section + 1]) void load(target.section + 1).catch(() => {});
})
.catch(() => {
if (generation !== mine) return;
playing = false;
events.onError?.("blocked");
});
})
.catch((error: unknown) => {
if (generation !== mine) return;
const message = error instanceof Error ? error.message : "error";
events.onError?.(message);
});
};
audio.onended = () => {
if (currentSection + 1 >= events.sections.length) {
playing = false;
position = null;
events.onPause?.();
events.onEnd();
return;
}
playFrom({ section: currentSection + 1, word: 0 });
};
audio.onerror = () => {
playing = false;
events.onError?.("error");
};
return {
play(from) {
playFrom(from ?? position ?? { section: 0, word: 0 });
},
pause() {
generation += 1;
stopClock();
try {
audio.pause();
} catch {
/* already gone */
}
playing = false;
events.onPause?.();
},
stop() {
this.pause();
position = null;
},
playing: () => playing,
position: () => position,
prefetch(index) {
if (events.sections[index]) void load(index).catch(() => {});
},
setRate(next) {
rate = Math.min(2, Math.max(0.5, next));
audio.playbackRate = rate;
},
dispose() {
generation += 1;
stopClock();
try {
audio.pause();
audio.removeAttribute("src");
audio.remove();
} catch {
/* already gone */
}
for (const clip of clips.values()) URL.revokeObjectURL(clip.url);
clips.clear();
},
};
}

77
lib/podcast-natural.ts Normal file
View file

@ -0,0 +1,77 @@
"use client";
/**
* The provider-voice preference: which TTS provider and voice the listener
* picked for the podcast player ("system" keeps the platform's own voices).
* One blob in localStorage, read through a tiny accessor.
*/
import { PROVIDERS, TTS_PROFILES, ttsProfile, type ProviderInfo } from "./providers";
const STORAGE_KEY = "freezmo.podcast-natural.v1";
export type NaturalPref = {
source: "system" | "natural";
provider: string;
voice: string | null;
model: string | null;
};
const FALLBACK: NaturalPref = { source: "system", provider: "", voice: null, model: null };
export function readNaturalPref(): NaturalPref {
if (typeof window === "undefined") return FALLBACK;
try {
const raw = window.localStorage.getItem(STORAGE_KEY);
if (!raw) return FALLBACK;
const parsed = JSON.parse(raw) as Partial<NaturalPref> | null;
if (!parsed || typeof parsed !== "object") return FALLBACK;
return {
source: parsed.source === "natural" ? "natural" : "system",
provider: typeof parsed.provider === "string" ? parsed.provider : "",
voice: typeof parsed.voice === "string" ? parsed.voice : null,
model: typeof parsed.model === "string" ? parsed.model : null,
};
} catch {
return FALLBACK;
}
}
export function writeNaturalPref(next: Partial<NaturalPref>): NaturalPref {
const merged = { ...readNaturalPref(), ...next };
try {
window.localStorage.setItem(STORAGE_KEY, JSON.stringify(merged));
} catch {
/* private mode */
}
return merged;
}
/** Providers with a TTS profile that the listener actually has a key for. */
export function ttsProvidersWithKeys(keys: Record<string, string | undefined>): string[] {
return Object.keys(TTS_PROFILES).filter((id) => Boolean(keys[id]));
}
/** The provider to use when the listener has not chosen one. */
export function defaultTtsProvider(keys: Record<string, string | undefined>, generation?: string): string {
const available = ttsProvidersWithKeys(keys);
if (available.length === 0) return "";
if (generation && available.includes(generation)) return generation;
/* Prefer OpenAI when both are configured: its voices are the better ones. */
if (available.includes("openai")) return "openai";
return available[0];
}
export function voiceForProvider(providerId: string, wanted: string | null): string {
const profile = ttsProfile(providerId);
if (!profile) return "";
if (wanted && profile.voices.some((voice) => voice.id === wanted)) return wanted;
return profile.voices.find((voice) => voice.male)?.id ?? profile.voices[0]?.id ?? "";
}
/** Provider labels for the picker (id → label). */
export function ttsProviderLabel(id: string): string {
return PROVIDERS.find((provider: ProviderInfo) => provider.id === id)?.label ?? id;
}
export { TTS_PROFILES };

View file

@ -21,6 +21,16 @@ import type { PodcastSection } from "./schemas";
export type PodcastVoice = { section: number; word: number }; export type PodcastVoice = { section: number; word: number };
/** Tidy text for the speech engine (see the note where it is used). */
export function normaliseSpoken(raw: string): string {
return raw
.replace(/[\u0000-\u001f\u007f-\u009f\u200b-\u200f\ufeff]/g, " ")
.replace(/…/g, ", ")
.replace(/\s*—\s*/g, " — ")
.replace(/\s+/g, " ")
.trim();
}
export type PodcastSpeechEvents = { export type PodcastSpeechEvents = {
sections: PodcastSection[]; sections: PodcastSection[];
locale: string; locale: string;
@ -129,10 +139,16 @@ export function createPodcastSpeech(events: PodcastSpeechEvents): PodcastSpeech
position = { section: target.section, word }; position = { section: target.section, word };
const from = sectionTokens[word].start; const from = sectionTokens[word].start;
const text = section.text.slice(from); /* Spoken text hygiene: engines click on control characters, stray
double spaces and unicode ellipses, and some pause oddly around them. */
const text = normaliseSpoken(section.text.slice(from));
const current = ++generation; const current = ++generation;
try { try {
window.speechSynthesis.cancel(); /* Only cancel when something is actually pending: an idle cancel makes
several engines pop. */
if (window.speechSynthesis.speaking || window.speechSynthesis.pending) {
window.speechSynthesis.cancel();
}
const utterance = new SpeechSynthesisUtterance(text); const utterance = new SpeechSynthesisUtterance(text);
utterance.lang = events.locale; utterance.lang = events.locale;
utterance.rate = events.rate; utterance.rate = events.rate;
@ -200,7 +216,12 @@ export function createPodcastSpeech(events: PodcastSpeechEvents): PodcastSpeech
events.onEnd(); events.onEnd();
return; return;
} }
speakFrom({ section: target.section + 1, word: 0 }); /* A short beat between parts: speaking into the engine's tail is
what makes the boundary pop on several platforms. */
window.setTimeout(() => {
if (generation !== current) return;
speakFrom({ section: target.section + 1, word: 0 });
}, 140);
}; };
utterance.onerror = (event) => { utterance.onerror = (event) => {

View file

@ -87,6 +87,64 @@ export const PROVIDERS: ProviderInfo[] = [
}, },
]; ];
/**
* Text-to-speech for the podcast player. Only OpenAI-compatible
* `/audio/speech` endpoints are wired (OpenAI, Groq and any local server the
* custom provider points at); the rest simply do not offer TTS. Voice ids come
* from each provider's catalogue — the male ones lead because a podcast host
* was the ask.
*/
export type TtsVoice = { id: string; label: string; male?: boolean };
export type TtsProfile = {
/** Default model for /audio/speech. */
model: string;
voices: TtsVoice[];
/** Response format to request (Groq speaks WAV, everything else MP3). */
format: "mp3" | "wav";
};
export const TTS_PROFILES: Record<string, TtsProfile> = {
openai: {
model: "gpt-4o-mini-tts",
format: "mp3",
voices: [
{ id: "onyx", label: "Onyx", male: true },
{ id: "echo", label: "Echo", male: true },
{ id: "ash", label: "Ash", male: true },
{ id: "fable", label: "Fable" },
{ id: "nova", label: "Nova" },
{ id: "coral", label: "Coral" },
{ id: "shimmer", label: "Shimmer" },
],
},
groq: {
model: "playai-tts",
format: "wav",
voices: [
{ id: "Fritz-PlayAI", label: "Fritz", male: true },
{ id: "Thunder-PlayAI", label: "Thunder", male: true },
{ id: "Mason-PlayAI", label: "Mason", male: true },
{ id: "Celeste-PlayAI", label: "Celeste" },
{ id: "Cheyenne-PlayAI", label: "Cheyenne" },
],
},
custom: {
model: "tts-1",
format: "mp3",
voices: [
{ id: "onyx", label: "Onyx", male: true },
{ id: "echo", label: "Echo", male: true },
{ id: "alloy", label: "Alloy" },
{ id: "nova", label: "Nova" },
],
},
};
export function ttsProfile(providerId: string): TtsProfile | null {
return TTS_PROFILES[providerId] ?? null;
}
export function providerInfo(id: string): ProviderInfo | undefined { export function providerInfo(id: string): ProviderInfo | undefined {
return PROVIDERS.find((provider) => provider.id === id); return PROVIDERS.find((provider) => provider.id === id);
} }

View file

@ -1,6 +1,6 @@
{ {
"name": "freezmo", "name": "freezmo",
"version": "0.7.5", "version": "0.7.6",
"private": true, "private": true,
"license": "GPL-3.0-only", "license": "GPL-3.0-only",
"scripts": { "scripts": {

View file

@ -9,7 +9,7 @@
* *
* Bumping CACHE_VERSION is how a new deploy invalidates old assets. * Bumping CACHE_VERSION is how a new deploy invalidates old assets.
*/ */
const CACHE_VERSION = "freezmo-v97"; const CACHE_VERSION = "freezmo-v98";
const SHELL_CACHE = `${CACHE_VERSION}-shell`; const SHELL_CACHE = `${CACHE_VERSION}-shell`;
const ASSET_CACHE = `${CACHE_VERSION}-assets`; const ASSET_CACHE = `${CACHE_VERSION}-assets`;

View file

@ -60,7 +60,13 @@ export function resolveChrome() {
export const CHROME = resolveChrome(); export const CHROME = resolveChrome();
export async function launchBrowser() { export async function launchBrowser() {
return chromium.launch({ executablePath: CHROME, args: ["--no-sandbox"] }); return chromium.launch({
executablePath: CHROME,
/* The podcast player plays generated audio through an <audio> element;
headless Chromium would refuse it without this flag (a real device has
the listener's own gesture, which is the policy's real gate). */
args: ["--no-sandbox", "--autoplay-policy=no-user-gesture-required"],
});
} }
/** axe-core source, injected into the page for the contrast audit. */ /** axe-core source, injected into the page for the contrast audit. */

View file

@ -368,6 +368,91 @@ try {
await context.close(); await context.close();
} }
/* 4b3. natural voices: the provider TTS behind the same player ------- */
{
const { context, page } = await makePage(browser, { width: 390, height: 844, deviceScaleFactor: 1, isMobile: true, hasTouch: true });
/* One second of silence as a real WAV — the player measures the clip and
times its words against it, so the header alone is not enough. */
const wav = (() => {
const sampleRate = 8000;
const samples = sampleRate;
const buffer = Buffer.alloc(44 + samples * 2);
buffer.write("RIFF", 0);
buffer.writeUInt32LE(36 + samples * 2, 4);
buffer.write("WAVE", 8);
buffer.write("fmt ", 12);
buffer.writeUInt32LE(16, 16);
buffer.writeUInt16LE(1, 20);
buffer.writeUInt16LE(1, 22);
buffer.writeUInt32LE(sampleRate, 24);
buffer.writeUInt32LE(sampleRate * 2, 28);
buffer.writeUInt16LE(2, 32);
buffer.writeUInt16LE(16, 34);
buffer.write("data", 36);
buffer.writeUInt32LE(samples * 2, 40);
return buffer;
})();
const bodies = [];
await page.route("**/api/tts", async (route) => {
bodies.push(route.request().postDataJSON());
await route.fulfill({ status: 200, contentType: "audio/wav", body: wav });
});
await page.addInitScript(() => {
localStorage.setItem(
"freezmo.providers.v1",
JSON.stringify({ keys: { openai: "sk-test" }, models: {}, customBaseUrl: "", generation: null }),
);
});
await seed(page, { decks: [], theme: "dark", extra: { "freezmo.podcasts.v1": JSON.stringify([EPISODE_RECORD]) } });
await page.goto(`${APP}/podcast/pod_local`, { waitUntil: "networkidle" });
await page.waitForTimeout(1000);
await page.getByRole("button", { name: "Voice" }).click();
await page.waitForTimeout(300);
await page.getByRole("button", { name: "Natural voice" }).click();
await page.waitForTimeout(400);
const voicesShown = await page.evaluate(() =>
[...document.querySelectorAll("button[aria-pressed]")].map((el) => el.textContent?.trim() ?? ""),
);
check(
"natural: the provider's voices are offered (male first)",
voicesShown.some((entry) => entry.includes("Onyx")) && voicesShown.some((entry) => entry.includes("male")),
JSON.stringify(voicesShown.slice(0, 5)),
);
await page.getByRole("button", { name: /Onyx/ }).click();
await page.waitForTimeout(300);
await page.getByRole("button", { name: "Close" }).click();
await page.waitForTimeout(200);
await page.getByRole("button", { name: "Play the episode" }).click();
/* The stub clip is half a second of silence: check while it is playing. */
await page.waitForTimeout(320);
const first = bodies[0];
check("natural: the voice is generated through the API", bodies.length >= 1, String(bodies.length));
check(
"natural: the request carries the section text and the picked voice",
Boolean(first) && String(first.text).startsWith("Daylight.") && first.voice === "onyx",
JSON.stringify({ voice: first?.voice, head: String(first?.text).slice(0, 24) }),
);
const audioState = await page.evaluate(() => {
const elements = [...document.querySelectorAll("[data-podcast-audio]")];
const audio = elements.at(-1);
return audio ? { count: elements.length, paused: audio.paused, blob: audio.src.startsWith("blob:") } : "no-audio-element";
});
check(
"natural: one audio element plays the clip",
typeof audioState === "object" && audioState.blob && audioState.paused === false && audioState.count === 1,
JSON.stringify(audioState),
);
check("natural: the position is remembered for later", (await page.evaluate(() => localStorage.getItem("freezmo.podcast-natural.v1")))?.includes("onyx"), "");
/* The word highlight rides the clip's real clock. */
const before = await page.evaluate(() => document.querySelector("[data-word-active]")?.textContent ?? "");
await page.waitForTimeout(1200);
const after = await page.evaluate(() => document.querySelector("[data-word-active]")?.textContent ?? "");
check("natural: the highlight follows the audio", before !== "" && after !== "", JSON.stringify({ before, after }));
await context.close();
}
/* 4c. the transcript follows by line, and yields to a manual scroll --- */ /* 4c. the transcript follows by line, and yields to a manual scroll --- */
{ {
const { context, page } = await makePage(browser, { width: 390, height: 700, deviceScaleFactor: 1, isMobile: true, hasTouch: true }); const { context, page } = await makePage(browser, { width: 390, height: 700, deviceScaleFactor: 1, isMobile: true, hasTouch: true });