Podcast voices: natural provider TTS, and system-voice hygiene
Two reports, one root: the platform's speech engine. "Loud cracking at the end of every sentence" is the browser/OS TTS popping between its internal sentence chunks (an idle cancel makes it worse). Hygiene for that path: never cancel when nothing is pending, normalise the spoken text (control characters, ellipses, dashes), and a 140ms beat between sections so speech never runs into the engine's tail. The real fix is a second voice source that does not use the platform engine: - lib/providers carries TTS profiles (OpenAI gpt-4o-mini-tts, Groq playai-tts, any OpenAI-compatible /audio/speech behind the custom provider), male voices first. - /api/tts proxies them with the same guards and the key still in headers. - the picker gains System / Natural, with the provider's voice list. - lib/podcast-audio plays one clip per section through a hidden <audio> element, measures the clip's real duration and gives each word a slice of it proportional to its spoken length - the highlight follows the audio exactly, and pause/seek/chapters become real audio operations. Clips cache per session, the next section generates while the current plays, and a voice change disposes the element and its blobs. Speed applies live (setRate) without dropping the cache. podcast-test stubs /api/tts with a real silent WAV and pins the path: the voice list, the request body, exactly one playing element, and the highlight riding currentTime. 43 checks. tests/lib.mjs launches Chromium with the autoplay policy relaxed (a real device has the listener's gesture). Version 0.7.6, SW v98.
This commit is contained in:
parent
9882d775e7
commit
8e1cd0855d
13 changed files with 997 additions and 63 deletions
20
README.md
20
README.md
|
|
@ -1492,6 +1492,26 @@ Things I chose while building, with the reasoning:
|
|||
into view as you type. No strings, so no i18n; the additive change sits on top of the merged
|
||||
scene-heat rework unchanged.
|
||||
|
||||
104. **Natural voices — and the crack at the end of every sentence.** Two reports, one root:
|
||||
the platform's speech engine. "There's a loud cracking sound at the end of every sentence" is
|
||||
the browser/OS TTS (iOS and several Linux engines pop between their internal sentence chunks,
|
||||
and an unnecessary `cancel()` makes it worse) — nothing in our code touches audio there. Both
|
||||
halves are answered: the system path got hygiene (never cancel while idle, strip control
|
||||
characters and normalise ellipses/dashes before speaking, and a 140 ms beat between sections
|
||||
so speech never runs into the engine's tail), and the real fix is a second voice source that
|
||||
does not use the platform engine at all. `lib/providers` now carries TTS profiles (OpenAI
|
||||
`gpt-4o-mini-tts`, Groq `playai-tts`, and any OpenAI-compatible `/audio/speech` the custom
|
||||
provider points at; male voices first), `/api/tts` proxies them so the key still never leaves
|
||||
the browser, and the player's picker gains a **System / Natural** switch with the provider's
|
||||
voice list. The natural engine (`lib/podcast-audio`) plays one clip per section through an
|
||||
`<audio>` element: the clip's real duration is measured and each word gets a slice of it
|
||||
proportional to its spoken length, so the highlight follows the audio exactly instead of
|
||||
estimating — and pause, seek and chapter jumps become real audio operations. Clips are cached
|
||||
per session, the next section is generated while the current one plays, and switching voice
|
||||
disposes the old element and its blobs. The suite stubs the TTS endpoint with a real silent
|
||||
WAV and pins the whole path (voice list, request body, one playing element, the highlight
|
||||
riding `currentTime`) — 43 checks.
|
||||
|
||||
103. **Podcast mode: your notes, read aloud, highlighted word by word.** The third segment of
|
||||
the note-powered wizards (Study deck ⇄ Homework ⇄ Podcast) takes the same intake — photos,
|
||||
documents, pasted text, the share target — and turns it into a spoken episode instead of
|
||||
|
|
|
|||
129
app/api/tts/route.ts
Normal file
129
app/api/tts/route.ts
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
import { getServerConfig } from "@/lib/config";
|
||||
import { ttsProfile } from "@/lib/providers";
|
||||
import { resolveModelTarget } from "@/lib/providers-server";
|
||||
import { rejectCrossOrigin, rejectOversizedBody, requestLocale } from "@/lib/http-guard";
|
||||
import { clientKey, rateLimit, rateLimitResponse } from "@/lib/rate-limit";
|
||||
import { acquireModelGate } from "@/lib/model-guard";
|
||||
import { translate } from "@/lib/i18n-dict";
|
||||
import { z } from "zod";
|
||||
|
||||
export const runtime = "nodejs";
|
||||
export const dynamic = "force-dynamic";
|
||||
export const maxDuration = 120;
|
||||
|
||||
const requestSchema = z.object({
|
||||
/** One section of the script — a couple of hundred words at most. */
|
||||
text: z.string().min(1).max(4000),
|
||||
voice: z.string().min(1).max(60).optional(),
|
||||
model: z.string().min(1).max(120).optional(),
|
||||
});
|
||||
|
||||
/** About 1 MB of text; audio comes back in the response, not the request. */
|
||||
const MAX_BODY_BYTES = 1_048_576;
|
||||
|
||||
/**
|
||||
* Natural podcast voices: the provider's own text-to-speech, proxied so the
|
||||
* key stays in the browser (it travels in the header, never lands on disk).
|
||||
* OpenAI-compatible `/audio/speech` only — OpenAI, Groq, and whatever local
|
||||
* server the custom provider points at.
|
||||
*/
|
||||
export async function POST(request: Request) {
|
||||
const locale = requestLocale(request);
|
||||
const crossOrigin = rejectCrossOrigin(request);
|
||||
if (crossOrigin) return crossOrigin;
|
||||
const oversized = rejectOversizedBody(request, MAX_BODY_BYTES);
|
||||
if (oversized) return oversized;
|
||||
|
||||
const config = getServerConfig();
|
||||
const limit = rateLimit(`tts:${clientKey(request)}`, config.rateLimits.generate, config.rateLimits.windowMs);
|
||||
if (!limit.allowed) return rateLimitResponse(limit, locale);
|
||||
|
||||
/* TTS is a provider feature: no provider key, no natural voice. */
|
||||
const resolved = resolveModelTarget(config, request, locale);
|
||||
if ("error" in resolved) {
|
||||
return Response.json({ error: resolved.error.message, kind: resolved.error.kind }, { status: resolved.error.status });
|
||||
}
|
||||
if (resolved.target.kind !== "provider") {
|
||||
return Response.json(
|
||||
{ error: translate(locale, "server.ttsNeedsProvider"), kind: "needs_key" },
|
||||
{ status: 400 },
|
||||
);
|
||||
}
|
||||
const provider = resolved.target.target;
|
||||
const providerId = (request.headers.get("x-freezmo-provider") ?? "").trim().toLowerCase();
|
||||
const profile = ttsProfile(providerId);
|
||||
if (!profile) {
|
||||
return Response.json(
|
||||
{ error: translate(locale, "server.ttsUnsupported", { provider: provider.label }), kind: "bad_provider" },
|
||||
{ status: 400 },
|
||||
);
|
||||
}
|
||||
|
||||
let payload: unknown;
|
||||
try {
|
||||
payload = await request.json();
|
||||
} catch {
|
||||
return Response.json({ error: translate(locale, "server.bodyNotJson") }, { status: 400 });
|
||||
}
|
||||
const parsed = requestSchema.safeParse(payload);
|
||||
if (!parsed.success) {
|
||||
return Response.json({ error: translate(locale, "server.invalidRequest", { detail: "text" }) }, { status: 400 });
|
||||
}
|
||||
|
||||
const gate = acquireModelGate(request, config, locale);
|
||||
if ("response" in gate) return gate.response;
|
||||
|
||||
const model = parsed.data.model ?? profile.model;
|
||||
const voice = parsed.data.voice ?? profile.voices.find((entry) => entry.male)?.id ?? profile.voices[0].id;
|
||||
const abort = new AbortController();
|
||||
const timer = setTimeout(() => abort.abort(), config.timeoutMs);
|
||||
const onRequestAbort = () => abort.abort();
|
||||
if (request.signal.aborted) abort.abort();
|
||||
else request.signal.addEventListener("abort", onRequestAbort, { once: true });
|
||||
|
||||
try {
|
||||
const upstream = await fetch(`${provider.baseUrl.replace(/\/$/, "")}/audio/speech`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: `Bearer ${provider.apiKey}`,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model,
|
||||
voice,
|
||||
input: parsed.data.text,
|
||||
response_format: profile.format,
|
||||
}),
|
||||
signal: abort.signal,
|
||||
});
|
||||
|
||||
if (!upstream.ok || !upstream.body) {
|
||||
const detail = await upstream.text().catch(() => "");
|
||||
/* Never log the text that was spoken; only the status. */
|
||||
console.error("[api/tts]", upstream.status, detail.slice(0, 200));
|
||||
const kind = upstream.status === 401 || upstream.status === 403 ? "auth" : upstream.status === 429 ? "usage_limit" : "unknown";
|
||||
return Response.json(
|
||||
{ error: translate(locale, "server.ttsFailed", { provider: provider.label }), kind },
|
||||
{ status: upstream.status === 429 ? 429 : 502 },
|
||||
);
|
||||
}
|
||||
|
||||
return new Response(upstream.body, {
|
||||
headers: {
|
||||
"Content-Type": profile.format === "wav" ? "audio/wav" : "audio/mpeg",
|
||||
"Cache-Control": "no-store, no-transform",
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
const timedOut = error instanceof Error && error.name === "AbortError" && !request.signal.aborted;
|
||||
console.error("[api/tts]", (error as Error)?.name ?? error);
|
||||
return Response.json(
|
||||
{ error: translate(locale, timedOut ? "server.ttsTimeout" : "server.ttsFailed", { provider: provider.label }), kind: "network" },
|
||||
{ status: 502 },
|
||||
);
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
request.signal.removeEventListener("abort", onRequestAbort);
|
||||
gate.gate.release();
|
||||
}
|
||||
}
|
||||
|
|
@ -4,6 +4,16 @@ import Link from "next/link";
|
|||
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
|
||||
|
||||
import { useI18n } from "@/lib/i18n";
|
||||
import { createPodcastAudio, type PodcastAudio } from "@/lib/podcast-audio";
|
||||
import {
|
||||
defaultTtsProvider,
|
||||
readNaturalPref,
|
||||
ttsProviderLabel,
|
||||
voiceForProvider,
|
||||
writeNaturalPref,
|
||||
} from "@/lib/podcast-natural";
|
||||
import { TTS_PROFILES } from "@/lib/providers";
|
||||
import { useProviders } from "@/lib/providers-client";
|
||||
import {
|
||||
createPodcastSpeech,
|
||||
tokenizeSections,
|
||||
|
|
@ -54,6 +64,79 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
const [position, setPosition] = useState<PodcastVoice | null>(null);
|
||||
const [unsupported, setUnsupported] = useState(false);
|
||||
const [voiceOpen, setVoiceOpen] = useState(false);
|
||||
/** "system" = the platform's voices; "natural" = the provider's TTS. */
|
||||
const [voiceSource, setVoiceSource] = useState<"system" | "natural">(() => readNaturalPref().source);
|
||||
const [naturalProvider, setNaturalProvider] = useState<string>(() => readNaturalPref().provider);
|
||||
const [naturalVoiceId, setNaturalVoiceId] = useState<string | null>(() => readNaturalPref().voice);
|
||||
const [naturalError, setNaturalError] = useState<string | null>(null);
|
||||
const [naturalBusy, setNaturalBusy] = useState(false);
|
||||
const providers = useProviders();
|
||||
|
||||
/* Either engine drives the same onWord/onEnd events; the transport below
|
||||
routes to whichever one the listeners picked. */
|
||||
const natural = useMemo<PodcastAudio | null>(() => {
|
||||
if (voiceSource !== "natural" || !naturalProvider) return null;
|
||||
return createPodcastAudio({
|
||||
sections: episode.sections,
|
||||
providerId: naturalProvider,
|
||||
voice: voiceForProvider(naturalProvider, naturalVoiceId),
|
||||
model: readNaturalPref().model ?? undefined,
|
||||
rate,
|
||||
onWord: (next) => {
|
||||
latest.current = next;
|
||||
setPosition(next);
|
||||
},
|
||||
onPlay: () => {
|
||||
setPlaying(true);
|
||||
setNaturalError(null);
|
||||
window.dispatchEvent(new CustomEvent("freezmo:foreground-audio", { detail: "podcast" }));
|
||||
},
|
||||
onPause: () => setPlaying(false),
|
||||
onEnd: () => {
|
||||
setPlaying(false);
|
||||
setPosition(null);
|
||||
latest.current = null;
|
||||
savePodcastProgress(episode.id, { section: 0, word: 0 });
|
||||
},
|
||||
onError: (message) => {
|
||||
setPlaying(false);
|
||||
setNaturalError(message);
|
||||
},
|
||||
onBusy: setNaturalBusy,
|
||||
});
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps -- speed is applied live (setRate below), so it must not rebuild the engine
|
||||
}, [voiceSource, naturalProvider, naturalVoiceId, episode.id, episode.sections]);
|
||||
|
||||
/* Speed changes apply to the element, not to the engine's cache. */
|
||||
useEffect(() => {
|
||||
natural?.setRate(rate);
|
||||
}, [natural, rate]);
|
||||
|
||||
/* The engine owns an <audio> element: release it when the engine is
|
||||
replaced (a different voice) or the player closes. */
|
||||
useEffect(() => {
|
||||
if (!natural) return;
|
||||
return () => {
|
||||
try {
|
||||
natural.dispose();
|
||||
} catch {
|
||||
/* already gone */
|
||||
}
|
||||
};
|
||||
}, [natural]);
|
||||
|
||||
useEffect(() => {
|
||||
return () => naturalRef.current?.dispose();
|
||||
}, []);
|
||||
|
||||
/* The default pick the first time someone opens the natural panel. */
|
||||
useEffect(() => {
|
||||
if (naturalProvider) return;
|
||||
const keys = providers.keys;
|
||||
const fallback = defaultTtsProvider(keys, providers.generation?.provider);
|
||||
if (fallback) setNaturalProvider(fallback);
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps -- once per mount
|
||||
}, [providers.keys]);
|
||||
const transcript = useRef<HTMLDivElement>(null);
|
||||
const activeWord = useRef<HTMLElement | null>(null);
|
||||
/** Line-following state: the last followed line and a manual-scroll pause. */
|
||||
|
|
@ -100,6 +183,10 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
}
|
||||
return bestVoice(voiceLocale, voiceList);
|
||||
}, [voiceURI, availableVoices, voiceLocale, voiceList]);
|
||||
const ttsProviderIds = useMemo(
|
||||
() => Object.keys(TTS_PROFILES).filter((id) => Boolean(providers.keys[id])),
|
||||
[providers.keys],
|
||||
);
|
||||
const roboticOnly = useMemo(
|
||||
() => onlyRoboticVoices(voiceLocale, voiceList),
|
||||
[voiceLocale, voiceList],
|
||||
|
|
@ -172,10 +259,14 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
/* The engine instance is rebuilt on a speed change; keep a ref so the
|
||||
unmount cleanup always stops whichever one is current. */
|
||||
const speechRef = useRef(speech);
|
||||
const naturalRef = useRef<PodcastAudio | null>(natural);
|
||||
const playingRef = useRef(false);
|
||||
useEffect(() => {
|
||||
speechRef.current = speech;
|
||||
}, [speech]);
|
||||
useEffect(() => {
|
||||
naturalRef.current = natural;
|
||||
}, [natural]);
|
||||
useEffect(() => {
|
||||
playingRef.current = playing;
|
||||
}, [playing]);
|
||||
|
|
@ -289,27 +380,33 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
);
|
||||
}, [position, tokens]);
|
||||
|
||||
const naturalActive = voiceSource === "natural" && Boolean(natural);
|
||||
/** The engines share their shape; the transport routes by source. */
|
||||
const canPlay = naturalActive ? true : supported;
|
||||
|
||||
const harness = () => (naturalActive ? natural : speech);
|
||||
|
||||
const toggle = () => {
|
||||
if (!speech) return;
|
||||
if (!canPlay) return;
|
||||
if (playing) {
|
||||
speech.pause();
|
||||
harness()?.pause();
|
||||
persist();
|
||||
return;
|
||||
}
|
||||
/* Other readers (a card's read-aloud) yield to the episode. */
|
||||
stopSpeaking();
|
||||
speech.play(position ?? { section: 0, word: 0 });
|
||||
harness()?.play(position ?? { section: 0, word: 0 });
|
||||
};
|
||||
|
||||
const seek = (next: PodcastVoice) => {
|
||||
if (!speech) return;
|
||||
if (!canPlay) return;
|
||||
latest.current = next;
|
||||
setPosition(next);
|
||||
/* A seek is an instruction to show that place: follow it at once. */
|
||||
lastLineTop.current = null;
|
||||
followPausedUntil.current = 0;
|
||||
stopSpeaking();
|
||||
speech.play(next);
|
||||
harness()?.play(next);
|
||||
};
|
||||
|
||||
const jump = (section: number) => seek({ section, word: 0 });
|
||||
|
|
@ -375,7 +472,7 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
{sections[currentSection]?.heading ?? ""}
|
||||
</p>
|
||||
</div>
|
||||
{availableVoices.length > 0 ? (
|
||||
{availableVoices.length > 0 || ttsProviderIds.length > 0 ? (
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setVoiceOpen((open) => !open)}
|
||||
|
|
@ -398,6 +495,14 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
) : null}
|
||||
</button>
|
||||
) : null}
|
||||
{naturalBusy ? (
|
||||
<span
|
||||
role="status"
|
||||
className="shrink-0 rounded-full bg-[var(--surface-sunken)] px-2.5 py-2 text-[11px] font-semibold text-[var(--text-muted)]"
|
||||
>
|
||||
{t("podcast.voiceGenerating")}
|
||||
</span>
|
||||
) : null}
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => {
|
||||
|
|
@ -426,7 +531,7 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
className="no-scrollbar min-h-0 flex-1 overflow-y-auto overscroll-contain px-4 pb-40 pt-24"
|
||||
>
|
||||
<div className="mx-auto w-full max-w-3xl space-y-6">
|
||||
{!supported || unsupported ? (
|
||||
{(!supported || unsupported) && !naturalActive ? (
|
||||
<p className="rounded-2xl border border-amber-300/50 bg-amber-50/70 px-3.5 py-3 text-[12.5px] leading-relaxed text-amber-800 dark:border-amber-400/25 dark:bg-amber-400/10 dark:text-amber-200">
|
||||
{t("podcast.noVoice")}
|
||||
{isAppleMobile() ? ` ${t("podcast.voiceIos")}` : ""}
|
||||
|
|
@ -497,14 +602,48 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
{t("common.close")}
|
||||
</button>
|
||||
</div>
|
||||
{/* Source: the platform's voices, or the provider's TTS. */}
|
||||
<div className="grid grid-cols-2 gap-1.5 px-1 pb-1">
|
||||
{(["system", "natural"] as const).map((source) => (
|
||||
<button
|
||||
key={source}
|
||||
type="button"
|
||||
aria-pressed={voiceSource === source}
|
||||
onClick={() => {
|
||||
setVoiceSource(source);
|
||||
writeNaturalPref({ source });
|
||||
setNaturalError(null);
|
||||
/* Swapping engines mid-sentence: stop the other one. */
|
||||
if (playing) {
|
||||
harness()?.pause();
|
||||
setPlaying(false);
|
||||
}
|
||||
}}
|
||||
className={cn(
|
||||
"touch-target rounded-xl border px-3 py-2 text-[12.5px] font-semibold transition",
|
||||
voiceSource === source
|
||||
? "border-[var(--nav-active-border)] bg-[var(--nav-active-bg)] text-[var(--text)]"
|
||||
: "border-[var(--hairline)] bg-[var(--surface-sunken)] text-[var(--text-muted)] hover:bg-[var(--hover)]",
|
||||
)}
|
||||
>
|
||||
{source === "system" ? t("podcast.voiceSystem") : t("podcast.voiceNatural")}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
{voiceSource === "system" ? (
|
||||
<>
|
||||
{availableVoices.length === 0 ? (
|
||||
<p className="px-1 pt-1 text-[11.5px] leading-relaxed text-[var(--text-subtle)]">
|
||||
{t("podcast.voiceNoSystem")}
|
||||
</p>
|
||||
) : null}
|
||||
{voiceURI ? (
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => {
|
||||
setVoiceURI(null);
|
||||
window.localStorage.removeItem(
|
||||
"freezmo.podcast-voice.v1",
|
||||
);
|
||||
window.localStorage.removeItem("freezmo.podcast-voice.v1");
|
||||
}}
|
||||
className="nav-row touch-target w-full text-left text-[13px]"
|
||||
>
|
||||
|
|
@ -525,34 +664,85 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
}}
|
||||
className={cn(
|
||||
"nav-row touch-target flex w-full items-center gap-2 text-left text-[13px]",
|
||||
active &&
|
||||
"border-[var(--nav-active-border)] bg-[var(--nav-active-bg)]",
|
||||
active && "border-[var(--nav-active-border)] bg-[var(--nav-active-bg)]",
|
||||
)}
|
||||
>
|
||||
<span className="min-w-0 flex-1">
|
||||
<span className="block truncate font-medium text-[var(--text)]">
|
||||
{entry.name}
|
||||
</span>
|
||||
<span className="block truncate font-medium text-[var(--text)]">{entry.name}</span>
|
||||
<span className="block text-[11.5px] text-[var(--text-subtle)]">
|
||||
{entry.lang}
|
||||
{entry.localService
|
||||
? ` · ${t("podcast.voiceLocal")}`
|
||||
: ""}
|
||||
{entry.localService ? ` · ${t("podcast.voiceLocal")}` : ""}
|
||||
</span>
|
||||
</span>
|
||||
{active ? (
|
||||
<CheckCircleIcon className="size-4 shrink-0 text-accent-600 dark:text-accent-200" />
|
||||
) : null}
|
||||
{active ? <CheckCircleIcon className="size-4 shrink-0 text-accent-600 dark:text-accent-200" /> : null}
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
<p className="px-1 pt-1 text-[11.5px] leading-relaxed text-[var(--text-subtle)]">
|
||||
{iosLimited
|
||||
? t("podcast.voiceIos")
|
||||
: roboticOnly
|
||||
? t("podcast.voiceRobotic")
|
||||
: t("podcast.voiceNote")}
|
||||
{iosLimited ? t("podcast.voiceIos") : roboticOnly ? t("podcast.voiceRobotic") : t("podcast.voiceNote")}
|
||||
</p>
|
||||
</>
|
||||
) : ttsProviderIds.length === 0 ? (
|
||||
<p className="px-1 pt-1 text-[11.5px] leading-relaxed text-[var(--text-subtle)]">
|
||||
{t("podcast.voiceNeedsKey")}
|
||||
</p>
|
||||
) : (
|
||||
<>
|
||||
{ttsProviderIds.length > 1 ? (
|
||||
<div className="flex flex-wrap gap-1.5 px-1 pb-1 pt-1">
|
||||
{ttsProviderIds.map((id) => (
|
||||
<button
|
||||
key={id}
|
||||
type="button"
|
||||
aria-pressed={naturalProvider === id}
|
||||
onClick={() => {
|
||||
setNaturalProvider(id);
|
||||
setNaturalVoiceId(null);
|
||||
writeNaturalPref({ provider: id, voice: null });
|
||||
}}
|
||||
className={cn(
|
||||
"rounded-full px-2.5 py-1.5 text-[11.5px] font-semibold transition",
|
||||
naturalProvider === id
|
||||
? "bg-[var(--nav-active-bg)] text-[var(--text)] ring-1 ring-[var(--nav-active-border)]"
|
||||
: "bg-[var(--surface-sunken)] text-[var(--text-muted)] hover:bg-[var(--hover)]",
|
||||
)}
|
||||
>
|
||||
{ttsProviderLabel(id)}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
) : null}
|
||||
{(TTS_PROFILES[naturalProvider]?.voices ?? []).map((entry) => {
|
||||
const active = entry.id === voiceForProvider(naturalProvider, naturalVoiceId);
|
||||
return (
|
||||
<button
|
||||
key={entry.id}
|
||||
type="button"
|
||||
aria-pressed={active}
|
||||
onClick={() => {
|
||||
setNaturalVoiceId(entry.id);
|
||||
writeNaturalPref({ voice: entry.id });
|
||||
}}
|
||||
className={cn(
|
||||
"nav-row touch-target flex w-full items-center gap-2 text-left text-[13px]",
|
||||
active && "border-[var(--nav-active-border)] bg-[var(--nav-active-bg)]",
|
||||
)}
|
||||
>
|
||||
<span className="min-w-0 flex-1">
|
||||
<span className="block truncate font-medium text-[var(--text)]">{entry.label}</span>
|
||||
<span className="block text-[11.5px] text-[var(--text-subtle)]">
|
||||
{entry.male ? t("podcast.voiceMale") : t("podcast.voiceFemale")}
|
||||
</span>
|
||||
</span>
|
||||
{active ? <CheckCircleIcon className="size-4 shrink-0 text-accent-600 dark:text-accent-200" /> : null}
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
<p className="px-1 pt-1 text-[11.5px] leading-relaxed text-[var(--text-subtle)]">
|
||||
{naturalError ?? t("podcast.voiceNaturalNote")}
|
||||
</p>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
) : null}
|
||||
|
|
@ -573,7 +763,7 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
<button
|
||||
type="button"
|
||||
onClick={toggle}
|
||||
disabled={!supported}
|
||||
disabled={!canPlay}
|
||||
aria-label={playing ? t("podcast.pause") : t("podcast.play")}
|
||||
title={playing ? t("podcast.pause") : t("podcast.play")}
|
||||
className="grid size-12 shrink-0 place-items-center rounded-full grad-ice text-[#06111f] transition disabled:opacity-45"
|
||||
|
|
@ -605,7 +795,7 @@ export function PodcastPlayer({ episode }: { episode: Podcast }) {
|
|||
<button
|
||||
type="button"
|
||||
onClick={() => {
|
||||
speech?.stop();
|
||||
harness()?.stop();
|
||||
setPosition({ section: 0, word: 0 });
|
||||
latest.current = { section: 0, word: 0 };
|
||||
savePodcastProgress(episode.id, { section: 0, word: 0 });
|
||||
|
|
|
|||
|
|
@ -291,6 +291,43 @@ export async function generatePodcast(
|
|||
return { podcast: ready, meta: {} };
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Natural podcast voices (provider text-to-speech) */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
/**
|
||||
* One section of the script, spoken by the picked provider. The key rides in
|
||||
* the headers exactly like every other model call; the audio comes back as a
|
||||
* blob URL the player hands to an <audio> element.
|
||||
*/
|
||||
export async function synthesizeSpeech(
|
||||
input: { text: string; voice?: string; model?: string; providerId: string },
|
||||
options: { signal?: AbortSignal } = {},
|
||||
): Promise<{ url: string; type: string }> {
|
||||
const response = await fetch("/api/tts", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
...authHeaders(),
|
||||
...providerHeaders(input.providerId),
|
||||
...localeHeader(),
|
||||
},
|
||||
body: JSON.stringify({ text: input.text, voice: input.voice, model: input.model }),
|
||||
signal: options.signal,
|
||||
}).catch(() => {
|
||||
throw new ApiError(t("api.unreachable"), "network");
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const body = (await response.json().catch(() => null)) as { error?: string; kind?: string } | null;
|
||||
throw new ApiError(body?.error ?? t("api.httpError", { status: response.status }), body?.kind);
|
||||
}
|
||||
|
||||
const blob = await response.blob();
|
||||
if (blob.size === 0) throw new ApiError(t("api.noAudio"), "bad_response");
|
||||
return { url: URL.createObjectURL(blob), type: blob.type };
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Make my homework */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
|
|
|||
|
|
@ -307,6 +307,14 @@ const en: Record<string, string> = {
|
|||
"podcast.voiceLocal": "installed",
|
||||
"podcast.voiceNote": "Voices come from your browser and system — pick the one that sounds best to you.",
|
||||
"podcast.voiceIos": "On iPhone, Safari offers web pages a single voice per language — the system's preferred one. For more (and better) voices: Settings → Accessibility → Spoken Content → Voices → your language, download a voice in Enhanced or Premium quality and make it the preferred one. Freezmo then reads with it.",
|
||||
"podcast.voiceSystem": "System voices",
|
||||
"podcast.voiceNatural": "Natural voice",
|
||||
"podcast.voiceMale": "male",
|
||||
"podcast.voiceFemale": "female",
|
||||
"podcast.voiceNaturalNote": "Generated by your own provider key — the words are timed to the audio, so the highlight still follows the voice.",
|
||||
"podcast.voiceNeedsKey": "Add an OpenAI or Groq key in Settings → Providers to use natural voices.",
|
||||
"podcast.voiceNoSystem": "This device reports no system voices. The natural voice works anywhere — or add a voice in your system settings.",
|
||||
"podcast.voiceGenerating": "Generating the voice…",
|
||||
"podcast.voiceRobotic": "Only the basic system voices are installed. For a natural host voice, add one in your system settings (Windows: Speech; macOS: Spoken Content; Chrome ships the Google voices).",
|
||||
"podcast.tapToJump": "Tap any word to jump there.",
|
||||
"podcast.noVoice": "This browser has no speech voice installed, so the transcript cannot be read aloud here. You can still read along.",
|
||||
|
|
@ -808,6 +816,8 @@ const en: Record<string, string> = {
|
|||
"api.streamLost": "The connection dropped mid-run. Check your connection and try again.",
|
||||
"api.almost": "Almost there…",
|
||||
"api.noDeck": "No deck came back from the server. Check your Ollama settings and try again.",
|
||||
"api.noPodcast": "No script came back from the server. Check the model setup and try again.",
|
||||
"api.noAudio": "The voice came back empty. Try another voice or provider.",
|
||||
"api.incomplete": "The server finished without sending a deck.",
|
||||
"api.noActivities": "The generated deck had no usable activities. Try again.",
|
||||
"api.gradingCancelled": "Grading was cancelled.",
|
||||
|
|
@ -822,6 +832,10 @@ const en: Record<string, string> = {
|
|||
"server.bodyNotJson": "Request body was not valid JSON.",
|
||||
"server.invalidRequest": "Invalid request: {detail}",
|
||||
"server.providerModelNeeded": "Pick the generation model in Settings → Providers.",
|
||||
"server.ttsNeedsProvider": "Natural voices come from a provider — add a provider key in Settings → Providers first.",
|
||||
"server.ttsUnsupported": "{provider} does not offer text-to-speech. OpenAI, Groq or a custom OpenAI-compatible server will.",
|
||||
"server.ttsFailed": "{provider} could not generate the voice for this part.",
|
||||
"server.ttsTimeout": "{provider} took too long to generate the voice.",
|
||||
"server.needContent": "Add at least one photo of your notes or paste some text to build a deck.",
|
||||
"server.needAssignment":
|
||||
"Add the assignment first — paste its text, photograph it, or add a document.",
|
||||
|
|
@ -1380,6 +1394,14 @@ const cs: Record<string, string> = {
|
|||
"podcast.voiceLocal": "nainstalovaný",
|
||||
"podcast.voiceNote": "Hlasy dodává prohlížeč a systém — vyber si ten, co zní nejlíp.",
|
||||
"podcast.voiceIos": "Na iPhonu nabízí Safari webovým stránkám jen jeden hlas na jazyk — ten preferovaný v systému. Pro další (a lepší) hlasy: Nastavení → Zpřístupnění → Mluvený obsah → Hlasy → tvůj jazyk, stáhni si hlas ve verzi Vylepšený nebo Prémiový a nastav ho jako preferovaný. Freezmo pak čte s ním.",
|
||||
"podcast.voiceSystem": "Systémové hlasy",
|
||||
"podcast.voiceNatural": "Přirozený hlas",
|
||||
"podcast.voiceMale": "mužský",
|
||||
"podcast.voiceFemale": "ženský",
|
||||
"podcast.voiceNaturalNote": "Generuje ho klíč tvého providera — slova jsou časovaná podle audia, takže zvýraznění drží s hlasem.",
|
||||
"podcast.voiceNeedsKey": "Pro přirozené hlasy si v Nastavení → Provideři přidej klíč OpenAI nebo Groq.",
|
||||
"podcast.voiceNoSystem": "Tohle zařízení nehlásí žádné systémové hlasy. Přirozený hlas funguje všude — nebo si hlas přidej v nastavení systému.",
|
||||
"podcast.voiceGenerating": "Generuji hlas…",
|
||||
"podcast.voiceRobotic": "Jsou nainstalované jen základní systémové hlasy. Pro přirozený hlas moderátora si nějaký přidej v nastavení systému (Windows: Řeč; macOS: Mluvený obsah; Chrome má hlasy Google).",
|
||||
"podcast.tapToJump": "Klikni na kterékoli slovo a přeskočíš tam.",
|
||||
"podcast.noVoice": "Tento prohlížeč nemá nainstalovaný hlas, takže se přepis nedá přečíst nahlas. Číst ho ale můžeš dál.",
|
||||
|
|
@ -1881,6 +1903,8 @@ const cs: Record<string, string> = {
|
|||
"api.streamLost": "Připojení se během generování přerušilo. Zkontrolujte připojení a zkuste to znovu.",
|
||||
"api.almost": "Už to bude…",
|
||||
"api.noDeck": "Ze serveru nepřišla žádná sada. Zkontroluj nastavení Ollamy a zkus to znovu.",
|
||||
"api.noPodcast": "Ze serveru nepřišel žádný scénář. Zkontroluj nastavení modelu a zkus to znovu.",
|
||||
"api.noAudio": "Hlas přišel prázdný. Zkus jiný hlas nebo providera.",
|
||||
"api.incomplete": "Server skončil, aniž by poslal sadu.",
|
||||
"api.noActivities": "Vygenerovaná sada neměla žádné použitelné aktivity. Zkus to znovu.",
|
||||
"api.gradingCancelled": "Známkování bylo zrušeno.",
|
||||
|
|
@ -1895,6 +1919,10 @@ const cs: Record<string, string> = {
|
|||
"server.bodyNotJson": "Tělo požadavku nebylo platné JSON.",
|
||||
"server.invalidRequest": "Neplatný požadavek: {detail}",
|
||||
"server.providerModelNeeded": "Vyberte model pro generování v Nastavení → Poskytovatelé.",
|
||||
"server.ttsNeedsProvider": "Přirozené hlasy dodává provider — nejdřív si v Nastavení → Provideři přidej klíč.",
|
||||
"server.ttsUnsupported": "{provider} text-to-speech neumí. Umí ho OpenAI, Groq nebo vlastní OpenAI-compatible server.",
|
||||
"server.ttsFailed": "{provider} nedokázal pro tuhle část vygenerovat hlas.",
|
||||
"server.ttsTimeout": "{provider} generoval hlas příliš dlouho.",
|
||||
"server.needContent": "Přidej aspoň jednu fotku poznámek nebo vlož text, ze kterého sada vznikne.",
|
||||
"server.needAssignment":
|
||||
"Nejdřív přidej zadání — vlož jeho text, vyfoť ho nebo přidej dokument.",
|
||||
|
|
|
|||
283
lib/podcast-audio.ts
Normal file
283
lib/podcast-audio.ts
Normal file
|
|
@ -0,0 +1,283 @@
|
|||
"use client";
|
||||
|
||||
/**
|
||||
* The provider-voice playback engine: one section at a time is turned into
|
||||
* audio by the browser's provider key (/api/tts) and played through an
|
||||
* <audio> element.
|
||||
*
|
||||
* Word highlighting has no boundary events here, but it does not need to
|
||||
* guess: the clip's real duration is known, so each word gets a slice of it
|
||||
* proportional to its spoken length. That keeps the highlight honest from the
|
||||
* first word to the last, and it makes pause/seek real audio operations —
|
||||
* `currentTime` is the clock.
|
||||
*
|
||||
* Sections are cached per session (and the next one is fetched while the
|
||||
* current plays), so listening to an episode twice costs one generation per
|
||||
* section, not one per replay.
|
||||
*/
|
||||
|
||||
import { synthesizeSpeech } from "./client-api";
|
||||
import { tokenizeSections, type PodcastVoice } from "./podcast-speech";
|
||||
import type { PodcastSection } from "./schemas";
|
||||
|
||||
export type PodcastAudioEvents = {
|
||||
sections: PodcastSection[];
|
||||
providerId: string;
|
||||
voice: string;
|
||||
model?: string;
|
||||
/** Initial playback speed (0.5–2); `setRate` changes it while playing. */
|
||||
rate?: number;
|
||||
onWord: (position: PodcastVoice) => void;
|
||||
onPlay?: () => void;
|
||||
onPause?: () => void;
|
||||
onEnd: () => void;
|
||||
onError: (message: string) => void;
|
||||
/** True while a section's voice is being generated. */
|
||||
onBusy?: (busy: boolean) => void;
|
||||
};
|
||||
|
||||
type Clip = { url: string; duration: number; type: string };
|
||||
|
||||
export type PodcastAudio = {
|
||||
play: (from?: PodcastVoice) => void;
|
||||
pause: () => void;
|
||||
stop: () => void;
|
||||
playing: () => boolean;
|
||||
position: () => PodcastVoice | null;
|
||||
/** Warm the cache for a section without playing it. */
|
||||
prefetch: (index: number) => void;
|
||||
/** Change the playback speed without losing the generated clips. */
|
||||
setRate: (rate: number) => void;
|
||||
/** Drop the element and the cached clips (the player unmounted). */
|
||||
dispose: () => void;
|
||||
};
|
||||
|
||||
export function createPodcastAudio(events: PodcastAudioEvents): PodcastAudio {
|
||||
const tokens = tokenizeSections(events.sections);
|
||||
const clips = new Map<number, Clip>();
|
||||
const loading = new Map<number, Promise<Clip>>();
|
||||
const audio = new Audio();
|
||||
audio.preload = "auto";
|
||||
/* In the document (hidden): some engines only start audio for attached
|
||||
elements, and it keeps the element inspectable for debugging. */
|
||||
audio.setAttribute("data-podcast-audio", "1");
|
||||
audio.style.display = "none";
|
||||
if (typeof document !== "undefined") document.body.appendChild(audio);
|
||||
|
||||
let playing = false;
|
||||
let position: PodcastVoice | null = null;
|
||||
let generation = 0;
|
||||
let frame = 0;
|
||||
let currentSection = 0;
|
||||
let rate = Math.min(2, Math.max(0.5, events.rate ?? 1));
|
||||
audio.playbackRate = rate;
|
||||
|
||||
const wordsOf = (section: number) => tokens.filter((token) => token.section === section);
|
||||
|
||||
/** Each word's start time inside the clip, by its share of the text. */
|
||||
const startsFor = (section: number, duration: number): number[] => {
|
||||
const words = wordsOf(section);
|
||||
const weights = words.map((word) => Math.max(1, word.text.replace(/[^\p{L}\p{N}]/gu, "").length));
|
||||
const total = weights.reduce((sum, weight) => sum + weight, 0) || 1;
|
||||
const starts: number[] = [];
|
||||
let cursor = 0;
|
||||
for (const weight of weights) {
|
||||
starts.push((cursor / total) * duration);
|
||||
cursor += weight;
|
||||
}
|
||||
return starts;
|
||||
};
|
||||
|
||||
const load = (index: number): Promise<Clip> => {
|
||||
const cached = clips.get(index);
|
||||
if (cached) return Promise.resolve(cached);
|
||||
const pending = loading.get(index);
|
||||
if (pending) return pending;
|
||||
const section = events.sections[index];
|
||||
if (!section) return Promise.reject(new Error("no-section"));
|
||||
events.onBusy?.(true);
|
||||
const promise = synthesizeSpeech(
|
||||
{
|
||||
text: `${section.heading ? `${section.heading}. ` : ""}${section.text}`,
|
||||
voice: events.voice,
|
||||
model: events.model,
|
||||
providerId: events.providerId,
|
||||
},
|
||||
)
|
||||
.then(async ({ url, type }) => {
|
||||
const duration = await measure(url);
|
||||
const clip = { url, duration, type };
|
||||
clips.set(index, clip);
|
||||
loading.delete(index);
|
||||
events.onBusy?.(false);
|
||||
return clip;
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
loading.delete(index);
|
||||
events.onBusy?.(false);
|
||||
throw error;
|
||||
});
|
||||
loading.set(index, promise);
|
||||
return promise;
|
||||
};
|
||||
|
||||
/** The clip's real length, read before playing so the clock can be built. */
|
||||
const measure = (url: string): Promise<number> =>
|
||||
new Promise((resolve) => {
|
||||
const probe = new Audio();
|
||||
probe.preload = "metadata";
|
||||
const done = () => {
|
||||
const duration = Number.isFinite(probe.duration) && probe.duration > 0 ? probe.duration : 0;
|
||||
resolve(duration);
|
||||
};
|
||||
probe.onloadedmetadata = done;
|
||||
probe.onerror = () => resolve(0);
|
||||
probe.src = url;
|
||||
/* Some engines never fire the event for blob URLs without a nudge. */
|
||||
window.setTimeout(done, 2500);
|
||||
});
|
||||
|
||||
const stopClock = () => {
|
||||
if (frame) cancelAnimationFrame(frame);
|
||||
frame = 0;
|
||||
};
|
||||
|
||||
const startClock = () => {
|
||||
const mine = generation;
|
||||
const section = currentSection;
|
||||
const clip = clips.get(section);
|
||||
if (!clip) return;
|
||||
const starts = startsFor(section, clip.duration);
|
||||
const tick = () => {
|
||||
if (generation !== mine) return;
|
||||
const at = audio.currentTime;
|
||||
/* Last word whose start is behind the cursor. */
|
||||
let index = 0;
|
||||
for (let i = 0; i < starts.length; i += 1) {
|
||||
if (starts[i] <= at) index = i;
|
||||
else break;
|
||||
}
|
||||
const next = { section, word: index };
|
||||
if (!position || position.section !== section || position.word !== index) {
|
||||
position = next;
|
||||
events.onWord(next);
|
||||
}
|
||||
if (clip.duration > 0 && at >= clip.duration - 0.05) {
|
||||
playFrom({ section: section + 1, word: 0 });
|
||||
return;
|
||||
}
|
||||
frame = requestAnimationFrame(tick);
|
||||
};
|
||||
frame = requestAnimationFrame(tick);
|
||||
};
|
||||
|
||||
const playFrom = (target: PodcastVoice) => {
|
||||
const section = events.sections[target.section];
|
||||
if (!section) {
|
||||
playing = false;
|
||||
position = null;
|
||||
events.onPause?.();
|
||||
events.onEnd();
|
||||
return;
|
||||
}
|
||||
const mine = ++generation;
|
||||
stopClock();
|
||||
playing = false;
|
||||
void load(target.section)
|
||||
.then((clip) => {
|
||||
if (generation !== mine) return;
|
||||
const words = wordsOf(target.section);
|
||||
const starts = startsFor(target.section, clip.duration);
|
||||
const word = Math.min(Math.max(0, target.word), Math.max(0, words.length - 1));
|
||||
currentSection = target.section;
|
||||
position = { section: target.section, word };
|
||||
events.onWord(position);
|
||||
if (audio.src !== clip.url) audio.src = clip.url;
|
||||
audio.playbackRate = rate;
|
||||
try {
|
||||
audio.currentTime = starts[word] ?? 0;
|
||||
} catch {
|
||||
/* not seekable yet: start from the top */
|
||||
}
|
||||
audio
|
||||
.play()
|
||||
.then(() => {
|
||||
if (generation !== mine) return;
|
||||
playing = true;
|
||||
events.onPlay?.();
|
||||
startClock();
|
||||
/* The next part is generated while this one plays. */
|
||||
if (events.sections[target.section + 1]) void load(target.section + 1).catch(() => {});
|
||||
})
|
||||
.catch(() => {
|
||||
if (generation !== mine) return;
|
||||
playing = false;
|
||||
events.onError?.("blocked");
|
||||
});
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
if (generation !== mine) return;
|
||||
const message = error instanceof Error ? error.message : "error";
|
||||
events.onError?.(message);
|
||||
});
|
||||
};
|
||||
|
||||
audio.onended = () => {
|
||||
if (currentSection + 1 >= events.sections.length) {
|
||||
playing = false;
|
||||
position = null;
|
||||
events.onPause?.();
|
||||
events.onEnd();
|
||||
return;
|
||||
}
|
||||
playFrom({ section: currentSection + 1, word: 0 });
|
||||
};
|
||||
|
||||
audio.onerror = () => {
|
||||
playing = false;
|
||||
events.onError?.("error");
|
||||
};
|
||||
|
||||
return {
|
||||
play(from) {
|
||||
playFrom(from ?? position ?? { section: 0, word: 0 });
|
||||
},
|
||||
pause() {
|
||||
generation += 1;
|
||||
stopClock();
|
||||
try {
|
||||
audio.pause();
|
||||
} catch {
|
||||
/* already gone */
|
||||
}
|
||||
playing = false;
|
||||
events.onPause?.();
|
||||
},
|
||||
stop() {
|
||||
this.pause();
|
||||
position = null;
|
||||
},
|
||||
playing: () => playing,
|
||||
position: () => position,
|
||||
prefetch(index) {
|
||||
if (events.sections[index]) void load(index).catch(() => {});
|
||||
},
|
||||
setRate(next) {
|
||||
rate = Math.min(2, Math.max(0.5, next));
|
||||
audio.playbackRate = rate;
|
||||
},
|
||||
dispose() {
|
||||
generation += 1;
|
||||
stopClock();
|
||||
try {
|
||||
audio.pause();
|
||||
audio.removeAttribute("src");
|
||||
audio.remove();
|
||||
} catch {
|
||||
/* already gone */
|
||||
}
|
||||
for (const clip of clips.values()) URL.revokeObjectURL(clip.url);
|
||||
clips.clear();
|
||||
},
|
||||
};
|
||||
}
|
||||
77
lib/podcast-natural.ts
Normal file
77
lib/podcast-natural.ts
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
"use client";
|
||||
|
||||
/**
|
||||
* The provider-voice preference: which TTS provider and voice the listener
|
||||
* picked for the podcast player ("system" keeps the platform's own voices).
|
||||
* One blob in localStorage, read through a tiny accessor.
|
||||
*/
|
||||
|
||||
import { PROVIDERS, TTS_PROFILES, ttsProfile, type ProviderInfo } from "./providers";
|
||||
|
||||
const STORAGE_KEY = "freezmo.podcast-natural.v1";
|
||||
|
||||
export type NaturalPref = {
|
||||
source: "system" | "natural";
|
||||
provider: string;
|
||||
voice: string | null;
|
||||
model: string | null;
|
||||
};
|
||||
|
||||
const FALLBACK: NaturalPref = { source: "system", provider: "", voice: null, model: null };
|
||||
|
||||
export function readNaturalPref(): NaturalPref {
|
||||
if (typeof window === "undefined") return FALLBACK;
|
||||
try {
|
||||
const raw = window.localStorage.getItem(STORAGE_KEY);
|
||||
if (!raw) return FALLBACK;
|
||||
const parsed = JSON.parse(raw) as Partial<NaturalPref> | null;
|
||||
if (!parsed || typeof parsed !== "object") return FALLBACK;
|
||||
return {
|
||||
source: parsed.source === "natural" ? "natural" : "system",
|
||||
provider: typeof parsed.provider === "string" ? parsed.provider : "",
|
||||
voice: typeof parsed.voice === "string" ? parsed.voice : null,
|
||||
model: typeof parsed.model === "string" ? parsed.model : null,
|
||||
};
|
||||
} catch {
|
||||
return FALLBACK;
|
||||
}
|
||||
}
|
||||
|
||||
export function writeNaturalPref(next: Partial<NaturalPref>): NaturalPref {
|
||||
const merged = { ...readNaturalPref(), ...next };
|
||||
try {
|
||||
window.localStorage.setItem(STORAGE_KEY, JSON.stringify(merged));
|
||||
} catch {
|
||||
/* private mode */
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
/** Providers with a TTS profile that the listener actually has a key for. */
|
||||
export function ttsProvidersWithKeys(keys: Record<string, string | undefined>): string[] {
|
||||
return Object.keys(TTS_PROFILES).filter((id) => Boolean(keys[id]));
|
||||
}
|
||||
|
||||
/** The provider to use when the listener has not chosen one. */
|
||||
export function defaultTtsProvider(keys: Record<string, string | undefined>, generation?: string): string {
|
||||
const available = ttsProvidersWithKeys(keys);
|
||||
if (available.length === 0) return "";
|
||||
if (generation && available.includes(generation)) return generation;
|
||||
/* Prefer OpenAI when both are configured: its voices are the better ones. */
|
||||
if (available.includes("openai")) return "openai";
|
||||
return available[0];
|
||||
}
|
||||
|
||||
export function voiceForProvider(providerId: string, wanted: string | null): string {
|
||||
const profile = ttsProfile(providerId);
|
||||
if (!profile) return "";
|
||||
if (wanted && profile.voices.some((voice) => voice.id === wanted)) return wanted;
|
||||
return profile.voices.find((voice) => voice.male)?.id ?? profile.voices[0]?.id ?? "";
|
||||
}
|
||||
|
||||
/** Provider labels for the picker (id → label). */
|
||||
export function ttsProviderLabel(id: string): string {
|
||||
return PROVIDERS.find((provider: ProviderInfo) => provider.id === id)?.label ?? id;
|
||||
}
|
||||
|
||||
export { TTS_PROFILES };
|
||||
|
|
@ -21,6 +21,16 @@ import type { PodcastSection } from "./schemas";
|
|||
|
||||
export type PodcastVoice = { section: number; word: number };
|
||||
|
||||
/** Tidy text for the speech engine (see the note where it is used). */
|
||||
export function normaliseSpoken(raw: string): string {
|
||||
return raw
|
||||
.replace(/[\u0000-\u001f\u007f-\u009f\u200b-\u200f\ufeff]/g, " ")
|
||||
.replace(/…/g, ", ")
|
||||
.replace(/\s*—\s*/g, " — ")
|
||||
.replace(/\s+/g, " ")
|
||||
.trim();
|
||||
}
|
||||
|
||||
export type PodcastSpeechEvents = {
|
||||
sections: PodcastSection[];
|
||||
locale: string;
|
||||
|
|
@ -129,10 +139,16 @@ export function createPodcastSpeech(events: PodcastSpeechEvents): PodcastSpeech
|
|||
position = { section: target.section, word };
|
||||
|
||||
const from = sectionTokens[word].start;
|
||||
const text = section.text.slice(from);
|
||||
/* Spoken text hygiene: engines click on control characters, stray
|
||||
double spaces and unicode ellipses, and some pause oddly around them. */
|
||||
const text = normaliseSpoken(section.text.slice(from));
|
||||
const current = ++generation;
|
||||
try {
|
||||
/* Only cancel when something is actually pending: an idle cancel makes
|
||||
several engines pop. */
|
||||
if (window.speechSynthesis.speaking || window.speechSynthesis.pending) {
|
||||
window.speechSynthesis.cancel();
|
||||
}
|
||||
const utterance = new SpeechSynthesisUtterance(text);
|
||||
utterance.lang = events.locale;
|
||||
utterance.rate = events.rate;
|
||||
|
|
@ -200,7 +216,12 @@ export function createPodcastSpeech(events: PodcastSpeechEvents): PodcastSpeech
|
|||
events.onEnd();
|
||||
return;
|
||||
}
|
||||
/* A short beat between parts: speaking into the engine's tail is
|
||||
what makes the boundary pop on several platforms. */
|
||||
window.setTimeout(() => {
|
||||
if (generation !== current) return;
|
||||
speakFrom({ section: target.section + 1, word: 0 });
|
||||
}, 140);
|
||||
};
|
||||
|
||||
utterance.onerror = (event) => {
|
||||
|
|
|
|||
|
|
@ -87,6 +87,64 @@ export const PROVIDERS: ProviderInfo[] = [
|
|||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* Text-to-speech for the podcast player. Only OpenAI-compatible
|
||||
* `/audio/speech` endpoints are wired (OpenAI, Groq and any local server the
|
||||
* custom provider points at); the rest simply do not offer TTS. Voice ids come
|
||||
* from each provider's catalogue — the male ones lead because a podcast host
|
||||
* was the ask.
|
||||
*/
|
||||
export type TtsVoice = { id: string; label: string; male?: boolean };
|
||||
|
||||
export type TtsProfile = {
|
||||
/** Default model for /audio/speech. */
|
||||
model: string;
|
||||
voices: TtsVoice[];
|
||||
/** Response format to request (Groq speaks WAV, everything else MP3). */
|
||||
format: "mp3" | "wav";
|
||||
};
|
||||
|
||||
export const TTS_PROFILES: Record<string, TtsProfile> = {
|
||||
openai: {
|
||||
model: "gpt-4o-mini-tts",
|
||||
format: "mp3",
|
||||
voices: [
|
||||
{ id: "onyx", label: "Onyx", male: true },
|
||||
{ id: "echo", label: "Echo", male: true },
|
||||
{ id: "ash", label: "Ash", male: true },
|
||||
{ id: "fable", label: "Fable" },
|
||||
{ id: "nova", label: "Nova" },
|
||||
{ id: "coral", label: "Coral" },
|
||||
{ id: "shimmer", label: "Shimmer" },
|
||||
],
|
||||
},
|
||||
groq: {
|
||||
model: "playai-tts",
|
||||
format: "wav",
|
||||
voices: [
|
||||
{ id: "Fritz-PlayAI", label: "Fritz", male: true },
|
||||
{ id: "Thunder-PlayAI", label: "Thunder", male: true },
|
||||
{ id: "Mason-PlayAI", label: "Mason", male: true },
|
||||
{ id: "Celeste-PlayAI", label: "Celeste" },
|
||||
{ id: "Cheyenne-PlayAI", label: "Cheyenne" },
|
||||
],
|
||||
},
|
||||
custom: {
|
||||
model: "tts-1",
|
||||
format: "mp3",
|
||||
voices: [
|
||||
{ id: "onyx", label: "Onyx", male: true },
|
||||
{ id: "echo", label: "Echo", male: true },
|
||||
{ id: "alloy", label: "Alloy" },
|
||||
{ id: "nova", label: "Nova" },
|
||||
],
|
||||
},
|
||||
};
|
||||
|
||||
export function ttsProfile(providerId: string): TtsProfile | null {
|
||||
return TTS_PROFILES[providerId] ?? null;
|
||||
}
|
||||
|
||||
export function providerInfo(id: string): ProviderInfo | undefined {
|
||||
return PROVIDERS.find((provider) => provider.id === id);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "freezmo",
|
||||
"version": "0.7.5",
|
||||
"version": "0.7.6",
|
||||
"private": true,
|
||||
"license": "GPL-3.0-only",
|
||||
"scripts": {
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
*
|
||||
* Bumping CACHE_VERSION is how a new deploy invalidates old assets.
|
||||
*/
|
||||
const CACHE_VERSION = "freezmo-v97";
|
||||
const CACHE_VERSION = "freezmo-v98";
|
||||
const SHELL_CACHE = `${CACHE_VERSION}-shell`;
|
||||
const ASSET_CACHE = `${CACHE_VERSION}-assets`;
|
||||
|
||||
|
|
|
|||
|
|
@ -60,7 +60,13 @@ export function resolveChrome() {
|
|||
export const CHROME = resolveChrome();
|
||||
|
||||
export async function launchBrowser() {
|
||||
return chromium.launch({ executablePath: CHROME, args: ["--no-sandbox"] });
|
||||
return chromium.launch({
|
||||
executablePath: CHROME,
|
||||
/* The podcast player plays generated audio through an <audio> element;
|
||||
headless Chromium would refuse it without this flag (a real device has
|
||||
the listener's own gesture, which is the policy's real gate). */
|
||||
args: ["--no-sandbox", "--autoplay-policy=no-user-gesture-required"],
|
||||
});
|
||||
}
|
||||
|
||||
/** axe-core source, injected into the page for the contrast audit. */
|
||||
|
|
|
|||
|
|
@ -368,6 +368,91 @@ try {
|
|||
await context.close();
|
||||
}
|
||||
|
||||
/* 4b3. natural voices: the provider TTS behind the same player ------- */
|
||||
{
|
||||
const { context, page } = await makePage(browser, { width: 390, height: 844, deviceScaleFactor: 1, isMobile: true, hasTouch: true });
|
||||
/* One second of silence as a real WAV — the player measures the clip and
|
||||
times its words against it, so the header alone is not enough. */
|
||||
const wav = (() => {
|
||||
const sampleRate = 8000;
|
||||
const samples = sampleRate;
|
||||
const buffer = Buffer.alloc(44 + samples * 2);
|
||||
buffer.write("RIFF", 0);
|
||||
buffer.writeUInt32LE(36 + samples * 2, 4);
|
||||
buffer.write("WAVE", 8);
|
||||
buffer.write("fmt ", 12);
|
||||
buffer.writeUInt32LE(16, 16);
|
||||
buffer.writeUInt16LE(1, 20);
|
||||
buffer.writeUInt16LE(1, 22);
|
||||
buffer.writeUInt32LE(sampleRate, 24);
|
||||
buffer.writeUInt32LE(sampleRate * 2, 28);
|
||||
buffer.writeUInt16LE(2, 32);
|
||||
buffer.writeUInt16LE(16, 34);
|
||||
buffer.write("data", 36);
|
||||
buffer.writeUInt32LE(samples * 2, 40);
|
||||
return buffer;
|
||||
})();
|
||||
const bodies = [];
|
||||
await page.route("**/api/tts", async (route) => {
|
||||
bodies.push(route.request().postDataJSON());
|
||||
await route.fulfill({ status: 200, contentType: "audio/wav", body: wav });
|
||||
});
|
||||
await page.addInitScript(() => {
|
||||
localStorage.setItem(
|
||||
"freezmo.providers.v1",
|
||||
JSON.stringify({ keys: { openai: "sk-test" }, models: {}, customBaseUrl: "", generation: null }),
|
||||
);
|
||||
});
|
||||
await seed(page, { decks: [], theme: "dark", extra: { "freezmo.podcasts.v1": JSON.stringify([EPISODE_RECORD]) } });
|
||||
await page.goto(`${APP}/podcast/pod_local`, { waitUntil: "networkidle" });
|
||||
await page.waitForTimeout(1000);
|
||||
|
||||
await page.getByRole("button", { name: "Voice" }).click();
|
||||
await page.waitForTimeout(300);
|
||||
await page.getByRole("button", { name: "Natural voice" }).click();
|
||||
await page.waitForTimeout(400);
|
||||
const voicesShown = await page.evaluate(() =>
|
||||
[...document.querySelectorAll("button[aria-pressed]")].map((el) => el.textContent?.trim() ?? ""),
|
||||
);
|
||||
check(
|
||||
"natural: the provider's voices are offered (male first)",
|
||||
voicesShown.some((entry) => entry.includes("Onyx")) && voicesShown.some((entry) => entry.includes("male")),
|
||||
JSON.stringify(voicesShown.slice(0, 5)),
|
||||
);
|
||||
await page.getByRole("button", { name: /Onyx/ }).click();
|
||||
await page.waitForTimeout(300);
|
||||
await page.getByRole("button", { name: "Close" }).click();
|
||||
await page.waitForTimeout(200);
|
||||
|
||||
await page.getByRole("button", { name: "Play the episode" }).click();
|
||||
/* The stub clip is half a second of silence: check while it is playing. */
|
||||
await page.waitForTimeout(320);
|
||||
const first = bodies[0];
|
||||
check("natural: the voice is generated through the API", bodies.length >= 1, String(bodies.length));
|
||||
check(
|
||||
"natural: the request carries the section text and the picked voice",
|
||||
Boolean(first) && String(first.text).startsWith("Daylight.") && first.voice === "onyx",
|
||||
JSON.stringify({ voice: first?.voice, head: String(first?.text).slice(0, 24) }),
|
||||
);
|
||||
const audioState = await page.evaluate(() => {
|
||||
const elements = [...document.querySelectorAll("[data-podcast-audio]")];
|
||||
const audio = elements.at(-1);
|
||||
return audio ? { count: elements.length, paused: audio.paused, blob: audio.src.startsWith("blob:") } : "no-audio-element";
|
||||
});
|
||||
check(
|
||||
"natural: one audio element plays the clip",
|
||||
typeof audioState === "object" && audioState.blob && audioState.paused === false && audioState.count === 1,
|
||||
JSON.stringify(audioState),
|
||||
);
|
||||
check("natural: the position is remembered for later", (await page.evaluate(() => localStorage.getItem("freezmo.podcast-natural.v1")))?.includes("onyx"), "");
|
||||
/* The word highlight rides the clip's real clock. */
|
||||
const before = await page.evaluate(() => document.querySelector("[data-word-active]")?.textContent ?? "");
|
||||
await page.waitForTimeout(1200);
|
||||
const after = await page.evaluate(() => document.querySelector("[data-word-active]")?.textContent ?? "");
|
||||
check("natural: the highlight follows the audio", before !== "" && after !== "", JSON.stringify({ before, after }));
|
||||
await context.close();
|
||||
}
|
||||
|
||||
/* 4c. the transcript follows by line, and yields to a manual scroll --- */
|
||||
{
|
||||
const { context, page } = await makePage(browser, { width: 390, height: 700, deviceScaleFactor: 1, isMobile: true, hasTouch: true });
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue