Welcome again! 👋🏾

If you haven’t already, I recommend reading Part 1 of this series to get a wholistic idea of what I was trying to accomplish when I set out to build Listen Better. That post contains a lot more detail on what exactly I built and what influenced my decisions as I built it. You’re currently reading a reflection on that process, so some of the technical details will still be relevant here.

Here’s a quick refresher: Listen Better is a web app that takes in French audio, translates and explains each sentence in it, and generates a new audio file featuring AI-generated podcast hosts who discuss the translations and explanations.

In this article, I’ll discuss the cost of creating an episode and offer some thoughts on the process of building the app.

Costs, you say? Spill the beans! 👀

Okay okay, I will! As you’ll recall from Part 1, there are 3 sets of services underpinning Listen Better: transcription using Deepgram, translation and explanation using OpenAI’s GPT-5, and audio generation using Google’s Gemini 2.5 Flash TTS. Quick side note, this “Gemini 2.5 Flash TTS” name is so annoying to pronounce and type. For the rest of this post I’ll just call it “G25T”.

The pricing of those 3 services is a mix of per-minute and per-character billing, so presenting the costs in that same manner won’t be particularly insightful here. More so, there’s obviously no fixed size for audio input sourced from all over the internet, there’s no fixed length on this project’s translation and explanation outputs, and consequently no fixed duration on audio output either. Thus, such intuitive anchor points don’t really exist here. For the numbers to make sense, it’s necessary to view them in the context of a unit that’s applicable to this project. That unit will be a “daily” episode.

A daily episode is the longest episode I can generate in one day, which is determined by the rate limits of the Gemini API. Of the 3 rate limit dimensions described on that page, requests per day (RPD) is the one that always blocks my progress 🥺. My account has a limit of 100 RPD, which isn’t bad for the deployed application, but was wayyy too low during development. It was frustrating at first but I had to become stoic about it because, as the saying goes, it eezz what it eezzzz. The daily episode tends be around 90 minutes on average, which is enough to fill my French listening cup on any given day.

With all that said, what then is the cost of a daily episode? Brace yourself… are you ready… on average, it costs a whopping $1.33! 😄 Does that surprise you? I thought this project would be very pricey so I went into it with the expectation that satisfactory output, whatever it would look like, would cost around $5 a day. So I was pleasantly surprised by that amount, even if it’s still higher than I’d like. You may already be wondering how that number breaks down across the 3 services, so here you go: Deepgram transcription contributes a mercifully insignificant 2 cents 😁, GPT-5 translation + explanation throws in a manageable 21 cents 🙂, whereas Gemini TTS greedily hogs the remaining $1.10 🫠. Frankly, I wasn’t surprised that TTS was the most expensive part, though I did expect the transcription to cost a lot more. To understand why I say that, let’s explore what that money actually buys.

The deployed application generates 3 audio segments for each sentence: an intro, the main translation + explanation, and an outro. 100 RPD therefore means about 33 sentences, though I round that down to 30 to create a little buffer. This number of sentences is a bit misleading because sentences can be long or short. The application actually combines shorter sentences up to a limit of 150 chars, and breaks up longer sentences down to around 200 chars. This strategy keeps the explanation within the 1500-2500 character range as mentioned in Part 1. So the more accurate statement is that the daily episode can contain explanations for about 30 chunks of transcribed French sentences from the original audio, with each chunk containing 150-200 characters of text. In reality, this represents around 260 seconds (~4.3 minutes) of input audio, on average. Knowing that, it makes sense why the transcription costs so little, right? This project simply doesn’t consume a lot of input audio in a single daily episode. If you look at the pricing pages of both the transcription and TTS services, you’ll notice that the Speech-to-Text transcription (pre-recorded, Nova-3 multilingual) costs 4.3 cents per minute of audio input, which is nearly three times more expensive than the TTS service’s 1.5 cents per minute of audio output. Ergo, doing less of the most expensive thing remains a great way to save costs 😀.

Can it be cheaper?

Simple answer: sort of, but the quality of output will almost certainly be worse. My notion of quality is admittedly quite squishy and poorly defined. It’s closer to “I know it when I hear it” than “it hits these benchmarks”. Why? Because the output of this project really is a matter of taste. So if you’re on the hunt for a TTS model to serve your needs, your mileage may indeed vary. Anyways, the cost of TTS is the elephant in the room here, as you just saw. So I’ll focus on that.

Gemini’s Batch API

The Batch API promises a 50% cost reduction for the same quality of output if you’re willing to accept delayed results. That would’ve been terrific here, and I went as far as to build out the logic to use the batch API… only to discover that it’s not supported for the TTS model 😞. Part 1 of this series captured some of the frustrations I experienced when working with the Google model, and this was yet another one. I console myself with the knowledge that I have the code ready to go if they ever decide to support the Batch API on that model.

Other proprietary TTS models

As a category, proprietary models generally gave me better results compared to open-source models. Here’s a list of other proprietary models I considered or tested, and a brief summary of the reasons I didn’t end up choosing them:

1. Google WaveNet TTS model: Google has a number of legacy TTS models, one of which is the WaveNet model. It’s cheaper than G25T, which is promising. However, if you listen to the audio sample of this model, I suspect you’ll agree with me that it’s a bit too robotic and monotonic for this project. Cheaper? Yes. Better? Nope.

2. Gemini 2.5 Pro TTS: as the name suggests, this is G25T’s more capable sibling. Great output, sometimes noticeably better than what I got from G25T. However, for double the price, “sometimes” just wasn’t good enough. Cheaper? Nope. Better? Often, but not often enough.

3. ElevenLabs: their v3 model is particularly impressive, with comparable results to the Gemini models. Sometimes I felt this model’s output was better, other times not. However… look at the pricing page and you’ll see why I didn’t choose this option. The Creator Plan is the cheapest plan with no hard cap on available minutes. It starts at 22 cents per minute, which is 15 times more expensive than G25T! More so, you only enjoy that price for a miserly 100 minutes, which is basically just one daily episode 😂. After 100 minutes, the price rises to 30 cents per minute, so 20x more expensive! 🥵 Even though the higher priced plans have lower unit prices, those unit prices are still many multiples of G25T’s unit price, and they require a volume of spend that I was certainly not going to commit to this project. Oh well, until we meet again, ElevenLabs.

4. OpenAI models: three of them, to be precise: gpt-4o-mini-tts, TTS and TTS HD. Only gpt-4o-mini-tts has a price that competes with G25T; it costs about the same. The other 2 are more expensive. More importantly, after testing the models in their playground, I felt that G25T gave me better control over emotional expression, and ultimately better output.

5. InWorld TTS models: 2 models, TTS-1 and TTS-1-Max. The first is half the price of G25T, whereas the second costs the same. In terms of output, TTS-1-Max predictably had better output than TTS-1, but not as good as G25T. I wasn’t able to dial in the French pronunciations or emotional expression as consistently as with G25T, so for the same amount of money it just didn’t make sense to use this one. Still good models though, and you can get a sample here.

Open-Source TTS models

I actually started out looking for open-source TTS models because I initially assumed that proprietary models would be too expensive to be worth the bother. As it turns out, open-source TTS models that support French were not as readily available as I expected. For the few that I did find, I either didn’t like their speech output, or didn’t have the hardware to test them. To bridge the hardware gap I considered using an inference provider, but after seeing the number of options to explore, and factoring in the unsatisfactory results from the models that were already accessible to me, I decided to try out proprietary models at that point. As you’ve already seen, I didn’t look back.

Drawbacks of the audio processing strategy

No control over changes to the underlying model

Model providers are always updating their models, and sometimes users have no other option than to accept such changes to keep using those models. After Listen Better first went live, Google published an update to G25T that they said would bring good tidings all across the board: “enhancements” that included “better expressivity”, “precision pacing”, “seamless dialogue”, and “significant improvements” to overall audio quality. Had Christmas 🎅🏾🎁 really come early? The key part was that the update would happen in place, with no action required from me. Crucially, I couldn’t opt out and take the change at my own time. In other words, with no extra effort on my part, everything would just get better overnight. If that sounds too good to be true… yeah, it was too good to be true.

The first problem was that the distinct male and female voices I had chosen started sounding more gender neutral more often, to the point that I wouldn’t even be sure who was speaking sometimes. After a few days I could tell that this was part of a wider issue where the new model now blends the roles and voices of the speakers. A line attributed to one speaker in the dialogue script sometimes gets read by the other speaker in the generated audio. And that’s not even the strangest part: occasionally the active speaker fully transitions from one of my selected voices to the other one literally mid-speech! Voice metamorphosis 😎. This problem was magnified by my usage of speaker names in the dialogue script, because I wanted the speakers to reference themselves by name to make the dialogue sound more natural. However, when Marie speaks a line that’s meant for Clément, and calls out her own name because Clément was scripted to call her by name, the listening ear can’t help but notice the proverbial record scratch. I eventually took out the names because of how annoying this got. At least without the names it just sounds like 2 people speaking alternately 🤷🏾‍♂️.

The second problem was that the new model would occasionally skip segments of the dialogue script and sometimes add extensive periods of silence in the generated speech. The same markup tags that I had used to introduce natural-sounding pauses between sentences and words seemed to cause the speakers to pause for much longer than expected. Sometimes they would pause for tens of seconds if not minutes, and in some cases, they would stop speaking entirely without reading out the full dialogue script. I ended up adding code to perform an additional check for extended silence on each generated audio file. Whenever more than 10 seconds of silence is detected, the audio gets regenerated. That’s not an ideal solution because extra generations means more money spent. Thankfully the problem is rare enough that this additional cost remains negligible.

All that said, was the update really so bad? Nope, not at all. In fact, with hindsight I can now say the update was good. I do believe the audio quality has improved, as well the model’s adherence to the prompt and script… well, at least when the audio is complete! The real problem was that I didn’t have the option of testing the new model and refining my strategy to take advantage of its strengths in a controlled environment, before pushing it to the deployed application. In a professional setting, that could’ve been a very big problem.

Non-deterministic LLM output

The same script content does not consistently generate the same audio content. Some words get mispronounced, others get swapped for closely-related alternatives, and some are skipped entirely (although without really changing the meaning of the overall sentence). This variability is not necessarily a bad thing because deterministic output can very quickly sound boring and monotonous. However, in this particular project, any word used in the generated speech that wasn’t part of the script is a potential source of misunderstanding for me, the end listener. Thus, script adherence was essential for keeping Listen Better usable. Thankfully, G25T is already quite good at sticking closely to the original script. With a few stern instructions added to the prompt, I have been able to keep the occurrence of unintended variations to a satisfactorily low level.

Transcription inaccuracies

Deepgram can sometimes identify the wrong words in the audio. When I’ve seen this happen, it’s been with words that sound the same but are spelled differently. Normally, the surrounding context would indicate what word was most likely said, and Deepgram generally handles that really well. The real issue here is that solving the problem is challenging unless there’s a preexisting canonical transcript to compare with. Thankfully, I haven’t really needed to solve this issue because of the very high accuracy of the transcriptions. In Listen Better, I’ve only ever noticed this issue twice, as in affecting 2 words, in over 28,000 transcribed words. Yeah, no urgent need to fix that! 🙂

Was the coding process a vibe?

Using coding agents, writing the code was as doable as any typical web-based project is these days. For the most part I accepted the UI choices of the coding agents (with some refinements, of course) because UI design is not one of my strengths. It would be accurate to say that a combination of Cursor, Codex, and Claude Code built the vast majority of the app! 😄 This doesn’t mean I handed over completely in the way that the “vibe coding” term suggests. I’ve been a developer for some years so I find it difficult to accept autogenerated code without building up the confidence that I know what it’s doing and not doing. So even though most of the code was written by coding agents, I set the direction and performed course correction by constantly reviewing and refining the outputs throughout the process.

Estimating the time I spent building Listen Better is tricky. It was not a full-time commitment, and I didn’t build from a pre-defined product specification. Nevertheless, if I could squash all the time together, I would estimate that it took me a month of full-time work to go from initial exploration to the current web app that I use daily. A big chunk of that time was spent learning a few new things, most notably TTS, RSS, agentic coding workflows, and self-hosting. I was also simultaneously making product and engineering decisions, some of which came more naturally to me than others, and most of which I didn’t bother about until after I started getting results from the TTS pipeline. Essentially, I created a working TTS pipeline then bolted a UI and RSS integration on top! 😆 Knowing what I know now, if the UI, product behavior, and TTS strategy are all specified in detail in advance, I believe a developer skilled at using coding agents can recreate the app in no more than 3 days of full-time work. That includes code review, refinement and testing, and a number of admin features not visible to regular users.

What was the biggest bottleneck I faced?

Simply put, wrangling the TTS models to get my desired output. Right from my earliest explorations of the concept for Listen Better, the first and most important signal to judge was the quality of generated audio output. My ideal TTS model would need to fluidly handle multilingual speech, support multispeaker audio generation, and provide tools for tweaking the tone and energy of the speakers according to my preferences. To properly assess how well a given model fit my objectives, I needed to spend some time prompting and prodding it until I felt confident of what its “best” output could be, even if I hadn’t yet gotten that level of output. This prompting and prodding exercise involved:

• doing a lot of good old “prompt engineering”

• changing the code continuously to match model features and APIs

• exploring the effects of controlling tone and emotion on the final output

• trying different tactics to

  • ensure words in each language were pronounced appropriately

  • maintain the prosody (meaning rhythm and intonation) of speakers across multiple generation attempts when hitting character limits

  • get the most natural-sounding dialogue, either by generating each speaker’s portion independently then stitching them together, or, if available, using the model’s built-in dialogue generation capabilities

This exploration looked a bit different for each model, and there’s no standard API or feature set for TTS generation. Overall, the process was much longer than I expected going in. It left me with the feeling that we’re still very much in the earliest innings of what can broadly be called voice AI, despite the outputs already sounding so good in some cases.

Rounding up

Listen Better was borne out of a very personal desire to improve my listening comprehension in a foreign language. Has this project been enough to fill that need? No, though it’s certainly been helpful. My vocabulary has grown faster than before I started using it. More so, hearing French words and expressions repeatedly from different audio sources has been reinforcing them in my subconscious in a way that I would not have managed unless I moved to France! Or some other French-speaking region. For now, the face twitch I mentioned in Part 1 hasn’t gone away 😄. Word by word, sentence by sentence, and in combination with other learning efforts, I’m optimistic that it will.

Beyond the language learning outcome, this project reacquainted me with today’s reality that long-form AI-generated content can only be created in small chunks that add up to a whole, often seconds or at best minutes at a time. In my case I had to stick to strict character limits then combine audio segments into full episodes. Input and output limits are not going away anytime soon, so learning to effectively work with them remains necessary.

And with that, I say a big « merci ! » to you for reading this post. Until next time! 👋🏾

Questions? Feedback? Nice words, or mean ones? Feel free to reach out to @CodeWithOz on all the socials, or on LinkedIn.

Ever since I started learning a new language, I’ve learned how much I take listening comprehension for granted. In my primary language, words float into my ears, and mere moments later, meaning has been conjured in my brain. And it’s almost always subconscious. Rather remarkable, isn’t it? Transferring that skill to a new language is, shall I say, quite a challenge. My listening experience often goes something like this:

I hear a sentence, now I’m trying to make sense of it. This word means that, that word means this, so the sentence probably means… oh crap, the next sentence is already tumbling out of Speaker’s mouth! Speaker is still saying more, I’m understanding even less, I’m increasingly uncomfortable, my face is twitching weirdly, dear Speaker slow down pleeeaseeee… reality sets in: I don’t understand the language, sigh. [Me]: “In English please?”

After facing this scenario a few times, I thought it may be helpful for me to start listening to audio where each sentence is followed by a translation and an explanation of what was said. Gives my brain enough time to process what was said, provides info that supports my learning, and hopefully improves my ability to comprehend at the speed of speech.

The more I thought about it, the more it sounded like an interesting project to build for myself. Well, I did decide to build it, and will now bring you up to speed on how that went. On y va !

What to build exactly?

I wanted a pipeline that would accept an audio file containing French speech, and transform that into another audio file featuring English explanations of that French speech. This pipeline needed 4 steps:

1. Transcribe the French audio input into text

2. Translate the French text into English

3. Generate the script of a dialogue that weaves the translation into an explanation

4. Generate audio output narrating the dialogue script

For the purpose of this writeup, I’ve implemented those steps in a simple typescript project that can be executed from the command line. The code lives in this repo, and I will reference some of the files and code snippets from there.

Transcribe audio input into text

This was the easiest part of this project. Deepgram provides a high-quality Speech-to-Text service that supports my desired language, French. They have a very generous $200 starter credit, so frankly I didn’t even bother to look elsewhere. After installing the JS SDK, I could easily transcribe the audio file using a single invocation of the transcribeFile method. After looking at the response schema of the underlying endpoint, I decided that I’d likely need the list of sentences and the list of words. The code that saves that data can be found in my transcribe function. To give you a sense of the output of this step, I used an audio clip that spans the first 37 seconds of a particular YouTube video… turn the volume down a bit if you want to watch it. Here you go 🙂. The output of transcribe for that audio clip looks like this:

An auspicious start! The words array is pretty long so it’s not shown fully in that snippet. If you’re craving some JSON to brighten up your day, view the full thing here.

On that note, Step 1 is done! ✅

Translate French text into English

The most obvious candidate for this step is a dedicated service like Google Translate or DeepL. However, I chose to use an LLM because modern LLMs use structured outputs to enable an almost unbounded range of output schemas. For this project, I wanted the following extras on top of the raw translation:

• a list of the verbs used in the sentence, with their conjugated and infinitive forms

• a list of other words like nouns, adverbs, and adjectives, each with its meaning

• a list of idiomatic expressions present in the sentence, each with its literal and contextual meaning

LLMs can provide all of these in one query response, and these days they are pretty great at raw translation too. Below is the Zod schema I used to capture my desired outputs:

Equipped with this schema, I tasked OpenAI’s GPT 5 Mini with handling the translations, using a system prompt that of course reflects the data in TranslationSchema. My translate function feeds the sentence to the LLM and gets the explained translation in return. Using the first sentence in the output from transcribe, I’m happy to report that the output of translate was as expected 🙂:

That’s just one sentence though. To process all the sentences, a simple loop will suffice… buttt as you may know, LLMs generally respond on a time scale of seconds. Stacking tens or maybe hundreds of sentences for sequential translation is not very efficient. Ideally this step should only last as long as the longest individual translation. In other words, the name of the game here is, say it with me, concurrency! My translateSentences function handles that using Promise.all. If you’re really keen to improve your French vocabulary 📚, go here to see the full set of translations for all 4 sentences.

And with that, let’s move on to Step 3. 🏃🏾‍♂️

Generate dialogue script featuring translation and explanation

Look, the path to completing this step proved to be muuch less straightforward than I imagined, my oh my. Essentially, the content of the script is directly influenced by the features of the text-to-speech (TTS) service used to narrate the script. These features vary by service provider, so I had to explore various paths and possibilities in search of satisfactory output. I’ll briefly try to give you a sense of what I mean.

Emotional expressiveness emerged as an important concern because I didn’t want to get bored out of my mind while listening to the explanations. There’s a cohort of TTS services that generate a level of monotone that Lex Fridman would be proud of. Thankfully, on the other end of the spectrum, some services provide knobs to make the speech sound less robotic, but even among these services, the knobs come in different shapes and sizes! 😩

There’s also the small matter of choosing the voices of the speakers. Each voice has its own profile of expressiveness, speed of speech, and appropriate use-cases. Even when using the same voice, maintaining prosody (fancy word for rhythm and intonation) across multiple turns can be challenging, or worse, impossible.

Additionally, my specific project needed to generate both French and English snippets close together, often within the same sentence. These snippets needed to be in the same voice, with appropriate pronunciation for each language, much like you’d expect when listening to a bilingual speaker.

And last but not least, the old truism applies here too: it’s all about the dollar bills yo! 💸 The models that performed best for my needs were proprietary and not cheap 😮‍💨. Even though some of the open-source models showed promise, hosting and serving them was not as straightforward as making API calls. The cost of serving and in some cases deploying the best of the open-source models was not low enough to justify using them over the proprietary models.

At this point, esteemed reader, you’re well within your rights to wonder what service I finally settled upon. Without further ado, the illustrious award of 🏅Featured in Uche’s Obscure Project🏅 goes to none… other… than… 🥁🥁🥁

🏆🎊 Gemini 2.5 Flash TTS! 🎊🏆

It supports dialogue such that you can build NotebookLM-style conversations if you really put your mind to it. That means, among other things, it’s pretty good at emotional expressiveness. In fact, of all the options I tested, it was the best at capturing emotional cues sprinkled throughout the script. It is very multilingual and has a nice range of voices, each of which is available in all the supported languages. It is therefore well-suited to mix French and English in the same voice, with appropriate accents and pronunciations… at least most of the time 🙈. Given that it’s an LLM, it enabled me to prompt my way to (sometimes partial) success with customized instructions. In terms of cost, this model was in the middle of the pack of the options I looked at, though it still isn’t cheap 😭. I could at least console myself knowing that it gave me the best results of all the options I considered.

Alrighty, we’ve gone a little while without seeing any code. I feel like a fish flapping around in open air 🎣. Time to dive back in! 👨🏾‍💻

I suspect that you may find the details of this part of the project a bit more interesting, so I’ll walk through my approach in more depth. generateExplanationScript is the function that will generate the script:

A few things are worthy of mention. First, the function returns an array of strings, rather than a single string. That structure made it easier to generate the audio in sensible chunks, in case a sentence’s full explanation runs longer than the character limits of the Gemini TTS API. If you’ve ever had to chunk large strings for any kind of processing, you’ve probably had to decide how best to truncate the string while maintaining coherence within the chunks. This was my way of preempting that decision.

Secondly, remember earlier when I spoke about knobs for tuning emotional expressiveness? The text in square brackets are examples of that. Gemini’s TTS is capable of picking up cues embedded in the script when the cues are presented as markup tags. And, for good measure, I’ll eventually explain exactly what those tags mean in the prompt I give to the LLM. For now, just note that the [English explanation] tag is more of an instruction to the LLM, telling it to read the subsequent text in English rather than French.

Following from that, the next notable thing is my heavy usage of French linking text coupled with the need to specify that some text should be read in English. The AI-generated speech needed to sound bilingual, but I noticed a quirk in all the TTS models I tried: the longer the text goes on in the same language, the stronger the speaker’s accent becomes in that language, and consequently, the worse it pronounces words in any other language. As a result, I often generated speech that featured very badly pronounced French words, because the speaker had progressively transitioned to a strong English accent by the time the French word came around. And no, explicitly specifying the language of the speaker via API options didn’t help much. To illustrate this phenomenon, the following script starts with 4 French words, goes on for much longer in English, then inserts 4 more French words close to the end:

Cette phrase veut dire: "This sentence is a demonstration of accent drift, which can happen with text-to-speech models. When a sentence starts in French, transitions to English, and spends much more time in English than in French, the speaker's accent starts to sound almost completely English. It's quite a strange and fascinating phenomenon. I think it's because of the way LLMs work. They are essentially prediction machines that try to guess the most likely next word for a given piece of text. So, for something which is generating audio, the more English it sees close to the point at which it needs to generate the next sound, the more it predicts that the sound should be read with an English accent. After all, an English speaker speaking English should sound English, no? Conversely, when the French text is closeby, it still predicts that the English should be pronounced with a French accent. But when the French is far behind a lot of English text, then you hear something like 'donnez-moi le livre', which doesn't sound as French as it should sound."

And here’s the generated audio. Listen carefully for how “donnez-moi le livre” is pronounced towards the end:

I know, esteemed reader, that you may not know a single word of French, let alone how French words should be pronounced. That’s totally fine! 😄 Take my word for it when I say that the speaker says “donnez-moi le livre” in a rather English-y way. That is exactly what I needed to prevent. My solution was to use French for all the linking text, to embed as much French as possible in-between the English translations. Very nice for me because it ended up providing even more French practice than I had planned 😏. Here’s what this strategy looked like for the verbs:

Once again, you don’t need to understand the French 🙂. You only need to understand that the words and meanings are interpolated using the general structure “{French words} mean {English meaning}”. And importantly, the text of each section is stitched together into one string before adding that final string to the explanationParts array, in line with the coherent chunking strategy mentioned earlier. The full definition of generateExplanationScript is available here, along with the definitions of replaceSlash and formatList.

To wrap up this step, here’s the output of generateExplanationScript for the first sentence in the transcription:

Great!

Combining those strings to form a coherent script is the domain of Step 4, so let’s head there tout de suite ! 🏃🏾‍♂️

Generate audio output narrating the dialogue script

Simply put, the goal here is to generate an audio file in which the speakers narrate the script from Step 3. This process needs to happen in 2 sub-steps:

• Send the API request that will generate the audio content

• Convert the API response into the desired audio format

Send audio generation API request

Given that we’re dealing with an LLM, I’ll start with the system prompt. As mentioned earlier, my TTS system prompt describes the meaning of the markup tags I use, and explains how they should be applied to the generated speech. If you read through the full prompt, you may have noticed something curious: it repeatedly emphasizes generating audio rather than text. Why do that if it’s for a TTS model?

Well, this is one of the pleasures of dealing with the guardrails Google has placed on some of their models. Long story short, the Gemini API server has an internal check that assesses the prompt to determine if it’s trying to trick the model to generate something other than audio. If that check concludes that your prompt is attempting such trickery, your request gets blocked ⛔. That’s reasonable in principle, but in reality the check is a bit too enthusiastic. The only way my prompt could consistently pass was to distribute repeated affirmations of the output type throughout the prompt. Of course this behavior is not documented anywhere, so the way I discovered it was by innocently sending my API requests, hitting unpredictable and confusing errors that messed up my results, banging my head against the wall for hours, scouring through GitHub issues for answers, finally discovering the behavior for myself with trial and error, then receiving confirmation of my discovery a few days late. Fun stuff.

Now we can switch focus to the API options used for the request. My intermediary of choice, LangChain’s google-webauth package, exposes two important API options:

• responseModalities maps to Modality, and is used to specify that the response should be audio (yet again, right?)

• speechConfig maps to MultiSpeakerVoiceConfig, and is used to define the names of the speakers and map them to specific voices.

The full code of my generateDialogue function lives here, and if you go through it you’ll notice another curiosity:

In other words, the system prompt is being injected into the HumanMessage instead of sending it as a standalone SystemMessage. Yep, that’s the result of yet another undocumented quirk of the Gemini TTS API. Like the earlier one, I discovered this one the hard way too. Moreover, you can see that humanMessageContent contains one more affirmation that the output type should be audio not text. At this point, if you’ve gotten the impression that making this part work reliably was a lot less intuitive and a lot more time-consuming than an API call should be, you’re not far off the mark! I’m not salty though, not at all… 😒

Convert API response to audio

What actually is the response from the API endpoint? The “Technical specifications” section of this page shows “Raw 16-bit PCM audio at 24kHz, little-endian” as the output format. The better way to be sure is actually to read the values specified in the mimeType property of the response, because docs pages can of course get outdated. Nevertheless, I can confirm that both are in sync at the time of writing 😀. If you’re wondering, PCM is the raw digital representation of the audio, so it’s not immediately usable by media players. A conversion step to a well-supported audio format is necessary, and for this project I picked the MP3 format.

The steps required here are to get a buffer representing the audio bytes, confirm the PCM format details from the MIME type, then generate the MP3 file from the PCM data. In Code This Means:

Note that the defaults for sample rate and bit depth come from the output format specified by the docs. convertAudioContentToMp3 accepts options for working directory and file base name, to allow customizing the output file’s location and name. The actual conversion will be performed by pcmBufferToMp3. It’s a function that writes the PCM data to a file, then uses FFmpeg to convert that PCM file into a separate MP3 file. If you’re interested, you can see the logic of the function here.

Chunking it all together

To finally generate the audio, the script chunks needed to be combined in a way that respected the character limit of 8000 bytes.

The first question was how many characters can fit into 8000 bytes? Simply put, it depends on the character encoding used to read those bytes. For this project, I assumed that the Gemini server supports and uses UTF-8 because that’s the standard used across the web. UTF-8 uses between 1-4 bytes per character, which makes the limit potentially as high as 8000 characters or as low as 2000 characters. Accented Latin characters like those found in French (é, ç, î, etc) use 2 or more bytes, so I was pretty certain that the practical limit would be between 4000 and 8000 characters. In reality, there’s a rather low probability of encountering a French sentence where accented characters are even up to half of the letters in the sentence, so I felt confident that I’d be fine anywhere below 6000 characters.

Armed with this info, the next question became: how much of that limit was already used up by the full prompt, excluding the script text? 2352 characters, to be exact. Therefore, the prompt was less than halfway there, which was great news. However, it still wasn’t fully clear how much dialogue could fit within the remaining space. So I asked myself a different question: how long can I listen to French speech before I can no longer follow what’s being said? In other words, going back to the description in the intro, what’s the point at which my face starts twitching weirdly? 😁 After some trial and error, I found a sweet spot at 200 characters of transcribed French text. As it turns out, the generated script featuring all the explanations uses between 1500-2500 characters when starting from around 200 characters. I rounded that up to 3000 characters to cover unexpectedly long explanations. That would mean the total input text could contain as many as ~5300 characters, which was still well within my projected safe limit of 6000 characters. Excellent!

With that established, I added some code to:

• take the script chunks from generateExplanationScript

• create new combined chunks that each do not exceed 3000 characters, and

• transform those combined chunks into MP3 files using generateDialogue followed by convertAudioContentToMp3

You can find that logic here, and enjoy the final mp3 files for each of the 4 sentences here. Want a preview already? The mp3 of the third sentence sounds like this:

That’s definitely a success in my book! 🙌🏾

The Real Thing

So, everything I’ve described so far has been to explain and showcase the principle of what I wanted. How about what I actually built and use daily? Well I’m glad you asked 😀 I present to you… Listen Better! Creative name, right?

At its core, Listen Better is a podcast-style audio feed featuring translations and explanations of different pieces of French audio. I source the input audio files from various corners of the interwebs. Each “episode” features 2 bilingual AI-generated hosts who dissect each sentence in the original audio, just as described in this writeup. The feed contains the full dialogue script for each episode so I can see and read what was discussed. I also created a dedicated RSS feed so I can listen in my podcast app. Additionally, I added a Telegram integration to send myself the full translated vocabulary of each sentence in the original audio file, as generated by Step 2. If any of this sounds interesting to you, the web app is literally at your fingertips 😀. If you’d like to do a few things differently, such as provide your own French audio, customize the output, or change the languages, feel free to reach out to me so we can talk about it.

If you listen to any of the episodes, you’ll notice a few extras not captured in this writeup:

• Each piece of generated audio is stitched together sequentially to form a continuous audio file that constitutes the episode.

• Each episode contains an intro and an outro, because I came to appreciate the value of at least half-decent transitions when listening to something informative.

• The original audio clips of the translated sentences are played before and after the explanations, to allow me hear the audio at regular speed and associate that with the explanation.

These extras are handled by some additional FFmpeg manipulation and a few extra API calls, but nothing fundamentally different from what I described above.

La Fin

So, coming back to the title of this post, can AI “solve” listening comprehension? I’m sure you already know the answer 😄 it certainly can’t right now! Who knows what the future holds though… but in the meantime, Listen Better demonstrates what’s possible today, which is not bad at all, if I do say so myself.

Is That All?

For how I built the project, yes. For all my thoughts on the project, no 😀. This writeup is only the first member of a 2-part series focused on this project. Would you like to know the cost of an episode, or how well this whole pipeline performs, or whether vibe coding can build you an equivalent in next to no time? I’ll share my thoughts on those topics and more when I publish Part 2 of this series. Watch this space! 👀

Until then, thank you for taking time out of your day to read this 😊. À la prochaine !

Questions? Feedback? Nice words, or mean ones? Feel free to reach out to @CodeWithOz on all the socials, or on LinkedIn.

You’ve probably heard this principle before: “don’t put blocking operations on the main thread”.

Until recently, I’d been working only with synchronous python for the best part of 3 years. Using an outdated version of Django will do that to you… So when I finally started working with asynchronous python, I failed, predictably, to apply that principle in my code.

Thankfully I’ve seen the error of my ways. This post is an attempt to pass along (some of) my understanding. On we go! 🤓

In search of responsive endpoints

FastAPI encourages the use of async endpoints. A simple FastAPI server with a basic endpoint that processes files may look like this:

I made the mistake of thinking that this server would be able to handle a new request while the process_files endpoint function is paused on the await upload_to_s3() step. The endpoint function is async after all, and upload_to_s3 is also an async function, even though the sleep operation inside it is not. Subconsciously, I expected Python to just handle whatever was necessary to run the sync operation without blocking the main thread.

To my surprise, the server would block and not process any new requests sent within each 5s span. The logs looked like this:

The server waited for 5 seconds while sleeping and didn’t start the new request until the first one was completed, and similarly remained blocked during the second request. Why did this happen?

It’s because python’s async-await implementation uses an event loop that processes only one task at a time. So when a blocking operation is on that event loop, nothing else will run on the loop until that operation is complete. Wrapping the operation in an async function won’t change that. An async function is not necessarily a non-blocking function.

Unblocking the server… the wrong way ❌

Before I fully understood what I wrote in the paragraph above, my first thought was to simply defer the upload operation until after the client receives the server response. FastAPI’s background tasks is a good way to try to achieve that. A background task can schedule upload_to_s3 to be executed after the server responds to the client, as follows:

background_tasks.add_task accepts the function to be executed as the first argument, followed by the args and/or kwargs that the function should be invoked with. As for how to pass background_tasks into the endpoint function, simply defining a parameter with the BackgroundTasks type is enough. FastAPI does the work of invoking the function with the appropriate argument.

Repeating the test with that code shows that the blocking problem didn’t actually go away 😭.

Even though the request finished processing before the upload started, the server still remained blocked on the upload and didn’t pick up the new request until afterwards. Whuttt?

The blocking task actually prevented the event loop from picking up the next scheduled task (i.e. the new request). More so, the client that made the initial request was actually kept waiting for the server response while the server was occupied with the blocking background task. So, for all intents and purposes, nothing changed 😞.

Okay then, what’s the correct way to go about this?

Unblocking the server… the right way ✅

Either make everything async, or push the sync stuff into a separate thread.

Double down on async 😎

This route requires finding or creating a version of the blocking operation that properly frees up the event loop when the operation is not executing python code. “Properly” means yielding control back to the event loop appropriately. That can be tricky to get right, so I generally prefer to search for a widely used async implementation of an operation rather than rolling my own. A lot of libraries and packages expose both async and sync versions of their available methods, so I’ve mostly not needed to look very far. For the sleep function used in this writeup, asyncio has a drop-in replacement for that:

After testing again with a few simultaneous requests, they all got handled without the server blocking.

Success! 🎉

If you can’t find an async alternative to your blocking operation, and you’re a cool kid feeling up to the task, you can always build your own async implementation of that operation. As a token of appreciation for reading this far, I recommend this guide as a place to start 🙂.

For any number of valid reasons, the async path may not be the right one to follow in a particular context. Not a problem though!

Give up and stay sync 🤷🏾‍♂️

Python’s concurrency implementation allows the execution of synchronous functions in a separate thread. This frees up the main thread to keep the event loop running unblocked. FastAPI’s concurrency module exposes a run_in_threadpool helper for achieving this. Just like background_tasks.add_task, run_in_threadpool accepts the function to be executed as the first argument, followed by the args and/or kwargs that the function should be invoked with. In Code This Means:

Notice that upload_to_s3 is no longer an async function, and it’s executed in a separate thread using run_in_threadpool. The pre-upload operations are also now performed synchronously in prepare_for_upload_synchronously. Lastly, background_tasks.add_task is still used to ensure that the execution happens after the endpoint returns a response.

Repeating the test of quick-fire requests showed similar behavior to the fully async technique:

The goodies 🛍️ don’t stop there though. There’s an even simpler way to get this same behavior that doesn’t involve calling run_in_threadpool: just pass the regular non-async function to background_tasks.add_task. Internally it figures out if the function is async, and runs it in a thread pool if so. Convenient! 🙌🏾

A note on thread safety

Executing logic in parallel using multiple threads introduces the possibility that more than one thread will access and try to modify the same resources at almost the same time, which can create some serious problems. Personally, I favor using multiple threads in scenarios where the thread operations are sufficiently isolated for my taste.

For example, I recently used the run_in_threadpool method for performing file conversion without saving the converted bytes to the filesystem. They were uploaded straight to s3 instead. The function that did this upload had no other side-effects, and the filenames in s3 were freshly generated UUIDs. Ergo, very small chance of a name collision leading to a mishap. Make a similar assessment for your use case and take any necessary precautions. Tread carefully when working with multiple threads!

Closing thoughts

A quick recap of the essentials I learned:

• defining a function with async does not make its operations non-blocking; you must make sure the operations are actually non-blocking

• when possible, avoid mixing blocking and non-blocking operations inside the same function; use threads for synchronous blocking work if necessary

• when using FastAPI, background tasks are a good way to defer non-essential chunks of work until after an endpoint has responded to the client

Lastly, it’s important to note that this writeup focuses on scenarios in which the background tasks are small enough to be safely executed within the same server as the FastAPI process, yet long enough to make a bad experience for the client hitting the endpoint. This writeup also contains an implicit assumption that custom sequencing of background tasks is not a requirement. Some or all of these may not be true for your use case. For larger workloads or workloads that require customized or consistent scheduling of tasks (eg FIFO task queues), tools like Celery and RQ are more appropriate.

Thanks for your time!

Questions? Feedback? Nice words, or mean ones? Feel free to reach out to @CodeWithOz on all the socials, or on LinkedIn.

Do you ever worry that the machines are close to taking over from us humans? Look no further than YouTube’s machine-generated closed captions for peace of mind.

That screenshot 👆🏾 is from a video uploaded in 2011. A decade and a half later, I still found it necessary to build a tool that makes sense of the closed captions in some videos I watch. Luckily for us, YouTube’s parent company brought the gift of LLMs to the world. Let’s go through my LLM-infused attempt at improving those gnarly transcriptions.

The AI Stack ⚙️

LangGraph

LangGraph is a framework for building the latest and greatest agentic workflows, making it a natural choice here of course. At a basic level, LangGraph represents an AI agent as a graph network where nodes perform actions and edges (the connections between nodes) dictate which nodes get executed in what order. A simple example is an agent with one node and two edges that connect the single node to the start and end of the graph, as seen below:

The chatbot node can be any function, whether or not it uses an LLM within. So it’s actually perfectly fine to build a LangGraph workflow that doesn’t use LLMs directly. For instance, a dev team may happily use LangGraph to orchestrate parts of their application knowing they’ll eventually spice things up with AI ✨.

State is another important concept to keep in mind. The state object is exactly what it sounds like: a place to preserve any data that’s relevant to the behavior of the agent. What’s the quote again… “with great freedom to save anything to state comes great responsibility to manage state updates wisely” 🤔 💭… or something like that anyways. LangGraph uses the reducer pattern for state management, so nodes can independently update specific parts of agent state without touching other parts. If you are familiar with state management in popular frontend frameworks, you probably already understand how this system works.

LangChain

LangGraph’s elder sibling. LangChain comes packed with lots of utilities and prebuilt functions that implement common patterns and best practices for building with LLMs. OpenAI is the model provider of choice for this writeup so I’ll be using LangChain’s OpenAI companion package, though Anthropic, Google and many others are perfectly capable alternatives too.

Tavily

The hero message on their landing page sums it up neatly: “Connect Your Agent to the Web”. Tavily provides a highly configurable suite of tools for getting structured results from the web. As we will see shortly, web search is a critical part of this agent’s internal operations.

The Agent 🤖

To keep things manageable, the objective of this agent is to correct misspellings of real-world entities that show up in the transcripts. That’s because the names of such entities get butchered a lot in those transcripts, though other sentence elements are often captured accurately enough now. Restoring the honor of the butchered entities is a worthy task for this agent. This focus on names means that this agent will not try to correct “imitated Jamaican vacation dot” to “Hey man. How’s your Jamaican vacation going?” as seen in the earlier screenshot. Rather, the agent should detect that “Jamaican” refers to a real-world entity and focus only on correcting it if it’s misspelled.

Broadly speaking, the agent works in 3 steps:

1. Extract named entities

2. Find canonical references for the named entities using web search

3. Replace incorrect names with canonical names where appropriate

To explore each step further, let’s get a basic outline of the agent:

Nothing fancy at this stage, just placeholders for the agent and its state object.

Step 1: Extract named entities

LLMs are a great fit for this step. They have gotten pretty darn good at extracting data from raw text. Plus, they’re also very good at formatting their output according to defined structures. The output of this step needs to go into a web search query, and an ideal query will contain both the named entity and some additional context to make the web search results a lil’ bit sharper. The agent must therefore extract named entities and relevant additional context from the transcript.

To kick things off, the first node in the agent will need the transcript text and an LLM to extract the relevant data. In Code This Means (1) the AgentState needs a property for the transcript text and (2) the node function needs an LLM to perform the extraction. Updating the agent’s state schema is easy enough:

And then comes the node function:

The extractor_node function performs the extraction by passing the transcript_text piece of state into the extractor_llm. Now the node must actually go into the agent’s graph:

The extractor is the entry point of the graph, though for now it’s also the exit point because it has no other friends in the graph 🥺.

Looking back at the system prompt for the LLM, there’s no guarantee of the structure and format of the LLM’s response. That needs to change:

The prompt now stipulates that the LLM must provide the name and context for each extracted entity in JSON format. Still, even if the LLM gets the format right, it’ll return text that contains JSON. To get the desired equivalent python objects, LangChain’s with_structured_output() method will try to convert the LLM’s text response into a defined Pydantic model.

Before going further with the agent logic, it’s a good idea to test the existing code. To do that, the graph needs to be compiled into an object that can be invoked:

The agent will be triggered by invoking the graph property. LangGraph provides 2 methods for this: .invoke() and .stream(). Both of them run the agent end-to-end with the provided input. However, .invoke() runs without interruption and returns the final complete state object when the run is complete, whereas .stream() returns an iterable containing every update to the agent’s state as the agent was running. Put differently, .stream() is more appropriate when the developer wants finer-grained control over each step of the agent’s logical flow. I prefer using the asynchronous versions .ainvoke() and .astream().

So then, what input do these methods accept? The initial state of the agent! Suppose the video transcript is the following:

"Hollywood director Kristoffer Nolen has announced his next big project, rumored to be another complex sci-fi thriller. Fans of Nolen’s earlier work are eager to see if this film will rival the success of ‘Inceptshun’ or ‘Dunkrik.’"

I’ll pass it into the agent like this:

Executing the file prints the following output:

Success! The LLM extracted 3 named entities (1 director’s name and 2 movie names) from the transcript, and provided some context around them too. The “Agent response: …” line shows the final state of the agent (from using .ainvoke()), which is the same as the initial state because the extractor node didn’t update the agent’s state.

Step 2 of the agent’s workflow needs the extracted data, so the extractor node needs to save that information to the agent’s state. In LangGraph, a node updates a piece of state by returning a dictionary containing a key for that piece of state. Saving the extracted data to state is therefore as easy as returning the extracted data in a dictionary with the appropriate key.

Now invoking the agent on that same transcript yields:

extracted_entities is now present, which means Step 1 is complete! ✅

Step 2: Find canonical references for the named entities using web search

Here’s the basic idea: search the web using outputs of the extractor, then verify the canonical names from the search results using another LLM.

In steps Tavily. Using the Tavily Search module, the agent can research the entity name along with the extracted context. In Code This Means:

The research_entity function constructs a web search query by simply concatenating the entity’s name and the extracted context, and takes only the top 3 results. Now the agent needs to get the canonical name from the search results. Big wall of code incoming! ⚠️🫣

Thankfully what the code does is not actually complicated. TavilySearchResult is just a convenience type based on the response schema of Tavily’s search endpoint. get_canonical_name formats the search results into a digestible string for the verifier LLM, invokes the LLM, and returns a structured VerifiedEntity after collecting the LLM’s response. The LLM is configured to output a BaseVerifiedEntity to make the process smoother.

The 2 new functions, research_entity and get_canonical_name, will be used by the next node in the graph, but… there could be any number of NamedEntity objects coming from the extractor node. How then can the agent handle this unbounded structure?

LangGraph’s Send API to the rescue! The Send API covers scenarios in which there’s an unknown number of parallel nodes (let’s call them workers) that will flow from a given node, and allows each worker to maintain its own state while also syncing to the state of the parent agent if desired. From the perspective of DemoEnrichmentAgent, each individual worker should receive a NamedEntity, coordinate with research_entity and get_canonical_name, then return a VerifiedEntity that can be saved to the agent’s state. We’ll return to the Send API shortly. First, the logic for the worker:

get_verified_entity_worker takes the named entity from its own WorkerState, uses research_entity and get_canonical_name to verify that entity, then updates the worker state with the verified entity. As you’ve probably already noticed, the verified_entities piece of WorkerState uses a slightly unusual Annotated[list, operator.add] type. The base list type is to keep track of all the verified entities. Annotating list with operator.add tells LangGraph to merge (by appending) each new list of items to the existing list of items for that state key. Without that annotation, the verified entities will be lost because LangGraph’s default behavior is to replace the previous list of values with the incoming list of values. Definitely don’t want the verified entities getting lost 🙅🏾‍♂️.

Lest we forget, the worker state is still isolated from the agent’s state. Would be really nice to have the verified_entities list synced up to the agent state… which is surprisingly trivial to achieve: define the same key in the agent state and LangGraph will keep them in sync.

With the state synced up, the Send API comes back into the picture. For this the agent needs a function that will spawn a worker for each extracted entity:

The first argument passed to Send() is the name of the node that will handle the requested operation (i.e. the worker), and the second argument is the piece of state that will be passed to that node (i.e. the WorkerState). However, get_verified_entity_worker is not actually a node in the graph. Well then, it’s time to fix that!

The graph now features a conditional edge that connects extractor to any number of get_verified_entity_worker nodes, as many as created by spawn_workers. Here’s the updated graph representation:

Now I don’t know about you, but for all that code we just went through, I feel like the graph should definitely look fancier 😒 [shakes fist].

Oh well, the thought of seeing the output of the updated agent brings a smile back to my face 🙂. Quick comment on the graph representation: the dotted lines between extractor and get_verified_entity_worker signify the 1-to-many relationship between the two nodes.

And now for the test! The same video transcript from earlier will suffice. The logs for this operation are quite a bit longer than those from the previous step, so I’ve pasted them here for your viewing pleasure 😍. They show the agent going through the research steps concurrently and then ending with the final state object that contains the verified entities. Smile definitely restored 😁. On we go to Step 3! 🏃🏾‍♂️

Step 3: Replace incorrect names with canonical names where appropriate

There are no prizes for guessing that, once again, an LLM is a capable tool for this step. A simple deterministic "Find and Replace” operation (like a regex) may conceptually seem to do the trick here. Turns out that strategy will quickly go off the rails because there’s no guarantee that every reference to a named entity will use the exact text that the extractor produced for that entity. It’s well and good to find and replace occurrences of “Kristoffer Nolen” with “Christopher Nolan”, but that won’t catch something like “Nolen’s” where “Kristoffer” is missing, or “Kristóffer Nolen” where an accented “ó” is present. With an appropriate prompt and some examples, an LLM can be coached to handle the inevitable variability quite well.

This agent will use a modified “Find and Replace” strategy. The agent will make successive scans over the text, use an LLM to replace any incorrect references it detects for each verified entity, and review the replacement work to ensure completeness. This strategy is not at all perfect, and I will discuss avenues for optimization. More on that to come.

Right now it’s time to scaffold the strategy. For each verified entity, the agent will:

• scan the transcript text and replace any occurrences with the canonical name. The agent needs a node for this.

• review the output of the replacement to make sure it was complete. The agent also needs a node for this.

• decide when to move on to the next verified entity. The agent needs a conditional edge for this, though with different behavior from the earlier one.

And on to the code 👨🏾‍💻:

get_verified_entity_worker now connects to replacement_reviewer, which compresses the unbounded structure back into one node after all the workers have completed their tasks. The new conditional edge stipulates that a return value of continue from continue_replacement_router will push the logical flow to replace_entity, whereas a return value of end will terminate the agent’s work. There’s also a normal edge connecting replace_entity to replacement_reviewer, which means that every time continue_replacement_router returns continue, a replacement will be attempted and subsequently reviewed. Voila! A decision-making loop in the agent. Now the graph looks like this:

Finally it looks more interesting! 🤓

Let’s flesh out the new functions in reverse order, starting with continue_replacement_router. This function’s behavior is relatively straightforward: if the agent has replaced all the verified entities, the task is done. Otherwise, the process should continue. A loop counter will be helpful to know which member of the verified_entities list the agent is working on.

continue_replacement_router simply signals if the loop counter has reached the end of the list of verified entities. Once again, the operator.add annotation tells LangGraph to merge updates into the existing value of the state key using an add operation. For an int type, this just means “increment by the new value”.

Next up is replacement_reviewer_node. To review effectively, the agent needs to keep track of the reviewed transcript text. I noticed, after some trial and error, that making multiple replacement attempts for each verified entity boosted the accuracy of the replacement work. To achieve that, the agent needs to track the number of attempts it’s made for the entity it’s currently processing. All this means that AgentState gets two new properties:

And replacement_reviewer_node gets the following logic (it’s kind of a lot but not too bad):

Once again, easier than the length suggests 🙂. replacement_reviewer_node uses replacement_reviewer_llm to check if the current entity has been fully replaced in the updated text. It includes a guard that skips the review when both replacement_loop_idx and replacement_pass_count are 0, which means no replacement has even happened. If the LLM concludes that the replacement was successful, or if 2 replacement attempts have been made for the entity, replacement_loop_idx gets incremented to move the agent to the next entity on the next round of the loop, and replacement_pass_count is reset to 0 in preparation for that. replacement_reviewer_llm is configured to always return a boolean using the ReplacementReviewOutcome structure.

Okay, one more function to go. Almost there! 🤏🏾

replace_entity_node basically just needs to establish the current entity it should work on, feed that entity along with the transcript text into an LLM that will do the replacement, then update the agent’s state with the new transcript text. One more big block of code, I promise it’s the last one 🙏🏾. Here it is:

As described above, entity_replacer_llm does the replacement, with output structured using TextReplacement. This node also updates replacement_pass_count in the agent’s state, because this operation is the replacement attempt.

Whew! That was a lot 😅. With all of those changes done, it’s time to take the agent for a spin. Fresh, real transcript data will be tested in short order, but for now let’s stick to Chris Nolan’s affairs to validate all the new code. The logs of this run are even longer than the last set, so feel free to feast your eyes on them here. They show the value of updated_transcript_text in the final state object, along with other markers of the agent’s work. Ergo, success! 🙌🏾

Looking closely at the logs, some interesting things happened in this run. First, the reviewer LLM correctly flagged up that the “Kristoffer Nolen” entity had not yet been fully replaced after the first attempt, and the second attempt finished off the job. Exactly what we want to see 👏🏾. Secondly, the replacer LLM inserted TRANSCRIPT_TEXT: into the updated transcript more than once. Thankfully it had corrected itself by the end, but that kind of variability is not welcome here. Some strategies that can minimize this issue will be discussed further below.

But now… it’s time for… 🥁🥁🥁 real data!

The Results 📊

I built this agent as part of a tool to summarize videos from Fabrizio Romano’s YouTube channel. Yes, I’m a football transfer news junkie, don’t judge me. A typical video from that channel is this one, about 8 minutes long. The video’s transcript contains around 1500 words using around 8300 characters. Running the transcript through this agent yielded… a recursion limit error 😭. My fancy “agentic” loop wins the award for causing that. Small matter though, the error page offers a simple solution: specify a recursion_limit that’s as high as you need it to be when calling .invoke() or .stream(). If you’re thinking that maybe, just maybe, this is a sign of an architectural problem, your instincts are not wrong. I’ll touch on that in a bit. After increasing the recursion limit and getting a successful run of the agent, I could show you some more exciting logs, but I’ll highlight some not-so-in-depth metrics instead.

One interesting metric for the text replacement step is the number of incorrect replacements for each extracted entity. This number hovered between 0-2 for each entity, and it includes partial and accented replacements as described earlier. It also includes replacements of variations of misspellings for the same entity, for example replacing “Antract Frankfurt” with “Eintracht Frankfurt” even though the extracted name was already the correct canonical name. 2 incorrect replacements per entity is as bad as I am willing to accept for my use-case, so this result just about falls within my limits. Here’s a per-entity breakdown in the format “number of incorrect replacements / total number of occurrences to replace” for a run in which 16 entities were extracted:

1. “Benjamin Chesco” → “Benjamin Sesko” - 1/11

2. “Manchester United” → “Manchester United F.C.” - 1/12

3. “Newcastle” → “Newcastle United” - 0/8

4. “Red Bull Leipzig” → “RB Leipzig” - 1/7

5. “Nicholas Jackson” → “Nicolas Jackson” - 2/7

6. “Chelsea” → “Chelsea FC” - 0/4

7. “Darwin Nunes” → “Darwin Núñez” - 2/10

8. “Al Hilal” → “Al Hilal” - 0/7

9. “Liverpool” → “Liverpool FC” - 0/4

10. “Ritsu Doan” → “Ritsu Doan” - 2/3

11. “Eintracht Frankfurt” → “Eintracht Frankfurt” - 0/2

12. “Jack Grealish” → “Jack Grealish” - 0/9

13. “Everton” → “Everton Football Club” - 0/3

14. “Tottenham” → “Tottenham Hotspur” - 0/1

15. “Southampton” → “Southampton” - 0/1

16. “Malik Fofana” → “Malick Fofana” - 0/2

Another way to view those same numbers is to calculate “number of correct replacements / total number of occurrences to replace” expressed as a percentage, as a measure of accuracy of the text replacement step. For this same run, the average accuracy per-entity is 90.2%. That suggests, from a ridiculously small sample size 😂, that the agent can do the replacement quite well most of the time.

One more interesting number is the rate at which the review step correctly identified when the replacement was complete or incomplete. This number can be calculated as “number of correct reviews / total number of reviews performed” and expressed as a percentage, as a measure of accuracy of the review step. Across the full run the agent performed 19 reviews rather than the maximum 32 possible, because the reviewer LLM concluded that some entities did not require a second replacement attempt. 15 of those reviews were correct, meaning 78.9% accuracy. Meh, definitely room for improvement there.

And on that note, it’s time to talk about getting better results.

The Upgrades(?) 👨🏾‍🔧

General rule of thumb: optimize only after measuring, and start optimizing where the measured results are worst. Aaand right after saying that, I’m going to violate that rule just a bit 😇 because my objective here is to provide some food 🥘 for thought 🧠. So I’ll speak broadly about different optimizations I either considered or applied while building this agent for my video summarizer.

Better prompts

Improving the agent’s LLM prompts is perhaps an obvious change. In this blog post I used very simplistic prompts that were intended to (1) conserve space and (2) maintain focus on what the system should be doing rather than how well it does its job. For my tool I used much more descriptive prompts that included a number of examples for how I wanted the LLMs to respond. If you’re wondering how to get highly descriptive prompts with appropriate examples and formatting, start by using ChatGPT or your fave chatbot to do the heavy lifting after describing your needs. As an example, here’s the final prompt I used for the text replacement step. Apart from “normal” prompting, RAG is a more advanced technique to inject domain-specific context into prompts, and the idea of “context engineering” is a more wholistic way to think of this process.

Model choice

Testing different models is another way to eek out better performance. Each model has a unique profile for cost and suitability to a given task, so there’s no substitute for trying out different models to see what works best for your use-case. For example, in the verifier step I was keen to use gpt-5-nano because of its low cost compared to other high-end OpenAI models, but I found that it performed noticeably worse compared to gpt-4o-mini. Similarly, I would’ve used OpenAI’s best available reasoning models for the review step but they were too expensive for this use-case. Lesser models and other optimizations could get me my desired results.

Reduce LLM workload

Steps 1 and 3 of this agent are variants of “needle in a haystack” challenges where the LLM has to find some data in a sea of text. One can generally assume that the smaller the haystack, the easier the LLM will find the needles. To this end, the next 5 subsections discuss different ways to reduce the amount of work sent to the LLM.

Summarizing/shortening input

You may have noticed that the final prompt I provided earlier specifies that a video summary will be one of the inputs to the LLM, not the full transcript text. That’s because my tool summarizes the transcript first, then passes the summary into the agent for enrichment. That step alone cut down around 60-70% of the transcript text sent to the agent, making the process faster, cheaper, and more accurate. Of course there’s a risk of losing some context relevant to some entities, but ultimately the outcomes balanced out pretty nicely for my needs.

Chunking

When dealing with large inputs it often makes sense to work with chunks of input at a time instead of all the inputs at once. However, the videos I worked with were short enough to not need chunking for better performance. So I didn’t need to pursue this strategy.

Deterministic + fuzzy search and replace

A combination of deterministic and fuzzy searching could be a viable strategy to identify the bits of text that need to be replaced in Step 3. Depending on how well it works, this strategy could augment if not replace the modified “find and replace” strategy used by DemoEnrichmentAgent. I didn’t pursue this path seriously because, frankly, I was more interested in learning how to build an agentic workflow than in making each step the best it could possibly be. My gut feeling is that deterministic + fuzzy search and replace would still not have been enough, but that remains to be seen.

One-shot replacement

This means replacing all the entities in one attempt, rather than using the replace-review loop that caused the recursion error. Once again, the final prompt I provided earlier shows that one-shot replacement is actually the strategy I use in my tool, not the replace-review loop. But why, you may ask? Well, with a small enough amount of text and with relatively few entities to replace, that step can be done by an LLM in one go. It comes with a slight dip in accuracy, but I was willing to accept it because of the time and cost savings. Choosing to summarize the transcript before feeding it into the agent made this a viable alternative to the replace-review loop.

Caching

Saving and reusing any outputs that the agent repeatedly generates can bring some welcome benefits. For example, the canonical names are good candidates for caching because they’re unlikely to change for a given input. With a store of such data, the agent’s logic can be modified to first check the cache and use results from there if available, rather than going to the web or relying on an LLM every time. However, caching is notoriously hard to get right if you’re not careful. For this agent, I decided not to implement caching because the potential time and cost savings were not large enough to risk my debugging sanity 🙂.

Conclusion 😌

If you made it this far, you have my sincere gratitude 🙇🏾‍♂️. You deserve a gift…

The complete code shown in this post is available on GitHub.

Now go forth and build your own agents! And let me know about them too 😉.

Questions? Feedback? Nice words, or mean ones? Feel free to reach out to @CodeWithOz on all the socials, or on LinkedIn.

Caddy is an awesome piece of software that puts SSL cert management on cruise-control, provides approachable yet flexible reverse proxying, and offers a powerful and configurable HTTP server, with some extra goodies for static files. And while providing all that value, Caddy still makes server management sooo much easier and more intuitive! If you can’t tell, I’m a fan of Caddy. I recently setup a server with Caddy, so this article features the tips and unblocks I picked up along the way. Let’s dive in!

Getting Started

If you’re new to Caddy, I highly recommend Caddy’s recommendation 🤭 that you get started in the following order:

• the Getting Started guide

• the Quick-starts guides

This is one of the rare occasions when the official guides are as clear as any others you’re likely to find. When you’re done with those guides, you can then consult the reference for the API or Caddyfile, depending on what you want to do and how you want to do it with Caddy. Let’s explore an example.

Use Case: Reverse Proxies Made Easy 🤌🏾

One of the reasons I love Caddy is the ease it brings to setting up a reverse proxy. For instance, putting reverse_proxy :9000 in a Caddyfile is enough to route all traffic to the application running on port 9000. You’ll often need something more complicated than that, yet the simplicity remains. To illustrate, imagine that you own the domain yourdomain.tld. You’ve built a helpful web application and you want it live on the internet at sub.yourdomain.tld. What will you do with the main yourdomain.tld site? Maybe it’s too much effort to worry about that right now. You do know, however, that you don’t want to redirect elsewhere. So if anyone chooses to visit yourdomain.tld, you decide to just show a very boring placeholder message in plain text: “Welcome to yourdomain.tld! Full website coming soon.” How to do this? 🤔

You could (1) capture all the traffic heading to the main domain and any subdomains you want to use, (2) use your helpful web app to serve the requests intended for sub.yourdomain.tld, and (3) show your boring message for all the remaining traffic. In Code This Means the following config in a Caddyfile will serve your needs:

*.yourdomain.tld, yourdomain.tld {
    @sub host sub.yourdomain.tld
    route {
        reverse_proxy @sub localhost:port
        respond "Welcome to yourdomain.tld! Full website coming soon."
    }
}

• *.yourdomain.tld, yourdomain.tld: matches incoming requests for all the domains and subdomains that should be handled by the directives inside this site block.

• @sub host sub.yourdomain.tld: further matches the subset of requests where the hostname is sub.yourdomain.tld, and assigns them the shorthand name @sub.

• reverse_proxy @sub localhost:port: routes all traffic matched by @sub to the application running on localhost:port.

• respond "Welcome to ...": responds with the specified plain text to all other requests not matched by @sub.

• route directive allows you to override the default order in which other directives are handled. For instance, by default Caddy gives the respond directive higher priority over the reverse_proxy directive. Without using the route directive, requests to sub.yourdomain.tld will get the plain text response instead of being served by your helpful web app. Of course that’s not the behavior you want here. You can therefore use route to specify your desired order of priorities, as shown above.

So, 7 lines of code. Not bad. Let’s briefly consider doing the same thing using perhaps the most popular reverse proxy solution, NGINX.

I’ll be honest: I’ve never gotten proficient with NGINX. That’s mostly because it can be rather verbose and unwieldy, increasing the likelihood that I’ll do the wrong thing. So, I hope you’ll forgive me for consulting with ChatGPT to generate an NGINX config that’s equivalent to the Caddyfile config above. Hallucinations 👻 are not welcome here, so I ran the chat bot’s answer through this validator to identify and fix any obvious problems. Without further ado, here’s the final snippet:

server {
    listen 80;
    listen [::]:80;
    server_name *.yourdomain.tld yourdomain.tld;

    # Redirect HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name *.yourdomain.tld yourdomain.tld;

    # SSL configuration
    ssl_certificate /path/to/your/certificate.crt;
    ssl_certificate_key /path/to/your/private.key;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    # Rule for sub.yourdomain.tld
    if ($host = sub.yourdomain.tld) {
        proxy_pass http://localhost:port;
        break;
    }

    # Default response for other subdomains and yourdomain.tld
    location / {
        return 200 "Welcome to yourdomain.tld! Full website coming soon.";
        default_type text/plain;
    }
}

😳😲😱

Bear in mind that Caddy provides automatic SSL out of the box. So the extra NGINX settings for SSL are actually required to match the absent SSL settings in the caddy config.

Well... easy choice for me. I prefer the Caddy way.

Going Live

As you may know, it’s pretty easy to do the wrong thing when manually setting up your live environment. So, I highly recommend reading through Digital Ocean’s guide to deploying Caddy for a live website, combined with the following troubleshooting notes.

Memory Requirements

Do you want to build the Caddy binary yourself? I might have some bad news for you: you need about 2GB of RAM for the build process to succeed. If you’re using a small DO droplet like the one I used, that’s definitely bad news because you don’t have that much RAM. Well then… is there good news? Yep 🙂‍↕️ you can build the binary on your computer using Go’s cross-platform compilation feature, as mentioned here. Then copy the binary to your server using rsync -avz /path/to/binary_file user@destination:/path/to/destination. user@destination represents your SSH username and server IP address.

Does building from source sound too stressful for you? Fear not — you can still download a prebuilt binary and start serving right away!

Uniform Firewalls

At the step for configuring the firewalls on your instance (i.e. sudo ufw allow …), avoid using ufw if you previously configured your firewall rules in a different place, such as in the DO dashboard. Instead, you should go back to that place and add 2 inbound rules for ports 80 and 443. Those ports correspond to HTTP and HTTPS, respectively. Here’s a screenshot of what this looks like in DO’s dashboard:

DNS Awareness

In your domain host’s DNS settings, you need to add CNAME alias records that define the subdomains you want Caddy to control. This step is critical. Without it, the subdomain traffic will never even reach your server instance for Caddy to handle as you desire. Setting up only the main A record for your domain is not sufficient.

Location, Location, Location!

Does your systemd setup fail when you try to enable the Caddy service with systemctl? If you’re using a Caddyfile, check whether you saved the file at the location described in the ExecStart and ExecReload lines of the general caddy.service file. At the time of writing, both lines default to /etc/caddy/Caddyfile. So if your Caddyfile is located elsewhere, you need to edit the caddy.service file on your system to point to the correct location.

Logging 🪵

You probably don’t need me to tell you that you should set up logging so you can track what Caddy is doing. Caddy’s logging philosophy is quite powerful, but you may not be familiar with it. Thankfully they have an explainer you can read through. Setting up logging is easy enough though:

• create the folder /var/log/caddy (use sudo if necessary)

• create the access.log file inside that folder

• give Caddy full control of the folder: sudo chown -R caddy:caddy /var/log/caddy

  • this ensures Caddy can write to the log file without permission issues

• add a log directive to your Caddyfile and specify access.log as the output file, like this:

    log {
        output file /var/log/caddy/access.log
    }
  

Logging setup complete ✅.

TLS Configuration 🤝

Depending on your cloud host provider, when starting Caddy you may get an error like the following:

parsing caddyfile tokens for 'tls': getting module named 'dns.providers.digitalocean': module not registered: dns.providers.digitalocean

If so, you need to confirm that your Caddy binary actually contains the necessary TLS module. To do that, run this command: caddy list-modules | grep -i <name-of-your-tls-plugin-provider>. If that command doesn’t find the TLS plugin provider, then the binary doesn’t include it. This happened to me when I downloaded a prebuilt Caddy binary from Caddy’s downloads page using curl on the server instance. For some reason, that command didn’t get the correct binary in that environment. My solution: download the binary to my computer via the browser, then copy the file to my server using the rsync command mentioned earlier. Easy peasy 😊. Of course if you built the binary yourself then you instead need to rebuild it with the necessary plugin included.

Other Cloud Providers

If you’re deploying to a server on a cloud host provider that’s not Digital Ocean, the setup is mostly the same provided your server is running Ubuntu linux. However, behind the scenes DO uses some juju for authentication and cert management. So the automatic TLS step must plug into that juju when hosting on DO. What does this mean for you? Simply put, you may be able to skip the automatic TLS step entirely if Caddy’s default TLS module works smoothly with your provider. If not, you will need to use a different and appropriate TLS plugin for that step. Don’t get too worried though. The TLS config steps may still be very similar because you generally want to achieve these 4 things:

• build (or download) Caddy with the TLS plugin for your cloud host

• get an auth token with permission to interact with your cloud host’s SSL juju

  • an account token with general read/write access should work, though you may want to limit the token’s scope to fit your needs

• set that token as an environment variable for the caddy start command

  • just like in the Digital Ocean example, you can update the Environment key in your caddy.service file with your token, in the form Environment=CLOUD_HOST_AUTH_TOKEN=your_token_here

• use that environment variable for the tls directive in your Caddyfile, like this:

    tls {
        dns <tls_plugin_name> {env.CLOUD_HOST_AUTH_TOKEN}
    }
  

You can then restart the caddy service and voila! Automatically renewing HTTPS!

Conclusion

Don’t you just love it when powerful software remains as friendly to use as it is capable? With Caddy you can derive great value right from the moment you start your local server, up until you’re configuring load balancers for a busy web app. There’s a range of use cases to consider, and an entire other half of Caddy (the API) that I didn’t even discuss! I urge you to explore the documentation to see if any of it can work for you.

Questions? Feedback? Nice words, or mean ones? Feel free to reach out to @CodeWithOz on all the socials, or on LinkedIn.

Suppose you want to start a simple postgres db instance on your computer (or a server). You could go through the manual installation, port configuration, user and role management, blah, blah, blah… or you could just use a postgres docker image. Let’s use a postgres docker image 🙂. You probably don’t want to lose all your db data if/when the docker container stops. For that you need a docker volume to persist the data in a location outside the docker container. In Code This Means you should run the following shell command:

docker volume create postgres

Creates the volume, names it postgres, done. Straightforward enough. Now you need to actually start the docker container and your postgres db instance. You should give the container a name, obviously. You should specify that it uses that volume you just created. You should specify that the container should be detached from your terminal, because you will close down that terminal sooner or later. That’s only for the container, how about for postgres? You should specify some basic values: username, password, database name, and port. You should specify the postgres image you actually want to run. Translating allll that into docker cli options, we get this shell command:

docker run -d \
  --name postgres \
  -v postgres:/var/lib/postgresql/data \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=my_db \
  -p 5435:5432 \
  postgres:13.4

Explaining very quickly:

• -d says detach from the terminal

• -v says the location in the container whose data should be persisted to the postgres volume you created earlier

• -e specifies environment variables inside the container

• -p specifies the host computer’s port (left side of :) that should be mapped to the container’s port (right side of :)

Now, here’s a docker-compose.yml file that combines the 2 commands you’ve used so far:

version: '3'

services:
  postgres:
    image: 'postgres:13.4'
    volumes:
      - 'postgres:/var/lib/postgresql/data'
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: my_db
    ports:
      - '5435:5432'

volumes:
  postgres:

I don’t even have to translate anything! Except services, maybe? You might have expected containers instead. Think of using Docker for setting up microservices 😉. It’s “services” because any particular service could be backed by multiple underlying containers. Composability, yay! To start your container, you already know what to do:

docker compose up -d

The -d option is still necessary to detach the container from your terminal.

And should you ever need to stop the container, docker compose down does that for you. If you’re determined to remove all traces of your activities 👻, add the -v option to delete your container and all its associated resources.

In light of all that, I’d explain the value prop of Docker Compose like this:

• define all your docker cli instructions and options in one place,

• in language that is more friendly and approachable,

• with the ability to define multiple containers and services together, and

• the ease of starting/stopping/managing them together and (roughly) in sync.

Now I (and perhaps you too) can appreciate why Docker Compose is so valuable for managing both simple and complex server infrastructure.

Questions? Feedback? Nice words, or mean ones? Feel free to reach out to @CodeWithOz on all the socials, or on LinkedIn.

2022 has been a year in which I've read a lot more books than usual, many thanks to the wonders of listening to audiobooks at 2x speed. I've wanted a way to catalogue these books so I thought I should build my own virtual bookshelf, because obviously there are no apps or websites that do this already, right? More so, I suspected that building a book shelf would be a good exercise in learning and practicing some CSS tricks that were not yet familiar to me. So without further ado, here's how it went! 📚🤓

Starting Point

My book shelf is modeled after the older design of iBooks as shown in this image:

Source: Cult of Mac

Each layer of the shelf is essentially a rectangular cuboid, so I started with this implementation of a cube and extended it as described in the rest of this article. Here's a copy of the finalized code in that article, which was my starting point:

Sub goal 1: stretch the cube into a rectangular cuboid

To do this, I started by creating 3 css variables to represent the height, width, and depth of the cube, and I substituted them into the existing code. To understand which variable should go where, I thought about the transforms that were applied to each of the faces from their initial starting position (upright facing the screen). Using the bottom face as an example, it had transform: translateY(100px) rotateX(-90deg);, which meant it first got pushed vertically downwards by half the height of the cube then rotated to face downwards. To reflect this I needed to change its translateY(100px) to translateY(calc(var(--height) * 0.5)). More so, due to the rotation, the edge that was previously the height had become the depth. So I updated the height property by setting it to the --depth variable. Reasoning similarly for the other faces, I ended up with the following changes:

:root {
  --height: 200px;
  --width: 200px;
  --depth: 200px;
}

.container {
  width: var(--width);
  height: var(--height);
  ...
}

.cuboid {
  width: var(--width);
  height: var(--height);
  ...
}

.cuboid__face {
  width: var(--width);
  height: var(--height);
  ...
}

.cuboid__face--front {
  transform: translateZ(calc(var(--depth) * 0.5));
}

.cuboid__face--back {
  transform: translateZ(calc(var(--depth) * -0.5)) rotateY(180deg);
}

.cuboid__face--left {
  transform: translateX(calc(var(--width) * -0.5)) rotateY(-90deg);
  width: var(--depth);
}

.cuboid__face--right {
  transform: translateX(calc(var(--width) * 0.5)) rotateY(90deg);
  width: var(--depth);
}

.cuboid__face--top {
  transform: translateY(calc(var(--height) * -0.5)) rotateX(90deg);
  height: var(--depth);
}

.cuboid__face--bottom {
  transform: translateY(calc(var(--height) * 0.5)) rotateX(-90deg);
  height: var(--depth);
}

At this point the cube still looked the same, but theoretically I could stretch it into a cuboid by simply setting the --width variable to a value that's higher than that of the --height variable. To test this, I used a value of 90% for --width, which made the cube look like this:

A rectangular cuboid! 🥳

Notice, however, that the left and right faces didn't quite adjust as expected. I initially tried to solve this, but after some head scratching I recognized that I only really needed the demarcations of the faces. In other words, if I had the front, back, top and bottom faces, I would be able to implicitly demarcate the left and right faces. Therefore, I could actually remove the side faces entirely, leaving the cuboid looking like this:

Revisiting the iBooks screenshot above, the shelf fills up the width of the viewport, so I needed to set the --width variable to 100%. Doing that made the cuboid look like this:

Something was clearly still off at the left and right edges of the viewport - the front face seemed to be overflowing the container. My first thought was that I needed to do some trigonometry to figure out how many pixels should be subtracted from the cuboid's width in order to constrain the front face to the viewport's width. Thankfully there proved to be a far less complicated solution once I remembered that the initial cube was made by moving all the faces, particularly the front and back faces, away from their starting position. Put differently, the 100% value is the width of the initial state of each face, which meant that the front face overflowed its container because it was "pulled forward" by half the cuboid's depth (recall that at this point the front face had transform: translateZ(calc(var(--depth) * 0.5));). To further illustrate this, I added another cuboid face that had no transforms so that it would occupy the initial position of the faces. You can see it in red in this screenshot:

That untransformed red face filled exactly the width of the container as the 100% value would suggest. Thus the solution became clear: the front face needed to be kept at the initial position and all the other faces needed to be "pushed back" based on the depth of the cuboid. Returning to the example of the bottom face, its transform at this point was transform: translateY(calc(var(--height) * 0.5)) rotateX(-90deg);, which meant it first got pushed vertically downwards by half the height of the cube then rotated to face downwards. The first step should actually be to push the face backwards by half the depth of the cuboid so that its final position would span from the bottom edge of the front face to the bottom edge of the back face after the translateY and rotateX are applied. In Code This Means a translateZ(calc(var(--depth) * -0.5)) was needed at the beginning of the existing transform. Reasoning similarly for the other faces led to these changes:

.cuboid__face--front {
}

.cuboid__face--back {
  transform: translateZ(calc(var(--depth) * -1)) rotateY(180deg);
}

.cuboid__face--top {
  transform: translateZ(calc(var(--depth) * -0.5)) translateY(calc(var(--height) * -0.5)) rotateX(90deg);
  height: var(--depth);
}

.cuboid__face--bottom {
  transform: translateZ(calc(var(--depth) * -0.5)) translateY(calc(var(--height) * 0.5)) rotateX(-90deg);
  height: var(--depth);
}

Notice that I removed the transform entirely from the .cuboid__face--front because the front face needed to be kept in its starting position as mentioned above. After these changes the cuboid looked like this:

Much better! 👌🏾

Sub goal 2: complete the shelf appearance

First, in the iBooks screenshot there's actually no top face. The flat horizontal base of the upper shelf covers what would be the top face. I chose to remove the top face entirely and extend the height of the back face such that it would hit the topmost part of the shelf container. The additional height didn't need to be precise, it just needed to be enough to cover the distance between the top edge of the back face and the top of the shelf container. 3rem proved to be sufficient:

.cuboid__face--bottom {
  --extra-height: 3rem;
  transform: translateZ(calc(var(--depth) * -0.5)) translateY(calc(var(--height) * 0.5)) rotateX(-90deg);
  height: calc(var(--depth) + var(--extra-height));
}

As a result of that change the cuboid looked like this:

The added height actually caused the face to grow downward instead of upward! 🤔 Why was that? It's all about the transforms baby! At this point the back face had a rotateY(180deg) in its transform, which meant that its top edge was facing downward and vice versa for its bottom edge. So the extra height got added at the top, but with the top edge facing downward. I rectified this easily by prepending a corresponding negative translateY to push the face down initially, knowing that the downward push would eventually turn upward because of the rotateY(180deg). In Code This Means:

.cuboid__face--back {
  --extra-height: 3rem;
  transform: translateY(calc(var(--extra-height) * -1)) translateZ(calc(var(--depth) * -1)) rotateY(180deg);
  height: calc(var(--height) + var(--extra-height));
}

Now the cuboid looks like this:

I then applied overflow: hidden on the shelf container to trim off the overflow at the top, after which I also needed to remove the border: 2px solid black; from the faces to make sure the alignment of the edges remained intact. The outcome was this:

Getting closer! Time to flesh out some of the finer details. First I added the thickness of the shelf floor with a simple div below the .container div, and gave it a slight height. Next I updated the background colors to use shades of brown that look closer to wood. I then added box shadows to give a bit more depth to the look of the shelf. Lastly I gave the front face a transparent background so that it wouldn't obscure the faces behind it, and I removed the text in the .cuboid__face divs. These changes left the cuboid looking like this:

Almost there!

Sub goal 3: adding books to the shelf

I needed to achieve two main objectives: ensure that the books overlay the back and bottom faces of the cube, and position them such that they would appear to be sitting on the floor of the shelf. The first objective was achieved by creating a .books-container div whose height was equal to the height of the cuboid, and giving it position: relative to ensure that it overlaid the cuboid container (because the .container was in its own layer due to having position: absolute). The second objective was achieved by adding a row of divs, each representing a book, within the .books-container, and adding a little padding-bottom to the .books-container to simulate the books standing in the middle of the shelf floor as in the iBooks screenshot. After these changes the shelf looked like this:

🙌🏾 🙌🏾

One last detail: the shadow at the top of the shelf that simulates the effect of light being blocked off by the "ceiling" of the shelf. I added that using a box shadow, leaving the shelf looking like this:

😍 ✅

Putting it all together, the code thus far is viewable in this codepen.

Conclusion

This was a fun little side project through which I learned about the perspective property and learned how to use and think of different CSS transforms (I sliced the air with my hands quite a lot because order of transforms is really important). One possible improvement would be to angle the top shadow on the left and right faces to give a more realistic depiction of how the shadows fall, as seen in the iBooks screenshot. I tried to achieve that with pseudoelements but didn't succeed in making it look realistic enough. The final live bookshelf showing my books is available at halfbaked.ucheoz.tech so feel free to have a look and let me know what you think. You can always reach me at @cinexa7254 on twitter and you can connect with me on LinkedIn.

Integration tests are used to test interrelated parts of an app. Some typical candidates for integration tests are code that interacts with a database and code that relies upon different standalone modules within a codebase. Often these tests will require you to know and/or control the output and side-effects of some of the interrelated modules, and this is where stubbing comes in. Very briefly, stubs allow you to create an alternate implementation of a function, with the added ability to track useful information such as when that function is executed, how many times it has been executed, and what arguments it was executed with. Therefore, stubs give you the ability to define how those interrelated modules will behave during your tests. So what do you do when you want to test a function that relies on values defined in other scripts? You could use one of the 2 packages I learned about this week: proxyquire and esmock.

proxyquire

The project's README summarizes its functionality succinctly:

Proxies nodejs's require in order to make overriding dependencies during testing easy while staying totally unobtrusive

In other words, proxyquire provides a way to require() modules and specify which of their imports are replaced by the custom functionality you have defined in your tests. I won't repeat basic code samples here because the README provides good examples. One feature I found particularly helpful is that proxyquire detects which of the require()d values are not stubbed, and ensures that the file continues to use the original implementation for those values. And even better, it gives you the flexibility to disable this behavior via the "call thru" option. Pretty neat!

Beware though, you will need to supply your stubbed dependencies using the exact same path that the parent file uses to import them. To illustrate this, consider a project with file structure as follows:

📦ballonDor
 📂modules
   - 📜helpers.js
   - 📜action.js
 📂test
   - 📜action.spec.js

The scripts have the following contents:

// modules/helpers.js
module.exports { predictWinner: () => Math.random() < 0.5 ? 'player1' : 'player2' };

// modules/action.js
const { predictWinner } = require('./helpers');

const playerNamesMap = new Map([
  ['player1', 'Kylian Mbappe'],
  ['player2', 'Erling Haaland'],
  ['player3', 'Karim Benzema']
]);
const getWinnerName = () => playerNamesMap.get(predictWinner());

module.exports = { getWinnerName };

We want to test that the getWinnerName function gets the correct name, so we do this by supplying a value ("player3") that it would normally not receive. We can see that predictWinner only returns "player1" or "player2", so we need a custom implementation of predictWinner that will return "player3". The path to the stubbed import should NOT be relative to the test file, but instead should be the exact same as the path used in the module file. In Code This Means that the correct way to stub the import would be as follows:

// test/action.spec.js
const { expect } = require('chai');
const sinon = require('sinon');

describe(`getWinnerName`, function () {
  it(`picks the correct name`, function() {
    const customPredictWinner = sinon.stub().returns('player3');
    const { getWinnerName } = proxyquire('../modules/action', {
      // "./helpers" is the same path used in modules/action.js
      './helpers': { predictWinner: customPredictWinner },
    });
    expect(getWinnerName()).to.equal('Karim Benzema');
    expect(customPredictWinner.called).to.be.true;
  });
});

Notice that the first argument of proxyquire is the correct path to the module being imported relative to the test file, but the customPredictWinner stub is supplied using the exact same path that's present in modules/action.js.

Unfortunately, proxyquire doesn't support ES module imports, which brings us to the second package: esmock.

esmock

esmock is very similar to proxyquire, providing essentially the same functionality but with a focus on environments where ES modules are imported using the import keyword rather than require. Its README gives a similarly straightforward description:

esmock provides native ESM import mocking for unit tests.

esmock also covers modules loaded with dynamic import()s by using a slightly different syntax, esmock.p('pathToDynamicallyImportedModule'). One potential gotcha with esmock is that it is always asynchronous, so all the assertions that rely on esmock imports will need to await the import statements and therefore be contained within async functions. Probably not a big deal in most scenarios, but something to be aware of nevertheless.

Conclusion

Both proxyquire and esmock do the same job quite well: they give you an efficient and easy way to control the behavior of imported dependencies in your tests. Before I discovered these options, I would use the much more tedious strategy of globally stubbing any imports whose behavior I wanted to control, and then resetting to the original behavior when necessary. With proxyquire and esmock, I can now do the opposite by preserving the original behavior except when I something custom, which is quite nice.

Happy learning!

I recently had the unenviable task of refactoring my employer's main frontend codebase to use ES6 modules. The codebase is reasonably large, with more than 100k lines of JavaScript code across more than 30 files. The files were not modular, meaning they defined their own variables and functions in the global scope, where logic in other files would expect to find those variables and functions... If you're cringing at that, yes that's one of the reasons why I had this task, to finally do away with the problems caused by that system. As you can probably imagine, the developer experience wasn't great with this codebase, and we had no way of removing inactive crufty code from the final build. This meant our builds were larger than they needed to be, and we couldn't significantly and efficiently reduce load time for the end user. So, there I stood, with the weight of these problems on my shoulders. esbuild was our bundler of choice, particularly because of its speed - that animation on its landing page is pretty convincing.

Lessons Learned

Thankfully I finished off the task conclusively, and the rest of this article is a reflection on the lessons I learned in the process.

1: Be thoughtful of when to import modules

Dynamic import() is a powerful way to defer loading scripts until they're needed, so it's important to take advantage of this. Top-level imports are perfect if they're absolutely needed when your bundle is parsed and executed, but with a bit more thought you may find that most of your imports are actually not strictly needed as top-level imports. A good bundler with tree-shaking will isolate the modules that are import()ed, thereby excluding them from your main js bundle and reducing the time required to parse and execute said bundle. Of course your code can get very messy if you're import()ing a lot of little functions, so some balance is needed. A good policy is to group together and import() modules that are limited to specific parts of your web app. In my case I used import() for modules that were specific to certain pages.

2: Beware of circular dependencies

Circular dependencies can be introduced very easily due to the reusable nature of modules, so there is value in being mindful of when exactly functions are executed. Depending on how your files are organized, it may be almost impossible to avoid some files importing each other. Thankfully esbuild is quite good at finding and ordering imported modules to minimize conflicts, and I imagine other bundlers are similarly effective in this regard. Nevertheless, for the gnarly problems that persist, I found it helpful to move the affected logic out of the module's global scope and into functions that are executed when necessary. In Code This Means:

// module1.js
import { objectOfInterest } from './module2';

export const anInterestingString = 'an interesting string';

export function someFunc() {
  const { prop } = objectOfInterest;
  return `value of "prop": ${prop}`;
}

someFunc();

// module2.js
import { anInterestingString } from './module1';

export const objectOfInterest = {
  foo: 'bar',
  prop: `check this out: ${anInterestingString}`
};

can be changed to:

// module1.js
export function someFunc() {
-  const { prop } = objectOfInterest;
-  return `value of "prop": ${prop}`;
+  const { getPropVal } = objectOfInterest;
+  return `value of "prop": ${getPropVal()}`;
}

// module2.js
export const objectOfInterest = {
  foo: 'bar',
-  prop: `check this out: ${anInterestingString}`
+  getPropVal: () => `check this out: ${anInterestingString}`
};

That is a contrived and simplified example that mimics an actual situation I faced. The idea is that objectOfInterest was being initialized with the value of anInterestingString from the other module, so the value of anInterestingString needed to be known when objectOfInterest was defined. Both modules depend on each other, hence the possibility of errors due to undefined/uninitialized values. However, by switching the definition of objectOfInterest to use a function that returns the desired string, the value of anInterestingString does not need to be known when objectOfInterest is defined, hence removing the possibility of conflicts.

3: Choose between minification and sourcemaps for better debugging

Consider disabling minification if sourcemaps worsen your debugging experience. A sourcemap allows the browser to reconstruct and present the original file in the browser's developer tools, but for different reasons you may be unsatisfied with the reconstructed output. In my case the reconstructed files did not get the same names of variables and functions as the original files, so I was essentially only getting beautified versions of the minified files. This made debugging quite difficult as you can imagine. Disabling minification gave me exactly what I needed because the variable names remained almost exactly the same, allowing me to traverse the code as easily as I would in my code editor. Keep in mind, however, that minification helps significantly reduce the size of your final bundle and also obfuscates your source code, so you probably only want to disable it for debug builds rather than production builds.

4: Preserve nested folders by putting a non-nested entrypoint first

If it is important to preserve your folder structure in the final build folder, then you should consider ordering your entrypoints such that those in the root of the project come before those in nested folders. This may be specific to esbuild because it seems like an implementation quirk, but I could be wrong. Consider a folder structure like this:

- src
  |
  - index.js
  - pages
    |
    - shop.js
  - funcs.js
  - helpers
    |
    - utils.js

index.js and pages/shop.js are entrypoints. esbuild accepts entry points as an array of file paths, and I found that using ['pages/shop.js', 'index.js'] caused the pages folder to not be recreated in the final build folder. Instead, the equivalent of shop.js was placed at the root of the build folder. In other words, the build folder looked like this:

- build
  |
  - index.js
  - shop.js
  ...

When I changed the order of entrypoints to ['index.js', 'pages/shop.js'], the original folder structure was maintained:

- build
  |
  - index.js
  - pages
    |
    - shop.js
  ...

Conclusion

ES modules are a great way to enforce modularity in a code base, and they allow bundlers to give you nice features like tree-shaking. Most, if not all, modern frameworks already revolve around using ES modules, so you may not even need to make a decision at this point. However if you're starting a project from scratch, I highly recommend that you organize your files using ES modules, even if you do not immediately plan on bundling your js. Modularity comes with other important benefits such as code isolation and often better readability, and ES modules provide us with a unified standard module system compared to some of the differing and sometimes incompatible module systems of the past. Don't hesitate to take advantage of this awesome feature!

Feel free to reach out to ask any questions, correct errors, or just say hi to me @cinexa7254 on Twitter. Thanks for your time!

function updateTransform(element) {
  element.style.transform = new DOMMatrix([val1, val2, val3, val4, val5, val6]);
}

You may be wondering how this matches up to the css transform functions that you already know, which involve syntax like transform: translate(10px, 50px) scale(0.9). The answer is that the familiar syntax is represented by a special matrix under the hood, so using DOMMatrix is our way of accessing and manipulating that matrix directly.

A simple example is that for an element with those same transforms I used above (translate(10px, 50px) scale(0.9)), the corresponding DOMMatrix looks like this:

{
    "a": 0.9,
    "b": 0,
    "c": 0,
    "d": 0.9,
    "e": 10,
    "f": 50,
    "m11": 0.9,
    "m12": 0,
    "m13": 0,
    "m14": 0,
    "m21": 0,
    "m22": 0.9,
    "m23": 0,
    "m24": 0,
    "m31": 0,
    "m32": 0,
    "m33": 1,
    "m34": 0,
    "m41": 10,
    "m42": 50,
    "m43": 0,
    "m44": 1,
    "is2D": true,
    "isIdentity": false
}

... 👀 that's complicated!

Despite how confusing that may look, there are 2 main takeaways I want you to have:

• there are 2 types of matrices, 2d and 3d. 2d requires 6 values to form the matrix, whereas 3d requires 16 values. 2d can represent simpler transforms (like my example above) and is represented by the properties from a-f in the object shown above, whereas 3d can represent more complex transforms and is represented by the properties from m11 to m44.
• you can directly change these values to update the transforms applied to your DOM nodes! This was the light bulb 💡 moment for me. I was trying to update the transformX and transformY of an element without using ugly regexes like /translate\(\d+,\d+\)/ (yuck! 🤮) because the element had its transform set by an external stylesheet.

My solution built on this second takeaway, particularly because you can construct a DOMMatrix using the existing transform of an element. In Code This Means:

function updateScaleUsingMatrix(element) {
  const currentTransformMatrix = new DOMMatrix(getComputedStyle(element).transform);
  // I can bump up the scale by just incrementing the numbers!
  currentTransformMatrix.a += 0.3;
  currentTransformMatrix.d += 0.3;
  element.style.transform = currentTransformMatrix;
  // or more explicitly,
  // element.style.transform = currentTransformMatrix.toString();
}

Additional Notes

• Incrementing numerical css values directly is part of the promise of the Typed OM API that's part of CSS Houdini. Unfortunately it's not yet fully implemented by browsers, most notably in Safari and Firefox. So you could view this matrix method as a fallback for those two browsers if you're already using Typed OM in the chromium-based browsers that support it.
• You could claim, fairly in my opinion, that this is not very intuitive. There's still the challenge of knowing which of those properties from a-f and m11-m44 represent the normal css transform functions that we're used to. For simpler transforms involving scale() and translate(), MDN has an example showing how scale() and transform() map to the matrix properties. There's also this package that will give you the familiar syntax of transform functions, so you can use the DOMMatrix as your intermediate only for manipulating the numbers. You can then convert back to the familiar transform functions to preserve readability. To be clear I have no affiliation with that package, I only came across it during my research. Most importantly, my recommendation is that you should test and see what properties of the 2d/3d matrix change when you perform your desired update, and then write your logic to update only that property.
• DOMMatrix has the alias WebKitCSSMatrix in some browsers. In Code This Means const getMatrix = window.DOMMatrix || window.WebKitCSSMatrix; can cover both possibilities.

So, I hope you found this as fascinating as I did. I punched the air and let out a loud "yussssss!" (yes I'm that type of dev 😂) when I realized this could work for me. Feel free to reach out to share your enthusiasm, ask any questions, correct errors, or just say hi to me @cinexa7254 on Twitter. Thanks for your time!

TL;DR

• You can use DOMMatrix to represent your css transforms and easily manipulate the numbers like any other javascript variable.
• You can map the familiar syntax of css transform functions to individual properties in the matrix either by testing manually to see what matrix properties change as you update the transform, or by using a package that does this for you, such as this one (I'm NOT affiliated with that project, I only came across it as part of my research).