Add batch size parameter to prevent context leaking of translator notes/hints

[thomasaull]

Problem

This is a follow up to #1728, since hints currently are not included in the request to a model at all (not when using the lingo.dev platform I think)

Given the following example:

{
  // Short form for "Year", used to display a date format, example in `en`: "DD-MM-YYYY". Don’t translate for CJK (Chinese, Japan, Korean) languages
  "DATE_FORMAT_SPECIFIER_YEAR": "Y",
  // Short form for "Month", used to display a date format, example in `en`: "DD-MM-YYYY"
  "DATE_FORMAT_SPECIFIER_MONTH": "M"
}

in my tests, using openai/gpt-oss-120b (openAI compatible provider, see #1729) there is some context leaking going on, even when trying to use a prompt like:

Translate the provided text from source: {source} to target: {target}. For each translation key, use only the hint with identical key. If no hint exists for a key, do not reuse any other hints

and translate to korean, the result it get is:

{
  "DATE_FORMAT_SPECIFIER_YEAR": "Y",
  "DATE_FORMAT_SPECIFIER_MONTH": "M"
}

even though only DATE_FORMAT_SPECIFIER_YEAR has the hint to not translate to CJK languages this get’s applied to DATE_FORMAT_SPECIFIER_MONTH aswell.

Looking at the reasoning it becomes clear, that the model is not able to follow the instruction to treat each hint for each translation completely individually. I tried many things, but could not find a reliable solution.

The way out is probably to translate each string one-by-one in individual requests to the LLM, preventing context leaking between translations completely. This of course has other trade-offs (increased API cost, slower, …) but it reliably translates to:

{
  "DATE_FORMAT_SPECIFIER_YEAR": "Y",
  "DATE_FORMAT_SPECIFIER_MONTH": "월"
}

Solution

Add a batch size parameter to the CLI to control how many translations strings are send to a LLM at once. A batch size of 1 would send each translation individually.

Visuals

Workarounds

No response

[maxprilutskiy]

We were lucky to solve the root cause with Lingo.dev Cloud, but technically – what you're describing – would be an amazing config param to add @thomasaull !

Would you like to work on it?

[Hellnight2005]

Hi, this issue doesn’t appear to be assigned yet. I’d be happy to work on it and can start today — could you please assign it to me?

[thomasaull]

@Hellnight2005 I’m a bit short on time, so from my side feel free! If you want I can provide my working implementation as a starting point.

(cc @maxprilutskiy)

[Hellnight2005]

Thanks @thomasaull, I’ll take this on. If you’re able to share your working implementation, I’d be happy to use it as a starting point.

[Hellnight2005]

Thanks @thomasaull I actually went ahead and implemented it.

I've added the --batch-size parameter to both the run and i18n commands. I verified that using --batch-size 1 correctly isolates the translation requests (one key per request), which should solve the context leaking issue you described.

I also verified it with unit tests and tested the CLI command end-to-end. I'll open a PR with the changes shortly

[thomasaull]

@Hellnight2005 Wow you’re quick 😅 I just finished applying my changes to the current main, not tested though and there could be errors since I created this against an older state of the main branch: main...thomasaull:lingo.dev:feature/cli-batch-size-parameter

Feel free to dig around, but since you have already implemented it, there is probably no need

[Hellnight2005]

Thanks @thomasaull

I went ahead and added the batch-size change and opened a PR. Would you mind taking a look and sharing your thoughts?

[thomasaull]

@maxprilutskiy Can someone from the lingo.dev team take a look aswell?