Should Edited MTL Be Allowed?

[Agnaa]

Also, having given some more time to think about it, I think we should expect MTLs to be reproducible. If a person consistently provides MTLs that are different from when other people use the same services, and are different in ways that serve their point, we should assume they were manipulated and punish them accordingly.

And to facilitate that, I think we should require people to specify which source of MTL they used, and if it was an LLM, what specific prompts they used.

 

[KLOL506]

Bump

 

[FinePoint]

and if it was an LLM, what specific prompts they used.

To actually confirm it was a real output from a particular LLM you'd also need the temperature, top_p, sampling parameters, etc.
(The entire point of token sampling is that the outputs are relatively unique.)

Knowing the prompt would definitely help us see how it generally translates something, but we'd ultimately not be able to tell if a particular word or phrase difference is due to manipulation or just random chance.

 

[Agnaa]

To actually confirm it was a real output from a particular LLM you'd also need the temperature, top_p, sampling parameters, etc.
(The entire point of token sampling is that the outputs are relatively unique.)

Knowing the prompt would definitely help us see how it generally translates something, but we'd ultimately not be able to tell if a particular word or phrase difference is due to manipulation or just random chance.

I didn't think modern chatbot-UI LLMs let you tinker with those.

If so, then yeah, we should receive those too.

 

[FinePoint]

I didn't think modern chatbot-UI LLMs let you tinker with those.

If so, then yeah, we should receive those too.

ChatGPT specifically (which we'd see most often) doesn't unless you use the API, but a lot of other services include at least temperature as an option.

For the sake of a rule, we'd probably just want to keep it vague, like: "The prompt you used and any other optional parameters if available."

We'd never be 100% certain, but of course we could run it an arbitrary number of times and if it never uses the same wording the user claims it did then it was probably manipulated.

 

[KLOL506]

@Antvasima You said you had some issues regarding AI translations with regards to hallucinations and stuff?

 

[Antvasima]

@Antvasima You said you had some issues regarding AI translations with regards to hallucinations and stuff?

Here is what I said:

As I mentioned elsewhere, I am currently strongly inclined to vote against using A.I.s for translations, as they notoriously inserts hallucinations into a small part of their scripts, and we wouldn't know which parts, or if they would catch nuances and choose the context-correct interpretations in cases of multiple meanings in Japanese or Chinese for example.

We would almost never know which 2% or more of the translations that an A.I. would just "lie" about.

Also, I am currently more inclined to agree with DontTalk and Agnaa about usage of A.I. images.

 

[FinePoint]

Even the latest AI models have a hallucination rate of at least 15% when given difficult tasks.

Whether that's higher than the error rate of a human translator, I wouldn't know, but definitely worth considering.

At the very least, it's worse than a human in that the AI is practically incapable of simply saying it doesn't know, rather than making things up.

 

[DontTalkDT]

Even the latest AI models have a hallucination rate of at least 15% when given difficult tasks.

Is a translation a difficult task, though? Wouldn't match my experience on the success rate of translation.

In any case, it's worth keeping in mind that an edited MTL isn't just passing a text through an AI. There's human oversight over the result. It needs very specific errors to fall through the net.

Anyway, my personal opinion is still that we shouldn't try to use AI to make translations on our own, especially not for source material that has an english translation already available.
However, I think for stories which exist entirely in MTL form in the english space, those should be fine unless they are proven problematic.

More specifically, my opinion is still this.

 

[FinePoint]

Is a translation a difficult task, though? Wouldn't match my experience on the success rate of translation.

That really depends on the language, context, and purpose. As a result studies on the subject give wildly different results.
For casual purposes I think AI is already more than sufficient.

I would argue that our use is rather technical, though, since we're especially concerned over the exact verbiage of rather complex topics.

So, I will use a study on a rather technical subject where exact verbiage also matters a lot: translated discharge instructions.
The results, in summary, were: when translating technical documents into Spanish, the error rate was similar to professional translators, just less fluent.
However, other languages, like Chinese, were markedly worse and more prone to errors across all metrics.

Of especial note, though, is that when AI did make a mistake (other than in Spanish), it was far more likely to be a very significant mistake that actually harms comprehension, such as skipping an entire section, or replacing a word with one that's nonsensical.

There's not a lot of studies on Japanese specifically, which I find we need most often. Three years ago studies were showing like 73% error. Anecdotally, companies in Japan often use AI for assistance now but definitely don't trust it enough to replace humans at this stage.

Tl;dr: For very common English-adjacent languages like Spanish, it would seem not, but for other more unique languages probably.

 

[Agnaa]

In any case, it's worth keeping in mind that an edited MTL isn't just passing a text through an AI. There's human oversight over the result. It needs very specific errors to fall through the net.

I wouldn't make such a strong claim, I think it can vary a lot. Edited MTL can mean using it for the first pass, then having a professional-quality translator go through checking that everything was translated well. Or it can mean a novice who doesn't know the source language reads the English text and tries to make sure that makes sense. Or it can mean skimming through the text to replace pronouns with a consistent translation.

 

[DontTalkDT]

That really depends on the language, context, and purpose. As a result studies on the subject give wildly different results.
For casual purposes I think AI is already more than sufficient.

I would argue that our use is rather technical, though, since we're especially concerned over the exact verbiage of rather complex topics.

So, I will use a study on a rather technical subject where exact verbiage also matters a lot: translated discharge instructions.
The results, in summary, were: when translating technical documents into Spanish, the error rate was similar to professional translators, just less fluent.
However, other languages, like Chinese, were markedly worse and more prone to errors across all metrics.

Of especial note, though, is that when AI did make a mistake (other than in Spanish), it was far more likely to be a very significant mistake that actually harms comprehension, such as skipping an entire section, or replacing a word with one that's nonsensical.

There's not a lot of studies on Japanese specifically, which I find we need most often. Three years ago studies were showing like 73% error. Anecdotally, companies in Japan often use AI for assistance now but definitely don't trust it enough to replace humans at this stage.

Tl;dr: For very common English-adjacent languages like Spanish, it would seem not, but for other more unique languages probably.

Well, as said, for verses that pull complicated metaphysical shenanigans where each word is important, I can see us saying that we don't want to deal with MTL. (Although it really isn't like human translation necessarily preserves those...) I just don't think "mage shoots fireball" is a complex situation.

Also, interestingly, the errors you point out, like skipping sections or inserting nonsense words, are exactly the kinds that actually wouldn't matter for an edited MTL. Because those are the kind of errors that would be noticed.

I wouldn't make such a strong claim, I think it can vary a lot. Edited MTL can mean using it for the first pass, then having a professional-quality translator go through checking that everything was translated well. Or it can mean a novice who doesn't know the source language reads the English text and tries to make sure that makes sense. Or it can mean skimming through the text to replace pronouns with a consistent translation.

Sure. I'm basically talking about the second case here, as that's what I usually see from MTL web novels. Although, it usually involves some more research into difficult passages.

 

[Vietthai96]

Idk much about other cases involves Other Latin languages to English translation since i'm not follows those stories. In case of Asian-Kanji languages like Japanese, Chinese to English translation, I'm on the DontTalkDT side, which is incident-based approach