Andy Burns <usenet@andyburns.uk> wrote:
>Adam H. Kerman wrote:

>>You also declared format=flowed.

>I used to set it specifically to do that, nowadays I can't see anywhere
>I could set it to not do that.

>>Thunderbird used the quote-level
>>indicator with the space separating it from quoted text but failed to
>>turn the quote into flowed text. There is no flowed text in that
>>followup. It also declared UTF-8 despite being 100% ASCII.

>The UTF-8 is a per-server setting, British English doesn't need much in
>terms of 8 bit characters, other than our currency symbol, I think we're
>past the days of expecting users to type GBP to avoid them. I've
>altered it to ISO-Latin-1, not sure if that will apply to this message,
>or only future ones?

I see

Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Also, in another header, your language and locale has been declared. I
don't know what the hell that's supposed to do for the reader. No
newsreader runs an article through Google Translate.

>>Crap like this doesn't break other newsreaders, but it's additional
>>examples of Thunderbird claiming to do things that it doesn't actually
>>get right.

Andy Burns <usenet@andyburns.uk> wrote:
>Andy Burns wrote:
>>Adam H. Kerman wrote:

>>> You also declared format=flowed.

>> I used to set it specifically to do that, nowadays I can't see anywhere
>> I could set it to not do that.

>the format=flowed setting only exists within the "here be dragons"
>about:config

The two-line quoted paragraph is flowed. I looked at the precursor
article; it was flowed there as well.

That tell us that despite the setting, Thunderbird won't flow unflowed
text in quote but will flow new text. Format=flowed has to be able to
address quoted text as oftentimes, the quote should be reformatted to
avoid creating a long line when the quote-level indicator is prepended
to the line, so it might as well be reformatted as flowed text if that's
what the newsreader is set to do. There just aren't newsreaders or Mail
clients that usefully create flowed text in quote.

Carlos E.R. <robin_listas@es.invalid> wrote:
>On 2024-02-02 18:47, Adam H. Kerman wrote:
>> Andy Burns <usenet@andyburns.uk> wrote:
>>> Carlos E.R. wrote:
>>>
>>>> The offending sequence is:
>>>>
>>>> 2E C2 A0  20 41
>>>> .            A
>>>>
>>>> C2 A0?  It should be just a "20 20".
>>>
>>> Looks like something tries to convert a sequence of two normal spaces to
>>> one non-breaking space (C2 A0) plus a normal space (20)
>>>
>>> But it then seems to get mangled into (C3 82) which is the Â
>>>
>>>
>>>
>>
>> Did you paste into a text editor in linux that wasn't told to convert
>> the Unix LF line boundary to CRLF boundary used on Usenet?

>Not me.

That is correct. You output from the composer the standard line boundary.

>I use Leafnode nntp proxy server, in Linux. I looked directly at the
>single file of the post using Midnight Commander Hex viewer.

Andy Burns <usenet@andyburns.uk> wrote:
>Adam H. Kerman wrote:

>>> this is just a straight reply to carlos's message

>> Again: Additional nonbreaking space characters were added. Every single
>> CRLF line boundary got changed to LF.

>> This followup isn't unconventional, Andy. It's nonstandard. The CRLF
>> line boundary has ALWAYS been the standard line boundary.

>I'm not doing anything to force "unixy" line breaks.

I force a deliberate translation mismatch in my terminal emulation, then
I see substituted characters in display for received characters that are
unexpected or cannot be translated given the settings. That's how I saw
the nonstandard line boundary.

I don't see any in this followup, and all of the nonbreaking spaces are
gone. It's pure ASCII, still declaring UTF-8 and 7-bit encoding!

>When I see your replies to my replies, it looks double-line spaced.

I quoted the nonstandard line break without translating it to the
standard line break, so that makes sense that the display would be
screwed up.

>It certainly doesn't look that way during my composition, or when I
>receive it back from the server, only if I manually decode the base64
>body, and paste into a hex viewer, *then* I see the 0A line breaks.

Oh for heaven's sake. Thunderbird needs to parse, THEN encode.

This is a huge hint. After decoding an image, are the line boundaries
supposed to be LF and not CRLF? I don't pretend any expertise here.

It's seeing some character sequence that's triggering an interpretation
that text is a binary and therefore to be encoded.

I haven't spotted the wrong line boundaries in other Thunderbird users'
articles.

>So unfortunately yes, TB is accumulating multiple ways it
>does "the wrong thing"

>I've set it for ISO-8859-1, it ignores me and uses UTF-8 anyway

>The moment it sees 8-bit characters it encodes in base64

>And it mutates "DOS" style line endings to "UNIX" style line endings

Adam H. Kerman wrote:

> The two-line quoted paragraph is flowed. I looked at the precursor
> article; it was flowed there as well.

The default about:config settings related to flowed are

mailnews.send_plaintext_flowed=true
mailnews.display.disable_format_flowed_support=false

which I haven't changed.

> That tell us that despite the setting, Thunderbird won't flow unflowed
> text in quote but will flow new text.

Yes, there is a certain clunkyness to editing quoted parts of a message,
it definitely acts as a bunch of lines, rather than one flowing
paragraph, I often select multiple lines and re-wrap it using ctrl-R

> Format=flowed has to be able to
> address quoted text as oftentimes, the quote should be reformatted to
> avoid creating a long line when the quote-level indicator is prepended
> to the line, so it might as well be reformatted as flowed text if that's
> what the newsreader is set to do. There just aren't newsreaders or Mail
> clients that usefully create flowed text in quote.

I can't remember whether format=flowed existed when I was using Forte
Agent, or whether it did much better?

I'll dip out of this thread now I think, and may hit the test groups
when I have chance ...

On 2024-02-02 20:33, Adam H. Kerman wrote:
> Andy Burns <usenet@andyburns.uk> wrote:
>> Adam H. Kerman wrote:

....

> I see
>
> Content-Type: text/plain; charset=UTF-8; format=flowed
> Content-Transfer-Encoding: 7bit
>
> Also, in another header, your language and locale has been declared. I
> don't know what the hell that's supposed to do for the reader. No
> newsreader runs an article through Google Translate.

It is just information for future features.

Like what dictionaries to activate, what salutation line to include, etc.

--
Cheers, Carlos.

"Carlos E.R." <robin_listas@es.invalid> wrote:

> On 2024-02-02 19:23, VanguardLH wrote:
>> "Carlos E.R." <robin_listas@es.invalid> wrote:
>>
>>> Me (my client) is not adding accented letters. Knuttle message has wrong
>>> chars and I proved it.
>>
>> Knuttle's posts use ASCII-7 (despite he specifies UTF-8). His post has
>> no accented characters. There are no wrong chars in his post.
>
> Yes there are, and I proved it. Look at the hex dump of his post,
> obtained without Thunderbird tinkering it, that I posted.
>
> Stop arguing the contrary, others are already on it.

Does anyone ELSE see non-ASCII characters in Knuttle's first post with
Message-ID of <updvvk$1k8be$1@dont-email.me>?

My client lets me see the raw source of messages, and I don't see any
non-ASCII in his posts. You're using Thunderbird, and all it has is to
view the raw source of articles is the same as mine. No "hex dump".

https://www.lifewire.com/view-source-of-message-thunderbird-1173188

In your <haet8kxo8f.ln2@Telcontar.valinor> article, you mention using
some "nnet" proxy leeching from some NNTP server. In the hex dump you
show from there, that is not from Knuttle's post. How do I know?
Because Knuttle was not indent quoting his own content.

Not in Knuttle's post __
\
00000500 0A 0A 3E 20 44 6F 63 78 20 69 73 20 ..> Docx is
0000050C 61 20 70 72 6F 70 72 69 65 74 61 72 a proprietar
00000518 79 20 66 6F 72 6D 20 62 65 6C 6F 6E y form belon
00000524 67 69 6E 67 20 74 6F 20 4D 69 63 72 ging to Micr
00000530 6F 73 6F 66 74 2E C2 A0 20 41 6E 79 osoft.  Any
0000053C 62 6F 64 79 20 77 69 74 68 20 61 20 body with a
00000548 4D 53 20 0A 3E 20 70 72 6F 67 72 61 MS .> progra
/
Not in Knuttle's post ____/

Greater-than signs are used for quote indentation. Knuttle wasn't
quoting himself within his same post. That was you adding the > chars
to cite his content in the quoted portion of your article. You grabbed
the wrong article to show a hex dump.

Knuttle posted this:

Docx is a proprietary form belonging to Microsoft. Anybody with a MS
progra...

No quoting chars (greater-than signs). He did not post what you show in
your hex dump, which was:

> Docx is a proprietary form belonging to Microsoft.* Anybody with a MS
> progra...

where * = U+0202.

The quoting characters (greater-than signs) were in your article, not
his. The A+inveredBreve was in your article, not his. How about doing
a hex dump on Knuttle's article? That would have no quoting chars in
his paragraph starting with "Docx is a proprietary".

Carlos E.R. <robin_listas@es.invalid> wrote:
>On 2024-02-02 20:33, Adam H. Kerman wrote:
>>Andy Burns <usenet@andyburns.uk> wrote:
>>>Adam H. Kerman wrote:

>...

>>I see

>>Content-Type: text/plain; charset=UTF-8; format=flowed
>>Content-Transfer-Encoding: 7bit
>>Also, in another header, your language and locale has been declared. I
>>don't know what the hell that's supposed to do for the reader. No
>>newsreader runs an article through Google Translate.

>It is just information for future features.

>Like what dictionaries to activate, what salutation line to include, etc.

In reading an article? None of that is relevant. It might be useful for
training an AI.

In followup, if the user isn't writing in his usual language, he'll have
to change some setting manually, like the dictionary. But the one thing
he won't do is change his locale, which is what the country code
indicates. The locale is relevant as a very broad hint of what dialect
of a language to expect but irrelevant to a user writing in a foreign
language that's certainly not based on his locale.

I'm just not seeing something happening automatically in the user's
newsreader to assist in reading the article.

VanguardLH <V@nguard.LH> wrote:

>Does anyone ELSE see non-ASCII characters in Knuttle's first post with
>Message-ID of <updvvk$1k8be$1@dont-email.me>?

I saw no non-ASCII characters at all.