Are text-only peers being too restrictive at <256k article size?

[Jesse Rehmer]

I have followed what seems like a general consensus among text-only
peers to limit article size at 128/256k when setting up peering
arrangements. However, I've begun a project to back fill my spool using
tools like pullnews and suck from a large spool.

I never changed INN's max. article size from 1000000. Out of curiosity,
after filling a few Big8 hierarchies from the upstream spool, I searched
the spool for articles >500k and to my surprise I've been missing out on
a lot of valid and (in my opinion) valuable articles.

I'm finding a ton of FAQs, some checkgroup messages, lots of source code
discussions, etc. that I've never seen before. I read a thread from a
few years ago where Russ Allbery mentions by not accepting articles up
to 1MB you may be missing out on quite a bit. From a very quick scan of
what I've found larger than 500k, he's right.

Seems I've been a bit hasty requesting peering parameters around a 256k
limit. I assume this common peering parameter was adopted to limit the
chances of misplaced binary articles coming through a feed, but am
wondering how "harmful" it is to have such low limits when I'm seeing a
high rate of valid communication that I believe many news admins would
want to carry?

Admittedly, it usually takes trial and error with peers using Cyclone
(and quite a few with Diablo who disable its internal article type
check) to properly configure their feeds not to leak binaries to
text-only peers, but I'm not seeing much of that these days.

Cheers,

Jesse

[The Doctor]

Post by Jesse Rehmer
I have followed what seems like a general consensus among text-only
peers to limit article size at 128/256k when setting up peering
arrangements. However, I've begun a project to back fill my spool using
tools like pullnews and suck from a large spool.
I never changed INN's max. article size from 1000000. Out of curiosity,
after filling a few Big8 hierarchies from the upstream spool, I searched
the spool for articles >500k and to my surprise I've been missing out on
a lot of valid and (in my opinion) valuable articles.
I'm finding a ton of FAQs, some checkgroup messages, lots of source code
discussions, etc. that I've never seen before. I read a thread from a
few years ago where Russ Allbery mentions by not accepting articles up
to 1MB you may be missing out on quite a bit. From a very quick scan of
what I've found larger than 500k, he's right.
Seems I've been a bit hasty requesting peering parameters around a 256k
limit. I assume this common peering parameter was adopted to limit the
chances of misplaced binary articles coming through a feed, but am
wondering how "harmful" it is to have such low limits when I'm seeing a
high rate of valid communication that I believe many news admins would
want to carry?
Admittedly, it usually takes trial and error with peers using Cyclone
(and quite a few with Diablo who disable its internal article type
check) to properly configure their feeds not to leak binaries to
text-only peers, but I'm not seeing much of that these days.
Cheers,
Jesse

Try

:Tm,<512000 .

[Richard Kettlewell]

Post by Jesse Rehmer
I have followed what seems like a general consensus among text-only
peers to limit article size at 128/256k when setting up peering
arrangements. However, I've begun a project to back fill my spool
using tools like pullnews and suck from a large spool.
I never changed INN's max. article size from 1000000. Out of
curiosity, after filling a few Big8 hierarchies from the upstream
spool, I searched the spool for articles >500k and to my surprise I've
been missing out on a lot of valid and (in my opinion) valuable
articles.

I track what the largest articles in my spool are and I don’t think I’ve
seen anything over 500Kbyte - locally at least, the largest tend to be
in the 100-120Kbyte region.

A lot of the larger articles are spam - 8 of a recent top 10 were copies
of the same article, for instance. For this reason I’ve now adopted a
much lower EMP threshold for large articles.

[Thomas Hochstein]

Post by Jesse Rehmer
I have followed what seems like a general consensus among text-only
peers to limit article size at 128/256k when setting up peering
arrangements.

That seems rather small.

Post by Jesse Rehmer
I'm finding a ton of FAQs, some checkgroup messages, lots of source code
discussions, etc. that I've never seen before.

Yep.

Post by Jesse Rehmer
I assume this common peering parameter was adopted to limit the
chances of misplaced binary articles coming through a feed, but am
wondering how "harmful" it is to have such low limits when I'm seeing a
high rate of valid communication that I believe many news admins would
want to carry?

It seems wrong to filter on size if you really want to filter on type.
Cleanfeed can block binaries in non-binary groups, regardless of size;
then you can accept text-only messages regardless of size, too.

-thh

[noel]

Post by Thomas Hochstein

Post by Jesse Rehmer
I have followed what seems like a general consensus among text-only
peers to limit article size at 128/256k when setting up peering
arrangements.

That seems rather small.

If you've got half megabyte text posts, they are not text but binaries
hiding as text, either that or its a year long thread and everyone is too
clueless to trim their damn posts, sure the odd FAQ post might get caught
up in it, but its gonna be rare.

[b***@ripco.com]

Post by noel

Post by Thomas Hochstein

Post by Jesse Rehmer
I have followed what seems like a general consensus among text-only
peers to limit article size at 128/256k when setting up peering
arrangements.

That seems rather small.

If you've got half megabyte text posts, they are not text but binaries
hiding as text, either that or its a year long thread and everyone is too
clueless to trim their damn posts, sure the odd FAQ post might get caught
up in it, but its gonna be rare.

I'm in total agreement.

Even at a 250k limit, anyone still on a 80x24 character screen (like me)
would be going through 130 pages.

The largest post I can find on my spool is just a bit over 210k and it's
just a faq that probably no one reads.

I really would like to know what posts Jesse found that would be of any
interest to anyone in the 500k to 1m range.

-bruce
***@ripco.com

[Nigel Reed]

On Sat, 18 Jun 2022 13:50:26 -0500

Post by Jesse Rehmer
I have followed what seems like a general consensus among text-only
peers to limit article size at 128/256k when setting up peering
arrangements. However, I've begun a project to back fill my spool
using tools like pullnews and suck from a large spool.
I never changed INN's max. article size from 1000000. Out of
curiosity, after filling a few Big8 hierarchies from the upstream
spool, I searched the spool for articles >500k and to my surprise
I've been missing out on a lot of valid and (in my opinion) valuable
articles.

Four of my ten peers (I'm always open to more) have requested
restrictions from 131972 to 512000. I have no restriction here.

[Borg]

Post by Jesse Rehmer
I have followed what seems like a general consensus among text-only
peers to limit article size at 128/256k when setting up peering
arrangements.

[...]

In the past I have used servers that limited posts to 32k and 64k. If I
were running a server today I would probably limit new posts to 64k. The
way I see it Usenet is for discussion, not encyclopedia publication.
Just my personal opinion ...

If someone is publishing a huge FAQ or something they think is really
important they can divide it into a series of posts. The work to create
such a large post is far in excess of the minor ten-second hassle of of
'head / tail -c 65536' or copypasta to two or three posts.

--

Borg