Re: Solution three (notes, not actual description yet): distributed nntp

[Erik Walthinsen <omega@omegacs.net>]

> This is a robust distributed model based on already-tested code. I
> haven't worked out the details yet, but I believe this is the correct
> path to go.
Ah, he sees the light ;-)  This was hashed out a couple hours ago on IRC,
and I'll see if I can explain why I believe this is the correct model.

Basically, the three-tiered model is flawed for several reasons:
1) the master is the ultimate single point of failure.  If it goes down,
we have to rearrange things, or sweat it out until it comes back.
2) the entire system has to be written to support 4 different
communication channels: master<->mirrors, and mirrors<->clients.  More
code, more bugs, more things to go wrong.
3) the data gets copied all over the place, and several machines (master +
mirrors) are storing data just for the sake of storing data.  This is
poor.
4) the existing news sites are already holding the information, why
abstract them behind another two layers?  It introduces more lag, more
resource usage, and generally more suckage.

The nntp-ish model is more sane, because:

1) *every* server is considered equal, and indeed, has the same software.
The cool thing is that this software already exists (cnews, inn, etc.),
and it is known to work well (see Usenet for nifty example).
2) the total amount of travel for the data is minimized, and is going
through known software.  Importing news from a spool to an sql db is
trivial, if we use a well-known format for the articles.  News::NNTPClient
and your choice of db module will chug through it in no time at all.  I
believe inn and cnews both have trigger mechanisms, making it a piece of
cake.
3) the whole system is much more robust, because nntp is designed to deal
with massive failures.  If the master server (the news host with no web
front end, or at least no *specific* front end) dies, auxilliary news
feeds deal with propagation of news around the failed server.

My general feeling is that this is the cleanest, and quickest way to get
things working.

> Please argue with me. :)
But it's so much easier to agree, especially when you get it right ;-)

TTYL,
    Omega

     Erik Walthinsen <omega@seul.org> - SEUL Project system architect
        __
       /  \                SEUL: Simple End-User Linux -
      |    | M E G A              Helping Linux become THE choice
      _\  /_                             for the home or office user