September 1996
Introduction
A few months ago, I decided to set up a local news server on my Linux Box, in order to read off-line the articles. Before then I read the news directly from the Usenet server of my Internet Provider, at the detriment of my phone bill…(Here, in France, even the local commnications are rather expensive)
Before trying to install and set up one of the two classical news servers (Inn and CNews), I browsed a bit the Web, looking for a possible other product. I then discovered a very powerful small package « Leafnode » written by Arnt Gulbrandsen. It can be found in source form at ftp://ftp.troll.no/freebies/leafnode/ and the home page of this project is accessible at http://www.troll.no/freebies/leafnode.html.
This package (leafnode-0.9.tar.gz, 29 Kb) contains three little programs, very easy to install, and to use:
- « Fetch » can feed a local news server from a remote Usenet server (university, ISP…). It also posts the outgoing articles, using classical NNTP requests.
- « Leafnode » is a USENET server, run by the inetd daemon, when a connection in required on the NNTP port, by a newsreader.
- « Texpire » is generally run daily from the crontab to erase the oldest articles from the news spool.
The main advantage of this system, is the transparent way it is inserted between the remote news server and the local newsreader. The remote Usenet server sees Leafnode exactly like a classical newsreader (like tin, trn, netscape,…) and the local news reader sees Leafnode just like a USENET server.
I have been happily using Leafnode for several months, and I would like to describe here the (very simple) steps to install, and configure it. Then I will give some hints to use it in a multi-users environment.
INSTALLING LEAFNODE
I’ve been using leafnode 0.8 for a few months, but I’ve recently upgraded to version 0.9. The process for installing it is the same as the previous version, but I’ve had a little problem, maybe due to my version of make, and I’ll describe how I have fixed it.
You first need to download the source code from ftp://ftp.troll.no/freebies/leafnode/. You’ll get leafnode-0.9.tar.gz.
Compiling the source code
Become root and do :
# cd /usr/local/src # tar -xzf ~/leafnode-0.9.tar.gz # cd leafnode-0.9/ # make
Everything must compile without any problem…
Installing leafnode
Make sure there is a « news » user and a « news » group on your system.
Then you can type:
# make install
If « make install » complains when making directories in /var/spool/news/message.id/, you may have the same problem as I’ve had. To fix it, I’ve modified the Makefile, to insert a part of the leafnode-0.8 Makefile:
replace the lines (in the « install: » section)
cd $(SPOOLDIR)/message.id
for a in 0 1 2 3 4 5 6 7 8 9
do for b in 0 1 2 3 4 5 6 7 8 9
do
mkdir ${a}${b}0 ${a}${b}1 ${a}${b}2 ${a}${b}3 ${a}${b}4
mkdir ${a}${b}5 ${a}${b}6 ${a}${b}7 ${a}${b}8 ${a}${b}9
done
by the line
mkdir -p $(SPOOLDIR)/message.id/{0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}
Be sure that the first character on the line is a tabulation. Then try again:
# make install
End of installation
At this point, Leafnode will have installed the following files on your system:
/usr/local/sbin/fetch /usr/local/sbin/leafnode /usr/local/sbin/texpire /usr/local/man/fetch.8 /usr/local/man/leafnode.8 /usr/local/man/texpire.8 /usr/lib/leafnode/config.example /var/spool/news/ … and a lot of subdirectories …
First you have to copy the file /usr/lib/leafnode/config.example to /usr/lib/leafnode/config and edit him, to put the name of your remote NNTP server in place of:
server = news.hiof.no
Edit the file /etc/inetd.conf, and look for a line like:
nntp stream tcp nowait root /usr/sbin/tcpd in.nntpd
then modify it like this:
nntp stream tcp nowait news /usr/sbin/tcpd /usr/local/sbin/leafnode
Depending on your configuration, there may be no TCP wrapper installed, so you could have to remove the « /usr/sbin/tcpd » part of this line.
and do
# killall -HUP inetd
First run
Connect to your Usenet provider, and, while being « root » or « news« , run:
$ fetch
You will have to wait for a moment, because Leafnode is asking the NNTP server the list of all the active groups. Once fetch ends, run a newsreader, as normal user, and ask him to contact the localhost. for example, with tin do:
$ export NNTPSERVER=localhost $ tin -r
(With Netscape you have to put localhost in « Options/Mail and News/Servers/NNTP server »)
You will get the list of all available newsgroups, then choose interesting ones, and read them. They will appear empty at this time. It’s normal.
As root run again fetch. It will download all the content of the previously read newsgroups. The first downloading will take a while, but the next will obviously be very much quicker.
Your local USENET server is installed!
CONFIGURING LEAFNODE
Now that leafnode run on your Linux box, you can configure some details:
In the file /usr/lib/leafnode/config, there are two fields you can edit:
- the expire value tells to Texpire the number of days an unread thread must be kept.
- the maxcount value limits the number of articles to download from a newsgroup, during a single execution of Fetch.
Refer to the comments in this file to choose the values (on my system I use expire=10 and maxcount=3000)
You must run Texpire from time to time, and a daily entry in the crontab seems to be the best choice:
00 03 * * * news /usr/local/sbin/texpire
This line is for a system-wide crontab (generally /etc/crontab maintained by root), but you will have to remove the username « news » on user crontab (/var/spool/cron/news).
You will have to run periodically fetch. If you have a permanent link with your news server, there’s an obvious solution: the crontab again, to run it once per hour for example.
00 * * * * news /usr/local/sbin/fetch
If you’re using a non-permanent ppp connection, you can insert fetch at the end of the ip-up shell script (see Linux Gazette 7 « Setting up PPP’s ip-up and ip-down scripts!« ).
Put the name « localhost » in /etc/nntpserver or set the environment variable (for example in /etc/profile):
export NNTPSERVER=localhost
USING LEAFNODE ON MULTI-USERS SITE.
There are any problems to avoid, especially if there are other users on your system, or if the leafnode server is on a local network. First you must make sure of the validity of the headers in the outgoing posts, but you also need to limit the list of the accessible newsgroups. Don’t forget that Fetch will download the whole content of a newsgroup if someone tries to read it. So, be very careful with newsgroups like alt.binaries.pictures…
Fortunately, we can use some little awk and shell scripts to verify and correct the outgoing posts, and to limit the local access to selected newsgroups.
Checking the outgoing posts
With some newsreaders, the « From: » field of the outgoing articles will be set to myname@my.station.on.my.lan and notmyname@my.internet.provider.com. With some of them you can configure the « From: » and « Reply-to: » fields, while the others need you to recompile them.
This problem can be worse if you have a Linux box with several users. Some of them can have misconfigured newsreader (sometimes on purpose..) and it may be safer to check the headers of the outgoing articles before posting them.
Here’s a small awk filter which allows a kind of masquerading of the « From: » line of an article. It will change the line "From:username@my.station.on.my.lan (user real name)" to a line "From: username@my.internet.provider.com (user real name)".
You may also ensure that username is correct (i.e. in a list of allowed users). The same perl script will help us to determine the correct articles. Otherwise it will add a line « *** Wrong From field – This article must be deleted *** » to the message .
#! /usr/bin/gawk -f
#
# /usr/local/sbin/change_article_from_domain
#
# awk script to change the domain name on the "From:"
# line of outgoing articles.
# If the username is not valid a message will be added
# at the bottom o the file, allowing a 'grep' to delete
# him.
BEGIN {
# replace with the correct domains
local_domain="my.station.on.my.lan"
real_domain ="my.internet.provider.com"
# insert here the name of your users allowed to post articles
# (may be just one)
valid_usernames["user1"]
valid_usernames["user2"]
must_be_deleted=0
}
/^From:/ {
gsub(local_domain, real_domain)
username=substr($2,1,index ($2, "@") - 1)
if (! (username in valid_usernames)) {
must_be_deleted=1
# you can also add a system command
# example : mail to newsmaster with
# the username of the wrong article
}
}
END {
if (must_be_deleted != 0) {
print "*** Wrong From field - This article must be deleted ***"
}
}
{
print
}
This script can be useful if you have up to, say, ten users, otherwise you’ll need to improve it in order to read the list of allowed users in an other file for example.
Now we will execute the above script on all the outgoing articles, sitting in /var/spool/news/out.going, then delete (or move to another directory) those with bad username.
#! /bin/bash # # /usr/local/sbin/modify_outgoing_articles cd /var/spool/news/out.going for i in * ; do /usr/local/sbin/change_article_from_domain < $i >/tmp/modified_articles/$i done rm -f * mv /tmp/modified_articles/* . rm -f `grep -l "*** Wrong From field - This article must be deleted ***" *`
(The backquote is used to catch the result of grep) Don’t forget to create a /tmp/modified_articles/ directory. This script cannot prevent the fake « From: » lines, when an authorized user is hidden behind an other correct username. This can not be easily done, and if you really don’t trust your users, you’ll have to use another Usenet package like Inn or Cnews.
Now all the outgoing articles will have a correct « From: » line.
Limiting the list of accessible news groups
The second important point to check out is the list of fetched newsgroups. If you haven’t got a huge disk space, it would be better to avoid downloading alt.binaries groups or alt.2600. for example… The problem is that fetch will download the content of each newsgroup corresponding to a file in /var/spool/news/interesting.groups, for example /var/spool/news/interesting.groups/comp.os.linux.announce
A file in this directory is touched by leafnode every time a user tries to read the content of the group. Are you sure that none of your users will try to have a look at alt.binaries.pictures.erotica…? So there are two possible solutions:
- You can edit the /usr/lib/leafnode/groupinfo file, in order to suppress the forbidden newsgroups. But this is not a very good solution, because fetch will re-create it during the next connection.
- You can suppress the forbidden groups from /var/spool/news/interesting.groups. This can be done by shell script, in many different ways, for example:
#! /bin/bash # # modify_interesting_groups cd /var/spool/news rm -f alt.* rm -f *windows* … touch comp.os.linux.announce touch comp.windows.x.announce …
you can also have a definite list of fetched newsgroups, for example in /var/spool/news/official.list, manually creadted:
# mkdir /var/spool/news/official.list # cd /var/spool/news/official.list # touch comp.os.linux.announce # touch comp.os.linux.answer # touch comp.lang.c.moderated
And this list will be copied in the interesting.groups directory by a script before each execution of fetch.
#! /bin/bash # # modify_interesting_groups cd /var/spool/news rm -f * copy /var/spool/news/official.list .
Now we have two scripts to insert before running fetch. For example in /etc/ppp/ip-up:
#! /bin/bash # # /etc/ppp/ip-up … /usr/local/sbin/modify_interesting_groups /usr/local/sbin/modify_outgoing_articles fetch
CONCLUSION
I think that Leafnode is a very interesting package for those (most of us) who are running Linux on a standalone box with intermitent connection to a Usenet server, or on a small local network with few users. It’s powerful and much simpler to install and to configure than Inn or Cnews, designed for bigger sites. Moreover it does not require any maintenance.
