How to make a local yum repository mirror.

By Hal Canary

Intro

Last week a friend of mine with a slow dial-up internet connection asked how he could download all of the newest updates for his FC1 system and take them home on a CD. I showed him how.

A small cluster of a router, one server and two workstations all have FC1 installed. Why not save upstream bandwidth by locally mirroring all updates? That's just what I did.

Let's get started. What you'll need: a server with httpd and plenty of extra hard-drive space (At least 5 GB). To retrieve the packages you'll need the rsync client. To automate the process you'll need the cron dæmon. And of course, you'll need to have yum on your clients.

Step one. Use rsync to mirror a repository.

I happen to have a directory on my server called /space. It's a user-writable location that doesn't get included in my weekly backups like /home does. I'll create a directory /space/mirror for all of my mirrors. Then I softlink the directory so that it shows up on my webserver.

$ mkdir -p /space/mirror
# ln -s /space/mirror /var/www/html/
$ ls -l /var/www/html/mirror
lrwxrwxrwx 1 root  root   18 Jan 25 03:04 /var/www/html/mirror -> \
/space/mirror

(I'm doing all of the following steps as a non-root user. Safer that way.)

Now I'll write a script that does the syncing. The script doesn't do much but call into rsync to do all of the heavy lifting. I wrote mirrors.kernel.org here, but you will want to find a closer mirror.

#!/bin/sh
# mirror-script.sh
# hal canary
DATE=`/bin/date +%Y-%m-%d`
OUTDIR='/tmp'
MIRROR=/space/mirror
[ -d $OUTDIR ] || mkdir -p $OUTDIR
OUTFILE=$OUTDIR/mirror-output-$DATE.txt

/bin/nice /usr/bin/rsync --verbose --progress \
   --stats --archive --partial \
   --exclude development/ \
   --exclude test/ \
   --exclude 1/SRPMS/ \
   --exclude 1/i386/iso/yarrow-SRPMS-disc1.iso \
   --exclude 1/i386/iso/yarrow-SRPMS-disc2.iso \
   --exclude 1/i386/iso/yarrow-SRPMS-disc3.iso \
   --exclude 1/i386/debug/ \
   --exclude updates/testing/ \
   --exclude updates/1/SRPMS \
   --exclude updates/1/i386/SRPMS/ \
   --exclude updates/1/i386/debug/ \
   mirrors.kernel.org::fedora/core/ $MIRROR/fedora/core/ \
   >> $OUTFILE

You'll notice that most of the arguments are exclude statements. I don't need to sync any of the source rpms, so I'll leave those out.

Now make that file executable and execute it.

$ chmod +x mirror-script.sh
$ ./mirror-script.sh

It will take some time to download all of those files. Come back in an hour or two. you can use 'tail -f mirror-output-.txt' to watch the progress.

Once you're done, try browsing to http://[YOUR_SERVER_NAME]/mirror to check that it works. (Assuming your http dæmon is up and running.)

Step Two: Automating the mirror

I made a file called $HOME/.crontab that contains the line:

15 1 * * * /space/mirror/mirror-script.sh

You can edit this file all you want. This will run the script at 1:15am every day. Then I ran the crontab program to make this a cron job:

$ crontab ~/.crontab

'crontab -l' tells you what jobs the cron dæmon will do for a particular user.

$ crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/home/hal/.crontab installed on Sun Jan 25 15:24:10 2004)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 ...
15 1 * * * /space/mirror/mirror-script.sh

Step Three: Making a CD copy of updates. (optional)

This should burn all contents of the update directory onto a CD, assuming cdrecord is setup (look at /etc/cdrecord.conf).

$ mkisofs -r -J \
  -V "Fedora Updates as of `/bin/date +%Y-%m-%d`" \
  /space/mirror/fedora/core/updates/1/i386/ \
  | cdrecord -v -

That should be it.

Step four: Setting up repositories.

Now, (as root) edit your /etc/yum.conf file on each machine you want to look at the mirror

[base]
name=Fedora Core $releasever - $basearch - Base
baseurl=http://[YOUR_SERVER_NAME]/mirror/fedora/core/1/i386/os/
gpgcheck=1
 
[updates-released]
name=Fedora Core $releasever - $basearch - Released Updates
baseurl=http://[YOUR_SERVER_NAME]/mirror/fedora/core/updates/1/i386/
gpgcheck=1

Then do an update just to check it out:

# yum update

Responses

yum-arch?

From: "Robert P. J. Day" <------@----------.--->
Date: Tue, 10 Feb 2004 09:14:30 -0500 (GMT-05:00)
To: "Hal Canary" <---@---.-------.----.--->
Subject: fedoranews: setting up a local yum repository

if you're explaining how to set up a local yum repository, don't you
somewhere have to run "yum-arch" to create the proper structure?  or
am i misunderstanding the purpose of that article?  thanks.

rday

If you get the headers/ directory with rsync, then yum-arch is redundant.

using an update cd

From: Curtis Rempel <------@-----.--->
Date: Fri, 20 Feb 2004 09:36:48 -0700
To: "Hal Canary" <---@---.-------.----.--->
Subject: re: How to make a local yum repository mirror

Hi,

I enjoyed your article on FedoraNEWS.ORG and got everything going - I
basically had the right idea but your article solidified things for
me.  Thanks!

One question - now that I've got all the updates on CD, I'm having
some trouble applying them to a machine w/o network access.  I've
mounted the CD and can see all the updates and I modified the
/etc/sysconfig/rhn/sources on that machine to point to /mnt/cdrom for
updates-released (i.e. the line is: yum updates-released /mnt/cdrom),
however, when I click on the update icon, it complains it cannot
access the updates: "The applet has been unable to access the
following information sources in its last attempts: updates-released @
/mnt/cdrom"

What little detail am I missing here?

Thanks!

Curtis

I've heard that yum has problems with using local files. Try starting httpd and linking to those files from /var/www:

sudo /sbin/service httpd start
sudo ln -s /mnt/cdrom /var/www/update

Then point yum at the http://localhost/update url.

(c) 2004 Hal Canary.
Verbatim copying of this article is permitted in any medium, provided this notice is preserved.
Responses are the property of the responders.

file modification time: 2004-02-20 18:30:44