HTTP Anti-Virus Proxy

HAVP has good specs and seems well structured, but it is the first piece of software I've seen in ages that requires mandatory locking. Most software that needs file locking either uses advisory locking in a rigorous manner, or implements locking on its own.

It is <b><i>very</i></b> inconvenient to have to create a new filesystem with mandatory locking enabled. Even a loopback file system is a pain, because it has to be pretty big (i.e., must be the max ever likely to be needed), and the space dedicated cannot be used for anything else.

Why was HAVP written to require it? Was it developed on a system that had it by default? Is it a C++ class/library (e.g. IOSTREAM) idiosyncrasy? How hard would it be to replace it with advisory locking (for example)?

Finally, do I really need it? I want to use HAVP with libclamav (or clamd if necessary).

Thanks
Paul Kosinski

If you read the design (http://www.server-side.de/documentation.htm) and understand the differences in locking, you will get it..

Locking is there to not let the scanner finish before we have downloaded file fully. Now if you look at advisory locking, how are you supposed to use it with external programs? They know nothing about HAVP process locks. Mandatory locking forces the OS itself to stop other processes from reading it.

Changing to advisory locking with libclamav could be possible because it is used internally in HAVP, but whats the point? Actually it is this simple to add mandatory locking support anywhere you want: "mount -o remount,mand /" - now you can use it anywhere you want on your root filesystem without even booting. I wonder why I haven't added this to documentation earlier.

Cheers,
Henrik

Henrik, your projeckt is very good but... but there is no use of your programm without mandatory locks.. i tried to use it under FreeBSD... useless.. couse users get empty pages.. instead of content.. 'couse i am not able to use mandatory locks under FreeBSD. and.. I did not find any such a system under FreeBSD.. that's a pity..
any comments?

sorry... forgot to mentioned... FreeBSD does not support MANDATORY LOCKING... ((((

HAVP 0.80 does not need mandatory locking on freebsd. But you loose the dynamic scanning features without it.

And really, it's not that hard to install a new Linux or Solaris box, is it?

Also you can try DansGuardian alpha version, it has virus scanning features and should work on FreeBSD. Nobody is forcing you to use HAVP on FreeBSD.

Cheers,
Henrik

That's what I was afraid of, that HAVP used mandatory locking so that it can lock files against simultaneous access by other programs that it has no knowledge of (and that might even be closed source and therefore impossible to analyze).

While it is clever to hard-lock the tail of the temp file to allow to scanner to operate in parallel, how can it be guaranteed that an arbitrary (closed source) scanner doesn't use O_NONBLOCK and abort a file scan if READ returns with EAGAIN? It might well do this to avoid freezing up when virus-scanning in batch mode.

Also, portability is reduced. Not to mention the BSDs, according to www.mjmwired.net/kernel/Documentation/mandatory.txt, SunOS 4.1.x, Solaris 2.x and HP-UX 9.x all behave differently in this regard

Finally, although the "mount -o remount,mand /" hack is tempting, and only files that have been treated to "chmod g-x g+S" are then supposed to be subject to mandatory locking, I am very reluctant to change the behavior of the entire file system on my production firewall, which has been running nicely for 488 days. (I'm not even sure if my custom built kernel supports it.)

I guess I'll have to see if I can use clamav without (mandatory) locking.

While you have some points, I don't really get why you have a problem? Not trying to be too harsh, but you sound a somewhat too theoretical unix geek.

First of all, if this works with all the 8 scanners that HAVP supports, why would there suddently be a differently behaving one?

Portability, who would _really_ use HAVP on HP-UX, Aix, BSDs etc? You don't even have many scanners for those platforms? 0.80 may even compile with those (FreeBSD tested), but it will not have the dynamic scanning without mandatory locking.

If your firewall is really on some production use and not at home, I wouldn't even install anything there (I don't understand why any software is installed on firewall, bad security). Otherwise, congrats for your uptime, but I don't really care..

Cheers,
Henrik

I guess that's due to having far more experience writing software (mainly compilers and operating systems) in a research/experimental environment (IBM Research) than my mere 8 years of commercial Unix software development (video servers at DEC and telephone switches at a startup bought by Lucent). In any case, I prefer to analyze the possible problems before the software is put into service.



I don't know how these scanners all behave, but if I were writing a scanner, I wouldn't have it hang on a locked file, as that would mean that daily batch scans would be difficult to manage. (I think most Windows scanners skip locked files, given Windows excessive use of locking.)




According to their website, ClamAV has been ported to at least FreeBSD, OpenBSD, AIX, OSF, Solaris, MacOS X, BeOS, and MS Windows (native, Interix, Cygwin, cygwin.dll based). And some people swear by FreeBSD as being more secure than Linux.




Firewall is a SOHO setup: I don't run the X server, or much else, and run whatever services non-root (dedicated user and group), or privilege separated, and chrooted when possible. I know that ideally there should be nothing else on the FW computer, but that would cost additional hardware, energy, and most important, office space and infrastructure.




Uptime is more a matter of trying to set a record than practicality: it ran 3 months "lights out" (long trip, VPN access), and survived at least 3 short power outages (UPS backup). So now I'm trying to see how long it can go, which means I don't want to make any far-reaching changes, like changing the root FS.

Given that, it turns out that I have an unused swap partition on the second drive. Maybe I'll build a separate FS there and try using "mount -o mand" on it.



Likewise, Paul

Might be true on Windows, but I haven't encountered this on any UNIX-platform. Wouldn't using non-blocking I/O to read files be more expensive (poll required?) and less portable? I'm not that experienced, but I wouldn't use it for a scanner. Maybe alarm call for a long timeout for read. I think there just isn't any case to worry about mandatory locked files on UNIXes, that would justify the extra calls.

Also this means all the n^2 other utilities that read files should do non-blocking, "just in case".

Cheers,
Henrik

As I understand O_NONBLOCK, it merely says that a read() operation (for example) won't block waiting for data, but rather return the EAGAIN error. You can wrap a simple loop around the read if you want to have it behave as "normal". For a <b><i>local</i></b> file read, I suspect that this wouldn't slow things down significantly, as EAGAIN wouldn't occur (much).

Usually, O_NONBLOCK is used for network I/O, in connection with select() when you want to serve a bunch of sockets from one process.

This is different from O_ASYNC, which uses the SIGIO signal to give truly asynchronous I/O. (I presume -- I've never used it.)

Windows is a different kettle of fish (rather smelly), since it seems to use mandatory locking everywhere, making things like backup difficult. I don't think most *nix utilities would have to use non-blocking read unless they performed critical functions like backup (or virus scanning) on large sets of files which were likely to have some that were mandatorily locked. So far, *nix tends not to have this.

Best wishes,
Paul

So, I created an EXT3 FS in a unused small partition I found, mounted it with "-o mand" and am now using it for the files HAVP needs to hard-lock.

Mandatory locking seems already to be in my 2.4.x kernel's EXT3 support by default, (I guess it isn't a build option after all), and there have been no stability problems in the few days I have been using it.

In general, I am impressed with HAVP! (I am using it between my centralized Privoxy server and the Internet.)

Best wishes,
Paul Kosinski