HTTP Anti-Virus Proxy

Official HAVP Support Forum
Registration disabled, I'm tired of spambots. E-mail havp@hege.li if you have questions.
HAVP project is pretty much frozen/abandoned at this time anyway.


All times are UTC + 2 hours [ DST ]




Post new topic Reply to topic  [ 12 posts ] 
Author Message
PostPosted: 05 Apr 2006 11:56 
HAVP 0.78 seems to work great for ftp servers accessed through a web browser.
But when I use the command "apt-get" (Debian Linux platform) to access servers such as ftp.uk.debian.org and ftp.fi.debian.org the downloads timeout at between 60% and 80% complete.

I get the following messages

Error reading from server - read (104 Connection reset by peer)
&
403 Server down from HAVP

the havp.log says
05/04/2006 09:03:07 (127.0.0.1) Could not read server body (ftp.fi.debian.org:21)
05/04/2006 09:04:41 (127.0.0.1) Could not read server header (ftp.uk.debian.org:21)

Any ideas as to what might be causing this behaviour?
and more importantly any ideas on how I can fix it?

Regards

Os


Top
  
 
 Post subject:
PostPosted: 05 Apr 2006 13:05 
Offline
HAVP Developer

Joined: 27 Feb 2006 18:12
Posts: 687
Location: Finland
Seems to be a problem with apt-get sending HTTP/1.1 pipelined requests. HAVP has some problems with them. I'll fix this is next version..

You can fix apt-get yourself by adding this line to /etc/apt/apt.conf:

Acquire::http::Pipeline-Depth 0;

Cheers,
Henrik


Top
 Profile  
 
 Post subject:
PostPosted: 05 Apr 2006 16:17 
Thanks for the quick response

Alas this doesnot seem to solve the problem for me, possibly because it is using the ftp method and not the http method.

Strangely if I change the ftp_proxy environment variable from http://localhost:8080 to ftp://localhost:8080 apt-get cannot even connect to the ftp servers.

Os


Top
  
 
 Post subject:
PostPosted: 05 Apr 2006 16:22 
Offline
HAVP Developer

Joined: 27 Feb 2006 18:12
Posts: 687
Location: Finland
It did fix it for me, there should be no reason it not to work after the pipeline setting. Though you seem to have more problems.. if you get 403 Server down, then it's possible something is down ;) I didn't try 0.78, but the newly released 0.79.. try if it fixes something.

And naturally ftp://localhost:8080 doesn't work. Proxies are always defined with http:// because they talk HTTP.

Oh and just to make sure.. you are using PARENTPROXY with HAVP, right? :D

Cheers,
Henrik


Top
 Profile  
 
 Post subject:
PostPosted: 05 Apr 2006 22:46 
I am using squid as the PARENTPROXY (I would happily use something different if I could find one that proxied ftp).

I have now upgraded to HAVP 0.79

ftp_proxy & FTP_PROXY are both set to http://localhost:8080/ in /etc/environment

I have added the line Acquire::http::Pipeline-Depth 0; to /etc/apt/apt.conf

Unfortunately the problem still remains (i.e. ftp transfer aborts halfway through), however the error message is a little different

"Error reading from server. Remote end closed connection"

I have tried with several different debian ftp servers.

Any other suggestions?

By the way, I tried usine SmartFTP from a windows host through the proxy using CONNECT. It generates an error in HAVP - "Invalid request from browser"

Thanks for your help

Os


Top
  
 
 Post subject:
PostPosted: 06 Apr 2006 06:36 
Something interesting from the havp logs

06/04/2006 04:24:05 127.0.0.1 ftp://ftp.de.debian.org/debian/dists/te ... ckages.bz2 Virus: BZip.ExceededFileSize

06/04/2006 04:24:20 127.0.0.1 ftp://ftp.de.debian.org/debian/dists/st ... ackages.gz Virus: GZip.ExceededFileSize

Does this indicate a virus has been detected? and could this potentially be the cause of the incomplete ftp download?

havp settings:
RANGE false
KEEPBACKBUFFER 200000 (The package files are all larger than this)
KEEPBACKTIME 5
MAXSCANSIZE 0
TRICKLING 30

Os


Top
  
 
 Post subject:
PostPosted: 06 Apr 2006 09:26 
Offline
HAVP Developer

Joined: 27 Feb 2006 18:12
Posts: 687
Location: Finland
Hehe yes that is the reason.. ClamAV is blocking files that exceed limits by default in 0.79.

You should change CLAMBLOCKMAX to false.

# Should we block files that go over maximum archive values?
# CLAMBLOCKMAX true

# Scanning limits for archives (filesize = MB):
# CLAMMAXFILES 1000
# CLAMMAXFILESIZE 10
# CLAMMAXRECURSION 8
# CLAMMAXRATIO 250

Cheers,
Henrik


Top
 Profile  
 
 Post subject:
PostPosted: 06 Apr 2006 13:53 
It works!

During an apt-get update the % complete counter for each package source is misleading - maybe it is calculated on time and estimated data rate. It appears to not reach anywhere near 100% before just disappearing. Maybe that this is due to the certain amount of data that HAVP holds back before sending in a burst.

If I whitelist the server the % complete counter works correctly, but the same number of bytes are transferred in total as before.

One thing I did notice is that I can't have

ftp.*.debian.org/debian/dists/*

in the whitelist as it generates an error -

06/04/2006 11:30:58 URLList - Too many wildcards in ftp.*.debian.org
06/04/2006 11:30:58 URLList invalid Domain: ftp.*.debian.org

I also tried usine SmartFTP from a windows host through an
"HTTP proxy using a CONNECT tunnel". It generates an error in HAVP - "Invalid request from browser". Any ideas as to why?

Thanks for all of your help

Os


Top
  
 
 Post subject:
PostPosted: 06 Apr 2006 13:58 
Offline
HAVP Developer

Joined: 27 Feb 2006 18:12
Posts: 687
Location: Finland
Osiris X wrote:
One thing I did notice is that I can't have

ftp.*.debian.org/debian/dists/*

in the whitelist as it generates an error -

06/04/2006 11:30:58 URLList - Too many wildcards in ftp.*.debian.org
06/04/2006 11:30:58 URLList invalid Domain: ftp.*.debian.org


I guess thats some limitation in the whitelist, maybe Christian will have a look as he did it.

Quote:
I also tried usine SmartFTP from a windows host through an
"HTTP proxy using a CONNECT tunnel". It generates an error in HAVP - "Invalid request from browser". Any ideas as to why?


CONNECT is meant for SSL requests, and your HAVP isn't compiled with the support. And anyways, all proxies allow only ports 443 and 563 by default with CONNECT. You are trying to use port 21 for FTP. You cannot scan the files like this anyway.

In any case, use HTTP rather than FTP for the mirrors. It is much more efficient. :)

Cheers,
Henrik


Top
 Profile  
 
 Post subject:
PostPosted: 06 Apr 2006 17:01 
Offline
HAVP Maintainer

Joined: 27 Feb 2006 19:08
Posts: 62
The domain of the white/blacklist is limited to one wildcard at the beginning.

*.debian.org would be ok

Also only complete domain names are supported.

e.g.

*ebian.org/

will not blacklist debian.org but ebian.org, xyz.ebian.org ...

I will try to allow "middle" wildcard in the next version.


Top
 Profile  
 
PostPosted: 25 Nov 2009 12:41 
Offline

Joined: 21 Nov 2009 13:30
Posts: 1
I need to set up a page where a client can login with a username password and access their FTP folder.? They also need an option to upload to their ftp folder. What's the best/easiest way to configure something like this?
_________________
external keyword tool ~ keyworddiscovery.com ~ keycompete.com ~ compete.com ~ webmasterworld.com


Last edited by eakinasila on 30 Nov 2009 08:16, edited 1 time in total.

Top
 Profile  
 
PostPosted: 26 Nov 2009 11:16 
Offline

Joined: 23 Apr 2008 09:36
Posts: 101
Hi eakinasila!

What are the credentials? With FTP-Client or explorer to handle the FTP file transfers?
Should it be possible to download files with explorer (perhaps with virus filter)?
In this case I would prefer apache2 and proftpd (frox for virus filter), but this is a matter of taste.

regards


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 

All times are UTC + 2 hours [ DST ]


Who is online

Users browsing this forum: Google [Bot] and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group