Subject: Summary: Uploading PDB files to a CGI script

From: "Robert Fraczkiewicz" <[email-masked]>
Date: Fri, 26 Jun 1998 11:34:33 -0500

Great "thank you" to all who replied to my requests (copies enclosed below)
regarding CGI scripts. Duplicate text has been removed for clarity.

Robert Fraczkiewicz
University of Texas Medical Branch

> Dear Netters:
>
> We would like to start a web service where people would upload atomic
> coordinates in PDB format through an HTML form (e.g., by pasting them
> to TEXTAREA, or direct upload of a disk file via <INPUT TYPE="file">
> tag) and get results  back to their browsers. The form and CGI script work
> wonderfully as long as the PDB data size stays below ~20000 characters, which
> is insufficient for any practical application. Above this number the server
> times out without even touching the CGI script.
>
> Is there a limit on the amount of data a web client can transfer to
> a web server through forms? Personally, I don't believe that. I am
> sure many of you had faced similar problem. Any hints will be greatly
> appreciated and summarized on the list.
>
> With best regards,
> Robert Fraczkiewicz
> University of Texas Medical Branch
>

-----------------------------------------------------------------------------

On Jun 23, 12:02pm, Bruce Luxon wrote:
> Subject: Re: CCL:Uploading PDB files to a CGI script
> Robert,
>
> What did you write the cgi-script in? I have some in .c that had this
> problem until I explicitly increased the number of chars in the
> message array in main.  I've never seen a web server-based limit on
> transfers - only software imposed limits.
>
> Bruce
>
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>   Bruce A. Luxon, Ph.D
>   Associate Professor
>   Sealy Center for Structural Biology
>   Dept. of Human Biological Chemistry & Genetics
>   University of Texas Medical Branch
>   Galveston, TX   77555-1157
>   (409)747-6802; Fax (409)747-6850
>   http://www.hbcg.utmb.edu/  http://www.nmr.utmb.edu/
>   mailto:[email-masked]
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>-- End of excerpt from Bruce Luxon

------------------------------------------------------------------------------

On Jun 23,  1:15pm, M Dominic Ryan wrote:
> Subject: Web forms
> The most reliable for me has been to get the browser to dump a file.  You
need
> to use the multipart form mechanism.  I think the textarea mode is limited
> by shell args limits.  We transfer pretty large files this way.
> __
> M. Dominic Ryan   (610)-270-6529     SmithKline Beecham Pharmaceuticals
> Internet:  [email-masked]     King of Prussia, PA
>-- End of excerpt from M Dominic Ryan

------------------------------------------------------------------------------

On Jun 23, 12:42pm, TJ O'Donnell wrote:
> Subject: RE: your CCL post
> Hi Robert-
>
> There is no inherent limit on the amount of data that
> can be sent through forms.  Whenever I expect to xfer
> a large amount of data, I do it via the <INPUT TYPE=FILE..>
> mechanism.  I have sucessfully transferred and processed
> files as large as several hundred Mbytes this way.
> The timeout is your problem, I think.  When I xfer
> large data files via an INTRAnet, it hardly ever fails.
> When I do it over a 28.8 modem, it hardly ever
> succeeds - the server times out.  It is simply a matter
> of making your server more patient.  You can configure
> this on the server side.  How you do it depends on which
> server you're using.
> --
> [email-masked]
> http://www.eecs.uic.edu/~tj
>-- End of excerpt from TJ O'Donnell

-----------------------------------------------------------------------

On Jun 23,  3:07pm, Jesus Castagnetto wrote:
> Subject: Re: CCL:Uploading PDB files to a CGI script
> Hello Robert,
>
> If you are using POST as the method to send the form, in theory
> there should not be a hard limit on the size, in practice
> the limit is only dependent on the  browser/OS combination.
>
> An alternative that will not suffer from this problem (AFIAK), is
> to use the <INPUT TYPE="file"> element so the user can send the
> file w/o needing to copy/paste the whole thing. You will need to
> modify your CGI to handle this. More info on this particular
> element can be found in the Netscape reference site. This is a tag
> that is understood by Netscape 3.x or higher, and Internet Explorer 4.x
>
> Good luck.
>
> =====
>      Jesus M. Castagnetto, Ph.D. - [email-masked]
> TSRI, Dept. of Molecular Biology |"Activation Energy: The useful
> 10550 N. Torrey Pines Rd. MBP326 | quantity of energy available
>        La Jolla, CA 92037        | in one cup of coffee."
>        Phone: 619-784-8582           Fax: 619-784-2289
>     E-Paper: http://www.ch.ic.ac.uk/ectoc/echet96/papers/003/
>         Lab: http://www.scripps.edu/research/metallo/
>   MetalloDB: http://metallo.scripps.edu/index.html
> Pilot Stuff: http://www.geocities.com/ResearchTriangle/Lab/1059/
>
>-- End of excerpt from Jesus Castagnetto

------------------------------------------------------------------------------

On Jun 24, 11:17am, Malcolm Gillies wrote:
>
> In general, no. I'd suspect network problems causing the HTTP connection
> to be prematurely broken. This may be caused or exacerbated by shortcomings
> in the TCP/IP implementation on the client and/or server machines. It's
> also possible that the TCP timeouts as defined by the operating system
> are too short.
>
> The solution to this problem depends a great deal on which operating system
> and web server software you are running.
>
> I would begin by trying to obtain some more detailed error messages from
> the web server -- it may be necessary to reconfigure it to produce more
> detailed error logging.
>
> good luck,
>
> Malcolm
> --
> Malcolm Gillies <[email-masked]>
> PhD student, Computational Medicinal Chemistry
> Dept Medicinal Chemistry, Faculty of Pharmacy,
> Utrecht University, The Netherlands
>-- End of excerpt from Malcolm Gillies

---------------------------------------------------------------------

On Jun 24,  2:54am, Paddy Kane wrote:
> Subject: Re: CCL:Uploading PDB files to a CGI script
>  Hi,
>
>  Unfortunately, I don't have an answer to your question, but I would be
interested in using the service when you get it running. Please post a message
to the CCL mailing list when you are ready to offer this service.
>
>  Kind rgds,
>  Paddy.
>
> ---
> Paddy Kane
> School Of Chemical Sciences
> Dublin City University
> Glasnevin
> Dublin 9
> Ireland.
>
> E-mail: [email-masked]
>-- End of excerpt from Paddy Kane

---------------------------------------------------------------------------

Later I have realized that the problem is somewhere else. Second posting
and replies to it are included below. Special thanks to Malcolm Gillies,
a man with genuine curiosity, who contributed his time and effort to
solve this puzzle.

On Jun 23,  6:27pm, Robert Fraczkiewicz wrote:
> Subject: CCL:CCL:Uploading PDB files to a CGI script
>
>
> Dear Netters:
>
> Thanks to all who replied to my inquiry. In my original posting I have
> reported a web server timing out when large files are uploaded via an
> HTML form.
>
> After some wrestling with the script I have figured out that the web
> server is basically OK and can accept data of any size. The problem is
> with Netscape browsers: server times out because browsers cannot handle
> large amounts of data from CGI programs. Let me present an example:
>
> This simple HTML form will let you upload any readable disk file into
> a CGI program:
>
> <HTML>
> <HEAD>
> <TITLE>Simple form</TITLE>
> <BODY>
> <FORM ENCTYPE="multipart/form-data" METHOD="POST" ACTION="your_CGI_script">
>  Enter PDB file name:<BR>
> <INPUT TYPE="file" NAME="pdbdata" SIZE=70></INPUT>
> <P>
> <INPUT NAME="s_submit" TYPE="SUBMIT" VALUE="SUBMIT"></INPUT>
> <INPUT NAME="s_reset"  TYPE="RESET"  VALUE="RESET FORM"></INPUT>
> </FORM>
> </BODY>
> </HTML>
>
> The CGI program I used takes the data and returns it to the browser
> as plain text (compile it and replace "your_CGI_script" by the executable
> pathname):
>
>
> #include <stdio.h>
> #include <stdlib.h>
>
> main()
> {
>     unsigned long i, n=0;
>     char *cl;
>
>     printf("Content-type:text/plain\n\n");
>     cl=getenv("CONTENT_LENGTH");
>     if(cl) n=atoi(cl);
>
>     for(i=0; i<n; i++) putchar(getchar());
>     putchar('\n');
>     fflush(stdout);
> }
>
>
> This works fine for small input files. For PDB data the threshold correspond
> to about ~300 atom coordinates. Submitting larger file locks the browser
> and causes the server to time out. When the "putchar" function is removed
> from the "for" loop (i.e., no data are returned to the browser) everything
> works fine. Same when the loop is slightly modified to output characters
> for "i" below certain threshold (10000 works here). Clearly, there is a
> miscommunication between the CGI program and the browser.
>
> I tried Netscape 3.04 and 4.02 on SGI, Netscape 3.0 on PC and Netscape 4.02
on
> Apple Macintosh. All of them show the same symptoms. Is it a bug, or am I
doing
> something wrong? Any hints will be greatly appreciated and summarized on the
> list.
>
> With best regards,
> Robert Fraczkiewicz
> University of Texas Medical Branch
>-- End of excerpt from Robert Fraczkiewicz

-------------------------------------------------------------------------

On Jun 24,  7:47pm, [email-masked] wrote:
> Subject: Re:  CCL:CCL:Uploading PDB files to a CGI script
> What if you did fflush more often?
>
> Bill Ross
>-- End of excerpt from [email-masked]

I tried fflush every 1000, 100 and 10 characters to no avail.

----------------------------------------------------------------------------

On Jun 25,  9:26am, Yongxing Liu wrote:
> Subject: Re: CCL:CCL:Uploading PDB files to a CGI script
> Hi,
>
>  A while ago, I down load a PERL script from the web for uploading files, My
> goal is putting a MD analysis program on the web and let people submit their
> trajactories and do the analysis. I soon found that it is difficult to
> uplaod 1Gig files for the analysis. So I gave up the idea.
>
> I just simply attatch the script and you may test it on your PDB files.
>
> By the way, try to look at
http://www.doe-mbi.ucla.edu/Services/Verify3D.html,
> It seems to me big pdb files works OK to upload to them. Maybe just e-mail
> them and ask what they are doing with their uploading.
>
> Best Wishes,
>
> Yongxing Liu
>
> ----------------------------------------------------------------------------
> ----
> #!/usr/bin/perl
> #
> # File Upload Script        Version 4.01
> # Created by Jeff Carnahan  [email-masked]
> # Created on: 4/8/95        Last Modified on: 08/07/96 11:20
> # Scripts Archive:          http://www.terminalp.com/scripts/
> #
> # Copyright (C) 1996 - Jeffrey D. Carnahan
> #   Copyright Information Can Be found in the attacted README file.
> #
> # ---------------------------------------------------------------------
> # Program Specific Quickie Notes:
> #     * Make Sure The First Line Is Pointing To The Correct Location Of Perl.
> #     * Make Sure This Program is chmodded with the permissions '755'.
> #     * Make Sure You Have The correct information in @refer.
> #
> #  Version:  Time Stamp:        History:
> #  ____________________________________________________________________
> #
> #      4.00  08/04/96 23:16     Revision Histroy Now Intact.
> #      4.00  08/04/96 23:16     Security hole regarding '../' paths
> #                               fixed.  Thanks to: Rus Berrett.  Mime
> #                               type error fixed.  Thanks to: Bob Stewart.
> #      4.01  08/07/96 11:20     Typo fixed in &NoOpen.  Thanks to Marco
> #                               Dings.
> #
> # ---------------------------------------------------------------------
> # Configurable Options Follow:
> #
> $userid        = "1003";
> $groupid       = "100";
> $url           = "http://linus.chem.wesleyan.edu/yliu/upload";
> $path          = "/usr3/people/yliu/public_html/upload";
> $overwrite     = "1";
> $index_rename  = "1";
> $rename_val    = "ppp";
>
>         @refer = ('www.terminalp.com','terminalp.com','207.86.130.85');
>
>
> #
> # End of Configurable Options.
> # ---------------------------------------------------------------------
> # ---------------------------------------------------------------------
> # Don't Modify Anything Past This Line Unless You Know What Your Doing!
> # ---------------------------------------------------------------------
> # ---------------------------------------------------------------------
> #
>
> &CheckHost;
>
> $| = 1;  # Set This To 1, so Perl doesn't Buffer the Data... =)
>
> if ($ENV{'REQUEST_METHOD'} eq "POST") {
>    read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
> } elsif ($ENV{'REQUEST_METHOD'} eq "GET") {
>         $buffer=$ENV{'QUERY_STRING'};
>   }
>
> $buffer =~ /^(.+)\r\n/;
> $bound = $1;
> @pairs = split(/$bound/, $buffer);
> print "Content-type: text/html\n\n";
>
> if (@pairs != 4) {
>     print "<HTML><HEAD><TITLE>Sorry!</TITLE></HEAD><BODY
BGCOLOR=\"#FFFFFF\">";
>     print "<H1>Sorry.</H1>";
>     print "<p>Sorry, your browser doesn't support the HTML 3.0 codes needed
> to upload files.";
>     print " Please try again with a browser that supports these functions,
> (Such as <a href=\"";
>     print
> "http://home.netscape.com/comprod/products/navigator/version_2.0/index.html\
> ">Netscape 2.0</a>.)\n";
>     print "</BODY></HTML>\n";
>     exit;
> }
>
>
> $agent = $ENV{'HTTP_USER_AGENT'};
> @var = split(/\r\n/,$pairs[1]);
> $filename = $var[3];
> if (($filename !~ /[A-Za-z_\-0-9\.]/) || ($filename =~ /\.\./)) {
>         print "<HTML><HEAD><TITLE>Sorry!</TITLE></HEAD><BODY
> BGCOLOR=\"\#FFFFFF\">";
>         print "<H1>Sorry.</H1>";
>         print "<p>Sorry, the file you want to upload contains illegal
> characters.  Please only use characters that include any combinations of
> letters, underscores (<i>_</i>), hyphens, numbers, or periods.";
>         print "<br>Please select another filename and try again.\n";
>         print "</BODY></HTML>\n";
>         exit;
> }
> if ($index_rename) {
>     $filename =~ s/index.html/$rename_val/g;
> }
>
> if ($pairs[2] =~ /Content-Type:/) {   # includes a MIME definition
>     $pairs[2] =~
s/^\r\n.+filename.+[^\w\.\%-]([\w\.\%-]+)"\r\n(.*\r\n)\r\n//;
>     $pairs[2] =~ s/\r\n$//;
> } else {
>     @var = split(/\r\n/,$pairs[2]);
>     $pairs[2] = $var[3];
> }
>
> if ($overwrite) {
>     open (OUTPUT, "> $path/$filename" ) || die &NoOpen($!);
>     print OUTPUT $pairs[2];
>     close (OUTPUT);
>     chmod (0644, "$path/$filename");
>     chown ($userid, $groupid, $file);
> } else {
>     if (-e "$path/$filename") {
>         print "<HTML><HEAD><TITLE>Sorry!</TITLE></HEAD><BODY
> BGCOLOR=\"#FFFFFF\">";
>         print "<H1>Sorry.</H1>";
>         print "<p>Sorry, the file you want to write to is already in use,
> and you are not allowed to overwrite it.";
>         print " Please select another filename and try again.\n";
>         print "</BODY></HTML>\n";
>         exit;
>     } else {
>         open (OUTPUT, "> $path/$filename" ) || die &NoOpen($!);
>         print OUTPUT $pairs[2];
>         close (OUTPUT);
>         chmod (0644, "$path/$filename");
>         chown ($userid, $groupid, $file);
>     }
> }
>
>
>
> print "<HTML><HEAD><TITLE>Thanks!</TITLE></HEAD><BODY BGCOLOR=\"#FFFFFF\">";
> print "<h1>Thanks!</h1>\n";
> print "<p>Thanks for uploading the file: $filename.  It is now known under
> the URL of: <a href=\"";
> print "$url/$filename\">$url/$filename</a>. If you would like to upload
> another file, click <a href=\"";
> print "$ENV{'HTTP_REFERER'}\">here</a> to return to the previous page.\n";
> print "<p><hr size=1>\n";
> print "<font size=-1><center><a
> href=\"http://www.terminalp.com/scripts/\">Obtain This Script</a> - <a
> href=\"mailto:jeffc\@terminalp.com\">jeffc\@terminalp.com</a></center></font
> >\n";
> print "</BODY></HTML>\n";
>
>
> sub NoOpen {
>     print "<HTML><HEAD><TITLE>Error!!</TITLE></HEAD><BODY
> BGCOLOR=\"#FFFFFF\">\n";
>     print "<H1>Error!</H1>\n";
>     print "<p>There was an error: <b><pre>@_[0]</pre></b>\n";
>     print "<br>Please make sure that the directory the file should be
> written to, <i>$path</i>, has been chmodded with the premissions
<i>777</i>.\n";
>     print "<br>Or, if you are trying to overwrite a existing file, check to
> see that the existing file has been chmodded with the premissions
> <i>666</i>.\n";
>     print "<p><hr size=1>\n";
>     print "<font size=-1><center><a
> href=\"http://www.terminalp.com/scripts/\">Obtain This Script</a> - <a
> href=\"mailto:jeffc\@terminalp.com\">jeffc\@terminalp.com</a></center></font
> >\n";
>     print "</BODY></HTML>\n";
> }
>
>
> sub CheckHost {
>     if ($ENV{'HTTP_REFERER'}) {
>         foreach (@refer) {
>             if ($ENV{'HTTP_REFERER'} =~ /$_/i) {
>                 $ok = 1;
>     	        last;
>         }   }
>     } else { # If There isn't a refering URL, then we'll assume it's ok.
>         $ok = 1;
>     }
>     if ($ok) { # It Checked, We Can Continue
>     } else {   # Nope... Someone's trying to hack their way in.
>         print "Content-type: text/html\n\n";
>         print "<HTML><HEAD><TITLE>Sorry!</TITLE></HEAD><BODY
> BGCOLOR=\"#FFFFFF\">";
>         print "<H1>Sorry.</H1>\n";
>         print "<p>Sorry, you are not allowed to post information to this
> server from <a href=\"";
>         print "$ENV{'HTTP_REFERER'}\">$ENV{'HTTP_REFERER'}</a>.  Please Use
> a document that is already";
>         print " on this server to upload a file.\n";
>         print "</BODY></HTML>\n";
>         exit;
>     }
> }
>
>-- End of excerpt from Yongxing Liu

------------------------------------------------------------------------------

On Jun 25,  4:07pm, Malcolm Gillies wrote:
> Subject: Re: CCL:CCL:Uploading PDB files to a CGI script
> Dear Robert,
>
> It looks like a buffering problem to me.
>
> While I reproduced your problem using the HTML and C included in
> your post on the setup here (Irix 6.2/NCSA httpd 1.5.2/Netscape 3.04),
> when I used the C code included below, it works fine.
>
> The difference is that I read and buffer all input before writing
> to stdout. Not quite sure why this prevents what looks like a dead-lock.
>
> #include <stdio.h>
> #include <stdlib.h>
>
> main()
> {
>     unsigned long i, n=0;
>     char *cl, *buf;
>     int c;
>
>     printf("Content-type:text/plain\n");
>     cl=getenv("CONTENT_LENGTH");
>     if(cl) {
>         n=atoi(cl);
>         printf("Content-length:%d\n",n);
>         printf("\n");
>
>         if(buf = malloc(n)) {
>             fread (buf, n, 1, stdin);
>             fwrite (buf, n, 1, stdout);
>         } else
>             printf("Out of memory\n");
>     } else {
>         printf("\n");
>         printf("I'm not playing with you!\n");
>     }
>
>     fflush(stdout);
> }
>
> cheers,
>
> Malcolm
>-- End of excerpt from Malcolm Gillies

-----------------------------------------------------------------------------

On Jun 25,  6:56pm, Malcolm Gillies wrote:
> Subject: Re: CCL:CCL:Uploading PDB files to a CGI script
> > You are right: buffering all input into a memory chunk created
> > by program prevents the deadlock. It works fine on my system.
> > I am going to submit a bug report to Netscape.
>
> Have you checked any other browsers?
>
> I suspect that it's the cgi that's deadlocking, and that it's
> buffering in the httpd which is causing it. But that's just a guess.
> I can't find any relevant info via Altavista or Dejanews.
>
> cheers,
>
> Malcolm
>-- End of excerpt from Malcolm Gillies

------------------------------------------------------------------------

On Jun 25,  7:08pm, Malcolm Gillies wrote:
> Subject: Re: CCL:CCL:Uploading PDB files to a CGI script
> Robert,
>
> >> Have you checked any other browsers?
> >
> > Yes. I tried Netscape browsers: 3.04, 4.02 and 4.05 for SGI IRIX,
> > 4.02 for Macintosh and 3.0 for Windows'95. Besides, I use different
> > server than you: Netscape Commerce httpd and stil reproduce this bug.
> > I have just submitted the bug report.
>
> I'm still suspicious that it's a pathological case inhererent in the
> CGI interface, rather than an actual bug. If I can organise it, I might
> try testing your original code against MS IE.
>
> But I should probably just take it easy ;-)
>
> >> I suspect that it's the cgi that's deadlocking, and that it's
> >> buffering in the httpd which is causing it. But that's just a guess.
> >> I can't find any relevant info via Altavista or Dejanews.
>
> cheers,
>
> Malcolm
>-- End of excerpt from Malcolm Gillies

-------------------------------------------------------------------

Since this problem was always reproducible irregardless of the version
number of Netscape browsers and was also confirmed by other users with
different HTTP server, I have submitted a bug report to Netscape.com.
I will post their answer as soon as I hear from them.

Robert Fraczkiewicz